~azzar1/unity/add-show-desktop-key

.. IVLE - Informatics Virtual Learning Environment

   Copyright (C) 2007-2009 The University of Melbourne

.. This program is free software; you can redistribute it and/or modify

   it under the terms of the GNU General Public License as published by

   the Free Software Foundation; either version 2 of the License, or

   (at your option) any later version.

.. This program is distributed in the hope that it will be useful,

   but WITHOUT ANY WARRANTY; without even the implied warranty of

   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

   GNU General Public License for more details.

.. You should have received a copy of the GNU General Public License

   along with this program; if not, write to the Free Software

   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA

*******************

System Architecture

*******************

IVLE is a complex piece of software that integrates closely with the 

underlying system. It can be considered part web service and part local system 

daemon. Due to the implementation of these parts it is tied to Apache Web 

Server (mainly due to the use of mod_python) and Linux.

Dispatch

========

IVLE uses mod_python_ to allow Python scripts to be called from Apache. We 

register the :mod:`ivle.dispatch` module as the ``PythonHandler`` in the 

associated VirtualHost, allowing us to intercept all HTTP requests to the web 

server.

The :mod:`ivle.dispatch` module is responsible for mapping requests from the 

client to the correct application plugin. Plugins can be specified by placing  

a :file:`*.conf` file into the :file:`/etc/ivle/plugins.d/` directory 

containing lines of the form :samp:`[{plugin_module}#{classname}]`.

.. TODO: Document Plugin Format and Routing Strings

In future, this may be ported to a WSGI (:pep:`333`) based dispatch to allow 

IVLE to be run on web servers other than Apache.

.. _mod_python: http://www.modpython.org/

Templating

----------

IVLE uses the Genshi_ XHTML template system to generate all HTML pages. We

have an inheritance-based "views" system. :class:`BaseView` is a class from

which all views derive.

There are 3 sub-types of :class:`BaseView` (more can be implemented if 

necessary):

* XHTML-Templated

    * browser, console, debuginfo, diff, forum, groups, help, home, logout, 

      settings, subjects, svnlog, tos, tutorial

* Raw byte streaming

    * download, server

* JSON service

    * consoleservice, fileservice, tutorialservice, userservice 

The apps each derive from one of the above.

.. note::

   IVLE used to write its HTML output as a raw stream to an output file, until

   it was refactored to use Genshi. All apps which haven't yet been refactored

   properly were ported to use the "raw byte streaming" view.

.. _Genshi: http://genshi.edgewall.org/

.. _ref-jail:

Jail System

===========

One of the main features of IVLE is it's ability to execute user's code in a 

customised environment that prevents access to other users files or underlying 

file system as well as placing basic resource limits to prevent users from 

accidentally exhausting shared resources such as CPU time and memory.

Trampoline

----------

To each user, it appears that they have their own private Unix filesystem 

containing software, libraries and a home directory to do with what they 

please. This is mainly done by the setuid root program ``trampoline`` which 

mounts the users home directory, sets up the users environment, jumps into the 

user's jail using the :manpage:`chroot(2)` system call and finally drops 

privileges to the desired user and group.

To prevent abuse, ``trampoline`` can only be used by root or one of the uids 

specified when trampoline is built by ``setup.py build`` (defaults to UID 33, 

www-data on Debian systems). Since it's one of two C programs involved in IVLE 

and runs setuid root it is rather security sensitive.

.. seealso:: Source code :file:`bin/trampoline/trampoline.c`

Base Image Generation

---------------------

All user jails share a common base image that contains the files required for 

both IVLE's operation and for executing user code. This base image is 

generated automatically by the ``ivle-buildjail`` script. This then calls the 

distribution dependant details in :mod:`ivle.jailbuilder` module. At present 

we only support building jails for Debian derived systems using 

:program:`debootstrap`.

The contents of the base image contains a few core packages required for the 

operation of IVLE - Python and the Python CJSON and SVN libraries. Other 

options that can be configured in :file:`/etc/ivle/ivle.conf` are the file 

mirror that debootstrap should use, the suite to build (such as hardy or 

jaunty), extra apt-sources, extra apt keys and any additional packages to 

install.

To prevent users from altering files in the base image we change the 

permissions of :file:`/tmp`, :file:`/var/tmp` and :file:`/var/lock` to not be 

world writeable and check that no other files are world writeable.

Finally we make the user dependent :file:`/etc/passwd` and 

:file:`/etc/ivle/ivle.conf` symlinks to files in the :file:`/home` directory 

so that they will be used when we mount a user's home directory.

Mounting Home Directories

-------------------------

To give the appearance of a private file system we need to merge together a 

user's local home directory with the base image.

To achieve this, IVLE uses the *bind mount* feature of Linux, which allows

directories to be accessible from another location in the file system. By

carefully bind-mounting the jail image as read-only and then bind-mounting the

user's :file:`/home` and :file:`/tmp` directory data over the top, we create a

jail with only three bind mounts and at virtually no file system overhead.

.. note::

   IVLE has historically used numerous solutions to this problem, which are

   chronicled here to avoid the same mistakes being made again.

   In the first release of IVLE this was done offline by hard-linking all the

   files into the target directory, but for a large number of users, this

   process can take several hours, and also runs the risk of exhausting

   the number of inodes on the underlying file system.

   The second solution was to use `AUFS <http://aufs.sourceforge.net/>`_ to

   mount the user's home directory over a read-only version of the base on

   demand. This was implemented as part of ``trampoline`` and used a secondary

   program ``timount`` (see :file:`bin/timount/timount.c`), run at regular

   intervals, to unmount unused jails. This used the :const:`MNT_EXPIRE` flag

   for :manpage:`umount(2)` (available since Linux 2.6.8) that only unmounts a

   directory if it hasn't been accessed since the previous call with

   :const:`MNT_EXPIRE`.

   While quite effective, AUFS appeared to cause NFS caching issues when IVLE

   was run as a cluster, and as its inclusion status in future Linux

   distributions is questionable, the developers elected to use the much older

   bind mount feature instead.

Entering the Jail

-----------------

Before running the specified program in the users jail we need to 

:manpage:`chroot(2)` into the users jail and update the processes environment 

so that we have the correct environment variables and user/group ids.

At this stage we also may apply a number of resource limits (see 

:manpage:`setrlimit`) to prevent run away processes (such as those containing 

infinite loops or "fork bombs") from exhausting all system resources. The 

default limits are on maximum address space (:const:`RLIMIT_AS`), process data 

space (:const:`RLIMIT_DATA`), core dump size (:const:`RLIMIT_CORE`), CPU time 

(:const:`RLIMIT_CPU`), file size (:const:`RLIMIT_FSIZE`) and number of 

processes that may be spawned (:const:`RLIMIT_NPROC`).

Unfortunately due to glibc's :manpage:`malloc(2)` implementation being able to 

allocate memory using :manpage:`mmap(2)`, :const:`RLIMIT_DATA` does not 

provide an effective limit on the amount of memory that a process can allocate 

(short of applying a kernel patch). Thus the only way to limit memory 

allocations is by placing limits on the address space, but this can cause 

problems with certain applications that allocate far larger address spaces 

than the real memory used. For this reason :const:`RLIMIT_AS` is currently set 

very large.

.. _ref-python-console:

Python Console

==============

IVLE provides a web based programming console, exposing similar features to 

Python's command line console. It is built around the

:file:`services/python-console` script, which opens up a socket on a random

port to which `JSON`_ encoded chat requests can be made.

A new console is typically launched on demand by the web client to the HTTP

API, which in turn calls the wrapper class :class:`ivle.console.Console` to

start a new console in the user's jail.

Subsequent requests from the same in-browser console connect to the existing

console process. This is achieved by storing a string on the client which

identifies the server address and port. The client then makes requests

through the load balancer, sending this string through to an arbitrary slave

which forwards the request to the identified console.

This means that all slaves need access to all ports on every other slave.

.. _JSON: http://json.org

.. _ref-usrmgt-server:

User Management Server

======================

The **User Management Server** is a daemon responsible for handling privileged 

actions on IVLE and should be launched along with IVLE. It is primarily 

responsible for:

* Creating user jails, Subversion repositories, and Subversion authentication 

  credentials.

* Creating group Subversion repositories.

* Rebuilding Subversion authorization files. 

Communication with the Server is done using the :ref:`Chat Protocol

<ref-chat>`.  To prevent unauthorized use, communication with the User

Management Server requires that a *shared secret* be used to communicate with

the server.  This secret is stored in the `magic` variable in the `[usrmgt]`

section of :file:`/etc/ivle/ivle.conf`.

The User Management Server is called almost exclusively from the 

:mod:`ivle.webapp.userservice` module.

.. seealso:: Source code :file:`services/usrmgt-server`

.. _ref-chat:

Chat Protocol

=============

**Chat** is our JSON_-based client/server communication protocol used in

communicating to :ref:`Python Console <ref-python-console>` processes and

:ref:`User Management Server <ref-usrmgt-server>`.  Since it is JSON-based it

can be called from either Python or JavaScript.

Protocol

--------

The protocol is a fairly simple client/server based one consisting of a single 

JSON object. Before communication starts a shared secret :const:`MAGIC` must 

be  known by both parties. The shared secret is then used to form a 

'keyed-Hash Message Authentication Code' to ensure that the content is valid 

and not been modified in transit.

The client request takes the following form::

    {

        "content": DATA,

        "digest": HASH

    }

where :const:`DATA` is any valid JSON value and :const:`HASH` is an string 

containing the MD5 hash of the :const:`DATA` appended to :const:`MAGIC` and 

then hex encoded.

The server will respond with a JSON value corresponding to the request.

If an error occurs then a special JSON object will be returned of the 

following form::

    {

        "type": NAME,

        "value": VALUE,

        "traceback": TRACEBACK

    }

where :const:`NAME` is a JSON string of the exception type (such as 

'AttributeError'), :const:`VALUE` is the string value associated with the 

exception and :const:`TRACEBACK` is a string of the traceback generated by the 

server's exception handler.

.. seealso:: Source code :file:`ivle/chat.py`

Version Control

===============

Along with traditional file system access, IVLE allows users to version their 

files using Subversion_. Much like how Subversion workspaces are used on a 

standard desktop, workspaces are checked out into users home directories where 

they can be manipulated through a series of AJAX requests to the 

``fileservice`` app.

Like all other user file system actions, version control actions need to be 

executed inside the user's :ref:`jail <ref-jail>`. Requests are made to the 

``fileservice`` app in :mod:`ivle.webapp.fileservice` which then calls the 

``fileservice`` CGI script using ``trampoline``. This script is simply a 

wrapper around :mod:`ivle.fileservice_lib` which actually contains the code to 

handle each of the actions.

Manipulation of the Subversion workspaces is done using the pysvn_ library. 

.. _Subversion: http://subversion.tigris.org/

.. _pysvn: http://pysvn.tigris.org/

Repositories

------------

Each user is allocated a Subversion repository when their :ref:`Jail 

<ref-jail>` is created by the :ref:`User Management Server 

<ref-usrmgt-server>`. Repository are stored in the location specified by 

``paths/svn/repo_path`` in :file:`/etc/ivle/ivle.conf` (by default 

:file:`/var/lib/ivle/svn/repositories/`). User repositories are stored in the 

:samp:`users/{USERNAME}/` subdirectory and group repositories in 

:samp:`groups/{SUBJECT}_{YEAR}_{SEMESTER}_{GROUP}`.

.. warning::

    While it would be possible to give users direct access to their repository 

    using Subversion's file backend, this would allow users to potentially 

    modify the history of any repository that they had access to. To ensure 

    repository integrity, all Subversion interaction must be done remotely.

Subversion WebDAV

-----------------

These repositories are served by Apache using ``mod_dav_svn`` allowing access 

over Subversion's WebDAV HTTP or HTTPS backends. Users are authenticated using 

a randomly generated key which is stored in the database and is made available 

to each user inside their jail (``svn_pass`` property inside 

:file:`/home/.ivle.conf`). This key is automatically provided when doing 

Subversion actions, but can be manually entered when accessing a user's 

repository from an external Subversion client such as with :samp:`svn checkout 

{svn_addr}/users/{USERNAME}/ workspace`.

Repository permissions for ``AuthzSVNAccessFile`` are automatically generated 

and placed in the file specified by the ``paths/svn/conf`` config option

(usually ``/var/lib/ivle/svn/svn.conf``) for user repositories and the

``paths/svn/group_conf`` option for group repositories (usually

``/var/lib/ivle/svn/svn-group.conf``). User authentication keys for

``AuthUserFile`` are stored in the file specified by ``paths/svn/auth_ivle``,

usually ``/var/lib/ivle/svn/ivle.auth``. These will be regenerated each time

user or group repository settings change.

Worksheets

==========

Worksheets provide a way for users to be able to attempt a set of coding 

exercises along with accompanying instructions. In the past worksheets were 

created directly using an XML format, but this has been deprecated in favour 

of being generated automatically from reStructuredText.

Worksheets are now stored in the database as a :class:`Worksheet` object (see 

:file:`ivle/database.py`).  This allows them to be treated with the same 

access permissions available to other objects and lays down the ground work 

for providing versioned worksheets.

Exercises

---------

When users submit an exercise, the user's solution is tested against a series 

of test cases which can be used to check if a solution is acceptable. Almost 

all the behavior for exercises is contained within 

:file:`ivle/webapp/tutorial/test/TestFramework.py`.

.. note::

    The TestFramework module is one of the oldest and most complicated in 

    IVLE, largely taken directly from the IVLE prototype. As such it has a 

    design that doesn't quite match the current architecture of IVLE, such as 

    using slightly different terminology and having a few testing facilities 

    that are untested or untested. It requires a substantial rewrite and 

    comprehensive test suite to be developed.

At the top level exists the :class:`Exercise` object (known as ``TestSuite`` 

in :file:`TestFramework.py`). This object encompasses the entire collection of 

tests for a given exercise and details such as the exercise name, provided 

solution and any "include code" (Python code available for all test cases, but 

not the user's submission).

Each exercise may contain one or more :class:`TestSuite` objects (known as 

``TestCase`` in :file:`TestFramework.py`. A test suite is a collection of 

tests that run with some sort of common input - be that stdin contents, a 

virtual file system configuration (presently disabled), inputs to particular 

function or defining the contents of one or more variables. A test suite will 

typically run until the first test case fails, but can be configured to 

continue running test cases even after one has failed. Exceptions raised by 

submitted code will typically cause the test to fail except if it is marked as 

an "allowed exception".

Individual units to be tested (something that can pass or fail) are contained 

within :class:`TestCase` objects (known as ``TestCaseParts`` in 

:file:`TestFramework.py`). A test case can test the value of source code text, 

the function return value (Will be ``None`` for scripts), stdout contents, 

stderr contents, name of any raised exception and contents of the virtual file 

system (presently disabled) of code submitted by users. These checks are 

contained in a :class:`TestCasePart`. In addition, a normalisation function or 

custom comparison function can be used instead of comparing the raw values 

directly.  By default, the value of each check will be ignored unless 

overidden by a test case part.

Database

========

Object Publishing

=================

URLs are resolved with a small IVLE-specific object publishing framework --

that is, resolution is implemented as traversal through an object graph. The

framework lives in :mod:`ivle.webapp.publisher`, and has an extensive test

suite.

This object graph is constructed by the dispatcher. Any plugin class deriving

from ViewPlugin will be searched for ``forward_routes``, ``reverse_routes``

and ``views`` sequences. Everything is class-based -- an object's routes

and views are determined by its class.

Forward routes handle resolution of URLs to objects. Given a source object

and some path segments, the route must calculate the next object.

A forward route is a tuple of ``(source class, intermediate path segments,

function, number of subsequent path segments to consume)``, or simply a

reference to a decorated function (see :mod:`ivle.webapp.admin.publishing`

for decoration examples). The function must return the next object in the

path.

A reverse route handles URL generation for an object. Given just an object,

it must return a tuple of ``(previous object, intermediate path segments)``.

This creates a chain of objects and path segments until the root is reached.

Due to IVLE's lack of a utility framework, reverse routes at the root of the

URL space need to refer to the root object with the magical

:mod:`ivle.webapp.publisher.ROOT`. 

Views are registered with a tuple of ``(source class, intermediate path segments,

view class)``.

In all of the above, "intermediate path segments" can either be a single

segment string, or a sequence of multiple strings representing multiple

segments.

.. note::

   While many applications prefer a pattern matching mechanism, this did not

   work out well for IVLE. Our deep URL structure and multitude of nested

   objects with lots of views meant that match patterns had to be repeated

   tediously, and views required many lines of code to turn a match into a

   context object. It also made URL generation very difficult.

   The simple object publishing framework allows views to be registered with

   just one line of code, getting their context object for free. URL

   generation now comes at a cost of approximately one line of code per class,

   and breadcrumbs are easy too. The reduced code duplication also improves

   robustness.