1
.. IVLE - Informatics Virtual Learning Environment
2
Copyright (C) 2007-2009 The University of Melbourne
4
.. This program is free software; you can redistribute it and/or modify
5
it under the terms of the GNU General Public License as published by
6
the Free Software Foundation; either version 2 of the License, or
7
(at your option) any later version.
9
.. This program is distributed in the hope that it will be useful,
10
but WITHOUT ANY WARRANTY; without even the implied warranty of
11
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12
GNU General Public License for more details.
14
.. You should have received a copy of the GNU General Public License
15
along with this program; if not, write to the Free Software
16
Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
22
IVLE is a complex piece of software that integrates closely with the
23
underlying system. It can be considered part web service and part local system
24
daemon. Due to the implementation of these parts it is tied to Apache Web
25
Server (mainly due to the use of mod_python) and Linux.
31
One of the main features of IVLE is it's ability to execute user's code in a
32
customised environment that prevents access to other users files or underlying
33
file system as well as placing basic resource limits to prevent users from
34
accidentally exhausting shared resources such as CPU time and memory.
40
To each user, it appears that they have their own private Unix filesystem
41
containing software, libraries and a home directory to do with what they
42
please. This is mainly done by the setuid root program ``trampoline`` (See
43
:file:`bin/trampoline/trampoline.c`) which mounts the users home directory,
44
sets up the users environment, jumps into the user's jail using the
45
:manpage:`chroot(2)` system call and finally drops privileges to the desired
48
To prevent abuse, ``trampoline`` can only be used by root or one of the uids
49
specified when trampoline is built by ``setup.py build`` (defaults to UID 33,
50
www-data on Debian systems). Since it's one of two C programs involved in IVLE
51
and runs setuid root it is rather secuity sensative.
56
All user jails share a common base image that contains the files required for
57
both IVLE's operation and for executing user code. This base image is
58
generated automatically by the ``ivle-buildjail`` script. This then calls the
59
distribution dependant details in :mod:`ivle.jailbuilder` module. At present
60
we only support building jails for Debian derived systems using
61
:program:`debootstrap`.
63
The contents of the base image contains a few core packages required for the
64
operation of IVLE - Python and the Python CJSON and SVN libraries. Other
65
options that can be configured in :file:`/etc/ivle/ivle.conf` are the file
66
mirror that debootstrap should use, the suite to build (such as hardy or
67
jaunty), extra apt-sources, extra apt keys and any additional packages to
70
To prevent users from altering files in the base image we change the
71
permissions of :file:`/tmp`, :file:`/var/tmp` and :file:`/var/lock` to not be
72
world writeable and check that no other files are world writeable.
74
Finally we make the user dependent :file:`/etc/passwd` and
75
:file:`/etc/ivle/ivle.conf` symlinks to files in the :file:`/home` directory
76
so that they will be used when we mount a user's home directory.
78
Mounting Home Directories
79
-------------------------
81
To give the appearance of a private file system we need to merge together a
82
user's local home directory with the base image. In the first release of IVLE
83
this was done off-line by hardlinking all the files into the target directory,
84
but for more than a handful of users this process could take several hours and
85
also ran the risk of exhausting inodes on the underlying file system.
87
The first solution was to use `AUFS <http://aufs.sourceforge.net/>`_ to mount
88
the user's home directory over a read-only version of the base on demand. This
89
was implemented as part of ``trampoline`` and used a secondary program
90
``timount`` (see :file:`bin/timount/timount.c`) run at regular intervals to
91
unmount unused jails. This uses the :const:`MNT_EXPIRE` flag for
92
:manpage:`umount(2)` (available since Linux 2.6.8) that only unmounts a
93
directory if it hasn't been accessed since the previous call with
96
While quite effective, AUFS appears to cause NFS caching issues when IVLE is
97
run as a cluster as well as questionable inclusion status in newer
98
distributions. The current system used in IVLE the much older *bind mount*
99
feature which allows directories to be accessible from another location in the
100
file system. By carefully read-only bind mounting the jail image and then bind
101
mounting the user's :file:`/home` and :file:`/tmp` directory data over the top
102
we can create a jail with only three bind mounts and at virtually no
108
Before running the specified program in the users jail we need to
109
:manpage:`chroot(2)` into the users jail and update the processes environment
110
so that we have the correct environment variables and user/group ids.
112
At this stage we also may apply a number of resource limits (see
113
:manpage:`setrlimit`) to prevent run away processes (such as those containing
114
infinite loops or "fork bombs") from exhausting all system resources. The
115
default limits are on maximum address space (:const:`RLIMIT_AS`), process data
116
space (:const:`RLIMIT_DATA`), core dump size (:const:`RLIMIT_CORE`), CPU time
117
(:const:`RLIMIT_CPU`), file size (:const:`RLIMIT_FSIZE`) and number of
118
processes that may be spawned (:const:`RLIMIT_NPROC`).
120
Unfortunately due to glibc's :manpage:`malloc(2)` implementation being able to
121
allocate memory using :manpage:`mmap(2)`, :const:`RLIMIT_DATA` does not
122
provide an effective limit on the amount of memory that a process can allocate
123
(short of applying a kernel patch). Thus the only way to limit memory
124
allocations is by placing limits on the address space, but this can cause
125
problems with certain applications that allocate far larger address spaces
126
than the real memory used. For this reason :const:`RLIMIT_AS` is currently set
132
IVLE provides a web based programming console, exposing similar features to
133
Python's command line console. It is built around python script
134
:file:`services/python-console` which opens up a socket to which `JSON`_
135
encoded chat requests can be made. A new console is typically from launched on
136
demand by the web client to the HTTP API, which in turn calls the wrapper
137
class :class:`ivle.console.Console` to start a new console in the user's jail.
139
.. _JSON: http://json.org
141
User Management Server
142
======================
144
The **User Management Server** is a daemon responsible for handling privileged
145
actions on IVLE and should be launched along with IVLE. It is primarily
148
* Creating user jails, Subversion repositories, and Subversion authentication
150
* Creating group Subversion repositories.
151
* Rebuilding Subversion authorization files.
153
Communication with the Server is done using the `Chat Protocol`_. To prevent
154
unauthorized use, communication with the User Management Server requires that
155
a *shared secret* be used to communicate with the server. This secret is
156
stored in the `magic` variable in the `[usrmgt]` section of
157
:file:`/etc/ivle/ivle.conf`.
159
The User Management Server is called almost exclusively from the
160
:mod:`ivle.webapp.userservice` module.
162
See :file:`services/usrmgt-server` for details.
167
**Chat** is our JSON_-based client/server communication protocol used in
168
communicating to `Python Console`_ processes and `User Management Server`_.
169
Since it is JSON-based it can be called from either Python or JavaScript.
173
The protocol is a fairly simple client/server based one consisting of a single
174
JSON object. Before communication starts a shared secret :const:`MAGIC` must
175
be known by both parties. The shared secret is then used to form a
176
'keyed-Hash Message Authentication Code' to ensure that the content is valid
177
and not been modified in transit.
179
The client request takes the following form::
186
where :const:`DATA` is any valid JSON value and :const:`HASH` is an string
187
containing the MD5 hash of the :const:`DATA` appended to :const:`MAGIC` and
190
The server will respond with a JSON value corresponding to the request.
191
If an error occurs then a special JSON object will be returned of the
197
"traceback": TRACEBACK
200
where :const:`NAME` is a JSON string of the exception type (such as
201
'AttributeError'), :const:`VALUE` is the string value associated with the
202
exception and :const:`TRACEBACK` is a string of the traceback generated by the
203
server's exception handler.
205
See :file:`ivle/chat.py` for details.
220
.. TODO: Not yet merged
added
removed
doc/dev/architecture.rst
