90
87
To each user, it appears that they have their own private Unix filesystem
91
88
containing software, libraries and a home directory to do with what they
92
please. This is mainly done by the setuid root program ``trampoline`` which
93
mounts the users home directory, sets up the users environment, jumps into the
94
user's jail using the :manpage:`chroot(2)` system call and finally drops
95
privileges to the desired user and group.
89
please. This is mainly done by the setuid root program ``trampoline`` (See
90
:file:`bin/trampoline/trampoline.c`) which mounts the users home directory,
91
sets up the users environment, jumps into the user's jail using the
92
:manpage:`chroot(2)` system call and finally drops privileges to the desired
97
95
To prevent abuse, ``trampoline`` can only be used by root or one of the uids
98
96
specified when trampoline is built by ``setup.py build`` (defaults to UID 33,
99
97
www-data on Debian systems). Since it's one of two C programs involved in IVLE
100
and runs setuid root it is rather security sensitive.
102
.. seealso:: Source code :file:`bin/trampoline/trampoline.c`
98
and runs setuid root it is rather secuity sensative.
105
100
Base Image Generation
106
101
---------------------
131
126
-------------------------
133
128
To give the appearance of a private file system we need to merge together a
134
user's local home directory with the base image.
135
To achieve this, IVLE uses the *bind mount* feature of Linux, which allows
136
directories to be accessible from another location in the file system. By
137
carefully bind-mounting the jail image as read-only and then bind-mounting the
138
user's :file:`/home` and :file:`/tmp` directory data over the top, we create a
139
jail with only three bind mounts and at virtually no file system overhead.
142
IVLE has historically used numerous solutions to this problem, which are
143
chronicled here to avoid the same mistakes being made again.
145
In the first release of IVLE this was done offline by hard-linking all the
146
files into the target directory, but for a large number of users, this
147
process can take several hours, and also runs the risk of exhausting
148
the number of inodes on the underlying file system.
150
The second solution was to use `AUFS <http://aufs.sourceforge.net/>`_ to
151
mount the user's home directory over a read-only version of the base on
152
demand. This was implemented as part of ``trampoline`` and used a secondary
153
program ``timount`` (see :file:`bin/timount/timount.c`), run at regular
154
intervals, to unmount unused jails. This used the :const:`MNT_EXPIRE` flag
155
for :manpage:`umount(2)` (available since Linux 2.6.8) that only unmounts a
156
directory if it hasn't been accessed since the previous call with
159
While quite effective, AUFS appeared to cause NFS caching issues when IVLE
160
was run as a cluster, and as its inclusion status in future Linux
161
distributions is questionable, the developers elected to use the much older
162
bind mount feature instead.
129
user's local home directory with the base image. In the first release of IVLE
130
this was done off-line by hardlinking all the files into the target directory,
131
but for more than a handful of users this process could take several hours and
132
also ran the risk of exhausting inodes on the underlying file system.
134
The first solution was to use `AUFS <http://aufs.sourceforge.net/>`_ to mount
135
the user's home directory over a read-only version of the base on demand. This
136
was implemented as part of ``trampoline`` and used a secondary program
137
``timount`` (see :file:`bin/timount/timount.c`) run at regular intervals to
138
unmount unused jails. This uses the :const:`MNT_EXPIRE` flag for
139
:manpage:`umount(2)` (available since Linux 2.6.8) that only unmounts a
140
directory if it hasn't been accessed since the previous call with
143
While quite effective, AUFS appears to cause NFS caching issues when IVLE is
144
run as a cluster as well as questionable inclusion status in newer
145
distributions. The current system used in IVLE the much older *bind mount*
146
feature which allows directories to be accessible from another location in the
147
file system. By carefully read-only bind mounting the jail image and then bind
148
mounting the user's :file:`/home` and :file:`/tmp` directory data over the top
149
we can create a jail with only three bind mounts and at virtually no
164
152
Entering the Jail
165
153
-----------------
185
173
than the real memory used. For this reason :const:`RLIMIT_AS` is currently set
189
.. _ref-python-console:
194
179
IVLE provides a web based programming console, exposing similar features to
195
Python's command line console. It is built around the
196
:file:`services/python-console` script, which opens up a socket on a random
197
port to which `JSON`_ encoded chat requests can be made.
199
A new console is typically launched on demand by the web client to the HTTP
200
API, which in turn calls the wrapper class :class:`ivle.console.Console` to
201
start a new console in the user's jail.
203
Subsequent requests from the same in-browser console connect to the existing
204
console process. This is achieved by storing a string on the client which
205
identifies the server address and port. The client then makes requests
206
through the load balancer, sending this string through to an arbitrary slave
207
which forwards the request to the identified console.
209
This means that all slaves need access to all ports on every other slave.
180
Python's command line console. It is built around python script
181
:file:`services/python-console` which opens up a socket to which `JSON`_
182
encoded chat requests can be made. A new console is typically from launched on
183
demand by the web client to the HTTP API, which in turn calls the wrapper
184
class :class:`ivle.console.Console` to start a new console in the user's jail.
211
186
.. _JSON: http://json.org
214
.. _ref-usrmgt-server:
216
188
User Management Server
217
189
======================
225
197
* Creating group Subversion repositories.
226
198
* Rebuilding Subversion authorization files.
228
Communication with the Server is done using the :ref:`Chat Protocol
229
<ref-chat>`. To prevent unauthorized use, communication with the User
230
Management Server requires that a *shared secret* be used to communicate with
231
the server. This secret is stored in the `magic` variable in the `[usrmgt]`
232
section of :file:`/etc/ivle/ivle.conf`.
200
Communication with the Server is done using the `Chat Protocol`_. To prevent
201
unauthorized use, communication with the User Management Server requires that
202
a *shared secret* be used to communicate with the server. This secret is
203
stored in the `magic` variable in the `[usrmgt]` section of
204
:file:`/etc/ivle/ivle.conf`.
234
206
The User Management Server is called almost exclusively from the
235
207
:mod:`ivle.webapp.userservice` module.
237
.. seealso:: Source code :file:`services/usrmgt-server`
209
See :file:`services/usrmgt-server` for details.
244
**Chat** is our JSON_-based client/server communication protocol used in
245
communicating to :ref:`Python Console <ref-python-console>` processes and
246
:ref:`User Management Server <ref-usrmgt-server>`. Since it is JSON-based it
247
can be called from either Python or JavaScript.
214
**Chat** is our JSON_-based client/server communication protocol used in
215
communicating to `Python Console`_ processes and `User Management Server`_.
216
Since it is JSON-based it can be called from either Python or JavaScript.
280
249
exception and :const:`TRACEBACK` is a string of the traceback generated by the
281
250
server's exception handler.
283
.. seealso:: Source code :file:`ivle/chat.py`
252
See :file:`ivle/chat.py` for details.
289
Along with traditional file system access, IVLE allows users to version their
290
files using Subversion_. Much like how Subversion workspaces are used on a
291
standard desktop, workspaces are checked out into users home directories where
292
they can be manipulated through a series of AJAX requests to the
295
Like all other user file system actions, version control actions need to be
296
executed inside the user's :ref:`jail <ref-jail>`. Requests are made to the
297
``fileservice`` app in :mod:`ivle.webapp.fileservice` which then calls the
298
``fileservice`` CGI script using ``trampoline``. This script is simply a
299
wrapper around :mod:`ivle.fileservice_lib` which actually contains the code to
300
handle each of the actions.
302
Manipulation of the Subversion workspaces is done using the pysvn_ library.
304
.. _Subversion: http://subversion.tigris.org/
305
.. _pysvn: http://pysvn.tigris.org/
311
Each user is allocated a Subversion repository when their :ref:`Jail
312
<ref-jail>` is created by the :ref:`User Management Server
313
<ref-usrmgt-server>`. Repository are stored in the location specified by
314
``[paths] [[svn]] repo_path`` in :file:`/etc/ivle/ivle.conf` (by default
315
:file:`/var/lib/ivle/svn/repositories/`). User repositories are stored in the
316
:samp:`users/{USERNAME}/` subdirectory and group repositories in
317
:samp:`groups/{SUBJECT}_{YEAR}_{SEMESTER}_{GROUP}`.
321
While it would be possible to give users direct access to their repository
322
using Subversion's file backend, this would allow users to potentially
323
modify the history of any repository that they had access to. To ensure
324
repository integrity, all Subversion interaction must be done remotely.
330
These repositories are served by Apache using ``mod_dav_svn`` allowing access
331
over Subversion's WebDAV HTTP or HTTPS backends. Users are authenticated using
332
a randomly generated key which is stored in the database and is made available
333
to each user inside their Jail (``svn_pass`` property inside
334
:file:`/home/.ivle.conf`). This key is automatically provided when doing
335
Subversion actions, but can be manually entered when accessing a users
336
repository from an external Subversion client such as with :samp:`svn checkout
337
{svn_addr}/users/{USERNAME}/ workspace`.
339
Repository permissions for ``AuthzSVNAccessFILE`` are automatically generated
340
and placed in the file located at ``[paths] [[svn]] conf`` for user
341
repositories and ``[paths] [[svn]] group_conf`` for group repositories. User
342
authentication keys for ``AuthUserFile`` are stored in the file located at
343
``[path] [[svn]] auth_ivle``. These will be regenerated each time user or
344
group repository settings change.
added
removed
doc/dev/architecture.rst
