352
Worksheets provide a way for users to be able to attempt a set of coding
353
exercises along with accompanying instructions. In the past worksheets were
354
created directly using an XML format, but this has been deprecated in favour
355
of being generated automatically from reStructuredText.
357
Worksheets are now stored in the database as a :class:`Worksheet` object (see
358
:file:`ivle/database.py`). This allows them to be treated with the same
359
access permissions available to other objects and lays down the ground work
360
for providing versioned worksheets.
366
When users submit an exercise, the user's solution is tested against a series
367
of test cases which can be used to check if a solution is acceptable. Almost
368
all the behavior for exercises is contained within
369
:file:`ivle/webapp/tutorial/test/TestFramework.py`.
372
The TestFramework module is one of the oldest and most complicated in
373
IVLE, largely taken directly from the IVLE prototype. As such it has a
374
design that doesn't quite match the current architecture of IVLE, such as
375
using slightly different terminology and having a few testing facilities
376
that are untested or untested. It requires a substantial rewrite and
377
comprehensive test suite to be developed.
379
At the top level exists the :class:`Exercise` object (known as ``TestSuite``
380
in :file:`TestFramework.py`). This object encompasses the entire collection of
381
tests for a given exercise and details such as the exercise name, provided
382
solution and any "include code" (Python code available for all test cases, but
383
not the user's submission).
385
Each exercise may contain one or more :class:`TestSuite` objects (known as
386
``TestCase`` in :file:`TestFramework.py`. A test suite is a collection of
387
tests that run with some sort of common input - be that stdin contents, a
388
virtual file system configuration (presently disabled), inputs to particular
389
function or defining the contents of one or more variables. A test suite will
390
typically run until the first test case fails, but can be configured to
391
continue running test cases even after one has failed. Exceptions raised by
392
submitted code will typically cause the test to fail except if it is marked as
393
an "allowed exception".
395
Individual units to be tested (something that can pass or fail) are contained
396
within :class:`TestCase` objects (known as ``TestCaseParts`` in
397
:file:`TestFramework.py`). A test case can test the value of source code text,
398
the function return value (Will be ``None`` for scripts), stdout contents,
399
stderr contents, name of any raised exception and contents of the virtual file
400
system (presently disabled) of code submitted by users. These checks are
401
contained in a :class:`TestCasePart`. In addition, a normalisation function or
402
custom comparison function can be used instead of comparing the raw values
403
directly. By default, the value of each check will be ignored unless
404
overidden by a test case part.
added
removed
doc/dev/architecture.rst
