~drizzle-trunk/drizzle/development

**********************************
test-run.pl - Drizzle testing tool
**********************************

Synopsis
========

**./test-run** [ *OPTIONS* ] [ TESTCASE ]

Description
===========

:program:`test-run.pl` (aka test-run, dtr, mtr) is used to execute tests
from the Drizzle test suite.  These tests are included with Drizzle 
distributions and provide a way for users to verify that the system will
operate according to expectations.

The tests use a diff-based paradigm, meaning that the test runner executes
a test and then compares the results received with pre-recorded expected 
results.  In the event of a test failure, the program will provide output
highlighting the differences found between expected and actual results; this
can be useful for troubleshooting and in bug reports.

While most users are concerned with ensuring general functionality, the 
program also allows a user to quickly spin up a server for ad-hoc testing
and to run the test-suite against an already running Drizzle server.

Running tests
=========================

There are several different ways to run tests using :program:`test-run.pl`.

It should be noted that unless :option:`--force` is used, the program will
stop execution upon encountering the first failing test. 
:option:`--force` is recommended if you are running several tests - it will
allow you to view all successes and failures in one run.

Running individual tests
------------------------
If one only wants to run a few, specific tests, they may do so this way::

    ./test-run [OPTIONS] test1 [test2 ... testN]

Running all tests within a suite
--------------------------------
Many of the tests supplied with Drizzle are organized into suites.  

The tests within drizzle/tests/t are considered the 'main' suite.  
Other suites are located in either drizzle/tests/suite or within the various
directories in drizzle/plugin.  Tests for a specific plugin should live in 
the plugin's directory - drizzle/plugin/example_plugin/tests

To run the tests in a specific suite::

    ./test-run [OPTIONS] --suite=SUITENAME

Running specific tests within a suite
--------------------------------------
To run a specific set of tests within a suite::

    ./test-run [OPTIONS] --suite=SUITENAME TEST1 [TEST2..TESTN]

Calling tests using <suitename>.<testname> currently does not work.
One must specify the test suite via the :option:`--suite` option.


Running all available tests
---------------------------
Currently, the quickest way to execute all tests in all suites is
to use 'make test' from the drizzle root.

Otherwise, one should simply name all suites::

    ./test-run [OPTIONS] --suite=SUITE1, SUITE2, ...SUITEN

Interpreting test results
=========================
The output of the test runner is quite simple.  Every test should pass.
In the event of a test failure, please take the time to file a bug here:
*https://bugs.launchpad.net/drizzle*

During a run, the program will provide the user with:
  * test name (suite + name)
  * test status (pass/fail/skipped)
  * time spent executing each test

At the end of a run, the program will provide the user with a listing of:
  * how many tests were run
  * how many tests failed
  * percentage of passing tests
  * a listing of failing tests
  * total time spent executing the tests

Example output::

    <snip>
    main.snowman                                                 [ pass ]       9
    main.statement_boundaries                                    [ pass ]      17
    main.status                                                  [ pass ]      12
    main.strict                                                  [ pass ]      50
    main.subselect                                               [ pass ]    6778
    main.subselect2                                              [ pass ]      51
    main.subselect3                                              [ fail ]
    drizzletest: At line 621: query 'select a, (select max(b) from t1) into outfile
    <snip>
    --------------------------------------------------------------------------------
    Stopping All Servers
    Failed 10/231 tests, 95.67% were successful.

    The log files in var/log may give you some hint
    of what went wrong.
    If you want to report this error, go to:
	http://bugs.launchpad.net/drizzle
    The servers were restarted 16 times
    Spent 64.364 of 115 seconds executing testcases

    drizzle-test-run in default mode: *** Failing the test(s): main.exp1 
    main.func_str main.loaddata main.null main.outfile main.subselect3 
    main.warnings jp.like_utf8 jp.select_utf8 jp.where_utf8
    
Additional uses
===============
Starting a server for manual testing
------------------------------------

:program:`test-run.pl` allows a user to get a Drizzle server up and running
quickly.  This can be useful for fast ad-hoc testing.

To do so call::

    ./test-run --start-and-exit [*OPTIONS*]

This will start a Drizzle server that you can connect to and query

Starting a server against a pre-populated DATADIR
--------------------------------------------------

Using :option:`--start-dirty` prevents :program:`test-run.pl` from attempting
to initialize (clean) the datadir.  This can be useful if you want to use
an already-populated datadir for testing.

Program architecture
====================

:program:`test-run.pl` uses a simple diff-based mechanism for testing.  
It will execute the statements contained in a test and compare the results 
to pre-recorded expected results.  In the event of a test failure, you
will be presented with a diff::

    main.exp1                                                    [ fail ]
    --- drizzle/tests/r/exp1.result	2010-11-02 02:10:25.107013998 +0300
    +++ drizzle/tests/r/exp1.reject	2010-11-02 02:10:32.017013999 +0300
    @@ -5,4 +5,5 @@
    a
    1
    2
    +3
    DROP TABLE t1;

A test case consists of a .test and a .result file.  The .test file includes
the various statements to be executed for a test.  The .result file lists
the expected results for a given test file.  These files live in tests/t 
and tests/r, respectively.  This structure is the same for all test suites.

test-run.pl options
===================

The :program:`test-run.pl` tool has several available options:

./test-run [ OPTIONS ] [ TESTCASE ]

Options to control what engine/variation to run
-----------------------------------------------

.. option:: --compress
   
   Use the compressed protocol between client and server

.. option:: --bench
   
   Run the benchmark suite

.. option:: --small-bench

   Run the benchmarks with --small-tests --small-tables

Options to control directories to use
-------------------------------------

.. option:: --benchdir=DIR          

   The directory where the benchmark suite is stored
   (default: ../../mysql-bench)
  
.. option:: --tmpdir=DIR

   The directory where temporary files are stored
   (default: ./var/tmp).

.. option:: --vardir=DIR  
         
   The directory where files generated from the test run
   is stored (default: ./var). Specifying a ramdisk or
   tmpfs will speed up tests.

.. option:: --mem 
   
   Run testsuite in "memory" using tmpfs or ramdisk
   Attempts to find a suitable location
   using a builtin list of standard locations
   for tmpfs (/dev/shm)
   The option can also be set using environment
   variable DTR_MEM=[DIR]

Options to control what test suites or cases to run
---------------------------------------------------

.. option:: --force                 
   
   Continue to run the suite after failure

.. option:: --do-test=PREFIX or REGEX
                        
   Run test cases which name are prefixed with PREFIX
   or fulfills REGEX

.. option:: --skip-test=PREFIX or REGEX
                        
   Skip test cases which name are prefixed with PREFIX
   or fulfills REGEX

.. option:: --start-from=PREFIX     

   Run test cases starting from test prefixed with PREFIX
   suite[s]=NAME1,..,NAMEN Collect tests in suites from the comma separated
   list of suite names.
   The default is: "main,jp"

.. option:: --skip-rpl              

   Skip the replication test cases.
   combination="ARG1 .. ARG2" Specify a set of "drizzled" arguments for one
   combination.

.. option:: --skip-combination      

   Skip any combination options and combinations files

.. option:: --repeat-test=n         
  
   How many times to repeat each test (default: 1)

Options that specify ports
--------------------------

.. option:: --master_port=PORT      

   Specify the port number used by the first master

.. option:: --slave_port=PORT      

   Specify the port number used by the first slave

.. option:: --dtr-build-thread=#    

   Specify unique collection of ports. Can also be set by
   setting the environment variable DTR_BUILD_THREAD.

Options for test case authoring
-------------------------------

.. option:: --record TESTNAME       

   (Re)genereate the result file for TESTNAME

.. option:: --check-testcases       

   Check testcases for sideeffects

.. option:: --mark-progress         

   Log line number and elapsed time to <testname>.progress

Options that pass on options
----------------------------

.. option:: --drizzled=ARGS           
 
   Specify additional arguments to "drizzled"

Options to run test on running server
-------------------------------------

.. option:: --extern                

   Use running server for tests

.. option:: --user=USER             

   User for connection to extern server

Options for debugging the product
---------------------------------

.. option:: --client-ddd            

   Start drizzletest client in ddd

.. option:: --client-debugger=NAME  

   Start drizzletest in the selected debugger

.. option:: --client-gdb            

   Start drizzletest client in gdb

.. option:: --ddd                   

   Start drizzled in ddd

.. option:: --debug                 

   Dump trace output for all servers and client programs

.. option:: --debugger=NAME         

   Start drizzled in the selected debugger

.. option:: --gdb                   

   Start the drizzled(s) in gdb

.. option:: --manual-debug          

   Let user manually start drizzled in debugger, before running test(s)

.. option:: --manual-gdb            

   Let user manually start drizzled in gdb, before running test(s)

.. option:: --manual-ddd            

   Let user manually start drizzled in ddd, before running test(s)

.. option:: --master-binary=PATH    
   
   Specify the master "drizzled" to use

.. option:: --slave-binary=PATH     

   Specify the slave "drizzled" to use

.. option:: --strace-client         

   Create strace output for drizzletest client

.. option:: --max-save-core         

   Limit the number of core files saved (to avoid filling up disks for 
   heavily crashing server). Defaults to 5, set to 0 for no limit.

Options for coverage, profiling etc
-----------------------------------

.. option:: --gcov                  

   FIXME

.. option:: --gprof                 

   See online documentation on how to use it.

.. option:: --valgrind              

   Run the *drizzletest* and *drizzled* executables using valgrind with 
   default options

.. option:: --valgrind-all          
   
   Synonym for :option:`--valgrind`

.. option:: --valgrind-drizzleslap  

   Run "drizzleslap" with valgrind.

.. option:: --valgrind-drizzletest  

   Run the *drizzletest* and *drizzle_client_test* executable with valgrind

.. option:: --valgrind-drizzled       

   Run the "drizzled" executable with valgrind

.. option:: --valgrind-options=ARGS 

   Deprecated, use :option:`--valgrind-option`

.. option:: --valgrind-option=ARGS  

   Option to give valgrind, replaces default option(s), 
   can be specified more then once

.. option:: --valgrind-path=[EXE]   

   Path to the valgrind executable

.. option:: --callgrind             

   Instruct valgrind to use callgrind

.. option:: --massif                

   Instruct valgrind to use massif

Misc options
------------

.. option:: --comment=STR           

   Write STR to the output

.. option:: --notimer               

   Don't show test case execution time

.. option:: --script-debug          

   Debug this script itself

.. option:: --verbose               

   More verbose output

.. option:: --start-and-exit        

   Only initialize and start the servers, using the
   startup settings for the specified test case (if any)

.. option:: --start-dirty           

   Only start the servers (without initialization) for
   the specified test case (if any)

.. option:: --fast                  

   Don't try to clean up from earlier runs

.. option:: --reorder               

   Reorder tests to get fewer server restarts

.. option:: --help                  

   Get this help text

.. option:: --testcase-timeout=MINUTES 

   Max test case run time (default 15)

.. option:: --suite-timeout=MINUTES 

   Max test suite run time (default 180)

.. option:: --warnings | log-warnings 

   Pass --log-warnings to drizzled

.. option:: --sleep=SECONDS         

   Passed to drizzletest, will be used as fixed sleep time