1
**********************************
3
**********************************
9
**./dbqp** [ *OPTIONS* ] [ TESTCASE ]
14
:program:`dbqp.py` is BETA software. It is intended to provide a standardized
15
platform to facilitate Drizzle testing.
17
The default mode is 'dtr' and is used to execute tests from the Drizzle
18
test suite. These tests are included with Drizzle distributions and
19
provide a way for users to verify that the system will operate according
22
The dtr tests use a diff-based paradigm, meaning that the test runner executes
23
a test and then compares the results received with pre-recorded expected
24
results. In the event of a test failure, the program will provide output
25
highlighting the differences found between expected and actual results; this
26
can be useful for troubleshooting and in bug reports.
28
The program is also integrated with the random query generator testing tool
29
and a 'rangden' mode is available - it will execute randgen tests when
30
provided a path to a randgen installation. Tests are organized similar to dtr
31
tests, but are .cnf file based.
33
A 'cleanup' mode is also available as a convenience - it will simply shutdown
34
any servers that may have been started via start-and-exit.
36
While most users are concerned with ensuring general functionality, the
37
program also allows a user to quickly spin up a server for ad-hoc testing
38
and to run the test-suite against an already running Drizzle server.
41
=========================
43
There are several different ways to run tests using :program:`dbqp.py`.
45
It should be noted that unless :option:`--force` is used, the program will
46
stop execution upon encountering the first failing test.
47
:option:`--force` is recommended if you are running several tests - it will
48
allow you to view all successes and failures in one run.
50
Running individual tests
51
------------------------
52
If one only wants to run a few, specific tests, they may do so this way::
54
./dbqp.py [OPTIONS] test1 [test2 ... testN]
56
Running all tests within a suite
57
--------------------------------
58
Many of the tests supplied with Drizzle are organized into suites.
60
The tests within drizzle/tests/t are considered the 'main' suite.
61
Other suites are located in either drizzle/tests/suite or within the various
62
directories in drizzle/plugin. Tests for a specific plugin should live in
63
the plugin's directory - drizzle/plugin/example_plugin/tests
65
To run the tests in a specific suite::
67
./dbqp.py [OPTIONS] --suite=SUITENAME
69
Running specific tests within a suite
70
--------------------------------------
71
To run a specific set of tests within a suite::
73
./dbqp.py [OPTIONS] --suite=SUITENAME TEST1 [TEST2..TESTN]
75
Calling tests using <suitename>.<testname> currently does not work.
76
One must specify the test suite via the :option:`--suite` option.
79
Running all available tests
80
---------------------------
81
Currently, the quickest way to execute all tests in all suites is
82
to use 'make test-dbqp' from the drizzle root.
84
Otherwise, one should simply name all suites::
86
./dbqp.py [OPTIONS] --suite=SUITE1, SUITE2, ...SUITEN
88
Interpreting test results
89
=========================
90
The output of the test runner is quite simple. Every test should pass.
91
In the event of a test failure, please take the time to file a bug here:
92
*https://bugs.launchpad.net/drizzle*
94
During a run, the program will provide the user with:
95
* test name (suite + name)
96
* test status (pass/fail/skipped)
97
* time spent executing each test
99
At the end of a run, the program will provide the user with a listing of:
100
* how many tests were run
101
* counts and percentages of total exectuted for all test statuses
102
* a listing of failing, skipped, or disabled tests
103
* total time spent executing the tests
108
30 Jan 2011 16:26:31 : main.small_tmp_table [ pass ] 38
109
30 Jan 2011 16:26:31 : main.snowman [ pass ] 42
110
30 Jan 2011 16:26:31 : main.statement_boundaries [ pass ] 47
111
30 Jan 2011 16:26:31 : main.status [ pass ] 51
112
30 Jan 2011 16:26:31 : main.strict [ pass ] 138
113
30 Jan 2011 16:26:43 : main.subselect [ fail ] 12361
114
30 Jan 2011 16:26:43 : --- drizzle/tests/r/subselect.result 2011-01-30 16:23:54.975776148 -0500
115
30 Jan 2011 16:26:43 : +++ drizzle/tests/r/subselect.reject 2011-01-30 16:26:43.835519303 -0500
116
30 Jan 2011 16:26:43 : @@ -5,7 +5,7 @@
117
30 Jan 2011 16:26:43 : 2
118
30 Jan 2011 16:26:43 : explain extended select (select 2);
119
30 Jan 2011 16:26:43 : id select_type table type possible_keys key key_len ref rows filtered Extra
120
30 Jan 2011 16:26:43 : -9 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL No tables used
121
30 Jan 2011 16:26:43 : +1 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL No tables used
123
30 Jan 2011 16:30:20 : ================================================================================
124
30 Jan 2011 16:30:20 INFO: Test execution complete in 314 seconds
125
30 Jan 2011 16:30:20 INFO: Summary report:
126
30 Jan 2011 16:30:20 INFO: Executed 552/552 test cases, 100.00 percent
127
30 Jan 2011 16:30:20 INFO: STATUS: FAIL, 1/552 test cases, 0.18 percent executed
128
30 Jan 2011 16:30:20 INFO: STATUS: PASS, 551/552 test cases, 99.82 percent executed
129
30 Jan 2011 16:30:20 INFO: FAIL tests: main.subselect
130
30 Jan 2011 16:30:20 INFO: Spent 308 / 314 seconds on: TEST(s)
131
30 Jan 2011 16:30:20 INFO: Test execution complete
132
30 Jan 2011 16:30:20 INFO: Stopping all running servers...
137
Starting a server for manual testing
138
------------------------------------
140
:program:`dbqp.py` allows a user to get a Drizzle server up and running
141
quickly. This can be useful for fast ad-hoc testing.
145
./dbqp.py --start-and-exit [*OPTIONS*]
147
This will start a Drizzle server that you can connect to and query
149
Starting a server against a pre-populated DATADIR
150
--------------------------------------------------
152
Using :option:`--start-dirty` prevents :program:`dbqp.py` from attempting
153
to initialize (clean) the datadir. This can be useful if you want to use
154
an already-populated datadir for testing.
156
NOTE: This feature is still being tested, use caution with your data!!!
158
Randgen mode / Executing randgen tests
159
---------------------------------------
161
Using :option:`--mode` =randgen and :option:`--randgen-path` =/path/to/randgen
162
will cause the randgen tests to execute. This are simple .cnf file-based
163
tests that define various randgen command lines that are useful in testing
164
the server. Test organization is similar to the dtr tests. Tests live in
165
suites, the default suite is 'main' and they all live in
166
drizzle/tests/randgen_tests::
168
./dbqp.py --mode=randgen --randgen-path=/path/to/randgen
170
A user may specify suites and individual tests to run, just as with dtr-based
171
testing. Test output is the same as well::
173
./dbqp --mode=randgen --randgen-path=/home/username/repos/randgen
174
Setting --no-secure-file-priv=True for randgen mode...
176
23 Feb 2011 11:42:43 INFO: Using testing mode: randgen
178
23 Feb 2011 11:44:58 : ================================================================================
179
23 Feb 2011 11:44:58 : TEST NAME [ RESULT ] TIME (ms)
180
23 Feb 2011 11:44:58 : ================================================================================
181
23 Feb 2011 11:44:58 : main.optimizer_subquery [ pass ] 134153
182
23 Feb 2011 11:45:03 : main.outer_join [ pass ] 5136
183
23 Feb 2011 11:45:06 : main.simple [ pass ] 2246
184
23 Feb 2011 11:45:06 : ================================================================================
185
23 Feb 2011 11:45:06 INFO: Test execution complete in 142 seconds
186
23 Feb 2011 11:45:06 INFO: Summary report:
187
23 Feb 2011 11:45:06 INFO: Executed 3/3 test cases, 100.00 percent
188
23 Feb 2011 11:45:06 INFO: STATUS: PASS, 3/3 test cases, 100.00 percent executed
189
23 Feb 2011 11:45:06 INFO: Spent 141 / 142 seconds on: TEST(s)
190
23 Feb 2011 11:45:06 INFO: Test execution complete
191
23 Feb 2011 11:45:06 INFO: Stopping all running servers...
195
A cleanup mode is provided for user convenience. This simply shuts down
196
any servers whose pid files are detected in the dbqp workdir. It is mainly
197
intended as a quick cleanup for post-testing with :option:`--start-and-exit`::
199
./dbqp.py --mode=cleanup
201
Setting --start-dirty=True for cleanup mode...
202
23 Feb 2011 11:35:59 INFO: Using Drizzle source tree:
203
23 Feb 2011 11:35:59 INFO: basedir: drizzle
204
23 Feb 2011 11:35:59 INFO: clientbindir: drizzle/client
205
23 Feb 2011 11:35:59 INFO: testdir: drizzle/tests
206
23 Feb 2011 11:35:59 INFO: server_version: 2011.02.2188
207
23 Feb 2011 11:35:59 INFO: server_compile_os: unknown-linux-gnu
208
23 Feb 2011 11:35:59 INFO: server_platform: x86_64
209
23 Feb 2011 11:35:59 INFO: server_comment: (Source distribution (dbqp_randgen))
210
23 Feb 2011 11:35:59 INFO: Using --start-dirty, not attempting to touch directories
211
23 Feb 2011 11:35:59 INFO: Using default-storage-engine: innodb
212
23 Feb 2011 11:35:59 INFO: Using testing mode: cleanup
213
23 Feb 2011 11:35:59 INFO: Killing pid 10484 from drizzle/tests/workdir/testbot0/server0/var/run/server0.pid
214
23 Feb 2011 11:35:59 INFO: Stopping all running servers...
219
:program:`dbqp.py`'s 'dtr' mode uses a simple diff-based mechanism for testing.
220
This is the default mode and where the majority of Drizzle testing occurs.
221
It will execute the statements contained in a test and compare the results
222
to pre-recorded expected results. In the event of a test failure, you
223
will be presented with a diff::
226
--- drizzle/tests/r/exp1.result 2010-11-02 02:10:25.107013998 +0300
227
+++ drizzle/tests/r/exp1.reject 2010-11-02 02:10:32.017013999 +0300
235
A test case consists of a .test and a .result file. The .test file includes
236
the various statements to be executed for a test. The .result file lists
237
the expected results for a given test file. These files live in tests/t
238
and tests/r, respectively. This structure is the same for all test suites.
243
The :program:`dbqp.py` tool has several available options:
245
./dbqp.py [ OPTIONS ] [ TESTCASE ]
253
.. option:: -h, --help
255
show this help message and exit
257
Options for the test-runner itself
258
----------------------------------
264
Set this to continue test execution beyond the first failed test
266
.. option:: --start-and-exit
268
Spin up the server(s) for the first specified test then exit
269
(will leave servers running)
271
.. option:: --verbose
273
Produces extensive output about test-runner state.
274
Distinct from --debug
278
Provide internal-level debugging output.
279
Distinct from --verbose
281
.. option:: --mode=MODE
284
We only support dtr...for now >;)
289
Record a testcase result
290
(if the testing mode supports it)
295
Don't try to cleanup from earlier runs
296
(currently just a placeholder) [False]
298
.. option:: --randgen-path=RANDGENPATH
300
The path to a randgen installation that can be used to
301
execute randgen-based tests
304
Options for controlling which tests are executed
305
------------------------------------------------
309
.. option:: --suite=SUITELIST
311
The name of the suite containing tests we want.
312
Can accept comma-separated list (with no spaces).
313
Additional --suite args are appended to existing list
316
.. option:: --suitepath=SUITEPATHS
318
The path containing the suite(s) you wish to execute.
319
Use on --suitepath for each suite you want to use.
321
.. option:: --do-test=DOTEST
323
input can either be a prefix or a regex.
324
Will only execute tests that match the provided pattern
326
.. option:: --skip-test=SKIPTEST
328
input can either be a prefix or a regex.
329
Will exclude tests that match the provided pattern
331
.. option:: --reorder
333
sort the testcases so that they are executed optimally
334
for the given mode [False]
336
.. option:: --repeat=REPEAT
338
Run each test case the specified number of times. For
339
a given sequence, the first test will be run n times,
340
then the second, etc [1]
342
Options for defining the code that will be under test
343
-----------------------------------------------------
347
.. option:: --basedir=BASEDIR
349
Pass this argument to signal to the test-runner
350
that this is an in-tree test (not required).
351
We automatically set a number of variables
352
relative to the argument (client-bindir,
353
serverdir, testdir) [../]
355
.. option:: --serverdir=SERVERPATH
357
Path to the server executable. [auto-search]
359
.. option:: --client-bindir=CLIENTBINDIR
361
Path to the directory containing client program
362
binaries for use in testing [auto-search]
364
.. option:: --default-storage-engine=DEFAULTENGINE
366
Start drizzled using the specified engine [innodb]
368
Options for defining the testing environment
369
--------------------------------------------
373
.. option:: --testdir=TESTDIR
375
Path to the test dir, containing additional files for
376
test execution. [pwd]
378
.. option:: --workdir=WORKDIR
380
Path to the directory test-run will use to store
381
generated files and directories.
382
[basedir/tests/dbqp_work]
384
.. option:: --top-srcdir=TOPSRCDIR
386
build option [basedir_default]
388
.. option:: --top-builddir=TOPBUILDDIR
390
build option [basedir_default]
394
By default, we symlink workdir to a location in shm.
395
Use this flag to not symlink [False]
397
.. option:: --start-dirty
399
Don't try to clean up working directories before test
402
.. option:: --no-secure-file-priv
404
Turn off the use of --secure-file-priv=vardir for
407
Options to pass options on to the server
408
-----------------------------------------
412
.. option:: --drizzled=DRIZZLEDOPTIONS
414
Pass additional options to the server. Will be passed
415
to all servers for all tests (mostly for --start-and-
419
Options for defining the tools we use for code analysis (valgrind, gprof, gcov, etc)
420
------------------------------------------------------------------------------------
424
.. option:: --valgrind
426
Run drizzletest and drizzled executables using
427
valgrind with default options [False]
429
.. option:: --valgrind-option=VALGRINDARGLIST
431
Pass an option to valgrind (overrides/removes default
434
Options for controlling the use of debuggers with test execution
435
----------------------------------------------------------------
441
Start the drizzled server(s) in gdb
443
.. option:: --manual-gdb
445
Allows you to start the drizzled server(s) in gdb
446
manually (in another window, etc
448
Options to call additional utilities such as datagen
449
------------------------------------------------------
453
.. option:: --gendata=GENDATAFILE
455
Call the randgen's gendata utility to use the
456
specified configuration file. This will populate the
457
server prior to any test execution