1
**********************************
2
dbqp.py - Drizzle testing tool
3
**********************************
8
**./dbqp** [ *OPTIONS* ] [ TESTCASE ]
13
:program:`dbqp.py` is BETA software. It is intended to provide a standardized
14
platform to facilitate Drizzle testing.
16
Currently, it only serves as a partial substitute for test-run.pl.
17
One can execute the test suite, record test cases, and use certain other options.
18
Other test-run.pl option such as gdb and valgrind are still in-development.
20
The current mode is 'dtr' and is used to execute tests from the Drizzle
21
test suite. These tests are included with Drizzle distributions and
22
provide a way for users to verify that the system will operate according
25
The dtr tests use a diff-based paradigm, meaning that the test runner executes
26
a test and then compares the results received with pre-recorded expected
27
results. In the event of a test failure, the program will provide output
28
highlighting the differences found between expected and actual results; this
29
can be useful for troubleshooting and in bug reports.
31
While most users are concerned with ensuring general functionality, the
32
program also allows a user to quickly spin up a server for ad-hoc testing
33
and to run the test-suite against an already running Drizzle server.
36
=========================
38
There are several different ways to run tests using :program:`dbqp.py`.
40
It should be noted that unless :option:`--force` is used, the program will
41
stop execution upon encountering the first failing test.
42
:option:`--force` is recommended if you are running several tests - it will
43
allow you to view all successes and failures in one run.
45
Running individual tests
46
------------------------
47
If one only wants to run a few, specific tests, they may do so this way::
49
./dbqp.py [OPTIONS] test1 [test2 ... testN]
51
Running all tests within a suite
52
--------------------------------
53
Many of the tests supplied with Drizzle are organized into suites.
55
The tests within drizzle/tests/t are considered the 'main' suite.
56
Other suites are located in either drizzle/tests/suite or within the various
57
directories in drizzle/plugin. Tests for a specific plugin should live in
58
the plugin's directory - drizzle/plugin/example_plugin/tests
60
To run the tests in a specific suite::
62
./dbqp.py [OPTIONS] --suite=SUITENAME
64
Running specific tests within a suite
65
--------------------------------------
66
To run a specific set of tests within a suite::
68
./dbqp.py [OPTIONS] --suite=SUITENAME TEST1 [TEST2..TESTN]
70
Calling tests using <suitename>.<testname> currently does not work.
71
One must specify the test suite via the :option:`--suite` option.
74
Running all available tests
75
---------------------------
76
Currently, the quickest way to execute all tests in all suites is
77
to use 'make test-dbqp' from the drizzle root.
79
Otherwise, one should simply name all suites::
81
./dbqp.py [OPTIONS] --suite=SUITE1, SUITE2, ...SUITEN
83
Interpreting test results
84
=========================
85
The output of the test runner is quite simple. Every test should pass.
86
In the event of a test failure, please take the time to file a bug here:
87
*https://bugs.launchpad.net/drizzle*
89
During a run, the program will provide the user with:
90
* test name (suite + name)
91
* test status (pass/fail/skipped)
92
* time spent executing each test
94
At the end of a run, the program will provide the user with a listing of:
95
* how many tests were run
96
* counts and percentages of total exectuted for all test statuses
97
* a listing of failing, skipped, or disabled tests
98
* total time spent executing the tests
103
30 Jan 2011 16:26:31 : main.small_tmp_table [ pass ] 38
104
30 Jan 2011 16:26:31 : main.snowman [ pass ] 42
105
30 Jan 2011 16:26:31 : main.statement_boundaries [ pass ] 47
106
30 Jan 2011 16:26:31 : main.status [ pass ] 51
107
30 Jan 2011 16:26:31 : main.strict [ pass ] 138
108
30 Jan 2011 16:26:43 : main.subselect [ fail ] 12361
109
30 Jan 2011 16:26:43 : --- drizzle/tests/r/subselect.result 2011-01-30 16:23:54.975776148 -0500
110
30 Jan 2011 16:26:43 : +++ drizzle/tests/r/subselect.reject 2011-01-30 16:26:43.835519303 -0500
111
30 Jan 2011 16:26:43 : @@ -5,7 +5,7 @@
112
30 Jan 2011 16:26:43 : 2
113
30 Jan 2011 16:26:43 : explain extended select (select 2);
114
30 Jan 2011 16:26:43 : id select_type table type possible_keys key key_len ref rows filtered Extra
115
30 Jan 2011 16:26:43 : -9 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL No tables used
116
30 Jan 2011 16:26:43 : +1 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL No tables used
118
30 Jan 2011 16:30:20 : ================================================================================
119
30 Jan 2011 16:30:20 INFO: Test execution complete in 314 seconds
120
30 Jan 2011 16:30:20 INFO: Summary report:
121
30 Jan 2011 16:30:20 INFO: Executed 552/552 test cases, 100.00 percent
122
30 Jan 2011 16:30:20 INFO: STATUS: FAIL, 1/552 test cases, 0.18 percent executed
123
30 Jan 2011 16:30:20 INFO: STATUS: PASS, 551/552 test cases, 99.82 percent executed
124
30 Jan 2011 16:30:20 INFO: FAIL tests: main.subselect
125
30 Jan 2011 16:30:20 INFO: Spent 308 / 314 seconds on: TEST(s)
126
30 Jan 2011 16:30:20 INFO: Test execution complete
127
30 Jan 2011 16:30:20 INFO: Stopping all running servers...
132
Starting a server for manual testing
133
------------------------------------
135
:program:`dbqp.py` allows a user to get a Drizzle server up and running
136
quickly. This can be useful for fast ad-hoc testing.
140
./dbqp.py --start-and-exit [*OPTIONS*]
142
This will start a Drizzle server that you can connect to and query
144
Starting a server against a pre-populated DATADIR
145
--------------------------------------------------
147
Using :option:`--start-dirty` prevents :program:`dbqp.py` from attempting
148
to initialize (clean) the datadir. This can be useful if you want to use
149
an already-populated datadir for testing.
151
NOTE: This feature is still being tested, use caution with your data!!!
156
:program:`dbqp.py` uses a simple diff-based mechanism for testing.
157
It will execute the statements contained in a test and compare the results
158
to pre-recorded expected results. In the event of a test failure, you
159
will be presented with a diff::
162
--- drizzle/tests/r/exp1.result 2010-11-02 02:10:25.107013998 +0300
163
+++ drizzle/tests/r/exp1.reject 2010-11-02 02:10:32.017013999 +0300
171
A test case consists of a .test and a .result file. The .test file includes
172
the various statements to be executed for a test. The .result file lists
173
the expected results for a given test file. These files live in tests/t
174
and tests/r, respectively. This structure is the same for all test suites.
179
The :program:`dbqp.py` tool has several available options:
181
./dbqp.py [ OPTIONS ] [ TESTCASE ]
186
.. option:: -h, --help
188
show this help message and exit
190
Options for the test-runner itself
191
----------------------------------
194
Set this to continue test execution beyond the first failed test
196
.. option:: --start-and-exit
198
Spin up the server(s) for the first specified test then exit
199
(will leave servers running)
201
.. option:: --verbose
203
Produces extensive output about test-runner state.
204
Distinct from --debug
208
Provide internal-level debugging output.
209
Distinct from --verbose
211
.. option:: --mode=MODE
214
We only support dtr...for now >;)
219
Record a testcase result
220
(if the testing mode supports it)
225
Don't try to cleanup from earlier runs
226
(currently just a placeholder) [False]
228
Options for controlling which tests are executed
229
------------------------------------------------
230
.. option:: --suite=SUITELIST
232
The name of the suite containing tests we want.
233
Can accept comma-separated list (with no spaces).
234
Additional --suite args are appended to existing list
237
.. option:: --suitepath=SUITEPATHS
239
The path containing the suite(s) you wish to execute.
240
Use on --suitepath for each suite you want to use.
242
.. option:: --do-test=DOTEST
244
input can either be a prefix or a regex.
245
Will only execute tests that match the provided pattern
247
.. option:: --skip-test=SKIPTEST
249
input can either be a prefix or a regex.
250
Will exclude tests that match the provided pattern
252
.. option:: --reorder
254
sort the testcases so that they are executed optimally
255
for the given mode [False]
257
.. option:: --repeat=REPEAT
258
Run each test case the specified number of times. For
259
a given sequence, the first test will be run n times,
260
then the second, etc [1]
262
Options for defining the code that will be under test
263
-----------------------------------------------------
265
.. option:: --basedir=BASEDIR
267
Pass this argument to signal to the test-runner
268
that this is an in-tree test (not required).
269
We automatically set a number of variables
270
relative to the argument (client-bindir,
271
serverdir, testdir) [../]
273
.. option:: --serverdir=SERVERPATH
275
Path to the server executable. [auto-search]
277
.. option:: --client-bindir=CLIENTBINDIR
279
Path to the directory containing client program
280
binaries for use in testing [auto-search]
282
.. option:: --default-storage-engine=DEFAULTENGINE
284
Start drizzled using the specified engine [innodb]
286
Options for defining the testing environment
287
--------------------------------------------
288
.. option:: --testdir=TESTDIR
290
Path to the test dir, containing additional files for
291
test execution. [pwd]
293
.. option:: --workdir=WORKDIR
295
Path to the directory test-run will use to store
296
generated files and directories.
297
[basedir/tests/dbqp_work]
299
.. option:: --top-srcdir=TOPSRCDIR
301
build option [basedir_default]
303
.. option:: --top-builddir=TOPBUILDDIR
305
build option [basedir_default]
309
By default, we symlink workdir to a location in shm.
310
Use this flag to not symlink [False]
312
.. option:: --start-dirty
314
Don't try to clean up working directories before test
317
.. option:: --no-secure-file-priv
319
Turn off the use of --secure-file-priv=vardir for
322
Options to pass options on to the server
323
-----------------------------------------
324
.. option:: --drizzled=DRIZZLEDOPTIONS
326
Pass additional options to the server. Will be passed
327
to all servers for all tests (mostly for --start-and-
331
Options for defining the tools we use for code analysis (valgrind, gprof, gcov, etc)
332
---------------------------------------------------------------------------------------
333
.. option:: --valgrind
335
Run drizzletest and drizzled executables using
336
valgrind with default options [False]
338
.. option:: --valgrind-option=VALGRINDARGLIST
340
Pass an option to valgrind (overrides/removes default
343
Options for controlling the use of debuggers with test execution
344
------------------------------------------------------------------
347
Start the drizzled server(s) in gdb
349
.. option:: --manual-gdb
351
Allows you to start the drizzled server(s) in gdb
352
manually (in another window, etc
added
removed
docs/testing/dbqp.rst
