« back to all changes in this revision
Viewing changes to docs/testing/test-run.rst
- browse files at revision 1901
- compare with another revision
- compare with revision 1929.1.8
- download diff
- download tarball
- view history from revision 1901
- Committer: Lee Bieber
- Date: 2010-11-03 16:59:57 UTC
- mfrom: (1900.1.3 build)
- Revision ID: kalebral@gmail.com-20101103165957-3s69xbr349fd9hz5
Merge Gustaf - Replaced macros with functions/templates
Merge Patrick - fix bug 669698: Need sphinx documentation for test-run.pl
Merge Toru - fix blitzdb VARCHAR issue
Merge Patrick - fix bug 669698: Need sphinx documentation for test-run.pl
Merge Toru - fix blitzdb VARCHAR issue
- files added:
- docs/testing
- files modified:
- docs/include.am
added
removed
docs/testing/test-run.rst
1
**********************************
2
test-run.pl - Drizzle testing tool
3
**********************************
4
5
Synopsis
6
========
7
8
**./test-run** [ *OPTIONS* ] [ TESTCASE ]
9
10
Description
11
===========
12
13
:program:`test-run.pl` (aka test-run, dtr, mtr) is used to execute tests
14
from the Drizzle test suite. These tests are included with Drizzle
15
distributions and provide a way for users to verify that the system will
16
operate according to expectations.
17
18
The tests use a diff-based paradigm, meaning that the test runner executes
19
a test and then compares the results received with pre-recorded expected
20
results. In the event of a test failure, the program will provide output
21
highlighting the differences found between expected and actual results; this
22
can be useful for troubleshooting and in bug reports.
23
24
While most users are concerned with ensuring general functionality, the
25
program also allows a user to quickly spin up a server for ad-hoc testing
26
and to run the test-suite against an already running Drizzle server.
27
28
Running tests
29
=========================
30
31
There are several different ways to run tests using :program:`test-run.pl`.
32
33
It should be noted that unless :option:`--force` is used, the program will
34
stop execution upon encountering the first failing test.
35
:option:`--force` is recommended if you are running several tests - it will
36
allow you to view all successes and failures in one run.
37
38
Running individual tests
39
------------------------
40
If one only wants to run a few, specific tests, they may do so this way::
41
42
./test-run [OPTIONS] test1 [test2 ... testN]
43
44
Running all tests within a suite
45
--------------------------------
46
Many of the tests supplied with Drizzle are organized into suites.
47
48
The tests within drizzle/tests/t are considered the 'main' suite.
49
Other suites are located in either drizzle/tests/suite or within the various
50
directories in drizzle/plugin. Tests for a specific plugin should live in
51
the plugin's directory - drizzle/plugin/example_plugin/tests
52
53
To run the tests in a specific suite::
54
55
./test-run [OPTIONS] --suite=SUITENAME
56
57
Running specific tests within a suite
58
--------------------------------------
59
To run a specific set of tests within a suite::
60
61
./test-run [OPTIONS] --suite=SUITENAME TEST1 [TEST2..TESTN]
62
63
Calling tests using <suitename>.<testname> currently does not work.
64
One must specify the test suite via the :option:`--suite` option.
65
66
67
Running all available tests
68
---------------------------
69
Currently, the quickest way to execute all tests in all suites is
70
to use 'make test' from the drizzle root.
71
72
Otherwise, one should simply name all suites::
73
74
./test-run [OPTIONS] --suite=SUITE1, SUITE2, ...SUITEN
75
76
Interpreting test results
77
=========================
78
The output of the test runner is quite simple. Every test should pass.
79
In the event of a test failure, please take the time to file a bug here:
80
*https://bugs.launchpad.net/drizzle*
81
82
During a run, the program will provide the user with:
83
* test name (suite + name)
84
* test status (pass/fail/skipped)
85
* time spent executing each test
86
87
At the end of a run, the program will provide the user with a listing of:
88
* how many tests were run
89
* how many tests failed
90
* percentage of passing tests
91
* a listing of failing tests
92
* total time spent executing the tests
93
94
Example output::
95
96
<snip>
97
main.snowman [ pass ] 9
98
main.statement_boundaries [ pass ] 17
99
main.status [ pass ] 12
100
main.strict [ pass ] 50
101
main.subselect [ pass ] 6778
102
main.subselect2 [ pass ] 51
103
main.subselect3 [ fail ]
104
drizzletest: At line 621: query 'select a, (select max(b) from t1) into outfile
105
<snip>
106
--------------------------------------------------------------------------------
107
Stopping All Servers
108
Failed 10/231 tests, 95.67% were successful.
109
110
The log files in var/log may give you some hint
111
of what went wrong.
112
If you want to report this error, go to:
113
http://bugs.launchpad.net/drizzle
114
The servers were restarted 16 times
115
Spent 64.364 of 115 seconds executing testcases
116
117
drizzle-test-run in default mode: *** Failing the test(s): main.exp1
118
main.func_str main.loaddata main.null main.outfile main.subselect3
119
main.warnings jp.like_utf8 jp.select_utf8 jp.where_utf8
120
121
Additional uses
122
===============
123
Starting a server for manual testing
124
------------------------------------
125
126
:program:`test-run.pl` allows a user to get a Drizzle server up and running
127
quickly. This can be useful for fast ad-hoc testing.
128
129
To do so call::
130
131
./test-run --start-and-exit [*OPTIONS*]
132
133
This will start a Drizzle server that you can connect to and query
134
135
Starting a server against a pre-populated DATADIR
136
--------------------------------------------------
137
138
Using :option:`--start-dirty` prevents :program:`test-run.pl` from attempting
139
to initialize (clean) the datadir. This can be useful if you want to use
140
an already-populated datadir for testing.
141
142
Program architecture
143
====================
144
145
:program:`test-run.pl` uses a simple diff-based mechanism for testing.
146
It will execute the statements contained in a test and compare the results
147
to pre-recorded expected results. In the event of a test failure, you
148
will be presented with a diff::
149
150
main.exp1 [ fail ]
151
--- drizzle/tests/r/exp1.result 2010-11-02 02:10:25.107013998 +0300
152
+++ drizzle/tests/r/exp1.reject 2010-11-02 02:10:32.017013999 +0300
153
@@ -5,4 +5,5 @@
154
a
155
1
156
2
157
+3
158
DROP TABLE t1;
159
160
A test case consists of a .test and a .result file. The .test file includes
161
the various statements to be executed for a test. The .result file lists
162
the expected results for a given test file. These files live in tests/t
163
and tests/r, respectively. This structure is the same for all test suites.
164
165
test-run.pl options
166
===================
167
168
The :program:`test-run.pl` tool has several available options:
169
170
./test-run [ OPTIONS ] [ TESTCASE ]
171
172
Options to control what engine/variation to run
173
-----------------------------------------------
174
175
.. option:: --compress
176
177
Use the compressed protocol between client and server
178
179
.. option:: --bench
180
181
Run the benchmark suite
182
183
.. option:: --small-bench
184
185
Run the benchmarks with --small-tests --small-tables
186
187
Options to control directories to use
188
-------------------------------------
189
190
.. option:: --benchdir=DIR
191
192
The directory where the benchmark suite is stored
193
(default: ../../mysql-bench)
194
195
.. option:: --tmpdir=DIR
196
197
The directory where temporary files are stored
198
(default: ./var/tmp).
199
200
.. option:: --vardir=DIR
201
202
The directory where files generated from the test run
203
is stored (default: ./var). Specifying a ramdisk or
204
tmpfs will speed up tests.
205
206
.. option:: --mem
207
208
Run testsuite in "memory" using tmpfs or ramdisk
209
Attempts to find a suitable location
210
using a builtin list of standard locations
211
for tmpfs (/dev/shm)
212
The option can also be set using environment
213
variable DTR_MEM=[DIR]
214
215
Options to control what test suites or cases to run
216
---------------------------------------------------
217
218
.. option:: --force
219
220
Continue to run the suite after failure
221
222
.. option:: --do-test=PREFIX or REGEX
223
224
Run test cases which name are prefixed with PREFIX
225
or fulfills REGEX
226
227
.. option:: --skip-test=PREFIX or REGEX
228
229
Skip test cases which name are prefixed with PREFIX
230
or fulfills REGEX
231
232
.. option:: --start-from=PREFIX
233
234
Run test cases starting from test prefixed with PREFIX
235
suite[s]=NAME1,..,NAMEN Collect tests in suites from the comma separated
236
list of suite names.
237
The default is: "main,jp"
238
239
.. option:: --skip-rpl
240
241
Skip the replication test cases.
242
combination="ARG1 .. ARG2" Specify a set of "drizzled" arguments for one
243
combination.
244
245
.. option:: --skip-combination
246
247
Skip any combination options and combinations files
248
249
.. option:: --repeat-test=n
250
251
How many times to repeat each test (default: 1)
252
253
Options that specify ports
254
--------------------------
255
256
.. option:: --master_port=PORT
257
258
Specify the port number used by the first master
259
260
.. option:: --slave_port=PORT
261
262
Specify the port number used by the first slave
263
264
.. option:: --dtr-build-thread=#
265
266
Specify unique collection of ports. Can also be set by
267
setting the environment variable DTR_BUILD_THREAD.
268
269
Options for test case authoring
270
-------------------------------
271
272
.. option:: --record TESTNAME
273
274
(Re)genereate the result file for TESTNAME
275
276
.. option:: --check-testcases
277
278
Check testcases for sideeffects
279
280
.. option:: --mark-progress
281
282
Log line number and elapsed time to <testname>.progress
283
284
Options that pass on options
285
----------------------------
286
287
.. option:: --drizzled=ARGS
288
289
Specify additional arguments to "drizzled"
290
291
Options to run test on running server
292
-------------------------------------
293
294
.. option:: --extern
295
296
Use running server for tests
297
298
.. option:: --user=USER
299
300
User for connection to extern server
301
302
Options for debugging the product
303
---------------------------------
304
305
.. option:: --client-ddd
306
307
Start drizzletest client in ddd
308
309
.. option:: --client-debugger=NAME
310
311
Start drizzletest in the selected debugger
312
313
.. option:: --client-gdb
314
315
Start drizzletest client in gdb
316
317
.. option:: --ddd
318
319
Start drizzled in ddd
320
321
.. option:: --debug
322
323
Dump trace output for all servers and client programs
324
325
.. option:: --debugger=NAME
326
327
Start drizzled in the selected debugger
328
329
.. option:: --gdb
330
331
Start the drizzled(s) in gdb
332
333
.. option:: --manual-debug
334
335
Let user manually start drizzled in debugger, before running test(s)
336
337
.. option:: --manual-gdb
338
339
Let user manually start drizzled in gdb, before running test(s)
340
341
.. option:: --manual-ddd
342
343
Let user manually start drizzled in ddd, before running test(s)
344
345
.. option:: --master-binary=PATH
346
347
Specify the master "drizzled" to use
348
349
.. option:: --slave-binary=PATH
350
351
Specify the slave "drizzled" to use
352
353
.. option:: --strace-client
354
355
Create strace output for drizzletest client
356
357
.. option:: --max-save-core
358
359
Limit the number of core files saved (to avoid filling up disks for
360
heavily crashing server). Defaults to 5, set to 0 for no limit.
361
362
Options for coverage, profiling etc
363
-----------------------------------
364
365
.. option:: --gcov
366
367
FIXME
368
369
.. option:: --gprof
370
371
See online documentation on how to use it.
372
373
.. option:: --valgrind
374
375
Run the *drizzletest* and *drizzled* executables using valgrind with
376
default options
377
378
.. option:: --valgrind-all
379
380
Synonym for :option:`--valgrind`
381
382
.. option:: --valgrind-drizzleslap
383
384
Run "drizzleslap" with valgrind.
385
386
.. option:: --valgrind-drizzletest
387
388
Run the *drizzletest* and *drizzle_client_test* executable with valgrind
389
390
.. option:: --valgrind-drizzled
391
392
Run the "drizzled" executable with valgrind
393
394
.. option:: --valgrind-options=ARGS
395
396
Deprecated, use :option:`--valgrind-option`
397
398
.. option:: --valgrind-option=ARGS
399
400
Option to give valgrind, replaces default option(s),
401
can be specified more then once
402
403
.. option:: --valgrind-path=[EXE]
404
405
Path to the valgrind executable
406
407
.. option:: --callgrind
408
409
Instruct valgrind to use callgrind
410
411
.. option:: --massif
412
413
Instruct valgrind to use massif
414
415
Misc options
416
------------
417
418
.. option:: --comment=STR
419
420
Write STR to the output
421
422
.. option:: --notimer
423
424
Don't show test case execution time
425
426
.. option:: --script-debug
427
428
Debug this script itself
429
430
.. option:: --verbose
431
432
More verbose output
433
434
.. option:: --start-and-exit
435
436
Only initialize and start the servers, using the
437
startup settings for the specified test case (if any)
438
439
.. option:: --start-dirty
440
441
Only start the servers (without initialization) for
442
the specified test case (if any)
443
444
.. option:: --fast
445
446
Don't try to clean up from earlier runs
447
448
.. option:: --reorder
449
450
Reorder tests to get fewer server restarts
451
452
.. option:: --help
453
454
Get this help text
455
456
.. option:: --testcase-timeout=MINUTES
457
458
Max test case run time (default 15)
459
460
.. option:: --suite-timeout=MINUTES
461
462
Max test suite run time (default 180)
463
464
.. option:: --warnings | log-warnings
465
466
Pass --log-warnings to drizzled
467
468
.. option:: --sleep=SECONDS
469
470
Passed to drizzletest, will be used as fixed sleep time
471
472
Loggerhead is a web-based interface for Breezy
Version: 2.0.1