« back to all changes in this revision
Viewing changes to docs/testing/test-run.rst
- browse files at revision 1892.4.1
- compare with another revision
- compare with revision 2472
- download diff
- download tarball
- view history from revision 1892.4.1
- Committer: patrick crews
- Date: 2010-11-01 23:18:55 UTC
- mto: (1900.1.2 build)
- mto: This revision was merged to the branch mainline in revision 1901.
- Revision ID: gleebix@gmail.com-20101101231855-1xt4ohcwzyxx1gdh
Added Sphinx docs for test-run.pl
- files added:
- docs/testing
- files modified:
- docs/include.am
added
removed
docs/testing/test-run.rst
1
**********************************
2
test-run.pl - Drizzle testing tool
3
**********************************
4
5
Synopsis
6
========
7
8
**./test-run** [ *OPTIONS* ] [ TESTCASE ]
9
10
Description
11
===========
12
13
:program:`test-run.pl` (aka test-run, dtr, mtr) is used to execute tests
14
from the Drizzle test suite. These tests are included with Drizzle
15
distributions and provide a way for users to verify that the system will
16
operate according to expectations.
17
18
The tests use a diff-based paradigm, meaning that the test runner executes
19
a test and then compares the results received with pre-recorded expected
20
results. In the event of a test failure, the program will provide output
21
highlighting the differences found between expected and actual results; this
22
can be useful for troubleshooting and in bug reports.
23
24
While most users are concerned with ensuring general functionality, the
25
program also allows a user to quickly spin up a server for ad-hoc testing
26
and to run the test-suite against an already running Drizzle server.
27
28
Running tests
29
=========================
30
31
There are several different ways to run tests using :program:`test-run.pl`.
32
33
It should be noted that unless :option:`--force` is used, the program will
34
stop execution upon encountering the first failing test.
35
:option:`--force` is recommended if you are running several tests - it will
36
allow you to view all successes and failures in one run.
37
38
Running individual tests
39
------------------------
40
If one only wants to run a few, specific tests, they may do so this way::
41
42
./test-run [OPTIONS] test1 [test2 ... testN]
43
44
Running all tests within a suite
45
--------------------------------
46
Many of the tests supplied with Drizzle are organized into suites.
47
48
The tests within drizzle/tests/t are considered the 'main' suite.
49
Other suitesare located in either drizzle/tests/suite or within the various
50
directories in drizzle/plugin. Tests for a specific plugin should live in
51
the plugin's directory - drizzle/plugin/example_plugin/tests
52
53
To run the tests in a specific suite::
54
55
./test-run [OPTIONS] --suite=SUITENAME
56
57
Running all available tests
58
---------------------------
59
Currently, the quickest way to execute all tests in all suites is
60
to use 'make test' from the drizzle root.
61
62
Otherwise, one should simply name all suites::
63
64
./test-run [OPTIONS] --suite=SUITE1, SUITE2, ...SUITEN
65
66
Interpreting test results
67
=========================
68
The output of the test runner is quite simple. Every test should pass.
69
In the event of a test failure, please take the time to file a bug.
70
71
At the end of a run, the program will provide the user with a listing
72
of:
73
* how many tests passed
74
* how many tests failed
75
* a listing of failing tests
76
* total time spent executing the tests
77
78
Example output::
79
80
--------------------------------------------------------------------------------
81
Stopping All Servers
82
Failed 10/231 tests, 95.67% were successful.
83
84
The log files in var/log may give you some hint
85
of what went wrong.
86
If you want to report this error, go to:
87
http://bugs.launchpad.net/drizzle
88
The servers were restarted 16 times
89
Spent 64.364 of 115 seconds executing testcases
90
91
drizzle-test-run in default mode: *** Failing the test(s): main.exp1
92
main.func_str main.loaddata main.null main.outfile main.subselect3
93
main.warnings jp.like_utf8 jp.select_utf8 jp.where_utf8
94
95
Additional uses
96
===============
97
Starting a server for manual testing
98
------------------------------------
99
100
:program:`test-run.pl` allows a user to get a Drizzle server up and running
101
quickly. This can be useful for fast ad-hoc testing.
102
103
To do so call::
104
105
./test-run --start-and-exit [*OPTIONS*]
106
107
..This will start a Drizzle server that you can connect to and query
108
109
Starting a server against a pre-populated DATADIR
110
--------------------------------------------------
111
112
Using :option:`--start-dirty` prevents :program:`test-run.pl` from attempting
113
to initialize (clean) the datadir. This can be useful if you want to use
114
an already-populated datadir for testing.
115
116
Program architecture
117
====================
118
119
:program:`test-run.pl` uses a simple diff-based mechanism for testing.
120
It will execute the statements contained in a test and compare the results
121
to pre-recorded expected results. In the event of a test failure, you
122
will be presented with a sample diff::
123
124
main.exp1 [ fail ]
125
--- drizzle/tests/r/exp1.result 2010-11-02 02:10:25.107013998 +0300
126
+++ drizzle/tests/r/exp1.reject 2010-11-02 02:10:32.017013999 +0300
127
@@ -5,4 +5,5 @@
128
a
129
1
130
2
131
+3
132
DROP TABLE t1;
133
134
A test case consists of a .test and a .result file. The .test file includes
135
the various statements to be executed for a test. The .result file lists
136
the expected results for a given test file. These files live in tests/t
137
and tests/r, respectively. This structure is the same for all test suites.
138
139
test-run.pl options
140
===================
141
142
The :program:`test-run.pl` tool has several available options:
143
144
./test-run [ OPTIONS ] [ TESTCASE ]
145
146
Options to control what engine/variation to run
147
-----------------------------------------------
148
149
.. option:: --compress
150
151
Use the compressed protocol between client and server
152
153
.. option:: ----bench
154
155
Run the benchmark suite
156
157
.. option:: --small-bench
158
159
Run the benchmarks with --small-tests --small-tables
160
161
Options to control directories to use
162
-------------------------------------
163
164
.. option:: --benchdir=DIR
165
166
The directory where the benchmark suite is stored
167
(default: ../../mysql-bench)
168
169
.. option:: --tmpdir=DIR
170
171
The directory where temporary files are stored
172
(default: ./var/tmp).
173
174
.. option:: --vardir=DIR
175
176
The directory where files generated from the test run
177
is stored (default: ./var). Specifying a ramdisk or
178
tmpfs will speed up tests.
179
180
.. option:: --mem
181
182
Run testsuite in "memory" using tmpfs or ramdisk
183
Attempts to find a suitable location
184
using a builtin list of standard locations
185
for tmpfs (/dev/shm)
186
The option can also be set using environment
187
variable DTR_MEM=[DIR]
188
189
Options to control what test suites or cases to run
190
---------------------------------------------------
191
192
.. option:: --force
193
194
Continue to run the suite after failure
195
196
.. option:: --do-test=PREFIX or REGEX
197
198
Run test cases which name are prefixed with PREFIX
199
or fulfills REGEX
200
201
.. option:: --skip-test=PREFIX or REGEX
202
203
Skip test cases which name are prefixed with PREFIX
204
or fulfills REGEX
205
206
.. option:: --start-from=PREFIX
207
208
Run test cases starting from test prefixed with PREFIX
209
suite[s]=NAME1,..,NAMEN Collect tests in suites from the comma separated
210
list of suite names.
211
The default is: "main,jp"
212
213
.. option:: --skip-rpl
214
215
Skip the replication test cases.
216
combination="ARG1 .. ARG2" Specify a set of "drizzled" arguments for one
217
combination.
218
219
.. option:: --skip-combination
220
221
Skip any combination options and combinations files
222
223
.. option:: --repeat-test=n
224
225
How many times to repeat each test (default: 1)
226
227
Options that specify ports
228
--------------------------
229
230
.. option:: --master_port=PORT
231
232
Specify the port number used by the first master
233
234
.. option:: --slave_port=PORT
235
236
Specify the port number used by the first slave
237
238
.. option:: --dtr-build-thread=#
239
240
Specify unique collection of ports. Can also be set by
241
setting the environment variable DTR_BUILD_THREAD.
242
243
Options for test case authoring
244
-------------------------------
245
246
.. option:: --record TESTNAME
247
248
(Re)genereate the result file for TESTNAME
249
250
.. option:: --check-testcases
251
252
Check testcases for sideeffects
253
254
.. option:: --mark-progress
255
256
Log line number and elapsed time to <testname>.progress
257
258
Options that pass on options
259
----------------------------
260
261
.. option:: --drizzled=ARGS
262
263
Specify additional arguments to "drizzled"
264
265
Options to run test on running server
266
-------------------------------------
267
268
.. option:: --extern
269
270
Use running server for tests
271
272
.. option:: --user=USER
273
274
User for connection to extern server
275
276
Options for debugging the product
277
---------------------------------
278
279
.. option:: --client-ddd
280
281
Start drizzletest client in ddd
282
283
.. option:: --client-debugger=NAME
284
285
Start drizzletest in the selected debugger
286
287
.. option:: --client-gdb
288
289
Start drizzletest client in gdb
290
291
.. option:: --ddd
292
293
Start drizzled in ddd
294
295
.. option:: --debug
296
297
Dump trace output for all servers and client programs
298
299
.. option:: --debugger=NAME
300
301
Start drizzled in the selected debugger
302
303
.. option:: --gdb
304
305
Start the drizzled(s) in gdb
306
307
.. option:: --manual-debug
308
309
Let user manually start drizzled in debugger, before running test(s)
310
311
.. option:: --manual-gdb
312
313
Let user manually start drizzled in gdb, before running test(s)
314
315
.. option:: --manual-ddd
316
317
Let user manually start drizzled in ddd, before running test(s)
318
319
.. option:: --master-binary=PATH
320
321
Specify the master "drizzled" to use
322
323
.. option:: --slave-binary=PATH
324
325
Specify the slave "drizzled" to use
326
327
.. option:: --strace-client
328
329
Create strace output for drizzletest client
330
331
.. option:: --max-save-core
332
333
Limit the number of core files saved (to avoid filling up disks for
334
heavily crashing server). Defaults to 5, set to 0 for no limit.
335
336
Options for coverage, profiling etc
337
-----------------------------------
338
339
.. option:: --gcov
340
341
FIXME
342
343
.. option:: --gprof
344
345
See online documentation on how to use it.
346
347
.. option:: --valgrind
348
349
Run the *drizzletest* and *drizzled* executables using valgrind with
350
default options
351
352
.. option:: --valgrind-all
353
354
Synonym for :option:`--valgrind`
355
356
.. option:: --valgrind-drizzleslap
357
358
Run "drizzleslap" with valgrind.
359
360
.. option:: --valgrind-drizzletest
361
362
Run the *drizzletest* and *drizzle_client_test* executable with valgrind
363
364
.. option:: --valgrind-drizzled
365
366
Run the "drizzled" executable with valgrind
367
368
.. option:: --valgrind-options=ARGS
369
370
Deprecated, use :option:`--valgrind-option`
371
372
.. option:: --valgrind-option=ARGS
373
374
Option to give valgrind, replaces default option(s),
375
can be specified more then once
376
377
.. option:: --valgrind-path=[EXE]
378
379
Path to the valgrind executable
380
381
.. option:: --callgrind
382
383
Instruct valgrind to use callgrind
384
385
.. option:: --massif
386
387
Instruct valgrind to use massif
388
389
Misc options
390
------------
391
392
.. option:: --comment=STR
393
394
Write STR to the output
395
396
.. option:: --notimer
397
398
Don't show test case execution time
399
400
.. option:: --script-debug
401
402
Debug this script itself
403
404
.. option:: --verbose
405
406
More verbose output
407
408
.. option:: --start-and-exit
409
410
Only initialize and start the servers, using the
411
startup settings for the specified test case (if any)
412
413
.. option:: --start-dirty
414
415
Only start the servers (without initialization) for
416
the specified test case (if any)
417
418
.. option:: --fast
419
420
Don't try to clean up from earlier runs
421
422
.. option:: --reorder
423
424
Reorder tests to get fewer server restarts
425
426
.. option:: --help
427
428
Get this help text
429
430
.. option:: --testcase-timeout=MINUTES
431
432
Max test case run time (default 15)
433
434
.. option:: --suite-timeout=MINUTES
435
436
Max test suite run time (default 180)
437
438
.. option:: --warnings | log-warnings
439
440
Pass --log-warnings to drizzled
441
442
.. option:: --sleep=SECONDS
443
444
Passed to drizzletest, will be used as fixed sleep time
445
446
Loggerhead is a web-based interface for Breezy
Version: 2.0.1