« back to all changes in this revision
Viewing changes to docs/testing/sql-bench.rst
- browse files at revision 2324.2.5
- compare with another revision
- compare with revision 2473
- download diff
- download tarball
- view history from revision 2324.2.5
- Committer: patrick crews
- Date: 2011-06-08 19:50:00 UTC
- mto: This revision was merged to the branch mainline in revision 2329.
- Revision ID: gleebix@gmail.com-20110608195000-arqp2wpsqifj1v79
Updated docs
- files modified:
- docs/testing/sql-bench.rst
added
removed
docs/testing/sql-bench.rst
6
6
7
7
Description
8
8
===========
9
dbqp's sql-bench mode allows a user to run the sql-bench testing suite.
9
dbqp can take advantage of two testing modes offered by the sql-bench tool.
10
11
The Drizzle code has a copy of this tool set in the tree and the test-runner offers two modes::
12
13
sql-bench modes
14
---------------
15
* sqlbench - runs the entire sql-bench test suite and can take a very long time (~45 minutes)
16
* crashme - runs the crash-me tool and reports failure if any tests should not pass
10
17
11
18
12
19
Requirements
36
43
37
44
38
45
sql-bench / dbqp tests
39
=====================
46
=======================
40
47
41
48
Currently, there are only two sql-bench test cases for dbqp. As one might expect, main.all_sqlbench_tests executes::
42
49
58
65
:option:`--force` is recommended if you are running several tests - it will
59
66
allow you to view all successes and failures in one run.
60
67
61
Running individual tests
62
------------------------
63
If one only wants to run a few, specific tests, they may do so this way::
68
At present, sql-bench output in a work in progress. It does report a simple pass/fail, but we are working on alternate ways of viewing / storing the results (and for other testing modes as well)
64
69
65
./dbqp --mode=sql-bench [OPTIONS] test1 [test2 ... testN]
66
70
67
71
Running all tests within a suite
68
72
--------------------------------
69
Many of the tests supplied with Drizzle are organized into suites.
70
71
The tests within drizzle/tests/randgen_tests/main are considered the 'main' suite.
72
Other suites are also subdirectories of drizzle/tests/randgen_tests.
73
74
To run the tests in a specific suite::
75
76
./dbqp --mode=sql-bench [OPTIONS] --suite=SUITENAME
77
78
Running specific tests within a suite
79
--------------------------------------
80
To run a specific set of tests within a suite::
81
82
./dbqp --mode=sql-bench [OPTIONS] --suite=SUITENAME TEST1 [TEST2..TESTN]
83
84
Calling tests using <suitename>.<testname> currently does not work.
85
One must specify the test suite via the :option:`--suite` option.
86
87
88
Running all available tests
89
---------------------------
90
One would currently have to name all suites, but the majority of the working tests live in the main suite
91
Other suites utilize more exotic server combinations and we are currently tweaking them to better integrate with the
92
dbqp system. The slave-plugin suite does currently have a good config file for setting up simple replication setups for testing.
93
To execute several suites' worth of tests::
94
95
./dbqp --mode=sql-bench [OPTIONS] --suite=SUITE1, SUITE2, ...SUITEN
73
At present, there is only one test case per suite for sqlbench and crashme modes - that is all that is needed for these tools.
74
To execute the sql-bench test suite::
75
76
./dbqp --mode=sqlbench
77
78
To execute the crash-me test suite::
79
80
./dbqp --mode=crashme
96
81
97
82
Interpreting test results
98
83
=========================
105
90
* test status (pass/fail/skipped)
106
91
* time spent executing each test
107
92
108
Example output::
109
110
20110601-191706 ===============================================================
111
20110601-191706 TEST NAME [ RESULT ] TIME (ms)
112
20110601-191706 ===============================================================
113
20110601-191706 readonly.concurrency_16 [ pass ] 240019
114
20110601-191706 max_req_lat_ms: 21.44
115
20110601-191706 rwreqps: 4208.2
116
20110601-191706 min_req_lat_ms: 6.31
117
20110601-191706 deadlocksps: 0.0
118
20110601-191706 tps: 150.29
119
20110601-191706 avg_req_lat_ms: 6.65
120
20110601-191706 95p_req_lat_ms: 7.02
121
20110601-191706 ===============================================================
122
20110601-191706 INFO Test execution complete in 275 seconds
123
20110601-191706 INFO Summary report:
124
20110601-191706 INFO Executed 1/1 test cases, 100.00 percent
125
20110601-191706 INFO STATUS: PASS, 1/1 test cases, 100.00 percent executed
126
20110601-191706 INFO Spent 240 / 275 seconds on: TEST(s)
127
20110601-191706 INFO Test execution complete
128
20110601-191706 INFO Stopping all running servers...
93
Example sqlbench output::
94
95
20110608-135645 ===============================================================
96
20110608-135645 TEST NAME [ RESULT ] TIME (ms)
97
20110608-135645 ===============================================================
98
20110608-135645 main.all_sqlbench_tests [ pass ] 2732007
99
20110608-135645 Test finished. You can find the result in:
100
20110608-135645 drizzle/tests/workdir/RUN-drizzle-Linux_2.6.38_9_generic_x86_64
101
20110608-135645 Benchmark DBD suite: 2.15
102
20110608-135645 Date of test: 2011-06-08 13:11:10
103
20110608-135645 Running tests on: Linux 2.6.38-9-generic x86_64
104
20110608-135645 Arguments: --connect-options=port=9306 --create-options=ENGINE=innodb
105
20110608-135645 Comments:
106
20110608-135645 Limits from:
107
20110608-135645 Server version: Drizzle 2011.06.19.2325
108
20110608-135645 Optimization: None
109
20110608-135645 Hardware:
110
20110608-135645
111
20110608-135645 alter-table: Total time: 42 wallclock secs ( 0.06 usr 0.04 sys + 0.00 cusr 0.00 csys = 0.10 CPU)
112
20110608-135645 ATIS: Total time: 22 wallclock secs ( 4.01 usr 0.26 sys + 0.00 cusr 0.00 csys = 4.27 CPU)
113
20110608-135645 big-tables: Total time: 24 wallclock secs ( 4.16 usr 0.22 sys + 0.00 cusr 0.00 csys = 4.38 CPU)
114
20110608-135645 connect: Total time: 31 wallclock secs ( 6.81 usr 4.50 sys + 0.00 cusr 0.00 csys = 11.31 CPU)
115
20110608-135645 create: Total time: 59 wallclock secs ( 2.93 usr 1.65 sys + 0.00 cusr 0.00 csys = 4.58 CPU)
116
20110608-135645 insert: Total time: 1962 wallclock secs (270.53 usr 66.35 sys + 0.00 cusr 0.00 csys = 336.88 CPU)
117
20110608-135645 select: Total time: 560 wallclock secs (23.12 usr 4.62 sys + 0.00 cusr 0.00 csys = 27.74 CPU)
118
20110608-135645 transactions: Total time: 21 wallclock secs ( 2.43 usr 1.98 sys + 0.00 cusr 0.00 csys = 4.41 CPU)
119
20110608-135645 wisconsin: Total time: 10 wallclock secs ( 2.11 usr 0.52 sys + 0.00 cusr 0.00 csys = 2.63 CPU)
120
20110608-135645
121
20110608-135645 All 9 test executed successfully
122
20110608-135645
123
20110608-135645 Totals per operation:
124
20110608-135645 Operation seconds usr sys cpu tests
125
20110608-135645 alter_table_add 18.00 0.02 0.00 0.02 100
126
20110608-135645 alter_table_drop 17.00 0.02 0.01 0.03 91
127
20110608-135645 connect 2.00 1.02 0.51 1.53 2000
128
<snip>
129
20110608-135645 update_rollback 3.00 0.26 0.23 0.49 100
130
20110608-135645 update_with_key 73.00 6.70 5.23 11.93 300000
131
20110608-135645 update_with_key_prefix 34.00 4.45 2.30 6.75 100000
132
20110608-135645 wisc_benchmark 2.00 1.49 0.00 1.49 114
133
20110608-135645 TOTALS 2865.00 310.26 79.94 390.20 2974250
134
20110608-135645
135
20110608-135645 ===============================================================
136
20110608-135645 INFO Test execution complete in 2735 seconds
137
20110608-135645 INFO Summary report:
138
20110608-135645 INFO Executed 1/1 test cases, 100.00 percent
139
20110608-135645 INFO STATUS: PASS, 1/1 test cases, 100.00 percent executed
140
20110608-135645 INFO Spent 2732 / 2735 seconds on: TEST(s)
141
20110608-135645 INFO Test execution complete
142
20110608-135645 INFO Stopping all running servers...
143
144
Example crashme output::
145
146
20110608-152759 ===============================================================
147
20110608-152759 TEST NAME [ RESULT ] TIME (ms)
148
20110608-152759 ===============================================================
149
20110608-152759 main.crashme [ fail ] 155298
150
20110608-152759 func_extra_to_days=error # Function TO_DAYS
151
20110608-152759 ###
152
20110608-152759 ###<select to_days('1996-01-01') from crash_me_d
153
20110608-152759 ###>2450084
154
20110608-152759 ###We expected '729024' but got '2450084'
155
20110608-152759 func_odbc_timestampadd=error # Function TIMESTAMPADD
156
20110608-152759 ###
157
20110608-152759 ###<select timestampadd(SQL_TSI_SECOND,1,'1997-01-01 00:00:00')
158
20110608-152759 ###>1997-01-01 00:00:01.000000
159
20110608-152759 ###We expected '1997-01-01 00:00:01' but got '1997-01-01 00:00:01.000000'
160
20110608-152759 ###
161
20110608-152759 ###<select {fn timestampadd(SQL_TSI_SECOND,1,{ts '1997-01-01 00:00:00'}) }
162
20110608-152759 ###>1997-01-01 00:00:01.000000
163
20110608-152759 ###We expected '1997-01-01 00:00:01' but got '1997-01-01 00:00:01.000000'
164
20110608-152759
165
20110608-152759 ERROR Failed test. Use --force to execute beyond the first test failure
166
20110608-152759 ===============================================================
167
20110608-152759 INFO Test execution complete in 158 seconds
168
20110608-152759 INFO Summary report:
169
20110608-152759 INFO Executed 1/1 test cases, 100.00 percent
170
20110608-152759 INFO STATUS: FAIL, 1/1 test cases, 100.00 percent executed
171
20110608-152759 INFO FAIL tests: main.crashme
172
20110608-152759 INFO Spent 155 / 158 seconds on: TEST(s)
173
20110608-152759 INFO Test execution complete
129
174
Loggerhead is a web-based interface for Breezy
Version: 2.0.1