dbqp / randgen integration…huzzah!

What is the big deal, you may ask?  Well, read on and all shall be revealed, intrepid reader ; )
As I mentioned an earlier post, our new test-runner – dbqp – allows us to define testing ‘modes’ which all utilize the same system and server management code.

One only has to define a testManager (what does a test look like / how to organize tests) and a testExecutor (how to execute / evaluate a test).  The aim is for dbqp to be a one-stop shop for test execution and to provide a clean and simple way to manage and expand this.

I have just added –mode=randgen to the test-runner.  The random query generator is a significant part of Drizzle’s testing strategy and we use a large number of tests with the tool.  Currently, they are executed and managed by drizzle-automation / Jenkins in our build system, but there has not been an easy way for users to execute these tests (outside of installing / learning drizzle automation or the randgen)…until now >; )

Documentation including requirements and setup instructions for the randgen can be found here
For the cost of about 300 lines of code (the new manager/executor files), we can now execute our randgen tests much the same way as we do our drizzletest suite!

./dbqp –mode=randgen –randgen-path=/home/user/repos/randgen
<snip>
25 Feb 2011 12:14:55 INFO: Using testing mode: randgen
25 Feb 2011 12:14:55 INFO: Processing test suites…
25 Feb 2011 12:14:55 INFO: Found 18 test(s) for execution
25 Feb 2011 12:14:55 INFO: Creating 1 testbot(s)
25 Feb 2011 12:14:55 INFO: Taking clean db snapshot…
25 Feb 2011 12:14:55 INFO: testbot0 server:
25 Feb 2011 12:14:55 INFO: NAME: server0
25 Feb 2011 12:14:55 INFO: MASTER_PORT: 9306
25 Feb 2011 12:14:55 INFO: DRIZZLE_TCP_PORT: 9307
25 Feb 2011 12:14:55 INFO: MC_PORT: 9308
25 Feb 2011 12:14:55 INFO: PBMS_PORT: 9309
25 Feb 2011 12:14:55 INFO: RABBITMQ_NODE_PORT: 9310
25 Feb 2011 12:14:55 INFO: VARDIR: drizzle/tests/workdir/testbot0/server0/var
25 Feb 2011 12:14:55 INFO: STATUS: 1
25 Feb 2011 12:15:00 : ================================================================================
25 Feb 2011 12:15:00 : TEST NAME                                               [ RESULT ]    TIME (ms)
25 Feb 2011 12:15:00 : ================================================================================
25 Feb 2011 12:15:00 : main.basic                                              [ pass ]         4376
25 Feb 2011 12:15:08 : main.blob                                               [ pass ]         7829
25 Feb 2011 12:15:12 : main.collations                                         [ pass ]         3989
25 Feb 2011 12:15:16 : main.combinations                                       [ pass ]         4169
25 Feb 2011 12:15:19 : main.create_drop                                        [ pass ]         3387
25 Feb 2011 12:15:34 : main.drizzledump_restore                                [ pass ]        14726
25 Feb 2011 12:15:42 : main.drizzledump_restore_rand                           [ pass ]         8289
25 Feb 2011 12:15:46 : main.limit_compare                                      [ pass ]         3535
25 Feb 2011 12:15:48 : main.many_indexes                                       [ pass ]         1725
25 Feb 2011 12:15:56 : main.optimizer_subquery                                 [ pass ]         8659
25 Feb 2011 12:16:03 : main.outer_join                                         [ pass ]         6464
25 Feb 2011 12:16:09 : main.outer_join_portable                                [ pass ]         6535
25 Feb 2011 12:16:12 : main.repeatable_read                                    [ pass ]         2906
25 Feb 2011 12:17:42 : main.select_stability_validator                         [ pass ]        90206
25 Feb 2011 12:17:47 : main.subquery                                           [ pass ]         4344
25 Feb 2011 12:19:29 : main.subquery_semijoin                                  [ pass ]       102766
25 Feb 2011 12:19:39 : main.subquery_semijoin_nested                           [ pass ]         9195
25 Feb 2011 12:19:42 : main.varchar                                            [ pass ]         3225
25 Feb 2011 12:19:42 : ================================================================================
25 Feb 2011 12:19:42 INFO: Test execution complete in 286 seconds
25 Feb 2011 12:19:42 INFO: Summary report:
25 Feb 2011 12:19:42 INFO: Executed 18/18 test cases, 100.00 percent
25 Feb 2011 12:19:42 INFO: STATUS: PASS, 18/18 test cases, 100.00 percent executed
25 Feb 2011 12:19:42 INFO: Spent 286 / 286 seconds on: TEST(s)
25 Feb 2011 12:19:42 INFO: Test execution complete
25 Feb 2011 12:19:42 INFO: Stopping all running servers…

Each testcase is a simple .cnf file that defines a few crucial elements:

[test_info]
comment = starts up a master-slave setup.  Still needs some validation work
[test_command]
command = ./gentest.pl –gendata=conf/drizzle/drizzle.zz –grammar=conf/drizzle/optimizer_subquery_drizzle.yy –queries=10 –threads=1
[test_servers]
servers = [[–innodb.replication-log=true],[–plugin-add=slave –slave.config-file=$MASTER_SERVER_SLAVE_CONFIG]]

The test_command is the command line that is passed to the randgen tool itself.  We have created most of these tests and have put together the command lines that serve our testing needs.  By organizing our tests in this fashion, it is now easy for anyone to use the tool to perform meaningful tests.

The test_servers section is interesting – it is a simple list of python lists.  Each sublist contains a set of server options that a server will need.  Servers are started in-order with the first defined server being considered the ‘master_server’.  As we can see in the example to test the new slave plugin, the files are amazingly simple and clean.

Some other added benefits are that dbqp’s –valgrind and –gdb options work the same with randgen tests as they would with drizzletest cases.  Addtionally, we will be able to incorporate the code coverage provided by the randgen in our gcov reports.

One final little feature worth noting is the –gendata option.

The randgen provides a feature called the random data generator / gendata.pl
By passing it a configuration file that defines a set of test tables (see the forge documentation for further details), it will connect to a server, create the test tables, and populate them according to specifications.

While I have other plans for this, one cool use is for ad-hoc testing.  By passing a simple command line + the use of –start-and-exit, one can have a nicely populated test database to play with:

./dbqp –mode=randgen –randgen-path=/home/user/repos/randgen –gendata=/home/user/repos/randgen/conf/drizzle/drizzle.zz –start-and-exit
Setting –no-secure-file-priv=True for randgen mode…
<snip>
25 Feb 2011 13:28:23 INFO: Using testing mode: randgen
25 Feb 2011 13:28:23 INFO: Processing test suites…
25 Feb 2011 13:28:23 INFO: Found 18 test(s) for execution
25 Feb 2011 13:28:23 INFO: Creating 1 testbot(s)
25 Feb 2011 13:28:23 INFO: Taking clean db snapshot…
25 Feb 2011 13:28:23 INFO: testbot0 server:
25 Feb 2011 13:28:23 INFO: NAME: server0
25 Feb 2011 13:28:23 INFO: MASTER_PORT: 9306
25 Feb 2011 13:28:23 INFO: DRIZZLE_TCP_PORT: 9307
25 Feb 2011 13:28:23 INFO: MC_PORT: 9308
25 Feb 2011 13:28:23 INFO: PBMS_PORT: 9309
25 Feb 2011 13:28:23 INFO: RABBITMQ_NODE_PORT: 9310
25 Feb 2011 13:28:23 INFO: VARDIR: drizzle/tests/workdir/testbot0/server0/var
25 Feb 2011 13:28:23 INFO: STATUS: 1
# 2011-02-25T13:28:23 Default schema: test
# 2011-02-25T13:28:23 Executor initialized, id GenTest::Executor::Drizzle 2011.02.2198 ()
# 2011-02-25T13:28:23 # Creating Drizzle table: test.A; engine: ; rows: 0 .
# 2011-02-25T13:28:23 # Creating Drizzle table: test.B; engine: ; rows: 0 .
# 2011-02-25T13:28:23 # Creating Drizzle table: test.C; engine: ; rows: 1 .
# 2011-02-25T13:28:23 # Creating Drizzle table: test.D; engine: ; rows: 1 .
# 2011-02-25T13:28:23 # Creating Drizzle table: test.AA; engine: ; rows: 10 .
# 2011-02-25T13:28:23 # Creating Drizzle table: test.BB; engine: ; rows: 10 .
# 2011-02-25T13:28:24 # Creating Drizzle table: test.CC; engine: ; rows: 100 .
# 2011-02-25T13:28:24 # Creating Drizzle table: test.DD; engine: ; rows: 100 .
25 Feb 2011 13:28:24 INFO: User specified –start-and-exit.  dbqp.py exiting and leaving servers running…
../client/drizzle -uroot -p9306 test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the Drizzle client..  Commands end with ; or \g.
Your Drizzle connection id is 3
Connection protocol: mysql
Server version: 2011.02.2198 Source distribution (dbqp_randgen_updates)
Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the buffer.
drizzle> show tables;
+—————-+
| Tables_in_test |
+—————-+
| A              |
| AA             |
| B              |
| BB             |
| C              |
| CC             |
| D              |
| DD             |
+—————-+
8 rows in set (0.001614 sec)
drizzle> select count(*) from dd;
+———-+
| count(*) |
+———-+
|      100 |
+———-+
1 row in set (0.000614 sec)

Please give it a try if you feel so inclined.  We will be working to integrate this better into our build and test systems (Jenkins / make targets / etc).  And remember ./dbqp –mode=cleanup for quick and easy cleanup when you are done playing! ; )