Monthly Archives: August 2010

Posted on by

Where to find randgen testing information / documentation

I know I write a lot about testing with the randgen. In case anyone is interested, I wanted to share some links to randgen documentation and information. It really is an amazing and powerful tool and has found many bugs in MySQL, MariaDB, and Drizzle.

  • It can generate queries faster than anyone could ever think of them.  This provides breadth of input testing and allows for stress testing (multi-threaded scenarios are supported)
  • A number of Validators are available to determine query correctness.  These include server-server comparison validators for testing result sets, execution time comparison validators for comparing query execution time, and others.  Custom validators are implemented fairly easily
  • Grammar-based testing allows for faster generation of test queries.  Rather than thinking of individual queries, one can focus on ‘patterns’ of queries and have the randgen generate hundreds of queries for the same amount of time and effort.

With that little sales pitch out of the way, here are the links to documentation.

  • MySQL Forge – general documentation
  • Drizzle wiki – information on how to run randgen tests and what to look for
  • DrizzleQA – links to information about all Drizzle testing

I will be writing more about how to use this tool in the future, but I’d definitely welcome feedback on what people are interested in.  The tool has applications for testing patches for regressions, validating upgrades based on current workloads, stress testing, etc. – feel free to let me know what you’d like to learn about.  Discussion is always welcome – stop by IRC (#drizzle) or drop us a line on our mailing list.

Posted on by

Testing restoration from drizzledump files

Currently, we have a lot of uses for drizzledump + restore:  upgrading Drizzle, backing up data, etc.  Since we are putting people in a position to rely on these tools, it seemed prudent that we improve our testing in this area.


Our plan was to include both test-suite tests as well as more vigorous testing with the randgen.  The test-suite tests are pretty straightforward and are in tests/t/drizzledump_restore.test.  You can get Drizzle here to check them out if reading test cases is your idea of fun : )  Of course, the randgen tests are the ones that do the interesting, high-volume stuff.


In a nutshell, the randgen test randomly generates any number of test tables in a database. The test tables have random composition (columns and datatypes used) and data population.  Once the tables have been generated, we call drizzledump, restore the tables from the dump file to a new database, then do a table-to-table comparison of the original and restored tables.  We report error and debugging information if we find a difference, and go on our happy way otherwise.


This test is now a part of our regular build and test system and runs on every push to trunk.  We make use of a standard, repeatable set of queries and we also make use of the randgen’s ability to randomize data and queries and use –seed=time for a smaller run of the same test.  This gives us a larger spread of tests each run.  Fortunately, the randgen is nice enough to report the seed value for repeatability in case we encounter a bug.
# 2010-08-17T21:32:20 Converting –seed=time to –seed=128208074
Feeding the randgen the same seed value for a grammar / gendata file combination will produce the same tables, data, and queries each time.


I’m happy to report that the test currently passes with flying colors.  Interestingly enough, I discovered what happens when you try to have several clients running drizzledump, creating and altering test tables, and restoring data from dumpfiles and it isn’t pretty.  Please note that our test is designed to run with only a single thread at a time : )


If anyone is interested in testing or anything else related to Drizzle, stop by IRC (#drizzle) or drop us a line on our mailing list.
Posted on by

Testing Drizzle with the random query generator

As some of you may know, part of Drizzle’s testing process involves the use of the random query generator (aka randgen aka RQG).  Any patches that want to make it into trunk must survive several randgen tests, among other things.

One of the first things I did upon joining the Drizzle team at Rackspace was to take a look at the RQG tests running in Hudson.  My evaluation included tuning up the existing tests and modifying some MySQL test grammars for use with Drizzle.  It should be noted that randomness is a double-edged sword – it is extremely powerful for finding bugs, but it also requires a certain amount of diligence to produce useful, valid queries.  Once my initial work was complete, we unleashed the new RQG tests and found the following bugs:

At the time of this writing, 6 of these are in ‘Fix Released’ status and 3 more are in work.  I’d like to extend a personal “Thank you” to Prafulla Tekawade for his great work – he has supplied the patches for 5 out of the 6 fixed bugs (!).  Additionally, thanks for Toru Maesaka for his continued work on BlitzDB (which is part of our regular randgen testing).

If anyone is interested in learning more about using the random query generator, I encourage you to check out launchpad.  General information can also be found here and here