Monthly Archives: January 2011

Posted on by

Introducing dbqp.py aka project_steve_austin

So, what is all this dbqp / steve_austin mess?
Short answer – it is a nascent test runner.  Currently it duplicates the functionality of test-run.pl, but it is implemented in what I hope to be a more adaptable and reusable manner.  Please note that I said ‘duplicate’.  We haven’t touched test-run.pl or changed anything about how that tool works.  We don’t want to shoot ourselves in the foot by breaking a currently useful tool.

steve_austin…really?
Yes, it is test-run.pl, but better…stronger…faster ; )

dbqp?
database quality platform.  The idea is that this isn’t meant to just run a single type of test.  The code is intended to facilitate the execution of other tests.

Why?
Let me give you an example – any test of a database that I can think of requires starting up one or more servers (not much to test otherwise).  However, no project provides a standardized way of allocating and managing servers for tests (until now).  test-run.pl has a start-and-exit option (so does dbqp), but it seems crazy to have to call an entirely separate tool just to start a server (and such a method provides no management or cleanup – it is on the individual test to deal with that).  To run the randgen on Drizzle, one must start a server with ./test-run –start-and-exit, then call the randgen, then shutdown the server when finished : (  The situation is slightly better for MySQL / MariaDB, but the ./runall script the randgen uses still calls mysql-test-run.  Other tools define their own startup and management code, but such code has never seemed easily re-used to me.  This isn’t meant to be a knock on anyone or anything.  It is simply my belief that ‘things can be better‘ ™.

With dbqp, I have separated server allocation / management, system management (setting environment vars, finding files, etc), and test managment.  By structuring things in this way, we can define different testing modes.  Each mode uses some combination of adjusting / searching the system, starting and managing servers, doing things to the servers, then evaluating the effect of these ‘things’.  At present we only have a ‘dtr’ mode which runs our standard test suite, but further modes can be defined by simply writing a custom testManager (what does a testcase look like and how to package the test info) and testExecutor (how to execute / evaluate a testcase).  This will allow us to do things like create a ‘randgen’ mode which will facilitate executing our randgen transaction log tests (which currently involve too many manual steps outside of our automated build system) and do other cool things.

It should be noted that this is what the current test-run.pl basically does – it looks for testcases, starts the servers required by the test, packages the data and calls drizzletest.cc to actually execute the test (same for mysql-test-run.pl and mysqltest.cc).  It just makes sense to redo this in a way that will allow us to do the same thing for other types of testing tools : )

How can I play with it?
It is in the tree.  There is a make target – ‘make test-dbqp’.  I have tested it on Ubuntu and Mac OS X and it takes ~5 minutes to run our tests (all 551 of them!)
The manual page is here.

Please note that it is still in development.  It is reasonably solid and is beta quality, but test-run.pl is still in the tree for a reason ; )  All tests currently pass and one can run a variety of test / suite combinations as well as record tests.  Other utility features such as gdb and valgrind are still being polished.  Rely on test-run.pl for now, play with dbqp.
I’m also still adding and improving things as I write this – many, many thanks to Lee Bieber and Vijay Samuel for being early guinea pigs.

Anything else?
I’d like to give credit where credit is due and thank Monty Taylor for proposing the GSOC project that got me started on this path.  He gets at least part of the credit for any goodness this may bring (all responsibility for anything FAIL is on my shoulders).

There are some interesting things to come – we’re also moving on to the next phase of replication testing now that our trx log is rock solid

Posted on by

Drizzle’s transaction log is passing all tests!

In case you missed it here, we are very proud to announce that Drizzle’s transaction log is passing all of our tests.  For quite some time, David Shrewsbury, Stewart Smith, and Joe Daly have been putting a lot of love into the log code.  Please don’t be fooled by Dave’s praise of QA now that the storm has passed…you should have heard the names he called me and the things he plotted when we were rooting these bugs out ; )  However, now that there is a permanent record of his words, I’ll be reminding him about this post the next time my testing becomes a pain in his posterior and I feel him giving me the stink-eye in IRC (heheh)

With that said, we really have been putting tons of effort into making the log rock-solid.  This code will serve as the foundation for Drizzle replication and we can now be assured that any replication solutions will have a reliable log that will reflect the state of the server.  We have been beating up the code with the random query generator.  We have concocted several grammars that throw a variety of queries, in transactions and standalone, at the server.  After we have made the master do some work – we use a variety of connections and per-connection query counts, we produce SQL from the log, populate a validation server with the SQL, then compare Drizzledump files to ensure they match.  You should really check out the transaction_reader utility in drizzle/plugin/transaction_log/utilities – it allows a user to view the raw trx log contents, produce SQL from the log contents, and a few other neat tricks.  I’ll be blogging a bit more about how we used this for testing and troubleshooting very soon.

The transaction log worked well in most cases, the majority of our problems were in rooting out strange behavior around deadlocks and rolled back transactions.  I’d like to once again thank Stewart Smith, of the spork most flaming, for his valued assistance in finding these annoying bugs : )  If anyone wants to take a look at the bugs we’ve killed – you can check them out here.

We are far from being done – some of our next tasks include testing RabbitMQ, tweaking randgen tests to make certain we are totally crash-safe, and a few other things.  In the meantime, our randgen trx log tests run against every branch we intend to merge into trunk, so we’re keeping a close eye on making sure it stays solid.  We’ll keep you posted as our replication testing moves along.  Please keep trying Drizzle and helping us to improve it with your bug reports and feedback.