So, what is all this dbqp / steve_austin mess?
Short answer – it is a nascent test runner. Currently it duplicates the functionality of test-run.pl, but it is implemented in what I hope to be a more adaptable and reusable manner. Please note that I said ‘duplicate’. We haven’t touched test-run.pl or changed anything about how that tool works. We don’t want to shoot ourselves in the foot by breaking a currently useful tool.
Yes, it is test-run.pl, but better…stronger…faster ; )
database quality platform. The idea is that this isn’t meant to just run a single type of test. The code is intended to facilitate the execution of other tests.
Let me give you an example – any test of a database that I can think of requires starting up one or more servers (not much to test otherwise). However, no project provides a standardized way of allocating and managing servers for tests (until now). test-run.pl has a start-and-exit option (so does dbqp), but it seems crazy to have to call an entirely separate tool just to start a server (and such a method provides no management or cleanup – it is on the individual test to deal with that). To run the randgen on Drizzle, one must start a server with ./test-run –start-and-exit, then call the randgen, then shutdown the server when finished : ( The situation is slightly better for MySQL / MariaDB, but the ./runall script the randgen uses still calls mysql-test-run. Other tools define their own startup and management code, but such code has never seemed easily re-used to me. This isn’t meant to be a knock on anyone or anything. It is simply my belief that ‘things can be better‘ ™.
With dbqp, I have separated server allocation / management, system management (setting environment vars, finding files, etc), and test managment. By structuring things in this way, we can define different testing modes. Each mode uses some combination of adjusting / searching the system, starting and managing servers, doing things to the servers, then evaluating the effect of these ‘things’. At present we only have a ‘dtr’ mode which runs our standard test suite, but further modes can be defined by simply writing a custom testManager (what does a testcase look like and how to package the test info) and testExecutor (how to execute / evaluate a testcase). This will allow us to do things like create a ‘randgen’ mode which will facilitate executing our randgen transaction log tests (which currently involve too many manual steps outside of our automated build system) and do other cool things.
It should be noted that this is what the current test-run.pl basically does – it looks for testcases, starts the servers required by the test, packages the data and calls drizzletest.cc to actually execute the test (same for mysql-test-run.pl and mysqltest.cc). It just makes sense to redo this in a way that will allow us to do the same thing for other types of testing tools : )
How can I play with it?
It is in the tree. There is a make target – ‘make test-dbqp’. I have tested it on Ubuntu and Mac OS X and it takes ~5 minutes to run our tests (all 551 of them!)
The manual page is here.
Please note that it is still in development. It is reasonably solid and is beta quality, but test-run.pl is still in the tree for a reason ; ) All tests currently pass and one can run a variety of test / suite combinations as well as record tests. Other utility features such as gdb and valgrind are still being polished. Rely on test-run.pl for now, play with dbqp.
I’m also still adding and improving things as I write this – many, many thanks to Lee Bieber and Vijay Samuel for being early guinea pigs.
I’d like to give credit where credit is due and thank Monty Taylor for proposing the GSOC project that got me started on this path. He gets at least part of the credit for any goodness this may bring (all responsibility for anything FAIL is on my shoulders).
There are some interesting things to come – we’re also moving on to the next phase of replication testing now that our trx log is rock solid