EXECUTE testing with randgen Transforms

A little while ago, we added EXECUTE functionality to the server.  This lets us either:

  • EXECUTE “SELECT * FROM TEST_TABLE”;
  • SET @var = “SELECT * FROM TEST_TABLE” ; EXECUTE @var;

We have added a new suite to our test tree and we’ve also started testing this with the randgen.  The random query generator allows for code known as a Validator.  This code executes on a per-query basis and can do whatever you need it to in order to verify a query.  Some examples:

  • ResultSetComparator – which compares result sets between two different servers for the same query.  Useful for checking result set correctness against a trusted validation server.  This tool, combined with the optimizer* grammars quickly found over 30 optimizer bugs in MySQL >:-)
  • ExecutionTimeComparator  – compares execution times between two servers.  This is useful when checking a patch for a regression, especially in the optimizer.

There is a special type of Validator known as a Transformer.  There are various Transforms that can be used on a query.  The randgen will alter the query in some way (a Transform).  Each Transform states how the result set should relate to the result set of the original query, for example a TRANSFORM_OUTCOME_SUBSET is expected when tacking on a LIMIT clause.  Some Transforms:

  • ExecuteAsSPTwice – This takes the original query, creates a MySQL stored procedure from it, then executes it twice in a row.  This was developed due to a MySQL bug
  • InlineSubqueries – Converts SELECT…WHERE col_int IN (SELECT col_int…) -> SELECT …WHERE IN (1,3,5) i.e. the actual values returned from the subquery

For Drizzle, we have created two new Transforms.  For each SELECT query generated by a given grammar, the randgen EXECUTE’s it as a string and as a variable.  I’m happy to report that the tests are passing with flying colors and will be added to our automated tests.

It was incredibly easy to create these new Transforms for the randgen.  Now, we get to try the functionality out against every SELECT we can generate via the randgen – we get to cover a lot more ground this way versus trying to craft these tests by hand (though we have added several such tests as previously noted).

Anyway, please feel free to kick the tires on this feature.  I leave it to you to check out EXECUTE…CONCURRENT ; )

3 thoughts on “EXECUTE testing with randgen Transforms

  1. Hi Jobin,

    We did indeed. There are still a few outstanding optimizer bugs, but these are edge cases. The majority of the bugs that we found in the optimizer were quickly killed by Prafulla Tekawade with his awesome patches : )

    You can read more about what we found / have killed here:
    http://www.wc220.com/?p=13

  2. Pingback: Patrick Crews: EXECUTE testing with randgen Transforms | Weez.com

Comments are closed.