So, Brian wrote a bug the other day, asking me to do more testing of Drizzle’s data dictionary.  Specifically, we wanted to look for how things behaved in a concurrent environment as this is often a killer for table functions / what people are most likely to forget.

What we came up with was the following plan:

1)  Generate a test that only looked at data dictionary tables, with several users generating the same queries.

2)  If step 1 looks good, we will slowly introduce background workloads (SELECT / UPDATE / etc) while we continue with the workload from step 1.

This resulted in a couple of new randgen grammars:

data_dict_concurrent_drizzle.yy – this grammar generates nothing but queries against the data dictionary tables.  At present, these are mostly of the variety:

• SELECT * FROM data_dictionary_table
• SHOW PROCESSLIST | VARIABLE | TABLE STATUS | etc

This is designed to stress the data dictionary, either alone or with another randgen process generating a background workload.

proclist_subquery_drizzle.yy – this grammar is the same as optimizer_subquery_drizzle (generating *nasty*, subquery-heavy SELECTs), but also allows for SHOW PROCESSLIST commands. This is mainly designed to stress the server / PROCESSLIST.  This grammar is nice as it is a single test that can just be run with several threads.

I am happy to report that in a data dictionary-only environment, the server was able to handle things very well.  I was running up to 100 connections, 100k queries per connection and things looked good.

The other tests are another matter.  While these are somewhat simple tests, they have proven highly effective so far:

• Bug #627733: Crash in InnodbTrxTool::Generator::populate_innodb_locks (this=0x7f26140046f0) at plugin/innobase/handler/data_dictionary.cc:269
• Bug #627742: Assertion failed – in drizzled::plugin::TableFunction::Generator::push (this=0x23876c0, arg=<value optimized out>, length=<value optimized out>) at drizzled/plugin/table_function.cc:185
• Bug #628398: Crash / segfault in copy_fields (join=0x284f868, end_of_records=false) at drizzled/sql_select.cc:6228
• Bug #628891: Crash / assertion failed – in drizzled::Diagnostics_area::set_eof_status (this=0x7f9f3c2c4258, session=0x7f9f3c2c3b10) at drizzled/diagnostics_area.cc:120

For each of these bugs, data dictionary queries were being executed while another query was also being processed.  It should be noted that our newest team member, Andrew Hutchings, had Bug#627742 fixed in less than 24 hours : )

I still have a few more scenarios to run through, but it appears that we have shaken out most of the bugs in this area.  Our next steps will be to install such tests in our build and test system to prevent regressions / catch new bugs and to fix the remaining crashes noted above.