Rulebase Systems Benchmarks 2009

Greetings:

With the forthcoming First Quarter Benchmarks approaching, I have been getting things set up to do the dreaded "Waltz" benchmarks. For the spreadsheet vendors, this holds no fear whatsoever since they can't really do this kind of thing using decision tables. But for vendors of tools such as Blaze Advisor, JRules, Jess, Drools, CLIPS, CLIPS/R2 and OPSJ this means, once again, submitting yourself to the never-ending battle between engineering and marketing.

Daniel Selman came up with a suggestion: Why not do something like http://www.spec.org/jAppServer2004/ ?? I took a look and, quite frankly, I'm not impressed for several reasons. (1) Each vendor has to be "trusted" to run the benchmark and report accurate results. I'm just too old and too jaded to trust vendors not to be swayed by marketing and the CxO gang to fudge on their reports and then dare anyone to challenge them on the results. (2) Application Server benchmarks are very generic and do not test the complexity of a rulebased system engine. (3) The results are confusing in the plethora of engines, number of cores used, etc. so that it's difficult to declare a "winner." Maybe this is what the marketing guys love about it; everyone is a winner because you can claim almost anything.

Here's the problem as I see it: If I claim to be 2.5M tall and nobody has a way to measure what I'm saying then how can you dispute it? Or, if everyone has their own measuring stick, then everyone can claim whatever they wish. I still favor one source, one measuring stick (however flawed) and one clear "winner" with rankings. The source is still open and anyone can do whatever they like in the way of challenging, but at least we would know who was fastest on whatever test we ran.

Now, all I'm asking is that you take a look at the SPEC benchmarks and see what you think. Most of you are already "approved" bloggers on this site so don't comment, just blog. If you have not been approved, just drop me an email and I'll put you on the list. But, let's get this done and out of the way before the end of the year if possible.

SDG
jco