Rulebase Systems Benchmarks 2009

Greetings:

With the forthcoming First Quarter Benchmarks approaching, I have been getting things set up to do the dreaded "Waltz" benchmarks. For the spreadsheet vendors, this holds no fear whatsoever since they can't really do this kind of thing using decision tables. But for vendors of tools such as Blaze Advisor, JRules, Jess, Drools, CLIPS, CLIPS/R2 and OPSJ this means, once again, submitting yourself to the never-ending battle between engineering and marketing.

Daniel Selman came up with a suggestion: Why not do something like http://www.spec.org/jAppServer2004/ ?? I took a look and, quite frankly, I'm not impressed for several reasons. (1) Each vendor has to be "trusted" to run the benchmark and report accurate results. I'm just too old and too jaded to trust vendors not to be swayed by marketing and the CxO gang to fudge on their reports and then dare anyone to challenge them on the results. (2) Application Server benchmarks are very generic and do not test the complexity of a rulebased system engine. (3) The results are confusing in the plethora of engines, number of cores used, etc. so that it's difficult to declare a "winner." Maybe this is what the marketing guys love about it; everyone is a winner because you can claim almost anything.

Here's the problem as I see it: If I claim to be 2.5M tall and nobody has a way to measure what I'm saying then how can you dispute it? Or, if everyone has their own measuring stick, then everyone can claim whatever they wish. I still favor one source, one measuring stick (however flawed) and one clear "winner" with rankings. The source is still open and anyone can do whatever they like in the way of challenging, but at least we would know who was fastest on whatever test we ran.

Now, all I'm asking is that you take a look at the SPEC benchmarks and see what you think. Most of you are already "approved" bloggers on this site so don't comment, just blog. If you have not been approved, just drop me an email and I'll put you on the list. But, let's get this done and out of the way before the end of the year if possible.

SDG
jco

RuleBased Forecasting

Greetings:

With the collaboration of three most excellent colleagues I have returned to the world of Quantitative Analysis and Forecasting. Finally!! Monica Adya was kind enough to send me the 1992 code that she did for seminal work that was co-authored by Fred Collopy and J. Scott Armstrong in 1992. It's all in C++ but I hope to begin by transferring to a combination of either Drools (or JRules or Advisor or OPS or JessJ or whatever) and Java. Or, perhaps, just leave the processing in C++ and abstract out the logic into CLIPS or ILOG Rules. (ILOG Rules is NOT free and, to my knowledge, they don't have a six-month version like JRules.)

I'm rather leaning to Drools and Java since (1) both are free and (2) it would be easier for the domain expert forecasters to read the reasoning in Drools than in C++ or Java. It would be nice if ILOG JRules or Rules, OPSJ or Blaze Advisor had some variant of an academic license that we could use for research. Although, JRules does have a six-month license that I might be able to keep extending for a while. Drools, JRules and OPSJ all have a similar syntax while CLIPS (C/C++) and Jess (Java) share a virtually identical syntax for the rules.

Probably I'll do the first cut in Drools and Java and see how that goes. If performance is adequate, meaning not over a minute or two to process large quantities of data, then I'll leave it alone. If it takes too long then maybe I'll move it to ILOG Rules or CLIPS.

Thanks again to Monica for having the foresight to have saved all of the old C++ stuff for so many years. :-)

SDG
jco

Comment by Richard Halsey

Richard Halsey, otherwise known as "Bubba De Kat" posted this as a comment but I thought that it should be up front for all to see. Enjoy:
=================
This looks interesting.

My area of interest lies with Machine Cognition. I have built a number of these systems and incorporated discrete rules, fuzzy rules, and Neural Nets in order to produce an agent-based system.

However, the last time I worked with this stuff was about 3 to 4 years ago - I retired to living along the Gulf of Mexico coast in Florida. I'll have to re-think where I was.
===============

Welcome

Greetings:

Welcome to the inauguration of the Expert Systems Consulting Group, ExSCG. (Everyone has a three- or four-letter acronym so we went for broke on five.) This site is for those who have "been there and done that" with any BRMS tool, rulebase engine, neural net, forward or backward chaining, whatever... Whether corporate or independent, if you would like to blog here or have us link to your blog, let us know. While we do not have a corporate tie-in, we certainly DO encourage those from the corporate world to blog with us.

Of special interest will be "non-business" applications; we already have plenty of sites that cater to the BRMS (Business Rule Management Systems) world so we certainly don't need another one. This site will be those who have a common interest; that of furthering the expertise, knowledge and experience of any kind of expert system, whether it be neural nets, Prolog or rulebased systems.

The rules here are simple - keep it technical, keep it interesting. No denigrating any vendor with random snide remarks but rather to point out both the pros and cons of any product BUT with substantiated testing. Anyone can comment, even anonymously, and experienced consultants will be invited bloggers. (Bloggers have to join Google for some reason.)

Hopefully, this will be a central point for Windows and Unix/Mac users, C/C++, C@, C#, Java and even COBOL rulebased and neural net advocates, corporate and independent consultants and architects, pretty much anything that is technical and AI. Who know? This might be the harbinger of a return to C/C++ rulebased systems.

BTW, if you did or did not attend the October Rules Fest in Dallas in October of 2008, be sure to visit http://ORF2009.blogspot.com to help plan the next episode.

SDG
jco
KnowledgeBased Systems Corporation
jameso@kbsc.com.

SDG

jco

jameso@kbsc.com