Pete’s Blog: Benchmarking the Benchmarks
In the world of low latency, it seems that benchmarks are headline news. Having readily available figures showing xxx microseconds and yyy hundreds of thousands of updates per second is a pretty sure way to get some press coverage for one’s product. Indeed, I find myself asking of vendors who are pushing their new datafeed handler, or complex event processing engine, “So, got any benchmarks for it?”
Until recently, the only game in town for benchmarks was Peter Lankford’s imposingly named (and cleverly named) Securities Technology Analysis Center (STAC), which makes a business out of running independent benchmarks for vendors, detailing the exact software and hardware components (or stack) used in the benchmark test.
Of course, while STAC is independent, not all of the benchmarks it runs are published. If the results don’t stack up so well (excuse the pun), then the vendor sponsoring the benchmark isn’t likely to make them public. So the news from STAC, while authentic and useful, is really just the good news. Indeed, STAC’s Lankford does point out that the published benchmarks should augment, not replace, specific benchmarks conducted by end users.
Intel has now come on stream with its own Low Latency Lab, with Streambase partner Datastream Analysis making use of it to benchmark calculations for algorithmic trading. Again, useful data to have. But since the lab is there to assist partners in porting their applications to the Intel architecture, one expects that any published results will show such endevours in good light.
It’s probably a pipe dream to expect any independent body to emerge that will really shake down low latency components and publish the results - the good, the bad and the ugly. For one thing, I suspect that support from vendors would be difficult to obtain.
But perhaps the next step might be the creation of some common benchmark standards, with input from financial markets practitioners, that would allow vendors, independent testers and end users to perform and compare results. Useful, and newsworthy, no?
Until next time … here’s some good music.
Technorati Tags: stac, securities technology analysis center, intel, data stream analysis, dsal, benchmarks
July 23rd, 2007 at 11:01 am
Thanks for the plug, Pete. Just wanted to let you and your readers know that help is on the way. As I write, STAC is working with capital markets customers and vendors to define standard benchmarks for a broad range of workloads. When the 1.0 spec is available, we’ll email everyone who has subscribed at www.STACresearch.com, so stay tuned.
Our initial focus is market data feed handling and distribution, followed closely by complex event processing (what STAC prefers to call “event processing platforms” or EPP, but I’ll save that for another blog). We eventually intend to develop standards based on other capital markets workloads such as execution and risk management–but one step at a time.
As for vendor support, two of the leading vendors in market data distribution (Reuters and Wombat) publicly voiced support for STAC developing standard benchmarks at last year’s well-attended High Performance on Wall Street event. I think you were there, Pete.
And every single EPP vendor with which I have discussed the plan (6 or 7 of them) has pledged its support. We also have the backing of many vendors below the application layer (OS, hardware, network, etc.). Intel, in fact, were the ones who sparked the public conversation about market data benchmarks at the High Performance show.
So, standard benchmark specifications are coming soon. But as you suspect, getting someone to invest the time and money to run those benchmarks periodically across a range of products without funding from vendors is a bigger challenge. Yes, we do this kind of thing privately for customers. In fact, customers can come to a multi-vendor STAC Lab like the one we just opened near Wall Street and work with us to compare products first hand in a standardized environment. But trading firms tend not to want to share these results with their competitors. So conducting public bakeoffs is an economic puzzle. Running a rigorous benchmark costs real money, and STAC distributes benchmark reports for free. We’ve considered creating a paid subscription service that would fund testing without vendor backing. However, we have not thought it would generate enough money to cover the costs. Getting vendors to support the release of benchmark data would also be more difficult in this model. I would be happy to be wrong about this and would encourage anyone who is interested in this model to email me at peter.lankford@STACresearch.com.
Meanwhile, I think that the public benchmark business we’re involved in has a lot of benefit for the industry, by encouraging vendors to out-perform each other. In fact, vendors are forming coalitions in order to maximize the performance of entire technology stacks. That’s exactly what customers want. These projects involve real research into how to get different layers of a solution to work together optimally. Like drug research, sometimes it succeeds, and often it fails. Yes, we report the successes, but careful readers will note that our reports document plenty of warts, too.
Thanks,
Peter Lankford
Director, STAC