Embedded CEP: An Intelligent Market Data Platform at Last
By Tony McManus, Wombat Financial Software and John Morrell, Coral8
The recent increases in global execution volumes are driving a common need among all market participants for high-speed business logic deployed in ever shrinking timescales. Because of this, real-time analytical event processing is now a cornerstone of trading technology in the multi-asset capital markets sector. In order to create and maintain a competitive advantage, these increasingly sophisticated trading institutions insist that their systems utilize a high-performance, low latency processing platform that is easy to develop, deploy, extend and integrate.
Complex Event Processing (CEP) technology is being hailed by many to be the solution to this new set of challenges. A core capability of any viable CEP solution is that it interacts well with its surrounding messaging infrastructure, which most vendors achieve through an adapter SDK. Built-in integration with stored data (relational and otherwise) is also important to provide access to reference data repositories, historical data or other contextual systems.
Beyond ease of integration, the CEP solution needs to meet a number of other key challenges. On the infrastructure side, it must be able to scale, guarantee high throughput and sub-millisecond latency, and deliver enterprise-class high availability and fault tolerance. And last, but not least, the CEP platform must offer a development environment that offers both a familiar, non-proprietary language and extensibility via common toolkits such as C++ or Java.
The current CEP software products continue to mature, with a number of software providers who could lay justifiable claim to having met the challenges outlined above. However, there is one problem that the majority of the vendors in this space are failing to address – tight integration of the CEP engine directly inside the market data backbone. Only with direct integration into the market data fabric can the CEP and market data infrastructure deliver the true “end-to-end” performance and low-latency characteristics required.
Embedded Adapters: The Critical Performance Factor
All too often when an independent message stream is directed into a CEP engine it is via a loosely-coupled adapter developed either by the customer or the CEP software supplier. In the market data context, the adapter is responsible for subscribing to the exchange data (via a third party direct or aggregated feed), ordering and eliminating duplicate messages, and translating the messages into the target format.
But, as always in low-latency systems, the devil is in the details. The architecture and low level implementation details of the adapter are critical to the success of a CEP-driven market data or trading solution. To maintain ultra low-latency, the adapter tasks must be performed in a matter of micro-seconds.
In the situation where a custom adapter provided by a CEP vendor interfaces with market data provided by a second vendor, the adapter will need to utilize the respective third party APIs. Considering this is a sector where latency is often measured in micro rather than milliseconds then it is imperative to look at: (a) the relationship between the two technology suppliers and (b) how the software products are integrated.
The ideal case is where the market data provider tightly integrates and offers the CEP vendor’s engine, and there is a strategic engineering partnership between the two which is designed to allow them to stream data straight off the wire and into the engine with minimal adapter intervention.
A worrysome scenario is where the vendor relationship and the adapter are “loosely-coupled.” Developing the adapter for extremely low latency requires full awareness of how the APIs can AND should work together, which cannot be achieved without a close engineering relationship. Some CEP vendors only support a loosely-coupled or “out-of-process” adapter model, where the adapter runs in a separate memory space from the CEP engine. Although this model may be more flexible, the increased latency incurred as a result of the extra data serialization and kernel to application space transfer is not tolerable.
Beyond Performance: Additional Benefits
Performance is one very important factor. Tight integration between a CEP engine and the market data infrastructure also enables faster algorithm development, swifter change, and better control of the algorithms and environment. The integrated architecture reduces the number of interfaces a customer would need to build and maintain. It also allows data model, message format and algorithm template metadata to be co-mingled in a common repository. These factors can dramatically speed the development and deployment cycles.
Having a common source of algorithms also makes change easier, development of the execution applications faster, and provides a controlled and auditable means to maintain the algorithms. It is similar to a service-oriented approach where the central CEP service in the market data infrastructure accelerates the deployment of the components around it.
The Support Relationship
Having your market data provided by one vendor and your CEP solution provided by another can unnecessarily complicate the sensitive area of problem resolution. For example, imagine that a development team has determined that the end-to-end latency between receiving the market data off the event bus to when the consequent output is published to target destinations is unacceptable. Who is to blame? Even getting a satisfactory level of communication between the parties is difficult. How difficult will it be to get the solution providers to accept that the problem resides within their domain, and then apply the resources to understand and fix the problem?
A single source for a complete and fully integrated CEP market data solution where a tight partnership exists between the vendors is a key differentiating factor. Managing a vendor relationship over a potentially long system lifecycle is not a trivial task for many customers and having a situation where all issues are owned, managed and maintained by a single point of contact is a luxury that is not to be taken for granted.
In addition, when the parties in a partnership deliver future innovation to their product set it is more likely to be integrated and delivered in a relatively seamless manner than if there is little or no strategic relationship. For example, it is expected that over the next twelve months the most enterprising market data providers will harness the new ultra performant networking architectures such as Infiniband and Remote Data Management Architectures (RMDA). When this revolution occurs how difficult will it be to upgrade the CEP infrastructure to benefit from the potentially huge increases in performance available. The single-source partnership solution will deliver the innovation fastest, and allow those customers able to gain the immediate benefits and competitive advantage.
Conclusion
The capital markets sector is beginning to benefit in a major way from the match of low latency market data provision and complex event processing. The key questions are: how do trading houses that wish to make the most of these opportunities go about evaluating the range of solutions available, and how do they then make an informed decision that will give them a lead on their competitors. Of course both the CEP and the market data solution must each meet the technical requirements that ensure they carry out their respective tasks sufficiently well. However, once these two technology sets are married it is no longer enough that they each perform well in isolation. The key factor now is that they deliver the goods in tandem and this must be guaranteed inherently through close system and company integration.
Technorati Tags: wombat, coral8, cep, messaging middleware
