Interview: Scalable Informatics Teams with Lucera for Low Latency

High frequency traders need extremely low latency hardware to be competitive. To learn more, we caught up with Joe Landman from Scalable Informatics, a leader in high performance solutions.

insideBIGDATA: Joe, you’ve been working with customers who need low-latency from the very beginning. Now you have an bundled an interesting solution together with Lucera in what might be described as a try-and-buy package. Can you tell us more?

Joe Landman

Joe Landman

Joe Landman: In short, we’ve been working with Cantor Fitzgerald’s spin-out Lucera to architect, build, and support a very low latency, very high performance IaaS. They have developed tools to enable an intelligent distributed market, and are offering this extreme performance and advanced FX capability to traders all over the world.

To help get things moving, we’ve put together an offer for Scalable Informatics customers in this space to get access to a dedicated platform with the Lucera tool set and massive cross connection capability, and try it out for a month. The customers get to use the best tools from Lucera on the fastest hardware we build. (Please see offer link, here).

While this is directed at traders, it could help out people looking at other very high performance market big data analytics with live and historical data. That very tight coupling between the market, data, code, and machine makes for a compelling system.

insideBIGDATA: Please enlighten me as to exactly what your company does.

Joe Landman: Scalable informatics is a leading supplier of big data analytical, cloud, and storage products, providing best in class performance and performance density. Using our tightly-coupled converged computing and storage appliance systems, we help customers achieve successful outcomes in their most pressing computational, storage, and cloud projects. Our appliances are simple to deploy and use, with world class support available from us and our partners.

Our siCloud platform, leveraging our Scalable OS, is in use as the basis of Lucera’s cloud platform. It provides the highest performance private cloud in market. We are building massive scale storage systems on our Unison platform for very high performance storage as well as for many PB erasure coded cold storage. Our time series analytics appliance Cadence sets industry standard records.  Our Resonance data warehouse appliance has demonstrated far better performance than competitive systems at a small fraction of the cost of these systems.

Our systems dominate benchmarks we participate in, usually against much larger collections of systems from better known companies.

insideBIGDATA: Who needs this technology?

Joe Landman: I’d say everyone … though for this particular offer, the target users are in trading.

But everything we are building for them is applicable for other high performance use cases. And as data grows exponentially fast over time, everything is becoming a high performance use case. Even if no one explicitly says it.

Big data computing is implicitly a high performance problem. Even if its not cast that way. To keep costs down, you need to focus on ramping up performance and performance density (performance per rack unit). Building out thousands of nodes of a big data processing engine when hundreds or less will do doesn’t make much economic sense. Building out massive scale of low performance inefficient systems where smaller numbers of higher performance systems would do the same job in a similar time at (far) lower cost, doesn’t make much economic sense.

Another of our theses is that for a fixed performance level, we can often provide that with (far) fewer systems which lowers TCO significantly. Or for those looking at a fixed cost basis, we provide much higher overall performance in that budget. Customers win either way.

Lucera is leveraging this to provide massive performance density to their customers, and we are encouraging trading folks to try this out.

insideBIGDATA: How can those in specifically the Big Data world use this technology?

Joe Landman: One of our core thesis is that “performance matters”, and high performance is very much a cost savings paradigm. We’ve argued for years, and benchmarks back us up again and again, that inefficient and poorly designed technologies actually increase your overall lifetime and acquisition costs. End users need many more inefficient systems to accomplish the same results that fewer of our systems are able to do in smaller packages.

Again, another core concept is that performance is an enabling technology. Without performance you are constrained in the types and frequency of the analytics you can do. With significant performance, you can design and execute new analysis that you might not have been able to even imagine doing in the past.

A prime example of this is in the world of genomics. In the very early days of sequencing, we had basically technological impediments to higher performance (older, slower technology), and few people could get complete sequences.  This would take years, and cost many hundreds of millions of dollars. Now, with far more efficient and faster technologies, we can performance full sequences in an order of magnitude of days, with a cost 5+ orders of magnitude lower, with far better accuracy. Now with a large collection of these sequences, we can perform analytical experiments on populations … something we’ve never dared dream about only a few years ago. These experiments may be able to provide insight into disease processes, ageing, and so forth.

The lesson from this is, performance reduces cost, increases what you can do with the technology, and allows you to think in terms of “what could I do with this” if you had it. This is why we’ve challenged people to think and dream exactly this at our site. If you have tremendous performance, what sort of analysis could you enable, that you can’t get to today?

This tremendous performance is exactly what Lucera is offering to users. Provide an unbeatable time to market, combined with a distributed exchange, excellent analytical tools.

insideBIGDATA: Any news you can tell us about that is coming down the pike for Scalable Informatics?

Joe Landman: We’ve got a number of exciting things in queue … most of which we can’t talk about just yet.  But I should note that if people were impressed with the 32 GB/s to and from RAIDed file systems on our siFlash at SC13.

We’ll be announcing a number of new and exciting things on our Unison line, specific to helping people with massive data sets and massive data flows, as well as cold storage systems. We’ll let you know more when we are ready to talk about it.

Resource Links: