The video below comes to us from the Strata Conference 2014: How Companies are Using Spark, and Where the Edge in Big Data Will Be.
While the first big data systems made a new class of applications possible, organizations must now compete on the speed and sophistication with which they can draw value from data. Future data processing platforms will need to not just scale cost-effectively; but to allow ever more real-time analysis, and to support both simple queries and today’s most sophisticated analytics algorithms. Through the Spark project at Apache and Berkeley, the developers have brought six years research to enable real-time and complex analytics within the Hadoop stack. Spark is a fast and general engine for large-scale data processing.
The speaker, Matei Zaharia is an assistant professor of computer science at MIT, and the initial creator of Apache Spark. He is currently on industry leave to start Databricks, a company commercializing Spark, where he is CTO.
Sign up for the free insideBIGDATA newsletter.