Finance – Where HPC Teams Up With Big Data

BigData_FinanceA group of researchers from three U.S. universities are leveraging high performance computing (HPC) systems in conjunction with Big Data systems to make a significant step forward in the rapid analysis of financial markets; in effect reducing processing from hours/days down to minutes.

The researchers from the University of Illinois, the Pittsburgh Supercomputer Center at the University of Pittsburgh and the San Diego Supercomputer Center at UC San Diego are making use of parallel processing supercomputer capacity provided via XSEDE – the Extreme Science and Engineering Discovery Environment -  to run analytics tasks, and have optimized their code to provide up to a 126x speedup compared to previous analysis undertaken in 2010.

That performance boost, which the researchers reported at the XSEDE conference in San Diego in July 2013, will allow them to analyze market characteristics across the entire Nasdaq equities market in just a couple of hours. This includes the impact of high frequency and other low-latency trading strategies, and could be used to detect whether unfavorable strategies, such as “quote stuffing,” have been deployed. The slides for the conference presentation can be downloaded HERE.

XSEDE provides the research community with access to a variety of computing resources, including supercomputers, as well as visualization and data analysis systems. The supercomputers used by this particular HFT (high-frequency trading) research are:

  • Blacklight at the Pittsburgh Supercomputing Center. Blacklight is an Intel-based SGI shared memory system intended for applications that require a large memory space for computational tasks.
  • Gordon at the San Diego Supercomputer Center. Gordon is a Flash-based supercomputer, also incorporating Intel chips, designed in partnership with Appro (now Cray) for data intensive workloads.
  • Stampede at the Texas Advanced Computing Center, University of Texas at Austin. Stampede – currently cited as the sixth most powerful supercomputer in the world – was designed in collaboration with Dell, and includes Intel multi-core and Xeon Phi many-core processors for highly parallel computational processing.

Trading firms, exchanges and regulators are likely to make increasing use of HPC and Big Data to implement parallel analytics of large financial market data sets, for pre-trade execution compliance and risk management applications.

Resource Links: