In this video from the 2013 Open Fabrics Developer Workshop, D.K. Panda from Ohio State University presents: High Performance RDMA-based Design for Big Data and Web 2.0 memcached.
You can check out more OFA videos at our Open Fabrics Workshop Video Gallery.
In this video from the Lustre User Group 2013 conference, Sage Weil from Inktank presents: An Intro to Ceph for HPC.
Ceph is a free software unified storage platform designed to present object, block, and file storage from a single distributed cluster. Ceph’s main goals are to be completely distributed without a single point of failure, scalable to the exabyte level, and freely-available. The data is seamlessly replicated, making it fault tolerant. Ceph is a software-based solution and runs on commodity hardware. The system is designed to be both self-healing and self-managing and strives to reduce both administrator and budget overhead.
Check out more presentations at our LUG 2013 Video Gallery.
Over at Admin HPC, Douglas Eadline writes that Hadoop could be the new corporate HPC for Big Data.
The growth of Hadoop and the hardware on which it runs has been increasing. Certainly it can be seen as a subset of HPC, offering a single yet powerful algorithm that has been optimized for a large number of commodity servers, with some crossover even into technical computing that could see further growth as things like YARN begin to give existing Hadoop clusters more HPC capabilities. Many companies are finding Hadoop to be the new Corporate HPC for big data.
Read the Full Story.
In this video from the 2013 Open Fabrics Developer Workshop, Sreev Doddabalapur from Mellanox presents: Accelerating Big Data over RDMA.
You can check out more OFA videos at our Open Fabrics Workshop Video Gallery.
In this video from the Lustre User Group 2013 conference, Josh Judd from Warp Mechanics presents: Warp Mechanics ZFS Array.
The WARP Mechanics 39830 is a turnkey network-attached non-volatile RAM + SSD system with industry-leading price, performance, and scalability. This system maximizes the IOPs performance for the most demanding application profiles. It is an ultra-dense space and power saving solution. This is optimal for large-scale IO intensive workloads with large live data sets. The 50x high capacity 2TB SSD modules per 4U enclosure are configured into five 10-disk RAID 6 sets to maximize protection and performance. Each RAID set has a two NV-RAM modules serving as write cache. These RAID sets are added to the overall ZFS storage pool and can be allocated to a nearly limitless number of any sized volumes presented to hosts. This yields a flexible 100TB of usable RAID protected SSD storage.
Check out more Lustre presentations at our LUG 2013 Video Gallery.
In this video from the Lustre User Group 2013 conference, Robert Read from Intel presents: Lustre on Amazon Web Services.
You can check out more Lustre presentations at our LUG 2013 Video Gallery.
This video looks at how Big Data is handled from the Large Hadron Collider (LHC).
The LHC produces millions of collisions every second in each detector, generating approximately one petabyte of data per second. None of today’s computing systems are capable of recording such rates, so sophisticated selection systems are used for a first fast electronic pre-selection, only passing one out of 10,000 events. Tens of thousands of processor cores then select 1% of the remaining events for analysis. Even after such drastic data reduction, the four big experiments, ALICE, ATLAS, CMS and LHCb, together need to store over 25 petabytes per year. The LHC data are aggregated in the CERN Data Centre, which performs initial data reconstruction is performed, and a copy is archived to long-term tape storage. Another copy is sent to several large data centres around the world.
In this video from the Lustre User Group 2013 conference, Christopher Morrone from Lawrence Livermore National Laboratory presents: Sequoia and the ZFS OSD.
Check out more presentations at our LUG 2013 Video Gallery.
In this podcast, the Radio Free HPC team discusses the recent buzz surrounding FGPAs. After being sidelined by accelerators, they’re increasingly being used in appliances.
Big vendors are talking about FGPAs not only for appliances but for general-purpose systems as performance assists. Are we headed back to the future? The guys discuss the ins and outs of FGPAs and why, in some cases, they could be a huge win for the organizations that implement them. But is the architecture flexible enough? For enterprise and Big Data, perhaps it is. If you need to perform the same algorithms over and over again, FGPAs could be a perfect fit. As with all things tech, there are a few cautionary notes to be sounded. Amassing more and more appliances can lead down a tricky road. Will their use in workload-optimized systems lead to vendor lock-in? Can you really teach an old FGPA new tricks? And can they be weaponized?
Most importantly: how are servers like cattle? Tune in to find out…
Download the MP3 * Subscribe on iTunes * RSS Feed
In Greek mythology the Titans, known as the elder gods, ruled the earth until the Olympians tossed them out of power.
At Oak Ridge National Laboratory, one of the Titans is not only still with us and doing well, but this modern day diety just got a boost that will help keep it king of the hill for some time to come.
The world’s fastest supercomputer, Titan is capable of delivering a peak capability of over 27 petaflops, ten times more powerful than previous generations of ORNL computers such as Jaguar. This week the Lab announced that it has selected DataDirect Networks to build the world’s fastest file storage system to ensure Titan’s ascendancy in the world of HPC.
Built around DDN’s SFA 12K-40, the system is being designed with 40 petabytes of raw capacity capable of “ingesting, storing, processing and distributing research data at unprecedented speed,” according to a DDN press release. The DDN system will work with the Lab’s Lustre parallel file system with a Lustre performance of over one terabyte per second to handle the demands of Titan’s 299,008 CPU cores.
The world’s toughest questions demand the toughest storage and the fastest technology to drive new levels of scientific insight. DDN has spent the better part of a decade engineering a platform that is built precisely and efficiently for today’s Big Data challenges,” comments Jean-Luc Chatelain, chief technology officer at DDN. “As applications everywhere – from energy exploration to climate modeling to energy efficient car manufacturing – continue to drive extreme levels of computational simulation and data analytics, we’re proud to provide the data storage technology that makes such innovation and economic competitiveness possible. We’re honored to continue our long-standing partnership with ORNL today and to be part of the future of Big Data and exascale computing tomorrow.”
ORNL points out that Titan is architecturally unique in a variety of ways, and is a showcase for tomorrow’s computational requirements as Big Data continues to make inroads into the enterprise.
When building the world’s fastest system for data intensive computing, we carefully considered all aspects of high-throughput I/O infrastructure and how efficient storage platforms can complement our supercomputer’s efficiency,” says Buddy Bland, project director for the Oak Ridge Leadership Computing Facility. “The ORNL and DDN teams have worked together to architect a file system designed to enhance the performance of our Titan supercomputer and enable our users to achieve unprecedented simulations and big data insights through massively scalable computing.”
The Olympians may have thought they dethroned the Titans, but as Yogi Berra once said, “It ain’t over till it’s over.” Titan at ORNL, with some help from DDN, rules supreme…at least for now.
Read the Full Story.
inside-bigdata.com is a production of insideHPC, LLC. © 2011-2013 Sitemap
