Processing the vast quantities of data produced by the SKA will require very high performance central supercomputers capable of 100 petaflops per second processing power. This is about 50 times more powerful than the most powerful supercomputer in 2010 and equivalent to the processing power of about one hundred million PCs.
In this podcast, the Radio Free HPC team discusses Lustre and LUG 2013 with Brent Gorda. Now part of Intel in their High Performance Data Division, Gorda was CEO of Whamcloud when the company was acquired last summer.
In this video from the Strata 2013 Conference, David Smith from Revolution Analytics describes the five stages of real-time analytics deployment, and the technologies supporting each stage, including Hadoop, R, and database warehousing systems. He also shares some best practices for setting up a the technology stack and processes for model deployment, based on some real-life case studies.
In case you weren’t aware of it, we are right in the middle of Big Data Week 2013. This is a global event featuring a series of community-led Big Data conferences, meetings, workshops, networking events and hackathons.
The event, which runs from April 22-28, has a somewhat British flavor, being produced by media 140 Worldwide headquartered in London. They anticipate that by the end of the week more than 5,000 Big Data practitioners from 20 countries and four continents will participate.
Typical events include:
Putting Data to Work (London) – A day of case studies, presentations, community showcases and debates hosted by Ed Dumbhill with the O’Reilly Strata Conference and O’Reilly Open Source Convention.
With more than 200 events, Big Data Week is a platform for thousands of data scientists, data technologists, data visualisers and leaders from the business, not-for-profit and governments sectors to share how Big Data is impacting their organisations,” according to the conference organizers. “‘Big data has gone from a buzz word to a business reality. Now is the moment to learn from each other to advance the art and science of harnessing data to benefit all aspects of society,’ said Kenneth Cukier, a co-author of the book Big Data: A Revolution That Will Transform How We Live, Work, and Think and the data editor at The Economist, who is keynoting an event on 25th April at Imperial College London.”
The full list of the 200+ events can be found here.
And here’s one of the week’s interesting activities: Teams from around the world will be tasked to find the world’s healthiest city using publically available data sources. They will develop models for ranking the health of a city using sources such as government statistics, weather patterns, Twitter sentiment and any other publically available data sets.
Will London be the winner? Only Big Data and time will tell.
The WARP Mechanics 39830 is a turnkey network-attached non-volatile RAM + SSD system with industry-leading price, performance, and scalability. This system maximizes the IOPs performance for the most demanding application profiles. It is an ultra-dense space and power saving solution. This is optimal for large-scale IO intensive workloads with large live data sets. The 50x high capacity 2TB SSD modules per 4U enclosure are configured into five 10-disk RAID 6 sets to maximize protection and performance. Each RAID set has a two NV-RAM modules serving as write cache. These RAID sets are added to the overall ZFS storage pool and can be allocated to a nearly limitless number of any sized volumes presented to hosts. This yields a flexible 100TB of usable RAID protected SSD storage.