Discovering Gold with Big Data Analytics and Data-Intensive Computing

Entries filed under “Storage”

Video: EIOW Exascale I/O Working Group

By Rich | Published May 18, 2013

In this video from the Lustre User Group 2013, Meghan McClelland from Xyratex presents: EIOW – Exascale I/O Working Group.

There is a fierce competition on the storage market to offer the best performing devices, with great management at a low price. The EIOW group, from the outset, decided that it would not attempt to offer an end-to-end solution, which would necessarily involve competing instead of working with storage providers. The focus of EIOW is on middleware to provide, for example, schemas describing data structure and layout, novel access methods to data for applications, a uniform data management infrastructure and a framework for the implementation of layered I/O software, similar in spirit to HDF5 as a specialized use of a parallel file system. We decided EIOW should be open, and have interfaces to layer on lower level storage infrastructure such as object stores, databases and file systems as provided by storage providers, to allow their expertise and leadership in this area to continue to benefit the HPC community.

Download the EIOW whitepaper and slides, or check out our LUG 2013 Video Gallery.


Also posted in Hardware, HPC, I/O, Lustre, Software, Video | Leave a comment

Slidecast: ClusterStor 1500 Departmental Scale-Out Storage for HPC

By Rich | Published May 14, 2013

In this slidecast, Ken Claffey from Xyratex describes the company’s new ClusterStor 1500 storage system. Designed for scale-out HPC storage solutions, the ClusterStor 1500 delivers HPC performance and efficiency with help from the Lustre file system.

Departments within larger organizations or medium-sized enterprises today, especially in the commercial, academic and government sectors, represent an underserved market. They need high-performance and scalable storage solutions that are cost-efficient, easy to deploy and manage and reliable even under heavy workloads,” said Ken Claffey, senior vice president of the ClusterStor business at Xyratex. “Growth in this market segment is being driven by the increasing adoption of simulation applications in a wide range of industries from car and aircraft design to chemical interactions and financial modeling. Traditional enterprise storage systems are simply not designed to meet the performance needs of these applications, so we engineered and built the affordable and modular ClusterStor 1500 to bring the performance power of Lustre to this underserved and growing market in the way that only ClusterStor can.”

With the ability to scale performance from 1.25GB/s to 110GB/s and raw capacity from 42TB to 7.3PB, ClusterStor 1500 is purpose-built to satisfy data intensive department level compute cluster needs, ClusterStor 1500 is designed to provide best in class scale-out storage for middle tier high performance computing environments. The ClusterStor 1500 solution features scale-out storage building blocks, the Lustre parallel filesystem and a comprehensive management platform that eliminates the guesswork usually associated with building and optimizing your own HPC storage solution.

Read the Full Story * View the slides * Download the MP3 * Subscribe on iTunes * Subscribe to RSS


Also posted in Business of Big Data, HPC, Lustre, Podcasts, Video | Leave a comment

Video: Aeon Computing Showcases Intelligent Disk Arrays with Lustre and ZFS

By Rich | Published May 11, 2013

In this video from the Lustre User Group 2013, Doug Johnson from Aeon Computing describes the company’s Hydra storage arrays that marry the speed of Lustre with the HA capabilities of ZFS.

The Hydra60 is a combination Lustre OSS (object storage server) and OST (object storage target) with two active/active failover nodes and shared storage in a single system chassis with an ultra dense 60 drive 6Gb SAS storage infrastructure. With a unified and zonable 6Gb SAS dual-ported backplane and drives the Hydra60 can sustain a remarkable performance while providing high-availability to volumes or object storage. With external interface options including FDR Infiniband, 40/10GbE 1Gb Ethernet and supporting Linux and Lustre releases 2.x the Hydra60 makes an excellent storage platform for Lustre performance with HA operation. The design of Hydra60 provides an affordable, redundant and resilient storage platform by leveraging RAIDZ thereby eliminating the cost of hardware RAID controller technology.”

For more on Lustre, check out our LUG 2013 Video Gallery.


Also posted in HPC, Lustre, Video | Leave a comment

Jeff Layton on Tools for Data Migration

By Rich | Published May 10, 2013

Over at HPC Admin, Dell’s Jeff Layton writes that with today’s explosive data growth, at some point you will have to migrate data from one set of storage devices to another. To help move things along, he provides an overview of data migration tools.

At some point during this growth spurt, you will have to think about migrating your data from an old storage solution to a new one, but copying the data over isn’t as easy as it sounds. You would like to preserve the attributes of the data during the migration, including xattrs (extended attributes), and losing information such as file ownership or timestamps can cause havoc with projects. Plus, you have to pay attention to the same things for directories; they are just as important as the file themselves (remember that everything is a file in Linux). In this article, I wanted to present some possible tools for helping with data migration, and I covered just a few of them. However, I also wanted to take a few paragraphs to emphasize that you need to plan your data migration if you want to succeed.

As referred to in Layton’s article, this video from the Lustre User Group 2013 features Marc Stearman from LLNL presenting: Sequoia Data Migration Experiences. Download the slides (PDF).

Read the Full Story.


Also posted in Hardware, Lustre, Software, Tape, Video | Leave a comment

Video: High Availability in Lustre

By Rich | Published May 6, 2013

In this video from the Lustre User Group 2013, John Fragalla from Xyratex presents: High Availability in Lustre.

Download the slides (PDF) or check out our LUG 2013 Video Gallery.


Also posted in Events, Hardware, HPC, I/O, Lustre, Software, Video | Leave a comment

Video: DataDirect Networks Update

By Rich | Published May 5, 2013

In this video from the Lustre User Group 2013, Jeff Denworth from DDN presents: DataDirect Networks Update.

DDN has developed a Hadoop solution that is all about time to value: It simplifies rollout so that enterprises can get up and running more quickly, provides typical DDN performance to accelerate data processing, and reduces the amount of time needed to maintain a Hadoop solution.” said Dave Vellante, Chief Research Officer, Wikibon.org. “For enterprises with a deluge of data but a limited IT budget, the DDN hScaler appliance should be on the short list of potential solutions.”

Download the slides (PDF) or check out our LUG 2013 Video Gallery.


Also posted in Business of Big Data, HPC, I/O, Lustre, Software, Video | Leave a comment

Indiana University Helps NASA Manage Big Data

By Rich | Published May 3, 2013

Indiana University has contributed Big Data expertise and infrastructure to NASA’s Operation IceBridge, a decade-long polar ice monitoring project.

For the past four years, IU Research Technologies, a cyberinfrastructure and service center affiliated with the Pervasive Technology Institute (PTI), has provided IT support for the Center for Remote Sensing of Ice Sheets (CReSIS), a National Science Foundation Science and Technology Center led by the University of Kansas. Kansas scientists provide NASA with the radar technology that measures the physical interactions of polar ice sheets in Greenland, Chile and Antarctica. IU experts bring innovative data management and storage solutions to the missions.

Essentially, IU has built a supercomputer that can fly,” said Rich Knepper, manager of IU’s campus bridging and research infrastructure team within Research Technologies. “During this current mission, our system provided analysis of radar data as the data was collected – in real time — allowing mission scientists to see the ice bed information as the plane flies over the Arctic.”

Read the Full Story.


Also posted in Public sector, Research | Leave a comment

Video: Lustre & ZFS go to Hollywood

By Rich | Published April 27, 2013

In this video from the Lustre User Group 2013 conference, Josh Judd from Warp Mechanics presents: Lustre & ZFS go to Hollywood.

WARP Mechanics Ltd. is a leading provider of high performance computing (HPC) solutions. The company mission is to bring these super computing technologies into broader IT markets. Each WARP product is factory-optimized for vertical markets such as public-sector “Big Science”, commercial Bio/Life, Cloud, or Media/Entertainment, and can be rolled out in a turn-key fashion.

Download the Slides (PDF).


Also posted in Hardware, Lustre, Software, Video | Leave a comment

Video: Warp Mechanics ZFS Array

By Rich | Published April 24, 2013

In this video from the Lustre User Group 2013 conference, Josh Judd from Warp Mechanics presents: Warp Mechanics ZFS Array.

The WARP Mechanics 39830 is a turnkey network-attached non-volatile RAM + SSD system with industry-leading price, performance, and scalability. This system maximizes the IOPs performance for the most demanding application profiles. It is an ultra-dense space and power saving solution. This is optimal for large-scale IO intensive workloads with large live data sets. The 50x high capacity 2TB SSD modules per 4U enclosure are configured into five 10-disk RAID 6 sets to maximize protection and performance. Each RAID set has a two NV-RAM modules serving as write cache. These RAID sets are added to the overall ZFS storage pool and can be allocated to a nearly limitless number of any sized volumes presented to hosts. This yields a flexible 100TB of usable RAID protected SSD storage.

Check out more Lustre presentations at our LUG 2013 Video Gallery.


Also posted in Events, Hardware, HPC, Lustre, Software, Video, ZFS | Leave a comment

SGI Infinite Storage Gateway Appliance Implements DMF in Minutes

By Rich | Published April 22, 2013

Today SGI announced the InfiniteStorage Gateway, an appliance that delivers virtualized data management to lower the cost of high volume storage. Why an appliance? SGI’s Floyd Christofferson described it as a way  to “install DMF within minutes and enable greatly simplified storage management for Big Data.”

The SGI InfiniteStorage Gateway reduces the dependency on high-cost primary storage by creating a virtualized storage fabric that can include any mixture of disk, tape, Zero-Watt Disk or MAID, and object storage. While appearing to users and applications simply as online data, SGI InfiniteStorage Gateway offers IT administrators the ability to keep data protected and online at a fraction of the cost of primary storage systems.

As data growth has continued to sky-rocket, IT organizations increasingly face the problem of infrastructure fragmentation, and the fact that their most expensive primary storage arrays are often used to house mostly inactive data,” said Laura DuBois, program vice president, IDC Storage Systems, Software and Solutions. “Data management is not only about the performance of active data today. It also must provide a seamless long-term strategy for all data that keeps costs at a minimum and reduces IT administrative burden without impacting users.”

With up to 276TB of onboard capacity in a single 4U appliance, the gateway automatically places data on any or all storage devices and locations based upon what works best for the access requirements and data protection policies.

Read the Full Story or View the Slides on Slideshare.


Also posted in Business of Big Data, DMF, Hardware | Leave a comment

Advertisement

NAS for Dummies Ad

View All Videos

inside-bigdata.com is a production of insideHPC, LLC. © 2011-2013 Sitemap