October 1997 Michael Cox and David Ellsworth publish “Application-controlled demand paging for out-of-core visualization” in the Proceedings of the IEEE 8th conference on Visualization. They start the article with “Visualization provides an interesting challenge for computer systems: data sets are generally quite large, taxing the capacities of main memory, local disk, and even remote disk. We call this the problem of big data. When data sets do not fit in main memory (in core), or when they do not fit even on local disk, the most common solution is to acquire more resources.” It is the first article in the ACM digital library to use the term “big data.”

Read the Full Story.

The post A Brief History of Big Data appeared first on Inside-BigData.

Lots of people and organizations are talking about big data, and the analytics side is getting a lot more attention than the storage site of the conversation. The potential of the analytics is fascinating, and I think we are still in the early days of its full potential. However, as the name implies, Big Data is big, and needs to be stored.

I got to thinking about this after an hour-long call with a customer that I only expected to take 5 minutes. He is quickly approaching 100 PB of data and does not have a plan for it. He has multiple disk and tape storage systems, with four different software solutions that manage portions of the data.  I don’t think the company ever expected to grow to this size when it made its software and hardware decisions over the last 10 years. They are now facing several major challenges:

• Knowledge
  • They have too many types of hardware and software for their staff to remain competent with all of them.
• Support
  • With a number of different systems, the amount of support contracts is difficult to manage, let alone deciphering the complexities of keeping everything running.
• Power
  • Power and cooling are crushing them.  The monthly bill is affecting the finances of the company and they are struggling to be able to obtain more power to grow.
• Data
  • In the end, this should all be about the data, but with the data spread across so many systems and technologies, it is difficult to access and use, at best.

This company is not alone in their challenges. And it has happened to far too many organizations out there. Data islands and different storage systems all made sense when they were deployed and IT looked at them as single, standalone solutions. Several years and a lot of growth later, and many companies that didn’t consider themselves a ‘data company’ now find themselves with Big Data. The challenge now is to figure out how to get out of the unplanned mess they are in and get things straightened out.

This is a challenge that users, integrators and manufactures should be working on for the next few years. As we talk about how to solve these problems, I think the first step is to focus on the data. The data is the reason we have all this storage and computing resources. There are a number of things being done to solve these challenges. I’ll be sharing more about this in future posts.

The post What Do You Do with Accidental Big Data? appeared first on Inside-BigData.

In this video from The Next Web Conference Europe 2013, Ken Cukier, Data Editor at the Economist describes how Big Data hype should not deter us from bringing this phenomenon to its full potential to change the world.

The post Ken Cukier on the How Big Data will Change the World appeared first on Inside-BigData.

In this video from the 2013 HPC User Forum, Steve Conway and Chirag Dekate from IDC present: High-Performance Data Analysis: HPC Meets Big Data.

Download the slides (PDF) or check out our HPC User Forum Video Gallery.

The post Video: High-Performance Data Analysis: HPC Meets Big Data appeared first on Inside-BigData.

Under the terms of the alliance, Splunk Hadoop Connect will link Splunk Enterprise to Cloudera Enterprise, Cloudera’s Hadoop distribution (and associated projects). Which is certainly good for Cloudera: any number of companies have released Hadoop distributions over the past couple months, crowding the marketplace, and making it all the more vital for individual firms to sign “alliances” and other contracts for their respective offerings. Such alliances could also prove vital for smaller firms seeking to hold the line, as it were, against IT giants such as IBM and SAP. The latter, of course, have untold millions of dollars and large amounts of other resources to deploy in the search for data-analytics customers; faced with that sort of competition, startups and midsize companies need to consider how partnerships can amplify the reach of their analytics products.

Read the Full Story.

The post Splunk and Cloudera Alliance Hints at New Big Data Landscape appeared first on Inside-BigData.

In this video from the 2013 HPC User Forum, Darren Schulte from Apixio presents: Knowledge-driven Outcomes for Healthcare.

Apixio delivers actionable insights from textual, scanned, and coded data for optimal risk management, population management, and revenue.”

Download the slides (PDF) or check out the HPC User Forum Video Gallery.

The post Apixio – Knowledge-driven Outcomes for Healthcare appeared first on Inside-BigData.

In this slidecast, Ken Claffey from Xyratex describes the company’s new ClusterStor 1500 storage system. Designed for scale-out HPC storage solutions, the ClusterStor 1500 delivers HPC performance and efficiency with help from the Lustre file system.

Departments within larger organizations or medium-sized enterprises today, especially in the commercial, academic and government sectors, represent an underserved market. They need high-performance and scalable storage solutions that are cost-efficient, easy to deploy and manage and reliable even under heavy workloads,” said Ken Claffey, senior vice president of the ClusterStor business at Xyratex. “Growth in this market segment is being driven by the increasing adoption of simulation applications in a wide range of industries from car and aircraft design to chemical interactions and financial modeling. Traditional enterprise storage systems are simply not designed to meet the performance needs of these applications, so we engineered and built the affordable and modular ClusterStor 1500 to bring the performance power of Lustre to this underserved and growing market in the way that only ClusterStor can.”

With the ability to scale performance from 1.25GB/s to 110GB/s and raw capacity from 42TB to 7.3PB, ClusterStor 1500 is purpose-built to satisfy data intensive department level compute cluster needs, ClusterStor 1500 is designed to provide best in class scale-out storage for middle tier high performance computing environments. The ClusterStor 1500 solution features scale-out storage building blocks, the Lustre parallel filesystem and a comprehensive management platform that eliminates the guesswork usually associated with building and optimizing your own HPC storage solution.

Read the Full Story * View the slides * Download the MP3 * Subscribe on iTunes * Subscribe to RSS

The post Slidecast: ClusterStor 1500 Departmental Scale-Out Storage for HPC appeared first on Inside-BigData.

In this video from the 2013 HPC User Forum, Radhika Subramanian from Emcien presents: The Big Data Paradigm Shift – Insight Through Automation.

Download the slides (PDF) or check out the HPC User Forum Video Gallery.

The post Video: The Big Data Paradigm Shift – Insight Through Automation appeared first on Inside-BigData.

There’s a big push to deliver SQL-analysis capabilities on top of Hadoop, and the talent shortage is just one reason. The second reason for the trend is that Apache Hive, Hadoop’s incumbent data warehousing infrastructure, offers a limited subset of SQL-like query capabilities and suffers from slow performance tied to behind-the-scenes MapReduce processing. Answering the call for broader, faster SQL querying on Hadoop are projects and initiatives including Cloudera Impala, EMC’s HAWQ query feature on the Pivotal HD distribution, Hortonworks Stinger, IBM Big SQL, MapR-supported Apache Drill, and Teradata SQL-H.

This is great stuff if you want to better understand the business opportunities out there in Big Data. Read the Full Story.

The post Five Big Business Opportunities for Big Data appeared first on Inside-BigData.

In this slidecast, Scott Gnau from Teradata Labs presents: Teradata Intelligent Memory.

The introduction of Teradata Intelligent Memory allows our customers to exploit the performance of memory within Teradata Platforms, which extends our leadership position as the best performing data warehouse technology at the most competitive price,” said Scott Gnau, president, Teradata Labs. “Teradata Intelligent Memory technology is built into the data warehouse and customers don’t have to buy a separate appliance. Additionally, Teradata enables its customers to buy and configure the exact amount of in-memory capability needed for critical workloads. It is unnecessary and impractical to keep all data in memory, because all data do not have the same value to justify being placed in expensive memory.”

How does Intelligent Memory work? This animation video does a good job of making this advanced technology look simple.

Read the Full Story * View the slides * Download the MP3 * Subscribe on iTunes * Subscribe to RSS

The post Slidecast: Teradata Rolls Out Intelligent Memory Technology appeared first on Inside-BigData.