More and more organizations are expanding their usage of Hadoop software beyond just basic storage and reporting. But while they’re developing increasingly complex algorithms and becoming more dependent on getting value out of Hadoop systems, they are also pushing the limits of their architectures,” said Bill Blake, senior vice president and CTO of Cray. “We are combining the supercomputing technologies of the Cray CS300 series with the performance and security of the Intel Distribution to provide customers with a turnkey, reliable Hadoop solution that is purpose-built for high-value Hadoop environments. Organizations can now focus on scaling their use of platform-independent Hadoop software, while gaining the benefits of important underlying architectural advantages from Cray and Intel.”

As you may recall, Cray acquired the CS300 cluster technology from Appro last year. This gives the company a more affordable cluster offering for markets that don’t require Cray’s low latency interconnect technology. Read the Full Story.

The post Cray Rolls Out Hadoop Cluster Solution appeared first on Inside-BigData.

In this slidecast, Jack Norris from MapR Technologies presents: Enterprise Grade NoSQL and Hadoop.

MapR M7 provides an enterprise-grade NoSQL solution that has tremendous scale advantages,” said John Schroeder, CEO and co-founder, MapR Technologies. “The fact that it is built-in with Hadoop is a game changer for organizations looking for the best platform to leverage Big Data and support the broadest set of mission-critical applications.”

Download the MP3 * View the slides * Subscribe on iTunes* Subscribe to RSS

The post Slidecast: MapR – Enterprise Grade NoSQL and Hadoop appeared first on Inside-BigData.

Enterprise users are looking for cost-effective and scalable tools to efficiently manage and quickly access large volumes of data to turn valuable information into actionable insight,” said Boyd Davis, vice president and general manager of Intel’s Datacenter Software Division. “The addition of the Intel Enterprise Edition for Lustre to our big data software portfolio will help make it easier and more affordable for businesses to move, store and process data quickly and efficiently.”

When paired with the Intel Distribution for Apache Hadoop, the Intel Enterprise Edition for Lustre software allows Hadoop to be run on top of Lustre, significantly improving speed in which data can be accessed and analyzed. This allows users to access data files directly from the global file system at faster rates and speeds up analytics time, providing more productive use of storage assets as well as simpler storage management.

The Intel Enterprise Edition for Lustre will be available in early in the third quarter of this year. Read the Full Story.

The post Hadoop Meets Lustre – Intel Rolls out Big Data Distribution for the Enterprise appeared first on Inside-BigData.

In this slidecast, Ed Boyajian from EnterpriseDB presents: Postgres – A Tipping Point?

PostgreSQL is the #1 enterprise-class open source database with a feature set comparable to the major proprietary RDBMS vendors and a customer list that spans every industry. EnterpriseDB’s Postgres Plus solutions let you confidently develop and deploy PostgreSQL-backed applications that scale all the way from embedded solutions to massive OLTP and data warehouse systems that serve thousands of users.”

View the slides * Download the MP3 * Subscribe on iTunes * Subscribe to RSS

The post Slidecast: Postgres – A Tipping Point? appeared first on Inside-BigData.

Hadoop spawns a new Java Virtual Machine* (JVM) for each MapReduce function on each slave node. This means that a large analytics job can result in the creation of thousands of individual JVMs. Because Hadoop does not share memory resources across nodes, each JVM and Java service must perform optimally. Reduced performance on any single node can hamper data analytics performance across the cluster.

Read the Full Story.

The post Optimizing Hadoop for Intel Architecture appeared first on Inside-BigData.

This system offers researchers a total of 48, eight-processor nodes on TACC’s Longhorn cluster to run Hadoop in a coordinated way with accompanying large-memory processors. A user on the system can request all 48 nodes for a maximum of 96 terabytes (TB) of distributed storage. What’s special about the Longhorn cluster at TACC isn’t simply the beefed-up hardware for running Hadoop; rather it’s the ability for researchers to leverage the vast compute capabilities of the center, including powerful visualization and data analysis systems, to further their investigations. The end-to-end research workflow enabled by TACC could not be done anywhere else, and as a bonus, researchers get access to the full suite of tools available at the center to do computational research.

According to TACC Research Associate Weijia Xu, the best part is that Hadoop is easy to use without requiring users to be experts. It handles a lot of the low-level computing behavior, so people don’t need to have a lot of knowledge about I/O or memory structures to get started.

Read the Full Story.

The post TACC’s Hadoop Cluster Makes Big Data Research More Accessible appeared first on Inside-BigData.

In this slidecast, Alex Gorbachev from Pythian presents a Practical Introduction to Hadoop. This is a great primer for viewers who want to get the big picture on how Hadoop works with Big Data and how this approach differs from relational databases.

View the slides * Download the MP3 * Subscribe on iTunes * Subscribe to RSS

The post Slidecast: A Practical Introduction to Hadoop appeared first on Inside-BigData.

inside Big Data: What are the main security issues with Hadoop distributions?

Jim Vogt: Hadoop, like many open source technologies such as UNIX and TCP/IP, was not created with security in mind. While the open source Hadoop community supports some security features –like Kerberos, the use of firewalls and basic HDFS permissions – these security features aren’t a mandatory requirement for a Hadoop cluster, making it possible for an organization to run entire clusters without deploying any security. At the same time, the distributed computing nature of Hadoop also presents a unique challenge – data is fluid in this type of environment, moving to and from different nodes and sometimes data is sliced into fragments and shared across multiple servers. This makes it incredibly difficult to secure with traditional security approaches. Finally, popular distributions of Hadoop – like Cloudera, Hortonworks, etc. – have little incentive to build out their security functionality. The business model for most distributions is built around the sales of professional services and support– not software. The open source distribution model also requires that any new feature developments obtain the blessing of the open source community, and this means that open source solutions will always lag behind commercial software solutions from companies, like Zettaset.

inside Big Data: To what degree is security a barrier to broader enterprise adoption of Hadoop and other big data technologies?

Jim Vogt: Enterprises are obviously moving to adopt Hadoop and Big Data technologies, but right now they’re limiting their deployment of the technology throughout their enterprise. Primarily due to complexity, management challenges and security. Any organization that stores or transacts sensitive information is going to be subject to the same compliance mandates and data security regulations that apply to their traditional data stores. This can be a gating factor for organizations looking to make broader use of Hadoop and Big Data technologies.

inside Big Data: Can’t Hadoop security be addressed with traditional perimeter security solutions and by compartmentalizing parts of the system behind firewalls?

Jim Vogt: In short, no. Perimeter security solutions and compartmentalization are not designed to address Hadoop’s unique distributed architecture. However, this does not stop incumbent data security vendors from believing firewalls are the best option for Hadoop and distributed cluster security. Some firewalls attempt to map IP to actual AD credentials, yet this requires specific network design. Even with special network configuration, a firewall can only restrict access on an IP/port basis, while knowing nothing when it comes to the Hadoop File System or Hadoop itself. In order to control access, data administrators would have to segregate sensitive data on separate servers. This approach is not only inefficient but also fundamentally incompatible with distributed file systems like Hadoop, since files are constantly being shifted from server to server. It would require the creation of a second Hadoop cluster to contain sensitive data, and even then would only provide two levels of security for the data.

inside Big Data: Does this mean that Big Data challenges can’t be met in a way that still meets with robust enterprise security requirements?

Jim Vogt: The solution is to bring the security closer to the data, and apply it within the cluster itself. This can be done using fine-grained access control such as RBAC and running it on every Hadoop node. Using commercial software to automate the installation and management of RBAC simplifies deployment, and eliminates much of the complexity that security professionals currently face with open source Hadoop products.

inside Big Data: How is Zettaset taking a different approach to security for Big Data?

Jim Vogt: Unlike the dominant BigData players in the market – Cloudera, MapR, Hortonworks – Zettaset is an enterprise software company. We sell software that enables enterprises to quickly deploy, secure and scale Hadoop clusters. Our Orchestrator software is distribution-agnostic, which means we can work with the leading Apache Hadoop-based distributions available today. We harden Hadoop to address policy enforcement, regulatory compliance, access control, and risk management within the cluster environment, delivering the security capabilities that IT security professionals expect in any enterprise.

The post Interview: Jim Vogt from Zettaset on Enterprise Security for Hadoop appeared first on Inside-BigData.

For Arun Murthy, the release manager for Hadoop 2.0, the most important change will be upgrading the MapReduce framework to Apache YARN, which will expand what software can be used in Hadoop and how much. Murthy, who is also YARN project lead and co-founder of Hortonworks, explained that “In Hadoop 1.0, everything was batch-oriented. In 2.0, you will now have multiple apps hitting the data inside all at once.” What YARN does, essentially, is divide the functionality of MapReduce even further, breaking the two major responsibilities of the MapReduce JobTracker component – resource management and job scheduling/monitoring – into separate daemons: a global ResourceManager and per-application ApplicationMaster.

Read the Full Story.

The post YARN to Spawn Big Data Breakthrough appeared first on Inside-BigData.

When we set out to build Hadoop 2.0, we wanted to fundamentally re-architect Hadoop to be able to run multiple applications against relevant data sets. And do so in a way where multiple types of applications can operate efficiently and predictably within the same cluster – this is really the reason behind Apache YARN, which is foundational to Hadoop 2.0. By managing the resource requests across a cluster, YARN turns Hadoop from a single application system to a multi-application operating system.

Read the Full Story.

The post Beyond Batch – Moving Hadoop to Multi App with Apache YARN appeared first on Inside-BigData.