I am currently wading through the massive task of trying to buy a house.  Searching is one part fun, one part frustrating.  My friends would say this shouldn’t come as a surprise, but I am looking for something a little different than normal.  I want some space for a workshop, a toy or two and maybe even a couple of goats.  At this point, you are asking what any of this has to do with government and big data.

When thinking about Big Data it can often be boiled down to the following steps:

• Gathering the data
• Storing the data
• Learning from the data
• Using the data

One thing governments of all sizes seem to be good at it is gathering data.  They definitely have lots of it. In fact some of it goes back farther than most companies have existed.  The scene from the end of Indiana Jones Raiders of the Lost Ark comes to mind, where the Ark gets put in a crate and stored away in a vast warehouse for further study.  As the camera pulls away, the crate gets lost in a sea of crates.

Currently there are islands of Big Data work in the government.  Research into the technology needed to analyze Big Data in the government is continuing.  The announcement in March 2012 of $200 million seeks to greatly improve the tools and techniques needed to access, organize and glean discoveries from huge volumes of data will undoubtedly teach us all some new things.  Looking at the details of the projects being worked on, it appears mostly focused on the analytics part of the larger Big Data world.

The wealth of public data various governments’ possess isn’t easy to use.  The GSA’s www.SearchUSA.gov is using Hadoop to provide search capabilities to government websites across the board.  It supports all levels of governments, from cities and states to federal agencies and multiple types of data.  This is a great example of what can be done to make data easier to get to.  I think there is a lot more opportunity for improvement in access to the data.

This leads me back to my house search.  I want to know what the zoning rules are, are there mineral leases, utility easements and right of ways on the land?  Can I build a barn where I want, or is there an underground power line there?  Some of the new data visualization concepts and technologies being discussed would be a great tool here.  Layering all the information that already exists about the property onto a map view would be fantastic.

Increasing the usability of the mass of public data through powerful searches, and visualization technologies is a challenge that will take time to solve.  The benefits and efficiencies could be great, both for government agencies and the public at large.

The post Big Data in Government appeared first on Inside-BigData.

I just got an email from the organizers of the NAB Show in Las Vegas this April about my registration confirmation. I’ve always enjoyed attending this show, as it has a lot of things you don’t see at the typical IT show. How many storage shows have provisions for helicopters to fly in and be displayed?

This show draws a wide cross section of organizations, with lots of educational seminars, as well as vendor displays. What they all have in common is data. The capture side creates the raw data, editors and post-production change the data, and broadcast distributes finished data.

It got me thinking, there is so much to see and do at this show, what do I want to make sure I learn at NAB? I will specifically be looking at four things:

• 4K and Beyond – Video takes space, and HD 1080P video takes lots of space. It has become so easy to capture HD video these days with things like the Go Pro Hero Cam (they should have a cool booth in the Expo Hall), I’ve got hours of footage from bike rides, autocrosses and other events consuming lots of storage. 4K makes 1080P looks small, and there is talk about what follows that. Increases in sensor resolutions will drive bigger video files not just in media, but also in security and other applications. I want to see what is coming, as storage systems need to be ready to hold the additional data.
• Digital Workflow – I think everyone looking at Big Data issues can learn a lot here. M&E has spent several years converting to digital file-based workflow. This means lots of huge, high value files that need to be analyzed for meta data creation, modified, rendered and distributed. Every year, they get a little more efficient at the process, something we can all learn from.
• Storage – Of course, as a storage guy, I am going to be interested in any new ways to use storage. There will be everything from extreme speed storage for playout at broadcast stations to long term archival storage of digital assets. Asset management of the archives ties in here and is equally important. A lot of the problems that are being solved for long term archive today for M&E will have a broader application in the near term.
• Data Movement – How does someone move a Terabyte of data, or a Petabyte securely and with confidence? I am very interested to see how different organizations are solving the data mobility challenge. With 4K video capture becoming more mainstream, and higher resolutions on the horizon, many entertainment companies are a generation or two ahead of the rest of technology users on this front. Lessons learned here will help us all.

There is a lot of potential to learn some interesting things at NAB this year. They are facing the same challenges many Big Data industries are, but they come at it without the preconceived notions of how IT is supposed to work. I think that gives them the potential to create some interesting solutions that can benefit us all.

The post Got Big Data? 4 Things to Look for at the NAB Show appeared first on Inside-BigData.

I noticed a funny thing the other day while on the Storage Networking World website, looking at the different things on the agenda. At the top of the list is the Big Data track the first day of the show. That’s pretty predictable, given everyone seems to be talking about Big Data these days.

Surprisingly, Hadoop isn’t mentioned once in any of the Big Data track session descriptions. Some of the Big Data Track sessions are about Data Analytics, bringing Big Data to the enterprise and where to start with Big Data – in all of this I am sure Hadoop will come up, but it’s interesting it was not mentioned in the titles or descriptions.

At most other events that have a Big Data focus you see Hadoop everywhere. In fact, the feedback from some people that went to the Strata Conference was Hadoop and Big Data are inseparable. It seems that many have begun to believe that Big Data = Hadoop… but does it?

If Big Data equals Hadoop, then Big Data equals Analytics. But Big Data isn’t that simple. Processing, programming, networking and storage have some type of implication to Big Data, and I am sure we will discover many more important aspects over the next year or two.

What were once simple tasks are much more complex when dealing is massive data sets. The impact to storage systems and networks when looking at data protection alone are beyond what most organizations have considered before. With data sets that can now be larger the many disk arrays, migrating to new systems is complex and time consuming.

SNW isn’t focused on Data Scientists, but on storage managers, and while Hadoop will surely be talked about, it won’t be the focus of the day. As time goes on more disciplines will start to look at the implication Big Data plays in their part of the IT ecosystem.

About Kevin Dudak

Kevin Dudak, a United States Air Force veteran, originally joined Spectra Logic in 2007 as a Product Manager, and brings more than 15 years of storage industry experience to his role. As product manager of BlueScale Software for T-Series Tape Libraries and nTier Disk Systems, Dudak is key in helping to define Spectra Logic’s role in big data and archive storage. Dudak possesses a diverse technology background drawn from years in the hardware and software storage market, where he has architected and overseen storage implementation and support. He earned both a bachelor of arts in economics and university studies from the University of New Mexico. During his spare time he enjoys aviation photography, working on classic cars, camping in Colorado and bike riding, and participates in the Registers Annual Great Bicycle Ride Across Iowa (RAGBRAI) each year.

The post Big Data Software – More Than Just Analytics appeared first on Inside-BigData.

The month of March brings longer days, warmer weather and the start of the spring trade show season.  There seem to be as many trade shows as there are interest and industries.  Last year, we saw a lot of people start talking about Big Data at these shows.  The trend most likely will continue, with Big Data taking a bigger share of the conversation.

Given the years I have been in the storage industry, it should come as no surprise that I tend to look at the storage part of Big Data. Over the last year we have heard a lot about the analytics side of Big Data.  It is exciting seeing all the amazing things we can do, and things we can learn from the massive amount of data we have at our finger tips these days. Without a doubt, we will continue to see much of the conversation focus on leveraging our data sets with tools like Hadoop. Sometimes, it seems we forget that Big Data is more than just the analytics; it is also about storing and managing potentially massive data sets.  2012 will see users and vendors starting to address the changes Big Data brings to storage.

The 2012 Tape Summit and the HPC Symposium kick off the season. The second annual Tape Summit is the gathering of top manufactures in the Data Tape, including drive, library, software and media companies; as well as press, analysts and bloggers. You don’t see tape and Big Data in the same conversation too often, but I think the tape industry will be looking to change that this year.  We will be hearing about Linear Tape File System (LTFS,) continued innovation in data management software and possibly the coming LTO6 and how all of these can have a big impact on storing lots of data.

The HPC Symposium will see presentations from some of the top organizations in the distributed high performance world. Many of the lessons the HPC world has learned over the last 5 years will make the adoption of Big Data easier and more effective.

I’ll be watching to see how LTFS might be a good answer to Big Data portability. We are seeing LTFS gain traction in some verticals like Media and Entertainment already. The question of how to move Petabytes of data, either to seed a cloud provider or just move to a different location has always been a problem. LTFS might just provide a good answer.

Dealing with massive data sets, be it integrity checking the data or protecting it is a struggle we all face at one time or another. We are starting to see a new crop of software vendors, some in the Active Archive Alliance, that are creating data storage environments.

Finally, with the expected shipment of LTO6 this calendar year, we will see a doubling of native capacity on media.  There should be performance improvements as well. Since the LTO consortium is attending Tape Summit, hopefully we will get more details on it, and how it might affect the economy of storing big data.

As March rolls on, we should start to see a lot of information coming out of events such as the HPC Symposium and the Tape Summit on not only how to analyze Big Data, but how to manage and store it when it isn’t being crunch.

About Kevin Dudak

Kevin Dudak, a United States Air Force veteran, originally joined Spectra Logic in 2007 as a Product Manager, and brings more than 15 years of storage industry experience to his role. As product manager of BlueScale Software for T-Series Tape Libraries and nTier Disk Systems, Dudak is key in helping to define Spectra Logic’s role in big data and archive storage. Dudak possesses a diverse technology background drawn from years in the hardware and software storage market, where he has architected and overseen storage implementation and support. He earned both a bachelor of arts in economics and university studies from the University of New Mexico. During his spare time he enjoys aviation photography, working on classic cars, camping in Colorado and bike riding, and participates in the Registers Annual Great Bicycle Ride Across Iowa (RAGBRAI) each year.

The post Thinking about Big Data on the Eve of the Spring Trade Show Season appeared first on Inside-BigData.