In this special guest feature, Spectra Logic’s Kevin Dudak writes that while the government is great at gathering data, accessing useful information remains difficult.
I am currently wading through the massive task of trying to buy a house. Searching is one part fun, one part frustrating. My friends would say this shouldn’t come as a surprise, but I am looking for something a little different than normal. I want some space for a workshop, a toy or two and maybe even a couple of goats. At this point, you are asking what any of this has to do with government and big data.
When thinking about Big Data it can often be boiled down to the following steps:
- Gathering the data
- Storing the data
- Learning from the data
- Using the data
One thing governments of all sizes seem to be good at it is gathering data. They definitely have lots of it. In fact some of it goes back farther than most companies have existed. The scene from the end of Indiana Jones Raiders of the Lost Ark comes to mind, where the Ark gets put in a crate and stored away in a vast warehouse for further study. As the camera pulls away, the crate gets lost in a sea of crates.
Currently there are islands of Big Data work in the government. Research into the technology needed to analyze Big Data in the government is continuing. The announcement in March 2012 of $200 million seeks to greatly improve the tools and techniques needed to access, organize and glean discoveries from huge volumes of data will undoubtedly teach us all some new things. Looking at the details of the projects being worked on, it appears mostly focused on the analytics part of the larger Big Data world.
The wealth of public data various governments’ possess isn’t easy to use. The GSA’s www.SearchUSA.gov is using Hadoop to provide search capabilities to government websites across the board. It supports all levels of governments, from cities and states to federal agencies and multiple types of data. This is a great example of what can be done to make data easier to get to. I think there is a lot more opportunity for improvement in access to the data.
This leads me back to my house search. I want to know what the zoning rules are, are there mineral leases, utility easements and right of ways on the land? Can I build a barn where I want, or is there an underground power line there? Some of the new data visualization concepts and technologies being discussed would be a great tool here. Layering all the information that already exists about the property onto a map view would be fantastic.
Increasing the usability of the mass of public data through powerful searches, and visualization technologies is a challenge that will take time to solve. The benefits and efficiencies could be great, both for government agencies and the public at large.
Jim Duffey writes that the Internet2 high-bandwidth network consortium designed to support collaborative ventures between universities, government research agencies and businesses, is ready to step up into the big data realm. 
