Big data refers to the analysis of huge information sets -- and the concept is picking up steam as organizations get better at capturing facts and figures about what they do, and as technology gets more adept at crunching that material into something useful. The idea, as anyone who’s seen the hit movie Moneyball knows, is to uncover relationships and patterns hidden in mountains of information, and use those insights to make better decisions and gain a competitive advantage.
This summer, Colorado intends to start sifting through data from the state’s departments of education, higher education, social services, corrections and a few others to get a better picture of why some students prosper and others struggle once finishing school. The state hopes to answer some fundamental questions. Does preschool or other forms of early childhood education make students more successful? What’s the impact on kids when a parent is incarcerated?
The plan relies on increasingly powerful technology known as master data management (MDM), which might help solve one of the biggest challenges to using big data: the fact that much of the information collected by public and private organizations is in sad shape. “We have a lot of really dirty data,” says Colorado CTO Sherri Hammons, who’s leading the project. Names are misspelled, addresses have typos and street numbers are entered into ZIP code fields, making it tough to match records from multiple agencies or programs to the same person, she says.
MDM software can analyze all of those slightly different bits of information and decide if they belong to one individual. The level of certainty can be adjusted depending on the use -- 85 percent certainty can be used for fishing licenses, for instance, but much better accuracy can be demanded for health records.
Colorado’s work on education analytics is part of a larger effort to improve access to data and standardize information sharing rules across the state. This spring, it will launch the Colorado Information Marketplace, an online clearinghouse for information collected by state and local governments.
Like many other government data sites, the Information Marketplace will offer access to public data from state agencies and participating localities, like Denver. But it’ll also be something of a dating service for state and local agencies that want to share information with one another. Agencies can list the types of data they’re willing to share, along with rules for who can see the information and how it can be used. The site will spell out standards for security, privacy and data formats, and offer templates for creating information-sharing agreements. “We’re trying to give a kick-start to people who want to share their data, but don’t know how,” Hammons says. “I think this thing will go viral because people are dying for information.”
Colorado officials say connecting the dots on data collected by multiple government programs ultimately may offer an entirely new perspective on public policy decisions. But for now, they’re moving cautiously. “We’re going to roll this out in baby steps,” says Hammons, adding that the state is paying close attention to security and privacy laws like the Health Insurance Portability and Accountability Act and the Family Educational Rights and Privacy Act.
But if they get it right, big data could completely change the game.
