Deep Data: The Key to Big Data Success

No doubt you've heard of "big data," but what about "deep data"? The answer, I'm afraid, is no. Don't be nervous, I'm not trying to force a new vocabulary on you. But in light of the recent and ongoing debate about the amount of data that business users need to collect and manage, I think the concept of deep data should be on the minds of business users who are concerned about the potential of their data. Badri Raghavan, CTO and chief data officer at FirstFuel, an analytics company specializing in energy efficiency in the building industry, has a unique perspective on this. The company's clients, including government agencies and energy organizations, are using FirstFuel's energy analytics services to drive the spread of greener, more cost-effective solutions in office environments, schools and other facilities.

In a phone interview, Raghavan talked about his view of "deep data" and how FirstFuel is turning it into a competitive advantage.

"What we call 'deep data' is really an amalgamation of multiple domain specializations -- in our case, the energy industry meets data science! -- designed to help technicians analyze a building's energy use from a macro-scale perspective," he told us.

The concept of deep data is inextricably linked to information density. "A given data stream can contain a lot of information," Raghavan said. "Conversely, it's possible for people to collect a large amount of data that lacks sufficiently conclusive content or information."

As you may have guessed, Raghavan himself doesn't subscribe to the practice of data collection or aggregating as much information as possible. But that's what a lot of organizations are doing right now, namely blindly aggregating sizeable data aggregates when they're not yet sure if it makes sense to do so.

The real heart of data collection is efficiency, or "utilizing the data assets that are already available. To accomplish this, we need to first identify the technical or business challenges we need to solve. Of the resources available to everyone, which data streams play the most important role?"

In the industry FirstFuel works in - namely, analyzing the energy consumption of large buildings - a single data stream often becomes the most important measurement.

"We'll take metering data as a scan of a building. Using our data science algorithms, we can analyze the health of the building, identify weaknesses and areas where there is still room for improvement."

This, he noted, is an excellent example of the type of deep data that actually works. Metered data is "a relatively compact stream of data, but it's very rich," and FirstFuel has been able to pinpoint the problem it's most interested in: finding out what's going on in energy consumption that doesn't prioritize efficiency.

Of course, the most important thing for many organizations is to figure out which data streams are most valuable to analyze, and then to combine them with other data to draw new conclusions.

FirstFuel has identified several types of data streams that are often the most potentially valuable.

"Metrological data can tell us a lot of information related to a building," Raghavan notes. "Next we started using high-resolution aerial imagery - yes, that's Google Earth - and we use a lot of that type of information in our work. From our perspective, it contains a wealth of potential information. It tells us what types of equipment are on the roofs of these buildings," and FirstFuel is able to use that to determine in general terms the total amount of energy that needs to be consumed by the corresponding building.

The analytics firm also takes data from the National Weather Service into account.

"We set out to set it up and bring it in incrementally. Wherever we can make an improvement in the conclusions of the information analysis, we take the relevant data streams into account."

And that, according to him, is the basic concept of deep data. "Instead of being confronted with vast sums of data ...... for long periods of time and trying to fish small pins out of them that symbolize valuable conclusions, as in the past, people can do deep research on relatively small data sets."

FirstFuel, for example, is perfectly capable of collecting a wide range of additional data - including information related to traffic flow and parking conditions, in addition to Twitter data streams - but the fact that there is simply no clear reason to drive them to choose such a laborious approach.

"Rather than jumping directly into a sea of big data that can be potentially analyzed, often ending up with little or no valuable information, we're more likely to get a lot more out of a relatively small volume of data - i.e., focusing on data that reflects a realistic picture of a building's condition - than we are to leap into a sea of big data that can be potentially analyzed, often ending up with little or no valuable information. actually reflect the objective condition of the building," Raghavan noted. "Having developed such a solution idea, we then progressively turn the idea into reality."

The above is what I shared with you about deep data The key to driving big data success, for more information, you can follow the Global Green Ivy to share more dry goods