No question about it – I’ve been in denial. I have not been willing to admit that Big Data is a relevant concept when it comes to network management. But the tide is overwhelming, and usage and definition of “Big Data” is metasticizing too fast. Resistance is futile. I’ve been assimilated into the Big Data collective. And somehow, I’m ok with that…
At first, Big Data was about off-line, unstructured business data, in larger volumes than normal, and managed/accessed/mined in new and interesting ways, using specific types of databasing and new analysis tools. The term “Big Data” is really quite silly, if you think about it. Haven’t there always been really big accumulations of data, even if they were in relatively limited clusters, buried deep inside data centers on fleets of storage arrays? Just think how much data the NSA must have on hand.
Because the term “Big Data” is so non-specific, many people have been confused by it’s connotations. The most natural has been to simply assume it is a reference to the sheer volume of data. By this measure, many management technologies, particularly performance management tools that generate and analyze gigabytes of metrics and measurements each day, would be Big Data applications.
It doesn’t help that many vendors have added to the confusion by twisting Big Data to their own marketing needs. For instance, I had one vendor try to tell me that the growing volumes of data-oriented mobile services was a Big Data problem. Puh-leeze!
What is more important, in my view, is the nature of the approach to data collection, management, and analysis. Big Data means large accumulations of structured or unstructured data, but it also means using certain specific technologies to hold that data – most notably Hadoop. And by this measure, network management vendors are starting to make noises about building direct Hadoop interfaces. Ok – so that is valid Big Data.
The other area where Big Data is increasing in relevance is in the area of analytics. My colleagues that cover Big Data from a Business Intelligence perspective know this area really well, including their most research on Operationalizing Big Data. There are a whole host of products on the market that are designed to crunch those Big Data sets and find insights and actionable information. Some of these very same approaches and technologies could be applied to network management (and infrastructure management, more broadly) in support of capacity planning and long-range service quality assessment. In fact, in the service provider sector, this is already the case, with a growing number of applications, such as customer experience management.
What has not been properly dealt with, in my view, is the need for Real-time Big Data analytics, for instance in support of automated IT and security operations. All of the data we can collect from and via networks has offline value, but also real-time value, if you can just elicit the important, actionable needles from the haystacks of metrics and measurements. This is the field of performance analytics, and it’s a fast-growing portion of performance management solutions as well as security monitoring and enforcement. Tom Nolle opined on this to some degree recently for SearchNetworking, but none of his use case examples are unique, in my view, and can be done with tools on hand today. But some real-time vendors are using Big Data approaches as part of their product architectures – such as what is being done by SevOne. And more announcements are coming.
I’d be the first to cheer if the term Big Data just plain went away, as the world begins to realize that this is nothing new. We’ve seen it before and solved it before – we are just doing it now in new and different ways. But that’s not how tech marketing and tech sales work, and so I’ll keep trying to help provide some sanity amidst the hype, though dialogues with practitioners and marketers, including a new research project that EMA will be launching shortly. Until then, resistance is futile. See you in the collective.