For the last several months, I’ve been heads-down on a research project that culminates over a year of investigation into the rise of what I have been calling “data-driven security”: security efforts informed and defined by their reliance on insight – and increasingly enabled by emerging technologies for large-scale data management, as well as by growing awareness of the value of data science.
Now that the report is published, what we’ve discovered over the course of the past year-plus should be of interest if you are one of the many who feel that lack of awareness stymies security efforts, regarding whether in security operations, in the technologies of defense, or in managing security strategy.
If you have followed my blog series on this trend (and posts since), you have the general outline of the study. In the research report, however, I go into greater depth on the topics covered in the blog series, including:
- The drivers motivating organizations to look to a wide variety of data sources to improve security tactics and get a better handle on strategy. (Hint: The blind should no longer lead the blind.)
- Security’s challenges of ETL, the extraction, transformation and loading of data (a maddening frustration we in security tend to minimize with pretty euphemisms like “data normalization”), and how data management technologies can help overcome one of the core obstacles to making more effective use of security data.
- The varieties of techniques organizations are exploring to derive new understandings from security-relevant data.
- How technology vendors and service providers are expanding capabilities through data-driven approaches.
- The role of technologies such as data warehousing, optimized data management techniques such as columnar systems, emerging NoSQL environments, data mining and analytic tools, collaborative analysis platforms, and why all these matter (or should) to security.
- And lest you think that last bullet excludes those organizations unable or unwilling to invest in a major data management undertaking for security: a discussion of how data-driven security will benefit “the rest of us.”
Added to this greater depth are findings from a survey we conducted among 200 respondents worldwide representing organizations of 1,000 personnel or more (since we felt that larger organizations were more likely to invest in the trend), with a mix of results both expected and – in some cases – surprising:
- When asked about the amount of security data they collect, nearly half (40%) of all respondents say they are already overwhelmed. Of the 49% of respondents reporting knowledge of security log and event data collection, 13% report collecting a terabyte or more each day. Some security product vendors, meanwhile, deal with more than 5 terabaytes of new threat data daily.
- At the same time, however, a whopping 73% of survey respondents said they would collect more security-related data, or a wider variety of data, if they could make use of it.
Why this seeming paradox?
- Only half (52%) of surveyed organizations claim to be “somewhat” confident they could detect an important security issue before it has an impact. Twelve percent are neither confident nor doubtful, while 7% report greater doubt than confidence.
- “Too difficult to distinguish legitimate from malicious activity” is the most frequently mentioned frustration with security practices (33% of all respondents).
- 28% say they are about as unsuccessful as they are successful at correlating security data to business impact. 4% are more unsuccessful than successful.
One of the more surprising findings was the extent to which organizations are exploring advanced approaches to data management in support of security efforts. I don’t mean just those normally associated with infosec, such as Security Information and Event Management (SIEM). Many seem to have concluded – erroneously – that the intersections of Big Data and security are primarily about SIEM. While it is true that 79% of those collecting security-relevant log and event data in our survey use SIEM and related technologies, the story certainly doesn’t end there – far from it. In fact, it even goes beyond more recent growth areas for security data, such as investigative and forensic platforms that enable security analysts to see threats more clearly:
- Half of all respondents indicated that data warehouse systems are already in use in support of security data management. 38% report the use of integrated analytical database platforms for this purpose.
- 55% employ Business Intelligence (BI) tools in connection with security efforts. 43% use, or have used, tools such as R for more advanced or flexible security data analysis.
- 28% plan to evaluate NoSQL platforms or environments such as Hadoop for security data management.
- Over one-third are already expanding their investment in both technologies and expertise specifically to increase capability in security data management and analysis, with over 40% more planning to do so in the next 1-3 years.
These are just a few of the findings that reflect an interest in data-driven security in the enterprise that may be more active than many may suspect. Clearly, this data will raise some questions. For me, the more tantalizing include:
- What is the extent to which respondents are actually “using” or exploring these technologies? Are these proof-of-concept pilots, or are they actually being deployed in production?
- Are these platforms and tools “owned” outright by security, or are security teams taking advantage of resources made available by other efforts within an organization, such as a BI initiative?
- How mature are these efforts, if in operation – and how have organizations been able to build the necessary maturity to make the most of the opportunity?
- And perhaps the most important: It’s interesting to learn that organizations are using advanced data management and analysis platforms – but what are they doing with all this data? What are the analytic techniques that matter? And what issues are organizations encountering in the responsible management of information that is likely sensitive?
My expectation would be that the answers to any of these questions today would say that most enterprises are still early in the evolution of data-driven security. Not so, however, for some technology vendors and service providers who are among the pioneers of data-driven approaches:
- The ability to mine enormous volumes of threat data to reveal new attacks and help pinpoint compromise have already been embraced by security technology vendors to help organizations deal with the scale of the threat landscape (or landscapes, considering the variety and diversity of threats).
- Putting the fruits of analysis directly to work in countermeasures that have a more direct and dynamic reliance on data sources is one way security technologies may be transformed by the trend.
- Cloud-based approaches offer a means to deliver this capability across a wide audience.
- Providers of managed and professional services, meanwhile, offer the expertise organizations will need to develop their strengths and capabilities in security data management and analysis.
We explore a number of examples of these developments in the report.
I hope to have the opportunity to follow up this research with further insight into the trend as it continues to unfold, and even more examples of the ways in which data science, which is influencing so much of technology development, is transforming the nature of security as well.
So stay tuned! More to come on this aspect of security evolution!