I’ve been looking over this year’s Black Hat agenda with an eye toward new thinking on a topic I’ve been blogging about over the last several months: the rise of data-driven security. By this I mean the increased leverage of data mining and analysis to deliver more detailed and accurate insight into the reality of security posture from large and often diverse data sets, and the ability to make more responsive use of that insight through tactics driven by dynamic data sources.

Though I did not see too much on the topic per se on the Black Hat program, I have noted a definite uptick in the interest of security professionals regardless, particularly in the potential impact of Big Data on security:

  • Hoff has been blogging on this theme lately, most recently with a very engaging example of how the performance of emerging Big Data techniques could better serve advances in the automation of security essential to its integration in high-performance environments – such as fast-moving trading systems, where a lot of money can be earned or lost in transactions measured in infinitesimal fractions of a second.
  • That post also has a pointer to James Arlen’s Black Hat talk, “Security When Nano-Seconds Count,” which sounds like it will (at the very least) highlight the need for this level of security performance in such environments.
  • A few weeks ago, GOTO Metrics closed $3 million in new venture funding and changed its name to Zettaset. In the press release announcing this news, the company described itself as an enabler of “massive data analytics for the enterprise via Hadoop, the open-source platform that provides reliable, scalable, distributed computing.”  While this in itself is not exactly unique (as those of you who follow my colleague Shawn Rogers are certainly aware), it places security use cases front-and-center alongside applications for health care, telecom and other industries. Taking a look at Zettaset’s executive team will help explain why. It not only includes some of the original SPI Dynamics crew (Web application security play now part of HP’s Application Security portfolio), but others with an infosec background not always seen in the Big Data world – which suggests some of the provocative directions Zettaset may well go…
  • Pervasive Systems‘ Innovation Lab recently showed me their concepts for advanced event processing – which could have significant application to security, including some of the concepts considered above regarding high-performance management of security data at scale. Within the communities of BI and data management, Pervasive is known, as it describes itself, as a vendor of “software to manage, integrate and analyze data, in the cloud or on-premises, throughout the entire data life cycle.” Beyond these communities, it deserves to be better known. Most legacy approaches to security-relevant data management do not (yet) take significant advantage of the technologies of Big Data. Pervasive’s RushScale technology leverages columnar/hybrid database architectures such as HBase, Cassandra, or other NoSQL techniques to take in data from log events to genomics to optimize data capture and processing, and output to analytic tools. Considering the volume of security-relevant data we already have challenges managing, these techniques deserve a closer look for their potential application to security – and they’ll likely get it.

Before we get too over-the-top with enthusiasm about Big Data’s security potential, however, Amrit Williams draws our attention to some important considerations about employing Big Data tools. As Amrit put it, “No doubt Hadoop (and its spawn) offers pretty radical improvements to how we manage and federate data, but it isn’t analytics and it doesn’t offer intelligence or knowledge without a lot of work.”

Although Hadoop is more a “them” than an “it” (Hadoop is essentially a toolset that includes highly scalable, distributed databases such as HBase and Cassandra, and it’s these that people often seem to have in mind when they reference Hadoop), Amrit’s point is well taken. Even though the nature of security-relevant data and analytics may differ from other aspects of the enterprise (I discuss some of these factors in the last three posts of the blog series), Amrit draws some painful parallels from the world of BI that we in security should consider soberly in weighing the value of Big Data techniques. He also makes a point that may be all too easily overlooked in heady speculation about what Big Data means for infosec: It’s one thing to optimize the collection and processing of data. It’s another thing entirely to make sense of it.

While there has been no little emphasis on analytics in the BI world, what will be the nature of security analytics that enable us to take advantage of Big Data trends more effectively in security – regardless whether at the “macro” level of human analysis, or in the “micro” realm of integrating high-performance security data analysis with environments such as those James Arlen will be talking about this week?

If your organization is actively exploring answers to such questions, I’d be very interested in hearing from you.

  • When I wrote about data synthesis platforms a few months back, I offered a few current examples in areas where data mining is already a factor in infosec, as well as some that suggest where security analytics may go tomorrow.
  • I’ve also seen some concrete application of security analytics that specifically leverage Hadoop, such as this presentation by Booz Allen Hamilton’s Paul Brown. (NB: Registration may be required. The video, recorded at this year’s Data Scientist Summit hosted by EMC’s Greenplum unit, begins with Martin Hall of Karmasphere – a vendor that describes itself as “Big Analytics for Big Data on Hadoop” – making the case for Hadoop in the service of data science. Paul’s part of the talk begins at 17:27. His demo of a network security application begins at 27:25.)

We are still, however, only at the very beginning of trends for leveraging Big Data techniques in the service of security, and taking on the tools to handle ever-larger data sets more efficiently may directly influence how security analytics may change in response.

These are just a few examples of ongoing developments and the provocative questions they raise that suggest how significant the impact of data-driven security may become. While this year’s Black Hat agenda isn’t exactly jam-packed with content on this theme, expect that to change, and not just at security conferences – perhaps quite a bit over the next several months.


Enhanced by Zemanta