The Future of SIEM – The market will begin to diverge

For the first day of the New Year I thought I would post my thoughts on the future of the SIEM market. Enjoy!

Security Information and event management technologies saw some consolidation in 2006. EMC acquired Network Intelligence, Novell acquired eSecurity, IBM acquired Micromuse, which had acquired Guardednet, and IBM also acquired Consul. Today there are lot’s of large, established broad-scoped vendors and point solution vendors trying to capture the roughly $300 million in revenue the SIEM market was estimated to be in 2006. How is this sustainable? What will happen to the market in the next 2-3 years?

First let’s define where the market has been. Initially these tools were designed for threat management against a noisy external threat environment, namely worms. Their orientation was primarily network and systems with real-time analysis of events to support incident response (Security Event Management). There were also vendors that provided long-term storage, historical analysis and trending against a large back-store to support forensic activities (Security Information Management). So we had real-time analysis to support incident response and long-term storage and historical analysis to support trend reporting and forensics. Threat drove sales and SEM was pushing the space, then the noisy threat environment quieted down and compliance became a larger issue for organizations. Auditors began looking for SIM functions, that is more long-term storage of data, and the vendors all scrambled to grab compliance dollars by adding compliance-oriented templates and repositioning their technology as a compliance solution. There were still some vendors that focused primarily on SEM or SIM, although the leading point solution vendors provided both with variable efficiency. The market, of course, wanted it all. So we now had convergence of SEM and SIM, thus was born the term SIEM that Mark Nicolett (Great analyst) and I coined at Gartner in 2005, although the actual convergence began en masse in 2004. At the time there was no separate log management market and we also included vendors that provided these functions, if they met certain inclusion criteria, in the SIEM market

Compliance buoyed a market that was nearing stagnation and it created requirements for integration with identity and access management systems (IAM), auditors were looking for user monitoring and auditing against critical servers and applications, not just visibility into threats against the network and devices. The vendors were all scrambling to add this user perspective to their solutions and large, established IAM vendors began to enter the market or expand their capabilities and integrate their technologies into their SIEM solutions. SIM, driven by compliance initiatives, became the main catalyst pushing the SIEM market. The one major anomaly to this trend was Cisco, which purchased Protego and began to sell it as MARS, a component of its Cisco Security Management Suite. MARS also has NBA-like functions (here) and is primarily a network centric, threat oriented SEM. As a side note Cisco deserves a lot of credit for taking an obscure and relatively unknown product in Protego and making it one of the most visible and arguably successful SIEM technologies on the market. Although Cisco will not disclose its penetration, it is fair to assume they have MARS deployed at 2-3,000 customers (yeah, yeah I know they are giving it away at huge discounts, but it is still there)

I took hundreds of SIEM inquiries in late 2005 and through 2006 and when I asked what the requirements were, that is threat oriented to support incident response (SEM) or long-term storage and analysis to support compliance or forensics (SIM), did they need a network view or a user view, were they trying to monitor activity against the network, servers, or applications, did they need a tool to support monitoring of activity for internal policy violations or for external attacks, etc, and on and on – well 9 out of 10 times the response was “yeah we want all of that”. The reality is that most organizations cannot support these deployments and many SIEM technologies were disasters not due to technology failures but because the customers did not properly define requirements or the scope of the deployment but the number one issue was they did not have a process to support the technology. Let me repeat this since it is so important most technologies fail in an enterprise due to lack of process not because of flaws in technology. It is not uncommon to see large deployments take 6-8 months longer than expected and to consume 250-500k more than anticipated, vendor bravado and aggressive selling tactics that bordered on unethical didn’t help either.

In addition to the SEM vs. SIM and network vs. user centric views there is another dynamic affecting the market. The majority of SIEM vendors filter the data, either at the point of collection or at an aggregation/correlation server. Log management systems collect all the data. The folks who filter claim that 80% of the log files are junk and they only provide the 20% that is relevant, the folks who collect it all say it is a requirement for compliance and how do you know what will be relevant tomorrow.

Today we have a market demanding a solution that addresses all their needs and vendors attempting to provide it. SEM/SIM functions delivered in a single converged tool to support network, system, application, and user centric views into the environment. We also have log management vendors, basically syslog servers on steroids, creating a lot of disruption with the traditional SIEM vendors, who are now positioning new solutions that provide log management appliances.

This market will begin to diverge in 2007 into a SEM market driven by network and systems oriented tools focused on the threat, configuration and policy compliance needs, pushed by vendors such as Cisco (expect competitors to acquire in 2007), Symantec, and NetIQ and a SIM market driven by user-centric auditing and monitoring, integrations with IAM systems, focused on regulatory compliance initiatives, pushed by vendors such as CA, IBM, and Novell (expect other IAM vendors to enter in 2007). Point solution vendors will struggle to address the broadest set of capabilities, and will also feel pressure from an increasingly disruptive set of log management vendors. Log management will break out as it own class of products and will see the biggest growth as folks realize that at the end of the day all they really wanted was a syslog server on steroids.