The silent explosion of log management

What's the fastest-growing data source at large organizations? Video? Maybe at YouTube, but not at Citibank. The answer is log files. Yup, those mundane text messages produced by every conceivable technology device are growing like a proverbial weed.

Why the log file explosion? First off, every IT shop has more and more devices to manage each day, and more devices mean more logs. In the past, most IT managers paid limited attention to logs, but this too has changed. Driven by security monitoring, regulatory compliance reporting, and IT troubleshooting, large organizations now aggregate, analyze, store, and archive terabytes of log data every month. Indeed, the log management snowball is rolling down a very steep and very snowy hill.

It's certainly time that the IT industry recognizes that log management is no longer a tactical domain of IT administrators. Rather, think of log management as the foundation of a Network Information System (NIS). Analysis of log data (as well as other data sources like NetFlow and SNMP) is quickly becoming the difference between effective IT security/operations management and flying blind.

As log management gains strategic importance, look for the big guys to swoop in. EMC/RSA and IBM are already there, and HP is working with SenSage on its play. There are also several strong pure plays including ArcSight, LogLogic, and LogRhythm. All this said, where is everyone else? Log management seems like a natural for Cisco Systems, especially in concert with its whole Data Center 3.0 initiative. Symantec is also missing from the party thus far, as is CA, McAfee, Microsoft, and loads of others.

Logs seem trivial, and log management appears like a tactical task at the bottom of the IT stack. Maybe in the past this was true, but in today's world, information is power and logs are device-specific information. Aggregate these logs, add some intelligence, and provide tools for analysis, and you get a great temperature reading of what's going on across the IT system. Seems pretty important to me.

Jon Oltsik is a senior analyst at the Enterprise Strategy Group.