Finding gold in your log files

Many studies have shown that the majority of security events and application errors would have been noticed earlier had the relevant log files been reviewed

Considering how much valuable information is available in log files, you'd think more companies would pay attention to them. Workstations, servers, firewalls, appliances, and other computer devices generate reams of event logs every day, and despite mountains of evidence showing their practical, cost-saving uses, logs often go ignored. A good log management system can help significantly with security, application troubleshooting, compliance, and systems management. If that's the case -- and it is -- why do logs and log management sometimes still get a bad rap?

It's understandable on many levels. First, logs can contain towering amounts of uninteresting, hard-to-decipher events, burying more useful information. In fact, without the appropriate tools and filters, logs can be nothing but noise -- and lots of it.

[ Get the full scoop on getting more value from your log files in the InfoWorld "Log Analysis Deep Dive" PDF special report. | Better manage your company's information overload with our Enterprise Data Explosion newsletter. ]

A standard Microsoft Windows computer can easily generate thousands of events each day even when things are humming along without a real problem. A thousand computers can generate tens of gigabytes of log files on a daily basis. I've seen enterprise event log collector tools bring robust networks to their knees. What's worse, many administrators would tell you that in a typical week, not a single issue requiring an immediate response was uncovered. "Talk about a waste of resources," they will tell you, even as valuable, useful data is passing under their eyes.

Diamonds in the rough

Log file review is rarely a management priority -- until it hits a tipping point or the auditors complain loud enough. Many studies have shown that the majority of security events and application errors would have been noticed earlier had the relevant log files been reviewed. Yet management tends to act as if logs aren't worth the time and effort to analyze, a dismissal that trickles down to overworked staff. Why mess with something that seems like a waste of time to all parties involved?

Another factor is simply human nature: Few people get excited about reviewing log files. The answer to "Hey, Johnny, what do you want to be when you grow up?" is never log file reviewer, even if a good log reviewer is actually worth his or her weight in gold.

So why should you or your company care about log files? Because they allow an IT organization to be proactive versus reactive. The typical IT department waits for calls for help before responding to problems. But by the time end-users call in, they are already frustrated, the event that prompted the call has typically entered a critical phase, and IT is forced to respond in the most inefficient manner possible.

Imagine how delighted your end-users would be if the help desk called them ahead of time to let them know they were having a hardware or software problem that was just starting to manifest itself. Wouldn't it be nice to catch hackers before they were successful? Can you imagine a world in which your purchasing department was alerted to buy additional hard drives before they ran out of free space?

Are log files a waste of time? The exact opposite is true. Logging, if appropriately configured and managed, will save you and your company time and money. The best-run organizations live on a diet of event log baselines and proactive responses, and you can too.

Log management 101

In a nutshell, logging allows you to quantitatively and proactively measure the overall health of your environment, from a security perspective, for auditing and compliance, for systems management, and for application tuning and troubleshooting. These basics will get you started.

Security monitoring. Most of the literature surrounding computer logging talks about monitoring events to lower your security risk. Logging can alert incident response teams to prevent malicious hacking in the first place -- or at least send in the cavalry as quickly as possible after an exploitative event has occurred to minimize damage and start foren­sic investigations.

Logging security events for intrusion detection and forensics, which is often the main reason administrators get into log management, requires specialized advice. You can start by reading NIST's Special Publication 800-92, "Guide to Computer Security Log Management." Released in September 2006, it's unusually easy to read for a NIST (National Institute of Standards and Technology) publication and extremely useful for deploying event log management systems in the real world. It's considered the gospel in this small corner of the computer security world.

The NIST guide steps through all of the essentials of log file management: identifying the threats and risks to your environment; determining policies for logging, auditing, and handling logs; collating, indexing, and normalizing logs for analysis; defining and generating alerts and actions for critical events; and defining reports and metrics for management review. From putting log management infrastructure and processes into place to reviewing and archiving logs, it leaves no stone unturned.

Auditing and compliance. As motives for instituting log management programs, auditing and compliance are becoming as important as traditional security requirements. Most industry regulatory guidelines now define specific security events that must be monitored. When the right audit policy has been enabled across all required computers, and the appropriate log management system is in place, most companies will pass that portion of a compliance review. On the other hand, the lack of an acceptable security auditing policy can raise suspicion that the right controls are lacking, which may have legal implications.

Systems management. The best-run shops understand the value of logging and use it for systems management. These organizations create baselines of normal operating activity and events, and they set up alerts triggered by excessive deviations. Many environments execute simple ping connectivity tests to monitor which devices are online and which unexpectedly dropped and need to be investigated. Other places embrace the full richness that logs provide.

If a hard drive begins to move too many bad sectors, even before a complete crash occurs, the log administrator has a replacement drive ready to roll. If network activity spikes unexpectedly, administrators are aware of the problem before the inevitable complaints about slowness arrive. A sustained traffic hit may be a worm or a denial-of-service attack. If a server or SAN crashes, the help desk knows about it before users start to call in.

Read more about how tap into log files in InfoWorld's free PDF report, "Log File Analysis Deep Dive," including:

* Application tuning and troubleshooting

* Choosing the right log management software

* The log management life cycle

* Pulling off a successful event management program

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Tags softwareArchitecture

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.
Roger A. Grimes

Roger A. Grimes

Show Comments

Brand Post

Most Popular Reviews

Latest Articles


PCW Evaluation Team

Luke Hill


I need power and lots of it. As a Front End Web developer anything less just won’t cut it which is why the MSI GT75 is an outstanding laptop for me. It’s a sleek and futuristic looking, high quality, beast that has a touch of sci-fi flare about it.

Emily Tyson

MSI GE63 Raider

If you’re looking to invest in your next work horse laptop for work or home use, you can’t go wrong with the MSI GE63.

Laura Johnston

MSI GS65 Stealth Thin

If you can afford the price tag, it is well worth the money. It out performs any other laptop I have tried for gaming, and the transportable design and incredible display also make it ideal for work.

Andrew Teoh

Brother MFC-L9570CDW Multifunction Printer

Touch screen visibility and operation was great and easy to navigate. Each menu and sub-menu was in an understandable order and category

Louise Coady

Brother MFC-L9570CDW Multifunction Printer

The printer was convenient, produced clear and vibrant images and was very easy to use

Edwina Hargreaves

WD My Cloud Home

I would recommend this device for families and small businesses who want one safe place to store all their important digital content and a way to easily share it with friends, family, business partners, or customers.

Featured Content

Product Launch Showcase

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?