Finding gold in your log files

Many studies have shown that the majority of security events and application errors would have been noticed earlier had the relevant log files been reviewed

Considering how much valuable information is available in log files, you'd think more companies would pay attention to them. Workstations, servers, firewalls, appliances, and other computer devices generate reams of event logs every day, and despite mountains of evidence showing their practical, cost-saving uses, logs often go ignored. A good log management system can help significantly with security, application troubleshooting, compliance, and systems management. If that's the case -- and it is -- why do logs and log management sometimes still get a bad rap?

It's understandable on many levels. First, logs can contain towering amounts of uninteresting, hard-to-decipher events, burying more useful information. In fact, without the appropriate tools and filters, logs can be nothing but noise -- and lots of it.

[ Get the full scoop on getting more value from your log files in the InfoWorld "Log Analysis Deep Dive" PDF special report. | Better manage your company's information overload with our Enterprise Data Explosion newsletter. ]

A standard Microsoft Windows computer can easily generate thousands of events each day even when things are humming along without a real problem. A thousand computers can generate tens of gigabytes of log files on a daily basis. I've seen enterprise event log collector tools bring robust networks to their knees. What's worse, many administrators would tell you that in a typical week, not a single issue requiring an immediate response was uncovered. "Talk about a waste of resources," they will tell you, even as valuable, useful data is passing under their eyes.

Diamonds in the rough

Log file review is rarely a management priority -- until it hits a tipping point or the auditors complain loud enough. Many studies have shown that the majority of security events and application errors would have been noticed earlier had the relevant log files been reviewed. Yet management tends to act as if logs aren't worth the time and effort to analyze, a dismissal that trickles down to overworked staff. Why mess with something that seems like a waste of time to all parties involved?

Another factor is simply human nature: Few people get excited about reviewing log files. The answer to "Hey, Johnny, what do you want to be when you grow up?" is never log file reviewer, even if a good log reviewer is actually worth his or her weight in gold.

So why should you or your company care about log files? Because they allow an IT organization to be proactive versus reactive. The typical IT department waits for calls for help before responding to problems. But by the time end-users call in, they are already frustrated, the event that prompted the call has typically entered a critical phase, and IT is forced to respond in the most inefficient manner possible.

Imagine how delighted your end-users would be if the help desk called them ahead of time to let them know they were having a hardware or software problem that was just starting to manifest itself. Wouldn't it be nice to catch hackers before they were successful? Can you imagine a world in which your purchasing department was alerted to buy additional hard drives before they ran out of free space?

Are log files a waste of time? The exact opposite is true. Logging, if appropriately configured and managed, will save you and your company time and money. The best-run organizations live on a diet of event log baselines and proactive responses, and you can too.

Log management 101

In a nutshell, logging allows you to quantitatively and proactively measure the overall health of your environment, from a security perspective, for auditing and compliance, for systems management, and for application tuning and troubleshooting. These basics will get you started.

Security monitoring. Most of the literature surrounding computer logging talks about monitoring events to lower your security risk. Logging can alert incident response teams to prevent malicious hacking in the first place -- or at least send in the cavalry as quickly as possible after an exploitative event has occurred to minimize damage and start foren­sic investigations.

Logging security events for intrusion detection and forensics, which is often the main reason administrators get into log management, requires specialized advice. You can start by reading NIST's Special Publication 800-92, "Guide to Computer Security Log Management." Released in September 2006, it's unusually easy to read for a NIST (National Institute of Standards and Technology) publication and extremely useful for deploying event log management systems in the real world. It's considered the gospel in this small corner of the computer security world.

The NIST guide steps through all of the essentials of log file management: identifying the threats and risks to your environment; determining policies for logging, auditing, and handling logs; collating, indexing, and normalizing logs for analysis; defining and generating alerts and actions for critical events; and defining reports and metrics for management review. From putting log management infrastructure and processes into place to reviewing and archiving logs, it leaves no stone unturned.

Auditing and compliance. As motives for instituting log management programs, auditing and compliance are becoming as important as traditional security requirements. Most industry regulatory guidelines now define specific security events that must be monitored. When the right audit policy has been enabled across all required computers, and the appropriate log management system is in place, most companies will pass that portion of a compliance review. On the other hand, the lack of an acceptable security auditing policy can raise suspicion that the right controls are lacking, which may have legal implications.

Systems management. The best-run shops understand the value of logging and use it for systems management. These organizations create baselines of normal operating activity and events, and they set up alerts triggered by excessive deviations. Many environments execute simple ping connectivity tests to monitor which devices are online and which unexpectedly dropped and need to be investigated. Other places embrace the full richness that logs provide.

If a hard drive begins to move too many bad sectors, even before a complete crash occurs, the log administrator has a replacement drive ready to roll. If network activity spikes unexpectedly, administrators are aware of the problem before the inevitable complaints about slowness arrive. A sustained traffic hit may be a worm or a denial-of-service attack. If a server or SAN crashes, the help desk knows about it before users start to call in.

Read more about how tap into log files in InfoWorld's free PDF report, "Log File Analysis Deep Dive," including:

* Application tuning and troubleshooting

* Choosing the right log management software

* The log management life cycle

* Pulling off a successful event management program

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Tags softwareArchitecture

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.
Roger A. Grimes

Roger A. Grimes

Show Comments

Most Popular Reviews

Latest Articles


PCW Evaluation Team

Jack Jeffries


As the Maserati or BMW of laptops, it would fit perfectly in the hands of a professional needing firepower under the hood, sophistication and class on the surface, and gaming prowess (sports mode if you will) in between.

Taylor Carr


The MSI PS63 is an amazing laptop and I would definitely consider buying one in the future.

Christopher Low

Brother RJ-4230B

This small mobile printer is exactly what I need for invoicing and other jobs such as sending fellow tradesman details or step-by-step instructions that I can easily print off from my phone or the Web.

Aysha Strobbe

Microsoft Office 365/HP Spectre x360

Microsoft Office continues to make a student’s life that little bit easier by offering reliable, easy to use, time-saving functionality, while continuing to develop new features that further enhance what is already a formidable collection of applications

Michael Hargreaves

Microsoft Office 365/Dell XPS 15 2-in-1

I’d recommend a Dell XPS 15 2-in-1 and the new Windows 10 to anyone who needs to get serious work done (before you kick back on your couch with your favourite Netflix show.)

Maryellen Rose George

Brother PT-P750W

It’s useful for office tasks as well as pragmatic labelling of equipment and storage – just don’t get too excited and label everything in sight!

Featured Content

Product Launch Showcase

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?