More data? Get used to it

IDC projects that by 2010, global data creation and storage will reach 988 exabytes

As part of the IT team at the Virginia State Police department in the U.S., Lt. Pete Fagan's job is to ensure that criminal investigators, police officers in the field and other authorities get the most accurate, timely and detailed crime-related information possible. Crime never stops, so the data volumes are huge, dynamic and getting bigger every day.

Criminal records often stretch back decades and include source material from countless systems in myriad formats. Multimedia content is a growing necessity. Metadata, a new necessity, is gobbling up precious disk space. Storage capacity, as you might imagine, is an ongoing problem for the department.

But that's not the half of it.

You see, Fagan's goal is to record every request for information on the police department's storage-area network. Not just a log notation of the request, but the entire transaction as it happened. According to the lieutenant, the system will store everything the requestor sees at the moment the request is fulfilled. If the requestor gets to view dozens of JPEGs, MP3 files, videos and other capacity-hogging content, Fagan wants all that information stored separately from the original data store.

Fagan isn't a data pack rat. He's simply doing his job with the tools at hand. For him, information in the moment, knowing exactly what was known and when, could prove crucial to the criminal justice process. Just having the most recent files won't do, because some criminal cases can hinge on when something was known and by whom. Because those criminal files are always in flux, Fagan needs a storage system that can function as a time machine to retrieve not just a facsimile of the past, but the actual files and data from another time. And he needs to store those exact past experiences for more than 300 million requests per year.

Fagan's time machine is emblematic of an impending storage crisis facing IT. More broadly, will it be a harbinger of loftier roles for CIOs? Or will it -- and projects like it -- ultimately condemn IT executives to being perceived as technologists with tactical, not strategic, value to the business?

Content explosion

The Virginia State Police department is in the midst of updating its 1TB storage infrastructure with new gear from Fujitsu to handle the estimated 11TB of data capacity that will be needed in the near future. While 11TB pales in comparison with the petabyte of data stored on average by Fortune 1,000 companies, the department's growth rate is much faster than larger organizations'. That puts it in the middle of the inexorable march to the global zettabyte storage requirement.

According to an IDC study, 161 exabytes of digital information were created and stored worldwide last year. (An exabyte, you'll recall, is 1 billion gigabytes.) IDC projects that by 2010, global data creation and storage will reach 988 exabytes, a mere 12 billion gigabytes shy of a zettabyte.

"The data explosion means the role of IT managers will expand considerably," said the report. The IDC report happened to be funded by EMC, and when I chatted about it with EMC executive Chuck Hollis, he speculated that a savvy CIO can take this growth as an opportunity to become "less of a technologist and more of an informationist." He argued that IT leaders can emerge as the primary arbiters of information standards and processes for companies -- much like CFOs raised themselves from lowly accountants to powerful executives sitting at the right hand of the CEO.

Wish it were true.

Even if IDC is right about storage demands skyrocketing in the next three years, you won't see CIOs casting a strategic eye on their companies' information policies or dictating data-retention policies to their business units. Instead, they'll be running around with their hair on fire trying to keep up with the ever-increasing amounts of information pouring into their shrinking corporate SANs. Oh sure, the CIO can outlaw certain file types or act as a trusted adviser who shows business leaders how to approach information life-cycle management. But a storage capacity crisis won't enhance IT's reputation. It will undermine it.

The only way CIOs will improve their standing during a global storage shortage is to ensure that their businesses don't suffer from it. I don't care how clever a CIO thinks a new information management scheme might be or how powerful he thinks it will make him. Such an approach will be meaningless to people like Lt. Fagan, who need data to do their jobs. And I wouldn't want to be the IT executive who tries to force a new storage policy on Fagan. After all, the lieutenant does carry a gun.

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.
Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Mark Hall

Show Comments

Cool Tech

Toys for Boys

Family Friendly

Stocking Stuffer

SmartLens - Clip on Phone Camera Lens Set of 3

Learn more >

Christmas Gift Guide

Click for more ›

Brand Post

Bitdefender 2019

This Holiday Season, protect yourself and your loved ones with the best. Buy now for Holiday Savings!

Most Popular Reviews

Latest Articles


PCW Evaluation Team

Aysha Strobbe

Microsoft Office 365/HP Spectre x360

Microsoft Office continues to make a student’s life that little bit easier by offering reliable, easy to use, time-saving functionality, while continuing to develop new features that further enhance what is already a formidable collection of applications

Michael Hargreaves

Microsoft Office 365/Dell XPS 15 2-in-1

I’d recommend a Dell XPS 15 2-in-1 and the new Windows 10 to anyone who needs to get serious work done (before you kick back on your couch with your favourite Netflix show.)

Maryellen Rose George

Brother PT-P750W

It’s useful for office tasks as well as pragmatic labelling of equipment and storage – just don’t get too excited and label everything in sight!

Cathy Giles

Brother MFC-L8900CDW

The Brother MFC-L8900CDW is an absolute stand out. I struggle to fault it.

Luke Hill


I need power and lots of it. As a Front End Web developer anything less just won’t cut it which is why the MSI GT75 is an outstanding laptop for me. It’s a sleek and futuristic looking, high quality, beast that has a touch of sci-fi flare about it.

Emily Tyson

MSI GE63 Raider

If you’re looking to invest in your next work horse laptop for work or home use, you can’t go wrong with the MSI GE63.

Featured Content

Product Launch Showcase

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?