In just three years, the bytes of data generated by businesses IT systems and devices, as well as digital cameras and mobile phones, will equal the number of grains of sand on the world's beaches, so says analyst house IDC.
This estimation reflects the proliferation of devices and systems used by businesses and consumers, said Stephen Minton, IDC's vice president of worldwide IT markets and strategies, during a briefing on Thursday.
Over the next few years, corporations will face tough decisions on how to store data, find information and comply with regulations, said Minton, who laid out a forecast of what IT managers can expect from what's being labelled the "information explosion."
It won't be an easy task. While 85 per cent of that data is predicted to come from consumers snapping photos, surfing Web pages and sending email, about 60 per cent of that consumer data will still cross corporate networks, Minton said.
Much of the data is unstructured, meaning it's not clearly labelled as to its content, such as photos, video and perhaps phone recordings, which makes it more difficult to use.
But technologies that enable deep analysis of the data are emerging, and could help business unlock what's important and improve their operations.
The mass of data companies are collecting - the US retailer Wal-Mart Stores for example, generates 1TB of transactional data a day - represents huge business opportunities "if companies can really get to a point where they can successfully analyse the data going back and forth on their network," Minton said.
Companies are "still a long way short of being able to analyse the unstructured data on their networks," he said. The data is coming from Web services such as the photo-sharing site Flickr, Skype and even the virtual world Second Life.
The business opportunities are the positive side. But the security concerns still abound, as well as regulatory compliance and liability worries.
According to data from the US Computer Emergency Response Team, the number of reported software vulnerabilities declined in 2003 and 2004, but surged in 2005 to around 6,000, an all-time high.
"This explosion in the number of applications and devices has created an explosion in the number of security vulnerabilities," Minton said.
The security and reliability of those IT systems has become so important that the IT department now has "a direct impact on the company's top and bottom line," Minton said. Customer experiences and the company's image are at stake when security fails, he said.
Accordingly, security has remained the top budget priority over the last five or six years, with 60 percent of companies placing it as the highest one in an IDC survey conducted earlier this year.
IDC is predicting that revenue from security and vulnerability management software products will grow by 20 percent this year over 2006, to US$2.27 billion. That far outpaces the 6 per cent to 7 per cent growth expected for the software industry as a whole.
When a report of a data breach hits the front page that "has a very bad impact on a company's brand image," Minton said.