Last November, a fire broke out in one of the buildings on ISTA Pharmaceuticals' main campus, forcing about 50 employees to move to another location on the property. After the building's sprinklers kicked in, the entire network had to be shut down because the water threatened the equipment carrying the company's inbound data traffic.
Managers and employees at the US-based ophthalmic pharmaceutical company handled the situation with composure, says IT Director Keith Bereskin. The company's network and core applications were back online within two hours, and only 10 of the affected employees had to stay away from their offices for more than three hours, according to Bereskin.
Not bad, "considering it wasn't something we formally talked about," he says.
ISTA's "mini-disaster" happened to coincide with a disaster recovery gap analysis being conducted at the company. In that analysis, a consultant discovered that the IT department, which oversees disaster recovery coordination, and the business divisions needed to communicate more effectively, says Bereskin.
The fire and the subsequent analysis helped spur ongoing discussions between Bereskin and his peers in various business departments to determine what their expectations would be during a recovery. Among other things, they're working to identify the data they would need right away and the systems and processes that would have to be restarted immediately.
"People have said to me that we should have had these discussions previously," says Bereskin. Now they do; Bereskin says he coordinates disaster planning meetings with his business peers several times a year.
The situation at ISTA highlights the types of communication problems that often exist among disaster recovery managers, business executives and line workers, according to disaster recovery experts. "The people side of disaster recovery planning is often overlooked," says John Linse, director of business continuity services at EMC. At many organizations, when it comes to communicating disaster recovery plans, "there's almost this 'shoot, ready, aim' kind of approach," says Linse.
For instance, one of EMC's Midwestern customers didn't have an effective disaster recovery plan in place when it suffered a power outage last June, so a US$7-an-hour security guard ended up being the one who made the decision to send home the 1,300 affected employees. The outage lasted two days and cost the company US$1.3 million in business, including estimated lost revenue for orders that couldn't be taken. Afterward, EMC helped the company craft a business continuity plan that included identifying key business processes that need to stay up during a disaster -- and which people are responsible for them.