Boeing subsidiary solves its storage problems

New EMC setup addresses slow Lotus Notes backups and disaster recovery needs

When the lease was up on its old storage equipment and it came time to consider a new platform, Aviall Services knew exactly what it needed.

The Dallas-based Boeing subsidiary and provider of after-market supply-chain management services for the aerospace, defense and marine industries had been stretching its EMC-based storage setup to the limit.

Not only were backups for its ERP and Lotus Notes messaging systems stretching far beyond their allocated windows, but restorations were so time-consuming, testing showed they would cost the firm thousands of dollars in lost time and productivity.

Russell Douglas, director of customer and supplier services information systems at the company, spoke about his storage experiences at the recent Network World IT Roadmap Conference and Expo in Dallas and in a follow-up interview.

"Our biggest problem was Lotus Notes," Douglas says. "It was taking approximately 24 hours to back up, so by the time one finished, another would begin, and our backups were always failing. With [the Sarbanes-Oxley Act], full backups have to be logged. When they fail, you have to log the reasons why, and they were failing all the time."

SOX also requires that Aviall test the restoration of critical systems every six months. "At the time, a restore of our ERP took nine hours," Douglas says, noting that full restorations were done via backup tapes. "Aviall is a 24/7 business, so you can imagine if we ever had to roll back an entire day's outage. That would cost us hundreds of thousands of dollars."

Aviall was hoping a new storage platform would not only free up more storage space and reduce critical backup and restoration times, but also position the firm to handle future virtualization and real-time disaster recovery needs.

"The [disaster-recovery] portion was primarily a push by Boeing, because if we're going to integrate with Boeing in defense and commercial aviation services, then we have to make sure we have a disaster-recovery plan that can bring us back up literally in a few hours," Douglas says.

Storage scorecard

The company hired a third-party consultant to build an RFP and gather relevant data from top-tier storage vendors, including EMC, Hitachi and Veritas (now owned by Symantec). The consultant, together with Aviall, also put together a weighted scorecard so Aviall's IT staff could assess vendors' offerings, including how each would fit with the firm's existing infrastructure.

The criteria studied included not only technical details -- the features and functions of a specific array; the backup solution; and the ability to support multiple tiers, virtualization, bare-metal recovery, tape backup and hierarchical storage management -- but also the vendor's overall financial outlook and technical road map.

Aviall then formed a committee of IT staff charged with using the new storage platform to make the final decision. At the time, the company's operations department was split, with one side managing databases and Unix servers, both AIX and Sun, and the other side responsible for managing SQL Server and a variety of NT and Novell servers. An equal number of staff from each team filled out the scorecards and had to defend their decisions to Douglas.

"That worked out, because really, they're the ones who have to live with the decision," Douglas says.

In the end, Aviall chose to go once again with EMC, upgrading to its Symmetrix DMX array. The company has never looked back. ERP backups that used to take five or six hours now take just one hour. And the nine-hour ERP restoration has been reduced to 1.5 hours. Similarly, Lotus Notes backups now take just a couple of hours, and have yet to fail.

"The old backups went from the database to tape, but now we go from disk to disk to tape," Douglas explains. "There is no tape contention. We go straight to disk, and then it rolls to tape in the background. That's a primary reason performance is so much better."

Page Break

The new setup better positions Aviall to build a more robust disaster-recovery plan because it supports near-real-time replication. "Before this, restores were completely from tape," Douglas says. "So we'd have to ship the tapes to a SunGard facility that had the right servers and backup equipment, and we'd restore everything from tape. At best, we were hoping we could restore everything in a week's time."

In 2008, Aviall is working to build a disaster-recovery collocation site that supports real-time data replication, reducing to just minutes the overall time to get up and running after a disaster. "We're hoping to obsolete that whole tape process by using site-to-site replication," he says. "If something happens to the primary site, all the data will just come up on the [disaster-recovery] site. And with dedicated, high-speed connections to the collocation site, we can make sure we have near real-time replication."

Hard-nosed negotiations

Douglas says Aviall receives all these benefits -- a doubling of its storage capacity, far more efficient backup and restorations, as well as real-time disaster recovery -- without increasing its overall storage costs.

"This new equipment that literally has twice as much storage and can outperform the old stuff, I wanted it for the same lease payment," he says. "You can imagine how hard those negotiations went."

Douglas says the terms are not that far-fetched, because chips and manufacturing equipment is so much less expensive now than when he leased the old equipment back in 2002. "That's why I had pretty high expectations," he says. "Plus, we knew our old EMC hardware would have a lot of residual value at the end of the lease."

In the end, Aviall got the price it wanted (it wouldn't share specifics with us). To top it off, Douglas negotiated a six-month window to migrate to the new equipment, before lease payments kicked in. "Each weekend beginning in April, we migrated something from the old array to the new array, and we were actually finished with the migration before the end of October," he says, noting that the lease payments began in November. "It was very smooth. That's one of my paradigms. We wanted to make sure there was no impact to the business, and what that really means is they were unaware the migration was going on."