Testing was completed in early November, Thibodeau says, although security testing is still ongoing. "There were no show-stoppers, so we're optimistic that we'll turn it on in December before the onslaught in January," he says. Other functionality will continue to be built through 2011. If NARA is rewarded its next appropriation of money, it expects to build the public-access capability within a year, Thibodeau says.
Other slowdown factors
Adam Jansen, president of Dkives Consulting, agrees that the phased approach is the way to go. Formerly the digital archivist for the state of Washington, Jansen built an electronic records archiving system for the state that serves 750 users and stores 75 million records -- from 150-year-old census books to e-mails accumulated over the most recent governor's eight-year term. The system stores a million Web pages from 400-plus agencies, and the state is about to release several hundred hours' worth of searchable full text, digital audio and tape of legislative committee hearings.
He calls NARA's project "a hugely ambitious project, and it's very difficult to bite off that big of a chunk all at once," he says. While with Washington's state government, Jansen says his team started with a few types of records and expanded from there. In four years, he says, the system went through three distinct iterations, with tweaking and reinventing along the way, especially when it came to ingesting records.
As for the ERA's bumpy history, Jansen also faults government bureaucracy and NARA's failure to seek out advice from others who had implemented such systema. "There were people who'd done research, and I'm not sure the lessons learned were researched and taken to heart," he says. "Having run a program similar to this for five years in Washington, I got almost no interaction with them despite efforts to do so."
But Thibodeau says when the project began in 1998, there was little information available. At that time, it took the agency two years just to research the feasibility of developing such a system, and it created a program management office to support it. "The biggest system we'd acquired before this was under $10 million," he says. "When you're doing something over $100 million, it's much more complicated, so we wanted to make sure we were competent to do it."
NARA also dedicated three years to eliciting and validating requirements, which culminated in inviting both the IT industry and the general public to comment on the requirements, he says. During this phase, NARA also organized two conferences, one for prospective users and another for industry, to discuss its plans and get feedback. "We wanted there to be no question of what we were building," Thibodeau says, claiming there have been no changes to the requirements over the course of the system build.
Countdown to January
Logan says the problem of managing electronic records won't be resolved until the government agencies themselves do a better job of electronic records management, including classifying, de-duplicating and purging data through the use of systems such as archiving, records and policy management, content monitoring/filtering, and content analytics tools.
Right now, she says, it's too easy to just keep buying more storage and keeping everything, and what's important to keep is intertwined with what's trivial. Not to mention that with no clear guidance or policy on data handling, she says, there's the risk of political appointees in outgoing administrations shredding data rather than turning it over.
"We've created a huge volume of stuff, and it's going to be impossible to sort it with any level of precision," she says. "The longer it sits around, the more you lose context and run the chance that the data formats will become extinct. I think the result will be a great loss of information for the future." Logan apologizes for seeming so pessimistic, "but I've been covering this for nine years, and the progress has been minimal."