FAA glitch shines spotlight on troubled telco project

A software problem led to numerous flight delays

The outage of a computer system used by airline pilots to file flight plans in the U.S will likely prompt a closer look at a US$2.4 billion telecommunications system that has grappled with numerous problems in the past.

The U.S. Federal Aviation Administration (FAA) offered few details Thursday about the exact nature of the glitch, which caused major delays and flight cancellations in airports across the country. But in a statement, the agency blamed a "software configuration problem" within the FAA Telecommunications Infrastructure (FTI) in Salt Lake City.

That problem brought down a system used mainly for traffic flow and flight planning services for about four hours this morning. The flight management system -- it's called the National Airspace Data Interchange Network (NADIN) -- was affected because it relies on FTI services to operate, the FAA said. There was no indication that the disruption was the result of a cyberattack, the FAA said.

FAA experts were investigating the outage and meeting with Harris Corp., the company that manages FTI to "discuss system corrections to prevent similar outages," the agency said.

In an e-mailed statement, a Harris spokesman said the company is working to "evaluate the interruption" to prevent future outages. "FTI has proven to be one of the most reliable and secure communications networks operating within the civilian government. Safety and security is our highest priority," the company said.

A spokeswoman for the Professional Airways Systems Specialists (PASS) union, which represents more than 11,000 FAA employees, told Computerworld the problem arose when scheduled maintenance on FTI in Los Angeles corrupted a router in Salt Lake City. A back-up router that should have kicked-in when the primary router went down failed to do so, resulting in the widespread outage, she said.

The $2.4 billion FTI program was introduced by the FAA in 2002 to replace seven FAA-owned and leased telecommunications networks. It provides a range of voice, data and video communication services for operations and mission support at more than 4,000 FAA and Defense Department facilities, according to Harris. The FTI network provides switching and routing services, as well as centralized infrastructure security monitoring services for the FAA.

An audit of the program released by the FAA's Inspector General last September cited concerns over delays in the project's implementation and doubts about the promised cost-benefits the network was supposed to yield. The report also noted several technical problems that had caused unscheduled outages to air traffic control operations.

"In some cases, these outages have involved simultaneous loss of both primary and back-up FTI services, which not only disrupts air travel but also creates potential safety risks," the inspector general report warned, pointing to several incidents in recent years.

On Sept. 25, 2007 for instance, all FTI services were lost at the Memphis Air Route Traffic Control Control Center (ARTCC), disrupting air traffic control for several hours and causing 566 flight delays, the report said. The problem stemmed from a "catastrophic failure" of an optical network ring that was supposed to offer built-in fault tolerance. The FAA was vulnerable to the same issues at Atlanta and in Jacksonville.

Another incident occurred on Nov. 9, 2007, when all primary and alternate FTI services were lost at Jacksonville, resulting in 85 flight delays. "We also found that when FTI outages occur, the services are not always restored within contractual timeframes," the inspector general's report said. In some cases, where services are supposed to be restored within three hours, Harris took twice as long to fix the problem. "Several areas remain critical watch items for decision makers as FAA moves forward with FTI," the report said.

Others have been critical of the program as well. PASS, for instance, has in the past voiced concern over safety and efficiency issues related to FTI.

PASS spokeswoman Kori Blalock Keller said the FAA needs to hold Harris accountable for the problems. "If they are going to provide service, we need to make sure they are reliable and they are quick" to respond to outages, she said. Although several FAA technicians were on hand in Salt Lake City today, they couldn't do much to help out because the FTI system is managed by Harris, she said.

According to Keller, the incident will likely prompt Congress to ask the FAA Inspector General for another review of the system. The National Air Traffic Controllers Association (NACTA) has also expressed frustration over FTI. After the failure in Memphis, the organization blasted the network as "unreliable [and] lacking suitable backup" and called it a source of "great frustration and deep concern" for FAA technicians and air traffic controllers. Bill Curtis chief scientist at CAST Software and co-author of the Capability Maturity Model used in software development today, said the outage highlights the havoc that can be created when something goes wrong in large, highly interconnected systems such as the FAA air traffic control system.

"It's not just one system, but a system of systems," he said. "If one of them starts behaving in a funny way, it starts propagating out and causes problems in other systems," said Curtis.

Join the newsletter!

Error: Please check your email address.
Rocket to Success - Your 10 Tips for Smarter ERP System Selection

Tags TelcoFAA

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Jaikumar Vijayan

Computerworld (US)
Show Comments


James Cook University - Master of Data Science Online Course

Learn more >




Back To Business Guide

Click for more ›

Most Popular Reviews

Latest Articles


PCW Evaluation Team

Louise Coady

Brother MFC-L9570CDW Multifunction Printer

The printer was convenient, produced clear and vibrant images and was very easy to use

Edwina Hargreaves

WD My Cloud Home

I would recommend this device for families and small businesses who want one safe place to store all their important digital content and a way to easily share it with friends, family, business partners, or customers.

Walid Mikhael

Brother QL-820NWB Professional Label Printer

It’s easy to set up, it’s compact and quiet when printing and to top if off, the print quality is excellent. This is hands down the best printer I’ve used for printing labels.

Ben Ramsden

Sharp PN-40TC1 Huddle Board

Brainstorming, innovation, problem solving, and negotiation have all become much more productive and valuable if people can easily collaborate in real time with minimal friction.

Sarah Ieroianni

Brother QL-820NWB Professional Label Printer

The print quality also does not disappoint, it’s clear, bold, doesn’t smudge and the text is perfectly sized.

Ratchada Dunn

Sharp PN-40TC1 Huddle Board

The Huddle Board’s built in program; Sharp Touch Viewing software allows us to easily manipulate and edit our documents (jpegs and PDFs) all at the same time on the dashboard.

Featured Content

Product Launch Showcase

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?