The entire ERAM program was a disaster that the FAA had been in denial about for years.  In 2009 they were proclaiming the program on time and on budget.

When they started trying to use ERAM at Salt Lake Center (ZLC) they found it was unstable and bug ridden.  But they had no way to test it other than on live air traffic.  So they did.

Because timelines were clearly more important to the FAA than safety, in spite of all the problems they went ahead with their plans to deploy ERAM at other facilities (including mine – Minneapolis Center or ZMP).

However, in the spring of 2010, it finally came to light how bad ERAM really was, and for a few months the FAA took a break from live testing of the system to work on fixing its many problems.  Then a few months later they started running it again on live traffic at its two keysites, Salt Lake Center (ZLC) and Seattle Center (ZSE).

But after continued testing and development, by spring of 2011, according to an Independent Operational Assessment (IOA), the system still wasn’t ready for further deployment.  The controllers’ union (NATCA) agreed with that assessment.

What was the FAA’s response to that critical assessment? – declare an In Service Decision (ISD) which meant it could be deployed at other facilities.

Some time ago I believed NATCA was setting itself up to be a fall guy/patsy for the failures of ERAM.  But when the FAA finally asked NATCA for real collaboration with the ERAM program the final nail was put in the coffin.

The instant the FAA asked NATCA to collaborate on ERAM, NATCA was placed in a lose/lose situation – they couldn’t claim to be a critical part of the National Airspace System (NAS) and yet not get involved with ERAM when the FAA conceded they needed help from its controller workforce.

But apparently NATCA also saw the program as an opportunity to be used for political gain.  NATCA believed they could step in and save ERAM, and thereby show how incompetent the FAA was and how smart they were.  Unfortunately they didn’t consider how messed up the ERAM program was.

When NATCA got the opportunity to become partners with the lemons that were ERAM, they decided to make lemonade.

However, that immediately turned NATCA’s involvement with ERAM into a conflict of interest.  They suddenly needed ERAM to succeed as much as the FAA did.  It was no longer about safety, but politics.

Today neither the FAA nor NATCA is willing to openly admit how much development ERAM still needs, as both have underlying political motives at heart; neither considers the controllers who have to work with it nor the overall safety of the air traffic system.

Case in point, for some time both Salt Lake Center (ZLC) and Seattle Center (ZSE) have been running versions of ERAM neither the FAA nor NATCA is will to run full time at other facilities because of all the known bugs (the known bug count is somewhere around 1000!).

But nonetheless, last month my facility ran the same version of ERAM on live air traffic two separate times for a total of 3 days.  But it was only on the weekend, and apparently people who fly on airplanes through ZMP’s airspace on the weekend don’t really count…

Don’t get me wrong – ERAM has made a lot of progress since the FAA last tried running it on live air traffic back in 2009.  But any air traffic computer system that has over 1000 known bugs isn’t fit for use on live air traffic.  If there are that many known bugs, one can only speculate as to the number of latent (unknown) bugs still in the software.

I’ve been accused of being an ERAM “hater” because of my views on it from the management side of the FAA, and more recently from the air traffic controllers’ union (NATCA) side.

As is typical of people wanting to avoid addressing facts, that’s a gross oversimplification that ignores my complaints and concerns about ERAM.

The truth is that I’m not opposed to ERAM.  What I object to, and have always objected to, is the way the ERAM program has progressed from its very inception.  It’s another one of the FAA’s many repeated examples of how not to upgrade a system that’s critical to how air traffic controllers do their jobs.

ERAM is an overly ambitious program, the likes of which the FAA has historically proved unable to successfully manage.  As proof consider the disastrous Advanced Automation System (AAS) including the Initial Sector Suite System (ISSS) in the 1980-90s and the problems with the Standard Terminal Automation Replacement System (STARS) program of late 1990′s/early 2000s.

Obviously the FAA thinks more of its ability as an organization to successfully manage these major programs than they’ve repeatedly demonstrated the ability (or lack thereof) in the past.

As I accept that as a given, what I object to most is alpha and beta testing software on controllers and the flying public.

I take my job, and its inherent safety critical aspects very seriously.

When the FAA first started running ERAM on live traffic by any standard it was alpha software.  It was unsafe and shouldn’t have been used on live air traffic, a fact that was eventually conceded, but not before it was used for many hours on the unsuspecting flying public.

Since then it has been improved, but the last time my facility ran ERAM (last month) the version of the software we ran had over 1000 known bugs.  By any reasonable standard it still can only be considered beta software.

ERAM was put into use well before it was ready, forcing air traffic controllers into the unfortunate position of working with and around its many shortcomings and problems.  Today that situation still exists.

Since air traffic uses many analogies with pilots, because of its current state controllers have been forced to act as “test pilots” with ERAM.  Test pilots volunteer for that duty, but controllers haven’t been given a choice to “test fly” ERAM.

Most pilots wouldn’t be comfortable flying an aircraft they didn’t think was airworthy.  But pilots are given the final authority as to whether or not to fly their aircraft.  Controllers on the other hand don’t have that choice, and many have very little confidence in the “airworthiness” of ERAM.

At the very least, no test pilot would put passengers in aircraft they were testing, but that’s exactly what the FAA has been doing with ERAM.

The software wasn’t tested adequately before acceptance and there was no way to fully test the software other than on live air traffic.  The fact that testing could only be performed on live air traffic was itself a fundamental design flaw of ERAM.

Additionally the design of ERAM clearly didn’t consider the things controllers need to do their jobs more safely and efficiently.  As I don’t know what tools the technicians use to keep that system running, I can only assume the same is true for the tools ERAM gives them.

What I do know is that whatever analysis (if any) was put into the examination of how controllers work and what tools would help them do their jobs better is absent from ERAM’s design.

One of the long-running FAA managers’ catchphrases for those criticising their system “upgrades” is that controllers are resistant to change.

While that’s true (controllers don’t want to re-learn how to do their jobs), it’s also true that I would love (like most other controllers) to have tools to enable me to do my job better.

Instead most often what we get are tools that make parts of our jobs more difficult.  Parts of ERAM actually make routine parts of controllers’ jobs more tedious.

Most notably, with ERAM, dropping a data tag on an aircraft that a controller is no longer required to monitor is a two step process instead of the current single step process.

Because attention to detail is important in air traffic control, distractions of any form are undesirable for controllers working air traffic.  However, ERAM has many ways that it creates distractions for controllers.

Bugs in ERAM are a distraction.  Wondering if and when ERAM will crash is a distraction.

But a new “feature” in ERAM also creates distractions for controllers.

Air traffic control is highly compartmentalized.  Controllers are delegated a specific section of airspace they are responsible for keeping airplanes moving safely and efficiently in.

There are numerous rules to make sure that the responsibility for keeping aircraft separated in a section of airspace falls on a single controller to avoid confusion.  Controllers cannot let aircraft enter another’s airspace without permission, and can only give control instructions to aircraft in their airspace.

However, for contingent safety reasons should they notice, controllers are also responsible for pointing out problems they see in adjacent sections of airspace.  But under the current HOST system though there are few ways for a controller to see or know for sure what is occurring in adjacent airspace.

Thus, in general controllers work their own airspace, separate their own traffic and assume the controllers working adjacent airspace are doing the same; a prudent and reasonable expectation.

But instead of adding tools to ERAM to help controllers better work their own airspace, ERAM added the “feature” of AOI (Area of Interest), expanding the areas in which controllers would be alerted to problems outside their airspace, adding more responsibility for controllers to monitor their “neighbor’s” airspace as well as their own.  The claim is that the AOI will help controllers work more efficiently near their sector boundaries, but because of the way the air traffic rules are written it also expands their responsibility outside their sector.

Although to a casual observer it might make sense to have more eyes watching traffic, it starts to blur the lines of compartmentalized responsibility that is one of the cornerstones of air traffic control.  The end effect is that the ERAM AOI has the potential to distract controllers from monitoring traffic within their own sector.

That’s not to mention the various known bugs, little glitches and quirks of ERAM that will be around for some time to come.  The expectation clearly is that controllers will get used to them and work around them.  They’re distractions nonetheless, and those distractions are detrimental to safety.

It was clear that the FAA has always driven more by timelines and budget than safety when it came to ERAM deployment, but now the controllers’ union has political motives as well with regards to ERAM, so they’re both willing to roll the dice and run ERAM on live traffic even though they know it’s not ready.

If ERAM is ready for use on live traffic, then why doesn’t the FAA and NATCA just use it at all the facilities 24/7?

The answer is obvious: because they know it’s not ready.

But they’re running it at more and more facilities nonetheless.

You can’t have your cake and eat it too.

If both the FAA and NATCA are willing to run ERAM full time at some facilities and part-time at others, how can they argue it’s not safe to use?  Air traffic and the people on the aircraft flying at facilities with less traffic, and at busier facilities on the overnight shifts and on the weekends are no less important than those flying elsewhere and at other times.

If it’s “safe enough” (i.e. “close enough for government work”) then I say just turn it on and let the chips fall where they may.  Otherwise we still shouldn’t be running it on live traffic.

They’re now planning on expanding the program further (again), in spite of the significant ongoing bugs the software still has.

ERAM is one of the cornerstones of the FAA much touted NextGen system, and was intended to replace the aging HOST computer system that runs the radar displays at the FAA’s enroute air traffic control centers.

They first started testing ERAM in the fall of 2009 at Salt Lake Center (ZLC), and quickly discovered that it wasn’t even close to being ready for use with live air traffic.  That didn’t stop them then from continuing to test the system on live air traffic for months, including expanding the testing to other facilities (including the facility where I work, Minneapolis ARTCC or ZMP).

Eventually, in the spring of 2010, the FAA temporarily stopped testing ERAM on live traffic.

I don’t believe that they stopped as much because of the problems as much as because those problems were starting to be noticed outside the FAA.  As long as the FAA was able to quietly and secretly test the bug-ridden ERAM software on the flying public, they were willing to do so.

After all, neither the program contractor, Lockheed Martin, nor the FAA thought its many problems precluded it being used on live air traffic; they figured the air traffic controllers could simply work around its many problems.

At that time the ERAM program had already fallen well behind schedule, and was rapidly going over budget.

However, the ongoing problems with the software eventually made it into the press, and the FAA was forced to slow down the ERAM program.

They stopped testing ERAM on live traffic, and spent months on further software development.  Then last fall the FAA started testing ERAM on live traffic again first at ZLC, and then ZSE.

While the project has made progress, it’s still fraught with problems.  They had a major problem this last December with ERAM at ZSE forcing them to revert back to the HOST computer system.

The system is also still experiencing what they call “regression” issues with the software, which means that whenever they fix some bugs, they create others, a problem they’ve had from the very beginning of the program.

Additionally, there are still some significant problems (critical bugs) with the system that would interfere with controllers’ abilities to keep airplanes safely moving through the airspace system.

Because of the continuing problems, a recent Independent Operational Assessment (IOA) on ERAM at the keysites determined that the program was not ready for further deployment at other facilities.

The controllers’ union, NATCA, who has been extensively involved helping test and debug the ERAM system, agreed with the findings of the IOA.

In spite of that however, the FAA decided to declare an In Service Decision (ISD) for ERAM, which means they can now start testing it on live air traffic at other facilities.  My facility (Minneapolis Center, ZMP) is next on the list to start testing.

Since the FAA decided to start deploying ERAM and testing it on live air traffic last year, well before it was ready for that purpose, it comes as no surprise to me whatsoever that they’re willing to do it again.

In fact, I predicted that a little less than a year ago.

FAA officials apparently admit “concerns” about the ERAM transition, but will the FAA choose to start using it again on live traffic before it’s ready regardless?

I’m betting they will, simple because they willingly and knowingly have already done that.

The entire ERAM program continues to fall further and further behind schedule, and is running millions over budget because of the continuing problems.  With the Federal budget showdown in Washington, it was just a question of time before the FAA managers in charge of ERAM started panicking again and rushing ahead with the program again.

If they can tell Washington and the taxpayers that ERAM is up and running, and deployed at more and more facilities, in spite of the problems it will make it look like they’ve done their jobs.

But unlike the last time they tried this gambit, this time plenty of people are paying attention.

The cost overruns and problems with ERAM have resulted in a Department of Transportation Office of Inspector General (IG) audit that was initiated last fall, the results of which are supposed to be completed this summer.

Considering the continuing problems, it begs the question: how much more time and money will the FAA (and the government) invest in ERAM?

But in the meantime, the ERAM program continues to be an FAA money pit.

The article states that ERAM:

…won’t reach nationwide operational readiness until August 2014 and will cost $330 million more than expected…

But it’s no news that the ERAM project has been delayed by problems, and those problems are costing the government (and the taxpayers) millions.

I’ve been patiently waiting to read the results of an Office of Inspector General audit on the ERAM program that was initiated in September of 2010.  I know that many of those working on the ERAM project at my facility were interviewed by the Inspector General last fall for their audit, so it will be interesting if and when that report appears what it says too.

(It’s unknown what the status of that audit currently is.)

Apparently the Federal Aviation Administration (FAA) also contracted out a review of ERAM in a letter dated June 1st of 2010 to the MITRE Corporation’s Center for Advanced Aviation System Development (CAASD).

FierceGovernmentIT got wind of the 197 page report completed in October of 2010, and eventually obtained a copy of it through a Freedom of Information Act (FOIA) request.  According to the FOIA reply dated January 12, 2011, the request was made only a few days after the review was completed, on October 12th, 2010, so it took the FAA three months to actually cough up a copy of the report.

Notably, each page of the report is marked,  “For Official Use Only.  Not approved for public release.”  (So much for transparency in government…)

MITRE somehow knew that the FAA wouldn’t want any report highlighting the problems with ERAM easily accessible to the general public.

The MITRE CAASD report is the result of an FAA request for an “independent review” of the ERAM program.  The “Tasking Letter” (Appendix A, page 161 of the report) was signed by both James H. Washington (FAA Acquisition Executive) and Richard L. Day (Senior VP, Operations).

Never mind that MITRE is hardly independent, nor impartial; they work primarily under contract for the Federal Aviation Administration.  From their own “About Us” section of their website:

MITRE has a longstanding relationship with the Federal Aviation Administration (FAA). Since 1958, the year MITRE first opened its doors, we have helped the FAA address the nation’s most critical aviation issues.

Our role was enhanced in 1990 when the FAA created a Federally Funded Research and Development Center and selected MITRE to operate it. The new entity is the Center for Advanced Aviation System Development (CAASD).

Even they couldn’t sugar-coat the state of the ERAM project, quantifying what those working with ERAM already know: that the program has more than its share of problems and is far from ready for nationwide use.

Why the FAA would need to spend more money for someone else to tell them their ERAM program is a mess, a program that’s already been bleeding funds for years and is falling further and further behind schedule, is beyond me.

Maybe it’s because the FAA higher ups are in disbelief that the program isn’t going well.  Maybe they were actually believing the press releases they were spewing as recently as last year that ERAM was on schedule and on budget.

Maybe those at FAA Headquarters felt misled by those managers in the field that were telling them that ERAM was progressing nicely; eventually found out it really wasn’t, and now doesn’t trust their own managers to give accurate reports on the status of the program.

Maybe it’s a way for the FAA to cover for the cost overruns and delays.  Now that the ERAM problems are no longer a secret, asking for reports and coming up with plans to address the problems buys them more time.

Finally, maybe it’s a tacit admission by the FAA that they’re in way over their heads and are now looking for an outside expert to tell them how to fix what ails the ERAM program, instead of listening to their program managers’ and the contractor’s (Lockheed Martin) claims that all is fine.

If the latter is true, it’s better late than never I guess.  That is, as long as they actually heed those reports, which they likely won’t.

They’ll do what the FAA always does:  cherry-pick just a few of the easier recommendations and follow them, claiming that they’re following the advice of the report (albeit only in small part), and otherwise continue to do business as usual.

After all, up until last spring many within the FAA thought the ERAM program was progressing just fine.  There are plenty within the FAA that still think ERAM is fine.

The project contractor, Lockheed Martin doesn’t care because once the FAA formerly accepted ERAM, the longer it takes the more money the contractor makes.

In the meantime, controllers at my facility (Minneapolis Center or ZMP ARTCC) are now getting a second course of training (refresher) in preparation for live ERAM use, which is supposed to happen again for us this spring.  ZMP will be the third facility to run ERAM on live traffic (again) since it’s an alternate key site.

We’ve now heard that one of the new “features” of ERAM (the addition of a Runway Visual Range/RVR field for each aircraft included in their flight plan) finally got removed because after years of working on it they still couldn’t get the data to properly pass to the facilities that needed the RVR.

Now that ERAM has become another FAA money pit, they’re starting to strip out things they can’t get to work and didn’t need to begin with.  Instead of working on the basic functionality needed to keep airplanes apart, they’ve been distracted all along with developing features controllers don’t need.

That’s what happens with non-controllers design a system without input from the experts who know what tools they need to do their jobs.

We’ve also heard that in meetings ERAM managers are now trying to decide what problems to spend money on to fix and which ones to ignore.

However, it was only a little over a month ago (December 14th, 2010) that while running ERAM on live air traffic, the second key site facility to go live with ERAM (Seattle Center or ZSE) had a series of circumstances that resulted in every single ERAM radar display to fail (red “X”) in succession.  That evening they reverted back to their old HOST computer system until software changes could be made to ERAM.

After those patches were made a few weeks later ZSE started running ERAM again on December 27th.

The event highlighted the probability that as ERAM is progressively deployed into facilities that run more and more traffic that there will be a higher likelihood of those series of yet undiscovered series of circumstances occurring with catastrophic results to the ERAM system.

From the FAA Administrator’s Fact Book (page 9):

Air Route Traffic Control Center Activity, Aircraft Handled Jan-Jun 2010:

Seattle Center (ZSE) 574,000

Salt Lake Center (ZLC) 632,000

Minneapolis Center (ZMP) 877,000

Atlanta Center (ZTL)  1,394,000

Seattle Center (ZSE) is the least busiest enroute center in the lowest 48 states, yet they still encountered a problem that crashed the entire ERAM system in the facility.

ERAM is getting better.  But the December 14th incident at ZSE proved it’s still not ready for use at the busier facilities.

That’s not to say that ERAM still doesn’t have its share of critical bugs, and it’s still likely not ready for deployment at the busier facilities elsewhere in the country.

I have heard of bugs that were corrected re-appearing in newer ERAM versions still, a problem they’ve been having from the very beginning.

ERAM also still has the potential to crash both an adjacent facility’s HOST computer system, as well as an adjacent ERAM computer system.

But both Salt Lake Center (ZLC) and Seattle Center (ZSE), the two key sites for ERAM, have been making extended continuous runs on ERAM, and we’re not hearing much about how that’s been going at all, even though my facility (Minneapolis ARTCC or ZMP) will be the next facility to run ERAM.

Maybe it’s been going great there.  Maybe all the critical bugs in ERAM have been corrected.  But considering the number of problems ERAM was having a year ago I highly doubt it.

So why aren’t we hearing more about what’s going on with ERAM at those facilities?

I believe it’s primarily due to habituation.

Have you ever walked into a room that had a peculiar odor?  After remaining in the room a while you won’t notice that odor anymore.

That’s a form of habituation.

Here’s the scientific definition:

1. The gradual decline of a response to a stimulus resulting from repeated exposure to the stimulus.

Air traffic controllers get habituated to working with defective equipment and procedures; it happens all the time.

They’ll initially notice and complain about the problems but they’ll invent and/or be ordered to use workarounds to address those problems.  Eventually those workarounds just become a way of doing business, in spite of the fact that the equipment and/or procedure is broken.

Often those procedures and equipment remain broken, never to be fixed.

Air traffic controllers are a pragmatic lot.  They have a pretty straightforward job to do:  keep the airplanes safely apart.

In spite of the faulty equipment and procedures controllers know they still have to do their jobs.

Although they’ll initially complain about faulty equipment and procedures, after a while they’ll give up and just deal with it.

The workarounds are always easier than trying to get the FAA to actually fix/correct a problem.

Of course once the controllers stop complaining about problems and filing reports on them, the FAA often chooses to believe they’ve been magically fixed, when they’re really just “out of sight, out of mind”.  Thus the problems go on forever without ever being addressed.

If controllers object to being forced to use new equipment and procedures that don’t work, the FAA simply accuses them of  “being resistant to change”.  Although controllers may be resistant to change, there isn’t a single controller I know that wouldn’t love to have better tools to make doing his job easier.  (The only caveat is that they want those tools to work properly.)

Ultimately controllers come to learn that complaining won’t change anything in the FAA, because those tasked with making decisions about using the faulty equipment and procedures don’t actually have to use them to accomplish their jobs.  Using the faulty equipment and/or procedures always becomes the air traffic controllers’ problem.

The managers in the FAA making decisions to start using the faulty equipment have completely different priorities than air traffic controllers, and they know that controllers are (generally) a smart lot and will figure out how to use the crappy equipment and procedures that are forced upon them (and always have).

If they don’t and something goes wrong, they can always use the air traffic controller as a scapegoat:  he failed to perform.

That’s why the FAA “formally accepted” ERAM from its contractor, Lockheed Martin, before it was ready for use in air traffic.  To those managers in the FAA, staying on schedule and budget was infinitely more important than whether or not the equipment worked.

Now the taxpayer is on the hook for paying for the numerous fixes ERAM needs before it can be used to safely separate airplanes.

After using ERAM for a while the controllers at ZLC and ZSE will become habituated to its problems.

They’ll figure out ways to deal with the more critical problems and ignore the rest of them.

It’s very similar to what happened with URET (User Request Evaluation Tool).  The FAA started deploying this paper flight strip replacement system to the enroute centers sometime around 1997, completing its deployment in 2006.

Most controllers like URET because it relieves them from the tedium of managing paper flight strips.

URET also has much improved route amendment functionality over the previous typed “6-7-10″ amendment controllers used to have to do.

But years later it still doesn’t work that well for some things, and still has numerous bugs in it that have yet to be corrected.

URET has this functionality called “conflict probe”.  It’s supposed to continuously detect whether or not aircraft are predicted to get too close together.  In such a case URET flags those flights for a controller’s attention.

However, the conflict probe only works accurately and in a timely manner maybe 85% of the time.  It sometimes creates false alerts (creating a distraction to controllers), or doesn’t create an alert when it should.

However, the FAA has made sure to tell controllers that URET is only a “tool” to keep airplanes separated; if URET is wrong controllers have to recognize that fact and ignore it.  The FAA has made it clear that it’s not legal for controllers to use URET to separate airplanes, and if a controller trusts URET and it’s wrong and aircraft get too close together it’s the controller’s fault.  (Remember what Otter said to Flounder?…)

URET also still doesn’t work in non-radar areas (areas without radar coverage, and yes, Virginia, there are still non-radar areas in the continental U.S.).  That’s because it lacks fix posting and time information (information controllers use in a non-radar environment to keep airplanes separated), only displaying aircraft position graphically.

And the FAA knows those graphical route displays aren’t accurate enough to be used to keep airplanes safely separated.  Because of that, URET can’t be used for separating airplanes.  It’s just a tool.

In those non-radar areas the controllers have to use paper flight strips to determine where airplanes are.

It also doesn’t work very well in other situations, including the fact that with URET there was no way to indicate whether an aircraft was on a controller’s frequency or not, a problem which was highlighted after the Northwest 188 incident.

(To date, despite promises, the FAA has yet to develop a tool for controllers specifically made for that purpose.  It’s still the controllers’ problem to figure out which workaround to use to indicate an aircraft is on their frequency.)

Old flights sometimes magically reappear in the URET aircraft list.  Some flights in URET don’t probe their routes which in turn disables the conflict probe on that flight.

URET bugs have contributed to operational errors at my facilities while controllers were trying to address the problem.  It’s the same peril that any new equipment like ERAM has:  while controllers are coping with the equipment problems they go “heads down” and are distracted from keeping the airplanes apart.

It’s what happened to Eastern Airlines Flight 401 that crashed in the Everglades in 1972 when the crew was distracted by troubleshooting a problem and didn’t realize their autopilot had disengaged.

But after years of using URET, the controller workforce has become habituated to its problems and shortcomings.  They rarely file automation discrepancy reports on the URET bugs and ignore the inaccurate conflict probes.

The computer code for URET is included in ERAM so it will have the same shortcomings and problems URET already has, but has been rebranded the En Route Decision Support Tool (EDST).

The FAA’s solution to the known equipment and procedural problems like those in URET is akin to the old doctor joke:  Patient:  “Doctor, it hurts when I do this.”  Doctor:  “Then don’t do that.”

The FAA is almost always more interested in “mitigation” schemes than actually addressing the problem itself.

If a controller is distracted by his defective equipment and he fails to keep airplanes apart it’s his fault; not the fault of the faulty equipment.  The FAA tells controllers of known bugs and problems and simply says, work around them and don’t get distracted by them.  That’s because it’s a lot easier (and cheaper) than actually fixing the problem.

The FAA loves to play semantic games with controllers dealing with defective equipment and/or procedures, calling them “workload” issues and not safety issues.  However, addressing problems increases controller workload, and increased workload can lead to “task saturation”.  Once that happens a controller is no longer safely working air traffic.

Workload aside, defective equipment and procedures are a distraction nonetheless.

Now that ZLC and ZSE are running ERAM the FAA is getting close to an In Service Decision (ISD), after which they will start running ERAM on live traffic at other facilities (including mine).

They still need to test and correct the problems with the non-Hosted Terminal functionality in ERAM, and my facility will be tasked with that job.

It’s a major known shortcoming of ERAM they discovered after the FAA had already accepted it.  More issues like that are likely to follow.

Although I have no doubt that ERAM is working better than it was the last time the FAA started using on it live traffic, I’m also sure that it has a long way to go before it’s suitable for use nationwide.

But over time more and more controllers will become habituated to ERAM’s shortcomings, just like they already have with so much of the FAA’s equipment and procedures.  In turn the FAA will think everything is working fine.

I think it’s already happening.

But I did want to write an update on the status of the FAA’s En Route Automation Modernization (ERAM) since I’ve written about it so much previously.

ERAM is the enroute air traffic control system computer replacement that’s supposed to be a cornerstone of the FAA’s much touted NextGen air traffic control system, although due to numerous bugs and problems with the ERAM program its testing on live traffic was suspended this spring.

Since my facility (Minneapolis ARTCC or ZMP) is no longer a key site for ERAM we’ve been spared runs on live air traffic for the time being.  But that also takes us out of the loop a bit when it comes to the status of ERAM development, even though we have been running live traffic on our backup computer system (EBUS/DARC) in the middle of the night (on the mid-shifts) in order to perform testing on ERAM.

The current two key sites for ERAM are Salt Lake Center (ZLC) and Seattle Center (ZSE) and they’re the only facilities authorized to run ERAM on live traffic.  Once ZLC and ZSE testing determine that ERAM is fit for use on live traffic again they will have an In Service Decision (ISD) after which non key sites can start running ERAM on live traffic as well.

ZMP was brought on board as a key site in part to be able to test the ERAM functionality with Non-Hosted Terminal (approach control) facilities, a configuration that apparently both ZLC and ZSE lacked.

Non-hosted Terminals are non-ERAM facilities adjacent to an ERAM facility that need to be able to receive and send flight plans back and forth to ERAM.  Since the two key sites didn’t have the adjacent facilities to be able to test this functionality, ZMP offered to do the job.

However, with all the problems they were having with ERAM it was eventually decided that the testing of the Non-hosted Terminal functionality could wait, and ZMP was no longer a key site.

This in not only indicative of how haphazard the ERAM development cycle has been, but of how far ERAM has still to go before it’s ready for deployment across the country.

Right now the FAA and the ERAM contractor, Lockheed Martin, are still working on ERAM to get it to a state where it can reliably perform basic functionality and not worrying about the extra functionality ERAM will need to work across the country.

That’s what they should have been doing initially, instead of rushing to use ERAM on live traffic simply to meet an unrealistic deployment schedule well before ERAM was able to perform basic functionality reliably.

The FAA is now tentatively planning on starting to deploy ERAM at other enroute facilities early next year although the “waterfall” (deployment schedule) is far from set in stone and thereby isn’t being published anywhere.

So much for the project being on time and on budget…

ZLC started testing ERAM on live traffic in August again and has completed some multi-day operational runs.  Since they still had some critical problems, they’re waiting for a new version of the software before they’ll make an attempt at other extended live runs and eventually a continuous 24/7 operational run again.  (The last time they tried this was back in March of this year.)

The ERAM software sounds like it’s improving, but they’re still apparently experiencing critical bugs with basic functionality such as missing aircraft computer identification numbers in data blocks and some tracking issues, as well as continued memory leaks that make the system sluggish and/or unstable after running for only a few hours.

The fact that they’ve been working continuously on ERAM bugs since this spring, when they stopped testing it on live traffic, and still have enough critical bugs to preclude a continuous operational run show that there was no way ERAM was fit and/or ready for use on live traffic last spring even though they were using it for that purpose.

It’s just too bad the FAA couldn’t come to that conclusion on their own, instead finally bowing to pressure from various sources to stop testing the faulty software on live traffic.

In the meantime they also finally developed a method to more easily transition back and forth from the current HOST computer system to ERAM they’re calling “shadow mode”.  Shadow mode has the ERAM system shadowing the HOST computer so that it can get flight plan data prior to switching over, making the entire transition a lot less work intensive for the air traffic controllers and a lot less messy.

This functionality should have been part of ERAM initially, but apparently the FAA and Lockheed Martin were delusional enough to believe that they were just going to turn ERAM on and everything would work great.

When that didn’t happen and they realized they would need to perform the transition back and forth from ERAM to HOST repeatedly they finally decided to develop a method to more easily do so.

That missing functionality is another indication of how screwed up both the FAA and Lockheed Martin’s approach to the ERAM program has been from the very beginning.

There’s functionality that should have been included from the very beginning that they’re just getting around to adding now.  There’s other functionality they didn’t seem concerned with and/or didn’t even know they needed (like the Non-hosted Terminal issue) because it wasn’t part of the original testing regimen.

Thus there’s sure to be more problems encountered as they find new issues they hadn’t thought about when they start deploying ERAM across the country, not the least of which is running ERAM with lots of air traffic.

The FAA also had intentions of decommissioning both its HOST computer system, and its backup computer system (EBUS/DARC) after ERAM was up and running.

However, since ERAM is the backup for ERAM (running the same software on multiple channels) if there is a critical bug in the software that causes one ERAM channel to crash, it would be fair to assume that the bug would occur on the other channels as well causing all the ERAM channels to crash in turn.

That means that until ERAM attains a much higher level of stability and reliability the FAA should maintain a legacy backup system (such as EBUS/DARC), but that’s not to say they will.

The ERAM software is the program that is supposed to replace the FAA’s HOST computer that currently runs its enroute controllers’ radar displays and processes flight plan information.

Because of the deployment schedule (the FAA likes to call it “waterfall”) for ERAM, it must be able to interface properly with those facilities running the current enroute computer system (the HOST computer).

Of course the plan is to eventually have all the facilities running ERAM, but the switchover won’t be instantaneous nationwide; some facilities will run ERAM before others.

However, there is at least one software bug in ERAM that they’ve known about since last fall and have not fixed, where under certain conditions the ERAM program had the potential to overload an adjacent facility’s HOST computer with flight plan data, eventually causing it to crash.

Although the bug in ERAM is difficult to replicate it hasn’t yet been corrected, and reared its ugly head not long ago during an ERAM test.  At that time a software technician noticed the problem developing and “pulled the plug”, disconnecting the HOST computer from the interface to an adjacent ERAM computer before it caused the HOST computer to crash.

Now one would think the the FAA and its ERAM contractor, Lockheed Martin, would have made tracking down and correction of this particular bug a priority a long time ago, given the damage it could do.

Instead, however, the FAA decided to patch the HOST computer (a computer system that’s quite stable) to protect it from this ERAM bug.

It’s another band-aid fix for the ERAM program, that continues to have bugs serious enough to cause a significant degradation in safety in the air traffic system.

However, the latest HOST patch won’t protect it from any of the other rare and/or yet undiscovered bugs in ERAM.  Given the number of known bugs in ERAM it’s likely that there are more that are capable of crashing adjacent HOST computers systems.

They were having so many problems tracking and fixing bugs with ERAM that they redesigned the bug-tracking and correction schemes this spring.  But the more things change, apparently the more they stay the same.

If the FAA now feels the need to start patching the HOST to protect it from ERAM bugs it certainly doesn’t say much for the state of the ERAM program itself.

At least in this case the FAA has decided it’s easier to patch the HOST as protection against ERAM bugs than it is to fix the bugs in ERAM itself.  And if that’s not the “cart pulling the horse” I don’t know what is…

As you may know the FAA stopped testing the ERAM software on live traffic in late March, due to the many serious problems they were experiencing with it.  I don’t believe it has been used on live traffic since.

Sometime in April some of the centers began testing version 2 of the ERAM software, which also involved some firmware updates to hardware.  We’ve gone to our backup computer system (EBUS/DARC) several times on the overnight/midshift since then to accomplish more ERAM testing in the background (as have many other facilities), but as far as I know no facility has started using ERAM to separate live traffic again yet.

Salt Lake Center is still slated to be the first enroute center to start running ERAM again on live air traffic, but with the increase in summer air traffic and given the problems with ERAM previously the FAA may have finally learned its lesson and be hesitant to roll the dice with ERAM again, at least at this time of year.

Later in May apparently in spite of the fact that ZMP is a keysite for ERAM, it was decided that our facility was not going to be part of the Initial Operational Testing and Evaluation (IOT&E) process, which gets us off the hook for any live runs with ERAM, at least for the time being.

Due to the problems with the way the FAA and the ERAM contractor (Lockheed Martin) were reporting and/or fixing bugs with the ERAM software previously, they’ve been working on changing that process.

They’re acknowledging that in the process of fixing bugs many of them would get worse, and/or they would create new bugs, so the end result was that they wouldn’t make any net progress.  Of course that’s common knowledge to anyone who was paying attention at the time, but it didn’t seem to phase those in the FAA overseeing the project.

They continued to test the faulty software on the flying public until word of how poorly the project was progressing got out, eventually including a report and recommendations from the Department of Transportation Inspector General.

Right now it sounds like they’re trying to optimize the Problem Reporting (PR) system for ERAM and how they determine whether or not ERAM is suited for use on live traffic.

Previously the FAA and the ERAM contractor, Lockheed Martin, weren’t too concerned with either, and although they should have had reasonable error reporting, correction and assessment processes in place when they started the ERAM project they obviously didn’t.  All of them should have existed well before they started testing the system on the flying public regardless.

At some point the FAA is going to start testing ERAM on live air traffic again.  Hopefully when they do so ERAM will be in a state more suited for the task than it was the last time the FAA tried using it for that.

In the meantime ERAM development continues quietly “under the radar” so to speak…

The Department of Transportation Inspector General (DOT IG), Calvin L. Scovel III, testifed before the House Subcommittee on Aviation on April 21st, 2010, about the status of the FAA’s entire NextGen project, including ERAM, stating that (my emphasis):

The $2.1 billion ERAM program will replace the existing hardware and software at facilities that manage high-altitude traffic.  ERAM,  however,  is experiencing software-related problems at FAA’s key initial operating site in Salt Lake City.  These problems include radar processor failures, problems in handing off traffic between controllers, and critical flight information being paired to the wrong aircraft.  FAA is spending about $14 million per month to resolve these problems and deploy ERAM at other sites.

and

While FAA does not believe the system to be fundamentally flawed, it has postponed the in-service and operational readiness decisions for ERAM at Salt Lake City by 6 months, both originally planned for December 2009.   We have not assessed the severity of the problems with ERAM,  but FAA officials are concerned about the ERAM transition at larger, more complex sites like Chicago and New York.  These locations have unique airspace and operational issues that will require adaptation of the system’s software to accommodate their needs.

And finally:

FAA  officials acknowledge that it is unlikely that all  20 systems will be fielded nationwide and controlling traffic on a regular basis by December 2010 as planned. FAA must take steps to ensure that problems with ERAM are resolved and make realistic adjustments to the program’s schedule.

You can see the entire DOT IG’s testimony here (which also discusses significant problems with other FAA projects, including the FAA’s Telecommunications Infrastructure/FTI program.)

Instead of simply re-releasing the warm and fuzzy press releases from the FAA on ERAM as news,now the media is  finally starting to report what’s really going on with the ERAM project.

Notably, the IG admits that they are trusting FAA officials to large degree about the details of the ERAM problems, as they admitted they didn’t actually examine the “severity of the problems with ERAM”.

There are many within the FAA that think the program is going just fine in spite of the problems, and while the “FAA does not believe the system to be fundamentally flawed” that may or may not be true.  Right now there seem to be some fundamental problems with tracking aircraft within the ERAM system.

The IG also suggested that the FAA must “make realistic adjustments to the program’s schedule” and “ensure that problems with ERAM are resolved” neither of which the FAA was previously willing to do by its own accord.

Doesn’t this sound a lot like trusting the fox to guard the hen house?

None of it really bodes well, because if the FAA and Lockheed Martin can’t get ERAM working at Salt Lake Center, and they know it’s going to require adaptation for the busier centers elsewhere in the country, how long will it really be until ERAM is actually ready for nationwide use?

FAA officials apparently admit “concerns” about the ERAM transition, but will the FAA choose to start using it again on live traffic before it’s ready regardless?

I’m betting they will, simple because they willingly and knowingly have already done that.

Early in the week of March 22, the FAA finally gave in to pressure to stop running the faulty versions of the ERAM software on live air traffic.

We were also being told that allegedly they were going to take the time to allow the program contractor (Lockheed Martin) to make software changes that would fix all the major ERAM bugs before running it again on live air traffic.

But early in April, it appeared that the FAA was considering having Salt Lake Center (ZLC) run the latest ERAM software version (U4), even though they knew it didn’t have all the necessary corrections to run reliably 24/7.

And at the same time Minneapolis Center (ZMP) had scheduled more ERAM live runs in mid April.  Eventually those tests were canceled, but it shows there are plenty of people in the FAA ERAM program that still don’t have any problem getting right back to their practice of running versions of ERAM software they know have major bugs.

Both indicate that the attitude for many within the FAA toward testing ERAM with known bugs on live traffic hasn’t changed at all, in spite of appearances that the program was going to proceed differently from this point out.  (Maybe those individuals just didn’t “get the memo”…)

Knowing the FAA I have to wonder if this delay/”review” is really just intended to placate those who were objecting to the way the ERAM program was progressing; not a genuine inclination to change the course of ERAM deployment.

Time will tell if the FAA gets right back to testing ERAM software versions they know have bugs.  But for the time being, they appear to be taking the time to do what they should have been doing all along, which is fixing the major known bugs before trying more tests on live traffic.

However, facilities continue to test ERAM in the background, forcing controllers to run live traffic on their inferior backup computer system (DARC/EDARC/EBUS).

Here’s a recap of notable ERAM Runs/Tests at the two ERAM keysites (note that after each of the failed tests, there were software updates made):

• October 3, 2009:  Salt Lake Center (ZLC)  has failed ERAM test on live traffic forcing fallback to HOST.
• January 30, 2009 to February 8, 2010 – Salt Lake Center (ZLC) extended live traffic ERAM run.  This controlled and sterile test apparently gives the FAA confidence that ERAM is ready for 24/7 use.
• February 17, 2010 to February 23, 2010 – Salt Lake Center (ZLC) starts what was intended as permanent 24/7 run on ERAM that was aborted with fallback to HOST due to major problems (“SD” Version).
• March 6, 2010 – March 8, 2010 – Seattle Center (ZSE) test running version T7(?); numerous major problems.
• March 14, 2010 – March 20, 2010 – Salt Lake Center (ZLC) running version T9, forced to fallback to HOST due to problems.

I would love to believe that the FAA is going to take a different approach to the entire ERAM program from here on out.  But I’m not holding my breath…

Within the last few days, apparently the FAA has decided to stop running the new software on live traffic and make an “assessment” of the program, although certainly not by its own accord.

Since last year, the FAA has been routinely running its En Route Automation Modernization (ERAM software), still under development, on live traffic, with many known critical bugs at three key sites, including including Salt Lake Center (ZLC), Seattle Center (ZSE), and Minneapolis Center (ZMP).

In spite of the fact that the FAA and the program contractor, Lockheed Martin, knew of many significant bugs with the software, the FAA opted to run the software nonetheless, often playing down the seriousness of the problems.

New versions of software addressed a few of the significant bugs while at the same time ignoring most of them.  That fact didn’t prevent the FAA from trying the latest versions on live air traffic anyway.

And predictably, the bugs that hadn’t been fixed in the latest software versions inevitably cropped up again, in some cases leading to a shutdown of the ERAM software with a fallback to the legacy HOST software.

What was the FAA’s primary motive for testing software that they knew had significant bugs?  Apparently in part to meet timelines and deadlines for the software deployment which is falling further behind schedule, as well as ensuring that the program’s contractor, Lockheed Martin, gets cash bonuses built into the contract.

Under pressure from the controller’s union, NATCA, who started telling congressmen and senators about the problems with ERAM, and those controllers helping work on the project, some of whom were becoming increasingly frustrated by the lack of progress in correcting significant known bugs with the software, (and this writer would like to believe some other negative publicity) the FAA eventually agreed to stop running it on live air traffic for the time being and instead make an assessment of the program.

The FAA knew that the more opposition grew to the program the harder it was going to be to “fly under the radar” and continue with its reckless approach.  Many of the critical problems they have had with ERAM weren’t corrected in spite of numerous software updates, negating the FAA’s claim that the program was progressing satisfactorily.

A large part of the problem was that the ERAM software clearly wasn’t developed to the point of being ready for use to separate live air traffic when the FAA started using it for exactly that.  Even if all the critical known bugs were magically fixed today, ERAM still probably wouldn’t be ready for use with real air traffic, but it certainly would be a lot more suitable for that task than it currently is.

I have no doubt that the FAA would have continued with its “damn the torpedoes” approach were it not for increasing opposition from the controller workforce and its union.

Unfortunately for the FAA, the more controllers that were exposed to running the ERAM program, the more visibility the problems got, none of which were going away.

That sorry state of the software was only going to lead to bad publicity that the FAA didn’t want.

It was months too late for the FAA to stop the current course of the entire ERAM program deployment, and I have no doubt that the FAA would never have stopped to make an assessment of the ERAM program on its own.  After all, they seemed to think everything was going fine.

The state of development for the entire ERAM program is obviously well behind where it should be, and although lots of others recognized that fact, as an organization the FAA was either in denial or oblivious. Considering what happened with the Initial Sector Suite System (ISSS) program decades ago, it’s obvious the FAA has learned nothing over the years (or has simply forgotten).

The only acceptable and responsible course of action now is for the FAA to force Lockheed Martin to correct all the known critical bugs before the ERAM software is accepted for further use in separating live air traffic.

Anything short of that will inevitably put us right back where we started, and likely back into the same dangerous and faulty cycle.

However, knowing the FAA I believe that this assessment will likely result in a decision which will amount to merely “trying to do better”, rather than a real commitment to fix the known problems before further using the flying public as beta testers/guinea pigs for the ERAM software.

That’s because the FAA has already demonstrated that it’s willing to run faulty software because they’re more concerned about the ERAM deployment schedule than anything else.

They’re panicking because the clock is ticking.  The HOST software contract has already been extended.  The ERAM program schedule continues to slip.  Failing to hold Lockheed Martin to a higher standard isn’t going to make the ERAM problems go away, but is conducive to staying (more) on schedule, in spite of the fact that it puts the flying public at risk.

The FAA has always assumed (and will continue to assume) that air traffic controllers will be able to work around ERAM’s deficiencies and keep airplanes separated nonetheless.

Keep in mind that none of this precludes the FAA from continuing to run/test the ERAM software; it merely won’t be used on live traffic for the time being at the key sites.

Testing of ERAM in the background will continue, and will likely reduce the margins of safety and increase controller workload by forcing them to work live traffic on its outdated DARC/EBUS backup computers without flight plan processing so that ERAM can use the interfaces between the various facility computer systems.

But unlike ERAM, at least the DARC/EBUS software properly tracks radar targets…