The ERAM software is the program that is supposed to replace the FAA’s HOST computer that currently runs its enroute controllers’ radar displays and processes flight plan information.

Because of the deployment schedule (the FAA likes to call it “waterfall”) for ERAM, it must be able to interface properly with those facilities running the current enroute computer system (the HOST computer).

Of course the plan is to eventually have all the facilities running ERAM, but the switchover won’t be instantaneous nationwide; some facilities will run ERAM before others.

However, there is at least one software bug in ERAM that they’ve known about since last fall and have not fixed, where under certain conditions the ERAM program had the potential to overload an adjacent facility’s HOST computer with flight plan data, eventually causing it to crash.

Although the bug in ERAM is difficult to replicate it hasn’t yet been corrected, and reared its ugly head not long ago during an ERAM test.  At that time a software technician noticed the problem developing and “pulled the plug”, disconnecting the HOST computer from the interface to an adjacent ERAM computer before it caused the HOST computer to crash.

Now one would think the the FAA and its ERAM contractor, Lockheed Martin, would have made tracking down and correction of this particular bug a priority a long time ago, given the damage it could do.

Instead, however, the FAA decided to patch the HOST computer (a computer system that’s quite stable) to protect it from this ERAM bug.

It’s another band-aid fix for the ERAM program, that continues to have bugs serious enough to cause a significant degradation in safety in the air traffic system.

However, the latest HOST patch won’t protect it from any of the other rare and/or yet undiscovered bugs in ERAM.  Given the number of known bugs in ERAM it’s likely that there are more that are capable of crashing adjacent HOST computers systems.

They were having so many problems tracking and fixing bugs with ERAM that they redesigned the bug-tracking and correction schemes this spring.  But the more things change, apparently the more they stay the same.

If the FAA now feels the need to start patching the HOST to protect it from ERAM bugs it certainly doesn’t say much for the state of the ERAM program itself.

At least in this case the FAA has decided it’s easier to patch the HOST as protection against ERAM bugs than it is to fix the bugs in ERAM itself.  And if that’s not the “cart pulling the horse” I don’t know what is…

The FAA tends to talk a good game when it comes to safety but then often fails to take action when it comes to improving safety within aviation and/or the air traffic system.

It’s also a fact that the FAA can be outright reckless when it comes to safety issues, as demonstrated by their willingness to test their En Route Automation Modernization (ERAM) software on the flying public while it was still ridden with bugs; well before it was suited for the task, apparently simply assuming the air traffic controllers using it would be able to deal with any software problems that arose and still keep the aircraft safely separated.

Only after word of how poorly the ERAM software was performing got out did the FAA decide to stop testing it on live traffic.  Now months later after continued development the software apparently still isn’t ready for use on live traffic.

I wrote some time ago how the FAA is really only motivated to address safety issues when they hit the headlines.

A recent example of that approach came last month when the news reported that the FAA hadn’t yet required the airlines to fix their cockpit windshield heaters even though the FAA had known for years that the heaters could start fires.

Not surprisingly, within a couple of weeks of the story appearing, the FAA finally took action, issuing an order to the airlines requiring they fix the problem.

It’s proof positive that the FAA tends to ignore safety issues until they become front page news.

So it’s curious that the FAA keeps inventing new safety reporting programs when they’ve demonstrated they’re not interested in being proactive when it comes to fixing safety related problems.

After years of attacking controllers for making mistakes, the FAA finally came up with a non-punitive safety reporting system in its Air Traffic Safety Program (ATSAP).

The ATSAP program is fantastic in that it finally gives controllers immunity from disciplinary action if they report safety related problems after years of being threatened or disciplined by FAA management for errors.  But it has also in large part bought the controller workforces’ silence when it comes to safety issues and operational errors.

It’s pretty clear that a vast number of the ATSAP reports are just being filed away and ignored by the FAA.  That’s because the program doesn’t have the authority to mandate any changes, and as the FAA routinely ignores outside safety recommendations (like those from the National Transportation Safety Board or NTSB), it’s sure not going to respond in any meaningful way to any internal safety program.

As long as those reports are kept internal to the FAA and out of the news, they’re fine with ignoring them.

Maybe that’s why the FAA creates their internal safety reporting systems:  to ensure those problems are kept internal and out of public view…

So it’s notable that on July 1st, we were briefed via a Memorandum that the FAA, cooperatively with the FAA air traffic controllers’ union (NATCA), and PASS (the union representing FAA technicians, inspectors, and other safety support staff) were introducing another safety reporting program called the “Partnership for Safety”.

According to the memo, the program:

…seeks to identify safety issues before an incident or accident occurs.  The Partnership for Safety program actively seeks your input on what problems or trends exist today in our air traffic control system that need to be fixed.  We are committed to working together to address these issues, and will spend the time, resources and energy to keep safety first.

But the FAA already has a relatively new safety reporting system in ATSAP.  Here’s what it says on the ATSAP website about the program:

The intent is to identify and report all events that may or did lead to a breakdown in safety, or increase risk to our operation. If we want to mitigate all safety risks, we need to identify and study the thousands of unreported events that may reveal the one critical safety event that could result in disaster.

The newest Partnership for Safety program appears to mirror the ATSAP program, and although only air traffic controllers have an associated Memorandum of Understanding (MOU) with the FAA that offers controllers immunity for reporting errors through ATSAP, the program is available to all FAA employees.

So why would the FAA spend the time and money to create another program that does what ATSAP is already supposed to be doing?

If employees have safety concerns, should they file reports under ATSAP, under the Partnership for Safety program, or both?

The new program is either an implicit admission that the FAA’s current safety reporting program (ATSAP) doesn’t really work, or is just more smoke and mirrors to make its employees and the general public think that the FAA is committed to safety (if one safety reporting program is good, than two is better!), when what they’re really committed to is simply business as usual.

Regardless of what the safety reporting system is, until the FAA has a safety program that legally binds them to correct safety related problems that are reported to them it will continue to turn a blind eye to them (or at least until those problems hit the headlines…).

Normally aircraft are all on filed routes, so controllers know exactly where they’re going.  They keep the airplanes apart based on those routings.

Once aircraft start turning off course to avoid cells of weather, the controllers’ task of keeping all the airplanes apart becomes much more complex.  When aircraft are deviating for weather controllers no longer know exactly where the aircraft are going.  Pilot requests to deviate and for turbulence reports also increase the controllers’ workload dramatically.

A sector that can handle a certain number of aircraft safely and efficiently under normal conditions can only handle a fraction of the aircraft safely and efficiently if they’re deviating around weather because of the added workload and complexity that occurs.

Since it’s important that air traffic flow efficiently (and safely) through the sectors, the FAA has orders regarding sector loading and efficiency that are contained in its Facility Operations Manual 7210.3.  (Note:  Even though I intertwine efficiency and safety because an efficient sector is a safe sector, for the record FAA managers claim there’s no correlation between the two.)

The enroute centers’ respective Traffic Management Units (TMU, overseen by the Air Traffic System Central Command Center, ATCSCC) are tasked with monitoring monitor traffic flow in sectors and ensuring that traffic flows smoothly.

One of TMU’s tasks, as outlined in the 7210.3, is to oversee the Monitor Alert Program (MAP).  As stated in that order:

Section 7. Monitor Alert Parameter
17-7-1.PURPOSE

The Monitor Alert Parameter (MAP) establishes a numerical trigger value to provide notification to facility personnel, through the MA function of the ETMS, that sector /airport efficiency may be degraded during specific periods of time. The efficiency of a functional position or airport in providing air traffic services is a shared responsibility of the TM team. That team consists of the ATCS(s), OS(s), and the TMU. These entities must monitor, assess and act on sector/airport loading issues to ensure that these NAS elements operate efficiently.

As specified in the same order, each enroute air traffic sector is assigned a numerical value that denotes the total number of aircraft it can work efficiently.  In the event traffic is predicted to approach or exceed that number, the order also specifies what action is supposed to be taken.

Our Front Line Managers (FLMs) monitor the MAP number continuously in the operational areas as well.

One of the many problems with the program is according to the order, the “TM team” is supposed to include the OS (Operations Supervisor), the TMU (Traffic Managment Unit) and the ATCS(s) (Air Traffic Control Specialists).  However, controllers are never briefed on the 7210.3; don’t read it; and most don’t generally even know the details of that particular order.

However, the order reads: “These entities must monitor, assess and act on sector/airport loading issues to ensure that these NAS elements operate efficiently.”  And who better to know the efficiency of a given sector at any point in time than the air traffic controller working it?

Unfortunately, at least at my facility, the ATCS(s) are effectively left out of the equation when it comes to the Monitor Alert.

If a sector goes “red”, or exceeds its efficient traffic loading index, the order also specifies what’s supposed to happen, albeit vaguely.  It says:

2. For red alerts – notify the affected area of the alert, indicating the expected impact and recommended action.
3. For yellow alerts – notify the affected area of the alert when analysis indicates that the ability of the sector to provide efficient air traffic services will be degraded due to abnormal operations.

What happens at my facility is that TMU calls the area supervisor and simply notifies them of the alert.  As far as I know they rarely if ever recommend any action.

In response to the notification sometimes the supervisor tells the controllers in the sector they’re going to get busy, or assigns additional controllers to work the sector, but rarely is anything else done.  If the MAP number exceeds the sector MAP by a large margin, or if the supervisor tells the controller and the controller objects strongly enough they may reroute some aircraft around the busy sector.  And then again they may not…

I would think it’s implied that some action would be taken when sectors went red or exceeded their peak efficiency volume, but TMU and FAA managers treat the Monitor Alert as mostly just an advisory program.  The order does say if a sector routinely goes red they’re supposed to examine what is causing that, but on a daily basis TMU and the FAA just treat it as an informational:   i.e. “Just FYI, you’re going to get busy.”

So it’s not as if anyone is actually going to take any action when traffic exceeds the number a sector can work efficiently, as per the order, but it’s supposed to be noted (and logged) at the very least.

The reason I bring this all up now is because when it comes to summer weather and deviating air traffic, the Monitor Alert Order says (my emphasis):

The ability of a functional position or airport to provide air traffic services may be affected by a variety of factors  (i.e., NAVAIDs, meteorological conditions,  communications  capabilities,  etc.); therefore MAP is a dynamic value which will be adjusted to reflect the capabilities of the functional position or airport.

and

The MAP value will be dynamically adjusted to reflect the ability of the functional position to provide air traffic service . During periods of reduced efficiency the MAP will be dynamically adjusted downward and conversely, when efficiency is improved, the MAP will be adjusted upward, but not to exceed the baseline or documented, adjusted value.

So based on that order, it’s pretty clear that when sectors have weather and deviating aircraft, the sector efficiency is reduced and the Monitor Alert number is supposed to be adjusted downwards to reflect that fact.

Note that the 7210.3 isn’t a “suggestion” manual.  It contains orders.  That would make one believe that it’s not optional, but mandatory.  It’s an order after all.

However, the FAA ignores that part of the Monitor Alert Order.

The FAA is quick to tell anyone who points out to them that they’re ignoring this part of the order that the Monitor Alert Order only concerns sector efficiency; not sector safety.

However, any controller will tell you that working too many airplanes at one time reduces the margins of safety; work too many airplanes and efficiency drops, as well as the controllers ability to work those airplanes safely.  There is only so much a controller can watch.

The National Transportation Safety Board (NTSB) agreed when they issued this Safety Recommendation A00_23-27 dated April 7, 2000, concerning a day when several operational errors occurred on a day there was weather impacting sectors in Cleveland Center.  They ignored the Monitor Alert Order then and many sectors got overloaded and resulted in several airplanes getting too close together.

At the time of the incident, the sector’s ability to handle traffic was degraded by weather and holding aircraft; therefore, the MAP should have been reduced to a number (significantly lower than 18) determined on the basis of these conditions, but it was not.

So the FAA has been ignoring that part of the Monitor Alert Order for over ten years!

At that time no Monitor Alert occurred at all, but even though alerts are generated now a vast number of them are simply ignored.

Additionally the report states (my emphasis):

The Safety Board’s investigation revealed that the Cleveland ARTCC has no procedure to ensure the necessary adjustment of sector MAP numbers; this lack of procedure degrades the system’s ability to issue appropriate monitor alert warnings.   During visits to other ARTCCs, Board investigators found that other traffic management units also did not adjust MAPs as required.   Therefore, the Safety Board believes that the FAA should direct ATC facilities using ETMS monitor alert software to establish procedures to ensure compliance with the dynamic MAP  adjustment  requirements contained  in  FAA  Order 7210.3, “Facility  Operation  and Administration.”

Ten years ago and the FAA is still ignoring its own orders, in spite of the fact that the NTSB called them out on it.

Remember when I mentioned earlier that FAA management states time and time again that the Monitor Alert Order concerns sector efficiency and not safety?

“the sector’s ability to handle traffic was degraded…”

The NTSB made a direct correlation between the MAP number being exceeded and getting aircraft too close together.  It’s simple math, but apparently the FAA can’t (or chooses not to) count.

In the report the NTSB recommended in part:

Direct air traffic control facilities using Enhanced Traffic Management System monitor alert software to establish procedures to ensure compliance with the dynamic Monitor Alert Parameter adjustment requirements contained in Federal Aviation Administration Order 7210.3, “Facility Operation and Administration.” (A-00-25)

Furthermore the NTSB noted in the report:

The excessive traffic level in the Bradford sector was discovered only as a consequence of the subsequent operational error investigations and would possibly not have been discovered or received management attention if the error had not occurred.  The FAA appears to have no formal process for the identification and investigation of situations in which a sector is subjected to excessive traffic demand when no otherwise reportable event takes place.

See no evil, hear no evil I guess…

The reality is that during the summer especially, when there is weather affecting sectors, the Monitor Alert Program isn’t implemented according to the FAA’s own orders in the 7210.3.  But that’s OK apparently because we haven’t killed anyone ignoring that particular order.

Yet.

The Tombstone Agency strikes again…

But it’s all fine with FAA management because they know full well that if something goes wrong they can (and will) simply blame it all on the controller(s) in the sector.

Here’s a response to a Unsatisfactory Condition Report (UCR)/safety report filed in my facility on a sector that went red on an overnight shift (a midshift) without any action taken to alleviate the amount of traffic in the sector at the time (my emphasis again):

UCR #521725, filed on May 3, 2006, alleges that FAA Management:

• Failed to provide adequate staffing on the mid-shift
• The Monitor Alert Program (MAP) displayed a yellow and then, a red alert status. The OMIC’s attempt to re-route some flights was not successful because the aircraft re-entered the sector at a later time and distance
• Supervision does not meet the same level as the day shifts.     (Reference FAA Order 7210.3, section 2-6-2a),
• MAP values were not adjusted as stated in FAA Order 7210.3, section 17-7-1, and
• NTSB recommendations, dated 4/7/00, (A-00-23 through A-00-27), the FAA did not comply with dynamic MAP adjustments or implement a reporting vehicle of sectors that became overloaded.

All the aforementioned issues are subject to management’s interpretation and implementation. This would include the recommendations offered by NTSB. Because the concerns are related to staffing and the interpretation of FAA Orders, including the MAP, these items are not subject to the UCR process. Therefore, “we consider this UCR is closed.”

Amazingly, the above response shows that not only does the FAA choose/”cherry-pick” which orders they actually implement, but in addition they feel free to “interpret” orders and simply ignore NTSB recommendations.

Keep in mind that at the time the above UCR was filed, the UCR program was the only formal safety reporting process within the FAA.  And it’s pretty clear that the FAA was more than willing to simply dismiss UCRs (at least when the subject of the UCR highlighted the fact that the FAA was intentionally violating its own orders).

As the response states,  the FAA chose to ignore the substance of the safety concerns in the report by simply claiming that the UCR process didn’t cover this sort of problem.

Lest you believe that malarkey, read the first section of the instructions for the UCR form:

2.     CONDITIONS TO BE REPORTED. A UCR (FAA Form 1800-1) should be prepared for situations involving safety or efficiency of equipment and operations and when any of the following conditions exist. UCR is not a substitute for an existing report.
a. Situations which may cause or contribute to accidents, incidents, or present a hazard to personnel and equipment.

What the FAA was really saying in the response was:  “We can’t and won’t reply substantively to the safety concerns you raised because we were cheating.  But we made the rules so we can break the rules.  Now go away.”

That’s the real safety culture of FAA management.

Not unlike our managers who chose to ignore the orders regarding the handling of a Northwest flight that lost communications with air traffic control (went NORDO) and overflew its destination aircraft last fall, FAA management obviously believes they’re above the rules (most of which they made), and that allegedly exist to make the air traffic system safe.

But by ignoring their own rules they’re rolling the dice with the safety of the air traffic system.

And the manager that wrote the above response that ignored/dismissed that safety complaint (UCR) four years ago?

He’s now in charge of our entire facility.

As you may know the FAA stopped testing the ERAM software on live traffic in late March, due to the many serious problems they were experiencing with it.  I don’t believe it has been used on live traffic since.

Sometime in April some of the centers began testing version 2 of the ERAM software, which also involved some firmware updates to hardware.  We’ve gone to our backup computer system (EBUS/DARC) several times on the overnight/midshift since then to accomplish more ERAM testing in the background (as have many other facilities), but as far as I know no facility has started using ERAM to separate live traffic again yet.

Salt Lake Center is still slated to be the first enroute center to start running ERAM again on live air traffic, but with the increase in summer air traffic and given the problems with ERAM previously the FAA may have finally learned its lesson and be hesitant to roll the dice with ERAM again, at least at this time of year.

Later in May apparently in spite of the fact that ZMP is a keysite for ERAM, it was decided that our facility was not going to be part of the Initial Operational Testing and Evaluation (IOT&E) process, which gets us off the hook for any live runs with ERAM, at least for the time being.

Due to the problems with the way the FAA and the ERAM contractor (Lockheed Martin) were reporting and/or fixing bugs with the ERAM software previously, they’ve been working on changing that process.

They’re acknowledging that in the process of fixing bugs many of them would get worse, and/or they would create new bugs, so the end result was that they wouldn’t make any net progress.  Of course that’s common knowledge to anyone who was paying attention at the time, but it didn’t seem to phase those in the FAA overseeing the project.

They continued to test the faulty software on the flying public until word of how poorly the project was progressing got out, eventually including a report and recommendations from the Department of Transportation Inspector General.

Right now it sounds like they’re trying to optimize the Problem Reporting (PR) system for ERAM and how they determine whether or not ERAM is suited for use on live traffic.

Previously the FAA and the ERAM contractor, Lockheed Martin, weren’t too concerned with either, and although they should have had reasonable error reporting, correction and assessment processes in place when they started the ERAM project they obviously didn’t.  All of them should have existed well before they started testing the system on the flying public regardless.

At some point the FAA is going to start testing ERAM on live air traffic again.  Hopefully when they do so ERAM will be in a state more suited for the task than it was the last time the FAA tried using it for that.

In the meantime ERAM development continues quietly “under the radar” so to speak…

The Department of Transportation Inspector General (DOT IG), Calvin L. Scovel III, testifed before the House Subcommittee on Aviation on April 21st, 2010, about the status of the FAA’s entire NextGen project, including ERAM, stating that (my emphasis):

The $2.1 billion ERAM program will replace the existing hardware and software at facilities that manage high-altitude traffic.  ERAM,  however,  is experiencing software-related problems at FAA’s key initial operating site in Salt Lake City.  These problems include radar processor failures, problems in handing off traffic between controllers, and critical flight information being paired to the wrong aircraft.  FAA is spending about $14 million per month to resolve these problems and deploy ERAM at other sites.

and

While FAA does not believe the system to be fundamentally flawed, it has postponed the in-service and operational readiness decisions for ERAM at Salt Lake City by 6 months, both originally planned for December 2009.   We have not assessed the severity of the problems with ERAM,  but FAA officials are concerned about the ERAM transition at larger, more complex sites like Chicago and New York.  These locations have unique airspace and operational issues that will require adaptation of the system’s software to accommodate their needs.

And finally:

FAA  officials acknowledge that it is unlikely that all  20 systems will be fielded nationwide and controlling traffic on a regular basis by December 2010 as planned. FAA must take steps to ensure that problems with ERAM are resolved and make realistic adjustments to the program’s schedule.

You can see the entire DOT IG’s testimony here (which also discusses significant problems with other FAA projects, including the FAA’s Telecommunications Infrastructure/FTI program.)

Instead of simply re-releasing the warm and fuzzy press releases from the FAA on ERAM as news,now the media is  finally starting to report what’s really going on with the ERAM project.

Notably, the IG admits that they are trusting FAA officials to large degree about the details of the ERAM problems, as they admitted they didn’t actually examine the “severity of the problems with ERAM”.

There are many within the FAA that think the program is going just fine in spite of the problems, and while the “FAA does not believe the system to be fundamentally flawed” that may or may not be true.  Right now there seem to be some fundamental problems with tracking aircraft within the ERAM system.

The IG also suggested that the FAA must “make realistic adjustments to the program’s schedule” and “ensure that problems with ERAM are resolved” neither of which the FAA was previously willing to do by its own accord.

Doesn’t this sound a lot like trusting the fox to guard the hen house?

None of it really bodes well, because if the FAA and Lockheed Martin can’t get ERAM working at Salt Lake Center, and they know it’s going to require adaptation for the busier centers elsewhere in the country, how long will it really be until ERAM is actually ready for nationwide use?

FAA officials apparently admit “concerns” about the ERAM transition, but will the FAA choose to start using it again on live traffic before it’s ready regardless?

I’m betting they will, simple because they willingly and knowingly have already done that.

Early in the week of March 22, the FAA finally gave in to pressure to stop running the faulty versions of the ERAM software on live air traffic.

We were also being told that allegedly they were going to take the time to allow the program contractor (Lockheed Martin) to make software changes that would fix all the major ERAM bugs before running it again on live air traffic.

But early in April, it appeared that the FAA was considering having Salt Lake Center (ZLC) run the latest ERAM software version (U4), even though they knew it didn’t have all the necessary corrections to run reliably 24/7.

And at the same time Minneapolis Center (ZMP) had scheduled more ERAM live runs in mid April.  Eventually those tests were canceled, but it shows there are plenty of people in the FAA ERAM program that still don’t have any problem getting right back to their practice of running versions of ERAM software they know have major bugs.

Both indicate that the attitude for many within the FAA toward testing ERAM with known bugs on live traffic hasn’t changed at all, in spite of appearances that the program was going to proceed differently from this point out.  (Maybe those individuals just didn’t “get the memo”…)

Knowing the FAA I have to wonder if this delay/”review” is really just intended to placate those who were objecting to the way the ERAM program was progressing; not a genuine inclination to change the course of ERAM deployment.

Time will tell if the FAA gets right back to testing ERAM software versions they know have bugs.  But for the time being, they appear to be taking the time to do what they should have been doing all along, which is fixing the major known bugs before trying more tests on live traffic.

However, facilities continue to test ERAM in the background, forcing controllers to run live traffic on their inferior backup computer system (DARC/EDARC/EBUS).

Here’s a recap of notable ERAM Runs/Tests at the two ERAM keysites (note that after each of the failed tests, there were software updates made):

• October 3, 2009:  Salt Lake Center (ZLC)  has failed ERAM test on live traffic forcing fallback to HOST.
• January 30, 2009 to February 8, 2010 – Salt Lake Center (ZLC) extended live traffic ERAM run.  This controlled and sterile test apparently gives the FAA confidence that ERAM is ready for 24/7 use.
• February 17, 2010 to February 23, 2010 – Salt Lake Center (ZLC) starts what was intended as permanent 24/7 run on ERAM that was aborted with fallback to HOST due to major problems (“SD” Version).
• March 6, 2010 – March 8, 2010 – Seattle Center (ZSE) test running version T7(?); numerous major problems.
• March 14, 2010 – March 20, 2010 – Salt Lake Center (ZLC) running version T9, forced to fallback to HOST due to problems.

I would love to believe that the FAA is going to take a different approach to the entire ERAM program from here on out.  But I’m not holding my breath…

Within the last few days, apparently the FAA has decided to stop running the new software on live traffic and make an “assessment” of the program, although certainly not by its own accord.

Since last year, the FAA has been routinely running its En Route Automation Modernization (ERAM software), still under development, on live traffic, with many known critical bugs at three key sites, including including Salt Lake Center (ZLC), Seattle Center (ZSE), and Minneapolis Center (ZMP).

In spite of the fact that the FAA and the program contractor, Lockheed Martin, knew of many significant bugs with the software, the FAA opted to run the software nonetheless, often playing down the seriousness of the problems.

New versions of software addressed a few of the significant bugs while at the same time ignoring most of them.  That fact didn’t prevent the FAA from trying the latest versions on live air traffic anyway.

And predictably, the bugs that hadn’t been fixed in the latest software versions inevitably cropped up again, in some cases leading to a shutdown of the ERAM software with a fallback to the legacy HOST software.

What was the FAA’s primary motive for testing software that they knew had significant bugs?  Apparently in part to meet timelines and deadlines for the software deployment which is falling further behind schedule, as well as ensuring that the program’s contractor, Lockheed Martin, gets cash bonuses built into the contract.

Under pressure from the controller’s union, NATCA, who started telling congressmen and senators about the problems with ERAM, and those controllers helping work on the project, some of whom were becoming increasingly frustrated by the lack of progress in correcting significant known bugs with the software, (and this writer would like to believe some other negative publicity) the FAA eventually agreed to stop running it on live air traffic for the time being and instead make an assessment of the program.

The FAA knew that the more opposition grew to the program the harder it was going to be to “fly under the radar” and continue with its reckless approach.  Many of the critical problems they have had with ERAM weren’t corrected in spite of numerous software updates, negating the FAA’s claim that the program was progressing satisfactorily.

A large part of the problem was that the ERAM software clearly wasn’t developed to the point of being ready for use to separate live air traffic when the FAA started using it for exactly that.  Even if all the critical known bugs were magically fixed today, ERAM still probably wouldn’t be ready for use with real air traffic, but it certainly would be a lot more suitable for that task than it currently is.

I have no doubt that the FAA would have continued with its “damn the torpedoes” approach were it not for increasing opposition from the controller workforce and its union.

Unfortunately for the FAA, the more controllers that were exposed to running the ERAM program, the more visibility the problems got, none of which were going away.

That sorry state of the software was only going to lead to bad publicity that the FAA didn’t want.

It was months too late for the FAA to stop the current course of the entire ERAM program deployment, and I have no doubt that the FAA would never have stopped to make an assessment of the ERAM program on its own.  After all, they seemed to think everything was going fine.

The state of development for the entire ERAM program is obviously well behind where it should be, and although lots of others recognized that fact, as an organization the FAA was either in denial or oblivious. Considering what happened with the Initial Sector Suite System (ISSS) program decades ago, it’s obvious the FAA has learned nothing over the years (or has simply forgotten).

The only acceptable and responsible course of action now is for the FAA to force Lockheed Martin to correct all the known critical bugs before the ERAM software is accepted for further use in separating live air traffic.

Anything short of that will inevitably put us right back where we started, and likely back into the same dangerous and faulty cycle.

However, knowing the FAA I believe that this assessment will likely result in a decision which will amount to merely “trying to do better”, rather than a real commitment to fix the known problems before further using the flying public as beta testers/guinea pigs for the ERAM software.

That’s because the FAA has already demonstrated that it’s willing to run faulty software because they’re more concerned about the ERAM deployment schedule than anything else.

They’re panicking because the clock is ticking.  The HOST software contract has already been extended.  The ERAM program schedule continues to slip.  Failing to hold Lockheed Martin to a higher standard isn’t going to make the ERAM problems go away, but is conducive to staying (more) on schedule, in spite of the fact that it puts the flying public at risk.

The FAA has always assumed (and will continue to assume) that air traffic controllers will be able to work around ERAM’s deficiencies and keep airplanes separated nonetheless.

Keep in mind that none of this precludes the FAA from continuing to run/test the ERAM software; it merely won’t be used on live traffic for the time being at the key sites.

Testing of ERAM in the background will continue, and will likely reduce the margins of safety and increase controller workload by forcing them to work live traffic on its outdated DARC/EBUS backup computers without flight plan processing so that ERAM can use the interfaces between the various facility computer systems.

But unlike ERAM, at least the DARC/EBUS software properly tracks radar targets…

ERAM, the replacement computer/display system for enroute center controllers, is still experiencing problems and the FAA continues to use it on live air traffic.  In other words, not much has changed.

The most critical ERAM bugs that I’m aware of (outside of outright display failures/lockups – big red “X”s) involve data block/tracking issues, some of which haven’t been corrected even though they’ve been known of for some time.  The absolute worst tracking bug is when a data block drops off a target and the accompanying flight plan is simultaneously deleted.  Other critical tracking bugs involve data blocks tracking on the wrong target.

One thing that has changed since I last wrote about ERAM is that I’ve since had the “opportunity” to work with it on live traffic.  Needless to say the experience didn’t change my opinion about the program.

We were told in advance to not “experiment” with ERAM while using it (presumably because of the myriad problems we might encounter).  Apparently it’s true that we’re really not testing the software any more right now; we’re just using it, while trying to not use it so much that we break it.  (Maybe at this juncture the FAA figures there are enough known bugs already and they don’t want to find more.)

Many of those involved with the ERAM project continue to believe and/or give outward appearances that things are going fine in spite of the persistent problems.  In fact, controllers are hearing less and less of the bugs in the software as time goes by, which has the added effect of making it appear that things are going better than they really are.

I guess it’s the old, “No news is good news” theory…

During the Initial Operational Capability (IOC) or first run on live traffic at Minneapolis Center (ZMP) they provided controllers with a list of known major bugs in ERAM, but they’re now no longer providing that information before live runs.  (Perhaps the list is too long…)  Either way it’s obvious that the FAA doesn’t think controllers need to know about the deficiencies of the system they’re supposed to use to keep airplanes separated (or doesn’t want to advertise them anyway).

ZMP had two operational runs with ERAM the first week of March and not surprisingly we experienced some repeats of some of the most significant ERAM bugs.

Mind you, most of the bugs we experienced weren’t new bugs; they were existing bugs they already knew about.

Instead of fixing the known bugs before running ERAM on more live traffic, the FAA continues to run software they know is faulty more often.

“Idealists” like me would like to believe that the FAA and Lockheed Martin would want to fix those known bugs once they were discovered before running the same versions on more live traffic.

But that’s not how the ERAM program is progressing at all.  That’s because the FAA and Lockheed Martin have agendas that don’t prioritize the safety of the flying public.

First, fixing all the bugs before running it more would take more time and cause the program to fall further behind schedule.

Another reason the FAA is now running software builds with known bugs on live traffic at the various key sites is to avoid refresher training for controllers as per the Memorandum of Understanding (MOU) between the FAA and the air traffic controllers’ union, NATCA.

The MOU says that:

“Basic and supplemental ERAM training shall be provided to all BUEs prior to implementation of ERAM at each facility.  If more than forty-five (45) days elapses between the time BUEs complete ERAM training and the actual implementation of ERAM at the facility, ERAM refresher training shall be provided prior to implementation.”

Apparently IOC equates to “implementation” in the MOU (although I don’t see that definition in the MOU).

So there you have it:  the FAA is now rushing ERAM into use at least in part so that it doesn’t have to re-train its controller workforce per the MOU.

Now that the Winter Olympics are over, Seattle Center (ZSE) is now also back in the ERAM game as a key site again.  ZSE just ran an extended live run less than a week ago that predictably also had its share of problems, simply because they were running the same software build everyone knew had existing bugs.

In spite of all the problems, the FAA continues to press towards an In Service Decision (ISD) after which the non-key sites (the rest of the enroute centers) can move towards their own IOCs, in spite of the fact that the longest period any of the key sites have run ERAM is for 8 days (and that under very controlled conditions).

So we’ve established that the FAA isn’t fixing significant/critical known bugs before running the ERAM software more often.  They’re telling controllers to tip-toe around ERAM while using it to create the illusion that things are going well, in spite of the many known bugs.  They’re withholding more and more information as time goes by and the project continues to go badly.

It’s obvious that the FAA is sticking with their approach that prioritizes the deployment schedule (which for the record continues to slip further and further behind), saving money on training, and pretty much anything else, over safety.

It’s just business as usual for the FAA…

The latest incident involved a veteran controller at New York’s JFK airport, who had his children relay some air traffic clearances on the radio frequency.

The JFK incident was the third in a string of recent air traffic control related incidents that made the headlines, including last summer’s mid-air collision near Teterboro airport in New Jersey of two VFR aircraft, as well as the incident last fall where Northwest 188 lost contact with air traffic control and eventually overflew its destination.

Always ready to put on the proper face to the media, the top levels of FAA management reacted to the latest incident with shock and outrage:

“This lapse in judgment not only violated FAA’s own policies but common sense standards for professional conduct. These kinds of distractions are totally unacceptable,” administrator Randy Babbitt said in the statement.

“…(violations) of FAA’s own policies…“?  “…standards for professional conduct…“?  Really, Mr. Babbitt?!

Let’s examine some facts about the three incidents:

In the case of the mid air collision the supervisor was on the clock but out of the facility running personal errands, which were apparently more important than his job.  In the case of Northwest 188, several managers decided to simply ignore orders.  And in the latest case, one or more supervisors apparently allowed an employee on two different days to let his children talk to airplanes on the radios.

In each and every incident, there was an FAA manager involved that wasn’t following the rules.  Do you think that’s just a coincidence?

The managers are the people supposed to be ensuring that the workers are following the rules, and what are they doing?  They’re breaking the rules themselves!

Does anyone remember this video?

The conduct of some of those managers is a violation of FAA Human Resources Policy which states in part (my emphasis):

An employee’s conduct on the job has a direct bearing on the proper and effective accomplishment of official duties and responsibilities. Employees are expected to approach their duties in a professional and business like manner and maintain such an attitude throughout the workday. It is also expected that employees will maintain a professional decorum at all times while in a temporary duty travel status or otherwise away from their regularly assigned post of duty, such as telecommuting, whether at home or at a telecommuting site, or attending training.

So much for following the rules and the higher standard FAA managers are allegedly held to.

Do you think a controller or two might have noticed managers in those cases intentionally violating FAA policies and acting unprofessionally?  Do you think they didn’t notice nothing happened to any of them for doing so?

Isn’t this called, “setting an example”?

Last year I wrote about the FAA’s “dumb luck” approach to safety, including the “customer service initiative” that ultimately led to two FAA safety inspectors turning into whistle-blowers when FAA managers ignored their concerns about problems with Southwest Airlines’ maintenance.

It was clear then that the FAA was only concerned about safety when the problems hit the headlines.

Then the FAA decided to reclassify air traffic control errors, turning many errors into non-events (and making it appear to the flying public we were having fewer errors).

They created a safety program (ATSAP) that allows controllers to anonymously report errors without fear of punishment, but which in turn also masks and allows the FAA to ignore many systemic problems.

Currently the FAA is testing its ERAM software, even with its many known bugs, on live air traffic.

And almost every time something makes it into the news that involves the FAA, an FAA spokesman quickly says, “Safety was never compromised.”

The FAA claims it’s an organization that’s passionate about safety, but there’s little to indicate it’s actually doing much to improve safety at all.  If anything it’s degrading safety more often than not.  It says one thing but does another.

So between managers not following FAA rules, and the many changes to FAA policies and procedures regarding air traffic safety and error reporting, should it really be a surprise that controllers may have gotten complacent?

And if they are complacent, aren’t they really just following direction and examples from the FAA management team?

I’m sure the FAA and the contractor Lockheed Martin will write it off as just another “glitch” (i.e. part of the development cycle), but it’s another glaring demonstration of how unreliable the ERAM software still is, even though the FAA continues to test it on live traffic, expecting air traffic controllers to simply work around its many problems and keep aircraft safely separated nonetheless.

ZLC started running ERAM on what was supposed to be a permanent basis on the morning of Wednesday, February 17.

They had previously completed an an eight day test that ended the first week of February, followed by a two week delay in which Lockheed Martin was supposed to correct the (known) bugs in the software before ZLC began using the new version permanently.

The latest failure shows that in spite of the software updates that obviously ERAM still has a long way to go before it’s fit to use on live traffic 24/7.

Notably the event marking the first enroute center to transition to ERAM full time came and went quietly.  Instead of calling in the media and having a press release (and having sheet cake), the FAA barely noted the occasion.

The complete lack of fanfare noting the first enroute center to start running ERAM full time shows that the FAA knows full well how unreliable/unstable the ERAM software still is.  At this point it’s clear they’re making deliberate efforts to not call any attention to the ERAM project.

After lots of boastful press from the FAA over ERAM early last year, including statements of how the program was on budget and ahead of schedule (even though it wasn’t), the FAA abruptly stopped talking about ERAM after significant problems running it at ZLC in a test last fall.

The FAA apparently learned its lesson then and now isn’t going to mention ERAM at all, instead choosing to continue testing and deploying ERAM quietly and keeping its fingers crossed that it won’t cause a news event.

Every time the FAA and Lockheed Martin complete another test without significant problems they seem to convince themselves the project is doing just fine.  After the eight day ZLC test they were convinced the software was ready for permanent use after just a little “tweaking”, even though it’s now clear that was far from the truth.

Last fall one of the problems that resulted in the aborted ZLC test was datablocks (the tag that displays the aircraft call sign and altitude as well as other information) wouldn’t track properly and sometimes ended up tagging up on the wrong target.

Guess what?  That problem still exists many months (and many updates) later.

The data block/tracking functionality is fundamental to an air traffic display system and is thereby safety-critical.  It’s disturbing that at this stage this basic functionality is still so unreliable in ERAM.

This may not be simply due to software bugs either; there may be some significant problems with the software tracking algorithms within ERAM, which from what I’ve heard are radically different from those used in the HOST computer system.

Here’s a list of some of the latest bigger problems with ERAM (and note that some of them, especially the tracking problems, aren’t new):

Interim Flight Plans – If a controller starts an interim flight plan (datablock only, no beacon code or routing) ERAM aggressively searches for the first target of opportunity to track. It may be a primary, or a beacon belonging to another aircraft.
Track Un-Pairing – Arbitrarily the datablock will disassociate from the beacon target. We are unable to determine what seems to cause it. We looked at RADAR sort boxes and ASR terminal RADAR feeds, and who knows what else. ERAM will not automatically re-pair the datablock and the target like HOST does. We see this happen frequently around SLC where limited datablocks create a bright large yellow spot over the airport. You can’t shut them off and it is easy for the un-paired datablock to disappear into the blob.
Track Swap – We had some instances of departures where ERAM switched datablocks on aircraft on completely different routes and entering different sectors.
Bogus Beacon Codes – Frequently ERAM will flash in the third line a bogus beacon code (like the aircraft is squawking an incorrect code) for one sweep and then it disappears.
Track Pairing – If ERAM associates a full datablock with an incorrect beacon, you have to track the datablock at least 32 miles away from the incorrect beacon for ERAM to accept the disassociation. Approximately 30 seconds has to pass before you can pair it with the correct beacon target.
Bogus Alerts – We see significant numbers of bogus alerts; MSAW, conflict probe in EDST (URET replacement), aircraft working is SUAs.
Inter Facility Handoffs to Vertically Stratified Sectors – If an aircraft changes altitude 30 minutes prior to exiting the facility, and the new altitude causes the aircraft to enter a different sector in the receiving facility, ERAM will hand the aircraft to the incorrect sector if you use the auto addressed handoff option (single alpha character followed by CID). You have to manually address the handoff to the correct sector.

Apparently the latest software version yet to be put into use isn’t intended to fix many of the aforementioned problems either; instead it addresses other bugs.

It will be interesting to see how the latest episode affects the entire ERAM project.

One way or the other it’s going to result in the project falling further behind schedule.

But I doubt very much that it will convince the FAA to stop testing the software on live traffic.