Salt Lake Center (ZLC) reverted back to the HOST computer system last night due to major problems after starting an ERAM run last week that was supposed to be permanent.
I’m sure the FAA and the contractor Lockheed Martin will write it off as just another “glitch” (i.e. part of the development cycle), but it’s another glaring demonstration of how unreliable the ERAM software still is, even though the FAA continues to test it on live traffic, expecting air traffic controllers to simply work around its many problems and keep aircraft safely separated nonetheless.
ZLC started running ERAM on what was supposed to be a permanent basis on the morning of Wednesday, February 17.
They had previously completed an an eight day test that ended the first week of February, followed by a two week delay in which Lockheed Martin was supposed to correct the (known) bugs in the software before ZLC began using the new version permanently.
The latest failure shows that in spite of the software updates that obviously ERAM still has a long way to go before it’s fit to use on live traffic 24/7.
Notably the event marking the first enroute center to transition to ERAM full time came and went quietly. Instead of calling in the media and having a press release (and having sheet cake), the FAA barely noted the occasion.
The complete lack of fanfare noting the first enroute center to start running ERAM full time shows that the FAA knows full well how unreliable/unstable the ERAM software still is. At this point it’s clear they’re making deliberate efforts to not call any attention to the ERAM project.
After lots of boastful press from the FAA over ERAM early last year, including statements of how the program was on budget and ahead of schedule (even though it wasn’t), the FAA abruptly stopped talking about ERAM after significant problems running it at ZLC in a test last fall.
The FAA apparently learned its lesson then and now isn’t going to mention ERAM at all, instead choosing to continue testing and deploying ERAM quietly and keeping its fingers crossed that it won’t cause a news event.
Every time the FAA and Lockheed Martin complete another test without significant problems they seem to convince themselves the project is doing just fine. After the eight day ZLC test they were convinced the software was ready for permanent use after just a little “tweaking”, even though it’s now clear that was far from the truth.
Last fall one of the problems that resulted in the aborted ZLC test was datablocks (the tag that displays the aircraft call sign and altitude as well as other information) wouldn’t track properly and sometimes ended up tagging up on the wrong target.
Guess what? That problem still exists many months (and many updates) later.
The data block/tracking functionality is fundamental to an air traffic display system and is thereby safety-critical. It’s disturbing that at this stage this basic functionality is still so unreliable in ERAM.
This may not be simply due to software bugs either; there may be some significant problems with the software tracking algorithms within ERAM, which from what I’ve heard are radically different from those used in the HOST computer system.
Here’s a list of some of the latest bigger problems with ERAM (and note that some of them, especially the tracking problems, aren’t new):
Interim Flight Plans – If a controller starts an interim flight plan (datablock only, no beacon code or routing) ERAM aggressively searches for the first target of opportunity to track. It may be a primary, or a beacon belonging to another aircraft.
Track Un-Pairing – Arbitrarily the datablock will disassociate from the beacon target. We are unable to determine what seems to cause it. We looked at RADAR sort boxes and ASR terminal RADAR feeds, and who knows what else. ERAM will not automatically re-pair the datablock and the target like HOST does. We see this happen frequently around SLC where limited datablocks create a bright large yellow spot over the airport. You can’t shut them off and it is easy for the un-paired datablock to disappear into the blob.
Track Swap – We had some instances of departures where ERAM switched datablocks on aircraft on completely different routes and entering different sectors.
Bogus Beacon Codes – Frequently ERAM will flash in the third line a bogus beacon code (like the aircraft is squawking an incorrect code) for one sweep and then it disappears.
Track Pairing – If ERAM associates a full datablock with an incorrect beacon, you have to track the datablock at least 32 miles away from the incorrect beacon for ERAM to accept the disassociation. Approximately 30 seconds has to pass before you can pair it with the correct beacon target.
Bogus Alerts – We see significant numbers of bogus alerts; MSAW, conflict probe in EDST (URET replacement), aircraft working is SUAs.
Inter Facility Handoffs to Vertically Stratified Sectors – If an aircraft changes altitude 30 minutes prior to exiting the facility, and the new altitude causes the aircraft to enter a different sector in the receiving facility, ERAM will hand the aircraft to the incorrect sector if you use the auto addressed handoff option (single alpha character followed by CID). You have to manually address the handoff to the correct sector.
Apparently the latest software version yet to be put into use isn’t intended to fix many of the aforementioned problems either; instead it addresses other bugs.
It will be interesting to see how the latest episode affects the entire ERAM project.
One way or the other it’s going to result in the project falling further behind schedule.
But I doubt very much that it will convince the FAA to stop testing the software on live traffic.