It’s been a while since my last entry and although I’ve had plenty to write about, I’ve been enjoying the summer here in Minnesota that will inevitably end all too soon.
But I did want to write an update on the status of the FAA’s En Route Automation Modernization (ERAM) since I’ve written about it so much previously.
ERAM is the enroute air traffic control system computer replacement that’s supposed to be a cornerstone of the FAA’s much touted NextGen air traffic control system, although due to numerous bugs and problems with the ERAM program its testing on live traffic was suspended this spring.
Since my facility (Minneapolis ARTCC or ZMP) is no longer a key site for ERAM we’ve been spared runs on live air traffic for the time being. But that also takes us out of the loop a bit when it comes to the status of ERAM development, even though we have been running live traffic on our backup computer system (EBUS/DARC) in the middle of the night (on the mid-shifts) in order to perform testing on ERAM.
The current two key sites for ERAM are Salt Lake Center (ZLC) and Seattle Center (ZSE) and they’re the only facilities authorized to run ERAM on live traffic. Once ZLC and ZSE testing determine that ERAM is fit for use on live traffic again they will have an In Service Decision (ISD) after which non key sites can start running ERAM on live traffic as well.
ZMP was brought on board as a key site in part to be able to test the ERAM functionality with Non-Hosted Terminal (approach control) facilities, a configuration that apparently both ZLC and ZSE lacked.
Non-hosted Terminals are non-ERAM facilities adjacent to an ERAM facility that need to be able to receive and send flight plans back and forth to ERAM. Since the two key sites didn’t have the adjacent facilities to be able to test this functionality, ZMP offered to do the job.
However, with all the problems they were having with ERAM it was eventually decided that the testing of the Non-hosted Terminal functionality could wait, and ZMP was no longer a key site.
This in not only indicative of how haphazard the ERAM development cycle has been, but of how far ERAM has still to go before it’s ready for deployment across the country.
Right now the FAA and the ERAM contractor, Lockheed Martin, are still working on ERAM to get it to a state where it can reliably perform basic functionality and not worrying about the extra functionality ERAM will need to work across the country.
That’s what they should have been doing initially, instead of rushing to use ERAM on live traffic simply to meet an unrealistic deployment schedule well before ERAM was able to perform basic functionality reliably.
The FAA is now tentatively planning on starting to deploy ERAM at other enroute facilities early next year although the “waterfall” (deployment schedule) is far from set in stone and thereby isn’t being published anywhere.
So much for the project being on time and on budget…
ZLC started testing ERAM on live traffic in August again and has completed some multi-day operational runs. Since they still had some critical problems, they’re waiting for a new version of the software before they’ll make an attempt at other extended live runs and eventually a continuous 24/7 operational run again. (The last time they tried this was back in March of this year.)
The ERAM software sounds like it’s improving, but they’re still apparently experiencing critical bugs with basic functionality such as missing aircraft computer identification numbers in data blocks and some tracking issues, as well as continued memory leaks that make the system sluggish and/or unstable after running for only a few hours.
The fact that they’ve been working continuously on ERAM bugs since this spring, when they stopped testing it on live traffic, and still have enough critical bugs to preclude a continuous operational run show that there was no way ERAM was fit and/or ready for use on live traffic last spring even though they were using it for that purpose.
It’s just too bad the FAA couldn’t come to that conclusion on their own, instead finally bowing to pressure from various sources to stop testing the faulty software on live traffic.
In the meantime they also finally developed a method to more easily transition back and forth from the current HOST computer system to ERAM they’re calling “shadow mode”. Shadow mode has the ERAM system shadowing the HOST computer so that it can get flight plan data prior to switching over, making the entire transition a lot less work intensive for the air traffic controllers and a lot less messy.
This functionality should have been part of ERAM initially, but apparently the FAA and Lockheed Martin were delusional enough to believe that they were just going to turn ERAM on and everything would work great.
When that didn’t happen and they realized they would need to perform the transition back and forth from ERAM to HOST repeatedly they finally decided to develop a method to more easily do so.
That missing functionality is another indication of how screwed up both the FAA and Lockheed Martin’s approach to the ERAM program has been from the very beginning.
There’s functionality that should have been included from the very beginning that they’re just getting around to adding now. There’s other functionality they didn’t seem concerned with and/or didn’t even know they needed (like the Non-hosted Terminal issue) because it wasn’t part of the original testing regimen.
Thus there’s sure to be more problems encountered as they find new issues they hadn’t thought about when they start deploying ERAM across the country, not the least of which is running ERAM with lots of air traffic.
The FAA also had intentions of decommissioning both its HOST computer system, and its backup computer system (EBUS/DARC) after ERAM was up and running.
However, since ERAM is the backup for ERAM (running the same software on multiple channels) if there is a critical bug in the software that causes one ERAM channel to crash, it would be fair to assume that the bug would occur on the other channels as well causing all the ERAM channels to crash in turn.
That means that until ERAM attains a much higher level of stability and reliability the FAA should maintain a legacy backup system (such as EBUS/DARC), but that’s not to say they will.
Good write up, wonder if there have been any updates from ZLC as to how ERAM testing is going?
Interesting note on shadow mode, it was actually started by ZLC themselves and two FAA persons from the Tech Center made it work. Lockheed didn’t care if hundreds of FP’s had to be reentered as long as it didn’t effect their bottom line.
As usual, George, thanks for the additional information!
It doesn’t come as any surprise that Lockheed Martin didn’t come up with the Shadow Mode function themselves; thus far they’ve had a hard time so far providing the real functionality in ERAM that controllers need (including an easy transition mode back and forth from HOST to ERAM).
We haven’t heard much about ZLC’s latest tests, although I did hear that during a test a few weeks ago when an adjacent center was running ERAM that the ERAM computer flooded our HOST with thousands of duplicate flight plans again (the very problem the HOST was patched for a month or so ago).
Therefore, there’s at least one other bug they’ve now experienced in ERAM that will cause it to send and resend a flight plan over and over again to an adjacent HOST computer.
Wonder how many times they’re going to try and patch the HOST system to protect it from an adjacent ERAM system…
How many times will they patch Host is an excellent question? I haven’t seen anything in the pipe coming along anytime soon. The clamps have been pretty much been applied to Host and unless there is a critical bug no fixes will be forth coming. The Host technical personnel had been reduced to under 8 for national support and of those most are assign to ERAM and are only pulled over to Host on a as needed basis.
The field continues to open problem records (INFO’s) against the Host system, why I don’t know, they don’t have anyone to look at them let alone fix them.
It’s just difficult to understand how the FAA allowed itself to get into the position of where ERAM may, have the same functionality and reliability as Host but in ten to twenty year from now. GPS you say, really? take at look at ZHU who is using ADS-B today, like right now. They adapted some virtual radars for the ADS-B and like magic radar tracks over the gulf. Imagine that!
Peace, George