Damn the Torpedoes/ERAMming Speed and DARC

Here’s another indication that the FAA knows that En Route Automation Modernization (ERAM) isn’t ready for use on live air traffic.

Air traffic controllers at my facility are getting extra training on the Direct Access Radar Channel (DARC) system, which is the current backup system for the primary HOST computer that processes and displays the radar information for enroute controllers.

(Note:  Somewhere along the line DARC became EDARC for “Enhanced” DARC, then Enhanced Back-Up Surveillance, or EBUS,  but my radar display still says “DARC” when its running and has essentially the same functionality in all versions, so that’s what I’m going to call it.)

FAA orders require that all controllers get yearly refresher training on the DARC system, just to be sure they know how to use it.

That’s because it’s the only backup to the HOST computer system.  If the HOST computer failed, controllers would revert back to DARC.  So it’s something controllers are supposed to know how to use, period.  That’s why we get yearly training on how to use it.

DARC also happens to be the backup for ERAM.

I’ve got a couple of problems with the FAA now giving extra training on DARC.

First, it’s obvious that they feel the yearly refresher is inadequate to ensure that controllers know how to use the backup system that they might have to use any day at any time should the primary computer system fail.  That’s that “safety based organization” (the FAA) I’ve been talking about showing its colors again.

Additionally, it’s pretty clear the FAA thinks ERAM is going to fail and wants to make extra sure controllers know how to use DARC.

ERAM actually has two separate channels/processors, that both have to fail to force controllers to revert back to DARC.  But I’d say the FAA seems to think this is a strong possibility; otherwise why would they worry about extra DARC training?

We’ve also been told that they discovered that running DARC is a way for them to “flush (reset)” ERAM should it become corrupted.

The thing that no one is talking about is that DARC lacks a lot of the functionality of the primary HOST computer system.  That results in an increased workload for controllers while working with DARC, and a lot more potential for a human error mistake and/or distractions.

There are actually two modes DARC can run in:  DARC Only mode, and DARC/HOST mode.  When they take the HOST computer down we run in DARC Only mode.

DARC Only mode lacks the capability to store and process flight plan and route information, which means controllers have to call the next controller to pass on flight information on each aircraft, and requires manual coordination for many of the functions controllers do with a simple computer entry while working on HOST.  Using DARC is much more cumbersome, and as it’s done infrequently and controllers are less proficient using it, ultimately a lot more work for controllers.

DARC essentially offers a primitive subset of the features controllers normally use with the primary computer system, HOST.  Thus it takes a lot longer for controllers to perform a lot of their job functions, and leaves them less time to watch for traffic and separate aircraft.

Anyone who thinks working on DARC is the same as working on HOST has never worked air traffic on DARC (or has forgotten what it was like to work on DARC).

In fact, FAA Order 7210.3 paragraph 8-1-1 specifically prohibits using DARC while the HOST computer is available, even for training, because DARC is an inferior system.

The air traffic manager shall not cause or permit the operational use of the Direct Access Radar Channel (DARC) solely for purposes of training when the primary operational system is available.

As newer controllers have noted time and time again after working traffic with DARC for the first time, put succinctly, “DARC sucks!”

The worst case scenario transition to DARC is an unscheduled/forced transition which inevitably results in chaos.

In such a case often all flight plan information on aircraft is lost as well.  Flight plan information is all information on a flight, most of which isn’t displayed on the controller’s radar display.  There is a lot of necessary information in the flight plan:  most importantly the route the aircraft will fly.

Ever since the the FAA started using electronic flight plan strips in the User Request Evaluation Tool (URET) to replace paper flight plan strips, this has been a known problem, even though the backup for HOST and URET officially is still paper strips.  Only to get paper strips you have to know the system is going to crash and get them in advance of the flight plan processing functions crapping out.

Here’s a DARC analogy for you:

Imagine you’re working on writing a word processing document on your computer with a deadline about ten minutes away, at which time your document needs to be completed and delivered to your editor.  You figure you have about 8 minutes work left at that time.

Then imagine your computer crashes and you can’t restart it.  No worries; you have a contingency plan.

You’ve got a different/backup computer you run simultaneously, that has a backup of the original document on it (hopefully).  It doesn’t have all the capabilities or ease of use of your first computer but you can edit a document in it.

So you switch to your backup computer, that should have the backup of your original document on it then make sure it resembles the document you were originally working with.  Once that’s done you can get back to writing your document again.

Only you had other reference material documents you were using to write your new document on your first computer, and those aren’t available on your backup computer.  You’ve ended up with some of the information you had been using to complete your task, but not all of it.

Never mind that, your deadline is approaching so you have to just do with what you’ve got.  Get it together and improvise.  You’re a pro; you can make this work!

If all goes well you will probably finish your document in time to get sent to your editor.  It likely won’t be your finest work of course, but hopefully you’re able to accomplish your job.

Only we’re not talking about a word processing document in the air traffic business…

Currently the HOST computer system is stable enough that a forced fallback (i.e. computer crash) to DARC is a rarity.  That’s not going to be the case with ERAM.

Before ERAM testing reared its head, in the past the enroute centers usually only ran DARC for testing of the HOST and updates, and then in the wee hours of the morning when the air traffic is reduced.  But these are planned, scheduled and orderly transitions to DARC however, so all the necessary preliminaries can be accomplished prior to transitioning to DARC.

With all the ERAM testing of late, the centers involved are running DARC much more often on the midshift (midnight to 6AM shift) solely to accomplish the ERAM testing.  It’s no longer being used as just a backup for the working primary computer system.

Because of all the problems and additional testing ERAM is now requiring, the FAA is forced to utilize its inferior backup computer system more regularly on real/live air traffic.

Twenty years ago when the HOST computer system was newer and a lot less stable it was taken down for maintenance more often on the midshift/overnight shift, a shift usually staffed only with a skeleton crew of controllers.

However, then we used to staff the A-side position (usually with air traffic controllers in training, “developmental controllers”) whose primary job was to coordinate the flight plan information while controllers worked the traffic on the radar scopes.

Nowadays the A-side position is usually no longer staffed (and never staffed on a midshift), so controllers working DARC pass flight plans as well as work the traffic.  If there’s any amount of traffic, controllers working DARC can get quite busy doing all the extra things DARC requires them to do.

And considering the limited staffing available on the midshifts there are usually very limited options to deal with the additional workload.

As I mentioned in the last entry, one of the problems they’ve had with ERAM is every software revision that fixes some bugs often creates just as many new ones.  That means every time they fix bugs in ERAM, centers run the backup computer system with a limited function set (DARC) so that they can retest the same ERAM functions/features again just to ensure the latest patches didn’t break something they previously had working.

The margins of safety in the air traffic system are already being compromised because of ERAM, and we’re not even actually using it yet.

4 comments

  1. ZLC has been running on ERAM full time since Saturday January 30th starting at 4 am local. There have no game changing issues. I believe the original plan was to run for 8-days and cut back to Host for the Olympics. Has now changed to unless some unknown issue arises ERAM will remain up at ZLC. ZLC has weathered the Monday, Tuesday rush.

  2. Thanks for the update, George!

    However, since you didn’t give details and to make sure people don’t think the current ZLC ERAM test is all sunshine and rainbows, we’ve heard from our ERAM guys that ZLC has already had over 300 problems (some of which might be duplicates) with ERAM, including over 40 instances of radar screens freezing (big red “X”).

    We’re also hearing that they’re rebooting the ERAM systems every day to avoid the added instability they experience running them for more than a day. I wonder if that was part of the design plan for a system intended to run 24/7…

    It’s pretty apparent that at this point likely nothing is going to prematurely end the ERAM test at ZLC, mostly because the project is already too far behind schedule.

    Anyway, I’ve said all along the FAA was going to use ERAM, regardless of whether it worked or not.

    I’m sure the controllers at ZLC are busting their butts to work with it in spite of all the problems, and the FAA will consider this latest test a success, as long as the entire system doesn’t crash and/or no one gets killed.

    After all, that’s the FAA safely culture. Smoke and mirrors… “Stay out of the media!”

    But that’s still not to say that ERAM should continue to be tested on real air traffic in its current state.

    It looks like on ZLC’s heels, my facility (ZMP) is scheduled to go IOC around midnight on Friday night, with the same version of the ERAM software ZLC is currently running, so we’ll likely get to see some of those bugs too.

  3. Thanks for that update. I sure the only thing standing between ERAM, FAA and Lockheeds complete failure are the ZLC controllers they desire kudos and hazard pay!

    Get tired of hearing the Washington types say “ERAM is 20 24” translation our system for the next 20 years 24 hours a day. Lockheed had them totally convinced that it never! need to be rebooted.

Leave a Reply

Your email address will not be published. Required fields are marked *