The FAA ground stop of Friday 13 November 2026 is being sold as an “old equipment” story. For anyone who runs production infrastructure, it reads like a change-management story, and most of the lessons were already written down in 2010.

Read this first. As of 16 November the FAA still says the root cause is undetermined. I tag each claim: [Official] said on record by FAA/DOT officials or the companies involved, [Reported] from engineers’ write-ups or press sources, [Inferred] my own reasoning. Sources are at the bottom.

What happened

L3Harris runs the FAA Telecommunications Infrastructure (FTI), the private network that carries controller–pilot voice and radar data between FAA facilities. It has been moving FAA lines from copper to fiber site by site. On Friday afternoon a technician installed a new router at an FAA radio site in El Dorado, Arkansas. Shortly after, the device began pumping out so much traffic that communications were disrupted at 17 of the 22 en-route centers that handle high-altitude traffic. A second suspect router sat at a radar site in Amarillo, Texas. [Reported]

The Transportation Secretary described it as a router flooding other routers with more than 1,000 connection attempts every four to five seconds. [Official] The FAA held all departures at 3:36 PM Eastern, with about 3,500 aircraft in the air, and lifted the stop at 8:15 PM. Once both sites were cut off from the network, the problems stopped. [Reported] Airlines kept cancelling thousands of flights through the weekend while aircraft and crews were out of position.

Investigators are focusing on a Juniper router feature that sends data more widely than necessary; the FAA has switched it off on all its Juniper routers. Juniper’s parent HPE says its equipment “was working as configured”. [Reported] The Secretary: “We haven’t determined what the fault was.” [Official]

Old routers explain fragility, not the trigger

The official narrative leans on age: obsolete equipment and a request for tens of billions in modernization money. Early reports even blamed routers built in 2001–2006. Age is real, but as engineer Dave Saunders put it, old routers don’t all fail at once. A nationwide event at 3:36 PM on a specific Friday needs a trigger and a propagation path. The trigger was a newly installed router during a change, and the vendor’s own statement points at configuration rather than a hardware fault. The propagation path was a network with nothing to stop one box’s traffic from reaching everyone else.

How one router takes down 17 centers

A dead fiber takes down one site. A box that floods is worse. Whatever the exact traffic turns out to be, every neighbor has to process it, and control-plane CPUs on older routers are far slower than their forwarding hardware. Starved routers miss keepalives, neighbors declare them dead, the network reconverges, and reconvergence adds more load. [Inferred] That is the non-linear failure RFC 3439 calls the amplification and coupling principles. The textbook defenses are old too: scope flooding and broadcast domains tightly, and filter and rate-limit what reaches each router’s CPU (RFC 6192).

We have seen this movie before

FTI, 19 November 2009: same network, same contractor, same shape

The DOT Inspector General’s report on that outage is worth reading in full. A Harris engineer (Harris is L3Harris’s predecessor) replaced a temporary transition router in Los Angeles and copied the configuration from the old unit without verifying it, missing a single no statement. Instead of the 21 facilities the change was supposed to touch, 129 were affected and about 75% of routes on the fiber network were blocked. Alarms never fired: a filter meant to silence one noisy router was suppressing alarms for every FTI router. Engineers then spent hours chasing the wrong problem.

An independent panel recommended, and the FAA accepted in April 2010: automated configuration tooling with independent verification, a model to simulate and test changes, end-to-end network visibility, and treating ADS-B and FTI as backups for each other. Harris said staff would actively monitor the network during maintenance on core routers.

And since then, on the same program

  • Dallas, September 2025. A telecom outage at the Dallas approach-control facility halted flights at both major airports for hours. The FAA blamed Frontier Communications and said L3Harris’s oversight “failed to ensure that redundancies in the system functioned properly.”
  • Philadelphia, September 2026. A primary circuit failed while the backup fiber had already been cut by a construction crew in New Jersey. About 9,500 US flights were delayed or cancelled.
  • GAO, September 2026. On the $12.5B modernization, auditors found no reliable lifecycle cost estimate, no integrated master schedule and 11,389 uncoordinated project schedules, a risk of “excessive disruptions to air traffic control operations”.

Elsewhere

  • Rogers, Canada, July 2022. A multi-phase core upgrade started at “High” risk; the risk algorithm downgraded one phase to “Low” because earlier phases went well, which skipped lab testing and senior approval. A route filter was deleted, core routers had no overload limit, and the national network fell over, 911 included.
  • CenturyLink, December 2018. One faulty module injected malformed packets into the management channel that were forwarded endlessly between nodes: 37 hours of outage and roughly 15 hours just to find the source. The FCC’s fixes included disabling unused features and alarming on CPU and memory peaks.
  • Facebook, October 2021. A maintenance command withdrew the backbone and a bug in the audit tool failed to stop it. Out-of-band access depended on the same backbone.
  • FAA NOTAM, January 2023. A contractor fixing the sync between primary and backup databases deleted files and corrupted the backup too.

What went wrong, and what good looks like

# What happened How it should be done
1 Friday afternoon, near peak traffic. The work was under way around 3 PM Eastern on a Friday, under an internal rule that small remote sites may be worked in daytime. [Official] Anything that touches a shared national transport goes in a low-traffic night window, never on Friday afternoon, with a freeze before holidays. Friday afternoon means fatigue, thin on-call cover and a weekend between you and the fix.
2 Risk rated by site size, not blast radius. The work “shouldn’t” have had that kind of impact, per the Secretary. [Official] Impact analysis based on what shares transport, control plane and flooding domain with the box being touched. Same mistake as Rogers.
3 New box, risky default left on. A feature that “broadcasts data more widely than necessary” was enabled; the vendor says the router worked “as configured”. [Reported] Golden configuration templates, peer review and a diff against the intended config before the box joins the network. Disable what you don’t use, which was also an FCC recommendation after CenturyLink.
4 No containment. One site’s traffic reached 17 of 22 centers. [Reported] Tightly scoped flooding domains, control-plane policing, rate and session limits, storm control on every router, and segmentation so a radio site cannot hurt a control center.
5 No fast rollback in evidence. More than four hours to restore; recovery came from cutting two sites off once they were found. [Official + Inferred] Old and new paths in parallel, abort thresholds agreed in advance, a rehearsed rollback measured in minutes, and a technician on site with authority to disconnect at the first alarm.
6 Shared-fate backups. Backup channels ride the same network, per the FAA Administrator. Dallas 2025 and Philadelphia 2026 were both backup failures. [Official + Reported] A truly independent path for emergency frequencies: different carrier, route and equipment. Recommended in 2010.
7 Weak change tracking. Officials reportedly had to review paper logs to learn who was working where; GAO found 11,389 uncoordinated schedules. [Reported] An electronic change calendar wired into the NOC, so “what changed in the last 30 minutes, anywhere?” takes seconds.
8 No visible live watch on the wider network during the work. Unknown whether neighbor alarms fired; in 2009 they were muted. [Inferred] Engineers watching neighbor CPU, sessions and alarms during the window, and alarm filters audited so they cannot mute a whole fleet. Promised in 2010.
9 A rushed, waived program. The target moved 21 months earlier to end of 2028, and an August 2025 waiver exempted it from nearly all acquisition rules. [Reported] Speed is fine, but every waiver needs a compensating control: more testing, smaller batches, stricter windows.
10 2010 lessons not closed. Same contractor family, same kind of transition work on FTI, same failure spreading far beyond the expected scope. [Inferred] Track audit recommendations to closure: independent configuration verification, a simulation lab, end-to-end visibility.

None of these items needs a $50 billion budget. Most of them need a change calendar, a night window, a reviewed config and a rollback someone has actually rehearsed.

Open questions

  • What exactly is the Juniper feature, and what traffic was flooding: routing updates, broadcast or multicast, authentication retries?
  • Who wrote the configuration for the new router, was it reviewed, and was it tested before going live?
  • What role did the Amarillo radar site play?
  • What risk rating did the change carry, who approved a daytime window, and did neighbor routers raise alarms that nobody acted on?

A production change checklist you can steal

Written for a mixed Linux fleet run with Ansible, but it works for any shared infrastructure.

Before

  • ☐ Blast radius mapped: what shares network, storage, auth, DNS or control plane with the target? Rate risk on that, not on how small the box is.
  • ☐ Window chosen for low traffic. No Friday afternoons, no pre-holiday changes, freeze periods respected.
  • ☐ One change per window, so you know what to roll back.
  • ☐ Configuration generated from templates and peer-reviewed, never copied by hand from the old box without a diff. New devices start from a hardened baseline with unused features off.
  • ☐ Tested in a lab or on a canary that resembles production (same OS, firmware, model).
  • ☐ Rollback written, timed and rehearsed. If it takes longer than the outage you can afford, redesign the change.
  • ☐ Go/no-go and abort criteria agreed in advance, with numbers (CPU, error rate, sessions, latency).
  • ☐ Stakeholders and on-shift operators notified; the person on site has authority to abort.
  • ☐ Out-of-band access to every affected device confirmed working and not dependent on what you are changing.

During

  • ☐ Ansible: --check --diff first, then serial: 1 or a small canary batch with max_fail_percentage: 0 before the rest.
  • ☐ For network gear, use confirmed commit or auto-revert timers (NETCONF confirmed commit, RFC 6241 section 8.4), so a change that cuts you off reverts itself.
  • ☐ Watch the neighbors, not just the target: CPU, sessions, error counters, alarms. Make sure no alarm filter is muting them.
  • ☐ Change logged live in the shared change calendar the NOC actually looks at.
  • ☐ Abort threshold hit means roll back now, then investigate.

After

  • ☐ Validate end-to-end service, not just “link is up”.
  • ☐ Soak period with someone watching before calling it done.
  • ☐ Close the ticket with what actually changed; update the CMDB or inventory.
  • ☐ Post-incident review for anything that surprised you, and track its recommendations to closure.

Design hygiene (do it once, benefit forever)

  • ☐ Control-plane protection, rate limits and storm control on every router and switch.
  • ☐ Unused features and services disabled.
  • ☐ Backups truly independent: different path, provider and equipment.
  • ☐ Management and monitoring networks that survive the failure of the production network.
  • ☐ End-of-life hardware flagged, so any change near it is automatically high risk.

Sources

Facts as of 16 November 2026. The FAA, FBI and L3Harris investigations are ongoing and some details here may change.

Tags: , , , , , ,

Leave a Reply

Your email address will not be published. Required fields are marked *