Monday, November 16, 2026

One Router Grounded a Nation. Would Your NOC Find It in Minutes?

On Friday, November 13, at 3:54 PM Eastern, the FAA stopped every departure in the
United States. The nationwide ground stop lasted for over four hours. More than 
15,000 flights were cancelled on Friday and Saturday, and more than 2,300 on
Sunday, before the FAA reported schedules returning to normal on Monday.

No one was hurt. The ground stop is a safety system, and it did its job. The failure 
is that it had to be used at all.
 
The next day, Transportation Secretary Sean Duffy gave the leading explanation.
During routine maintenance at a remote FAA site, a contractor was moving
telecommunication lines from copper to fiber. A router there began flooding the 
FAA network with more than 1,000 connection attempts every four to five seconds.
Duffy said it "might have caused" the outage; the investigation is still open. It 
would not be the first time: in 2009 a malfunctioning router slowed the entire 
US air traffic control system, and that problem stayed unresolved for four hours.
 
Whatever the final report says, the lesson for every network team is the same:
 
A fault at one remote site should stay at that site. When it doesn't, your
monitoring is the last line of defense, and it should name the device in
minutes, not hours.
 
So ask the question now, before your own Friday afternoon: if one router in your
network started flooding it, how long would your NOC take to find it?

1. Why One Device Can Stop a Network

A router sending 1,000 connection attempts every few seconds does not need to
break anything itself. The damage happens next door. The devices it talks to
spend their CPU and memory answering it. Session tables fill up. Links that
carried normal traffic are suddenly full. Real traffic gets queued, then dropped,
and services that depend on it start to time out.

That is why a flood is so hard to read from the outside. The first symptoms
appear far from the cause: an application stops responding in one place, a
link looks congested in another, alarms arrive from devices that are working
exactly as designed. The flooding router itself may look healthy.

The good news is that a flood is loud, if you know where to listen. It shows up
in at least four places:

  - Flow data. One source suddenly tops the list of talkers, with a huge
    number of short connections.
  - Interface counters. Packet and byte rates on its links jump far above
    their usual baseline.
  - The devices it floods. Sessions, CPU and error counters climb on its
    neighbors.
  - Alerts. Traps and syslog messages start arriving, often from the
    neighbors rather than the culprit.

Your tools already collect most of this. The question is whether they put it
together fast enough. That is the job of network monitoring.

2. Audit What Your Tools See

Before you need it, walk through this checklist with your own NOC tools:

  - Top talkers. Can you see, in one view, which device is sending the
    most connections right now, across the whole network and not one site
    at a time?
  - Baselines. Do your thresholds know what normal looks like for each
    link? A rate that is ten times normal at a quiet remote site may still
    be below a fixed, network-wide threshold.
  - Remote sites. Are your small, remote locations polled and exporting
    flows as carefully as your core? Changes there are often made by
    people who are not on your team.
  - Alert routing. When the neighbors start complaining, does the alert
    reach someone who can act, or does it join a flood of other alerts?
  - Change windows. Does a maintenance window raise your attention, or
    quietly mute alarms for the site where the work is going on?
  - Time to culprit. From the first alert, how many minutes until
    someone can name the device and the port? Have you ever measured it?

The last question is the one that matters, and it is the hardest to answer
honestly. You can't measure it by breaking your production network. You can
measure it in a lab.

3. Stage the Flood in a Lab

MIMIC Simulator lets you build the network first and break it on purpose,
without touching anything real.

Build a network replica. Start from a few thousand simulated devices: core,
distribution and many small remote sites. The Topology Designer draws the
whole layout in minutes. Each simulated device answers SNMP like the real
thing, so your NOC tools discover and poll it as usual.

Pick the culprit. Choose one router at a remote site and let it misbehave:

  - With the MIMIC NetFlow Simulator or MIMIC sFlow Simulator, its
    flow records report the flood: one source, a huge number of short
    connections, rising every second.
  - With the MIMIC SNMP Simulator, you change the rates of its
    interface counters in real time, so its utilization jumps exactly as
    a flooding device's would. You can do the same on its neighbors, and
    take interfaces down when they give up.
  - Traps and syslog messages arrive from the neighbors, just as the
    alarms would in a real event.

Watch your tools. Start a timer when the flood starts. Note when the
first alert fires, when a person first sees it, and when someone can name the
router and the port. That last number is your real time to culprit.

The video below shows the same idea in a security setting: MIMIC generating
DDoS and port-scan traffic, and ElastiFlow picking it out. For the full
walkthrough, see our ElastiFlow post.




If your tools name the router in minutes, you have proof. If they take
longer, you've found the gap in a lab, not on a Friday afternoon.

4. Make It a Drill

One test is a snapshot, and not sufficient. Networks and tools change every week:
a new site, a firmware upgrade, a new alert rule, a new person on the night shift.
Run the flood again after every change that matters, and keep the times.

The same test lab is also a training ground. Your NOC staff can practise on a
crisis that looks real, at full scale, without risk to the production network.
Customers have run NOC crisis training on 10,000 simulated devices for years.
A drill like this turns "we think we'd catch it" into a number you can show.

Find the Router Before It Finds You

On November 13, a single router at a remote site may have been enough to
stop a nation's air traffic for four hours. Your network has remote sites,
contractors and maintenance windows too.

Build the network in MIMIC, stage the flood, and time your tools. If you'd
like help setting up the dry run, tell us what your network looks like.

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.