United States. The nationwide ground stop lasted for over four hours. More than
Sunday, before the FAA reported schedules returning to normal on Monday.
No one was hurt. The ground stop is a safety system, and it did its job. The failure
During routine maintenance at a remote FAA site, a contractor was moving
telecommunication lines from copper to fiber. A router there began flooding the
monitoring is the last line of defense, and it should name the device in
minutes, not hours.
network started flooding it, how long would your NOC take to find it?
1. Why One Device Can Stop a Network
break anything itself. The damage happens next door. The devices it talks to
spend their CPU and memory answering it. Session tables fill up. Links that
carried normal traffic are suddenly full. Real traffic gets queued, then dropped,
and services that depend on it start to time out.
appear far from the cause: an application stops responding in one place, a
link looks congested in another, alarms arrive from devices that are working
exactly as designed. The flooding router itself may look healthy.
in at least four places:
- Flow data. One source suddenly tops the list of talkers, with a huge
number of short connections.
- Interface counters. Packet and byte rates on its links jump far above
their usual baseline.
- The devices it floods. Sessions, CPU and error counters climb on its
neighbors.
- Alerts. Traps and syslog messages start arriving, often from the
neighbors rather than the culprit.
Your tools already collect most of this. The question is whether they put it
together fast enough. That is the job of network monitoring.
2. Audit What Your Tools See
Before you need it, walk through this checklist with your own NOC tools:
- Top talkers. Can you see, in one view, which device is sending the
most connections right now, across the whole network and not one site
at a time?
- Baselines. Do your thresholds know what normal looks like for each
link? A rate that is ten times normal at a quiet remote site may still
be below a fixed, network-wide threshold.
- Remote sites. Are your small, remote locations polled and exporting
flows as carefully as your core? Changes there are often made by
people who are not on your team.
- Alert routing. When the neighbors start complaining, does the alert
reach someone who can act, or does it join a flood of other alerts?
- Change windows. Does a maintenance window raise your attention, or
quietly mute alarms for the site where the work is going on?
- Time to culprit. From the first alert, how many minutes until
someone can name the device and the port? Have you ever measured it?
The last question is the one that matters, and it is the hardest to answer
honestly. You can't measure it by breaking your production network. You can
measure it in a lab.
3. Stage the Flood in a Lab
MIMIC Simulator lets you build the network first and break it on purpose,
without touching anything real.
Build a network replica. Start from a few thousand simulated devices: core,
distribution and many small remote sites. The Topology Designer draws the
whole layout in minutes. Each simulated device answers SNMP like the real
thing, so your NOC tools discover and poll it as usual.
Pick the culprit. Choose one router at a remote site and let it misbehave:
- With the MIMIC NetFlow Simulator or MIMIC sFlow Simulator, its
flow records report the flood: one source, a huge number of short
connections, rising every second.
- With the MIMIC SNMP Simulator, you change the rates of its
interface counters in real time, so its utilization jumps exactly as
a flooding device's would. You can do the same on its neighbors, and
take interfaces down when they give up.
- Traps and syslog messages arrive from the neighbors, just as the
alarms would in a real event.
Watch your tools. Start a timer when the flood starts. Note when the
first alert fires, when a person first sees it, and when someone can name the
router and the port. That last number is your real time to culprit.
The video below shows the same idea in a security setting: MIMIC generating
DDoS and port-scan traffic, and ElastiFlow picking it out. For the full
walkthrough, see our ElastiFlow post.
If your tools name the router in minutes, you have proof. If they take
longer, you've found the gap in a lab, not on a Friday afternoon.
4. Make It a Drill
One test is a snapshot, and not sufficient. Networks and tools change every week:
a new site, a firmware upgrade, a new alert rule, a new person on the night shift.
Run the flood again after every change that matters, and keep the times.
The same test lab is also a training ground. Your NOC staff can practise on a
crisis that looks real, at full scale, without risk to the production network.
Customers have run NOC crisis training on 10,000 simulated devices for years.
A drill like this turns "we think we'd catch it" into a number you can show.
Find the Router Before It Finds You
On November 13, a single router at a remote site may have been enough to
stop a nation's air traffic for four hours. Your network has remote sites,
contractors and maintenance windows too.
Build the network in MIMIC, stage the flood, and time your tools. If you'd
like help setting up the dry run, tell us what your network looks like.





