IoT Sensors Need to be Managed

In order to be adopted, the components of the Internet of Things must be
manageable and managed, according to this Gartner report .
Specially in light of the revelations of vulnerabilities in commercial
networking equipment, such as the recent CERT advisory regarding Netgear
routers, CIOs are going to be hesitant of adding to the exposure to hackers
with this new class of networked gadgets.

Whether the sensors themselves have management interfaces, or the
network infrastructure (edge gateways, wireless access points, ...), the IoT
environment consists not just of the data plane, but must consider the
management plane.

MIMIC IoT Simulator provides a comprehensive, integrated framework for
simulating large IoT environments by providing common network
management APIs to simulated devices, such as SNMP, command line
interfaces (CLI), Web services, NetFlow, etc.  By combining the required
interfaces into your simulation you can exercise your management
plane for intrusion detection, fault and performance monitoring while
generating desired payloads to your IoT platform for large data analytics
and anomaly detection.

From original post

MIMIC MQTT Simulator for testing IoT Anomaly Detection

Anomaly detection for IoT is a challenge of both infrastructure monitoring and big data.

For an example of the former, if a sensor fails to PING for a while, it can be
assumed to be down or unreachable. This can be detected with traditional
network management applications, this scenario is shown with MIMIC 
NetFlow Simulator generating flows to ELK.

On the other end of the spectrum is a sensor that is malfunctioning by
generating too much data, as the highlighted green node in the Kibana
graph below.

Data generated by your IoT sensors are a special case data source for
Anomaly Detection.

For reference, check these white papers

https://www.bosch-si.com/internet-of-things/iot-downloads/iot-analytics-white-paper/anomaly-detection.html

https://aws.amazon.com/blogs/iot/anomaly-detection-using-aws-iot-and-aws-lambda/

https://www.oreilly.com/ideas/the-elements-of-anomaly-detection-in-the-internet-of-things

https://software.intel.com/en-us/articles/change-and-anomaly-detection-framework-for-internet-of-things-data-streams

Database techniques can be used to populate your data repository for
priming an anomaly detection algorithm, but only real-time generation
of precisely tailored data verifies that end-to-end processing works
as intended.

MIMIC MQTT Simulator can simulate large numbers of heterogeneous
sensors generating desirable data patterns in real-time over MQTT. For
example, you can have miriads of sensors generating MQTT payloads
containing a "normal" pattern, and instruct a small subset of them to
"misbehave" predictably, then observe how long it take to detect this
anomaly.

By deterministically varying the anomaly patterns in the simulator you are
able to tune and regression test iterations in your detection algorithm.
You are able even to explore boundary conditions of the infrastructure
requirements, such as message rates, failure conditions, etc.

Simulate thousands of Bosch sensors with MQTT Simulator

You can use the Bosch XDK Cross Domain Development Kit to connect your
Bosch sensor implementation to your IoT platform of choice. But how do you
load test with thousands or hundreds of thousands of sensors?

Here is a Youtube video that shows this in real-time.

We simulated the sample sensor in MIMIC MQTT Simulator with the sample
JSON in under one minute by just doing a copy/paste of the message from
the web page into MIMIC.

The "Subscriber" in the screenshot shows the unmodified message received
by the Mosquitto subscriber as the first message.

Then we modified certain fields to return different values. In the screenshot
they are the underlined "sn" and "value" fields.

Starting 1000 sensors to generate those values to the broker took another
minute. The "MIMICview" shows 5000 sensors configured, and 1000 sensors
started. The "Broker" terminal shows the IP addresses of the connecting
clients.

Dell/EMC performs complete testing using MIMIC Simulator

EMC has multiple copies of MIMIC Simulator. There is a team of 60 engineers
using MIMIC for scalability, performance, feature and regression testing.

Using MIMIC, they are able to simulate a lab with thousands of switches,
routers, firewalls, load balancers, and hosts from Cisco, Juniper, F5, Riverbed
and many other vendors at a fraction of the cost of hardware. They can also
recreate their customer provided  scenarios to support them better, and
certify newer devices with Smarts.

The engineers use MIMIC’s user-friendly GUI in addition to the scripting
interface. They are able to write scripts in multiple languages (Tcl, Java, Perl,
Python, C++, PHP); they can easily change simulations dynamically and
create interesting scenarios like "interface down" or "core router down", and
test disastrous conditions. They are able to query simulated devices
using SNMPv1, SNMPv2c or SNMPv3, and verify the same information using
command line interfaces (CLI) via Telnet/SSH. They can get the system
configuration using Cisco IOS commands, and then get the same information
using SNMP. They can also generate NetFlow (v5, v9, IPFIX, NBAR) flows.
Since all of these are integrated in MIMIC, they see very consistent data, just
like real devices

For details see our case study.

MIMIC MQTT Simulator and Elk Stack

Thinking of feeding MQTT messages of your sensors into an
Elk Stack via a MQTT input plugin?

You could deploy hardware sensors or test clients for your testbed.
Alternatively, MIMIC MQTT Simulator provides a large virtual sensor
network of up to 100,000 sensors per host, so you can rapidly prototype
different types of sensor scenarios and scale up cost-effectively.

MQTT performance methodology using MIMIC MQTT Simulator

Performance of small-scale environments never predicts behavior of
large-scale deployments. But, it is too expensive to setup large numbers
of your MQTT sensors to load test your IoT back-office platform, including
MQTT broker and client applications.

With MIMIC MQTT Simulator, it is simple to create large sensor simulations
(you can run up to 100,000 simulated sensors on a single server)
to verify performance.

But how do you guarantee that your performance tools are not impacting
your tests? For example, if you determine that your broker handles a load
generated by your load generator with satisfactory response, are you sure
that the load generator is not slowing down the whole test?

The methodology is to use MIMIC to simulate a large environment with
synthetic background throughput (the "load rig"), then verify your performance
requirements (eg. maximum round-trip delay) either with a small number of
your real-world sensor, or with another MIMIC setup measuring end-to-end
latency (the "measurement rig"). That way you are sure the synthetic load is not
impacting your measurement setup except through the system under test, the broker.

In the screenshot above we are running 10 sensors with an end-to-end
measuring instrumentation, and the end-to-end delay is graphed in the
bottom graph. It shows minimum, average and maximum delay for messages
from those sensors to a subscriber running in the same MIMIC.

From another MIMIC instance, we keep adding a synthetic load onto the
MQTT broker under test, from 0 to 1000 in steps of 100. The upper graph
shows the size of the background load over the 15 minutes of the test.
Each background load sensor publishes at 1 message per second, so the
throughput is the same as the number of sensors. This is trivial to
change in MIMIC to conform to your real-world expectations.

As you can see, the delay is only slightly increasing over time, except
for 2 notable bumps at 600 and 1000 sensors. It is trivial to repeat the
scenario, and verify that indeed there is a reproduceable problem. You
would never know if you did not do the tests.

(This post has been updated by this newer post.)

How MIMIC Simulator creates large flow matrices

Traffic matrices have been around for a long time as detailed in this
article. A cell in the matrix denotes a quantity of traffic flowing from a
source to a destination. Partitioning each of the source and destination
dimensions into interesting subsets allows to draw conclusions about
traffic flowing into and out of those subsets. For example, traffic flowing
into and out of a company network, or along geographic or organizational
boundaries.

With the advent of NetFlow and Cisco's NBAR the 2-dimensional source
IP/destination IP (network traffic) matrices have been extended to add
the application dimension: not only do we know how much traffic flows
between network nodes, but also which kind. For example, we can
quantify how much Youtube traffic flows in and out of the company network
vs. plain web-surfing or shopping.

MIMIC NetFlow Simulator can simulate any flow matrix for network
performance and security monitoring. In a vast sea of normal traffic
flows slight deviations can be added deterministically to test and
demonstrate performance monitoring features or security policies.

For example, MIMIC is being used to shape application traffic for
StealthWatch, Scrutinizer, Intermapper, and even in the on-demand
demo cloud for Cisco's dCloud.

Test your management application with hundreds of Dell iDRACs

Testing your configuration management application or data center
orchestration software for large scalability requires you to setup a
large lab with many servers.

In the case of Dell servers, the Dell iDRAC is a card with out-of-band
control and monitoring of a Dell server. Rather than buying many servers
and hardware, you can simulate the iDRAC and server in MIMIC Web Simulator.

The screenshot below shows the Dell OpenManage Essentials application
managing 100 simulated Dell servers. The simulation allows complete
control to achieve whatever testing or demo scenario you want.

Once simulated in MIMIC, each of these devices are available on
the network as if they are real devices. Your management applications
can access them and have a complete control over their configurations.
You can easily start, stop them and change their parameters at
run time. You can customize the simulation at run-time to introduce
faults.

SNMP Simulators: build vs. buy

Faced with the prospect of simulating large network management
environments, the typical build vs. buy dilemma emerges: should you pay
for a commercial solution, or build one of your own?

There are several free open-source simulators available, promising to tilt
that equation in favor of "build": since a seemingly viable alternative is
already available, you'll just modify what's out there to suit your needs.

This will work out in only the fewest scenarios: the simplest small-scale
simulations.  For most other uses, there are many pitfalls:

1. Meeting the requirements: You have to evaluate the best open-source
for your purposes, and you risk making a sub-optimal choice. You might
pick one, only to find out it does not handle your changing requirements;

2. Developer training: You have to train your developers on the chosen
solution(s), not only from a end-user perspective, but down to the nitty
gritty design, source code and build procedures. You are essentially
investing in something other than your core competency;

3. Quality: Open-source might be high-quality for commodity software
(Linux, OpenOffice), but in highly-specialized areas it usually has few
developers/maintainers, goes out of date, and results in lower-quality
software.

4. Scalability: Designs that work for a small number of simulated entities
usually will not for large numbers due to worse-than-linear behavior of
algorithms. This will usually result in unacceptable performance.

5. Beyond SNMP: Network management applications these days use more
than SNMP to access the managed entities. This requires an integrated
solution that simulates not only the SNMP interface, but also command-line
interfaces (Telnet, SSH), flow-based protocols (NetFlow, sFlow), Web services
(XML, SOAP, REST, etc).

6. Budget for software maintenance: The initial development task is
usually a small part of the total effort. At least one developer has to
continuously maintain and upgrade the software, post messages on some
user groups for getting it built and deployed but also try to get timely
answers for resolving bugs.

 The end result is usually that you'll spend way more effort on the
"build" decision than the cost of the commercial solution would have
been to start with, impacting time-to-market, quality, support, etc.

Many large companies have already chosen to buy instead of spending
effort re-inventing the wheel. Their thinking is that it is better
to spend time and energy behind the companies' core product and services.
Do they know something you don't?

MIMIC SNMP Simulator has been on the market for 20 years. Simulators is all
that Gambit does and has hundreds of man years invested in. The end result
is a mature, high-quality, high-performance simulator that runs at most of
the large networking vendors world-wide. For example, reference customers
are vendors such as Lancope and Xirrus, and end-users such as Pepco and
SITA.

Combined with the other simulators in the MIMIC Simulator Suite you can
have a large realistic environment for development, testing, demo and
training of your network management application.

MIMIC runs on physical servers or the cloud and has been adapted to many
unforeseen uses. For example, Cisco runs MIMIC in combination with Prime
Infrastructure in their on-demand cloud platform dCloud.

Simulate your RelayR IoT environment with MIMIC IoT Simulator

Don't have different types of sensors to test / prototype / deploy your
IoT application on RelayR? Looking to scale without the expense?

Now you can quickly simulate different types of sensors to send MQTT
payload messages at arbitrary rate to your application.  In this
screenshot we quickly setup 4 different sensor types showing dynamic
values changing in the RelayR console.

MIMIC IoT Simulator can simulate unlimited number of sensors of
unlimited variety with unlimited scenarios, all in software. You can go
beyond the requirements so that there will be no surprises after
deployment.

Scale your IBM Bluemix IoT testing with MIMIC MQTT Simulator

Setting up large IoT environments for development, testing and training
is expensive and time-consuming.

Now you can easily scale your IoT environment for testing with
IBM Bluemix with MIMIC MQTT Simulator. MIMIC simulates large
numbers of sensors generating arbitrary MQTT payloads at desired rates
to prototype application policies, test performance under stress, and
develop sales demonstrations and training curricula.

Here we see 10 simulated MQTT sensors connected to Bluemix generating
messages and responding to commands. With MIMIC, what you can do with
10 sensors you can do with 100,000 sensors from one server or virtual
machine on either private or public cloud.

Quickly prototype your subscriber client with MIMIC and a public MQTT broker

The public MQTT brokers at

http://moxd.io/2015/10/public-mqtt-brokers/

allow you quickly to test your MQTT clients. The problem is: so do
many other people. Thus, the messages you receive are completely
outside your control.

By using simulated sensors with MIMIC MQTT Simulator, you can
rapidly prototype your MQTT subscriber application to scale your
IoT environment. For example, this screenshot shows 100 simulated
sensors publishing arbitrary payload messages at predefined rate,
and a mosquitto subscriber client displaying them.

At the bottom left, the "10000-ft view" of the MIMICView GUI
allows you to monitor and control thousands of simulated sensors.
In this case we see the icons of 100 active sensors.
The MIMIC Log Window with diagnostic messages at bottom right
displays the traced MQTT activity of one of the sensors.
The Mosquitto subscriber at top right displays the received
JSON-formatted messages on one of the topics. The tcpdump
capture in the middle top shows the packets exchanged between
simulated sensors and broker.

Drive automated regression tests with MIMIC and Jenkins

FYI, we have verified that MIMIC Simulator works with Jenkins Continuous Integration Server

https://jenkins.io/2.0/

by easily configuring it to run some of our MIMIC regression tests. You
should be able to take advantage of this platform to drive the automated
regression tests of your network management app or IOT application with
MIMIC.

The attached screenshot shows the log of one of the regression tests which
traverse through various SNMP MIBs. You can run any of the scripts
provided in the MIMIC script library. Since MIMIC can be controlled from
multiple scripting languages, it can easily be controlled from a continuous
delivery platform like Jenkins to setup and exercise your application
features under test.

Anatomy of a virtual sales demo appliance for network management

Anatomy of a virtual sales demo appliance for network management

Overview

Traditionally, providing a sales demo for network management software meant setting up a 
physical environment featuring the desired variety of networking equipment to cover the sales 
demonstration to the particular target customer.

Furthermore, most sales scenarios required complicated setup, and required travel to the 
customer site, or trade shows, or time-sharing of central facilities in a sales demo lab or 
web-site. These limitations made such demos cost prohibitive, resulting in small-scale, static 
demos that did not materially impact sales conversion.

With the advent of ubiquitous virtual machines (VMs) and network simulation, sales demo labs 
can be provisioned cost-effectively by implementing virtual demo appliances on either public or 
private clouds. Bundling the management app and MIMIC Simulator on a VM image, allows the 
sales staff to instantiate as many demo labs as needed at any point in time, and for the sales 
associate to access the demo environment over the Internet whenever and as long as needed. 
Realistic self-service sales demonstrations can be provisioned automatically.

Benefits of cloud based sales

Besides the obvious benefits of reduced sales cycle and higher conversion due to targeted 
demonstration of a variety of features, higher customer satisfaction and more customer loyalty 
(thus, repeat business), here are some additional advantages of appliance-based sales:

    Multiple on-demand demo and training sessions can be scheduled without
    worrying about conflict or sharing.

    On-demand pre-sales training can be used for application
    evaluation. With a larger variety of scenarios, demos can be
    tailored to the customers needs.

    Many potential customer take a long time to setup the test lab
    and install the application for the evaluation. An online demo can
    get them started on playing with application earlier so they can
    already get familiarized with the features. When the lab gets ready,
    they can directly start on testing the specific needs. This reduces
    the sales cycle significantly.

    Partner training reduces the support cycle and improves closing rate
    by better demonstrating features that specifically solve customer
    requirements

Implementation

MIMIC Simulator is a simulation software suite including SNMP Simulator, WEB Simulator, 
NetFlow/sFlow Simulator, designed to accurately create virtual environments to interoperate 
with network management applications. This solution is already deployed in an on-demand 
cloud offering called Cisco dCloud, with on-demand sales labs for Cisco Prime Infrastructure. 
This offering leverages the SNMP, NetFlow and Cisco IOS simulation features of MIMIC toward 
high-impact sales demonstrations.

 

Other such sales demo appliances can easily be implemented as an on-demand, single-tenant 
offering. Similarly to traditional software installations, the network management application is 
installed in a VM environment along with MIMIC Simulator. The simulator transparently provides 
a scalable, dynamic network environment to be managed by the application. The customer and 
even the sales associate driving the demo usually do not know what is running behind the 
scenes.

Many R&D and Quality Assurance departments already test network management applications 
against MIMIC Simulator. They setup the simulator to provide predictable, repeatable, 
regressionable scenarios that run the management software through its paces. Those 
scenarios can be adapted for sales demos.

An extra customization adds a simple demonstration scaffold to provide the sales staff a 
user-friendly choice of scenarios to demonstrate. Each scenario configures the simulator to 
create a particular network configuration and traffic patterns to impact the management 
application. For example, a faulty configuration or an attack scenario can be simulated to show  
troubleshooting procedures.

Once a snapshot of this VM lab is saved, any number of independent instances can be 
provisioned for the required demo periods. The cloud framework isolates each of the VM labs 
from each other, preventing them from affecting each other.

Summary

A combination of MIMIC Simulator and network management application bundled on a virtual 
machine provides cost-effective, on-demand sales demonstrations tailored to customer needs. 
Deploying this appliance on the public cloud scales the demo resources at incremental and 
predictable cost, enabling precise cost/benefit analysis when integrated into the sales CRM 
environment.

Train your network operators like airline pilots with MIMIC Simulator

When commercial pilots train for disasters it's not with Microsoft Flight
Simulator, but something more like this realistic simulator.

You too can avoid outages like the recent Southwest grounding by training
your network operators to handle unforeseen scenarios with the industry-
leading MIMIC Simulator.

Create network labs that approximate your production network and play
out complex scenarios on demand to implement / test / train network
management policies. Train your IT staff with all the latest devices and
various network environments. That way they are prepared when a disaster
happens and resolve it quickly.

Large topologies with MIMIC for development, testing, sales demos

Would you prefer a topology like this

or like this

to verify / demonstrate your network management application?

With MIMIC Simulator Enhanced Topology Wizard, you can create many
different synthetic, large-scale, verifiable networks in minutes to
drive development, testing, sales demonstrations of your configuration
management application that visualizes topologies.

For example, here are different 500-node topologies:

Too symmetric? Spend a couple of minutes more and create something like this:

Long-term latency test with MIMIC MQTT Simulator

If your large-scale IoT application has latency requirements, you want
to do long-term testing to ensure your environment satisfies those
requirements.

We ran the end-to-end latency test as shown in this published video
with MIMIC MQTT Simulator against a third-party MQTT broker setup
overnight, and graphed the latency reported by our subscriber client
action script with Grafana.

The setup simulates 3000 MQTT clients, of which one is the subscriber
that measures round-trip delay of published messages to unique topics.
This tests a common fan-out scenario.

We did not expect to find anything interesting. The attached picture
shows a good steady state, with average and max latency times below
and around 500 msec (respectively) ... most of the time.

But, there are 2 instances where max latency times are above 1 second.

If your IoT application requirement is latency times below 1 second,
you'd be worried, and would need to start investigating. Without this
test you would never know about any long-term problems in your setup.

MQTT Simulator enables large-scale Azure IoT testing

We have recently simulated many sensors with MIMIC MQTT Simulator
publishing arbitrary messages to a subscriber client connected to
Azure IoT Hub as a proof of concept of large-scale testing of IoT application
deployment using this platform.

MIMIC MQTT Simulator interoperates with major brokers

In addition to our prior published reports, we recently simulated
thousands of sensors to 3 more brokers

VerneMQ

RabbitMQ

EMQTT

with thousands of arbitrary messages per second published.

Regardless which broker you select, you can perform prototyping, scalability,
robustness and security testing of your IoT application with
MIMIC MQTT Simulator prior to deployment.

How is realistic MQTT testing with MIMIC different from simplistic testing?

Most simple test apps use a for loop to create connections to a MQTT broker,
using only one single IP address for a large number of client sessions.

By using one IP address per session with MIMIC MQTT Simulator just like in

real world scenarios (when do real sensors share IP addresses?) we
uncovered a flaw in third-party MQTT brokers.

They would fail after a small number of simulated sensors (typically 1000
connections).

Only after adjusting the ARP cache parameters on the broker systems did
we overcome this problem.

MIMIC allows you to create realistic IoT pre-deployment testing scenarios
using IPv4 or IPv6 addressing, just like in the real world.

MIMIC MQTT Simulator publishing synthetic load to HiveMQ

MIMIC MQTT Simulator is simulating 40,000 connected sensors.
The HiveMQ MQTT Broker connected with them as if they are physical
sensors. As shown in the graphs, MIMIC is generating messages at the
rate of around 4,000 messages / second.

In this scenario it has been running for many hours.
The 40001st connection is a third-party subscriber client. This shows
that MIMIC simulated sensors are no different than the real one.

MIMIC allows you to stage precise pre-deployment tests for HiveMQ and
other IoT applications to make sure they handle the expected load and
provide the needed performance. In addition, you can easily customize
message payloads of simulated sensors for rapid prototyping, and create
pathological scenarios to minimize risks.

MIMIC MQTT Simulator for rapid prototyping of Amazon AWS IoT applications

A simulated sensor in MIMIC MQTT Simulator is generating JSON
messages to Amazon AWS IoT . The AWS subscriber client is showing
the received messages.

MIMIC allows rapid prototyping of many publishers with custom payloads
and custom message rates. By simulating thousands of such sensors you
can benchmark your performance with AWS IoT

Setting up a benchmarking environment to test the performance
of large-scale IoT applications is specially cumbersome. It needs a large
number of sensors, each with different IP address. Each type of sensor
sends out different messages at different rates and with unique content.

MIMIC MQTT Simulator creates such an environment with thousands of
sensors. In this example we have simulated an Intel sensor that sends
messages to the AWS IoT gateway. You can see in the image that the
messages are received by the AWS IoT subscriber client. The MIMIC log
window shows the connection-related transactions.

This can be easily multiplied to simulate thousands of sensors to create an
IoT lab with the desired variety of sensors. You can use the environment to
benchmark the performance matrix for your IoT applications.

Benchmark your MQTT application with MIMIC MQTT Simulator

Now you too can benchmark your MQTT application with MIMIC MQTT Simulator.

Here the IBM MessageSight MQTT broker is handling 100,000 connections
from MIMIC simulated MQTT sensors publishing messages at around 10,000
/ second to one subscriber client.

Everything in the MIMIC MQTT Simulator is runtime-configurable to suit
your custom requirements.

MIMIC MQTT Simulator interoperates with IBM MessageSight

In a little over 10 minutes we start 20,000 simulated sensors connecting
to the IBM MessageSight MQTT Broker. Each sensor on its own IP address
is publishing messages at the rate of 1 / minute to its own topic, for
a total of around 340 / second. Since the broker is forwarding those to
the lone subscriber, the overall message rate is doubled as shown.

With MIMIC MQTT Simulator you can simulate up to 100,000 (yes, one
hundred thousand) sensors from one host.

MIMIC Simulator simulates Internet of Things

The number of Internet of Things devices by 2020 is forecast between
20+ BILLION devices according to Gartner and 38 BILLION according to
Juniper Research.

MIMIC MQTT Simulator supports simulating up to 100k (yes, 100000)
sensors from one host. You can generate any message for any sensor type
with any frequency to test your brokers and subscribers. Any number of
hosts can be controlled to provide unlimited scalability.