Wednesday, August 1, 2018

Video: Monitor end-to-end latency of your IoT Application with 10,000 Sensors

This 5-minute Youtube video shows how to monitor response time to a
MQTT broker in an IoT Application with 10,000 active publishers.


This is not only important in the selection process, but also for ongoing
monitoring / troubleshooting, as outlined in our previous blog post
"IoT Sensors Need to be Managed", and this Gartner report.

We are following the testing methodology outlined in our previous post
"MQTT performance methodology using MIMIC MQTT Simulator" to
minimize the interference between the test equipment and the system
under test.

We are using the open-source Node-RED flows published in our Github
repository to measure and graph end-to-end latency between a publisher
and subscriber once a second for several brokers.

This simulates the latency between your sensors that are publishing
telemetry and your application that is consuming the telemetry.

We are using MIMIC MQTT Simulator to deploy 10,000 publisher clients
to the MessageSight broker on our intranet.

When we turn the switch on, the first 2,000 publishers are starting.
We'll time lapse this process for brevity. Notice how we are graphing
the number of active sensors sending telemetry.

The graphs show the latency for 4 brokers. Only the bottom line for
the MessageSight is being influenced with our active sensors. The
others are purely for control, to make sure our measuring and graphing
is correct. In particular, the mosquitto line should be steady, since
it is doing nothing.

The public broker graphs will be unpredictable.

Notice how the blue MessageSight line is mostly steady around 0
milliseconds.  The white mosquitto line is steady around 50 milliseconds,
and should remain so for the duration of the experiment. (It turns out
the reason for the 50 ms delay is explained here).

If your application has real-time requirements, then response time
is a vital parameter to monitor. Even if not, then response time
degradation can point to problems in your setup, specially at high
scale.

As more sensors become active, the blue latency graph becomes more
erratic. This is expected, as the broker is doing more work. In this
experiment, each sensor is only sending a small message every second,
and you can see the messages per second at the bottom of the MIMICview
graphical user interface.

There are many variables that could impact the latency: the distance
between sensors, brokers and applications, the message profile, that
is the average size of the payload and frequency of the messages, the
QOS of the messages, the topic hierarchy being published to, the number
of subscriber clients and their performance, the retention policy for
messages, and many more. Only your particular requirements would tell
whether the performance is acceptable for you.

(This is a follow-up to our earlier video).
 

Monday, July 30, 2018

Track end-to-end latency to your MQTT broker

Tracking end-to-end latency (response time) is useful in selecting your
MQTT broker, as well as monitoring your IoT platform / broker performance.

Check this real-time latency dashboard at

http://latency.iotsim.io/

which tracks end-to-end response time every second to a number of
public MQTT brokers over the internet. The local intranet mosquitto broker
measurement is for comparison.

The NODE-RED source code is at

https://github.com/gambitcomminc/nodered-mqtt-latency


Thursday, July 19, 2018

MIMIC MQTT Simulator enables rapid prototyping for Telit IoT Platform

If you want to prototype or test your solution with Telit's IoT Platform, then
you can use MIMIC MQTT Simulator to simulate a large number of sensors
and gateways.

In our initial effort we simulate a single sensor generating programmatic,
customizable, predictable, reproducible telemetry to Telit. With MIMIC what
can be done with one sensor can easily be scaled to thousands or even
millions of sensors. Complex scenarios can be setup once, and reproduced
at will thereafter. You can investigate different choices very quickly.

In the screenshot below, the light value changes randomly, but the
temperature value is changed predictably on-demand at run-time.


Friday, June 29, 2018

Testing IoT platform resilience with MIMIC IoT Simulator

With higher scalability of IoT environments in the face of outsider access
over the internet comes the requirement of proving resilience of your
IoT infrastructure. How do you know your IoT application can withstand
expected, and unexpected loads, recover from faults, and resist malicious
attacks?

Even in the absence of hackers, devices break and misbehave all the time.
How does your infrastructure handle the so-called crying baby, ie. a sensor
publishing too frequently, and how can be assured it does not interfere with
legitimate telemetry?

There are plenty of references about this requirement, eg. this post about
Azure throttling:

" We’re already dealing with serious-scale connectivity when we talk about the 
Internet of Things, and we impose the throttling limits on IoT Hub to protect 
against what otherwise looks like Denial of Service (DoS) attacks on the 
service."

or this Gartner report which contains

"... An IoT solution may be made up of hundreds or thousands of
devices. To test all of the devices in their real environments may be prohibitively
expensive or dangerous. However, you also need to ensure that your IoT 
platform and back-end systems can handle the load of all of those devices and 
correctly send and receive data as necessary."



Some platforms document their throughput testing, eg. Solace, or throttling
policies, like for Azure and Amazon.  For others this information is hard to
find.

But even if they are documented, throughput is highly dependent on your
specific application profile, and policies are subject to change at any
time, and you will likely run into some problem in production. Unless you
thoroughly tested before deployment, and can rely on QOS guarantees.

To test your specific performance requirements you will need to setup a
test that recreates your specific message patterns, connectivity patterns,
scale, message processing, etc, not just once, but continually.

MIMIC IoT Simulator is specifically designed for rapid development,
regression testing, continuous tuning, thorough training, compelling
demonstration of large-scale IoT environments.

MIMIC provides a virtual IoT lab with unlimited scale and a flexible,
programmable simulation framework to customize your performance testing.

As a quick example, we did a simple test to run a sensor at 10 messages
/ second to a broker, which ran indefinitely. Then we increased the rate
to 100 messages / second, and in multiple separate tests it was
disconnected after 20 minutes. A look at the packet exchange with
wireshark uncovered an explicit disconnect initiated at the broker.

(Original post at our blog page)

Wednesday, June 27, 2018

Virtual IoT Simulator Lab for Thingworx

In our quest for MIMIC MQTT Simulator to interoperate with a wide variety of
Internet of Things platforms and applications we have completed a scenario
where simulated sensors are feeding real-time, precisely controllable
telemetry via MQTT to PTC's Thingworx Industrial Connectivity.

With MIMIC you can simulate end-to-end large numbers of sensors and
actuators to create the complex scenarios for rapid development,
complete testing, thorough tuning and compelling demo and training
of IoT applications like Thingworx.

This 2-minute Youtube video demonstrates how a simulated sensor is
producing arbitrary and precise telemetry as charted in a hosted
Thingworx Foundation instance. This will now be extended to add
edge processing of a large number of sensors.


Friday, June 22, 2018

MIMIC MQTT Lab for Crouton Dashboard

Crouton is an open-source MQTT subscriber dashboard application based on
Node.js . It relies on its own JSON-based protocol layered on top of MQTT to
deliver dashboard features.

MIMIC MQTT Simulator is a scalable, customizable, predictable, dynamic
simulation platform designed allow rapid development / testing / deployment
/ tuning / training / demonstration of large-scale  Internet of Things
applications.

We have tailored one of our MQTT Labs to interact with Crouton through any
supported public broker. With it you can quickly explore Crouton features,
since you can customize payloads to the Crouton requirements instead of
low-level coding. Since values are dynamically changeable at runtime, you
can investigate dashboard deployment under a variety of scenarios. By
running many sensors, you can determine how scalable Crouton is.

This one-minute Youtube video shows it in action.


Wednesday, June 20, 2018

MIMIC MQTT Lab and IOTA MAM

IOTA is *the* DLT (distributed ledger technology) for the Internet of Things.
The facility for distributing telemetry from sensors is called MAM (Masked
Authentication Messaging), for details see their blog.

We were able to transfer dynamically changing MQTT telemetry generated
from our Bosch sensor simulation in our MIMIC MQTT Lab sent to the public
MQTT broker broker.hivemq.com, then through IOTA MAM to the IOTA
tangle in real-time.

You can check it yourself at this online IOTA MAM Explorer,   enter this
string

MGFJOZACIOWFEKGZARLWYPVUZGSDCAZPVPYACGCZJADGQWHFJLG9ZJGXNFIMVTCXMJUEEXUTEE9DDXHTC

where it says "enter root-address here...", optionally disable the "Pretty
print (JSON)" button, and press the search button.

This is a proof-of-concept meant to show that you can publish end-to-end
any telemetry from sensors to the IOTA tangle in real-time to be consumed
by your IoT application.

See this in action in this Youtube video.

You can also check the transactions on the ledger at https://thetangle.org/.