Friday, September 21, 2018

Fire-and-forget vs. complete control of MQTT message generation

Free, open-source MQTT performance tools follow the "fire-and-forget"
paradigm: you configure message generation in advance, then fire the
tool to perform the load testing.

This is fine for static scenarios, but when are real-world scenarios
static? Message characteristics change all the time: sensors connect and
disconnect, you get Denial-of-Service (DOS) attacks or mal-functioning
sensors (the "crying baby" scenario). How will you test that your
application load balancer (ALB) throttles misbehaving connections in
real-time?

MIMIC MQTT Simulator lets you change any message characteristic
(frequency, topic, payload, QOS, etc) at any point in real-time, not just for
one sensor, but also for many thousands if not millions. This allows you to
create anomaly scenarios that simpler tools don't allow you to.

In this 4-minute Youtube video we show our open-source "MQTT Topic
Statistics" mqtt-stats  subscriber client connected to a IBM MessageSight
broker instance. MIMIC MQTT Simulator is used to generate different
message loads dynamically during the video, verified by mqtt-stats.

In the simulator we first start 10 simulated Sparkplug sensors. Each
sensor publishes to 3 topics (NBIRTH, DBIRTH and DDATA) in order to generate
telemetry. By default each is sending 1 DDATA message per second.

In the steady-state (when all sensors are registered) mqtt-stats shows the 30
topics and 10 active topics (see in video). We show the 10 NBIRTH, DBIRTH,
and DDATA topics as defined in the Sparkplug specification. At this point we
can clear out the topics with File -> New to only show the active DDATA
topics. All sensors are showing a rate of 1 message / second.

Now we can change message rates dynamically, we pick 2 sensors to send 5, then
10, then 20 messages / second. Then we pick 3 different sensors and change their
frequency to 50 / second. And yet a different one with 100 / second. Then we reset
all 10 back to 1 message / second.  What we did interactively you can script
programmatically with the MIMIC API in 6 languages.

The rest of the video show a total of 1000 sensors being monitored by the
mqtt-stats client.


Thursday, September 13, 2018

Node-RED and 1000 simulated Sparkplug B sensors

In a previous post we discussed a technique to reduce the number
of tracked telemetry in the Node-RED IoT application, which only graphed
interesting temperature values from 20 simulated Bosch sensors.

In this experiment, MIMIC MQTT Simulator is simulating 1000 Sparkplug
sensors sending telemetry to Node-RED, graphing 10 of them in real-time.
We notice that each sensor has unique, expected telemetry.



This open-source Github repository has the flow source code to generate
these graphs.



The MQTT input node sends messages to a decode-protobuf node, which
decodes the Sparkplug messages coming from the simulated sensors and
passes them on as JSON message objects. The next node switches between
NBIRTH / DBIRTH / DDATA messages. The first 2 just count the arrivals. The
DDATA messages are graphed only for the first 10 sensors, else there would
be too many lines in the graph. Unknown messages are counted as well.
We implemented a simple message performance instrumentation by
resetting the message count every second.

Monday, September 10, 2018

Graphing arbitrary Sparkplug telemetry in Ignition

We followed this Ignition tutorial to create a real-time graph of simulated
telemetry exported from MIMIC MQTT Simulator.

This short Youtube video shows the 3 independently controlled temperature
values being graphed in the Easy Chart. We had to apply this workaround
to get the tags to be graphed.



With MIMIC, you can export arbitrary, precise, customizable telemetry in
real-time to any MQTT-based IoT platform for thousands to millions of
simulated sensors over the internet or in your network edge.

(This is a followup to our previous post.)

Wednesday, August 29, 2018

Testing MQTT Telemetry with arbitrary function

Do you need to test your IoT platforms against arbitrary telemetry data from
a variety of sensors?

You can easily to generate ANY data you want from ANY sensor you want
using MIMIC IoT Simulator.

The below screenshot shows a NODE-RED subscriber client graphing a
simulated sensor publishing data of a Sine function with configurable
amplitude. In this case we just kept expanding the amplitude every minute.



The code to achieve that is shown and consist of a total of 10 lines including
comments and the effort in minutes.

Since the amplitude is parametrized, with the same code you could have
multiple sensors sending the same function, but different amplitudes at any
point in time. Any other parameters can be made unique for each simulated
sensor.

The next screenshot shows 4 such sensors with different amplitudes
simultaneously. Any more would clutter the single graph too much.


Wednesday, August 1, 2018

Video: Monitor end-to-end latency of your IoT Application with 10,000 Sensors

This 5-minute Youtube video shows how to monitor response time to a
MQTT broker in an IoT Application with 10,000 active publishers.


This is not only important in the selection process, but also for ongoing
monitoring / troubleshooting, as outlined in our previous blog post
"IoT Sensors Need to be Managed", and this Gartner report.

We are following the testing methodology outlined in our previous post
"MQTT performance methodology using MIMIC MQTT Simulator" to
minimize the interference between the test equipment and the system
under test.

We are using the open-source Node-RED flows published in our Github
repository to measure and graph end-to-end latency between a publisher
and subscriber once a second for several brokers.

This simulates the latency between your sensors that are publishing
telemetry and your application that is consuming the telemetry.

We are using MIMIC MQTT Simulator to deploy 10,000 publisher clients
to the MessageSight broker on our intranet.

When we turn the switch on, the first 2,000 publishers are starting.
We'll time lapse this process for brevity. Notice how we are graphing
the number of active sensors sending telemetry.

The graphs show the latency for 4 brokers. Only the bottom line for
the MessageSight is being influenced with our active sensors. The
others are purely for control, to make sure our measuring and graphing
is correct. In particular, the mosquitto line should be steady, since
it is doing nothing.

The public broker graphs will be unpredictable.

Notice how the blue MessageSight line is mostly steady around 0
milliseconds.  The white mosquitto line is steady around 50 milliseconds,
and should remain so for the duration of the experiment. (It turns out
the reason for the 50 ms delay is explained here).

If your application has real-time requirements, then response time
is a vital parameter to monitor. Even if not, then response time
degradation can point to problems in your setup, specially at high
scale.

As more sensors become active, the blue latency graph becomes more
erratic. This is expected, as the broker is doing more work. In this
experiment, each sensor is only sending a small message every second,
and you can see the messages per second at the bottom of the MIMICview
graphical user interface.

There are many variables that could impact the latency: the distance
between sensors, brokers and applications, the message profile, that
is the average size of the payload and frequency of the messages, the
QOS of the messages, the topic hierarchy being published to, the number
of subscriber clients and their performance, the retention policy for
messages, and many more. Only your particular requirements would tell
whether the performance is acceptable for you.

(This is a follow-up to our earlier video).
 

Monday, July 30, 2018

Track end-to-end latency to your MQTT broker

Tracking end-to-end latency (response time) is useful in selecting your
MQTT broker, as well as monitoring your IoT platform / broker performance.

Check this real-time latency dashboard at

http://latency.iotsim.io/

which tracks end-to-end response time every second to a number of
public MQTT brokers over the internet. The local intranet mosquitto broker
measurement is for comparison.

The NODE-RED source code is at

https://github.com/gambitcomminc/nodered-mqtt-latency


Thursday, July 19, 2018

MIMIC MQTT Simulator enables rapid prototyping for Telit IoT Platform

If you want to prototype or test your solution with Telit's IoT Platform, then
you can use MIMIC MQTT Simulator to simulate a large number of sensors
and gateways.

In our initial effort we simulate a single sensor generating programmatic,
customizable, predictable, reproducible telemetry to Telit. With MIMIC what
can be done with one sensor can easily be scaled to thousands or even
millions of sensors. Complex scenarios can be setup once, and reproduced
at will thereafter. You can investigate different choices very quickly.

In the screenshot below, the light value changes randomly, but the
temperature value is changed predictably on-demand at run-time.