Thursday, December 6, 2018

How to test the accuracy of your IoT applications

While implementing your MQTT-based application's back-end processing,
such as real-time archiving, analytics, edge-processing, graphing, how
do you  verify that it will store and process all received telemetry messages,
with no messages missing, no extra messages, no bytes altered, in the
correct order? Furthermore, it is really hard to make sure that it works at
required scale and speeds, eg. at millisecond granularity.

You need to come up with a performance test that goes beyond simple
load testing and handles each of the test requirements (message integrity,
sequencing and frequency).

To test this case, you can apply the following features found in MIMIC MQTT Simulator:

1. copy the existing Bosch XDK simulation as detailed in an earlier post .
This ships with MIMIC, and generates a realistic JSON payload similar to

{
 "sn":"20:19:AB:F4:00:01",
 "data":{
  "acc":{"x":26,"y":32,"z":1012,"unit":"mG"},
  "gyro":{"x":1220,"y":-6835,"z":-2319,"unit":"mdeg/s"},
  "mag":{"x":40,"y":1,"z":-4,"unit":"uT"},
  "light":{"value":999,"unit":"mLux"},
  "temp":{"value":"100","unit":"mCelsius"},
  "pressure":{"value":98897,"unit":"Pascal"},
  "humidity":{"value":39,"unit":"%rh"}
 }
}


By copying it to a new simulation, you can now customize it without
disturbing the original simulation.

2. change the frequency of the generated messages to 1 msec;

3. change the number of messages to exactly 1000, not one more, not one
less;

4.  change the JSON payload function to return light values in sequence;

5. change the QOS to 2.

This is a total of about 5 lines of change to the existing simulation
configuration.

Once you configure a MIMIC sensor with this simulation, it generates the
1000 messages, and you are able to verify that they are all there in the
correct order. This entire exercise should only take about 15 minutes,
as it did when we tried it


Follow-up tests can be to scale up to run multiple sensors (as many as
required), as detailed in this previous post, and for longer periods of time
(see also this post), with payloads that resemble those in your application
(see this previous post).


Monday, November 26, 2018

Arbitrary telemetry to ELK/Logstash/Kibana

In a previous post we saw how MIMIC MQTT Simulator was used to
generate arbitrary telemetry to the Ignition IoT platform.

In this post we experimented with sending telemetry data with
values in a sine-wave pattern to the MQTT plugin for ELK (Elasticsearch,
Logstash, Kibana).


The Bosch sensor simulation provided with the MIMIC MQTT module
gives rolling motion indications commonly seen in masthead devices,
including wind sensor, wind transducer, and other weather devices.

In the simulation script, the sine wave or a constant value is easily
assigned to accelerometer or gyro values in the lower left window text
editor.

In the ELK illustration, the search engine based application draws the
sine wave as a heat map between peaks and troughs in the upper right
window as steady air current is detected. For comparison, a prominent,
near constant value of banking angle is shown in the lower right window
indicating a calm, windless air condition.


Friday, October 19, 2018

Simulated security cameras to Domoticz

The FOSS (free open-source software) Domoticz home automation system
supports the MQTT IoT protocol to interoperate with a variety of devices
as detailed on their Wiki page.

We setup the MQTT gateway to monitor a Dahua IP camera we got at
Home Depot as detailed in their hardware compatibility page and recorded
the MQTT session.

Then we duplicated this simulation manyfold with MIMIC MQTT Simulator,
and were able to simulate VideoMotion events from any number of simulated
cameras as shown in the screenshot below.


Can you tell which is real and which is simulated?

This 1-minute Youtube video shows which we are manipulating in real-time.
You can generate any video motion events in any order with any frequency
to test your security policies of your IoT application.

Friday, September 21, 2018

Fire-and-forget vs. complete control of MQTT message generation

Free, open-source MQTT performance tools follow the "fire-and-forget"
paradigm: you configure message generation in advance, then fire the
tool to perform the load testing.

This is fine for static scenarios, but when are real-world scenarios
static? Message characteristics change all the time: sensors connect and
disconnect, you get Denial-of-Service (DOS) attacks or mal-functioning
sensors (the "crying baby" scenario). How will you test that your
application load balancer (ALB) throttles misbehaving connections in
real-time?

MIMIC MQTT Simulator lets you change any message characteristic
(frequency, topic, payload, QOS, etc) at any point in real-time, not just for
one sensor, but also for many thousands if not millions. This allows you to
create anomaly scenarios that simpler tools don't allow you to.

In this 4-minute Youtube video we show our open-source "MQTT Topic
Statistics" mqtt-stats  subscriber client connected to a IBM MessageSight
broker instance. MIMIC MQTT Simulator is used to generate different
message loads dynamically during the video, verified by mqtt-stats.

In the simulator we first start 10 simulated Sparkplug sensors. Each
sensor publishes to 3 topics (NBIRTH, DBIRTH and DDATA) in order to generate
telemetry. By default each is sending 1 DDATA message per second.

In the steady-state (when all sensors are registered) mqtt-stats shows the 30
topics and 10 active topics (see in video). We show the 10 NBIRTH, DBIRTH,
and DDATA topics as defined in the Sparkplug specification. At this point we
can clear out the topics with File -> New to only show the active DDATA
topics. All sensors are showing a rate of 1 message / second.

Now we can change message rates dynamically, we pick 2 sensors to send 5, then
10, then 20 messages / second. Then we pick 3 different sensors and change their
frequency to 50 / second. And yet a different one with 100 / second. Then we reset
all 10 back to 1 message / second.  What we did interactively you can script
programmatically with the MIMIC API in 6 languages.

The rest of the video show a total of 1000 sensors being monitored by the
mqtt-stats client.


Thursday, September 13, 2018

Node-RED and 1000 simulated Sparkplug B sensors

In a previous post we discussed a technique to reduce the number
of tracked telemetry in the Node-RED IoT application, which only graphed
interesting temperature values from 20 simulated Bosch sensors.

In this experiment, MIMIC MQTT Simulator is simulating 1000 Sparkplug
sensors sending telemetry to Node-RED, graphing 10 of them in real-time.
We notice that each sensor has unique, expected telemetry.



This open-source Github repository has the flow source code to generate
these graphs.



The MQTT input node sends messages to a decode-protobuf node, which
decodes the Sparkplug messages coming from the simulated sensors and
passes them on as JSON message objects. The next node switches between
NBIRTH / DBIRTH / DDATA messages. The first 2 just count the arrivals. The
DDATA messages are graphed only for the first 10 sensors, else there would
be too many lines in the graph. Unknown messages are counted as well.
We implemented a simple message performance instrumentation by
resetting the message count every second.

Monday, September 10, 2018

Graphing arbitrary Sparkplug telemetry in Ignition

We followed this Ignition tutorial to create a real-time graph of simulated
telemetry exported from MIMIC MQTT Simulator.

This short Youtube video shows the 3 independently controlled temperature
values being graphed in the Easy Chart. We had to apply this workaround
to get the tags to be graphed.



With MIMIC, you can export arbitrary, precise, customizable telemetry in
real-time to any MQTT-based IoT platform for thousands to millions of
simulated sensors over the internet or in your network edge.

(This is a followup to our previous post.)

Wednesday, August 29, 2018

Testing MQTT Telemetry with arbitrary function

Do you need to test your IoT platforms against arbitrary telemetry data from
a variety of sensors?

You can easily to generate ANY data you want from ANY sensor you want
using MIMIC IoT Simulator.

The below screenshot shows a NODE-RED subscriber client graphing a
simulated sensor publishing data of a Sine function with configurable
amplitude. In this case we just kept expanding the amplitude every minute.



The code to achieve that is shown and consist of a total of 10 lines including
comments and the effort in minutes.

Since the amplitude is parametrized, with the same code you could have
multiple sensors sending the same function, but different amplitudes at any
point in time. Any other parameters can be made unique for each simulated
sensor.

The next screenshot shows 4 such sensors with different amplitudes
simultaneously. Any more would clutter the single graph too much.


Wednesday, August 1, 2018

Video: Monitor end-to-end latency of your IoT Application with 10,000 Sensors

This 5-minute Youtube video shows how to monitor response time to a
MQTT broker in an IoT Application with 10,000 active publishers.


This is not only important in the selection process, but also for ongoing
monitoring / troubleshooting, as outlined in our previous blog post
"IoT Sensors Need to be Managed", and this Gartner report.

We are following the testing methodology outlined in our previous post
"MQTT performance methodology using MIMIC MQTT Simulator" to
minimize the interference between the test equipment and the system
under test.

We are using the open-source Node-RED flows published in our Github
repository to measure and graph end-to-end latency between a publisher
and subscriber once a second for several brokers.

This simulates the latency between your sensors that are publishing
telemetry and your application that is consuming the telemetry.

We are using MIMIC MQTT Simulator to deploy 10,000 publisher clients
to the MessageSight broker on our intranet.

When we turn the switch on, the first 2,000 publishers are starting.
We'll time lapse this process for brevity. Notice how we are graphing
the number of active sensors sending telemetry.

The graphs show the latency for 4 brokers. Only the bottom line for
the MessageSight is being influenced with our active sensors. The
others are purely for control, to make sure our measuring and graphing
is correct. In particular, the mosquitto line should be steady, since
it is doing nothing.

The public broker graphs will be unpredictable.

Notice how the blue MessageSight line is mostly steady around 0
milliseconds.  The white mosquitto line is steady around 50 milliseconds,
and should remain so for the duration of the experiment. (It turns out
the reason for the 50 ms delay is explained here).

If your application has real-time requirements, then response time
is a vital parameter to monitor. Even if not, then response time
degradation can point to problems in your setup, specially at high
scale.

As more sensors become active, the blue latency graph becomes more
erratic. This is expected, as the broker is doing more work. In this
experiment, each sensor is only sending a small message every second,
and you can see the messages per second at the bottom of the MIMICview
graphical user interface.

There are many variables that could impact the latency: the distance
between sensors, brokers and applications, the message profile, that
is the average size of the payload and frequency of the messages, the
QOS of the messages, the topic hierarchy being published to, the number
of subscriber clients and their performance, the retention policy for
messages, and many more. Only your particular requirements would tell
whether the performance is acceptable for you.

(This is a follow-up to our earlier video).
 

Monday, July 30, 2018

Track end-to-end latency to your MQTT broker

Tracking end-to-end latency (response time) is useful in selecting your
MQTT broker, as well as monitoring your IoT platform / broker performance.

Check this real-time latency dashboard at

http://latency.iotsim.io/

which tracks end-to-end response time every second to a number of
public MQTT brokers over the internet. The local intranet mosquitto broker
measurement is for comparison.

The NODE-RED source code is at

https://github.com/gambitcomminc/nodered-mqtt-latency


Thursday, July 19, 2018

MIMIC MQTT Simulator enables rapid prototyping for Telit IoT Platform

If you want to prototype or test your solution with Telit's IoT Platform, then
you can use MIMIC MQTT Simulator to simulate a large number of sensors
and gateways.

In our initial effort we simulate a single sensor generating programmatic,
customizable, predictable, reproducible telemetry to Telit. With MIMIC what
can be done with one sensor can easily be scaled to thousands or even
millions of sensors. Complex scenarios can be setup once, and reproduced
at will thereafter. You can investigate different choices very quickly.

In the screenshot below, the light value changes randomly, but the
temperature value is changed predictably on-demand at run-time.


Friday, June 29, 2018

Testing IoT platform resilience with MIMIC IoT Simulator

With higher scalability of IoT environments in the face of outsider access
over the internet comes the requirement of proving resilience of your
IoT infrastructure. How do you know your IoT application can withstand
expected, and unexpected loads, recover from faults, and resist malicious
attacks?

Even in the absence of hackers, devices break and misbehave all the time.
How does your infrastructure handle the so-called crying baby, ie. a sensor
publishing too frequently, and how can be assured it does not interfere with
legitimate telemetry?

There are plenty of references about this requirement, eg. this post about
Azure throttling:

" We’re already dealing with serious-scale connectivity when we talk about the 
Internet of Things, and we impose the throttling limits on IoT Hub to protect 
against what otherwise looks like Denial of Service (DoS) attacks on the 
service."

or this Gartner report which contains

"... An IoT solution may be made up of hundreds or thousands of
devices. To test all of the devices in their real environments may be prohibitively
expensive or dangerous. However, you also need to ensure that your IoT 
platform and back-end systems can handle the load of all of those devices and 
correctly send and receive data as necessary."



Some platforms document their throughput testing, eg. Solace, or throttling
policies, like for Azure and Amazon.  For others this information is hard to
find.

But even if they are documented, throughput is highly dependent on your
specific application profile, and policies are subject to change at any
time, and you will likely run into some problem in production. Unless you
thoroughly tested before deployment, and can rely on QOS guarantees.

To test your specific performance requirements you will need to setup a
test that recreates your specific message patterns, connectivity patterns,
scale, message processing, etc, not just once, but continually.

MIMIC IoT Simulator is specifically designed for rapid development,
regression testing, continuous tuning, thorough training, compelling
demonstration of large-scale IoT environments.

MIMIC provides a virtual IoT lab with unlimited scale and a flexible,
programmable simulation framework to customize your performance testing.

As a quick example, we did a simple test to run a sensor at 10 messages
/ second to a broker, which ran indefinitely. Then we increased the rate
to 100 messages / second, and in multiple separate tests it was
disconnected after 20 minutes. A look at the packet exchange with
wireshark uncovered an explicit disconnect initiated at the broker.

(Original post at our blog page)

Wednesday, June 27, 2018

Virtual IoT Simulator Lab for Thingworx

In our quest for MIMIC MQTT Simulator to interoperate with a wide variety of
Internet of Things platforms and applications we have completed a scenario
where simulated sensors are feeding real-time, precisely controllable
telemetry via MQTT to PTC's Thingworx Industrial Connectivity.

With MIMIC you can simulate end-to-end large numbers of sensors and
actuators to create the complex scenarios for rapid development,
complete testing, thorough tuning and compelling demo and training
of IoT applications like Thingworx.

This 2-minute Youtube video demonstrates how a simulated sensor is
producing arbitrary and precise telemetry as charted in a hosted
Thingworx Foundation instance. This will now be extended to add
edge processing of a large number of sensors.


Friday, June 22, 2018

MIMIC MQTT Lab for Crouton Dashboard

Crouton is an open-source MQTT subscriber dashboard application based on
Node.js . It relies on its own JSON-based protocol layered on top of MQTT to
deliver dashboard features.

MIMIC MQTT Simulator is a scalable, customizable, predictable, dynamic
simulation platform designed allow rapid development / testing / deployment
/ tuning / training / demonstration of large-scale  Internet of Things
applications.

We have tailored one of our MQTT Labs to interact with Crouton through any
supported public broker. With it you can quickly explore Crouton features,
since you can customize payloads to the Crouton requirements instead of
low-level coding. Since values are dynamically changeable at runtime, you
can investigate dashboard deployment under a variety of scenarios. By
running many sensors, you can determine how scalable Crouton is.

This one-minute Youtube video shows it in action.


Wednesday, June 20, 2018

MIMIC MQTT Lab and IOTA MAM

IOTA is *the* DLT (distributed ledger technology) for the Internet of Things.
The facility for distributing telemetry from sensors is called MAM (Masked
Authentication Messaging), for details see their blog.

We were able to transfer dynamically changing MQTT telemetry generated
from our Bosch sensor simulation in our MIMIC MQTT Lab sent to the public
MQTT broker broker.hivemq.com, then through IOTA MAM to the IOTA
tangle in real-time.

You can check it yourself at this online IOTA MAM Explorer,   enter this
string

MGFJOZACIOWFEKGZARLWYPVUZGSDCAZPVPYACGCZJADGQWHFJLG9ZJGXNFIMVTCXMJUEEXUTEE9DDXHTC

where it says "enter root-address here...", optionally disable the "Pretty
print (JSON)" button, and press the search button.

This is a proof-of-concept meant to show that you can publish end-to-end
any telemetry from sensors to the IOTA tangle in real-time to be consumed
by your IoT application.

See this in action in this Youtube video.

You can also check the transactions on the ledger at https://thetangle.org/.

Friday, June 8, 2018

Free, online MIMIC MQTT Lab for Samsung Artik

We have recently released this free, online MIMIC MQTT Lab for the
Samsung Artik IoT platform and it is available on the Artik marketplace.

In under 3 minutes you get a virtual lab with a simulation of a IoT
control system based on the MQTT standard.

You can investigate:
  • authentication and access of MQTT-based sensors to Artik in your own account
  • telemetry flowing from sensors to defined device types in Artik
  • charting of that telemetry based on device manifest
  • control of actuators from defined rules in Artik
  • pathological conditions that deviate from the normal steady-state

Check out this Youtube video for the entire process from an empty Artik
account.



Thursday, May 24, 2018

MIMIC MQTT Simulator supports Sparkplug IoT standard

Cirrus Link has published an open-source IIoT (Industrial IoT) standard
called Sparkplug which specifies a framework for stateful data exchange
between things (sensors, actuators, gateways) and applications via the
MQTT protocol.

Inductive Automation's Ignition platform with their MQTT Edge module
implements this standard to supply an industrial-strength scalable
IoT application environment. The eco-system of devices that connects
to Ignition via MQTT must support the Sparkplug state machine.

MIMIC MQTT Simulator is designed to perform large-scale development /
testing / proof-of-concept / deployment or training on IoT platforms,
including Ignition. We have applied the open source Sparkplug library
to simulated sensors in MIMIC to integrate a large number of things
into Ignition.

This screenshot shows  10 simulated Bosch sensors publishing unique
temperature values to Ignition.

(we have posted a follow-up article.)

Monday, March 19, 2018

Gartner Report: A Guidance Framework for Testing IoT Solutions

Take a look at this report  by Gartner's Research VP Sean Kenefick
about  testing IoT solutions. It mentions MIMIC Simulator as a test tool.

"... An IoT solution may be made up of hundreds or thousands of devices. To
test all of the devices in their real environments may be prohibitively
expensive or dangerous. However, you also need to ensure that your IoT
platform and back-end systems can handle the load of all of those devices
and correctly send and receive data as necessary."

MIMIC IoT Simulator enables rapid development, testing, tuning,
deployment and training of large-scale, heterogeneous IoT environments.

Friday, March 9, 2018

Train on Dell OpenManage Network Manager with MIMIC Simulator

The task to set up a lab for training your staff on Dell OpenManage
Network Manager
is expensive and time-consuming. You have to
acquire the equipment, set it up, maintain it and customize it to
reproduce scenarios that you want to practice on. This has been
covered in a previous post.

This is where a network simulator like MIMIC Simulator can help.
Rather than setting up a physical lab that reproduces your production
environment, and scheduling time on this limited resource, you can
import it into MIMIC and multiply it many-fold so that each
of your operators can train on exactly the scenarios you need.



For example, these were scenarios we setup quickly with the simplest
out-of-the-box network that ships with MIMIC: it discovered the
topology shown



Since MIMIC simulates dynamic values at runtime, the performance
monitoring shows realistic values for instrumented resources:



and



We then setup some alarms and they show



To investigate who is generating the top traffic, we could pinpoint
the source of the problems.



Ask yourself how much effort this simple exercise is in your network.
Prepare your staff for contingencies before it costs you much more
in network downtime.

Thursday, March 1, 2018

Integrating Samsung Artik and MIMIC MQTT Simulator via NODE-RED

The Samsung Artik IoT platform exports an API that you can use
to programmatically retrieve and change your device information.

This REST API can then be used in a variety of ways to control your
IoT implementation. In our case, we used it to define the devices that
interact with Artik, eg. for this Youtube video .

One of the ways is through the Node-RED graphical programming
environment. This small setup




allows us to retrieve the device information for our purposes with a
couple of clicks. From an initial state as shown in the Node-RED dashboard



by entering the single Bearer Token of the Artik API, it retrieves the
UserID, registered Device Types and Devices with a couple of clicks on
the trigger nodes.



By saving this information into a file, we can define those devices
as simulated sensors in MIMIC MQTT Simulator

Wednesday, February 14, 2018

Scaling your Node-RED dashboard with MIMIC MQTT Simulator

 

 

Overview

This article shows how you can scale your Node-red visualization to a
large number of sensors. We'll go beyond the tutorials that are readily
available to apply Node-RED to a common Internet of Things (IoT)
scenario. You'll see that what works for one sensor will not work for
many, and a strategy for improving it.

We start by visualizing telemetry for one sensor. It readily shows in
the textual, gauge and chart widgets of the Node-RED dashboard. But,
when we extend it to multiple sensors, the widgets are overrun with
values. We offer one possible solution by changing the problem statement.

The accompanying Youtube video shows this in real-time.

MQTT Lab

We'll be using the MIMIC MQTT Lab accessible on the Internet for free.
20 simulated sensors are publishing MQTT telemetry to the public
iot.eclipse.org broker.  We have seen this lab in previous articles and
videos.


 

 

Single sensor telemetry

We are using Node-RED to visualize our sensor telemetry with the
dashboard plug-in, and I'll go through it in detail.



First, on the left we see a MQTT input node labelled Single sensor
subscribed to a single topic from the public iot.eclipse.org broker.


It feeds to a debug node labelled msg.payload that lets us see what is being received.



In the debug tab we see the JSON payload of our standard simulated Bosch
sensor with telemetry containing acceleration, humidity, pressure and
temperature. Let's focus on temperature.

We can change the temperature at any point in time through the Agent 
Variable Store dialog in MIMIC. Let's do this now. Later we'll visualize the
changes.


We can link the debug node to different stages of our flow and see
what happens.

Next comes a json node, which converts the JSON in the payload to a
Javascript object. Let's link the debug node and see what is in the
object. You can see that the temperature value is accessible at
msg.payload.data.temp.value .


The JSON node then feeds into the Messages node, which is a
counter which counts the messages flowing through it. It feeds into
our dashboard to display the number of messages received. We see it
slowly incrementing.



Let's now link the json node to the rest of the flows to visualize
more of the telemetry.

I'll start by linking the json node to the NOOP node. This is just
a convenience node that will later let me do some easy re-linking.
It just passes the message straight through to some other nodes.


First is the Samples counter node, it counts the number of samples
arriving. For now, this is exactly the same as the number of MQTT
messages received.



We can see it in the Samples dashboard widget that is placed in the
Telemetry group in the Home dashboard.


This is what the dashboard looks like.

The SN number text widget underneath the samples widget extracts
the serial number that was received.


The temperature text widget shows the numeric value of the temperature
received.


The gauge and chart widgets need a single value extracted out of the
payload, which is what the Extract temp node does.




Let's change the temperature again in the MIMIC lab and see the
visualization.


As soon as we click Ok in the MIMIC Agent Variable Store dialog, the
value changes from 20000 mCelsius to 10000.



Many sensors

So far so good for one sensor. But, if you want to use this for many,
it will not work. Let's try by switching from the single sensor input
node to the many sensor input node labelled Bosch sensors.


First, we see many more messages received as shown in the debug
tab. The simulated sensors in our MQTT lab are only generating 1
message every 10 seconds on average, so you can see how easy it is
to overwhelm the collector with messages.



Second, the gauge only shows the value for the last received telemetry,
and the chart now becomes a jumble of lines for the different sensors.
It's hard to discern anything.


Let's now modify this setup to make this scenario a little more useful.
Let's say all the sensors below 80 degree celsius are normal, and
we only want to visualize the sensors that are running too hot.


We can do this with the switch node. It only sends along messages
that have a temperature value higher than 80 degrees.


Let's link it in, and see what happens.

I use an Inject node named clear stats to inject an empty message to
clear the stats.


Even though further messages are being received. No samples are reported.



Let's now change one of the sensors to an abnormally high temperature.


As soon as it reports, it is visualized.



We saw how MIMIC MQTT Simulator helps in scaling your NODE-RED
visualization.