Emergency Shutoff scenario: Losant IoT Platform

In previous posts we examined how different platforms, such as
Losant, Samsung Artik, IBM Watson,  allow implementing a real-time,
dynamic, bi-directional IoT control system: imagine a manufacturing floor
with machines heating up, and a cooling system to cool them down. A sensor
on each machine reports current temperature to the platform, which
if high enough, turns on the cooling system, at which point machines
cool down, and when temperatures are low enough turns the A/C off.

This is all fine in a steady state scenario, where everything behaves
as it should. But when does equipment not break ever?

We used MIMIC MQTT Simulator to create a pathological scenario to make
one of the simulated machines misbehave, and keep heating up even after
the A/C is on.

An emergency shutoff rule on the platform kicks in to prevent the machine

from overheating and shuts off everything, or just the machine in question.

Hope you have tested that in your control system...

This 2-minute Youtube video shows this in action with the Losant IoT platform.

Update 6/12/2019: This more recent video shows more sensors and Losant
dashboard features. 

Real-time simulated control system: Losant

Our minimal standard for a scalable, real-time, dynamic, bi-directional
demonstration of an IoT platform is:

- multiple devices, at least dozens, preferably thousands (it is easy to
provision a few devices, not so easy to provision many);

- arbitrary telemetry (for heterogeneity, the devices dictate the payload,
not the platform);

- easy rules to control your environment, preferably with no programming;

- real-time processing, preferably at the edge;

- bi-directional flow: telemetry generated by sensor-type devices, commands
accepted by actuator-type devices

For this we have come up with a minimal pattern of a control system
like that of a data center with equipment that heats up, causing sensors
to generate telemetry with rising temperature, and a cooling system
controlled by an actuator, to cool them down. The control system receives
temperature values from many sensors, which if any of them cross a high
threshold, turns the cooling system on. Once the equipment cools down
beyond a low threshold, it triggers the actuator to turn off. In the steady
state, the temperatures reported will go up and down predictably.

We have implemented this demo on third-party platforms, like
Samsung Artik  or IBM Watson.

In this post we detail the demo setup for the Losant platform as shown
in this 1-minute Youtube video:

- we setup 2 device recipes, one for our sensors and one for the actuator;

- the payload cannot be arbitrary, but must conform to Losant's template;

- the sensors generate light and temperature data;

- the actuator generates on/off state data, and accepts on/off command;

- once we configured the authentication parameters to our application
context, the devices were able to interoperate with Losant

This screenshot shows the telemetry generated by one of the sensors.

The light value changes randomly, like that of a flickering light. The
temperature value rises and falls within the thresholds. The following
screenshot shows the workflows that control the actuators:

Once the steady-state works, we can implement and test rules that handle
pathological scenarios, such as the equipment not cooling down.


Update 6/12/2019: This more recent video shows more sensors and Losant
dashboard features. 

How to test the accuracy of your IoT applications

While implementing your MQTT-based application's back-end processing,
such as real-time archiving, analytics, edge-processing, graphing, how
do you  verify that it will store and process all received telemetry messages,
with no messages missing, no extra messages, no bytes altered, in the
correct order? Furthermore, it is really hard to make sure that it works at
required scale and speeds, eg. at millisecond granularity.

You need to come up with a performance test that goes beyond simple
load testing and handles each of the test requirements (message integrity,
sequencing and frequency).

To test this case, you can apply the following features found in MIMIC MQTT Simulator:

1. copy the existing Bosch XDK simulation as detailed in an earlier post .
This ships with MIMIC, and generates a realistic JSON payload similar to

{
 "sn":"20:19:AB:F4:00:01",
 "data":{
  "acc":{"x":26,"y":32,"z":1012,"unit":"mG"},
  "gyro":{"x":1220,"y":-6835,"z":-2319,"unit":"mdeg/s"},
  "mag":{"x":40,"y":1,"z":-4,"unit":"uT"},
  "light":{"value":999,"unit":"mLux"},
  "temp":{"value":"100","unit":"mCelsius"},
  "pressure":{"value":98897,"unit":"Pascal"},
  "humidity":{"value":39,"unit":"%rh"}
 }
}


By copying it to a new simulation, you can now customize it without
disturbing the original simulation.

2. change the frequency of the generated messages to 1 msec;

3. change the number of messages to exactly 1000, not one more, not one
less;

4.  change the JSON payload function to return light values in sequence;

5. change the QOS to 2.

This is a total of about 5 lines of change to the existing simulation
configuration.

Once you configure a MIMIC sensor with this simulation, it generates the
1000 messages, and you are able to verify that they are all there in the
correct order. This entire exercise should only take about 15 minutes,
as it did when we tried it


Follow-up tests can be to scale up to run multiple sensors (as many as
required), as detailed in this previous post, and for longer periods of time
(see also this post), with payloads that resemble those in your application
(see this previous post).

Arbitrary telemetry to ELK/Logstash/Kibana

In a previous post we saw how MIMIC MQTT Simulator was used to
generate arbitrary telemetry to the Ignition IoT platform.

In this post we experimented with sending telemetry data with
values in a sine-wave pattern to the MQTT plugin for ELK (Elasticsearch,
Logstash, Kibana).


The Bosch sensor simulation provided with the MIMIC MQTT module
gives rolling motion indications commonly seen in masthead devices,
including wind sensor, wind transducer, and other weather devices.

In the simulation script, the sine wave or a constant value is easily
assigned to accelerometer or gyro values in the lower left window text
editor.

In the ELK illustration, the search engine based application draws the
sine wave as a heat map between peaks and troughs in the upper right
window as steady air current is detected. For comparison, a prominent,
near constant value of banking angle is shown in the lower right window
indicating a calm, windless air condition.

Simulated security cameras to Domoticz

The FOSS (free open-source software) Domoticz home automation system
supports the MQTT IoT protocol to interoperate with a variety of devices
as detailed on their Wiki page.

We setup the MQTT gateway to monitor a Dahua IP camera we got at
Home Depot as detailed in their hardware compatibility page and recorded
the MQTT session.

Then we duplicated this simulation manyfold with MIMIC MQTT Simulator,
and were able to simulate VideoMotion events from any number of simulated
cameras as shown in the screenshot below.

Can you tell which is real and which is simulated?

This 1-minute Youtube video shows which we are manipulating in real-time.
You can generate any video motion events in any order with any frequency
to test your security policies of your IoT application.

Fire-and-forget vs. complete control of MQTT message generation

Free, open-source MQTT performance tools follow the "fire-and-forget"
paradigm: you configure message generation in advance, then fire the
tool to perform the load testing.

This is fine for static scenarios, but when are real-world scenarios
static? Message characteristics change all the time: sensors connect and
disconnect, you get Denial-of-Service (DOS) attacks or mal-functioning
sensors (the "crying baby" scenario). How will you test that your
application load balancer (ALB) throttles misbehaving connections in
real-time?

MIMIC MQTT Simulator lets you change any message characteristic
(frequency, topic, payload, QOS, etc) at any point in real-time, not just for
one sensor, but also for many thousands if not millions. This allows you to
create anomaly scenarios that simpler tools don't allow you to.

In this 4-minute Youtube video we show our open-source "MQTT Topic
Statistics" mqtt-stats  subscriber client connected to a IBM MessageSight
broker instance. MIMIC MQTT Simulator is used to generate different
message loads dynamically during the video, verified by mqtt-stats.

In the simulator we first start 10 simulated Sparkplug sensors. Each
sensor publishes to 3 topics (NBIRTH, DBIRTH and DDATA) in order to generate
telemetry. By default each is sending 1 DDATA message per second.

In the steady-state (when all sensors are registered) mqtt-stats shows the 30
topics and 10 active topics (see in video). We show the 10 NBIRTH, DBIRTH,
and DDATA topics as defined in the Sparkplug specification. At this point we
can clear out the topics with File -> New to only show the active DDATA
topics. All sensors are showing a rate of 1 message / second.

Now we can change message rates dynamically, we pick 2 sensors to send 5, then
10, then 20 messages / second. Then we pick 3 different sensors and change their
frequency to 50 / second. And yet a different one with 100 / second. Then we reset
all 10 back to 1 message / second.  What we did interactively you can script
programmatically with the MIMIC API in 6 languages.

The rest of the video show a total of 1000 sensors being monitored by the
mqtt-stats client.

Node-RED and 1000 simulated Sparkplug B sensors

In a previous post we discussed a technique to reduce the number
of tracked telemetry in the Node-RED IoT application, which only graphed
interesting temperature values from 20 simulated Bosch sensors.

In this experiment, MIMIC MQTT Simulator is simulating 1000 Sparkplug
sensors sending telemetry to Node-RED, graphing 10 of them in real-time.
We notice that each sensor has unique, expected telemetry.

This open-source Github repository has the flow source code to generate
these graphs.

The MQTT input node sends messages to a decode-protobuf node, which
decodes the Sparkplug messages coming from the simulated sensors and
passes them on as JSON message objects. The next node switches between
NBIRTH / DBIRTH / DDATA messages. The first 2 just count the arrivals. The
DDATA messages are graphed only for the first 10 sensors, else there would
be too many lines in the graph. Unknown messages are counted as well.
We implemented a simple message performance instrumentation by
resetting the message count every second.