MQTT 5 Testing

MQTT 5.0 is the next version of the ISO standard MQTT 3.1.1 Internet
of Things protocol.

We used the latest version of MIMIC MQTT Simulator to test several MQTT
brokers with MQTT 5 support, and in a short time have discovered some
differences in protocol behavior between them.

Test 1 - CONNECT with Maximum Packet Size set to low value

In the case of 0, one broker disconnects immediately, and the other sends a
CONNACK with Reason Code. In this case, the spec clearly says

It is a Protocol Error to include the Maximum Packet Size more than once, or for the value to be set to zero.

Both behaviors are legal, since protocol errors must result in disconnect, but
at the discretion of the server it may send a CONNACK with reason code, as
detailed here.

Curiously, if the value is 10, again one broker disconnects immediately, but
the other does not send anything, presumably because the CONNACK would
have violated the limit. This results in a timeout at the client.

Test 2 - PUBLISH with empty topic name and no topic alias

In this case, one broker disconnects and the other sends PUBACK with
Reason Code. In this case, the spec clearly says

It is a Protocol Error if the Topic Name is zero length and there is no Topic Alias.

Protocol errors must result in disconnect as detailed here.

Test 3 - PUBLISH to wildcard topic

For example, to handle the error of a PUBLISH with QOS = 1 to an illegal
wildcard topic, one broker returns a PUBACK with the Reason Code of 0x90,
whereas another returns a DISCONNECT with that Reason Code. This
behavior is expected with QOS 0, but since it is not defined as a protocol
error, for QOS 1 and 2 the PUBACK and PUBREC response should be used.

This is what our logs show:


INFO  02/06.10:58:25 - MQTT[AGT=1] - sent PUBLISH (110 bytes)
INFO  02/06.10:58:25 - MQTT[AGT=1] - rcvd PUBREC rc=0x90 Reason String The topic is not valid: ...

vs.

INFO  02/06.11:01:25 - MQTT[AGT=1] - sent PUBLISH (110 bytes)
INFO  02/06.11:01:25 - MQTT[AGT=1] - rcvd DISCONNECT reason 0x90 (Reason String PUBLISH with wildcard character (#/+) was sent.)

Test 4 - PUBLISH with Payload Format property

For a PUBLISH with property Payload Format set, the spec says

A Server MUST send the Payload Format Indicator unaltered to all
subscribers receiving the Application Message [MQTT-3.3.2-4]

One of the brokers discards the Payload Format Indicator property if it is 0.
If this property is missing, it can be assumed to be 0. While not breaking
anything, this is technically incorrect.

Conclusion

Since MQTT 5 is so recent, you need to test many features of your selected
MQTT brokers and IoT applications with different scenarios to mitigate
costly surprises before deployment. MIMIC MQTT Simulator can really help
with that.

Real-time telemetry throttling demo for MQTT.Cool

MQTT.Cool is a web gateway that you put in front of any MQTT broker to
boost its security, performance, and architecture. It allows IoT applications
to connect to your MQTT broker from anywhere on the Internet, even behind
the strictest corporate firewalls and proxies, without sacrificing security.
MQTT.Cool will automatically throttle the data flow for each client, to adapt
to any network congestion. Simply put, MQTT.Cool's unique feature
of optimizing the real-time data flow, rather than forwarding it as a dumb
pipe, is one of the biggest advantages for a Customer's application.

What better way to demonstrate its features than to create an online
demo page that lets you visualize and interact with a sample client?
The open-source client lets you control throttling globally or individually
to see the effect on the data.

The real-time data is generated by MIMIC MQTT Simulator with arbitrary,
customizable, scalable, predictable telemetry. We deployed 10 sensors on
our MQTT Lab to generate a variety of telemetry patterns that is throttled
by MQTT.Cool.

400,000 MQTT connections to IBM MessageSight for real-time performance testing

As part of our MQTT broker and IoT Platform testing program, we had
previously connected 100,000 simulated connections to IBM MessageSight.

Now, with the latest MIMIC MQTT Simulator 18.10 running in Docker
instances on 2 VirtualBox VMs (16GB RAM and 4 CPUs each), we managed
to start 200,000 simulated sensors connecting to IBM MessageSight 2.0 in
under 15 minutes.

Contrary to the prevailing testing methodologies, where either dummy
MQTT sessions are opened (not doing anything other than maintaining
the session with PING responses), or the vast majority of clients subscribe
rather than publish, each of our clients simulates a sensor publishing
telemetry. To test the actual performance of the topic switching, you can
now add your expected number of subscriber on the expected topic
hierarchies.

With this fast turnaround we can now do timely investigation of the
real-time performance impact of other variables such as:

1) unencrypted vs. TLS

2) varying message frequency

3) varying payload sizes with arbitrary content (JSON, binary, etc)

4) different QOS levels

5) varying topic hierarchies

6) varying subscription levels

7) impact on end-to-end latency

As always, each of the 200,000 sensors can be controlled individually or in
a group in real-time to affect payload contents, payload frequency,
connection status to verify handling of pathological conditions by the
IoT platform.

Update 1/8/2019: We have achieved 300,000 simultaneous connections.
Do you see where this is going?

Update 1/29/2019: We have achieved 400,000 simultaneous connections
on the newly released MessageSight 5.

Emergency Shutoff scenario: Losant IoT Platform

In previous posts we examined how different platforms, such as
Losant, Samsung Artik, IBM Watson,  allow implementing a real-time,
dynamic, bi-directional IoT control system: imagine a manufacturing floor
with machines heating up, and a cooling system to cool them down. A sensor
on each machine reports current temperature to the platform, which
if high enough, turns on the cooling system, at which point machines
cool down, and when temperatures are low enough turns the A/C off.

This is all fine in a steady state scenario, where everything behaves
as it should. But when does equipment not break ever?

We used MIMIC MQTT Simulator to create a pathological scenario to make
one of the simulated machines misbehave, and keep heating up even after
the A/C is on.

An emergency shutoff rule on the platform kicks in to prevent the machine

from overheating and shuts off everything, or just the machine in question.

Hope you have tested that in your control system...

This 2-minute Youtube video shows this in action with the Losant IoT platform.

Real-time simulated control system: Losant

Our minimal standard for a scalable, real-time, dynamic, bi-directional
demonstration of an IoT platform is:

- multiple devices, at least dozens, preferably thousands (it is easy to
provision a few devices, not so easy to provision many);

- arbitrary telemetry (for heterogeneity, the devices dictate the payload,
not the platform);

- easy rules to control your environment, preferably with no programming;

- real-time processing, preferably at the edge;

- bi-directional flow: telemetry generated by sensor-type devices, commands
accepted by actuator-type devices

For this we have come up with a minimal pattern of a control system
like that of a data center with equipment that heats up, causing sensors
to generate telemetry with rising temperature, and a cooling system
controlled by an actuator, to cool them down. The control system receives
temperature values from many sensors, which if any of them cross a high
threshold, turns the cooling system on. Once the equipment cools down
beyond a low threshold, it triggers the actuator to turn off. In the steady
state, the temperatures reported will go up and down predictably.

We have implemented this demo on third-party platforms, like
Samsung Artik  or IBM Watson.

In this post we detail the demo setup for the Losant platform as shown
in this 1-minute Youtube video:

- we setup 2 device recipes, one for our sensors and one for the actuator;

- the payload cannot be arbitrary, but must conform to Losant's template;

- the sensors generate light and temperature data;

- the actuator generates on/off state data, and accepts on/off command;

- once we configured the authentication parameters to our application
context, the devices were able to interoperate with Losant

This screenshot shows the telemetry generated by one of the sensors.

The light value changes randomly, like that of a flickering light. The
temperature value rises and falls within the thresholds. The following
screenshot shows the workflows that control the actuators:

Once the steady-state works, we can implement and test rules that handle
pathological scenarios, such as the equipment not cooling down.

How to test the accuracy of your IoT applications

While implementing your MQTT-based application's back-end processing,
such as real-time archiving, analytics, edge-processing, graphing, how
do you  verify that it will store and process all received telemetry messages,
with no messages missing, no extra messages, no bytes altered, in the
correct order? Furthermore, it is really hard to make sure that it works at
required scale and speeds, eg. at millisecond granularity.

You need to come up with a performance test that goes beyond simple
load testing and handles each of the test requirements (message integrity,
sequencing and frequency).

To test this case, you can apply the following features found in MIMIC MQTT Simulator:

1. copy the existing Bosch XDK simulation as detailed in an earlier post .
This ships with MIMIC, and generates a realistic JSON payload similar to

{
 "sn":"20:19:AB:F4:00:01",
 "data":{
  "acc":{"x":26,"y":32,"z":1012,"unit":"mG"},
  "gyro":{"x":1220,"y":-6835,"z":-2319,"unit":"mdeg/s"},
  "mag":{"x":40,"y":1,"z":-4,"unit":"uT"},
  "light":{"value":999,"unit":"mLux"},
  "temp":{"value":"100","unit":"mCelsius"},
  "pressure":{"value":98897,"unit":"Pascal"},
  "humidity":{"value":39,"unit":"%rh"}
 }
}


By copying it to a new simulation, you can now customize it without
disturbing the original simulation.

2. change the frequency of the generated messages to 1 msec;

3. change the number of messages to exactly 1000, not one more, not one
less;

4.  change the JSON payload function to return light values in sequence;

5. change the QOS to 2.

This is a total of about 5 lines of change to the existing simulation
configuration.

Once you configure a MIMIC sensor with this simulation, it generates the
1000 messages, and you are able to verify that they are all there in the
correct order. This entire exercise should only take about 15 minutes,
as it did when we tried it


Follow-up tests can be to scale up to run multiple sensors (as many as
required), as detailed in this previous post, and for longer periods of time
(see also this post), with payloads that resemble those in your application
(see this previous post).

Arbitrary telemetry to ELK/Logstash/Kibana

In a previous post we saw how MIMIC MQTT Simulator was used to
generate arbitrary telemetry to the Ignition IoT platform.

In this post we experimented with sending telemetry data with
values in a sine-wave pattern to the MQTT plugin for ELK (Elasticsearch,
Logstash, Kibana).


The Bosch sensor simulation provided with the MIMIC MQTT module
gives rolling motion indications commonly seen in masthead devices,
including wind sensor, wind transducer, and other weather devices.

In the simulation script, the sine wave or a constant value is easily
assigned to accelerometer or gyro values in the lower left window text
editor.

In the ELK illustration, the search engine based application draws the
sine wave as a heat map between peaks and troughs in the upper right
window as steady air current is detected. For comparison, a prominent,
near constant value of banking angle is shown in the lower right window
indicating a calm, windless air condition.