Thursday, December 1, 2022

MIMIC MQTT Lab: Test MQTT 5 support on AWS IoT Core

 AWS recently announced MQTT 5 support for AWS IoT.

We tested it in less than 5 minutes with MIMIC MQTT Lab AWS . You can do the same to make sure your
AWS IoT application uses the latest MQTT 5 features such as properties in PUBLISH messages, etc. 
Check the 2-minute Youtube video that shows the MQTT 5 CONNACK with new reason code:
CONNACK rc=0x00 Session Expiry Interval 0,Receive Maximum 100,Maximum QoS 1,Retain Available 1,Maximum Packet Size 149504,Topic Alias Maximum 8,Wildcard Subscription Available 1,Subscription Identifiers Available 0,Shared Subscription Available 1,Server Keep Alive 50



When we connect with the disallowed QOS 2, we get a new self-explanatory error code:
CONNACK rc=0x9b Reason String CONNACK:QOS 2 is not supported:861b3462-65d8-ba70-5472-63869294a5a1

and when we send a malformed PUBLISH (empty topic and topic alias):

INFO  12/02.10:53:07 - MQTT[AGT=3916] - sent CONNECT (51 bytes)
INFO  12/02.10:53:07 - MQTT[AGT=3916] - rcvd CONNACK rc=0x00 Session Expiry Interval 0,Receive Maximum 100,Maximum QoS 1,Retain Available 1,Maximum Packet Size 149504,Topic Alias Maximum 8,Wildcard Subscription Available 1,Subscription Identifiers Available 0,Shared Subscription Available 1,Server Keep Alive 50
INFO  12/02.10:53:08 - MQTT[AGT=3916] - sent PUBLISH (126 bytes)
INFO  12/02.10:53:08 - MQTT[AGT=3916] - rcvd DISCONNECT reason 0x82 (Reason String DISCONNECT:Data in packet does not conform to MQTT specification:19ec6dc1-0b50-888c-6c3e-3be26faee968)



Monday, November 21, 2022

MQTT performance testing - Best Practices

The MIMIC Simulator performance testing methodology attempts to overcome
common problems with published performance benchmarks, specially in the 
IoT arena. In this article we examine one recently published report and discuss 
how to make it better.
 
The main problem with any performance test is that the results apply only to the 
specific test scenario. If the test scenario is carefully selected, the results will be 
relevant for a wide variety of situations. If the test report is good, then the exact 
methodology is documented, so you can evaluate it, and determine whether the 
results can be useful for you. For example this report

https://www.researchgate.net/publication/354610718_Stress-Testing_MQTT_Brokers_A_Comparative_Analysis_of_Performance_Measurements

performed one test scenario for an uncommon situation of a small set (3) of high-
frequency publishers, and 15 mosquitto_sub subscribers. Plain text MQTT is only 
used in trivial situations, and there is no indication that TLS transport is measured.
Latency measurements suffer from the time synchronization problem on different 
systems.
 
Specifically, it says right at the beginning in the abstract
 
"The evaluation of the brokers is performed by a realistic test scenario"

 but then, in section 4.1.1. Evaluation Conditions:

"

Number of topics:                   3
                                    (via 3 publisher threads)
Number of publishers:               3
Number of subscribers:              15 (subscribing to all 3 topics)
Payload:                            64 bytes
Topic names used to publish large
number of messages:                 ‘topic/0’, ‘topic/1’, ‘topic/2’
Topic used to calculate latency:    ‘topic/latency

"
 
so rather than testing a large-scale environment, a small set (3) of high-frequency 
publishers, and 15 mosquitto_sub subscribers was used. In our experience, no 
recent broker has any problem with less than 1000 publishers.

Second, in section 4. the subscriber back-end is detailed:

"The subscriber machine used the “mosquitto_sub” command line
subscribers, which is an MQTT client for sub- scribing to topics and
printing the received messages. During this empirical evaluation, the
“mosquitto_sub” output was redirected to the null device (/dev/null)
"

using a the simple mosquitto_sub client which is single threaded. In addition,
the subscribers subscribe to all topics, probably the wildcard topic #. So, out of 
many code paths in the broker, the least commonly used is tested. If your 
application uses a topic hierarchy, with different subscribers subscribing to 
different topic trees, then topic matching performance needs to be exercised.

Third, while QOS 0, 1 and 2 seem to be tested, only a single payload size 
was used, and there is no indication that TLS transport is measured.

Fourth, they attempt to measure latency correctly, ie. section 4.1.2

"Latency is deļ¬ned as the time taken by a system to transmit a message
from a publisher to a subscriber
"

but their methodology is flawed, since it is almost impossible to synchronize the 
clocks on 2 separate systems to millisecond accuracy and in table 6 the latencies 
are in the 1ms range. So, the measurements rely on unknown synchronization. 
For an example of the MIMIC latency testing methodology see this blog post.




Friday, November 18, 2022

Migrating from shuttered IBM Watson IoT platform

In a previous article simulation was recognized as helping prevent IoT project failures
so prevalent in the industry.
 
With the recent announcements of the shuttering of the Google IoT Core and 
IBM Watson IoT platforms, we can suggest that MIMIC IoT Simulator can be used 
to help migrate from the obsolete IoT platforms to a new offering by:


1) running a facsimile of your environment in MIMIC

2) staging migration to the new platform

3) testing requirements at various scales to make it future-proof

before you impact your production network.

Thursday, November 17, 2022

How to scale your MQTT lab to 1000 sensors in minutes

TL;DR Money saved: $40,000. Time saved: immeasurable.

We needed to create a MQTT lab with 1000 sensors to test a subscriber client with
realistic telemetry. The open-source client tracks any key value, and alerts if any
arbitrarily pre-selected value exceeds a threshold.

We bought 1 real Shelly Plus H&T sensor for $40.

After you have configured it, it sends MQTT messages to the broker, but only 
every time the temperature and humidity changes. So, to test our application, we 
would have had to run to the refridgerator quite often to make it change the 
temperature.

As you can see from the screenshot


it sends JSON payloads, but very infrequently. In our case, after 6 minutes


So, every time we wanted a message, we needed to change the temperature.

To accelerate development, we used MIMIC.

First we just captured the messages with wireshark, recorded into MIMIC MQTT Simulator
and generated messages whenever and however we wanted. Rather than waiting for minutes, we can
send any message with any value in seconds, speeding up development time. Then we multiplied
the sensor 1000-fold, quickly reaching the required scalability at no additional cost.

This video shows the process in 2 minutes:


Money saved: $40,000. Time saved: immeasurable.

Thursday, October 20, 2022

IoT platform demo requirements


IoT platform software is complicated leading to project delays and frequent failures. To 
select the right platform for their needs, the customer has to learn and evaluate features such 
as data acquisition, analytics, graphing, database integration, alarming, etc.

Product evaluation is usually done in a couple of steps, starting with a survey leading to practical 
hands-on trial, each further filtering from the wide assortment of products. The initial evaluation needs 
to whittle down to a couple of candidates for deeper investigation before a proof-of-concept effort. 

Free brochures, videos, and other static marketing collateral can only go so far to get a customer
interested in the IoT platform software. What is needed is a more engaging, interactive format.
Thus, in order to best advertise platform features, an initial product demo has to be
  1. free - to get the foot in the door, you should not be charged. Further, the customer
    should not need to give a credit card number, or any kind of personal information
  2. immediate - the customer should get to the demo in a couple of clicks, and the instructions
    should not consist of more than a couple of lines. In total, the demo should be done in a
    couple of minutes.
  3. always on - in a global economy, the demos should be accessible 24 hours a day, 7 days
    a week.
  4. interactive - the demos should go beyond replay of videos, and consist of live interactions
    for each customer.
  5. dynamic - the customer should be able to view the features you want to show off in the
    most compelling way and be most realistic.
  6. predictable - the demo should behave the same way each time they visit, so that you can
    control it, describe it, and it matches the customer's expectations.
  7. diverse - every customer scenario is unique, requiring different platform features and
    types of sensor hardware. The demos should showcase the various features that the customer wants.
  8. scalable - scaling up an IoT application is the hardest part. The demo should prove
    how the platform enables large scale applications.


For examples of demos with MIMIC IoT Simulator that we have provided to our partners 

see also this blog post .



Tuesday, September 13, 2022

MIMIC SNMP Simulator and neteXpose DNA


MIMIC SNMP Simulator can simulate details of each device to the finest level. In 
this integration, netExpose DNA discovered the chassis details of a simulated 
Alcatel switch, allowing them to test their management application with all sorts
of device scenarios.




Friday, August 26, 2022

Migrating to Google IoT Core alternatives


If you are contemplating migrating from Google IoT before it retires in a year, you 
might want to use MIMIC MQTT Simulator and its SaaS offering for Google IoT 
https://mqttlab.iotsim.io/gcp to design / develop / test your migration strategy.
 
MIMIC MQTT Simulator allows you to simulate a large number of your assets that 
connect to Google IoT Core, then migrate them to the alternative of your choice.
You can duplicate your current IoT environment in MIMIC to make sure the new 
IoT platform handles your specific requirements.