Is IoT Device Shadow Right for You?

When building an IoT platform, AWS IoT Core offers many out of the box services to help get you up and running faster. One of those services is the IoT Device Shadow.

Amazon’s description of shadow IoT is:

The AWS IoT Device Shadow service adds shadows to AWS IoT thing objects. Shadows can make a device’s state available to apps and other services whether the device is connected to AWS IoT or not. AWS IoT thing objects can have multiple named shadows so that your IoT solution has more options for connection your devices to other apps and services

There is more than one solution to any problem, however, and just because something is easier doesn’t necessarily mean it is the right solution for your particular application. In this post, we explore the pros and cons of utilizing the IoT Device Shadow and compare it to an alternative structure using DynamoDB to store the state of a device.

Use Of IoT Device Shadow AWS

A typical use case for the IoT Core Device Shadow involves using it to start a firmware update. When a device comes online, the physical state of the device is pushed to the reported state of the IoT Device Shadow:

A client (such as a mobile app or web dashboard) can see that the device is online because the IoT Device Shadow exists and the reported state is ‘online’. The IoT Device Shadow data can be accessed through a custom API or through the AWS SDK. When the client wants to upgrade the firmware, it sets the desired state of the shadow to the firmware version it would like to upgrade to. The difference between the reported ‘firmwareVersion’ and the desired ‘firmwareVersion’ causes a delta to generate, and a message to be sent on the $aws/things/<thingName>/shadow/update/delta managed MQTT topic¹:

This delta message is consumed by the device, and it takes action to have its reported state match the desired state (in this case, update the firmware and report the new version).

DynamoDB as a Store of Device State

The IoT Core Device Shadow is a great service and a way to get a lot of functionality with minimal development effort. However, using the device shadow frequently or having large fleets that use it several times per day can cause costs to rise significantly over time.² We can design an architecture using DynamoDB, API Gateway, and Lambda that gives us similar functionality to that of the IoT Device Shadow but is more cost effective.

The mechanics of a firmware update are slightly different in this alternative approach. When the device comes online, it publishes a JSON representation of its state onto a non-shadow IoT Core MQTT topic. The payload will need to contain an attribute that indicates the source of the message (“source”: “device”). An IoT Rule on this topic does an update operation which places the payload into the DynamoDB table. The partition key of this table is the device’s unique serial number. When the update operation occurs, this generates a DynamoDB Stream record with the eventName equal to INSERT or MODIFY (depending on if the device settings already exist in the table). The stream processing lambda does not forward this message to the device on an MQTT topic because the source of the change was the device. To initiate the firmware update, an external client sends a post request through API Gateway, which updates the firmware version attribute in Dynamo for the targeted device. This generates a stream record of type UPDATE with source attribute = ‘api’. The stream processing lambda forwards this message to the device on an MQTT topic because the source of the change is not the device.

Pros and cons of each approach

So which should you choose? Below are a few things you might want to consider when weighing these options:

Pros of IoT Core

• Out-of-the-box operation with no extra resources or custom code required
• Good visibility on data flow to and from the device through the shadow. Using the shadow makes it easy to see when something went wrong, as the desired and reported states will not sync up if a delta message is not processed by the device
• Less complexity in your system architecture which can help avoid introducing bugs

Cons of IoT Core

• Any operation involving the IoT Device Shadow is priced the same at $1.25 per million operations. A million operations may sound like a lot, but in the shadow example workflow above, provisioning a device and updating it’s firmware consumes 5 operations. If the state of your device is frequently changing and you have a large number of IOT devices, the shadow can become expensive quickly
• The IoT Device Shadow is only available in one availability zone in the region your system is deployed in. This is a vulnerability because any outage of that availability zone would leave you blind to the state of your fleet.³

Pros of DynamoDB approach

• For an architecture with DynamoDB acting as the source of truth on the state of device, the cost will be about 25% less than using the shadow. The cost for MQTT messages is $1.00 per million, DynamoDB writes are $1.25 per million, and the rule execution adds another $0.15 for a total of $2.40 per million state updates. But reads are only $0.25 per million, making the cost of 1 million reads and 1 million writes 25% cheaper using this architecture ($2.65 per million vs $3.50 per million). Storage and lambda execution costs should be negligible with 25Gb free storage per month in DynamoDB and lambda executions happening only on external state updates, not device initiated ones.
• You can reduce the cost of utilizing this alternative approach further by using IoT Basic Ingest
• DynamoDB acts as a central location for your fleet’s device configuration and state. IoT IoT Device Shadows aren’t consolidated which can make it more time consuming to inspect multiple IoT Device Shadows at once.
• DynamoDB offers an intuitive user interface. Developers may not value this feature all that much, but non-developer data clients benefit from seeing the data represented in a familiar row-column table layout instead of a JSON object
• The device data will be replicated across three availability zones by default when stored in DynamoDB, which gives you peace of mind as a solutions architect

Cons of DynamoDB Approach

• It will take more work on behalf of your developers to implement this approach and there will be more infrastructure to support
• More time spent on developing a coherent MQTT topic design, as data will be written on custom topic names instead of the AWS supported IoT Device Shadow topics
• Less visibility into potential systemic problems. Keeping the real state of the device in sync with the DynamoDB representation is a critical concern in this architecture. There could be further improvement on this design to implement a device acknowledgement mechanism that confirms that external state changes are actually reflected on the device.⁴

Conclusion

Overall, using the IoT Core IoT Device Shadow is extremely simple and brings a lot of utility to the table. With careful design, the costs associated with using the IoT Device Shadow can be managed. But if IoT Device Shadow cost calculations or the lack of multi-availability zone redundancy are a cause for concern for you, it is good to know that there are other solutions available that can meet most of your needs, keep costs under control, and offer some built-in benefits. For more AWS architecture inspiration, check out the Cloud Prose Blog by Trek10 here.

¹See https://docs.aws.amazon.com/iot/latest/developerguide/device-shadow-mqtt.html for more details

²See device shadow pricing structure: https://aws.amazon.com/iot-core/pricing/

³https://docs.aws.amazon.com/iot/latest/developerguide/disaster-recovery-resiliency.html

⁴An acknowledgement mechanism would be a way for the device to indicate a state update request from an external client was implemented in the firmware. This could involve a giving each client generated change a unique ID, and when the firmware receives this desired state change on an MQTT topic, it would respond on a dedicated acknowledgement MQTT topic with a simple payload (ex: {‘status’: True, ‘changeId’: <unique change id>}). An IoT Rule on this endpoint could then update the device settings dynamo table with an attribute that confirms the change was synced successfully by the device.