In a technical environment without a global standard, IoT device manufacturers, integrators, and platform providers are facing difficulties keeping up with the massive amounts of different payload formats, APIs, and proprietary protocols.
The open source project Eclipse Vorto addresses this problem by providing cloud-based editors to abstract vendor-specific device payloads as re-usable Vorto Function Blocks. These are then aggregated to describe a whole device in the form of a Vorto Information Model. Information Models and Function Blocks are written in vortolang, a simple grammar to define interfaces between a physical device and its digital twin counterpart. IoT solutions communicate with physical devices only via these abstract Function Blocks and their co-related data schema. In this way, IoT solutions become de-coupled from the plethora of different device data formats, APIs, and encodings. But how to convert the device data to these abstract Function Block interfaces? Simple: with so-called Vorto Mapping Specifications, which contain all the necessary instructions in order to harmonize device-specific payloads.
Payload normalization in general
Normalization of data can be handled on different system nodes, depending on the requirements of the IoT use case. This separation allows entities to keep full control over where they transform their proprietary into normalized data.
- Normalization on a device node
In this scenario, the IoT device is a smart device, providing added services utilizing the onboard sensors. An example is a Bosch smart oven that provides built-in analytics on normalized data; data that could be aggregated in a cloud data lake.
- Normalization on a gateway node
A gateway connecting multiple sensors or devices by various protocol drivers (BLE, GPIO, etc), harmonizes the device data in order to provide gateway functionality, such as analytics features or other business-specific functionality.
- Normalization on an IoT platform node
Very much like in a gateway node, a cloud IoT platform normalizes incoming telemetry device data from various protocol adapters (MQTT, CoAP, etc) in order to provide value added services to northbound IoT solutions. Examples are device management or data analytics features.
- Normalization on an application node
An application would normalize data in order to stay agnostic of supported devices and stay more focused on the application-specific business features, rather than implementing technical, device-specific data decoders.
Sometimes a device quite simply is not capable of carrying out payload mapping. But there are also other factors that can play a role. Think about the limitations that come with sending data to the back-end via a mobile connection. Often, you want to keep the amount of data you are transmitting to a minimum. The problem is that after mapping, the device payload is more bloated due to the normalization and conversion from binary data and similar information. To save on bandwidth, it can therefore make more sense to perform payload mapping on a different system node; on the platform for instance.
When payload mapping is not performed on the device, a different entity is needed that can execute this step. You have to use a mapping engine, which enables payloads to be transformed on either an intermediary – such as a gateway – the platform, or directly in the application.
As a German computer science student, I have taken on assignments in 3 different departments at Bosch over the last one and a half years. I’m particularly interested in Open Source and EduTech technologies. I believe that the IoT and automation have a huge potential to both change and improve the way people live, work, and enjoy life. A passionate learner and developer, I’m always keen to learn new skills and tooling. In addition to my regular work, I have built and now maintain the world’s largest free open-source automation bot for Instagram. In my free time, I enjoy climbing outings with friends and travel in foreign countries.
How Eclipse Vorto addresses normalization
Eclipse Vorto offers a runtime library that can be configured via a Vorto Information Model for the device. It includes a mapping specification that enriches the Information Model with device-specific payload conversion rules. In practice, the runtime library takes the device payload as an input and then outputs the converted and normalized payload, by applying the mapping specification.
The mapping specification is managed and version controlled in the Vorto Repository, alongside the Information Model for the device. This makes it possible to re-use the mapping specification for other use cases, regardless of whether the normalization takes place on the device, the gateway, the platform, or the application node. The mapping library is currently supported on the Java and Node.js platform. Read more about the Vorto Payload Mapping library.
Example: Normalizing industry data using Eclipse Vorto
In order to clarify the concepts described earlier, we will look at a specific example of how the Vorto Information Model and the mappings are used.
In our case, we send CSV data from a permanent-magnet synchronous motor (PMSM) to an Eclipse Hono MQTT connector.
The Vorto Payload normalization middleware consumes the data from Eclipse Hono, pipes it through the Vorto Payload Mapping Engine, and exposes the normalized device data as an AMQP topic. Any AMQP 1.0 subscriber can now retrieve harmonized device data and process it regardless of the connected device using a digital twin solution such as Bosch IoT Things.
In our example, we use Eclipse Ditto, an open source digital twin service, that receives the normalized Eclipse Vorto data and stores it. The Vorto Dashboard then requests the data from the Eclipse Ditto digital twin API and renders the data nicely using pre-defined Eclipse Vorto compliant UI widgets.
If you want to see how the whole pipeline of mapping different device payloads with the Vorto Mapping Engine works, feel free to head over to the Vorto Dashboard Demo.
This article only gives a brief introduction to Payload Mappings. If you want to go through the whole process of setting up this pipeline for your own use cases, you can use our detailed tutorial to do so. It will explain how to map your custom device payloads step by step.