IoT applications share a common set of requirements such as flexible and scalable data management in addition to device and event management. An IoT platform has to fulfill these key requirements to provide a good foundation for IoT applications. From a data management perspective, devices and events are two distinct types of data that an IoT application platform must be able to manage. Additionally, the data management strategy for IoT applications has to answer the challenges that device and event data is presenting (e.g. data volumes, data schemas).
From self-contained sensors to systems of systems
If you look at IoT use cases, you will see that there are two central data entities:
- Devices: Most IoT applications are built around some kind of physical device. The types of devices can vary greatly, from shopping carts to hand-held power tools to large trucks.
- Events: Managing events generated by the devices is critical to most IoT applications. These events can range from technical events like a machine failure to a customer’s expression of a buying interest at a specific retail location.
Efficient management of devices and events is depending on the creation of independent hierarchies, with communication between each layer. I see these three hierarchy levels:
- Typically small, self-contained hardware elements like sensors, controllers, etc.
- Devices are larger, more powerful entities, like a machine, a vehicle, or a power plant; a device can connect to multiple hardware elements through a local network, using specialized software drivers for different types of hardware. The device typically connects to the backend application platform via a mobile carrier network (GSM, EDGE) or via a direct internet connection.
- Systems of systems are multiple devices and can be grouped together to form what is called a system of systems, e.g. an assembly line consisting of multiple machines, a power grid consisting of multiple power plants, etc.
In my opinion, the benefit of using well-defined hierarchies of devices in an IoT application is that the underlying application platform (also often called software) can provide many required features out-of-the-box, regardless of the specific type of asset. These features include:
- Device communication: Efficient communication between the different hierarchy levels, including events sent from remote devices to the backend, as well as operations initiated from the backend and sent to and executed by the remote devices.
- Management backend: Includes a central device database, data analytics, secure event management, business rules execution, business process execution, and a management console for technical device administrators.
Focus on data management strategies
From a data management point of view, an IoT platform must therefore manage two distinct types of data in the backend:
- Device instances and hierarchy: Each device becomes an individual entry in the central device database, including information about its position in the device hierarchy, as well as additional attributes and properties. The information about the device hierarchy is critical in order to enable efficient communication with all devices.
- Events related to individual devices: Each device entry contains a complete recording of all events related to the device, including technical events (e.g. failure notifications, etc.), as well as business events (e.g. a vehicle is crossing a geo-fence).
Device and event data is imposing huge challenges on the data management strategies for IoT applications. Let me highlight three for you:
- Data volumes: Depending on time intervals and numbers of devices, data volumes for the event history can become overwhelmingly large. Imagine ten million smart meters submitting a meter reading every 15 minutes for 20 years.
- Data schemas: In addition to structured data such as meter readings, many IoT applications also have to deal with semi-structured and un-structured data. For example, an event might include additional information such as a digital image taken to document the event, or an un-structured reading in an unknown technical format submitted by a specialized sensor.
- Data schema evolution: Especially in large IoT deployments it will be impossible to ensure that all devices in the field will always have the same configurations or versions of hardware and/or software. Consequently, a data management solution must be able to handle different versions of device and event data in parallel. In addition, there is often the need to blend device and event data with backend enterprise systems to enable analytics and create automated processes. All of these systems can change quickly, so it is vital to be able to seamlessly evolve the schema.
If you want to get more information on how you can benefit from NoSQL-based data management, please download our “IoT and Big Data” whitepaper.