IoT Offerings in Market – How to integrate IoT Devices with a Unified Data Platform on GCP 

By Mandar Chaudhari | @intelia | August 22

Introduction

The Internet of Things (IoT) describes the network of physical objects (things) that are embedded with sensors, software, and other technologies for the purpose of connecting and exchanging data with other devices and systems over the internet without user involvement. Examples of these devices or things can be a person with a heart monitor implant, a farm animal with a biochip transponder, an automobile that has built-in sensors to alert the driver, a smart thermostat in your office or a smart device mounted on the power lines to monitor surroundings of power lines and towers.

 

Overview of IoT Use Cases

IoT applications and devices are used in all major industries in the world today. This includes Manufacturing, Automotive, Transportation and Logistics, Retail, Public Sector, Healthcare, Energy sector and many more. Organizations in a variety of industries are using IoT to operate more efficiently, better understand customers to deliver enhanced customer service, improve decision-making and increase the value of the business. One of the use cases where intelia was part of with one of our energy sector clients was to connect and collect IoT data from devices mounted over the power lines. These Dynamic Line Monitor (DLM) devices captures sensors data related to ambient temperature, heartbeat, vegetation distance from power lines, sag of the lines between the towers etc and send this telemetry data to the Google cloud on short interval of time using Google’s IoT core service. Raw telemetry data was collected in GCP BigQuery (Datawarehouse service) which is then further processed and analysed to derive meaningful insights and live dashboards for key stakeholders.

How IoT works and Why are IoT data treated differently?

An IoT ecosystem consists of web-enabled smart devices that use embedded systems, such as processors, sensors, and communication hardware, to collect, send and act on data they acquire from their environments. IoT devices share the sensor data they collect by connecting to an IoT gateway or other edge device where data is either sent to the cloud to be analysed.

IoT data in considered as one of the real-time or near-real-time data source, due to which it provides data-driven insights to help better manage the business. IoT data is categorised as small packets of data sent frequently to the cloud hence If we take into consideration of 3V’s of Big Data, Volume is not big here, but Velocity is extremely high. For Example, each device mounted on power lines sending data of few MBs to the cloud every 5 minutes of interval or every device mounted in Delivery vehicle is sending sensor data of 10-50 MBs every 5 minutes. To process such data, we need scalable architecture which consists of components that can auto-scale based on demand and only charge for what you use.

How can you integrate IoT devices to GCP

Google Cloud provides IoT Core service to connect cloud environment to these IoT devices.

The main components of Cloud IoT Core are the device manager and the protocol bridges:

  • A device manager for registering devices with the service, so you can then monitor and configure them
  • A device manager for registering devices with the service, so you can then monitor and configure them
  • Two protocol bridges (MQTT and HTTP) that devices can use to connect to Google Cloud Platform

Once you register new device in IoT Core’s Device Registry, device can send the data (JSON data containing unique device ID) over MQTT (Message Queue Telemetry Transport) protocol to Cloud Pub/Sub topic which can be further load the data into BigQuery using either Cloud Function or Cloud Dataflow. There is also latest Pub/Sub feature called Write to BigQuery Subscription which can directly load data into BigQuery tables without any dependencies to write separate code. Alternatively, An IoT device can send the data directly to GCP Cloud Storage over HTTP protocol. This data can then be loaded into BigQuery using Cloud Function for further analysis. Once all data arrives in BigQuery, Unified Data Platform is built on top of variety of raw telemetry data categorised as Fact data and Device Metadata categorised as Dimension data, it can be further transformed into proper Data Modelling layer. Raw data can be further joined and processed to form Consumption layer. You can also send the commands or update device configuration (e.g., firmware version) to the IoT device from Google Cloud via IoT Core service. However, Currently, Cloud IoT Core supports commands over MQTT only (not HTTP).

Migrating from IoT Core – Alternatives in the market

As per Google’s announcement, Google IoT Core is being retired on August 16, 2023, there are several offering in the market. Google Cloud offers a host of partner-led solutions, built on Google Cloud, that meet the needs of IoT customers. Explore different options on this link

  • One of the options is ClearBlade. ClearBlade is an IoT, Edge, and AI software company driving enterprise digital transformation through its IoT Core, IoT Enterprise, and Intelligent Assets offerings on Google Cloud. ClearBlade IoT Core is a replacement product that provides the same functionality as GCP IoT Core. ClearBlade IoT Core offers 1:1 product parity including a Device Table, Security, MQTT+ Messaging, Service Integrations, Edges, and Monitoring. ClearBlade also provides a free “1-click” migration tool that fully automates the move to ClearBlade IoT Core. This is best option if you do not want to spend too much time in thinking and rebuilding your existing IoT Core architecture and simply use ClearBlade IoT Core easy integration to replace Google IoT Core.
  • ThingsBoard is an open-source IoT platform for data collection, processing, visualization, and device management. ThingsBoard is suitable alternative to Google IoT Core and Professional Edition provides much more features of full IoT application enablement capabilities.
  • Losant is another option which we can use to replace Google’s IoT Core functionality to replace the Device registry and register devices and it also easy integrations with Cloud Pub/Sub or BigQuery to send raw data to GCP.

Conclusion

There are several benefits of IoT (Internet of Things) data analysis, and it encourages companies to rethink the ways they approach their businesses and gives them the tools to improve their business strategies. IoT data can be easily ingested into Google Cloud using one of the Google IoT Core’s technology partner option available in the market. Partner that has similar offerings as Google IoT Core components can be easily integrated with Pub/Sub to send data over to the cloud and vice versa.