Using AWS backend to perform Data Visualization of virtualized IoT devices, Hands-on Guide

This tutorial is about the implementation of a system capable to monitor data that comes from virtualized sensors. The system is composed of different tools offered by Amazon Web Services, that permits the correct visualization of different information coming from simulated IoT Devices.

An Overview

The system I'm going to show you has different components:

• Virtualized IoT Environmental Stations, Python scripts that publish to the MQTT broker different metrics.
• Amazon's IoT Core and MQTT Broker, Which permits us to catch and forward the information published by the virtual sensors.
• Amazon's DynamoDB, Which stores this information.
• Amazon's Lambda Functions, That fetch and process data from the database.
• Amazon's API Gateway, Which is used in order to access the results of our Lambda functions from the Web Application.
• Serverless Web Application hosted on Amazon's S3, Which permits the visualization of the most interesting metrics from your personal device.

The following image resume the structure and communications of the system

Before we get deeper into the tutorial, in the following link is possible to check out the full code that comprehends:

• Python Scripts to run the virtual environmental stations.
• Web Application complete build and source code.
• Lambda Functions code that permits us to fetch and process data.

The following is an explanatory video about the functionality and structure of the project:

Now let's start with the first steps of the tutorial.

Step 1: Create Amazon IoT Core Thing

This first is a basic step: Create an instance of a "Thing" in Amazon IoT Core. Since this step is really straight forward and takes into account a lot of basic steps, I will recommend following the steps presented in the link below, until the chapter about "Configure and Test Rules".

This tutorial will help you create the Thing instance that we will use to communicate with the MQTT broker. An important passage is the one that warns you to store the certificate, that we will use later. Once you followed the tutorial above, you should have the following situation:

Step 2: Python Virtualized Devices

Now that we have stored certificates and keys, we will start working on our virtualized sensor using Python. We can use Python to create virtual sensors using AWS Python IoT SDK.

In order to do that, you can follow the instruction in the following GitHub Repository:

Once we have installed the SDK we can proceed with the creation of the scripts that will virtualize our sensor. Let's check the script present in the GitHub repo linked in the beginning.

We use this function in order to store the important communication parameters, remember to set the certificates variables with the correct path and to pick from AWS MQTT the endpoint host. Remember to choose accurately your topic, since we will use it later to access the information.

After we have the function that uses the SDK calls in order to instaurate connection with the MQTT:

Then we have the function that generates some random sensor metrics, this will be called in order to furnish new values to the system.

In the end we have a simple building message function, and finally the main function that publishes the message to the topic we selected every 10 seconds.

After this setup, we could be able to reach our MQTT broker and publish the message under "/topic"

Step 3: DynamoDB and IoT Core Rule

We have achieved the important step of publishing the data into the MQTT broker, but now we need persistence. In order to build a persistence layer we can set up a new DynamoDB table that will store each message we publish under a certain topic. Let's head to DynamoDB in our AWS console and create a new table:

Once we set up the new table, setting up "DateTime" as the primary key, we will have a new table in "/Tables" with the name we decided to give to it.

We now need to redirect our published messages from the MQTT to our DynamoDB table. We can do that using an IoT Core rule. Since this is also a very mechanic step I suggest you follow this detailed guide, and to use our "topic" and DynamoDB table instead of the one presented in the guide.

Once we have set up the rule and the DynamoDB table we are halfway through, but we have realized a persistence layer capable of storing data published in our topic into our database table.

Step 4: Lambda Function

The next step is about retrieving that data. We will create lambda functions that are capable to fetch and process the DynamoDB contents. In order to create a Lambda Function you can just go to Lambda from the AWS search bar and "create a new function". In this case, I used "NodeJs" but lambda function can use different languages since in this case are practically stand-alone programs that have to generate output from our tables.

In my project I used two different functions:

• station_values Which retrieved the latest values of the sensors of our two stations.
• sensor_values Which retrieved the last hour values generated for each metric.

You can have a look at those functions following the GitHub link I suggested in the beginning, now let's focus on the general aspects those functions have :

• Data Fetching, we use CynamoDB client in order to communicate with the tables. in this case I decided to use the function "scan" returning all the values of the table.

• Data Processing, using a really simple script in order to take the latest value of our two environmental stations

• Data Output, were we pass our processed data in JSON format

A response will be structures in the following way:

We will be able to use the output of this script in a GET function accessible from the internet in the next steps.

If you have some problem with the connection between Lambda function and DynamoDB it may be the case you have not the permission to make them communicate. You can follow these steps in order to solve it.

Step 5: Gateway API

As we said in the previous chapter, we need to access using some REST calls the output of our Lambda functions. AWS API Gateway offers this service creating an API that can be accessed using HTTP. We can select "Create new API" in the main dashboard of API Gateway.

After that we click on "Build REST API" and we will have the following situation.

You can call it as you want, select "Endpoint Type/Edge optimized" and then click create. you will now have a practically blank screen, select "Actions/Create Method". Select in the drop-down menu the voice "GET" and create the method. you should now have the following screen:

Here you can link the Lambda function you need to have access and then proceed to create it. Once created is important in order to access it to enable CORS. you can do that from the actions drop-down menu selecting GET as method.

Once we have finished this step, we can proceed to deploy our API selecting "Deploy API" and creating a new stage name.Once you deployed your API you should have the API link, accessible via http like below:

Those "Invoke URL" will display the API call, in other words the result of the Lambda function we connected to it.

Step 6: Serverless Web Application and last steps

We reached the final step, we now have access to our processed data in an accessible form using Gateway API invoke URL.

Now we can create our web application in order to use those data. Since structuring the layout of the web application is not the purpose of this tutorial, I will just suggest you take a look at the repository I linked at the beginning where you can find my project and the ReactJS web application I built.

The important step I will point out is the one regarding the usage of our API calls. Once you created them, you will have access to the data produced by the lambda and you can build an application with your favorite framework and use it to display the information. My application example:

Now in order to host our our public domain we will create a new S3 bucket from S3 console. Once we gave it a name, we will upload our application build using the upload console

Another important step is granting public access to our web application. After uploading it we can go to "Permissions" and edit "Block public access" to OFF.

The last step is creating a static website hosting and we can do that going into "Properties" and selecting "Static Website Hosting/use this bucket to host a website"

Now we are able to display the information on our public domain!

Conclusions

Thank you for having spent your time reading this and trying to set up your IoT Data Visualization system. This is data-centric approach is just one of the possible approaches possible using AWS. This project is part of the First assignment of Internet of Things course at Sapienza University.