DEVELOP A SERVERLESS ACTIVITY RECOGNITION MODEL USING AWS
In recent times, also due to the current health emergency, crowdsensing has been considered as a powerful tool to combat the CoVid-19 pandemic.
What is crowdsensing exactly? And how easily can be integrated into our technology?
Described as a technique whose goal is to analyze a large amount of data coming from a group of people, crowdsensing is indeed a great tool that can adapt very well to the devices we use in our daily life, such as smartphones. This guide will show how using the integrated accelerometer within our cellular phone is possible to build and set-up an activity recognition model which will tell whether the user is moving or not. To do this, we need a powerful cloud infrastructure which will allow us to receive, store and analyze those data.
THINKING THE ARCHITECTURE
Like the previous article I've written, the infrastructure has been developed using AWS. The picture below shows the architecture of the crowdsensing model
which consists of the following components:
- Mobile client: HTML5 application which will send the measures of the mobile phone integrated accelerometer to the cloud. To achieve this goal, we need to use methods and interfaces provided by the Generic Sensor API, a framework also used by Javascript to expose sensor data consistently.
- Dashboard: HTML5 web application which will show all the results stored within the system. Data can be filtered within the last hour.
- API Gateway: platform provided by AWS which allows us to define a WebSocket for managing connections by all the mobile clients of the system.
- Lambda: computing service that runs code in response to events and automatically manages the computing resources required by that code. It allows us to create a serverless infrastructure since AWS will dynamically manage the resources and their allocation, leaving us the only duty of developing the code which will compute the sensor's data.
- DynamoDB: NoSql database provided by AWS which will be used to store users data to be seen later to the dashboard.
In particular, our activity recognition model will have two operating modes:
- Edge-Based: the front-end application will compute the activity status of the user internally and will send only the resulting activity to the cloud. The user will also have a personal dashboard to see his statistics about application usage.
- Cloud-Based: the front-end application will only send the measures retrieved from mobile phone accelerometers to the cloud, which in turn will run the activity model to send back the resulting activity status to the user.
The solution can be found at this GitHub repository.
DESIGNING THE MOBILE APPLICATION
Once we figured out how the system will work, we start to build the web application which will be used at the edge of the architecture by the users. As said before, we need to use the Generic Sensor API implemented within the Javascript language. In particular, we will read values from gyroscope by the usage of the Accelerometer interface. There exist several implementations: we will rely on the LinearAccelerationSensor subclass since we don't need to consider the contribution of a gravity force in our activity recognition model.
The sensor initialization will happen on the onload event listener of the associated index.js file of the web application, which is the running script loaded by the HTML5 page: (we take into consideration the edge-based mode, but the same reasoning can be applied to the cloud-base files)
let sensor = new LinearAccelerationSensor({ frequency: 1 });
var measures = [];
sensor.start();
sensor.onreading = () => {
// display measures to html page
measures.push({x:sensor.x, y:sensor.y, z:sensor.z, ts:Date.now()});
// do model (if edge-based)
// send data to aws
};
sensor.onerror = (event) => {
console.log(event.error.name, event.error.message);
};
This snippet shows how the HTML5 page will create a sensor object with a sampling frequency of 1Hz, i.e. values are read from the accelerometer every second.
The sensor object has an event listener where we perform the following tasks:
- we display the loaded values into the HTML page
- if we are in the edge-based mode, we perform the model with the stored data,
- we send a message to AWS WebSocket in order to send the retrieved samples
However, up to now, we don't know if the user has decided to give the browser permission to use the mobile phone integrated sensors. The Generic Sensors API can be used along with the Permissions API, which can read the current permission state to assess whether the browser can read values from the accelerometer or not.
Before starting the application we need to check by the usage of the Navigator interface if the accelerometer permission has been granted by the system. If the corresponding state of the sensor is not denied, we can initialize the sensor object.
navigator.permissions.query({ name: 'accelerometer' }).then(result => {
if (result.state === 'denied') {
// print denied error
return;
}).catch(error => {
// print api not supported error
});
BUILDING THE MODEL
When we are in the edge-based mode, we need to perform the activity recognition model at the user application side, so it's a good way to introduce it now. An equivalent version of this model has been deployed to AWS Lambda and it will be shown later.
There are various research activities that have studied many approaches to do complex activity recognition models which recognize different human activities: in this case, we will distinguish whether the user is moving or is standing still. This model has been implemented in the model.js resource file.
First, we can compute the Signal Magnitude Area (SMA) of a set of samples within 10 seconds using the following function:
function sma(measures) {
let sum = 0;
measures.forEach(measure => {
sum += Math.abs(measure.x) + Math.abs(measure.y) + Math.abs(measure.z);
})
return sum / measures.length;
}
Then the following function will tell whether the user is moving or not by checking if the resulting SMA is greater than a pre-defined threshold, i.e.
function getStatus(sma){
return sma >= 1.5;
}
So to compute the model inside our edge-based solution we need just to call those two functions, and send the resulting status to the cloud.
SENDING DATA THROUGH THE NETWORK
Now we have a simple and effective model that tells us whether the user is moving or not. The next step is to open a connection to the AWS Websocket to send the collected data. Websocket (WS) is a technology that implements a two-way interactive session between two parties over a network, and it has been implemented in Javascript by WebSocketAPI.
// Create WebSocket connection.
const socket = new WebSocket('wss://<your url>');
The initialization will open a connection to the WS endpoint, where we will send our data as a string by the usage of the send function. The payload of the message will be depending on the operating mode. For the edge-based mode, we will need to specify the user status:
{
action: "store",
message: {
ts: Date.now(),
id: id,
status: status,
isCloudBased: false
}
}
while for the cloud-based mode we will send only the measures of the sensors:
{
action: "store",
message: {
ts: Date.now(),
id: id,
measures: measures,
isCloudBased: true
}
}
The field isCloudBased will tell to the cloud back-end whether to apply the activity recognition model or not; while the action field will be used by the cloud to decide which procedure can read the message field
The last remark is about the user identifier: it is a session-based identifier, which is stored inside the Storage of the browser in order to be used for other purposes, such as the edge-based dashboard. This is the first action taken by the HTML5 page:
var id = sessionStorage.getItem('crowdSensingId');
// ...
if (!id) {
console.log("No id found on sessionStorage. Generating ...");
id = generateId();
sessionStorage.setItem('crowdSensingId', id);
}
The generateId function is a basic implementation of the Universally Unique Identifier (UUID) format and can be found on the model.js file.
DEPLOYING THE ACTIVITY RECOGNITION APPLICATION
The application has been deployed automatically from my repository using GitHubPages. Here you can find the cloud-based solution and the edge-based solution. Some devices and/or browsers such as Safari for Apple smartphones and tablet may not support the Generic Sensor API.
BUILDING A SERVERLESS ARCHITECTURE ON AWS
Since we need an HTTPS endpoint to send in a secure way the sensors data, we can't use Elastic Beanstalk to deploy a server with a WS API interface. However, cloud technologies come with tools to build serverless applications, i.e. microservices managed directly by AWS to an HTTPS endpoint. This result can be achieved on AWS by the usage of API Gateway and Lambda.
CREATING A WS ENDPOINT ON API GATEWAY
First of all, we need to create a WebSocket which will receive data from users and will send the corresponding statistics to the dashboard. This can be done from the AWS management console tool:
In particular, based on the action field of the incoming payload, the WS will do different procedures. The CrowdSensingAPI will manage two actions:
- store, to save the samples sent by the users on DynamoDB. If the isCloudBased field is set to true, the action will perform the activity recognition model described before.
- getSamples, used by the dashboard to view the resulting data. It can be related to a specific user or within the last hour of actions.
For each route, we need to specify to send the response back to the client and, most importantly, a procedure to follow.
USING LAMBDA FUNCTION FOR APPLICATION LOGIC
Once we have defined the available routes that our API can publish, we need to define a logic behind them. This is possible by the usage of Lambda functions, i.e. a small portion of code that runs within the AWS Lambda container.
First of all, from the AWS console we need to define a new function. Considering the procedure that stores data into our database (we will see later its features):
Along with the function definition, we have created an IAM Role called crowdsensingrole who has granted access to use DynamoDB with reading and writing capabilities. This role is associated with this function, so every time it will be executed under the crowdsensing identity. Now we can write the function within the IDE provided by the Lambda service:
const Model = require('./model');
const AWS = require('aws-sdk');
const docClient = new AWS.DynamoDB.DocumentClient({region: 'us-east-1'});
exports.handler = function(event, context, callback){
let message = JSON.parse(event.body).message;
let model = new Model();
let sma = message.isCloudBased ? model.sma(message.measures) : null;
let status=message.isCloudBased ? model.getStatus(sma): message.status;
var params = {
TableName: "crowd_sensing",
Item:{
"ts": message.ts,
"id": message.id ,
"info": message.isCloudBased ? {
"measures": message.measures,
"sma": sma,
"status": status
} : {"status" : status}
}
};
docClient.put(params, function(err, data) {
if (err) {
callback(err, null);
} else {
if(message.isCloudBased){
callback(null,{"statusCode":200,"body":message.status});
}
}
});
};
The Model object contains the function we have defined for the mobile application before and will apply the activity recognition model if the request comes from a cloud-based client: in this particular case, we will store also the measures and the corresponding SMA. This can be done after the request message is parsed from the event parameter, and send back to the API with the callback function.
Now we need to go back to the API Gateway platform and associate the function storeSamples to the store route.
The same steps can be done in order to define a function behind the getSamples route. In this case, however, we need to retrieve data from the storage:
exports.handler = function(event, context, callback){
console.log('processing event: %j', event);
let message = JSON.parse(event.body).message;
console.log('processing message: %s', event);
let params = message.lastHour==true ? lastHourParams : latestParams;
if(message.id){
if(!params.FilterExpression && !params.ExpressionAttributeNames && !params.ExpressionAttributeValues){
params['FilterExpression'] = "";
params['ExpressionAttributeNames'] = {};
params['ExpressionAttributeValues'] = {};
}
params['FilterExpression'] = "#id = :uuid"+(params['FilterExpression']!="" ? " and " : "")+params['FilterExpression'];
params.ExpressionAttributeNames["#id"] = "id";
params.ExpressionAttributeValues[":uuid"] = message.id;
}
docClient.scan(params, function(err, data) {
let crowd = {};
if (err) {
callback(err, null);
}
else {
data && data.Items && data.Items.forEach(function(item){
if(!crowd[item.id]){
crowd[item.id] = [];
}
crowd[item.id] && crowd[item.id].push({
'ts': item.ts,
'measures': item.info.measures || null,
'sma': item.info.sma || null,
'status': item.info.status
});
});
callback(null,{"statusCode":200,"body":JSON.stringify(crowd)});
}
});
};
This is possible by using the scan function provided by the Document Client object, with a filter expression contained within the params variable depending on whether the client has asked the latest values or the values from the last hour only. If the identifier is specified within the request body, we add this condition to the filter. We'll go more into details about the parameter provided to the scan function within the next section.
BUILDING A PERSISTENT STORAGE USING DYNAMODB
We just saw that maintaining our data is a crucial aspect of the solution since we need to retrieve them for the dashboard component. For this purpose, we will rely on the NoSQL Database provided by AWS, DynamoDb. We just create a table named crowd_sensing where:
- the timestamp (ts) is the partition key,
- the user identifier (id) is the sort key.
Going back to the getSamples function, we can retrieve all the tuples from the table with the following definition
var latestParams = {
TableName: "crowd_sensing"
};
while if we need to retrieve all the tuples within the last hour, we need a more complex definition of parameters:
var lastHourParams = {
TableName: "crowd_sensing",
FilterExpression: "#ts between :ts1 and :ts2 ",
ExpressionAttributeNames:{
"#ts": "ts"
},
ExpressionAttributeValues: {
":ts1": new Date().setHours(new Date().getHours()-1),
":ts2": Date.now()
}
};
As we have seen before, the filtering on the user identifier is added only if the request comes from an edge-based solution.
DISPLAYING ALL THE VALUES TO A DASHBOARD
The last thing to do is to show the values retrieved from the sensors of all the involved users or, in case of the edge-based solution, display to a particular user the statistics of his session. Just like the sensor application, we need to open a WebSocket connection to retrieved data from the getSamples API every two minutes. So in the file index.js of the cloud-based dashboard, we will do the following, on the onload event listener:
// Create WebSocket connection.
socket = new WebSocket('wss://<your-ws-url>');
// Connection opened
socket.addEventListener('open', (function (event) {
console.log("connected to aws websocket!");
//first request
socket.send(JSON.stringify(request));
}).bind(this));
// Listen for messages
socket.addEventListener('message', (function (event) {
crowd = JSON.parse(event.data);
if (crowdKeys && crowdKeys.length != 0) {
crowdKeys.forEach(item => {
// draw results on dom
});
}
else {
document.getElementById("warning").innerHTML="No samples found";
}
}).bind(this));
socket.addEventListener('error', (function (err) {
let errorDOMElement = document.getElementById("error");
errorDOMElement.hidden = false;
errorDOMElement.innerHTML = "WebSocket error";
}).bind(this));
tid = setInterval((function () {
// every 2 minutes call aws to check if new samples have been processed
socket.send(JSON.stringify(request));
}).bind(this), 120000);
where the request object will tell the system whether to check data within the last hour or not.
var request = {
action: "getsamples",
message: {
lastHour: false // latest value by default
}
};
For the edge-based dashboard, we need to view the activity of a single user, which is the one who has launched in the same session the activity recognition model. To do this, we need to retrieve the identifier from the Session Storage.
var id = sessionStorage.getItem('crowdSensingId');
// ...
window.onload = function () {
if (!id) { // render only if id has been found within local storage
// print error
return;
}
// init page
}
DEPLOYING THE DASHBOARD
Just like the activity recognition application, both edge-based and cloud-based dashboard can be found on Github Pages.
LAST REMARKS
A demonstration of the system can be found on the video below:
This article has been made to address the topics of the fourth assignment provided by the IoT course held at Sapienza University - Rome (IT). I hope that this reading will be useful for all my colleagues and for others to understand how the development of an activity recognition model architecture works.
