A Serverless App to Process Streaming Data

There is certainly a joy in building things when they are serverless.  Combining  serverless capabilities with stream processing services that can continuously capture gigabytes of data per second from different sources, variety of use cases from actionable insights gathered from IT logs to track public reaction  in  social  media  feeds  can be  covered.

Here is a test architecture for a Serverless App that uses user input to  start  collecting streaming text data from twitter (one ‘still’ easily accessible source of streaming data),  filters the result based on a desired hashtag and finally performs a sentiment analysis and  visually compare the number of negative and positive in a dynamic chart.  

The main sequence of events in this structure are:

1.  User's request will be resolved by Route53 to a Webapp uploaded as a single html file in a S3 bucket configured for static web hosting  plus some *.js source code with proper CORS configurations.

2. API Gateway which is the waitress here and delivers the user's request. Upon receiving a tracking request with a hashtag a Lambda function starts Codebuild to run a simple Node.js application that gathers streaming tweets from twitter API and pushes them to Kinesis Stream shards.

3. Kinesis Stream with only one shard is enough to gather 1MB/sec or 1000 records/sec, which adds up to $0.017 per hour if data is kept less than 24 hours in the Stream. Streams are then connected to Kinesis Firehose that loads streaming data to S3.

4. S3, upon receiving data, triggers another Lambda function that populates a Dynamodb table with one tweet per item. This table triggers the final stage of architecture where AWS Comprehend starts analysing each tweet and writes back its sentiment value to the Dynamodb table.

5. User request to compare the the number of tweets with positive and negative content will be proxied by API Gateway to this table and Chart.js will output beautiful and dynamic chart versus time.

Now that the App is ready we can try it by a simple question: “Does Twitter like Trump”? While the answer is quite clear let’s see how bad the situation is:

Just to be fare let's follow another hashtag: