How Shipbird leveraged AWS Lambda

Lately we moved our services into Elastic Beanstalk (EB) to allow easier deployments to entire fleet of EC2s behind a load balancer. Moving into EB makes our deployment less error prone, allowed us to do blue-green deployments, etc.

Previously running on self-managed EC2 instances we were scheduling our backend tasks using CRON on a particular EC2 instance.

Moving into EB we had 2 options:

• Either we run the CRON on one of the EC2 instances inside EB.
• OR we run CRON outside the EB environment as we always did.

Running CRON outside the EB environment meant we needed to manage an EC2 instance and manage its lifetime. (Less preferable). So we decided to go with trying to run our CRON inside the EB environment. Here is what we needed to achieve that:

• Only one EC2 instance can run the CRON jobs. Not every instance that gets launched in the auto-scaling group. This instance was the Leader instance.
• The Leader instance cannot go out of commission (due to scale down). If it did a new Leader should elected by the EB environment.
• An easy way to detect Leader EC2 instance and run container_commands that will launch the cron jobs on the Leader.

Luckily EB has a way to detect and run commands on the leader instance. But with a blue-green deployment we would end up twice the number of CRON jobs setup (one in each environment). We could have created environment specific CRON jobs that only schedules tasks within the environment's EC2 instances. But that meant some fancy shell scripts to detect the environment's URL and then creating the CRON jobs.

Also the Leader election apparently was flaky during scaling down events, leaving the EB environment sometimes without a Leader.

We started exploring other options like SQS and Lambda.

AWS SQS

Simple Queue Service is a message queuing mechanism that allows one to put messages into the queue and have some other part of the code get the message. A Delay Queue enables delaying the delivery of the message by maximum 15 minutes.

One way to get the CRON tasks running was to to use a delay queue and add a message into the queue, wait for it to be available, then en-queue a new message. Wait and repeat. The issue with this mechanism was that, there needs to be seeding for the first message and electing the server that was going to be responsible for doing that inside the EB environment. Again the leader issue came up. The other option was to run our own EC2 instance to do the seeding. (Less preferable).

AWS Lambda

"AWS Lambda is a server-less compute service that runs your code in response to events and automatically manages the underlying compute resources for you".

Basically it runs micro-code on a fleet of machines that AWS has at its disposal. The code runs for a brief period of time and one can choose the amount of memory needed for the code to run.

AWS Lambda is essentially a sophisticated brokering service that finds idle instances to run short lived custom code, thus effectively leasing machines to short term tenants.

Lambdas can be triggered by various events. One of them is CloudWatch Events - Schedule. The schedule can be 1 minute, 5 minutes, 15 minutes, 1 hour, 1 day or a custom CRON schedule.

There we had it all. A dependable scheduler that,

• Did not depend on us running EC2 instances
• Was outside the EB environment

We grouped our backend tasks into the following 3 categories.

• Quarter Hourly (Called every 15 minutes indefinitely)
• Hourly (Called every hour indefinitely)
• Daily (Called every day indefinitely)

We created 3 APIs to be invoked by 3 Lambda functions. Each Lambda function called its respective API using Node JS http module.

Inside each of the 3 APIs we added logic to determine which task should be run each time the QuarterHourly, Hourly or Daily API got called. For example we have a task to send background check orders at 9 in the night and retrieve results at 10 in the night. So we added the 2 tasks inside the Hourly API and by checking the current timestamp's hour we could determine when to send the background check orders and when to retrieve the results. Similarly we have tasks that run on certain days of the month. So we put those tasks under the Daily API call.

What does Shipbird do?

"Shipbird uses crowdsourcing technology to do inner city deliveries using its fleet of drivers (called birds). The drivers are people just like you and me wanting to make money on the side during their commute to work or working weekends."