Debouncing webhooks with Talos and AWS CDK

Here at Bared Footwear we use Sanity to manage our content and Gatsby Cloud to generate static builds. Sanity is a really powerful, flexible, and easy to use CMS which has the ability to call a webhook whenever a document is published (or unpublished).

Recently I enabled the Sanity/Gatsby Cloud integration which adds Gatsby Cloud webhook URLs to Sanity, however we quickly noticed that our build pipeline was getting jammed with build requests. We have a number of integrations that sync data between various systems and Sanity, so every time a document was synced, it would send another build request.

Gatsby does an okay good job of cancelling unnecessary builds, but it still meant that our pipeline was running almost continuously when it didn't really need to.

So I created Talos, the webhook debouncer. In Greek mythology Talos was a giant automaton made of bronze who protected Europa from pirates and invaders.

The architecture of Talos is pretty simple: Keep build requests in a queue and check that queue every 15 minutes. If there are requests in the queue, call the build webhook and purge the queue. If there aren't, do nothing.

NOTE: Luckily, in our case the webhook payload doesn't matter - if in your case it does, Talos may need some modification, or may not be fit for purpose at all.

I really love CDK, it makes it incredibly easy to create, configure, and maintain AWS infrastructure in a language that's familiar to engineers already, like Typescript, C#, or Python. YAML can get in the bin.

The first step to customising Talos for your use case is to create a config.json file in the root. It only has 2 configurable properties:

• The Cron schedule - how often the Lambda function should check the queue
• Webhook URLs to debounce - you should be able to have as many as you want

"cron": {
  "schedule": "cron(0/15 * * * ? *)"
},
"webhookUrls": {
  "production": "https://webhook.domain/production-webhook",
  "staging": "https://webhook.domain/staging-webhook"
}        

The next step is to create the stack in CDK:

export class TalosStack extends Stack 
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props)
  }
}        

After doing some basic checks on the configuration file, we create a single API Gateway instance:

const api = new RestApi(this, `talos-rest-api`, 
  deployOptions: {
    tracingEnabled: true,
  },
})        

Next we create the SQS queues for each webhook defined in the config:

Object.entries(webhookUrls).forEach(([key]) => {
  const messageQueue = new Queue(this, `talos-queue-${key}`)
  queues.push(messageQueue)
})        

As with most things on AWS, we then need to give the API Gateway permission to send a message to SQS:

const credentialsRole = new Role(this, 'talos-role', 
  assumedBy: new ServicePrincipal('apigateway.amazonaws.com'),
})
	

const policy = new Policy(this, 'talos-send-message-policy', {
  statements: [
    new PolicyStatement({
      actions: ['sqs:SendMessage'],
      effect: Effect.ALLOW,
      resources: queues.map(({ queueArn }) => queueArn),
    }),
  ],
})
	

credentialsRole.attachInlinePolicy(policy)        

Finally we need to create unique endpoints under API Gateway for each webhook, each of these endpoints is a Lambda integration that can receive messages from SQS and purge the queues.

Assume the rest of the code is wrapped in this forEach block:

Object.entries(webhookUrls).forEach(([key, value], index) => {
  ...
}        

Create the endpoints:

const webhookUriHash = crypt
  .createHash('md5')
  .update(JSON.stringify({ [key]: value }))
  .digest('hex')
const webook = api.root.addResource(webhookUriHash)
const queue = queues[index]

webook.addMethod(
  'POST',
  new AwsIntegration({
    service: 'sqs',
    path: `${Aws.ACCOUNT_ID}/${queue.queueName}`,
    integrationHttpMethod: 'POST',
    options: {
      credentialsRole,
      passthroughBehavior: PassthroughBehavior.NEVER,
      requestParameters: {
        'integration.request.header.Content-Type': `'application/x-www-form-urlencoded'`,
      },
      requestTemplates: {
        'application/json': `Action=SendMessage&MessageBody=$util.urlEncode($input.body)`,
      },
      integrationResponses: [
        {
          statusCode: '200',
          responseTemplates: {
            'application/json': `{'success': true}`,
          },
        },
	  ],
    },
  }),
  { methodResponses: [{ statusCode: '200' }] }
)        

Now add the Lambda integration and add it to the webhook:

const lambdaPolicy = new PolicyStatement()
lambdaPolicy.addActions('sqs:PurgeQueue', 'sqs:ReceiveMessage')
lambdaPolicy.addResources(queue.queueArn)
	

const lambdaFunction = new Function(this, `talos-sqs-cron-lambda-${key}`, {
  functionName: `talos-sqs-cron-lambda-${key}`,
  handler: 'handler.handler',
  runtime: Runtime.NODEJS_14_X,
  code: new AssetCode(`./lambda`),
  memorySize: 512,
  timeout: Duration.seconds(10),
  initialPolicy: [lambdaPolicy],
  environment: {
    SQS_URL: queue.queueUrl,
    SQS_REGION: Stack.of(this).region,
    WEBHOOK_URL: value,
  },
})


webook.addMethod('GET', new LambdaIntegration(lambdaFunction, {}))        

As you can see the Lambda function is simply picked up from the ./lambda directory, luckily CDK already knows how to deal with Typescript so you can write your Lambda in Typescript without any extra configuration or tooling!

import { ReceiveMessageCommand, PurgeQueueCommand, SQSClient } from '@aws-sdk/client-sqs
import axios from 'axios'
	

const handler = async function (event: any, context: any) {
  try {
    const queueUrl = process.env.SQS_URL
    const webhookUrl = process.env.WEBHOOK_URL
    const client = new SQSClient({ region: process.env.SQS_REGION })
    const params = {
      AttributeNames: ['SentTimestamp'],
      MaxNumberOfMessages: 10,
      MessageAttributeNames: ['All'],
      QueueUrl: queueUrl,
      VisibilityTimeout: 20,
      WaitTimeSeconds: 0,
    }
	

    if (!webhookUrl) {
      return {
        statusCode: 500,
        headers: {},
        body: {
          error: 'Webhook URL not specified',
        },
      }
    }
	

    try {
      const data = await client.send(new ReceiveMessageCommand(params))
	

      if (data.Messages) {
        const body = data.Messages[0].Body
        await axios.post(webhookUrl)
        await client.send(
          new PurgeQueueCommand({
            QueueUrl: queueUrl,
          })
        )
      }
	

      return {
        statusCode: 200,
        headers: {},
        body: {
          messages: data.Messages?.length || 0,
        },
      }
    } catch (err) {
      return {
        statusCode: 500,
        headers: {},
        body: {
          queueUrl,
          error: err,
        },
      }
    }
  } catch (err) {
    return {
      statusCode: 500,
      headers: {},
      body: {
        error: err,
      },
    }
  }
}
	

export { handler }        

Now we want to add a cron schedule to call the Lambda function every 15 minutes (as per our configuration):

const rule = new Rule(this, `talos-cron-${key}`, 
  schedule: Schedule.expression(cron.schedule),
})
	

rule.addTarget(new LambdaFunction(lambdaFunction))        

And the final step is to output our brand new SQS URLs and API Gateway webhooks which we can add to our Sanity webhook configuration, replacing the old Gatsby Cloud URLs:

new CfnOutput(this, `talos-queue-url-${key}`, 
  value: queue.queueUrl,
  description: 'The URL of the SQS queue',
  exportName: `talos-queue-url-${key}`,
})
	

new CfnOutput(this, `talos-webhook-url-${key}`, {
  value: `https://${api.restApiId}.execute-api.${Stack.of(this).region}.amazonaws.com/prod/${webhookUriHash}`,
  description: 'The URL of the API gateway',
  exportName: `talos-webhook-url-${key}`,
})        

We have been running Talos for over a month now and it has been faultless. SQS is one of the oldest AWS products on the market, so it was always going to be reliable. So if you've disabled the Sanity/Gatsby Cloud integration because your build system was being overwhelmed, try Talos and let me know how it works for you!

Download or fork Talos @ Github: https://github.com/struct78/talos

We are also hiring! Apply now!