JSON compression for AWS API Gateway, Lambda and Node.js 10.x with zlib

When you have a large JSON data payload (probably queried out of a database), the basic idea is that you'll want to compress it before sending it over the wire as much as possible; to increase network performance (desktops, tablets and phones have plenty of power these days to decompress payloads for you). The payload limit for AWS API Gateway and Lambda is currently: 6291556 bytes. To get around this and increase network performance, I wanted to get a basic since of what data compression algorithms were natively available to Node.js developers via zlib (this comes with all Node.js installations).

Here's a code snippet that shows how to take a simple JSON blob and compress it with 3 different zlib available compression algorithms (it will print out the byte length, too):

// access the zlib module
const zlib = require('zlib')

// create a JSON blob
const json = JSON.stringify({turn: 'this JavaScript object into JSON'})

// print out the JSON blobs' byte length from largest to smallest
// after it is compressed (I'm using sync processes here for ordering):
console.log(`gzip compression: ${zlib.gzipSync(json).byteLength}`)
console.log(`zlib compression: ${zlib.deflateSync(json).byteLength}`)
console.log(`brotli compression: ${zlib.brotliCompressSync(json).byteLength}`)

When you run this program via Node.js in your terminal, you'll see this as output:

gzip compression: 37
zlib compression: 25
brotli compression: 21

As you can see, Google's brotli compression algorithm is the best outright as far as compression size goes (https://github.com/google/brotli).

However, if you were to run this in your Node.js 10.x AWS Lambda, you'll get an error that looks like this: "zlib.brotliCompressSync is not a function". In a nutshell, I really wished that AWS would be very clear on the exact version of Node.js that is running in AWS Lambda. We can deduce that they aren't using the LTS 10.16.0 (the latest 10.x version) since this function was just introduced since that version.

Work around for that? Yes, just install it as a 3rd party dependency for your Node.js Lambda deployment: https://www.npmjs.com/package/brotli.

Additional info on this topic:

1.) Why don't we stream the response back using Node.js Stream API? https://forums.aws.amazon.com/thread.jspa?threadID=248560. You can and to save on memory is generally a good approach to reduce system memory issues (I really just wanted to see what we could do with a JSON blob in memory and compress it down a lot to get past a raw payload limitation).

2.) I wanted to give a shout out to this project: https://www.npmjs.com/package/pako. It's a really awesome compression package for JSON that's also recommended with Node.js!

3.) Be aware of grabbing large amounts of data in Node.js (from a DB) because you'll run out of your memory quick if the query returns enough data to hit the MB limit, however, using the info from point 1.) above, you'll notice that even if you can stream a lot of data over the wire (manage memory issues), response times for API Gateway are limited to 30 second max and potentially less in the future; so e.g. it's not going to save you.

4.) So regardless if you're streaming or batching data, API Gateway and Lambda are super limited for big data tasks. If you stream and fix memory issues, you run into timeout issues from either tools depending on the use case. If you don't stream, you can blow memory up quickly.

Conclusion:

AWS Lambda and API Gateway are great together to server speedy/optimized and cacheable REST endpoints (not going to get into GraphQL or WebSocket support here). You'll most likely run into a lot of different limitations for 'big data' related items, so it's not advised to use these tools for this purpose (AWS: I'm waiting on no memory, compute, timeouts and network limitations in the future! =). EDIT: Lambda is great for big data tasks that are to be handled potentially as 'parts' of big data on an event basis (if you just use Lambda and not with API GW*)

Thanks for the read and I hope that you learned some super basic stuff about how to compress JSON with Node.js, zlib and some of the runtime limitations of Node.js 10.x in the current AWS API Gateway and Lambda ecosystem.