How to add dependencies to a Lambda Function in AWS
The Lambda Python runtimes includes the AWS SDK for Python (Boto3) and its dependencies, but this does not include a lot of frequently used libraries. Quite recently, I had to work with pandas in AWS Lambda, and quickly realized that it's not as easy as simply saying import pandas as pd or saying pip install pandas. AWS has their own way of adding dependencies, and that's through something called "Layers".
A Lambda layer is generally a .zip folder that contains supplementary resources, which can be for library dependencies, a custom runtime, or configuration files.
My aim with this post is to provide an easy and straightforward way to add Layers to a Lambda function, that 100% works, because when I first tried, I wasn't able to find adequate guidance. I'll show you how to add pandas, but this approach can be applied for any dependency. In fact, you'll technically learn how to add three different dependencies at once with this article!
1. Download the files for the dependencies you need
2. Unzip the wheel file
Use the command wheel unpack <filename>
Recommended by LinkedIn
3. Make a new folder named "python"
Make a new folder named "python" and move the contents from inside of the unzipped folder from the previous step, into this new folder.
4. Repeat steps 1-3 to download numpy and pytz since pandas is dependent on numpy. *** Don't remake "python" folder
Make sure to add the files from the unzipped folders to the python folder. At this point, your python folder should have all the required filed for pandas, numpy and pytz. ***pytz wouldn't have "cp123" type tags but rather has "py2" and "py3" which indicates that it is suitable for any Python 2 or 3 versions.
5. Zip up python file.
6. Create a Layer to upload dependencies
While creating a layer to, select the same runtime and architecture as your lambda function. You can choose up to 15 runtimes. This is where you'll upload the file you just zipped in the previous step.
7. Add the layer to the lambda function
When you add the layer, select "Custom layers" and select the Layer you just created.
8. Run your Lambda Function
Now, if you run your Lambda function, it should execute without any errors!
I wanted to express my gratitude for your insightful post. It provided much-needed clarity for me as I tried to create a Lambda’s custom layer for the first time. I'd like to contribute a bit more to the discussion based on my own experience. While attempting the suggested approach of downloading packages from pypi.org, I encountered some challenges. Specifically, for my lambda function, I needed to install several packages, each with its own set of dependencies. Keeping track of these dependencies became somewhat complex. Instead, I opted to install the Windows Subsystem for Linux (WSL) (https://www.ssl.com/how-to/enable-linux-subsystem-install-ubuntu-windows-10) and followed these steps: 1. Installed the necessary Python version using WSL (sudo apt install python3.9). 2. Installed all required packages to a Python folder using pip (pip install --target python_folder requirements.txt). 3. Zipped this folder and proceeded with the steps outlined in your guide. This alternative approach proved to be highly effective in managing dependencies and simplifying the process. Thank you once again for your valuable insights.