How to Create an AWS Lambda Layer for Python 3.12 with Specific Dependencies

When working with AWS Lambda, managing dependencies can be a bit tricky, especially for packages that require native libraries or are platform-specific. One solution to this is to use Lambda layers. In this tutorial, I’ll walk you through how to create an AWS Lambda layer that contains necessary Python dependencies, such as langchain, sqlalchemy, pymysql, and more.

Step 1: Understanding the Requirements

Lambda layers are used to package libraries and dependencies separately from your Lambda function code, allowing you to reuse them across multiple functions. However, AWS Lambda has its own specific runtime environment that is based on Amazon Linux 2. Therefore, when installing dependencies, you need to ensure that they are compatible with this environment. This is where the pip install command with specific flags comes into play.

Here is the command you will use to install the required packages for your Lambda layer:

pip install --platform manylinux2014_x86_64 --target=python --implementation cp --python-version 3.12 --only-binary=:all: --upgrade langchain_aws langchain sqlalchemy pymysql langchain-community langchain-core

Step 2: Breakdown of the Command

Let’s break down the command and the significance of each flag:

  • --platform manylinux2014_x86_64: AWS Lambda runs on Amazon Linux, which is based on manylinux2014. This flag ensures that the installed packages are compatible with Lambda’s execution environment.
  • --target=python: This specifies the target directory where the dependencies will be installed. In Lambda layers, Python dependencies must reside in a folder named python/, so the target is set to python.
  • --implementation cp: This tells pip to look for packages compatible with CPython, the implementation used by AWS Lambda.
  • --python-version 3.12: Lambda currently supports Python 3.12, and this flag ensures that the correct version of dependencies is installed for that version.
  • --only-binary=:all:: This flag instructs pip to install only precompiled binary versions of the packages. This is crucial since Lambda’s environment doesn’t support compiling dependencies during runtime.
  • --upgrade: Ensures that you get the latest versions of the packages.
  • The list of packages: langchain_aws, langchain, sqlalchemy, pymysql, langchain-community, and langchain-core are the dependencies required for your Lambda function.

Step 3: Creating the Lambda Layer

Follow these steps to create the Lambda layer:

Set Up a Virtual Environment (Optional but Recommended): Creating a virtual environment ensures that your system’s global Python environment remains clean.

python3 -m venv layer-env
source layer-env/bin/activate

Run the pip Command: Run the command we discussed earlier to install the required dependencies:

pip install --platform manylinux2014_x86_64 --target=python --implementation cp --python-version 3.12 --only-binary=:all: --upgrade langchain_aws langchain sqlalchemy pymysql langchain-community langchain-core

After running the command, the python/ directory will contain all the necessary libraries and files.

Zip the python/ Directory: AWS Lambda requires that your layer be uploaded as a zip file.

zip -r9 layer.zip python/

Upload the Layer to AWS Lambda:

  • Go to the AWS Management Console.
  • Navigate to Lambda > Layers > Create Layer.
  • Provide a name for your layer, upload the layer.zip file, and select the compatible runtime (Python 3.12).

Use the Layer in Your Lambda Function: Once the layer is created, you can add it to your Lambda function under the “Layers” section.