AWS - Lambda functions and Secrets Manager performance

Using the AWS Secrets Manager, we can safely store secrets that are required by a Lambda function. Use-cases include security keys for encrypting, decrypting and signing payloads. I wrote a how-to article a few months back on the topic (see https://www.garudax.id/pulse/utilizing-secrets-manager-aws-nalin-jayasuriya). 

It is crucial that AWS Lambda functions perform their function quickly when utilized in interactive applications, such as in my hobby project PasswordWiser (www.passwordwiser.com).

When an AWS Lambda function is started and run, it may process a single request or several requests, depending on how frequently requests are received and processed. Think of this concept as being similar to when running an application in a managed containerized environment such as OpenShift. 

Initial retrieval of a secret from the AWS Secrets Manager could sometimes take up to a full second (i.e., a 1000 milliseconds). This I determined this by looking at the AWS CloudWatch logs. Even a second's delay is a long time in an interactive application's response. So, anything we can do to speed up retrieval was needed.

In late 2022, AWS introduced a Lambda function extension called the 'AWS-Parameters-and-Secrets-Lambda-Extension'. In this article I will describe how to utilize this new extension to significantly improve the performance in an AWS Lambda function that is utilizing AWS Secrets.

The first step is to add the new extension to our Lambda function. We do this by adding a layer in the Lambda function we want to optimize. Select the 'AWS-Parameters-and-Secrets-Lambda-Extension' as the layer.

When called by our Lambda function, this extension calls the AWS Secrets Manager (given the calling context has the required permissions), caches the result for a short period of time (e.g., 3 minutes), and returns the secret to us. On subsequent calls, the cached value will be returned quickly. For just how long the caching is performed is configurable. AWS runs this extension on HTTP port 2773 in the same execution environment as the Lambda function.

Since it was hard to find example C# .NET code for the HTTP call to fetch the cached secret from the extension, I had to experiment and write the code myself. Here I am sharing the code so that you wouldn't need to reinvent the wheel!

During runtime, the Lambda function has the AWS session token available via environment variable AWS_SESSION_TOKEN. This value must be sent to the extension so that it has the security context. We send this in the HTTP header. We send the name of the secret we need to retrieve via query-string parameters.

Sample code: C# .NET 6.0:

public class GetSecretResult
 {
    public string SecretString { get; set; }
 }

.
.
.
public string GetSecretByKey(ILambdaLogger logger, string secretName)
{
   var awsSessionToken = Environment.GetEnvironmentVariable("AWS_SESSION_TOKEN");

   var cacheClient = new HttpClient();
   cacheClient.DefaultRequestHeaders.Add("X-Aws-Parameters-Secrets-Token", awsSessionToken);
   var url = $"http://localhost:2773/secretsmanager/get?secretId={secretName}";
   var cacheResponse = cacheClient.GetAsync(url).Result;

   if (cacheResponse.StatusCode == HttpStatusCode.OK)
     {
        var contents = cacheResponse.Content.ReadAsStringAsync().Result;
        if (string.IsNullOrEmpty(contents))
          {
              var detail = $"{nameof(GetSecretByKey)}. Error, secret '{secretName}' is empty.";
              logger.LogError(detail);
              throw new Exception(detail);
          }

        var contentData = JsonSerializer.Deserialize<GetSecretResult>(contents);
        return contentData.SecretString;
      }
     else
      {
         logger.LogInformation($"{nameof(GetSecretByKey)} failed. HTTP response {cacheResponse}");
      };
    throw new Exception($"{nameof(GetSecretByKey)}. Failed to get secret.");
}
        

With the above approach, I noticed an amazing performance improvement in retrieving the secrets by looking at the CloudWatch logs for the Lambda function. When multiple requests are processed by the same Lambda execution instance, the time taken was less than 50 milliseconds!

So, if you too have a Lambda function that utilizes the Secrets Manager, try out the the new AWS-Parameters-and-Secrets-Lambda-Extension.