Securing AWS Lambda at Scale

https://peirstom.github.io/blog/securing-aws-lambda-at-scale

Serverless gives teams insane velocity, but that speed collapses if permissions sprawl or runtime attacks go undetected. This field guide documents how I run Lambda programs that are fast, compliant, and battle-tested.

1. Model the Blast Radius Before Writing Code

Start with a simple inventory: which teams own the functions, what data they touch, and which accounts/environments they live in. I map those details into an ADR (architecture decision record) so everyone agrees on the desired trust boundaries.

From there, design the minimum IAM surface. Each function receives its own execution role scoped to exactly the AWS API calls it needs, and access to parameters is delegated by resource tag. CloudFormation stacks (or CDK constructs) enforce those relationships so the deploy pipeline fails if a developer tries to grant *:* access.

Field Notes

• Tag Lambda functions and IAM roles with Environment, DataClassification, and Owner for later policy scoping.
• Block wildcard resources in IAM by using aws:requestTag conditions inside service control policies.

2. Bake Guardrails into CI/CD

A Lambda that ships without scrutiny will accumulate permissions. I wire the pipeline so that every merge request runs: (a) IaC scanning (cfn-nag + Checkov) for CloudFormation/CDK templates, (b) policy linting via iam-live to spot unused actions, and (c) unit tests that exercise the handler with mocked context.

The pipeline publishes a signed SAR (Serverless Application Repository) layer containing baseline observability tooling—AWS Powertools, structured logging, and a custom metric decorator—so every function gets the same safeguards by default.

Field Notes

• Fail the build when a function-env pair lacks mandatory environment variables such as POWERTOOLS_SERVICE_NAME.
• Use GitLab/GitHub protected environments so only security reviewers can approve production Lambda deploys.

3. Instrument Runtime Protections

Better IAM reduces risk, but you still need detection. Each function ships with AWS Lambda Extensions that forward logs to AWS Security Lake in near real time. We layer AWS WAF on API Gateway/Lambda URLs and enable AWS Shield Advanced for mission-critical endpoints.

Runtime monitoring looks for privileged API calls (for example ssm:GetParameter against secrets) that deviate from an established baseline. When they do, EventBridge triggers an automated isolation playbook that revokes the execution role, pages the on-call, and snapshots logs for forensics.

Field Notes

• Enable Lambda insights with enhanced metrics so cold-start spikes, memory thrash, and throttles surface quickly.
• Route detections to Security Hub with custom finding types so they participate in organization-level metrics.

4. Close the Feedback Loop

Security controls only stick when teams see outcomes. Every Friday the DevSecOps crew reviews IAM usage reports (from Access Analyzer) with product squads. If an action hasn’t been used in 30 days it’s pruned. If a function repeatedly hits concurrency limits, we investigate whether retry storms could mask an incident.

Finally, we run quarterly GameDays that simulate key compromise, runaway costs, and supply-chain attacks on Lambda layers. Those drills refine our runbooks and validate that alarms reach humans as fast as machines.

Field Notes

• Export IAM Access Advisor data to Athena for trend analysis across thousands of roles.
• Document remediation steps in runbooks stored next to the code so context travels with each function.

https://peirstom.github.io/blog/securing-aws-lambda-at-scale