Tracking OS & application package vulnerabilities in Amazon Machine Images (AMI) and container images has different considerations as they are used by ephemeral compute resources in AWS cloud environment. Few such considerations are:
Figure 1: Considerations for AMI & Container Image Vulnerability Management
Many cloud workload protection (CWP) vendors provide capabilities to handle different aspects of workflow for assessing OS & application package vulnerabilities in AMI & container image & real-time tracking of risk score. However, an external vulnerability orchestration tool like DefectDojo would be still required for lifecycle management of vulnerability assessment findings.
Figure 2: Vulnerability Management Capabilities in CWP Vendor and in DefectDojo
Tenets for setting up OS & application package vulnerability assessment finding management using DefectDojo:
- Grouping of findings per AMI name and per container image name. For each AMI/container image, group findings based on component/packages instead of CVEs as it helps identify unique finding and helps generating SBOM of the components
- Data model in DefectDojo - organization's business units mapped to products in DefectDojo, AWS accounts belonging to individual business units mapped to engagement in DefectDojo. Each assessment cycle to gather vulnerability assessment data, is mapped to test cycle in DefectDojo NOTE: this is assuming there is a test type defined in DefectDojo corresponding to the CWP vendor; if not, create one
- Vulnerabilities in container images that were actively used for creating an ECS/EKS container instance (either using self hosted worker nodes or in AWS Fargate), mapped to test cycles in AWS accounts in which these container instances were instantiated. For container images that were in AWS ECR repository but did not have an active container instance, then findings in those images were grouped in a special DefectDojo engagement
Figure 3: DefectDojo Data Model for Tracking Container Image Vulnerability Findings
- Use GitHub Action to implement pipeline to consume APIs provided by the CWP vendor to gather vulnerability data and ingest to DefectDojo using DefectDojo APIs
Learning while setting up this workflow:
- In the vulnerability data ingestion pipeline, create each vulnerability assessment finding in DefectDojo with a unique ID based on AMI/container image name, component/package name and severity (for AMIs related unique IDs, append AWS account number as AMIs can be shared across AWS accounts). There was also string appended to the unique ID to identify whether it is AMI related finding or related to active image or non-active or container image that was pulled through ECR public repository. Implement lifecycle management of finding status in this GitHub Action workflow to manage lifecycle management of finding status
- CWP vendor API rate limit will greatly influence how the GitHub Action would be setup. Also, consider time taken to process all OS & application package CVEs (which could be from NVD, RHSA, ALSA, GHSA) in a multi-account AWS environment with workloads using different compute options; separate GitHub Action jobs would be required to overcome job execution duration limit of 6 hours
- If the patching process for ephemeral resources is just starting, then first discuss with devOps teams on the approach to bring down total number of findings before creating Jira tickets. Critical & High severity findings can be taken up initially for patching; consider leveraging 'finding grouping' functionality in DefectDojo to create Jira tickets. This will help avoid overwhelming devOps team's Jira boards
- If the SAST scans are to exclude dev dependency packages, then make sure the container images don't have findings related to those packages included in dev dependencies. Work with devOps team to update the container image building pipeline for excluding dev dependencies in the built container images
- Enabling SSO based access to DefectDojo UI pages and having a well-defined user lifecycle management process
- If endpoints are being populated in DefectDojo based on VM host details or container cluster details, then add pipeline to refresh this data as they are ephemeral in nature
- May need a separate Python script to gather metrics to track effort & effectiveness of the vulnerability management program. These metrics could be collected to determine overall program health & effectiveness in risk reduction efforts. Following could be collected for this measurement - total number of open findings & total number of closed findings; total number of findings open last month & total number of findings closed last month; % of findings closed within SLA (defined based on finding severity)
- (More updates pending)