DevSecOps and Secret Handling

Preventing secrets from being committed into source code repositories requires a multi-stage process without which your organization may inadvertently leak credentials or access keys that lead to your next data breach. Above, we identify the key areas you should consider as you develop a comprehensive prevention strategy to strengthen your secret management practices.

Secret management requires a strategy:

• Assess & Remediate: identify what secrets already exist within your code bases and develop a plan
• Deploy: deploy a secure storage platform to store secrets
• Train: Developers and DevOps engineers need to be familiar and trained with how to use the secret store
• Build: A process to prevent secrets from being accidentally pushed into code repositories needs to be established within your CI/CD pipeline to get to the proactive #shiftleft level of maturity.

Assess & Remediate

Identify code repositories; owners; and an assessment methodology

Develop a comprehensive process, Tooling and Remediation Runbooks, and consider all stages of the assessment from identification to cleanup. Modus has helped many customers with this process.

This should include:

• Identify the automated scanning tools you will use.
• Include the repository owners in this process as part of your planning so they understand what is expected of them.
• Share the process with stakeholders.

When developing an initial assessment and remediation plan, most organizations are surprised to find just how many egregious instances of spilled credentials and secrets actually have occurred and are not prepared for the volume and level of effort required to resolve these initial problems.  For this, we suggest leveraging your existing vulnerability management process if they already exist. 

If a vulnerability management program does not already exist which you can leverage consider developing a process built around a standard incident response playbook which would include the following major functions:

1. Preparation: Tell stakeholders what you are doing and why you are doing it. 
2. Identification: Run automated scan tools against code repositories. Tools such as Trufflehog, git-secret, and Horusec all offer scan engines that can help you identify secrets across code bases. Manually validate results
3. Containment: Integrate cleanup into existing SDLC process and tools e.g. Jira and develop run books ie (remove and rotate the secret + task security teams with validating the secret has not been abused)
4. Eradication: Code needs to be refactored so that embedded secrets are removed.
5. Recovery: After refactoring, the updated code needs to be released.
6. Lessons Learned: Identify why this issue occurred and how it can be prevented.

Deploy & Train

Secrets need to be deployed securely into environments

Where do we store secrets and safely access them?

• Cloud-Native tools such as AWS Secrets Manager
• Third-party tools such as HashiCorp Vault 
• Inject secrets into environments from chosen tool
• Open source tools exist but have limits

The deployment must include proactive training for engineers and developers.

As part of your strategy to prevent secrets from being committed into source control systems, you must have a secure platform for managing those secrets many options exist to satisfy requirements however while evaluating and selecting a technology you should consider the following:

• Will the solution provide a centralized view of all secrets; how often they are rotated; and how they are leveraged within your organization?
• Does the solution integrate with access provisioning processes to automate key rotation? And does that solution provide for fine-grained access management so that secrets are not over-provisioned?
• Does the system provide audit capability so you can quickly identify who added a secret; and how it was used?
• Does the system provide coverage of all the areas that you need? Will the solution work with Lambda, Terraform, Kubernetes, Github, Bitbucket, and or GitLab?

Once a system is vetted, selected, provisioned, brought online a comprehensive effort to document usage of the system as a standard and train engineers and developers to use the system must be made.

Build

Getting to DevSecOps requires creating automation to prevent secrets from being committed in the first place

Prevent secrets from making it into source control

Your first line of defense is ensuring API keys, passwords and similar don’t end up in source control and while we list this step last; once you've gone through the process of automating checks and building the foundation of secret management this is your first line of defense.

• Train your staff 
• Use pre-commit hooks. A variety of tools exist including:
• AWS Labs git-secrets
• Talisman pre-commit hook library 
• Include Document your standards around handling secrets

Many vendors offer in-repository secrets scanning and remediation tools

Major vendors such as GitLab and GitHub can scan and lock secrets directly in source control

• Reduces need for custom CI/CD pipelines
• Can alert and auto-lock secrets
• Often expensive

Building custom pipelines for secret scanning

If not shifting left, custom pipelines can be built

• Open-source tools for scanning pull requests e.g. git-secrets
• Integration of existing SAST tools with some secrets detection capabilities e.g. SonarQube 
• Block pull-requests and build pipelines on failure

Summary

Eliminating secrets from source control systems and throughout serverless and containerized systems requires a comprehensive strategy. That strategy must include a process to identify your current state; technology to securely store secrets; training for engineers and developers, and build automation to automate the prevention of secrets moving forward. 

Learn more about how Modus Create can help identify and solve the security challenges in your development pipelines by visiting https://moduscreate.com/services/security/