Bootstrapping Microservices using Gitlab and HashiCorp Vault

Overview

At Workfront, one of our primary goals for 2020 is the modernization of our platform. To that end, we are heavily invested in the disaggregation of our monolithic architecture into independently scalable microservices. One of the first challenges we faced when taking on this endeavor was determining how to enable teams to deploy their services and manage their application secrets with as little friction as possible.

To support the needs of our customers, Workfront has adopted a cloud-agnostic approach to developing and hosting our software. Specifically, we currently offer our SaaS solution hosted on either on AWS or GCP. Rather than rely on proprietary offerings from each cloud vendor, we decided to standardize our microservice efforts around the popular cloud-agnostic container orchestration framework, Kubernetes.

A key benefit of standardizing around Kubernetes is the fact that we can share underlying infrastructure between our microservices while maintaining isolation through the use of namespaces. Specifically, we can create a namespace for each service and leverage the built-in RBAC functionality of Kubernetes to logically isolate one service from another.

Implementation

All of the above is great in theory, but how do you actually implement namespace isolation in an automated fashion? Our solution, dubbed internally as the "k8s-bootstrapper", solves this problem by integrating HashiCorp Vault and Gitlab CI/CD pipelines.

This tool borrows some core principles of Kubernetes in its implementation. Specifically, it relies heavily on declarative documents written in YAML to outline the expected state and is idempotent in nature. Each microservice has a "manifest" YAML file which declares the following information about the service:

• Name of the service
• Kubernetes environments where the service will be deployed
• Usernames of team members who should have access to Kubernetes environments and application secrets stored in Vault
• The Gitlab repository(s) associated with the microservice

Using this information, the k8s-bootstrapper Gitlab CI/CD pipeline performs the following actions for each registered microservice:

• Creates a key/value store for the service in Vault
• Creates a namespace in each Kubernetes environment listed in the manifest file
• Creates service account/role/role binding in each Kubernetes environment to have full access to the service's namespace and nothing else (in most cases)
• Uploads the namespace-specific Kubernetes credentials tied to the service account created in the previous step to the service’s Vault key/value store
• Grants all users listed in the manifest access to the Vault key/value store using their Okta username as their identity
• Creates Vault AppRole credentials with access to the Vault key/value store and adds them as protected environment variables in the service's associated Gitlab repository(s). These credentials are used for fetching secrets programmatically as part of the service's Gitlab CI/CD pipelines.

Each of the above steps are implemented in such a way that they can be run any number of times with no side-effects. This allows the pipeline to execute these actions on each commit, ensuring that the actual state matches the desired state outlined in the service's manifest file.

Developer's Perspective

So what does all this look like from a developer's perspective, anyway? Let's walk through the general flow with an example:

Sarah is a team lead and is starting work on a new microservice. She starts by creating a Gitlab repository and committing her code. She soon realizes she needs the ability to deploy her team's code to an environment for testing and creates a merge request into the k8s-bootstrapper with a manifest file that looks something like this:

service: some-new-service
users:
  - sarahsmith
  - joeschmoe
  - johnfilmore
  - alexmiller
repositories:
  - group: core
    name: some-new-service
environments:
  - kubernetes-dev

Once the request is reviewed, approved, and merged by an administrator the Gitlab CI/CD pipeline runs through the steps outlined above. Sarah and the rest of the members of their team can now access the newly created "some-new-service" Vault key/value store through the Vault user interface or their well-documented API. There they can store secrets specific to their application without worrying about the underlying encryption mechanism, access control policies, or persistence strategy.

What's more, the credentials for accessing the different Kubernetes environments are stored in the service's Vault key/value store. These credentials can be fetched as part of their service's Gitlab CI/CD pipeline to deploy their application.

Additional Benefits

Taking this approach allows us to also have an audit log since, after all, it is just a git repository. If we ever want to know when a service was registered or a certain user was granted access to a service's secret store all that is needed is to perform a simple "git blame" command.

Additionally, we are able to control access to production Kubernetes environments by requiring merge requests be approved by architects, SREs, security, and other concerned parties. Through the use of merge request templates in Gitlab we are able to provide developers a template where they have the opportunity to answer questions around their strategies for monitoring/alerting, high availability, data persistence, and scalability as part of the merge request.

Finally, Gitlab scheduled jobs allow us to define a job that runs every n days for rotating both Kubernetes and Vault AppRole credentials with no impact to end users (developers).

Conclusion

Gitlab CI/CD pipelines allow almost anything to be automated. Vault's support for fine-grained access controls coupled with a secure-by-nature architecture make it ideal for persisting application secrets. Both Gitlab and HashiCorp Vault are great products on their own, but combining the two has allowed Workfront to balance both automation and security in a straight-forward manner.