The term Secrets Management is trending in the DevOps realm, and you may be struggling to understand the problem it solves. You may be thinking, “What am I missing here?” So let’s get back to the basics and look at a few simple examples of secrets management to better understand the use case.

; TLDR this is where you should be keeping all of your passwords, API Keys, private keys, etc.

What is it that we are trying to solve with Secrets Management?

At the core, we are trying to protect sensitive information, such as credentials, private keys, tokens, etc., from leaking out of our system or being abused by someone to gain unauthorized access.

Back in the old days, you had one web server and one database server in a simple, predictable, and easy to manage configuration. You had only one file where you kept the password (the connection string), and that was it. Then the team grew along with the product, and now you have to share passwords between developers and different server environments.

You shared passwords by emails, Google Docs, or hardcoded to the connection string on Git. Then, the SSL private key came into the picture, and things started to get a little more complicated. Oh, and just to keep things simple, the same passwords were used both for development and in production.

Today, you have multiple instances of that web app, which is now stateless running on docker, talking to other microservices, using more and more databases, and supporting services. Where do you keep these credentials now? Hardcoded?

This is where the concept of Secrets Management comes to play. It’s how you deploy your applications in different environments without hard-coding or keeping them in clear text in your application or infrastructure code.

There are multiple “layers” of Secrets Management solutions, where, like everything else, there are shortcomings and trade-offs to consider.

Our requirements for a Secrets Management solution

Depending on your company’s risk profile, size, stage, and regulation, you may have different requirements from a Secrets Management system. The following is a short list of things to consider when looking for a solution:

1. Passwords have to be encrypted at rest (not the REST API rest).
2. Allow you to attach them at scale.
3. Automatically attach / load on new instances of your application.
4. Access to secrets is restricted.
5. Support for changing secrets or secret rotation.

The Secrets Management “Layers”

Various technical solutions can be considered “Secrets Management.” Some are pretty straight forward, and you may already be using them, and others like the Hashicorp Vault are full-scale applications that require their own deployment.

.env Files

These are files that you place in the file system next to your application as settings that contain credentials. You may keep a dedicated file for each environment, and then copy or mount it during deployment. The .env files overwrite or override the default app settings. For example, in Java’s Spring, you can keep multiple versions of your application.properties file, one for each environment, and then place these files where the application expects them.

If you can find a way to securely keep and distribute the files to the servers, such as using Ansible Vault, you can somehow consider it as Secrets Management.

When you want to scale or use containers, the delivery of the .env files becomes complicated, which leads us to the next layer.

Docker / Kubernetes Secrets

In the container world, you have stateless containers, auto-scaling, and all the fun stuff you read on Medium. Loading .env files to containers may work with docker-compose by mounting a volume, but when you have nodes going up and down and have stateless containers without permanent storage, well, it gets complicated.

Fortunately, container orchestration platforms, like Kubernetes and even Docker Swarm, give you an option for keeping secrets in the form of key/value database that you can later attach to your containers in the form of: 1. Environment variables. 2. Mount as files. 3. API access

Now that a service is managing the secrets, there are a few benefits:

1. You need to authenticate to gain access to these secrets. A pod has to have service account permissions to mount a secret.
2. Secrets are centrally stored, for the most part, encrypted at rest, and can serve dynamic resources without your intervention.
3. You now have the option to use API calls to access the secrets from the application, which allows you better support for key and password rotation. Another benefit is that the secret is not loaded to the container as a clear-text file, which makes your life a little easier when it comes to security reviews (though that is not 100% correct due to the secret zero problem).

Using Kubernetes Secrets is a step forward up to a certain scale, but what happens when you have multiple clusters or having other non-containerized services running?

That’s where services such as AWS Secrets Manager, and Hashicorp Vault come in to save the day.

AWS Secrets Manager

Amazon’s solution is one of the first and more mature cloud solutions for secrets management, so I’m going to use it as an example, but you can find similar services in Google’s cloud platform and Azure.

If you are heavily invested in AWS, you are already using IAM (Identity Access Management), and there is a good chance that you rely on managed services such as RDS, EKS, Redshift, etc. That means that you can enjoy the benefits of the ecosystem, and take security to a higher level by utilizing the AWS Secrets Manager service.

The AWS Secrets Manager service is a secure and encrypted key-value storage that you can access via an API. Like any other AWS service, authentication and authorization are done by IAM, which makes this service highly accessible (in a good way) from your infrastructure. You can centrally store credentials, access keys and certificates, and control who can have access to them while keeping a full audit log.

Having a single service that handles all of your secrets across your infrastructure takes a considerable burden off your hands - Dynamic resources such as Auto Scaled instances or containers are fetching their own secrets at startup and runtime. You don’t have to go through all of your services to update a password.

One area that makes AWS Secrets Manager shine is when dealing with password rotation. If you ever had to deal with compliance, this is a big headache that you know you should be doing, but almost no one does. AWS Secrets Manager can rotate your database or one of many other service passwords. Usually, this is a deployment “event” that requires planning and has risk attached.

To sum some of the benefits of using a managed cloud service such as AWS Secrets Manager:

1. A single source of truth for your secrets.
2. IAM for authentication and authorization for accessing secrets.
3. Audit log - required for compliance.
4. Secrets rotation - In my opinion, this is one of the strong points.
5. Easily accessible from all of your cloud environments.
6. Compared to other solutions, such as running your own Vault, saves you the cost of administration.

There are some drawbacks to consider:

1. To enjoy most of the benefits, you have to modify your applications to use the AWS Secrets Manager API. Otherwise, you will not be able to use features like database password rotation without service interruptions.
2. Vendor lock-in - For hybrid cloud or on-prem architectures, this may not be a suitable solution.
3. Single point of failure - If there is a service disruption AWS Secrets Manager, your whole system can be affected. Rare as it is, it’s still something to consider.

The Vault

When you are running on multiple cloud providers, working on-prem, or don’t want to lock yourself down to just one cloud provider, the Hashicorp Vault is a great option. It’s pretty massive, and there is a steep learning curve, which requires thoughtful planning, but in my opinion, it’s worth it.

The Hashicorp Vault is an encrypted key-value storage, that is similar to (and can incorporate) the Consul service. You can use the Vault command line or an API to authenticate and then make requests to store or retrieve secrets.

Other than just storing the secrets, the Vault can also:

1. Create dynamic credentials to your databases - the Vault can create a username and password for each of your application instances.
2. Handle secrets rotation.
3. It makes revocation easy.
4. Act as a Certificate Authority and dynamically issue certificates to your services.
5. Allow you to create custom backends that utilize the benefits of dynamic secrets - For example, create dynamic credentials to your service.
6. It can use Kubernetes as its authentication backend, which makes it a little more like AMI, where a pod can use its authentication token for accessing the Vault.
7. You can control the “unseal” policy.

But again, as always, there are some drawbacks to using the Vault:

1. The hidden cost of maintaining this service.
2. More resources to run the Vault.
3. Steep learning curve.
4. Increased complexity of the system.
5. If the vault cluster is down, depending on your usage, it may cause an outage.

What is the right solution for me?

Well, having A solution in place is a start. I wouldn’t recommend implementing a full-blown vault to support a small application, or if you have a small team to support it. I suggest you go over the list above and consider the pros and cons of each option and just go for it.