May 26, 2020

Be rid of database passwords!

I recently published a demo of Hashicorp Vault’s dynamic secret feature.

You can follow along with the demo using all the commands here: https://github.com/florx/secrets-are-hard-demo

If you’re more of a text person than a video person - here’s what I covered and why they’re so cool.

What are they? #

Dynamic secrets are credentials that don’t exist until you request them. Once you request access to a specific system (e.g. a database), the centralised secret management system (in this case Vault) will generate credentials and serve them.

These credentials will be time-limited, and if the requesting system doesn’t check in to say it’s still using them, they will be automatically revoked. This whole process of generation, leasing, and revoking, means any secrets are only available for a very short time and are only kept in memory.

Why is it important? #

Static secrets, the more conventional way of hardcoding a set of credentials (e.g. a username and password to a database) are all too easy to share. These credentials are also very difficult to rotate, do you create a new user, make sure you gave them the same permissions, then swap out the username and password. Or do you simply change the password, how do you change the password and config at the same time without any interruption of service?

Dynamic secrets solves a bunch of these problems; credentials are uniquely generated per client that needs (and is allowed) access. Permissions can be tightly controlled and changed easily (the next time it’s requested, a new set of permissions is given out). And there’s no need to think about password rotation; it happens every time the service requests credentials.

But in production? #

Yes, on my current project, we run this in production for all of our relational databases. We had the classic problem where developers had “borrowed” the username and password from services in production to access the database, to debug issues. So when someone left the project, we needed to roll the passwords - and got stuck in the problem space above.

We tried it out in our development environment for a few weeks, and after we’d tweaked the permissions (we were too aggressive at first, it stopped our migrations from being able to alter tables), once we got it working well - we then rolled it out to all our relational databases across all of our estate.

How do I try it out? #

In the demo video above, I go through these steps, which are also on my GitHub repository: https://github.com/florx/secrets-are-hard-demo. To follow along, you may want to clone it, so you have the various docker-compose files and test data.

$ git clone git@github.com:florx/secrets-are-hard-demo.git

Prerequisites #

You’ll need a few things first:

PostgreSQL command-line tools (brew install postgresql)
Vault (https://www.vaultproject.io/downloads.html)
Docker Desktop (https://www.docker.com/products/docker-desktop)

Setup #

First, we need to start a Vault server to play around with. This instance has basically no security enabled, so please don’t run it in production like this!

$ vault server -dev

Next up in our environment setup steps, we need a database to test with. I’ve picked PostgreSQL simply because I’m most familiar with it. The docker-compose file postgres.yml has all the config we need to start it, with default credentials admin/supersecret.

$ docker-compose -f postgres.yml up

Almost there with setup, let’s populate the database with some random data, then we can tell if our dynamic secrets and permissions actually worked! Naturally, we wouldn’t normally inline the password into an environment variable, but this makes the command copy/pasteable.

$ PGPASSWORD=supersecret psql -h localhost -U admin -d users -a -f users.sql

Last bit of setup, we want to set up our Vault CLI so we can use the vault command and it’ll correctly talk to the one we just setup.

$ export VAULT_ADDR='http://127.0.0.1:8200'

Setup dynamic secrets! #

We first need to enable the secrets database engine, this will allow us to tell Vault how to access and give us credentials to our database.

$ vault secrets enable database

Next, we want to give Vault a way to access our database; we tell it what plugin, roles and connection_url to use, plus some credentials to log in to that database.

This config is how Vault will connect to create and revoke users, for us to access it later.

# Give Vault instructions on how to contact our database
# Note that both allowed_roles=* and sslmode=disable are not secure, and both should not be used in production.
$ vault write database/config/users-database \
    plugin_name=postgresql-database-plugin \
    allowed_roles="*" \
    connection_url="postgresql://{{username}}:{{password}}@localhost:5432/users?sslmode=disable" \
    username="admin" \
    password="supersecret"

The last step of configuring Vault is to describe our role. We can have as many roles per database as we’d like.

In this example, I’ve picked a super limited read-only role. The creation_statements specify in SQL how to create a new user for this specific database type, and any permissions you wish to assign to that user.

The default_ttl and max_ttl define how long the credential is allowed to last before it needs to be renewed (default), and before it is revoked completely (max).

# Create Vault database role for limited read only access to all tables
$ vault write database/roles/read-only-users-database-human-role \
    db_name=users-database \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN \
    PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
        GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="10h"

That’s all the config! Now if we need to get access to this database, we can read database/creds/read-only-users-database-human-role. Vault will automatically connect to our database, then create a brand new username and password for us. This credential will be time-limited to 1 hour initially, but allowed to renew the lease for up to 10 hours. After the 10 hours are up, the user will be removed, and any connections booted from the database.

$ vault read database/creds/read-only-users-database-human-role

Test it out! You’ll be allowed to select * from users but not anything that will change the data, e.g. delete from users.

$ PGPASSWORD=<password-from-above> psql -h localhost -U <user-from-above> -d users

Wrap up #

As you’ve seen from the demo, you no longer have to worry about developers, or services having static credentials to databases. Additionally, their access control can be very limited down to codified Vault roles, and it’s effortless to add a new role if there’s a different access pattern.

We run this in production for all of our relational databases and love it. Give it a try and let me know how you get on! @florx

Kudos