Implementing TLSA or HPKP for certificate pinning while using automated certificate authorities such as Let’s Encrypt can be tricky. These notes explain how I do it on my servers, using GnuTLS to do the heavylifting.

Certificate pinning is a technique allowing server operators to specify an identifying characteristic about the certificate to be used to establish TLS connections. My favorite use case involves identifying the server certificate itself – that is, the leaf certificate – so that the verification mitigates the risk of compromising the issuing CA.

However this can be a bit tricky as by default, Certbot generates a new key-pair and CSR automatically. This would be fatal with HPKP, which precludes the browser from updating the certificate pinning until the validity window has expired. We need to somehow allow Certbot to do its thing while at the same time, maintaining the same set of keys around so that pinning can work.

As a measure of safety it’s always a good idea to authorize more than one certificate. This allows for an easy recovery path in case the private key of your active certificate is compromised or lost. I like authorizing 4 at a time. This gives me a few tries to resolve any issues that can arise. Hopefully.

Generate multiple CSRs

I’ll use GnuTLS’ certtool to generate suitable CSRs. The goal is to obtain a group of certificate keys and CSRs, which can then be submitted for signing by the CA. In order to do this, I conceived a simple filesystem layout that you can see at https://github.com/nerdlem/certs – This allows me to keep my certificates and keys in a safe offline location.

Follow the instructions in the README.md file to create a directory for your new certificate, do the rsync and edit template.conf to suit. Then simply run make and ensure that the process generated the CSRs and keys as expected.

During normal operation, your server will need the CSR, private and public key for the current certificate and the remaining public keys. The remaining private keys must be kept in a separate safe location, so that you can do a key rotation in case your current key is compromised or lost.

A nice thing about GnuTLS is that the files it generates include a human-readable dump of the PEM-encoded material. This makes the files easier to inspect while maintaining compatibility with OpenSSL and other TLS implementations that adhere to the PEM specification.

Generate / renew Let’s Encrypt certificate

To achieve full control over the process, I’ll have certbot issue certificates periodically via cron. These certificates will need to be moved to the expected location and services restarted accordingly. This is a slightly more involved process than the standard automated process, but one that would ensure that I can use certificate key pinning effectively over time.

To carry on this task, I added certbot-auto-renew.sh to my repo at https://github.com/nerdlem/certs, which takes care of renewing the certificates. A few files and directories are needed though.

CSR and public keys

Those are kept under directories of the form /etc/letsencrypt/seed/<domain>/. You can easily upload the required material via a command like this

make HOST=${yourserver} upload

Webroot parameters

To manage the authentication for issuing certificates, you need to keep a file certbot-params on directories of the form /etc/letsencrypt/webroot/<domain> that will contain the required command line parameters to complete the certbot authentication. Here’s the file I’m currently using my lem.click certificate:

--webroot -w /path/to/my/webroot -d lem.click,lem.link,blog.lem.link,blog.lem.click,www.lem.link,www.lem.click

Perhaps later I’ll add hooks to support other forms of authentication which might negate the need for this, but for now this is good enough for what I need.

Crontab entry

I use a crontab(5) file like the following to take care of my certificate rotation.

31 */12 * * * root /usr/bin/certbot-auto-renew.sh
27 3 * * *    root MINDAYS=10 /usr/bin/auto-check.sh

The auto-check.sh is a simple script that uses the ssl-cert-check utility to verify certificate expiration. It’ll email me if certificates get dangerously close to expiration.

This is how I’m managing my certificates and I hope this is useful for you. If you liked this post, please take the time to share it so that others can also find it. Comment or ideas? Hit me up on Twitter.