Managing multiple sets of certificates with Let’s Encrypt and Certbot does not have to be complicated. This post contains some of my notes about managing servers with Nginx, Sendmail and Dovecot along with Let’s Encrypt certificates.
Managing permissions
Properly sharing certificates among services require some permission gymnastics to keep everybody happy. Sendmail by default complains if certificates or private keys are publically readable, as do other programs. Opting for the paranoid approach, I adjusted permissions so that only appropriate users within my servers have access to the certificates and keys.
The simplest method I’ve found, is creating a common group and add users smssp, smmta and www-data along with other users who need access to the certificate material. I’ll use that group later to modify file ownership appropriately.
addgroup certs
adduser lem certs
adduser smmsp certs
adduser smmta certs
adduser www-data certs
I also wrote a script I call /etc/certbot-renew-hook which takes care of fixing group ownership and permissions for me. I’ll put this script to use a bit later.
#!/bin/sh
# Ensure that all certificates and related files managed by certbot have proper
# permissions and group ownership
set -e
LEROOT=/etc/letsencrypt/archive
for domain in ${RENEWED_DOMAINS}; do
[ -d ${LEROOT}/${domain} ] && \
( find ${LEROOT}/${domain} -type f | xargs --no-run-if-empty chown smmta:certs ;
find ${LEROOT}/${domain} -type f | xargs --no-run-if-empty chmod o-rwx )
done
exit 0
Generating the certificates with Let’s Encrypt
If you want to request wildcard certificates or authenticate via DNS, you might want to check my post about wildcard certificates and DNS authentication with LetsEncrypt.
My setup shares certificates among servers and services. I like to use the webroot authorization mechanism, which allows me to feed the appropriate webroots for each domain I’m generating certificates for. I then run a command similar to this:
$ sudo certbot certonly --webroot \
-w ${BLOG_WEBROOT} \
-d lem.click,www.lem.click,⋯ \
-w ${LIBERTAD_WEBROOT} \
-d libertad.link,⋯
A successful run looks like this…
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for lem.click
⋮
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0001_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0001_csr-certbot.pem
And produces these files:
$ ls -l /etc/letsencrypt/live/lem.click
-rw-r--r-- 1 root root 543 Aug 20 17:49 README
lrwxrwxrwx 1 root root 33 Aug 20 17:49 cert.pem -> ../../archive/lem.click/cert1.pem
lrwxrwxrwx 1 root root 34 Aug 20 17:49 chain.pem -> ../../archive/lem.click/chain1.pem
lrwxrwxrwx 1 root root 38 Aug 20 17:49 fullchain.pem -> ../../archive/lem.click/fullchain1.pem
lrwxrwxrwx 1 root root 36 Aug 20 17:49 privkey.pem -> ../../archive/lem.click/privkey1.pem
$ ls -l /etc/letsencrypt/archive/lem.click
-rw-r--r-- 1 root root 2122 Aug 20 17:49 cert1.pem
-rw-r--r-- 1 root root 1647 Aug 20 17:49 chain1.pem
-rw-r--r-- 1 root root 3769 Aug 20 17:49 fullchain1.pem
-rw-r--r-- 1 root root 1704 Aug 20 17:49 privkey1.pem
At this point, I change the permissions with /etc/certbot-renew-hook. Later, when Certbot renews my certificates automatically, the hook script will maintain the permissions I need for everything to continue working.
# RENEWED_DOMAINS=lem.click /etc/certbot-renew-hook
# ls -l archive/lem.click/
total 16
-rw-r----- 1 smmta certs 2122 Aug 20 17:49 cert1.pem
-rw-r----- 1 smmta certs 1647 Aug 20 17:49 chain1.pem
-rw-r----- 1 smmta certs 3769 Aug 20 17:49 fullchain1.pem
-rw-r----- 1 smmta certs 1704 Aug 20 17:49 privkey1.pem
My Let’s Encrypt installation provided a simple crontab file at /etc/cron.d/certbot, to which I added the commands required to restart Sendmail, Dovecot and Nginx upon renewing the certificates. The command look like this and you can customize it for your own environment:
certbot -q --renew-hook /etc/certbot-renew-hook renew && systemctl restart nginx.service && systemctl restart sendmail.service && systemctl restart dovecot.service
Configuring Sendmail
I would like my Sendmail instances to use the certificate I created for lem.click – I placed all hostnames related to mail transmission in that certificate, so I would be presenting valid certificates to other MTAs and to my SMTP clients.
To do this, simply place the proper symlinks from /etc/mail/tls to the full certificate chains that certbot provided and restart Sendmail so that the new key material is loaded and ready to use:
cd /etc/mail/tls
rm -f *.crt *.key *.pem
ln -s /etc/letsencrypt/live/lem.click/fullchain.pem sendmail-client.crt
ln -s /etc/letsencrypt/live/lem.click/fullchain.pem sendmail-server.crt
ln -s /etc/letsencrypt/live/lem.click/privkey.pem sendmail-common.key
systemctl restart sendmail.service
I can simply verify that the new certificate (and the complete chain) is being served by Sendmail using the openssl command:
# openssl s_client -starttls smtp -connect mx.libertad.link:25
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = lem.click
verify return:1
---
Certificate chain
0 s:/CN=lem.click
i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
2 s:/O=Digital Signature Trust Co./CN=DST Root CA X3
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
⋮
Notice how the chain includes our newly created certificate siged by the intermediate, the intermediate certificate and the self-signed root CA. The root CA does not seem to be required, as this would be known to the peer.
Configuring Dovecot
In my Dovecot setup I placed all the certificate-related files under /etc/dovecot/tls, although you can simply load the right certificates and keys from the configuration file. The relevant snippet of my configuration is this:
ssl_cert = </etc/dovecot/tls/server-cert.pem
ssl_key = </etc/dovecot/tls/server-key.pem
Dovecot has the usual knobs for configuring SSL/TLS which are not relevant to this post. Defaults should work well.
After making the above setting, ensure that the proper key material is symlinked where Dovecot expects it and restart the service, all running as root:
[ -d /etc/dovecot/tls ] || mkdir -p /etc/dovecot/tls
cd /etc/dovecot/tls
rm -f *.crt *.key *.pem
ln -s /etc/letsencrypt/live/lem.click/fullchain.pem server-cert.pem
ln -s /etc/letsencrypt/live/lem.click/privkey.pem server-key.pem
systemctl restart dovecot.service
Again we check that the chain is present. Dovecot is not sending the self-signed root CA, which is fine as this should be already in the trust store fo the client.
# openssl s_client -connect imap.lem.click:993
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = lem.click
verify return:1
---
Certificate chain
0 s:/CN=lem.click
i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
⋮
Configuring Nginx
Nginx configuration is tricky for virtual hosted environments. In order for TLS to work here, the client and server need to agree on which certificates to use. Here’s where Server Name Indication comes in. Think of this as a low-level ServerName directive but for TLS.
To start, I symlinked all the certificates under /etc/nginx/tls because I like consistency. Then I provided mnemonic symlinks for each certificate group. This is handily done by these commands, running as root:
[ -d /etc/nginx/tls ] || mkdir -p /etc/nginx/tls
cd /etc/nginx/tls
for d in /etc/letsencrypt/live/*; do ln -s $d; done
With the symlinks in place, go into each one of your server definitions and name the certificate and key files. This will automatically enable SNI support in Nginx.
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name lem.link ⋯;
⋮
ssl_certificate /etc/nginx/tls/lem.click/fullchain.pem;
ssl_certificate_key /etc/nginx/tls/lem.click/privkey.pem;
⋮
}
Note the use of fullchain.pem so that the web server returns the complete certificate chain, allowing clients to authenticate the endpoint certificate. The only thing left to do is restart Nginx to force reloading of the new cert material.
sudo systemctl restart nginx.service
Then you can easily use openssl to verify that the expected certificates are being used.
# openssl s_client -connect lem.click:443
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = athena.pics
verify return:1
---
Certificate chain
0 s:/CN=athena.pics
i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
⋮