I recently decided to set up my website to use HTTPS. Configuring Github Pages - which hosts this blog - was easy . I just had to follow the Github Pages documentation on troubleshooting custom domains which directed me to update the
A records in veryjoe.com’s domain configuration, then check a checkbox.
Dokku - which hosts my side project web apps - should have been just as easy. There is Dokku plugin called dokku-letsencrypt which lets you automatically register and configure Let’s Encrypt SSL certificates. It promises to get you set up with just 3 commands, but I ran into two issues.
- It fails if you’ve misconfigured your app’s domain in Dokku.
- It fails if your apps have no network listeners.
I’m not going to dig into the precise causes of these errors in this post, since I didn’t do that in real life. But in the unlikely event that you are currently dealing with the same issues, hopefully this helps you debug.
1. Dokku domain misconfiguration
The first site I tried to configure was js.apps.veryjoe.com. Running
dokku letsencrypt js produced following error:
ACME server returned an error: urn:acme:error:malformed :: The request message was malformed :: Error creating new authz :: Name does not end in a public suffix
This suggested something was up with Dokku’s Domain name configuration. I investigated with
=====> js domains information Domains app enabled: true Domains app vhosts: js.apps Domains global enabled: true Domains global vhosts: apps root@apps:~# dokku domains:add-global
Apparently I had misconfigured Dokku, supplying it with just
apps as the domain name instead of
apps.veryjoe.com, so the plugin was trying to request a certificate for
js.apps, and Let’s Encrypt wasn’t having it.
I tried to reset the global domain configuration, but that didn’t actually change the configuration for the
root@apps:~# dokku domains:remove-global apps.veryjoe.com -----> Removed apps root@apps:~# dokku domains:add-global apps.veryjoe.com -----> Added apps.veryjoe.com root@apps:~# dokku domains:report =====> js domains information Domains app enabled: true Domains app vhosts: js.apps Domains global enabled: true Domains global vhosts: apps.veryjoe.com
I figured that perhaps I just had to clear the app-specific configuration too, and I was right.
root@apps:~# dokku domains:clear js root@apps:~# dokku domains:report =====> js domains information Domains app enabled: true Domains app vhosts: js.apps.veryjoe.com Domains global enabled: true Domains global vhosts: apps.veryjoe.com root@apps:~# dokku letsencrypt js =====> Let's Encrypt js ... -----> Certificate retrieved successfully.
In conclusion, Name does not end in a public suffix is pretty self explanatory. You just need to get your domain names in order!
2. missing network listeners
Next I tried diff.apps.veryjoe.com. The error follows (emphasis added):
Unable to reach http://diff.apps.veryjoe.com/.well-known/acme-challenge/zWbBBKkpaQD-6O0NXO8FWktijAIKSpDMWI-2K9MlrVw: HTTPSConnectionPool(host=’diff.apps.veryjoe.com’, port=443): Max retries exceeded with url: /.well-known/acme-challenge/zWbBBKkpaQD-6O0NXO8FWktijAIKSpDMWI-2K9MlrVw (Caused by SSLError(CertificateError(“hostname ‘diff.apps.veryjoe.com’ doesn’t match ‘js.apps.veryjoe.com’“,),))
diff.apps.veryjoe.com was not successfully self-verified. CA is likely to fail as well!
Generating new certificate private key CA marked some of the authorizations as invalid, which likely means it could not access http://example.com/.well-known/acme-challenge/X. Did you set correct path in -d example.com:path or –default_root? Is there a warning log entry about unsuccessful self-verification? Are all your domains accessible from the internet? Failing authorizations: https://acme-staging.api.letsencrypt.org/acme/authz/rx8o_z-L66gQgOVHjWWUoLDlIb9vOEPzTBqPvxnYkEM
Let’s Encrypt tries to access a file which the dokku-letsencrypt plugin hosts in order to prove that I own the domain, but it can’t find it.
Debugging the network error
First I tried looking up “dokku letsencrypt hostname doesn’t match” and found this issue which recommended following some instructions in the plugin’s readme. They didn’t work but I noticed something suspicious while I was following them:
root@apps:~# dokku proxy:ports-add diff http:80:5555 ! No web listeners specified for diff
No web listeners specified suggests some kind of network configuration error. I googled around and found a github issue and a tutorial which connected nginx to issues with dokku-letsencrypt. So why was
diff broken in this way but not
Eventually I ran
root@apps:~# dokku network:report =====> alcoholculator network information Network bind all interfaces: false Network listeners: =====> diff network information Network bind all interfaces: false Network listeners: =====> js network information Network listeners: 172.17.0.4:5000 Network bind all interfaces: false =====> paint network information Network listeners: 172.17.0.2:5000 Network bind all interfaces: false =====> thumbnailer network information Network listeners: 172.17.0.3:5000 Network bind all interfaces: false
Now I had a hypothesis:
alcoholculator should also break, and
paint should succeed.
Unfortunately, I hit a snag at this point.
Avoiding Let’s Encrypt rate limits
Let’s Encrypt only lets you try to register certificates a few times every three hours. Running
dokku letsencrypt paint:
There were too many requests of a given type :: Error creating new registration :: too many registrations for this IP: see https://letsencrypt.org/docs/rate-limits/
The solution suggested by the link is to use the staging environment while you experiment.
dokku config:set --no-restart paint DOKKU_LETSENCRYPT_SERVER=staging dokku config:set --no-restart alcoholculator DOKKU_LETSENCRYPT_SERVER=staging
Solving the issue
At this point I was able to prove my hypothesis. Running against their staging environment,
paint successfully registered for a certificate, and
alcoholculator encountered the same error as
As I began to search for issues related to Dokku’s static site buildpack and letsencrypt, I realized that that the latest version of the buildpack actually lives at https://github.com/dokku/buildpack-nginx now. The instructions there tell you to just put a file called
.static in the root of your repository. I did that for
diff and pushed the changes. Now it had a network listener:
root@apps:~# dokku network:report diff =====> diff network information Network bind all interfaces: false Network listeners: 172.17.0.6:5000
dokku-letsencrypt diff ran successfully against the staging environment:
root@apps:~# dokku letsencrypt diff -----> Certificate retrieved successfully.
I decided that it wasn’t worth digging into why exactly the old version was breaking things. Sometimes things break, you update them, and then they’re fixed 🤷♂️.
alcoholculator in the same way, I just had to wait 3 hours for Let’s Encrypt’s rate limits to reset and then I was able to set everything up.
Ultimately both of the problems here were of my own making: misconfiguration and outdated dependencies. The dokku-letsencrypt plugin is pretty great (this tutorial suggests that a Let’s Encrypt certificate would be significantly more painful to set up without it!) but no software can completely prevent user error. With decent error messages and warnings, I was able to web-search my way to solutions. I hope you enjoy your secure connection to my side projects!