SSL Renewals That Just Work: How We Got There

At 02:13 on a Tuesday, an alert popped up in our Discord channel. A client's site was about to lose its HTTPS lock, and the last invoice for a £80 SSL renewal sat in his inbox. He was staring at a 2am email, wondering whether the site would stay online. That moment made us ask: why should any customer have to intervene when a certificate expires?

The answer was simple. We could replace a manual, yearly process with a fully automated ACME workflow that runs without human input. Below we walk through the steps we took, the tools we built, and the monitoring we rely on to keep the cycle smooth.

From Yearly Invoices to Daily Checks

A traditional SSL purchase costs around £80 per year per domain, and renewal requires logging into a control panel, generating a CSR, and clicking "activate". In practice, most owners forget until the certificate is within a few days of expiry. The result is a frantic night-time scramble, or an unexpected browser warning that drives users away.

We started by mapping the exact points where manual work entered the process:

1. Invoice generation - a billing event that happens once a year.
2. CSR creation - a command that must be run on the server.
3. CA interaction - uploading the CSR, waiting for validation, downloading the cert.
4. Web server reload - applying the new cert without downtime.

Each step added latency and risk. Our goal was to collapse them into a single automated loop that runs well before a certificate reaches its 90-day limit.

The ACME Automation Loop

Let's Encrypt provides free, domain-validated certificates that last 90 days. The ACME protocol lets a host request, validate, and download a cert automatically. Our implementation follows the same pattern we outlined in our earlier post about Let's Encrypt automation, but with a few practical tweaks for reliability.

Renewal Timing

We query the expiry date of every active certificate twice a day. If the certificate is within 30 days of expiry, we start a renewal request. This gives us a two-week safety margin even if the ACME server throttles us or a DNS change propagates slowly.

Challenge Placement

The ACME HTTP-01 challenge requires a file at /.well-known/acme-challenge/. Our nginx pod for each site serves that path directly from a temporary directory. The steps are:

# Create a temporary challenge directory
mkdir -p /var/www/challenges

# Tell flame-bubble to bind-mount it into the site container
flame-bubble --bind /var/www/challenges:/var/www/challenges

# Write the challenge token (provided by the ACME client)
echo "TOKEN_DATA" > /var/www/challenges/TOKEN

Once the file is in place, Let's Encrypt validates it, issues the cert, and we store the new PEM bundle in /etc/ssl/tch/DOMAIN/. The old cert is archived for audit purposes.

Nginx Reload

Our pod-nginx runs as an OpenRC-supervised process (Alpine Linux has no systemd) that can be refreshed without dropping connections:

rc-service pod-nginx reload

Because the reload happens after the new cert is written, there is never a window where the site serves an expired cert.

Continuous Monitoring

Automation is only as good as the monitoring that watches it. We built three layers of checks:

1. flame-watchman runs every 5 minutes. It scans the certificate directory, confirms the notAfter date, and flags any cert that is less than 30 days from expiry. If a renewal failed, it creates a ticket in our internal queue.
2. flame-sentinel sends a 60-second heartbeat to a Discord channel. If the heartbeat stops, we know the node is unreachable and can act before any certs on that node are at risk.
3. External probe: a lightweight Go script queries each site over HTTPS and verifies the chain. It runs from a separate monitoring VM in the US node, ensuring we catch issues that might be hidden behind internal routing.

When a failure is detected, we receive a Discord message that reads:

[SSL] Renewal failed for blog.example.com
Reason: DNS propagation delay (TTL 300s)

Because the alert arrives during business hours, we can intervene with a single command to retry the renewal. In practice, the failure rate is under 0.2 % per month, and most of those are resolved automatically on the second attempt.

What Customers Experience

From a client's perspective, nothing has changed. Every site is delivered with HTTPS enabled from day one, and the lock icon stays green. There is no "click to enable SSL" button because the feature is baked into the provisioning script. If a customer adds a new subdomain, the next daily scan will pick it up, request a certificate, and have it live within an hour.

We do not provide wildcard certificates; each domain and subdomain gets its own certificate. This mirrors Let's Encrypt's default behavior and keeps the renewal scope small, which reduces the impact of rate limits.

Trade-offs We Acknowledge

Automating SSL renewal does not eliminate all risk. The 90-day validity means that a broken ACME client could affect many sites at once. Our monitoring mitigates this, but we still advise customers to keep their DNS records stable and to avoid rapid CNAME changes that can cause validation failures.

We also choose not to offer third-party paid certificates. Those often come with longer lifespans but also higher cost and a manual renewal process that we deem unnecessary for our target audience of freelancers and small agencies.

Looking Ahead

Our next improvement will be a built-in fallback that switches a site to a self-signed cert for a brief window if all renewal attempts fail. That will keep the site reachable over HTTPS, albeit with a warning, until we can intervene. It's a small addition that further reduces the chance of a user seeing an "invalid certificate" error.

The journey from a yearly £80 invoice and a 2 am panic email to a zero-action automatic process took a few months of coding, testing, and monitoring tweaks. The result is a system that keeps certificates fresh, sites secure, and owners asleep at night.

If you're curious about the exact commands we run or want to see the monitoring dashboard, feel free to open a ticket in the customer portal. The tooling is our own, documented for customers, and we're happy to share snippets that don't expose secrets.