How I run production infrastructure as a one person consultancy

There is a question everyone has about hiring one person to build and run something, and almost none of them ask it out loud. It goes something like: this is fine while he is here, but what happens after he leaves?

It is a fair question. A one person consultancy has no rota, no ops team and no on call escalation. If the answer to 'who is watching this' is 'me, when I remember', then what you bought is not a product. It is a liability with a launch date.

So here is what I actually run, how it is watched, and what I have got wrong along the way. The worked example is a small tool I wrote for myself called Binnacle. It exists because two things that matter were not being checked by anything, and I only found out by looking.

What is being run

Two hosts.

The first is a single well specified server that I own and control. It runs every application: my own products, the marketing sites, and a staging environment for each. It also runs the monitoring stack (Prometheus, Grafana, Loki). Everything is deployed with Kamal and fronted by kamal-proxy, which is the only thing on the box that listens on the public internet. Nothing reaches an application except through it.

The second is a very small cloud box, on a different network with a different provider. It runs two things: Binnacle, and Uptime Kuma. That is all it does, and it does it on purpose.

The separation is the first rule. A monitor cannot live on the thing it watches, because a host cannot report its own death. If the main server goes down, the thing that tells me about it has to be somewhere else.

At the time of writing that is nineteen projects across production and staging, every public domain polled, and every Postgres database backed up nightly.

The two gaps

I already had Prometheus, Grafana, Loki and Uptime Kuma. On paper that is a lot of monitoring for one person. In practice two things were not covered at all, and I did not notice for a while because nothing was telling me.

Nothing recorded deploys. Kamal deploys are a command you run in a terminal. Once the terminal is closed, there is no record of what version went out, to which host, when, or how long it took. When something broke, 'what changed and when' was a question I could not answer without archaeology in a git log.

Nothing confirmed the backups. A cron job runs at three in the morning, finds every Postgres container on the host, runs pg_dump, compresses it and uploads it to S3. It had been running for months. The script ended in an unconditional exit 0. That means cron recorded success every single night, whether the backups had landed or not. There was no notification path. Nothing had ever checked that a backup existed, was recent, and was not an empty file.

I want to be plain about this because it is the kind of thing consultancies do not put in articles. A backup you have never verified is a hope, not a backup. I had a hope.

What Binnacle does

Binnacle is one Go binary and a Postgres database. It does the two things nothing else covered, and it links into everything else.

Deploy history. Every tracked project has a Kamal post-deploy hook. It is the same file in every project, byte for byte. The only thing that differs is a token in the project's secrets, which tells Binnacle which project the deploy belongs to. When a deploy finishes, the hook posts the version, the hosts, who ran it and how long it took. The deploy shows up on the dashboard within a second.

The hook has a five second timeout and always exits zero. Kamal aborts a deploy if a hook fails, so a Binnacle outage must never be able to break a deploy. The monitor is allowed to be down. The deploys are not allowed to notice.

Backup verification. For every project with a database, Binnacle lists the S3 prefix and checks three things. The newest backup is recent (within 26 hours, to suit a 3am cron). It is over a minimum size, which catches a successful dump of an empty database. And roughly seven backups landed in the last seven days, which catches intermittent failures that a freshness check alone would miss.

Binnacle's AWS credentials can list the bucket and nothing else. It cannot download a backup. If it is ever compromised, the attacker learns filenames and sizes.

Health and certificate checks. Every endpoint is polled concurrently over HTTPS. The certificate expiry is read from the TLS handshake on the same request, with a warning at fourteen days. Projects often have more than one domain, each with its own certificate, and each is checked on its own.

Alerts. One Slack message when an incident opens. One when it recovers. Never one per failed check. A monitor that cries wolf gets muted, and a muted monitor is worse than none. The alerting logic is a pure state machine with no I/O, and it is the most heavily tested part of the codebase, because it is the part most likely to be subtly wrong.

The rollback line. When something is broken, the dashboard shows the current version, the previous known good version, and the exact kamal rollback command to paste. It does not run it. Binnacle holds no credentials for any host. A web page that can deploy is, if compromised, root on the box. The hard part during an incident is knowing which version to go back to, not typing the command.

Why it links into Grafana rather than replacing it

The first thing I considered building was a dashboard. Then an uptime monitor. I rejected both, because Grafana and Uptime Kuma already existed and already worked.

The actual problem was that they were going unused. Onboarding a new project into them cost too much, and nothing ever drove me to them with context. When something went wrong I would think 'the logs are in Loki somewhere' and then spend five minutes remembering the right label and the right time range.

So Binnacle stores metrics and logs for nothing. It stores the three stable identifiers per project that Grafana and Loki need, and every project page carries links into them with the container and the time range already applied. Deploys, backups and health live in Binnacle. Everything else lives where it already lived, one click away, with the filter already set.

That is a smaller tool than the one I nearly built, and it gets used. The one I nearly built would have been a third dashboard nobody looked at.

The second opinion, and the day it did not help

Uptime Kuma runs next to Binnacle on the small box, watching every production endpoint and kamal-proxy itself. That is deliberate redundancy. Binnacle once silently stopped alerting because of a malformed webhook, and Kuma was what noticed. Overlapping alerts are the price of a second opinion.

Then on 2 September the estate went down for about 65 minutes after a reboot. The cause was a boot order race on port 443 between two services that both wanted it, and the fix is two small systemd units that are now documented, with the date, and the exact commands to confirm a boot is healthy.

But here is the part that matters for this article. Both monitors detected the outage correctly. Both posted to Slack. Both got a 200 OK back. And neither reached me.

Two monitors delivering to one medium are not two opinions. That was the real gap, and no amount of monitoring on the host would have closed it. What is missing is a second delivery route, not a third monitor. That is the next thing on the list, and I would rather say so than pretend the setup is finished.

What 'supported after launch' actually means

This is the answer to the question nobody asks.

When I build or run something for a client, the same kit goes with it. Whether it lives on my hosts or on theirs, it ships with:

• A deploy hook, so every release is recorded with a version, a performer and a timestamp. 'What changed' has an answer.
• Backups that are verified, not backups that are configured. There is a difference, and I have been on the wrong side of it.
• Health and certificate checks on every domain, with a warning before a certificate expires rather than a morning after.
• A rollback line ready to paste, with the previous known good version already worked out.
• A written topology. What runs where, how each thing is reached, and how each thing is watched. Every claim carries the date it was last checked, and anything not verified is marked as such rather than left to look confirmed.

That last one is the real answer to 'what happens after he leaves'. The documentation is written so that someone who is not me could take the estate over. The monitor holds no credentials, so handing it to another engineer does not hand them the keys. The hook is a plain shell script that anyone can read in a minute.

None of this is clever. Most of it is a small Go binary and a habit of writing down what I checked and when. But it is the difference between software that has a person behind it and software that has an email address.

If you are weighing this up

The question behind 'what happens after he leaves' is a question about risk, and it deserves a concrete answer rather than reassurance. Mine is above. If you have something in mind, start with the scoping sprint. One week, £1,950 + VAT, and you get the spec, the architecture, the delivery plan and a quote. If we go ahead, the full amount is credited against the work. If we do not, everything the week produced is yours to take elsewhere.