Skip to content
Serverküche
Search

Loading search …

Applications Difficulty: Intermediate

Healthchecks: Monitor Cron Jobs & Backups (Dead Man's Switch)

Notices when a backup does NOT run: Healthchecks monitors cron jobs via a dead man's switch and alerts on failure – self-hosted behind Traefik.

· Last updated: · 9 min read ·Duration: approx. 45 minutes
Table of contents

The most dangerous failures are the silent ones: the nightly backup hasn’t run for three weeks – and no one notices until the data is actually needed. Uptime Kuma tells you when a service is up. Healthchecks tells you when a job didn’t run. That difference is what saves data.

What are we building?

A self-hosted Healthchecks server (v5.0.3) behind Traefik that works as a dead man’s switch: every monitored job “checks in” after a successful run with a short HTTP ping. If that ping is missing (because the job crashed, the server was off, or the cron entry is gone), Healthchecks raises an alert. By the end you monitor your Restic backups, database dumps and any other recurring job with it – and get notified before the absence becomes a problem.

Prerequisites

🍳 Recommendation Ad

VPS 1000 G12.5

4 vCores · 8 GB RAM · 128 GB SSD

from €14.50/month

Healthchecks is frugal and runs happily alongside your other stack.

Go to netcup →

💶 1 month free:

Single use, valid for VPS 1000 G12.5. Redeem in the cart →

💶 €5.95 voucher for new netcup customers: always valid · not for domains or VPS Lite

All netcup vouchers →

Step by step

Step 1: Understand the principle – monitoring, inverted

Classic monitoring actively asks: “Does the service respond?” Healthchecks inverts that: the job checks in with the server. Each check has a unique ping URL. After a successful run the job calls that URL. Healthchecks expects the ping within a defined window (period) plus a tolerance (grace time). If the ping doesn’t arrive in time, the check goes “down” and Healthchecks alerts. That’s the dead man’s switch: it’s not the presence of a signal that triggers the alarm, but its absence.

Step 2: Create the Compose file

Healthchecks is a Django application; we run it with SQLite – perfectly sufficient for a typical self-hosting scale. Create the project:

Terminal
mkdir -p /opt/healthchecks/data && cd /opt/healthchecks
chown -R 1000:1000 data

The compose.yaml – replace YOUR_DOMAIN and generate your own SECRET_KEY (openssl rand -hex 32):

YAML
services:
  healthchecks:
    image: healthchecks/healthchecks:v5.0.3
    restart: unless-stopped
    user: "1000:1000"
    volumes:
      - ./data:/data
    environment:
      SITE_ROOT: https://YOUR_DOMAIN
      SITE_NAME: Serverkueche Healthchecks
      ALLOWED_HOSTS: YOUR_DOMAIN
      DEBUG: "False"
      REGISTRATION_OPEN: "False"
      DB: sqlite
      DB_NAME: /data/hc.sqlite
      SECRET_KEY: "YOUR_RANDOM_KEY"
    networks: [proxy]
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.hc.rule=Host(`YOUR_DOMAIN`)"
      - "traefik.http.routers.hc.entrypoints=websecure"
      - "traefik.http.routers.hc.tls.certresolver=le"
      - "traefik.http.services.hc.loadbalancer.server.port=8000"

networks:
  proxy:
    external: true

With SQLite, DB_NAME is mandatory

Without DB_NAME: /data/hc.sqlite Healthchecks creates the database in a non-writable location and fails to start with “unable to open database file”.

Without REGISTRATION_OPEN, sign-up is open to everyone

Healthchecks defaults REGISTRATION_OPEN to True. Without the line above, the login page carries a Sign Up button, and anyone who knows your address can create an account on your instance. An account created that way really does end up in the database – even if sending the confirmation mail fails for lack of SMTP. With REGISTRATION_OPEN: "False" the button disappears and /accounts/signup/ answers with 403 Forbidden.

Step 3: Start and log in

Terminal
docker compose up -d

The image ships a health check; Traefik only routes once the container is healthy – that takes just a few seconds. Check:

Terminal
docker compose ps --format 'table {{.Name}}\t{{.Image}}\t{{.Status}}'
Ausgabe
NAME                          IMAGE                              STATUS
healthchecks-healthchecks-1   healthchecks/healthchecks:v5.0.3   Up 42 seconds (healthy)

On start, Healthchecks checks whether the hostname from SITE_ROOT is listed in ALLOWED_HOSTS. If it isn’t, the container aborts with the error hc.api.E002 and sits in a Restarting loop – Traefik finds no running service, and your domain only returns a Traefik 404. Both variables therefore have to carry the same hostname, with ALLOWED_HOSTS written without https://.

Healthchecks does not ship an account – on start the image only sets up the database. So create your access once yourself (replace the password):

Terminal
docker compose exec healthchecks python manage.py createsuperuser \
  --email admin@YOUR_DOMAIN --password YOUR_STRONG_PASSWORD
Ausgabe
System check identified some issues:

WARNINGS:
?: (hc.api.W002) No SMTP configuration, cannot send email
	HINT: See https://github.com/healthchecks/healthchecks#sending-emails
?: (mail.W001) Your MAILERS setting has no 'default' entry. Sending email without a valid mailer will fail.
	HINT: Add a 'default' entry to MAILERS.
Superuser created successfully.

Both warnings are normal at this point: no mail server is configured yet. You add that in step 7 – it makes no difference to creating the account.

SUPERUSER_EMAIL has no effect

Many guides list SUPERUSER_EMAIL and SUPERUSER_PASSWORD as environment variables in the Compose file. The official image does not read them – before starting its services, its startup script only runs manage.py migrate. Anyone relying on that ends up at a login page with no account behind it. The createsuperuser route above is the reliable one.

Now open https://YOUR_DOMAIN/ and log in with those credentials:

The Healthchecks login page under your own HTTPS domain
The self-hosted Healthchecks login – no Sign Up button, because REGISTRATION_OPEN is set to False

Step 4: Create a project and your first check

After login create a project via New Project… (e.g. “Serverküche”) and inside it a first check via Add Check. Give it a descriptive name, tags and a schedule – period = expected interval between two runs (for a daily backup: 1 day), grace time = how long Healthchecks waits after the due time before alerting (e.g. 1 hour). The overview shows all checks with status, ping URL and last ping:

The Healthchecks overview with several checks, ping URLs and status indicator
The check overview: a green tick for \"up\", plus ping URL, schedule and last ping

Each check gets its own ping URL of the form https://YOUR_DOMAIN/ping/<UUID>. Clicking a check opens the detail page with instructions, history and status:

The detail page of a check with ping URL, current "up" status and event log
The detail page: ping URL, current status and the log of received pings

Step 5: Have a job send the ping

Now the core. You make your job call the ping URL after a successful run. The simplest example – at the end of your script:

Terminal
curl -fsS -m 10 --retry 5 https://YOUR_DOMAIN/ping/YOUR_CHECK_UUID

-fsS keeps curl quiet but reports errors; -m 10 aborts after 10 seconds; --retry 5 catches brief network hiccups. Even better: report the job’s exit code, so a failed run shows up as an error immediately instead of “no ping”:

Terminal
#!/bin/bash
URL="https://YOUR_DOMAIN/ping/YOUR_CHECK_UUID"
# ... your actual job runs here ...
restic backup /important/data
# report the exit code to Healthchecks (0 = ok, otherwise failure)
curl -fsS -m 10 --retry 5 "$URL/$?"

Step 6: Monitor Restic backups

This is the showcase. If your Restic backups run via a systemd timer, you add the ping at the end of the backup script. If the backup doesn’t run (timer disabled, server off, script crashed), the ping is missing – and after the grace time expires, Healthchecks alerts. That way you learn about a dead backup within hours, not at data-loss time.

Use start and failure signals

Healthchecks can do more than “done”: a ping to .../ping/UUID/start before the job additionally measures the run time, a ping to .../ping/UUID/fail actively reports a failure. So you see not just whether but also how long a job ran – useful for spotting backups that are getting slower over time.

Step 7: Set up notifications

An alert is only useful if it reaches you. Under Integrations you connect channels: email, ntfy, Gotify, webhooks and many more. For the self-hosting stack ntfy is the obvious choice – push to your phone, without a third-party service. Set up at least one channel and assign it to your checks, otherwise the “down” status stays silent.

If you also want to be alerted by email, the image is still missing your mail server’s credentials. They belong in the environment block of the compose.yaml:

YAML
      EMAIL_HOST: mail.YOUR_DOMAIN
      EMAIL_PORT: "587"
      EMAIL_HOST_USER: YOUR_SMTP_USER
      EMAIL_HOST_PASSWORD: YOUR_SMTP_PASSWORD
      EMAIL_USE_TLS: "True"
      DEFAULT_FROM_EMAIL: healthchecks@YOUR_DOMAIN

After docker compose up -d, check whether Healthchecks accepts the settings:

Terminal
docker compose exec healthchecks python manage.py check
Ausgabe
System check identified no issues (0 silenced).

Without EMAIL_HOST the same command reports (hc.api.W002) No SMTP configuration, cannot send email instead – and sending via manage.py sendtestemail aborts with MailerDoesNotExist: "The mailer 'default' is not configured.".

On a netcup server, SMTP is closed at first

netcup assigns new servers the “netcup Mail block” policy by default, which blocks outgoing ports 25, 465 and 587. Healthchecks then reports no configuration error – delivery simply runs into a timeout. You remove the policy from the server yourself in the SCP, see set up the netcup firewall. If you’d rather not, stick with ntfy: it runs over HTTPS and is unaffected by the block.

When things go wrong

Container won’t start: “unable to open database file”. DB_NAME: /data/hc.sqlite is missing or the data folder isn’t owned by UID 1000. Set the path and chown -R 1000:1000 /opt/healthchecks/data (see the warning in step 2).

The container shows Restarting and the domain returns 404. docker compose logs shows (hc.api.E002) The hostname in settings.SITE_ROOT is not found in settings.ALLOWED_HOSTS. Put exactly the hostname from SITE_ROOT into ALLOWED_HOSTS – without https:// – and restart with docker compose up -d.

Login fails with 403 (CSRF error on submit). Healthchecks then treats the request as plain HTTP, and Django rejects the form because the https://… origin doesn’t match. The image detects HTTPS via the X-Forwarded-Proto header that Traefik sends – if it’s missing, for instance behind a differently configured proxy, you get this error. A CSRF_TRUSTED_ORIGINS variable doesn’t help: Healthchecks doesn’t read it.

The check won’t turn “green” even though the job runs. Check whether the curl ping actually runs and hits the right UUID: curl -v https://YOUR_DOMAIN/ping/UUID should return OK. On the detail page the log shows whether and from which IP pings arrive.

I get no notification on “down”. No integration channel is assigned, or (for email) the SMTP settings are missing. Set up a channel under “Integrations” and assign it to the check.

Your problem is not listed? Search all error messages →

Maintenance & backups

  • Updates. Occasionally bump the image tag (healthchecks/healthchecks:v5.0.3) to the current version and docker compose up -d; the database migrates automatically at start. Since v5.0 the image runs gunicorn instead of uWSGI – if your Compose file has a custom command: calling uwsgi, change it to /opt/healthchecks/run.sh; the file from this tutorial needs no change. Your normal update process handles the rest.
  • Backup. The entire state lives in the SQLite file under data/ – add it to your Restic backup. A small but neat side effect: Healthchecks then monitors the backup that backs it up – close the loop by having the backup job also ping a Healthchecks check.
  • Who watches the watchman? Healthchecks itself has to be running to alert. So add it to Uptime Kuma as an HTTP monitor – the two tools then cover each other: Kuma checks that Healthchecks is reachable, Healthchecks checks that your jobs ran.

You might also like