Building a small uptime monitor, and the gaps its 71 tests missed

A small uptime monitor is one Node process with a timer per URL, a GET with a 10-second timeout, and an SQLite file of results. I built one in 25 commits on its first day. All 71 of its tests pass today, and reading it six months later I still find five gaps that no test covers.

I built it for a client, to watch their production URLs. It is single-user and password-protected, and it posts to Slack when something goes down. This log covers the build. The database choice has its own post, shipping SQLite to production on purpose, from a different app.

One Fastify process, a timer per monitor, one SQLite file

The design spec is 237 lines of Markdown. One Docker container runs one Fastify process, and that process does three jobs: it serves the built React dashboard, it answers the REST API, and it runs the ping worker. The worker keeps a Map of setInterval handles, one per active monitor, with intervals from 30 seconds to an hour. Each tick sends a GET, compares the status code to the expected one, and writes a row to ping_results. An edit through the API restarts that monitor’s timer, so a change needs no server restart.

The first commit, the spec, landed at 12:36 on 2026-04-01. By midnight there were 25 commits: schema, auth, CRUD, the ping worker, the dashboard, a Docker build, a /health route for the host’s health check, monitor groups, response-time limits, SSL expiry checks, 90-day retention and a failure threshold. The repository has 60 commits over 15 days in total. After day one came Slack alerts, multi-region agents, keyword checks, maintenance windows, a public status page, and an analytics page with PDF export.

The spec’s “not in v1” list had four items: Slack alerts, SSL checks, keyword checks and multi-user support. Three shipped within five days, and multi-user never did. The fastest reversal was not on that list. Under data retention, the spec said “Indefinite — all ping results and incidents kept forever”, and the cleanup worker arrived ten hours after the spec.

Three failures to open an incident, one success to close it

The first ping worker opened an incident on the first failed check. That is too sensitive for a production site on the public internet: one dropped connection and the team gets an alert. The same night, each monitor got a threshold, three by default, and the worker now opens an incident only when the last three pings all failed. One good ping closes it.

The asymmetry is deliberate. A false alarm costs attention, but a late recovery notice costs nothing. It has one side effect that the dashboard does not show. The incident row takes its started_at from the third failure, not the first. At the default 60-second interval, every incident duration is two minutes too short, and so is the mean time to recovery on the analytics page. The uptime percentage counts every failed ping, so a single blip lowers uptime but never appears as an incident. The two numbers measure different things.

90-day retention, and the results table it does not cover

At a 60-second interval, one monitor writes 1,440 rows a day and 129,600 rows in 90 days. At 30 seconds, it writes twice that. The cleanup worker runs at startup and every six hours, and deletes ping_results rows older than RETENTION_DAYS (default 90).

Multi-region support arrived the next day. A spoke agent runs in another region, pings its assigned monitors, buffers the results, and posts them to the hub every 10 seconds. They go into a second table, regional_ping_results. The cleanup worker was written before that table existed and was never changed. Regional results are kept forever, and they grow with the number of regions: a monitor checked from three regions at 60 seconds writes 4,320 rows a day that nothing deletes. The retention test passes because it tests the table that has retention.

A 90-day uptime bar drawn from 10,000 rows

The monitor detail page draws a 90-day uptime bar, one cell per day, and a response-time chart with 24-hour, 7-day and 30-day views. Both come from one request: GET /monitors/:id/history, with no date range. The route defaults to 90 days and caps the result at 10,000 rows, newest first.

At a 60-second interval, 10,000 rows are 6.94 days. So the 90-day bar has about seven days of data and about 83 grey “No data” cells, and the 30-day chart shows one week. At 30 seconds, it is three and a half days. The page also polls every 30 seconds, so it fetches those 10,000 rows twice a minute while it is open. The dashboard’s 30-day uptime percentage does not have this problem, because it is a count(*) in SQL. The bar has it because it counts rows in the browser.

The fix is to aggregate where the data is: one query that groups by day and returns 90 rows, not 10,000.

Two timestamp formats in one SQLite database

The hub writes timestamps with SQLite’s datetime('now'), which gives 2026-10-15 14:05:34: UTC, but with a space and no Z. The spoke agents send JavaScript’s toISOString(), which gives 2026-10-15T14:05:34.000Z. Both formats are in the same database, and SQLite compares them as strings.

That caused two client-side fixes. On day one, every time displayed one hour early, because a browser parses a timestamp without Z as local time, and the UK was on summer time. A utcDate() helper now adds the Z when it is missing. On 2026-04-15, the last commit fixed a chart axis that showed “Invalid Date”: the chart stored locale-formatted strings and parsed them back, which fails outside US date order.

One server-side effect is still there. The dashboard and history routes build their date limits with toISOString() and compare them to the hub’s stored values. A space sorts before a T, so in SQLite '2026-07-17 23:59:59' >= '2026-07-17T00:00:01.000Z' is false. Every hub row from the boundary day falls outside the range, and the dashboard’s “last 30 days” figure covers 29 days plus part of the current one. If I started again, I would store epoch milliseconds as integers and format only at display time.

A timeout that stops at the headers

Each ping starts a 10-second AbortController timeout and clears it as soon as fetch resolves. fetch resolves when the headers arrive, not the body. For most monitors that is enough, because the worker never reads the body. A keyword check does read it: after the status code matches, the worker calls response.text() and looks for the expected word. By then nothing can abort the request.

I tested this with a local server that sends a 200 and the first bytes of a page, then never ends the response. The ping code got its headers after 9 ms. response.text() failed after 301,106 ms with UND_ERR_BODY_TIMEOUT, the default 300-second body timeout in Node’s built-in fetch (Node 24.13).

The result row is worse than the wait. The worker sets isUp to true when the status code matches, before it reads the body. The catch block records the error message and the elapsed time, but it does not set isUp back to false. So a page that stalls in the middle of its body is written as up: status 200, a response time of about five minutes, and the error terminated. The timer does not wait for the ping either. At a 60-second interval, about five pings to the same stalled URL run at once. The fix is two lines: clear the timeout after the body is read, and set isUp = false in the catch. The spoke agent has a copy of the same ping code, with the same two lines missing.

Agents that never see an edited monitor

The hub restarts a monitor’s timer when the monitor is edited. A spoke agent does not. Every 60 seconds it fetches its monitor list and compares the IDs with the timers it runs. It stops timers for IDs that are gone and starts timers for IDs that are new. A monitor whose ID is in both lists keeps its old timer, and that timer still holds the old URL, interval, expected status and keyword. An edited regional monitor runs with its old settings in every region until the agent restarts. Nothing on the hub shows this, because a regional result row stores the monitor ID, not the URL or the settings it was checked against.

What I would keep in a small uptime monitor

I would keep the shape: one process, one SQLite file in WAL mode, a timer per monitor, and a short spec as the first commit. It took one day to get a working monitor, and nothing in the list above is a reason for a queue or a second database.

Three of the five gaps have one thing in common. Each is a place where a later feature added a second copy of something: a second results table, a second timestamp format, a second ping loop in the agent. The tests cover the first copy. The check I take from it: when a feature adds a table, a format or a loop, search for every place that handles the first one, and make the new one go through it too.