My load test was lying to me, and I almost believed it

Before I moved Impulseye's backend onto a real €10/month server, I wanted a number: how many monitors can this box actually handle before it falls over? So I built a load-testing rig to find out. The rig gave me an answer. The answer was wrong, twice, and both times for reasons that had nothing to do with the server.

The setup

The idea was simple: run the real, unmodified scheduler and worker code against a fully isolated Docker stack — its own Postgres, its own Redis, and a dummy HTTP target standing in for “the internet” so the test measured my own scheduler-to-worker pipeline, not some third party’s network variance. Seed N fake monitors, watch the queue for a few minutes, report whether throughput kept up with the schedule or the backlog grew.

Nothing about that plan was wrong. The bugs were in the twenty percent of the rig I’d written quickly because it “was just test scaffolding.”

Bug one: the target that wasn’t there

The first real run reported almost total failure. Every single check — regardless of which of the 500 seeded monitors it was for — came back as a timeout, at almost exactly the same latency: ~5000ms, the axios timeout I’d set. Not slow. Not occasionally failing. Uniformly, precisely timing out, as if nothing on the other end was answering at all.

That uniformity was the tell, but it took me a minute to trust it over the more obvious-looking explanation. The queue was backing up, throughput was near zero — every surface-level signal said “the worker fleet can’t keep up.” It took reading the actual container list, not just the metrics, to notice the real story:

The crash itself was almost funny once I found it. The dummy target was a five-line Express app: match any path, respond 200 after a short random delay.

app.get("*", (req, res) => {
  const delay = 20 + Math.floor(Math.random() * 130);
  setTimeout(() => res.status(200).send("ok"), delay);
});

That bare "*" wildcard is completely standard Express 4. But the Dockerfile pinned express@5, and Express 5 ships a newer path-to-regexp that no longer accepts an unnamed wildcard — it throws at startup instead:

PathError [TypeError]: Missing parameter name at index 1: *
    at pathToRegexp (path-to-regexp/dist/index.js:274:5)

The container died before it ever bound to a port, and because nothing was watching it, it just stayed dead. Every one of the 500 seeded monitors spent the entire test run trying to reach a service that had never actually started. The fix was a one-line syntax change — Express 5 wants the wildcard named, e.g. /*splat— plus a restart policy so a future crash can’t silently sit there for an entire test run again.

Bug two: the schema that was quietly wrong

With the dummy target actually running, the numbers improved a lot — small monitor counts passed cleanly. But past a few thousand simulated monitors, the same pattern I’d already been burned by once came back: throughput fell behind, the BullMQ queue backlog climbed into the thousands, and it really did look like I’d found a genuine capacity ceiling. This time the target was definitely running. So I believed it. For a while.

What made me doubt it was the CPU graph. If the worker fleet were genuinely maxed out, I’d expect to see it pinned near 100%. Instead it sat around 15–20% the entire time — comfortably idle, even while the queue was supposedly drowning. A backlog with spare CPU sitting right next to it isn’t a capacity problem. It’s a sign something else is the bottleneck.

That something else was sitting in the Postgres logs the whole time, three errors deep on effectively every single check:

ERROR:  column "quiet_hours_enabled" does not exist
ERROR:  column "user_id" does not exist
ERROR:  column "is_active" does not exist

My alert-lifecycle code queries a monitor’s maintenance windows, notification preferences, and active channels on every check that changes state. The throwaway schema I’d hand-written for the isolated test database was missing three columns those queries expected. Every check was throwing three failed queries before it could finish — and Postgres logging a full failed-statement dump, on a table it was hitting hundreds of times a second, is real CPU and disk I/O that had absolutely nothing to do with how many monitors the scheduler could actually handle.

What the box could actually do

With both bugs fixed, the real numbers were almost anticlimactic: clean passes at 2,000, 4,000, and 8,000 simulated monitors, with the worker fleet barely breaking a sweat. That’s comfortably past anything I expect to need for a while — and it’s the number I actually trust, because by then I’d watched the test lie to me twice and knew what a genuine bottleneck versus a broken harness looked like.

• Uniform failure, no spread — timeouts clustering at almost exactly the configured limit meant nothing was responding, not that something was slow.
• Backlog growing with CPU sitting idle— a real capacity ceiling saturates the resource that’s actually constrained. Spare CPU next to a growing queue means the bottleneck is somewhere else.
• Read the logs of the thing under test, not just its metrics. Both bugs were sitting in plain sight in container status and Postgres error logs the entire time — the throughput graph alone never would have pointed at either one.

The actual lesson

I’d been careful to run the real scheduler and worker code, completely unmodified, against the test stack — that part of the rigor was right. What I’d treated casually was everything around it: the dummy target, the throwaway schema, the things that felt like scaffolding instead of code worth trusting. A load test is only as honest as its stand-ins, and a broken stand-in doesn’t fail loudly. It just quietly hands you a number that looks exactly like the answer to a completely different question.