Skip to content
Clownmind
Engineering

Rehearse peak season in July or find out about it in November

Load testing against production-shaped data twice before peak has caught more genuine incidents for us than any amount of monitoring.

MOMarcus OyelaranEngineering lead8 April 20266 min

Monitoring tells you about the incident while it is happening. A rehearsal tells you about it in July, when fixing it costs a sprint instead of a trading day.

What a rehearsal involves

Production-shaped data, not synthetic fixtures. Traffic patterns modelled on last year's actual curve, including the spike shape rather than a flat plateau. And the full path — CDN, application, database, third-party calls — because the thing that breaks is almost never the component you were worried about.

We run two: one in late July to find the structural problems, one in September to confirm the fixes and catch anything the summer's releases introduced.

What the rehearsals typically find

  • A cache key that includes a session identifier, making the cache useless under load.
  • A third-party call in the critical path with no timeout, which turns their bad day into yours.
  • Database connection pool limits that are fine at 3x normal traffic and catastrophic at 8x.
  • A queue with no dead-letter handling, quietly discarding work when it backs up.

Every one of those is boring to fix in July. Each of them is a very bad afternoon in November.

The runbook is the deliverable

The rehearsal produces a document: what breaks first, at what load, what the symptom looks like in the dashboards, and what to do about it. That runbook is what someone on call reads at 2am, and it is worth more than the load test results themselves.

Keep reading

Bring us the messy version

Thirty minutes with an engineer who has done this before.

No sales sequence. One reply from a senior engineer, usually within a working day.

Want an opinion on your architecture before you commit budget?

Thirty minutes with a principal engineer. No deck, no discovery call disguised as a sales call.