Skip to content

Schema change rehearsal

Every migration runs twice.

The first time against a copy of production, with real traffic replayed over it. The second time for real — and by then you already know what it locks.

Opens the pack's source on GitHub — Bellwether has no live product to open.

  • Postgres 12–17
  • MySQL 8
  • Runs in your VPC
  • No agent on the primary
Rehearsal 4192Hold
ALTER TABLE orders ADD COLUMN settled_at timestamptz

orders · primary-eu-2

Rows touched
48,213,904
Counted on the mirror, not estimated from the planner.
Traffic replayed
2.1%
A slice weighted to match the live read/write mix.
Longest lock
1.94s
ACCESS EXCLUSIVE, taken while the index builds.
Observed p95
340ms
Checkout writes queued behind the lock in rehearsal.
Writes by hourquiet 02:00–05:00

The index build blocks checkout writes for 1.94s. At 14:00 that is roughly 2,700 queued writes — enough to trip the payment timeout. Re-run after 02:20, or build the index CONCURRENTLY and keep the window under 40ms.

Loading operating figures

How it works

Three steps, and none of them touch the primary.

  1. 01

    Connect

    Point Bellwether at a read replica. Nothing is installed beside the database you are already worried about.

  2. 02

    Rehearse

    The pending statement runs against a copy at production scale, with a weighted slice of live traffic replayed over the top of it. Production's index bloat, production's row widths, production's contention.

  3. 03

    Verdict

    Pass, hold or refuse — with the lock it took, how long it held, and the hour where it would cost least.

What it reports

A planner estimate is a guess. This is a measurement.

Replay, not simulation

The rehearsal runs your statement against a real copy at production scale. A number from the query planner is a guess about all of it.

Planner estimate
40ms
Observed in rehearsal
1.94s

Lock forecast

Which lock, on which relation, for how long, at which hour. The answer is a distribution across the day, not one number pretending the load is flat.

Quiet-window finder

Bellwether reads a fortnight of traffic and names the windows where this particular lock would cost the least.

The reversal is rehearsed too

The down path replays on the same copy. A migration you cannot reverse is not ready, whatever the up path did — and two-thirds of down paths have never been run once.

Teams

The Tuesday‑morning rule, retired.

We retired the Tuesday-morning rule because we finally knew how long the lock would last. The rule only ever existed because nobody could answer that.
Ifeoma AchebeStaff Engineer · Kestrel Freight
Two of our down paths did not work. We learned that on a copy, at four in the afternoon, instead of at two in the morning.
Tomás BeltránDatabase Reliability · Orrery Health

Rehearse the next one.

Point Bellwether at a replica. First verdict in about ten minutes.

Reads a replica · installs nothing on the primary · no card · opens the pack's source, not a live demo