Did my deploy slow down my site? How to know for sure

You shipped on Tuesday. On Thursday the site feels a bit slower, or a customer says so, or a chart wiggles the wrong way. Was it the deploy? Or is it just a busy day, a slow phone, a bad network?

Most teams never get a straight answer to this. They squint at a graph, run Lighthouse a few times, and move on. Here’s how to actually find out.

The short answer

Compare real visits to the new release with real visits to the release before it. Do it for one page on one kind of device at a time, using the typical visit (the 75th percentile) rather than the average. Only call it slower when both sides have enough visits and the change is bigger than everyday noise.

The question isn’t “is the site slow?”. It’s “is this release slower than the last one?”.

Each part of that sentence is there for a reason. Leave one out and you’ll blame the wrong release, or miss the right one.

Why one test can’t tell you

The first thing most of us do is run Lighthouse or PageSpeed Insights before and after. That’s a good habit for catching obvious problems, but it can’t settle the question.

1. 01Lab scores move on their own. Run Lighthouse five times on the same page and you’ll get different numbers. Server load, the test machine and the network all change between runs. A few points either way is normal, so a small drop after a deploy might be nothing.
2. 02It’s one visit on one machine. Your visitors use old Android phones, fast laptops, hotel Wi-Fi and 4G on a train. The slow ones are the ones who complain, and a single lab run doesn’t include them.
3. 03It can’t click. Interaction to Next Paint (INP), the metric for how fast the page reacts to taps and clicks, needs a real person. A lab test has none, so a heavier click handler won’t show up.

We wrote more about this gap in Your Lighthouse score is green. Why are your Core Web Vitals failing?

Compare releases, not days

The next idea is to compare this week with last week in your monitoring tool. That’s closer, but dates mix things together. A Monday has different visitors from a Saturday. A newsletter brings a wave of phone users. If you deployed twice this week, a “this week” chart blends both releases with whatever ran before them.

Instead, tag every visit with the release it loaded: the commit or version that was live when the page was served. Then the comparison is clean. Every visit to v41 on one side, every visit to v40 on the other. The days and hours fall away, and the only thing that changed between the two groups is the code.

This is also why Google Search Console can’t answer the question. Its Core Web Vitals report is a rolling 28-day window, so a bad release only drags the number down slowly, and by the time it turns red you may have shipped five more times.

Split by page and device

A site-wide number hides the change you’re looking for. Say your release made the pricing page slower on phones. Your homepage gets ten times the traffic and didn’t change, so the site-wide number barely moves. Or traffic shifted: more people landed on a heavy page this week, and the site looks slower even though no page got worse.

So compare the same page, on the same kind of device. Phones and desktops behave so differently that mixing them is like averaging the weather in two cities. Start with your busiest pages and the pages your release touched.

Then use the 75th percentile, not the average. The p75 is the visit that’s slower than three quarters of the others: a typical slow-ish visit. A few awful visits can’t drag it around the way they drag an average, and it’s the same number Google uses to judge Core Web Vitals.

How big a change is real?

Even with clean groups, two releases never get exactly the same numbers. You need two rules before you call anything slower.

1. 01Enough visits on both sides. With a handful of visits, one person on a slow connection decides the result. We wait for at least 30 visits per page and device on each release before saying anything.
2. 02A change bigger than the noise. We only count a change if it’s more than 5% of the old number, or a minimum amount for that metric, whichever is larger: 100ms for LCP and FCP, 20ms for INP, 50ms for TTFB and 0.01 for CLS. The minimum stops tiny wobbles on an already fast page from counting.

Here’s what that looks like for one page on phones:

LCP got 180ms slower, past its 120ms bar, so it regressed. INP moved by 12ms, under its 20ms floor, so that’s noise. CLS jumped by 0.04, well past 0.01. Something on that page now shifts around while it loads, maybe an image that lost its size or a banner that arrives late.

These rules are a practical bar, not a statistics test. They’re strict enough that you don’t chase ghosts, and simple enough that you can check the maths yourself.

Doing it yourself

You can build this with free tools. Roughly:

• Add Google’s open-source web-vitals library to your pages. It reports LCP, INP, CLS, FCP and TTFB from real visitors.
• Send each result to your own endpoint along with the page path, a device type (phone, tablet, desktop) and the release: your commit hash or version number, baked in at build time.
• Store the rows somewhere you can query, like a database table or a spreadsheet if traffic is small.
• For each page and device, take the 75th percentile of each metric for the new release and for the one before it.
• Apply the two rules above, and look again a day later if either side is still short of visits.

It works, but it’s a weekend project that turns into a thing you maintain: bots to filter, Safari and Firefox not reporting some metrics, single-page apps that change pages without a full load, and someone has to remember to check after every deploy.

That’s the job we built CodeGecko for. A small script collects real visits, your deploy pipeline tells us when a release goes out, and each release gets a verdict per page and device: Improved, Regressed or Unchanged, with the commits that shipped next to it. The methodology lists every rule, and the demo shows a release that regressed.

A checklist

• Don’t trust one Lighthouse run either way. Use it to spot problems, not to decide whether a deploy hurt.
• Group real visits by release, not by date.
• Look at one page on one kind of device at a time, starting with your busiest pages and the ones you changed.
• Compare the 75th percentile, not the average.
• Wait for enough visits on both releases before deciding.
• Only call it slower when the change beats the noise: 5% or the metric’s floor, whichever is larger.
• When something regressed, read that release’s commits first. The cause is usually in there.

NourBuilding CodeGecko