Appearance
Git Bisect for Performance Regression
Finding the commit that broke a test is one thing. Finding the commit that made a query 300ms slower or a build 40s longer is another. Correctness bisect asks "does it work?" and gets a binary answer. Performance bisect asks "is it fast enough?" — and that question needs statistics, thresholds, and noise handling.
This guide covers performance regression bisect for a DevOps engineer who needs to catch latency, throughput, memory, and build-time regressions before they hit production.
Why a DevOps Engineer Needs This
- A deploy went out. P99 latency jumped from 200ms to 600ms. You have 80 commits to search.
- A CI pipeline that used to take 4 minutes now takes 9. Nobody noticed for three weeks.
- A database migration merged two weeks ago made a pagination query 10x slower. No test failed — it was just slow.
- A compiler flag change reduced binary size by 2% but doubled link time.
Performance regression bisect finds the exact commit that introduced a measurable degradation, even when no correctness test fails.
How Performance Bisect Differs from Correctness Bisect
| Dimension | Correctness bisect | Performance bisect |
|---|---|---|
| Test result | Binary pass/fail | Continuous measurement (ms, bytes, ops/s) |
| Sample size | 1 run per commit | N runs per commit (statistical) |
| Decision logic | Exit code 0 vs non-zero | Threshold comparison + confidence |
| Noise tolerance | None needed | Must handle CPU scaling, GC, ASLR |
| Duration | Log2(N) * test duration | Log2(N) * test duration * samples |
| Common failure | Flaky test | Measurement variance |
Core Methodology
1. Establish the baseline
Before you bisect, know what "good" looks like in numbers:
bash
# At the known-good commit (e.g., v2.3.0)
git checkout v2.3.0
hyperfine --warmup 3 --min-runs 10 './bench.sh' --export-json /tmp/baseline.jsonExtract the baseline metric:
bash
BASELINE_MEAN=$(jq '.results[0].mean' /tmp/baseline.json)
BASELINE_STDDEV=$(jq '.results[0].stddev' /tmp/baseline.json)2. Measure at each candidate commit
A performance bisect test script does not exit 1 on failure. It exits 1 when the measured value exceeds the threshold:
bash
#!/bin/bash
# scripts/bisect-perf.sh
set -euo pipefail
THRESHOLD_PCT=10 # Allow 10% regression
SAMPLES=5 # Number of runs per commit
BASELIME_MEAN=1.45 # seconds — from known-good commit
# --- Build step ---
make clean || exit 125
make -j"$(nproc)" || exit 125
# --- Measure ---
# Use hyperfine for stable timing with warmup
HYPERFINE_OUT=$(mktemp)
hyperfine --warmup 2 --runs "$SAMPLES" \
--export-json "$HYPERFINE_OUT" \
'./run-bench.sh' 2>/dev/null || exit 125
# --- Compare ---
CURRENT_MEAN=$(jq '.results[0].mean' "$HYPERFINE_OUT")
rm -f "$HYPERFINE_OUT"
# Calculate regression ratio
REGRESSION=$(echo "scale=4; $CURRENT_MEAN / $BASELIME_MEAN" | bc)
THRESHOLD=$(echo "scale=4; 1.0 + $THRESHOLD_PCT / 100.0" | bc)
echo "Commit: $(git rev-parse --short HEAD)"
echo "Baseline mean: ${BASELIME_MEAN}s Current mean: ${CURRENT_MEAN}s Ratio: ${REGRESSION}"
if (( $(echo "$REGRESSION > $THRESHOLD" | bc -l) )); then
echo "REGRESSION detected: ${REGRESSION}x of baseline"
exit 1 # bad commit
else
exit 0 # good commit
fi3. Statistical noise mitigation
A single run per commit is useless for performance. Even with the same binary, wall-clock time varies due to:
- CPU frequency scaling (turbo boost, thermal throttling)
- ASLR and memory layout randomization
- Scheduler interference (cron, background services)
- JIT warmup and GC pauses (JVM, V8, LuaJIT)
- Disk cache state (cold vs warm)
Multiple runs with warmup. Always discard the first 2-3 runs (JIT warmup, cache fill). Use the median, not the mean — the median is robust to outliers from GC pauses or scheduler jitter.
bash
hyperfine --warmup 3 --runs 10 --style=basic './target/release/myapp bench'Pin CPU frequency. On bare-metal runners or VMs with cpufreq control:
bash
sudo cpupower frequency-set --governor performanceWithout this, turbo boost can make the same commit appear 15% faster or slower depending on thermal history.
Pin to a single core. Prevents scheduler migration variance:
bash
taskset -c 1 hyperfine --warmup 3 --runs 10 './bench.sh'Use --min-runs with --show-output. For short benchmarks (<100ms), run count matters more than warmup. Hyperfine adaptively increases runs until the confidence interval is tight enough.
4. Threshold strategies
Choosing the right threshold is the hardest part of performance bisect. Too tight and every commit looks like a regression. Too loose and real regressions slip through.
Percentage threshold (simple). Flag any commit where mean exceeds baseline by X%:
bash
THRESHOLD_PCT=5 # 5% regression allowedStandard deviation threshold (robust). Flag when the current mean is more than N standard deviations from the baseline mean. This automatically accounts for the benchmark's inherent variance:
bash
# baseline.json contains stddev
BASELINE_MEAN=$(jq '.results[0].mean' /tmp/baseline.json)
BASELINE_STDDEV=$(jq '.results[0].stddev' /tmp/baseline.json)
Z_SCORE=3 # 99.7% confidence for normal distributions
# Flag if current_mean > baseline_mean + Z_SCORE * baseline_stddev
THRESHOLD=$(echo "$BASELINE_MEAN + $Z_SCORE * $BASELINE_STDDEV" | bc)Absolute threshold (critical paths). For SLO-bound operations, use an absolute wall:
bash
# API endpoint must serve in under 500ms P99
MAX_LATENCY_MS=500Distribution comparison (advanced). Use a two-sample Kolmogorov-Smirnov test or Mann-Whitney U test to compare the current commit's measurement distribution against the baseline. This catches changes in variance even when the mean hasn't moved much. Tools like asv (Airspeed Velocity) implement this automatically.
5. Multi-criteria regression
A single benchmark can regress in multiple dimensions. Latency might stay flat while memory spikes. Track multiple metrics:
bash
#!/bin/bash
set -euo pipefail
BASELINE_LATENCY=1.45 # seconds
BASELINE_MEMORY=128 # MB
BASELINE_BINARY_SIZE=15 # MB
# Build
make -j"$(nproc)" || exit 125
# Measure latency
LATENCY=$(hyperfine --warmup 2 --runs 5 \
--export-json /tmp/perf.json \
'./run-bench.sh' 2>/dev/null && \
jq '.results[0].mean' /tmp/perf.json)
# Measure peak memory (Linux)
/usr/bin/time -v ./run-bench.sh 2>/tmp/mem.txt >/dev/null
MEMORY=$(grep 'Maximum resident set size' /tmp/mem.txt | awk '{print $NF}')
MEMORY_MB=$((MEMORY / 1024))
# Measure binary size
BINARY_SIZE=$(du -m ./target/release/myapp | cut -f1)
# Score: exit 1 if ANY metric exceeds threshold
REGRESSED=0
if (( $(echo "$LATENCY > $BASELINE_LATENCY * 1.1" | bc -l) )); then
echo "REGRESSION: latency ${LATENCY}s (baseline ${BASELINE_LATENCY}s)"
REGRESSED=1
fi
if [ "$MEMORY_MB" -gt $((BASELINE_MEMORY * 110 / 100)) ]; then
echo "REGRESSION: memory ${MEMORY_MB}MB (baseline ${BASELINE_MEMORY}MB)"
REGRESSED=1
fi
exit $REGRESSEDCI/CD Integration Patterns
Pattern 1: Nightly performance regression detection
Run a full performance bisect on a schedule. No human needed until a regression is found:
yaml
# .forgejo/workflows/perf-bisect-nightly.yaml
name: Nightly performance regression bisect
on:
schedule:
- cron: '0 3 * * *' # 3 AM daily
jobs:
perf-bisect:
runs-on: dedicated-bench-runner
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # full history for bisect
- name: Run performance bisect
run: |
# Compare HEAD against last known-good tag
LAST_GOOD=$(git tag --list 'v*' --sort=-version:refname | head -1)
scripts/bisect-perf-manage.sh run \
--good="$LAST_GOOD" \
--bad=HEAD \
--worktree=/mnt/ramdisk/bisect-wt \
scripts/bisect-perf.sh
- name: Report regression
if: failure()
run: |
echo "Performance regression detected between $LAST_GOOD and HEAD"
git bisect logPattern 2: PR performance gate (bounded)
Run a performance gate on PRs, but only on the merge base — not a full bisect:
yaml
- name: Performance gate
run: |
# Compare PR HEAD against merge-base with main
BASE=$(git merge-base origin/main HEAD)
git checkout "$BASE"
hyperfine --warmup 2 --runs 5 --export-json /tmp/base.json './bench.sh'
git checkout HEAD
hyperfine --warmup 2 --runs 5 --export-json /tmp/head.json './bench.sh'
python3 scripts/compare-perf.py /tmp/base.json /tmp/head.json --threshold=5A full bisect on every PR is too slow. The gate tells you "this PR regressed something" — then you bisect post-merge to find the exact commit.
Pattern 3: Dedicated performance runner
Performance bisect needs a stable environment. Shared CI runners with noisy neighbours produce unusable measurements:
- Dedicated bare-metal instance with CPU governor pinned to
performance. - Docker with
--cpuset-cpusto pin to specific cores. - RAM disk for build artifacts (
/mnt/ramdisk) to eliminate disk I/O variance. - Same kernel version as production. Kernel scheduler changes affect benchmark results.
bash
# Container with CPU pinning and no swap
docker run --rm \
--cpuset-cpus="0-3" \
--memory=8g \
--memory-swap=8g \
--volume /mnt/ramdisk/bisect:/workspace \
perf-bench-image \
scripts/bisect-perf.shPattern 4: Isolated worktree with ramdisk
For long bisect sessions, use a worktree on a RAM disk to eliminate filesystem noise:
bash
scripts/bisect-perf-manage.sh run \
--good=v2.3.0 \
--bad=HEAD \
--worktree=/mnt/ramdisk/bisect-wt \
--samples=5 \
--threshold=5 \
scripts/bisect-perf.shThe worktree shares the .git database with the main repo but the working tree lives on RAM. Build artifacts, temp files, and benchmarks all run from RAM. When the bisect finishes, remove the worktree and the RAM disk is gone.
Language-Specific Patterns
Go benchmarks
bash
#!/bin/bash
set -euo pipefail
# Build
go build ./cmd/bench || exit 125
# Run Go benchmark, extract ns/op
go test -bench=. -benchtime=100x -count=5 ./pkg/sorting 2>/dev/null \
| grep 'BenchmarkSort' \
| awk '{print $3}' \
| sed 's/ns\/op//' \
> /tmp/bench-results.txt
# Calculate median
sort -n /tmp/bench-results.txt \
| awk '{a[NR]=$1} END {print a[int(NR/2)]}' \
> /tmp/median-ns
CURRENT=$(cat /tmp/median-ns)
BASELINE=4523000 # ns from known-good commit
THRESHOLD=$(echo "$BASELINE * 1.05" | bc)
if (( $(echo "$CURRENT > $THRESHOLD" | bc -l) )); then
exit 1
fi
exit 0Node.js / TypeScript
bash
#!/bin/bash
set -euo pipefail
npm ci || exit 125
npm run build || exit 125
# Use hyperfine to run the Node.js benchmark
hyperfine --warmup 3 --runs 10 \
--export-json /tmp/perf.json \
'node dist/bench.js' 2>/dev/null || exit 125
CURRENT_MEAN=$(jq '.results[0].mean' /tmp/perf.json)
BASELINE_MEAN=0.850 # seconds
if (( $(echo "$CURRENT_MEAN > $BASELINE_MEAN * 1.05" | bc -l) )); then
exit 1
fi
exit 0Rust benchmarks
bash
#!/bin/bash
set -euo pipefail
# Build with release flags
cargo bench --no-run 2>/dev/null || exit 125
# Run the specific benchmark, extract time
cargo bench -- bench_my_query 2>/dev/null \
| grep 'bench_my_query' \
| grep -oP '[\d.]+(?= ns/op|\d+ ns/op)' \
| tail -1 > /tmp/ns
CURRENT=$(cat /tmp/ns)
BASELINE=1250000 # ns from known-good
if (( $(echo "$CURRENT > $BASELINE * 1.05" | bc -l) )); then
exit 1
fi
exit 0Python (pytest-benchmark)
bash
#!/bin/bash
set -euo pipefail
pip install -e . -q || exit 125
# pytest-benchmark writes a JSON file with all results
pytest tests/perf/ --benchmark-only \
--benchmark-json=/tmp/bench.json 2>/dev/null || exit 125
# Extract the mean of the key test
CURRENT_MEAN=$(jq '.benchmarks[] | select(.name == "test_query_latency") | .stats.mean' /tmp/bench.json)
BASELINE_MEAN=0.032 # seconds
if (( $(echo "$CURRENT_MEAN > $BASELINE_MEAN * 1.1" | bc -l) )); then
exit 1
fi
exit 0Advanced Techniques
Adaptive sample sizing
Short benchmarks need more runs to overcome noise. Long benchmarks need fewer. Adjust sample count dynamically based on the coefficient of variation (CV = stddev/mean):
bash
# After first 3 runs, compute CV
# If CV > 0.05 (5% variance), run 5 more samples
# If CV > 0.10, run 10 more samples
# If CV < 0.02, stop — stable enoughHyperfine does this automatically with --min-runs and the adaptive stopping criterion.
Cold-cache bisect
Some regressions only appear on cold cache (first request after deploy). Simulate this by dropping caches before each measurement:
bash
# Requires root or sudo (or CAP_SYS_ADMIN in containers)
echo 3 | sudo tee /proc/sys/vm/drop_caches
# Also drop the application's internal cache
rm -rf /tmp/app-cache/*Note: dropping caches adds ~200ms of noise per run. For sub-millisecond benchmarks this dominates. Use cold-cache bisect only for whole-application startup or first-request latency.
Multi-commit skip for known-broken builds
If commits in the range are known to not compile or have broken tests, pre-mark them as skipped to avoid wasting bisect iterations:
bash
# Read a list of known-broken SHAs from a file
while read -r SHA; do
git bisect skip "$SHA"
done < known-broken-commits.txtParallel performance bisect
For monorepos with multiple independent services, bisect each service's performance in parallel using separate worktrees:
bash
# Worktree 1: payments service perf
scripts/bisect-perf-manage.sh run \
--good=v2.3.0 --bad=HEAD \
--worktree=/mnt/ramdisk/bisect-payments \
--path=services/payments \
scripts/bisect-perf-payments.sh &
# Worktree 2: checkout service perf
scripts/bisect-perf-manage.sh run \
--good=v2.3.0 --bad=HEAD \
--worktree=/mnt/ramdisk/bisect-checkout \
--path=services/checkout \
scripts/bisect-perf-checkout.sh &
waitEach worktree shares the .git database so there is no duplicate object storage. The --path flag restricts each bisect to commits touching that service's directory, narrowing the search space dramatically.
Pitfalls
1. Noise from CPU frequency scaling
The single biggest source of false positives. Always pin the CPU governor to performance on dedicated runners. On shared runners, accept that you need more samples and wider thresholds.
2. Noise from background processes
Systemd timers, log rotation, cron jobs, and monitoring agents all add jitter. On a dedicated performance runner:
bash
# Pause non-essential services during the bisect
sudo systemctl stop cron.service
sudo systemctl stop logrotate.timer
sudo systemctl stop sysstat-collect.timer3. JIT warmup
JIT-compiled languages (Java, V8, LuaJIT, .NET) need warmup runs before the steady-state measurement. The first 3-10 runs are always slower. Hyperfine's --warmup handles this.
Worse: if the bisect lands on a commit that changes JIT behaviour (e.g., inlining thresholds), the warmup curve itself changes. Always run at least 10 iterations and compare the median of runs 5-10, not the overall mean.
4. GC pressure
Languages with tracing GC (Go, Java, C#, Lua) can have unpredictable pauses that dominate a single measurement. Use the median, not the mean. A single GC pause can double the mean of 10 runs but the median stays stable.
5. Disk cache state
The first run after a checkout is always slower because the disk cache is cold. Hyperfine's --warmup handles this too. For storage benchmarks, explicitly warm the file cache:
bash
# Read all benchmark data into page cache
cat data/*.bin > /dev/null6. Comparing across different hardware
Never compare performance numbers from different machines, different kernel versions, or different CPU governors. If you bisect on a dev laptop, the absolute numbers mean nothing. Always bisect on the same class of hardware as production.
7. Threshold too tight for the noise level
A 1% threshold is noise on all but the most tightly controlled environments. Start at 10%, then tighten as you understand the variance. Use the Z-score approach (threshold = baseline_mean + N * baseline_stddev) to auto-tune.
8. Binary search on non-monotonic performance
Performance is not always monotonic. A commit might be faster, then the next slower, then the next faster again. Git bisect assumes a single transition from good to bad. If performance oscillates, bisect gives unreliable results. Mitigate by:
- Running each candidate 3 times and taking the median.
- Using a wider threshold (15-20%) to filter out minor oscillation.
- If oscillation persists, bisect is the wrong tool — use
git log --binaryor manual search instead.
Comparison: Performance Bisect vs Correctness Bisect Decision Table
| Scenario | Use correctness bisect | Use performance bisect |
|---|---|---|
| Test started failing | Yes | No |
| Build broke | Yes | No |
| API latency doubled | No | Yes |
| Memory usage grew 50% | No | Yes |
| Binary size increased | No | Yes |
| Startup time tripled | No | Yes |
| Throughput dropped | No | Yes |
| Flaky test (intermittent) | No | Yes (to check if variance changed) |
Quick commands
bash
# Record baseline at a known-good tag
git checkout v2.3.0
hyperfine --warmup 3 --min-runs 10 './bench.sh' --export-json /tmp/baseline.json
BASELINE=$(jq '.results[0].mean' /tmp/baseline.json)
# Run the bisect with a performance test script
scripts/bisect-perf-manage.sh run \
--good=v2.3.0 \
--bad=HEAD \
--worktree=/mnt/ramdisk/bisect-wt \
--threshold=5 \
--samples=5 \
scripts/bisect-perf.sh
# Manual step-by-step performance bisect
git bisect start HEAD v2.3.0 --
git bisect run scripts/bisect-perf.sh
git bisect log
git bisect resetSee also
docs/devops-workflows/git-bisect-run.md— automated correctness regression bisectdocs/devops-workflows/git-bisect-ci-triage.md— basic bisect setup for CIdocs/recipes/bisect-perf-quick.md— one-page cheat sheet for performance bisectscripts/bisect-perf-manage.sh— automation script for performance bisectscripts/bisect-run-manage.sh— automation script for correctness bisect