Skip to content

Git Bisect for Performance Regression ​

Finding the commit that broke a test is one thing. Finding the commit that made a query 300ms slower or a build 40s longer is another. Correctness bisect asks "does it work?" and gets a binary answer. Performance bisect asks "is it fast enough?" — and that question needs statistics, thresholds, and noise handling.

This guide covers performance regression bisect for a DevOps engineer who needs to catch latency, throughput, memory, and build-time regressions before they hit production.

Why a DevOps Engineer Needs This ​

  • A deploy went out. P99 latency jumped from 200ms to 600ms. You have 80 commits to search.
  • A CI pipeline that used to take 4 minutes now takes 9. Nobody noticed for three weeks.
  • A database migration merged two weeks ago made a pagination query 10x slower. No test failed — it was just slow.
  • A compiler flag change reduced binary size by 2% but doubled link time.

Performance regression bisect finds the exact commit that introduced a measurable degradation, even when no correctness test fails.

How Performance Bisect Differs from Correctness Bisect ​

DimensionCorrectness bisectPerformance bisect
Test resultBinary pass/failContinuous measurement (ms, bytes, ops/s)
Sample size1 run per commitN runs per commit (statistical)
Decision logicExit code 0 vs non-zeroThreshold comparison + confidence
Noise toleranceNone neededMust handle CPU scaling, GC, ASLR
DurationLog2(N) * test durationLog2(N) * test duration * samples
Common failureFlaky testMeasurement variance

Core Methodology ​

1. Establish the baseline ​

Before you bisect, know what "good" looks like in numbers:

bash
# At the known-good commit (e.g., v2.3.0)
git checkout v2.3.0
hyperfine --warmup 3 --min-runs 10 './bench.sh' --export-json /tmp/baseline.json

Extract the baseline metric:

bash
BASELINE_MEAN=$(jq '.results[0].mean' /tmp/baseline.json)
BASELINE_STDDEV=$(jq '.results[0].stddev' /tmp/baseline.json)

2. Measure at each candidate commit ​

A performance bisect test script does not exit 1 on failure. It exits 1 when the measured value exceeds the threshold:

bash
#!/bin/bash
# scripts/bisect-perf.sh
set -euo pipefail

THRESHOLD_PCT=10   # Allow 10% regression
SAMPLES=5          # Number of runs per commit
BASELIME_MEAN=1.45 # seconds — from known-good commit

# --- Build step ---
make clean || exit 125
make -j"$(nproc)" || exit 125

# --- Measure ---
# Use hyperfine for stable timing with warmup
HYPERFINE_OUT=$(mktemp)
hyperfine --warmup 2 --runs "$SAMPLES" \
  --export-json "$HYPERFINE_OUT" \
  './run-bench.sh' 2>/dev/null || exit 125

# --- Compare ---
CURRENT_MEAN=$(jq '.results[0].mean' "$HYPERFINE_OUT")
rm -f "$HYPERFINE_OUT"

# Calculate regression ratio
REGRESSION=$(echo "scale=4; $CURRENT_MEAN / $BASELIME_MEAN" | bc)
THRESHOLD=$(echo "scale=4; 1.0 + $THRESHOLD_PCT / 100.0" | bc)

echo "Commit: $(git rev-parse --short HEAD)"
echo "Baseline mean: ${BASELIME_MEAN}s  Current mean: ${CURRENT_MEAN}s  Ratio: ${REGRESSION}"

if (( $(echo "$REGRESSION > $THRESHOLD" | bc -l) )); then
  echo "REGRESSION detected: ${REGRESSION}x of baseline"
  exit 1  # bad commit
else
  exit 0  # good commit
fi

3. Statistical noise mitigation ​

A single run per commit is useless for performance. Even with the same binary, wall-clock time varies due to:

  • CPU frequency scaling (turbo boost, thermal throttling)
  • ASLR and memory layout randomization
  • Scheduler interference (cron, background services)
  • JIT warmup and GC pauses (JVM, V8, LuaJIT)
  • Disk cache state (cold vs warm)

Multiple runs with warmup. Always discard the first 2-3 runs (JIT warmup, cache fill). Use the median, not the mean — the median is robust to outliers from GC pauses or scheduler jitter.

bash
hyperfine --warmup 3 --runs 10 --style=basic './target/release/myapp bench'

Pin CPU frequency. On bare-metal runners or VMs with cpufreq control:

bash
sudo cpupower frequency-set --governor performance

Without this, turbo boost can make the same commit appear 15% faster or slower depending on thermal history.

Pin to a single core. Prevents scheduler migration variance:

bash
taskset -c 1 hyperfine --warmup 3 --runs 10 './bench.sh'

Use --min-runs with --show-output. For short benchmarks (<100ms), run count matters more than warmup. Hyperfine adaptively increases runs until the confidence interval is tight enough.

4. Threshold strategies ​

Choosing the right threshold is the hardest part of performance bisect. Too tight and every commit looks like a regression. Too loose and real regressions slip through.

Percentage threshold (simple). Flag any commit where mean exceeds baseline by X%:

bash
THRESHOLD_PCT=5   # 5% regression allowed

Standard deviation threshold (robust). Flag when the current mean is more than N standard deviations from the baseline mean. This automatically accounts for the benchmark's inherent variance:

bash
# baseline.json contains stddev
BASELINE_MEAN=$(jq '.results[0].mean' /tmp/baseline.json)
BASELINE_STDDEV=$(jq '.results[0].stddev' /tmp/baseline.json)
Z_SCORE=3  # 99.7% confidence for normal distributions

# Flag if current_mean > baseline_mean + Z_SCORE * baseline_stddev
THRESHOLD=$(echo "$BASELINE_MEAN + $Z_SCORE * $BASELINE_STDDEV" | bc)

Absolute threshold (critical paths). For SLO-bound operations, use an absolute wall:

bash
# API endpoint must serve in under 500ms P99
MAX_LATENCY_MS=500

Distribution comparison (advanced). Use a two-sample Kolmogorov-Smirnov test or Mann-Whitney U test to compare the current commit's measurement distribution against the baseline. This catches changes in variance even when the mean hasn't moved much. Tools like asv (Airspeed Velocity) implement this automatically.

5. Multi-criteria regression ​

A single benchmark can regress in multiple dimensions. Latency might stay flat while memory spikes. Track multiple metrics:

bash
#!/bin/bash
set -euo pipefail

BASELINE_LATENCY=1.45   # seconds
BASELINE_MEMORY=128     # MB
BASELINE_BINARY_SIZE=15 # MB

# Build
make -j"$(nproc)" || exit 125

# Measure latency
LATENCY=$(hyperfine --warmup 2 --runs 5 \
  --export-json /tmp/perf.json \
  './run-bench.sh' 2>/dev/null && \
  jq '.results[0].mean' /tmp/perf.json)

# Measure peak memory (Linux)
/usr/bin/time -v ./run-bench.sh 2>/tmp/mem.txt >/dev/null
MEMORY=$(grep 'Maximum resident set size' /tmp/mem.txt | awk '{print $NF}')
MEMORY_MB=$((MEMORY / 1024))

# Measure binary size
BINARY_SIZE=$(du -m ./target/release/myapp | cut -f1)

# Score: exit 1 if ANY metric exceeds threshold
REGRESSED=0
if (( $(echo "$LATENCY > $BASELINE_LATENCY * 1.1" | bc -l) )); then
  echo "REGRESSION: latency ${LATENCY}s (baseline ${BASELINE_LATENCY}s)"
  REGRESSED=1
fi
if [ "$MEMORY_MB" -gt $((BASELINE_MEMORY * 110 / 100)) ]; then
  echo "REGRESSION: memory ${MEMORY_MB}MB (baseline ${BASELINE_MEMORY}MB)"
  REGRESSED=1
fi

exit $REGRESSED

CI/CD Integration Patterns ​

Pattern 1: Nightly performance regression detection ​

Run a full performance bisect on a schedule. No human needed until a regression is found:

yaml
# .forgejo/workflows/perf-bisect-nightly.yaml
name: Nightly performance regression bisect
on:
  schedule:
    - cron: '0 3 * * *'  # 3 AM daily

jobs:
  perf-bisect:
    runs-on: dedicated-bench-runner
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # full history for bisect

      - name: Run performance bisect
        run: |
          # Compare HEAD against last known-good tag
          LAST_GOOD=$(git tag --list 'v*' --sort=-version:refname | head -1)
          scripts/bisect-perf-manage.sh run \
            --good="$LAST_GOOD" \
            --bad=HEAD \
            --worktree=/mnt/ramdisk/bisect-wt \
            scripts/bisect-perf.sh

      - name: Report regression
        if: failure()
        run: |
          echo "Performance regression detected between $LAST_GOOD and HEAD"
          git bisect log

Pattern 2: PR performance gate (bounded) ​

Run a performance gate on PRs, but only on the merge base — not a full bisect:

yaml
- name: Performance gate
  run: |
    # Compare PR HEAD against merge-base with main
    BASE=$(git merge-base origin/main HEAD)
    git checkout "$BASE"
    hyperfine --warmup 2 --runs 5 --export-json /tmp/base.json './bench.sh'

    git checkout HEAD
    hyperfine --warmup 2 --runs 5 --export-json /tmp/head.json './bench.sh'

    python3 scripts/compare-perf.py /tmp/base.json /tmp/head.json --threshold=5

A full bisect on every PR is too slow. The gate tells you "this PR regressed something" — then you bisect post-merge to find the exact commit.

Pattern 3: Dedicated performance runner ​

Performance bisect needs a stable environment. Shared CI runners with noisy neighbours produce unusable measurements:

  • Dedicated bare-metal instance with CPU governor pinned to performance.
  • Docker with --cpuset-cpus to pin to specific cores.
  • RAM disk for build artifacts (/mnt/ramdisk) to eliminate disk I/O variance.
  • Same kernel version as production. Kernel scheduler changes affect benchmark results.
bash
# Container with CPU pinning and no swap
docker run --rm \
  --cpuset-cpus="0-3" \
  --memory=8g \
  --memory-swap=8g \
  --volume /mnt/ramdisk/bisect:/workspace \
  perf-bench-image \
  scripts/bisect-perf.sh

Pattern 4: Isolated worktree with ramdisk ​

For long bisect sessions, use a worktree on a RAM disk to eliminate filesystem noise:

bash
scripts/bisect-perf-manage.sh run \
  --good=v2.3.0 \
  --bad=HEAD \
  --worktree=/mnt/ramdisk/bisect-wt \
  --samples=5 \
  --threshold=5 \
  scripts/bisect-perf.sh

The worktree shares the .git database with the main repo but the working tree lives on RAM. Build artifacts, temp files, and benchmarks all run from RAM. When the bisect finishes, remove the worktree and the RAM disk is gone.

Language-Specific Patterns ​

Go benchmarks ​

bash
#!/bin/bash
set -euo pipefail

# Build
go build ./cmd/bench || exit 125

# Run Go benchmark, extract ns/op
go test -bench=. -benchtime=100x -count=5 ./pkg/sorting 2>/dev/null \
  | grep 'BenchmarkSort' \
  | awk '{print $3}' \
  | sed 's/ns\/op//' \
  > /tmp/bench-results.txt

# Calculate median
sort -n /tmp/bench-results.txt \
  | awk '{a[NR]=$1} END {print a[int(NR/2)]}' \
  > /tmp/median-ns

CURRENT=$(cat /tmp/median-ns)
BASELINE=4523000  # ns from known-good commit
THRESHOLD=$(echo "$BASELINE * 1.05" | bc)

if (( $(echo "$CURRENT > $THRESHOLD" | bc -l) )); then
  exit 1
fi
exit 0

Node.js / TypeScript ​

bash
#!/bin/bash
set -euo pipefail

npm ci || exit 125
npm run build || exit 125

# Use hyperfine to run the Node.js benchmark
hyperfine --warmup 3 --runs 10 \
  --export-json /tmp/perf.json \
  'node dist/bench.js' 2>/dev/null || exit 125

CURRENT_MEAN=$(jq '.results[0].mean' /tmp/perf.json)
BASELINE_MEAN=0.850  # seconds

if (( $(echo "$CURRENT_MEAN > $BASELINE_MEAN * 1.05" | bc -l) )); then
  exit 1
fi
exit 0

Rust benchmarks ​

bash
#!/bin/bash
set -euo pipefail

# Build with release flags
cargo bench --no-run 2>/dev/null || exit 125

# Run the specific benchmark, extract time
cargo bench -- bench_my_query 2>/dev/null \
  | grep 'bench_my_query' \
  | grep -oP '[\d.]+(?= ns/op|\d+ ns/op)' \
  | tail -1 > /tmp/ns

CURRENT=$(cat /tmp/ns)
BASELINE=1250000  # ns from known-good

if (( $(echo "$CURRENT > $BASELINE * 1.05" | bc -l) )); then
  exit 1
fi
exit 0

Python (pytest-benchmark) ​

bash
#!/bin/bash
set -euo pipefail

pip install -e . -q || exit 125

# pytest-benchmark writes a JSON file with all results
pytest tests/perf/ --benchmark-only \
  --benchmark-json=/tmp/bench.json 2>/dev/null || exit 125

# Extract the mean of the key test
CURRENT_MEAN=$(jq '.benchmarks[] | select(.name == "test_query_latency") | .stats.mean' /tmp/bench.json)
BASELINE_MEAN=0.032  # seconds

if (( $(echo "$CURRENT_MEAN > $BASELINE_MEAN * 1.1" | bc -l) )); then
  exit 1
fi
exit 0

Advanced Techniques ​

Adaptive sample sizing ​

Short benchmarks need more runs to overcome noise. Long benchmarks need fewer. Adjust sample count dynamically based on the coefficient of variation (CV = stddev/mean):

bash
# After first 3 runs, compute CV
# If CV > 0.05 (5% variance), run 5 more samples
# If CV > 0.10, run 10 more samples
# If CV < 0.02, stop — stable enough

Hyperfine does this automatically with --min-runs and the adaptive stopping criterion.

Cold-cache bisect ​

Some regressions only appear on cold cache (first request after deploy). Simulate this by dropping caches before each measurement:

bash
# Requires root or sudo (or CAP_SYS_ADMIN in containers)
echo 3 | sudo tee /proc/sys/vm/drop_caches

# Also drop the application's internal cache
rm -rf /tmp/app-cache/*

Note: dropping caches adds ~200ms of noise per run. For sub-millisecond benchmarks this dominates. Use cold-cache bisect only for whole-application startup or first-request latency.

Multi-commit skip for known-broken builds ​

If commits in the range are known to not compile or have broken tests, pre-mark them as skipped to avoid wasting bisect iterations:

bash
# Read a list of known-broken SHAs from a file
while read -r SHA; do
  git bisect skip "$SHA"
done < known-broken-commits.txt

Parallel performance bisect ​

For monorepos with multiple independent services, bisect each service's performance in parallel using separate worktrees:

bash
# Worktree 1: payments service perf
scripts/bisect-perf-manage.sh run \
  --good=v2.3.0 --bad=HEAD \
  --worktree=/mnt/ramdisk/bisect-payments \
  --path=services/payments \
  scripts/bisect-perf-payments.sh &

# Worktree 2: checkout service perf
scripts/bisect-perf-manage.sh run \
  --good=v2.3.0 --bad=HEAD \
  --worktree=/mnt/ramdisk/bisect-checkout \
  --path=services/checkout \
  scripts/bisect-perf-checkout.sh &

wait

Each worktree shares the .git database so there is no duplicate object storage. The --path flag restricts each bisect to commits touching that service's directory, narrowing the search space dramatically.

Pitfalls ​

1. Noise from CPU frequency scaling ​

The single biggest source of false positives. Always pin the CPU governor to performance on dedicated runners. On shared runners, accept that you need more samples and wider thresholds.

2. Noise from background processes ​

Systemd timers, log rotation, cron jobs, and monitoring agents all add jitter. On a dedicated performance runner:

bash
# Pause non-essential services during the bisect
sudo systemctl stop cron.service
sudo systemctl stop logrotate.timer
sudo systemctl stop sysstat-collect.timer

3. JIT warmup ​

JIT-compiled languages (Java, V8, LuaJIT, .NET) need warmup runs before the steady-state measurement. The first 3-10 runs are always slower. Hyperfine's --warmup handles this.

Worse: if the bisect lands on a commit that changes JIT behaviour (e.g., inlining thresholds), the warmup curve itself changes. Always run at least 10 iterations and compare the median of runs 5-10, not the overall mean.

4. GC pressure ​

Languages with tracing GC (Go, Java, C#, Lua) can have unpredictable pauses that dominate a single measurement. Use the median, not the mean. A single GC pause can double the mean of 10 runs but the median stays stable.

5. Disk cache state ​

The first run after a checkout is always slower because the disk cache is cold. Hyperfine's --warmup handles this too. For storage benchmarks, explicitly warm the file cache:

bash
# Read all benchmark data into page cache
cat data/*.bin > /dev/null

6. Comparing across different hardware ​

Never compare performance numbers from different machines, different kernel versions, or different CPU governors. If you bisect on a dev laptop, the absolute numbers mean nothing. Always bisect on the same class of hardware as production.

7. Threshold too tight for the noise level ​

A 1% threshold is noise on all but the most tightly controlled environments. Start at 10%, then tighten as you understand the variance. Use the Z-score approach (threshold = baseline_mean + N * baseline_stddev) to auto-tune.

8. Binary search on non-monotonic performance ​

Performance is not always monotonic. A commit might be faster, then the next slower, then the next faster again. Git bisect assumes a single transition from good to bad. If performance oscillates, bisect gives unreliable results. Mitigate by:

  • Running each candidate 3 times and taking the median.
  • Using a wider threshold (15-20%) to filter out minor oscillation.
  • If oscillation persists, bisect is the wrong tool — use git log --binary or manual search instead.

Comparison: Performance Bisect vs Correctness Bisect Decision Table ​

ScenarioUse correctness bisectUse performance bisect
Test started failingYesNo
Build brokeYesNo
API latency doubledNoYes
Memory usage grew 50%NoYes
Binary size increasedNoYes
Startup time tripledNoYes
Throughput droppedNoYes
Flaky test (intermittent)NoYes (to check if variance changed)

Quick commands ​

bash
# Record baseline at a known-good tag
git checkout v2.3.0
hyperfine --warmup 3 --min-runs 10 './bench.sh' --export-json /tmp/baseline.json
BASELINE=$(jq '.results[0].mean' /tmp/baseline.json)

# Run the bisect with a performance test script
scripts/bisect-perf-manage.sh run \
  --good=v2.3.0 \
  --bad=HEAD \
  --worktree=/mnt/ramdisk/bisect-wt \
  --threshold=5 \
  --samples=5 \
  scripts/bisect-perf.sh

# Manual step-by-step performance bisect
git bisect start HEAD v2.3.0 --
git bisect run scripts/bisect-perf.sh
git bisect log
git bisect reset

See also ​

  • docs/devops-workflows/git-bisect-run.md — automated correctness regression bisect
  • docs/devops-workflows/git-bisect-ci-triage.md — basic bisect setup for CI
  • docs/recipes/bisect-perf-quick.md — one-page cheat sheet for performance bisect
  • scripts/bisect-perf-manage.sh — automation script for performance bisect
  • scripts/bisect-run-manage.sh — automation script for correctness bisect

Git Master — Git Reference for Every Skill Level