git bisect run — Automated Regression Hunting¶

git bisect run turns a binary search into a single command. Give it a script that exits 0 for "good" and non-zero for "bad", and Git does the rest. No manual checkout, no staring at prompts. For a DevOps engineer triaging CI failures at scale, it's "check 40 commits by hand" vs "go make coffee."

The existing git-bisect-ci-triage.md covers getting started with bisect. This guide goes deep on run, the automation workhorse, with patterns for real-world CI pipelines, flaky tests, multi-module repos, and Docker-based builds.

How bisect run works¶

git bisect start
git bisect bad HEAD            # current state is broken
git bisect good v2.3.0         # known working tag
git bisect run ./test-script.sh

What happens:

Git checks out the midpoint between good and bad.
Runs ./test-script.sh with no arguments.
Exit code 0 = "good" — narrows the search to later commits.
Any non-zero = "bad" — narrows to earlier commits.
Loop until exactly one commit is left. That's your culprit.

The script runs inside the bisect checkout. It has full filesystem access. Whatever it needs (node_modules, Go toolchain, make artifacts) must be built from scratch or cached externally.

Exit code contract¶

Get the exit codes right. Bisect reads one return value:

Exit code	Meaning	Bisect action
0	Good (passed)	Search moves to later range
1-124	Bad (failed)	Search moves to earlier range
125	Skip this commit	Runs `git bisect skip`; tries next candidate
126	Script cannot be run	Aborts bisect
127	Script not found	Aborts bisect
128+	Fatal signal	Aborts bisect

Exit codes 1-124 all mean "bad." There is no fine-grained distinction. If you need different handling for build failures vs test failures, write that logic into your script and map both to exit code 125 (skip) for non-regression failures.

Exit code 125 is your escape hatch for commits that cannot be evaluated. A broken build, missing dependency, or infrastructure outage all map here. Bisect will skip that commit and try the next candidate.

Building a robust test script¶

A production-ready bisect script does four things:

Build the project from the checked-out commit.
Run the reproduction case (test, benchmark, integration check).
Map results to exit codes (0 = good, 1-124 = bad, 125 = skip).
Clean up artifacts to avoid cross-contamination.

#!/bin/bash
# scripts/bisect-test.sh — template for bisect run
set -euo pipefail

# --- Build ---
# If the build fails, skip this commit (regression in build itself is noise)
if ! npm ci 2>/dev/null; then
  echo "SKIP: build failure at $(git rev-parse HEAD)"
  exit 125
fi

# --- Run the test ---
if npm run test:regression -- --grep="checkout" 2>&1; then
  # Test passed = commit is good
  exit 0
else
  # Test failed = commit is bad
  exit 1
fi

Build failures map to 125 because they don't tell you anything about the regression. The build itself may be broken for unrelated reasons at that commit.

Handling build artifacts¶

Bisect checkouts mutate the working tree. If builds leave artifacts, they can contaminate the next iteration:

#!/bin/bash
set -euo pipefail

# Clean artifacts from any previous bisect iteration
rm -rf node_modules dist .next 2>/dev/null || true

npm ci || exit 125
npm run build || exit 125
npm run test:regression && exit 0 || exit 1

Alternatively, use git clean -fdx at the top of the script. It removes all untracked files including any cache you placed in the working tree.

Using temporary directories¶

For zero-contamination builds, work in a temp dir:

#!/bin/bash
set -euo pipefail

WORK=$(mktemp -d)
trap "rm -rf $WORK" EXIT

cp -r . "$WORK"
cd "$WORK"

npm ci || exit 125
npm run build || exit 125
npm run test:regression && exit 0 || exit 1

This is slower because it copies the whole working tree, but it eliminates all state leakage. Best for long bisect sessions where the cost of a false positive outweighs the copy time.

Pattern: Bisect run with Docker¶

When the test environment has complex dependencies (specific DB version, service mesh, kernel module), isolate the entire evaluation in a container:

#!/bin/bash
# scripts/bisect-docker.sh
set -euo pipefail

# Build the Docker image for this commit
IMAGE_TAG="bisect-$(git rev-parse --short HEAD)"

if ! docker build -t "$IMAGE_TAG" -f ci/Dockerfile.bisect . 2>/dev/null; then
  echo "SKIP: docker build failed at $(git rev-parse HEAD)"
  exit 125
fi

# Run the test in a clean container, destroying it after
if docker run --rm "$IMAGE_TAG" npm run test:regression; then
  docker image rm "$IMAGE_TAG" >/dev/null 2>&1 || true
  exit 0
else
  docker image rm "$IMAGE_TAG" >/dev/null 2>&1 || true
  exit 1
fi

The container gives each commit a hermetic test environment. Each commit runs in isolation. The image build time is the cost, mitigated by mapping 125 on build failure so you skip commits that can't even build.

Pattern: Run against a specific file or path¶

When the regression lives in a single module or file, narrow the bisect scope:

#!/bin/bash
set -euo pipefail

# Only run tests for the payments module
cd services/payments || exit 125

npm ci || exit 125
npm run test:unit || exit 1

exit 0

The git bisect start command already lets you scope the search to commits touching a path:

git bisect start -- services/payments
git bisect bad HEAD
git bisect good v2.3.0
git bisect run ./scripts/bisect-payments.sh

This reduces the search space. A 500-commit full-repo bisect becomes 30 commits for a single module.

Pattern: Multi-criteria regression scoring¶

Sometimes one test passes but another fails, and you want to classify the failure severity:

#!/bin/bash
set -euo pipefail

mvn compile -q 2>/dev/null || exit 125

# Critical test
mvn test -pl core -Dtest="CriticalFlowTest" -q && CRITICAL_PASS=true || CRITICAL_PASS=false

# Performance test
mvn test -pl perf -Dtest="ScaleTest" -q && PERF_PASS=true || PERF_PASS=false

if $CRITICAL_PASS && $PERF_PASS; then
  exit 0  # good
elif ! $CRITICAL_PASS; then
  exit 1  # bad — critical test failed
else
  exit 0  # perf regression is acceptable for this search
fi

The exit code from the script is binary (0 vs non-zero), so you encode the decision tree inside the script. This pattern is useful when you're tracking a specific regression but the project has other flaky or non-critical tests.

Pattern: Flaky test tolerance with retry¶

#!/bin/bash
set -euo pipefail

npm ci || exit 125
npm run build || exit 125

for attempt in 1 2 3; do
  if npm run test:regression; then
    exit 0  # passed — commit is good
  fi
  echo "Attempt $attempt failed, retrying..."
  sleep 2
done

# All 3 attempts failed — commit is bad
exit 1

Three attempts with a 2-second gap between them. If even one passes, the commit is marked good. This tolerates transient CI infra issues while still catching real regressions.

Adaptive retry with result logging¶

#!/bin/bash
set -euo pipefail

npm ci || exit 125
npm run build || exit 125

RESULTS=""
for attempt in 1 2 3; do
  if npm run test:regression 2>/dev/null; then
    RESULTS="${RESULTS}pass "
  else
    RESULTS="${RESULTS}fail "
  fi
done

# Log the attempt pattern for later analysis
echo "HEAD=$(git rev-parse HEAD) results=$RESULTS" >> ../bisect-retry-log.txt

# If even one attempt passed, call it good
if [[ "$RESULTS" == *"pass"* ]]; then
  exit 0
fi
exit 1

The log file lives outside the checkout (note ../bisect-retry-log.txt) so it persists across bisect checkouts. After the run completes, review the log for flaky patterns.

Pattern: Skip unreachable commits¶

Some commits won't build across the entire range — package.json versions that don't resolve, compiler toolchain mismatches, infrastructure changes.

#!/bin/bash
set -euo pipefail

if [ ! -f "package.json" ]; then
  echo "SKIP: no package.json at $(git rev-parse HEAD)"
  exit 125
fi

# Check if the expected tool exists
if ! command -v node &>/dev/null; then
  exit 125
fi

npm ci || exit 125
npm run test:regression && exit 0 || exit 1

Exit 125 is not a failure — it's an acknowledgement that this commit can't be evaluated. Bisect logs the skip and continues.

Skip commit logging¶

Bisect saves skipped commits in .git/BISECT_LOG. After a run:

git bisect log | grep -E "^(# skip:|# bad:|# good:)"

Review the skip pattern. If more than 50% of commits were skipped, your test script is too brittle — stabilize the build step or widen the good/bad range.

Pattern: Bisect across merge-heavy history¶

By default, bisect follows the full commit graph including merges. This can confuse the search when a merge brings in dozens of commits that don't affect the regression. Use --first-parent to follow only the mainline:

git bisect start --first-parent
git bisect bad HEAD
git bisect good v2.3.0
git bisect run ./test.sh

This walks only the first-parent chain. Merges appear as single commits. You lose granularity inside the merge but gain speed and clarity. Use when the regression is almost certainly in mainline work, not in a merged side branch.

Pattern: Running bisect run from any repo state¶

Bisect changes your working tree with each checkout. Running inside its own worktree keeps the rest of your repo safe:

git worktree add ../bisect-runner HEAD
cd ../bisect-runner

git bisect start
git bisect bad HEAD
git bisect good v2.3.0
git bisect run ../scripts/test-plg.sh

git bisect reset
cd -
git worktree remove ../bisect-runner

The test script must use absolute paths or paths relative to the worktree (like ../scripts/), because bisect changes the working directory with each commit checkout.

Pattern: Parallel bisect runs on different modules¶

For a monorepo where multiple independent modules regressed, run parallel bisects in separate worktrees:

for module in payments inventory shipping; do
  git worktree add "../bisect-$module" HEAD
  cd "../bisect-$module"
  git bisect start -- "services/$module"
  git bisect bad HEAD
  git bisect good v2.3.0
  git bisect run "./scripts/test-$module.sh" &
  cd -
done

wait
echo "All bisect runs complete"

Each bisect runs in its own shell process, in its own worktree, with its own module's test suite. They share the .git/objects database so no object duplication. On a modern 8-core machine, 3 parallel bisects finish about 3x faster than serial.

Replaying a bisect session¶

When you need to reproduce a bisect result or share it with a colleague, save the log:

git bisect log > /tmp/bisect-run-2025-04-12.log

# Later — replay the exact same search
git bisect replay /tmp/bisect-run-2025-04-12.log

Replay is deterministic. It re-checks out every commit the original run touched. Useful when: - You want to verify a result before escalating. - A colleague says "I found the bad commit but can't reproduce." - You need to confirm the fix commit actually resolves the regression.

Using `bisect run` in CI pipelines¶

Forgejo/GitHub Actions/GitLab CI can trigger a bisect run automatically when a regression is detected:

# .forgejo/workflows/bisect-regression.yaml
on:
  workflow_dispatch:
    inputs:
      bad_ref:
        description: "Known bad ref (default: HEAD of main)"
        required: false
        default: "main"
      good_ref:
        description: "Known good ref (tag or SHA)"
        required: true

jobs:
  bisect:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0    # full history needed
      - name: Run bisect
        run: |
          git bisect start
          git bisect bad origin/${{ inputs.bad_ref }}
          git bisect good ${{ inputs.good_ref }}
          git bisect run ci/scripts/bisect-runner.sh
      - name: Report result
        run: |
          git bisect log

The CI runner needs full history (fetch-depth: 0 by default is shallow). Partial clones with blob:none work since bisect only needs tree metadata, not blob contents.

CI bisect with Forgejo¶

- name: Automated bisect
  run: |
    git config remote.origin.fetch "+refs/*:refs/*"
    git fetch origin
    git bisect start
    git bisect bad origin/main
    git bisect good refs/tags/${{ github.event.inputs.good_tag }}
    git bisect run ci/bisect-ci.sh 2>&1 | tee /tmp/bisect-result

The refs/*:refs/* fetch ensures all refs (including tags) are available for the good boundary.

Debugging a bisect run¶

When bisect run gives a strange result, inspect the full search path:

# After the run completes
git bisect log

# Or during the run — open another terminal in the worktree
tail -f .git/BISECT_LOG

The log shows every commit evaluated, the script's exit code, and the decision (good, bad, or skip). Look for:

Consecutive bad commits. The range may be wrong, and the "good" boundary might actually be bad.
Excessive skips. Test script is too brittle. Stabilize the build step first.
Short search. Fewer than 3 steps for hundreds of commits means the good and bad boundaries are too close.

Forcing a specific commit¶

If you suspect a commit is mislabeled by bisect, confirm it manually:

git checkout <suspect-sha>
./your-test-script.sh
echo $?

If exit code contradicts what bisect recorded, your test script is non-deterministic or the environment changed between runs.

Terminating a bisect run early¶

git bisect reset

This restores HEAD to the original branch and exits the bisect state. Use it when: - The test script is producing garbage results. - You realize the good/bad ranges are wrong. - The build takes too long per commit (switch to a narrower range or lighter test).

Useful Aliases¶

[alias]
    brun  = bisect run
    blog  = bisect log
    bplay = bisect replay
    breset = bisect reset

Comparison: Manual bisect vs bisect run¶

Aspect	Manual `git bisect`	`git bisect run`
Human at keyboard	Required for every step	Only to start and evaluate result
Handles 100+ commits	7+ manual checkouts	One command
Flaky tests	You decide per commit	Must build tolerance into script
CI integration	Unsuitable	Native (script-based)
Reproducibility	Low (human decisions vary)	High (same script, same result)