Skip to content

Git Bisect: Automated CI Failure Triage

git bisect performs a binary search across the commit graph to find the exact commit that introduced a regression. For a DevOps engineer running CI pipelines, this is faster and more precise than manual rollback or re-running the full matrix by hand.

Why a DevOps Engineer Needs This

  • A CI pipeline goes red at 3 AM; git log --oneline shows 80 commits since the last green tag. Manual checkout is painful.
  • git bisect run ./test-ci.sh automates the search; each checkout runs the test, bisect narrows to the bad commit in ~log2(N) steps (e.g. 7 runs for 100 commits instead of 100).
  • Combine with git worktree to isolate bisect checkouts so your main branch and editor state stay untouched.
  • Perfect for identifying the commit that broke a specific integration test, introduced a dependency conflict, or regressed a benchmark.

Quick Start

# Mark the known bad commit (current failing build) and a good base
# (last green tag or release)
git bisect start
git bisect bad HEAD
git bisect good v2.3.0

# Or in one line using ranges
git bisect start
git bisect bad HEAD
git bisect good v2.3.0

# After binary search completes
git bisect log   # review the full search path
git bisect reset # return to original branch

Pattern 1: Fully Automated Bisect with a Test Script

#!/bin/bash
# scripts/test-pass.sh — exit 0 = good, non-zero = bad
set -euo pipefail
npm ci >/dev/null 2>&1
npm run test:integration -- --grep="checkout"

chmod +x scripts/test-pass.sh
git bisect run ./scripts/test-pass.sh

When finished, git bisect prints the first bad commit SHA, message, and the exact line of the script that failed.

Pattern 2: Bisect with Worktree Isolation

# Keep your main checkout untouched
git worktree add ../bisect-runner HEAD
git worktree add ../bisect-runner -b bisect-isolation main

# Inside the isolated worktree
cd ../bisect-runner
git bisect start
git bisect bad HEAD
git bisect good v2.3.0
git bisect run ./test-pass.sh
git bisect reset

Pattern 3: Bisect on a Specific File / Dependency Change

When a single file like Dockerfile or package.json is the prime suspect, use git log -S or git log -G first:

git log -S "POSTGRES_VERSION=" --oneline -- Dockerfile
git log -G "redis" --oneline -- package-lock.json

Then narrow the bisect range to only the commits touching that context.

Pitfalls

  • Dirty worktree — git bisect will refuse with You need to resolve your current index first. Commit or stash first.
  • Merge commits — git bisect skips merges by default. If the regression is inside a merge, use git bisect skip selectively or reproduce manually.
  • Flaky tests — if the test is non-deterministic, bisect run can mis-label. Stabilize the test script first (e.g. pin seed, mock time, disable concurrency).
  • Binary artifacts — if the build produces binaries per commit, clean artifacts between iterations (git clean -fdx or make clean) in the test script.

Useful Aliases

# ~/.gitconfig
[alias]
    bisect-auto = !git bisect run ./scripts/test-pass.sh
    bisect-log  = bisect log

Verification

# After a run completes, verify the identified commit
git show $(cat .git/BISECT_LOG | head -1 | awk '{print $NF}')
git log --oneline $(cat .git/BISECT_LOG | head -1 | awk '{print $NF}')..HEAD

When Not to Use Bisect

  • The regression spans multiple unrelated commits; manual rollback with git revert is simpler.
  • The build takes 30+ minutes per commit; use a lighter reproduction script or binary search on a smaller matrix.