Skip to content

Git debugging and diagnostics with GIT_TRACE ​

CI pipelines, Git hooks, and large-repo operations fail silently or with cryptic output ("fatal: protocol error", "bad object", "did not send all necessary objects"). The default Git error message tells you that something broke, not where or why. Git's built-in trace system exposes the internal state — protocol negotiation, pack transfer, credential exchange, ref resolution — across 9 environment variables.

This guide covers each variable, what it reveals, and the single-shot invocation that drops a complete diagnostics bundle for a failing CI job.


Quick capture: full diagnostics bundle ​

When a pipeline step fails with an opaque Git error, run the failing command under a blanket trace:

bash
# Collect every trace category in one session
GIT_TRACE=1 \
GIT_TRACE_PACKET=1 \
GIT_TRACE_SETUP=1 \
GIT_TRACE_PERFORMANCE=1 \
GIT_TRACE_SHALLOW=1 \
GIT_CURL_VERBOSE=1 \
git fetch origin main 2>&1 | tee git-trace-$(date +%s).log

The single git-trace-<ts>.log file contains the entire negotiation and transport story.


Trace variables reference ​

GIT_TRACE=1 ​

Prints every invoked Git command, subcommand, and external helper call — the call tree. When a large CI script runs 50 git calls and one of them fails 20 steps in, GIT_TRACE shows you exactly which invocation returned the non-zero exit code and how long each call took to start.

bash
# All Git commands and their arguments
GIT_TRACE=1 git log --oneline -1

# Sample output:
# 19:30:01.123456 git.c:470   trace: built-in: git log '--oneline' '-1'
# 19:30:01.125432 read-cache.c:2188 trace: index uses .gitmodules, setup work-tree

GIT_TRACE_PACKET=1 ​

Dumps every packet sent and received over the Git wire protocol — the full conversation between client and server during fetch, push, clone, or ls-remote. Indispensable when the error is "protocol error", "expected old/new/ref", "did not receive packfile", or any server -- client mismatch.

bash
# Full wire protocol dump
GIT_TRACE_PACKET=1 git fetch origin main 2>&1 | head -40

# Sample:
# 19:30:01.456789 pkt-line.c:80   trace: packet: git< 0000000000000000000000000000000000000000 HEAD refs/heads/main
# 19:30:01.456790 pkt-line.c:80   trace: packet: git< 0000
# 19:30:01.458901 pkt-line.c:80   trace: packet: git> want 7b3c8a9e5f1d2c4e6f8a0b1c2d3e4f5a6b7c8d9e
# 19:30:01.458902 pkt-line.c:80   trace: packet: git> done
# 19:30:01.461234 pkt-line.c:80   trace: packet: git< ACK 7b3c8a9e5f1d2c4e6f8a0b1c2d3e4f5a6b7c8d9e common
# 19:30:01.461235 pkt-line.c:80   trace: packet: git< NAK

A want line tells you exactly which commit the client requested — if it's an old SHA that doesn't exist on the remote, the negotiation was stale. NAK without ACK means the server has none of your advertised refs, hinting at a forced-push divergence or shallow boundary.

GIT_TRACE_SETUP=1 ​

Shows Git's working-directory discovery — how setup_git_directory() figured out where .git/ is, which config files it loaded, and in what order. Debugs "not a git repository" errors when a CI runner's GIT_DIR or current directory is wrong.

bash
GIT_TRACE_SETUP=1 git status

# Sample:
# trace: setup: found gitdir: /builds/project/.git
# trace: setup: setup_git_directory: gitdir=/builds/project/.git (cwd=/builds/project/app)
# trace: setup: config /etc/gitconfig
# trace: setup: config /builds/project/.git/config

GIT_TRACE_PERFORMANCE=1 ​

Records wall-clock time for each step — reading the index, loading config, traversing the object graph, writing the pack. When a clone that should take 30 seconds takes 6 minutes, this tells you where the time went.

bash
GIT_TRACE_PERFORMANCE=1 git status 2>&1

# Sample:
# 19:30:01.481234 read-cache.c:1769 trace: performance: 0.002345 s  (read_cache)
# 19:30:01.489012 diff-lib.c:1695 trace: performance: 0.005678 s  (diff-index --cached)
# 19:30:01.491234 status.c:1067 trace: performance: 0.002123 s  (run_diff_files)

The cumulative column tells you the slowest step. In large monorepos, read_cache dominates when the index has 100k+ entries.

GIT_TRACE_SHALLOW=1 ​

Debugs shallow clone boundary negotiation, unshallow commits, and graft resolution. Use when --depth=1 clones fail with "fatal: error in object" or the shallow boundary doesn't match the server's.

bash
GIT_TRACE_SHALLOW=1 git fetch --depth=1 origin main

# trace: shallow: new_depth = 1
# trace: shallow: sending shallows: abc123...
# trace: shallow: got unshallow points: xyz789...

GIT_CURL_VERBOSE=1 ​

Everything cURL sends and receives — DNS resolution, TLS handshake, HTTP headers, transfer speed, redirects, and error codes. For HTTPS remotes, this is the single most useful diagnostic when clones/fetches/pushes fail.

bash
GIT_CURL_VERBOSE=1 git fetch origin main 2>&1

# Sample:
# *   Trying 192.168.1.100:443...
# * Connected to git.local.sneaksquid.xyz (192.168.1.100) port 443
# * TLS 1.3 handshake complete
# > GET /hermes/git-master.git/info/refs?service=git-upload-pack HTTP/1.1
# > Host: git.local.sneaksquid.xyz
# > Accept: */*
# < HTTP/1.1 401 Authorization Required
# < WWW-Authenticate: Basic realm="Git"
# * Authorization failed

Common patterns from GIT_CURL_VERBOSE:

CURL outputLikely issue
SSL certificate problem: self-signed certificateInternal CA cert missing from CI container
401 Authorization Required every timeCredential helper not configured, token expired, or no cached auth
HTTP 417 Expectation FailedReverse proxy rejects Expect: 100-continue header
Transfer closed with N bytes remainingProxy timeout or interrupted connection
Empty reply from serverServer closed connection before finishing response
schannel: next InitializeSecurityContext failedWindows certificate store doesn't trust the CA

GIT_TRACE_PACK_ACCESS=1 ​

Records every object access with the source file and offset — loose object or packfile, byte position, delta base resolution. Extremely verbose. Use when suspecting a corrupt pack or slow delta resolution.

bash
GIT_TRACE_PACK_ACCESS=1 git rev-list --all --objects --count

GIT_TRACE_CURL_NO_DATA=1 ​

Suppresses the actual HTTP body (pack data) from GIT_CURL_VERBOSE output while keeping headers. When GIT_CURL_VERBOSE dumps too much binary pack data to scroll through useful header content.

bash
GIT_CURL_VERBOSE=1 GIT_TRACE_CURL_NO_DATA=1 git fetch origin

GIT_TRACE_REDIRECTS (Windows) ​

On Windows, traces path resolution for git.exe redirecting to git-wrapper.exe or using GIT_TEXT_DOMAIN_DIR. Rarely needed on Linux CI runners but important for mixed-platform devops.


Practical diagnostics scenarios ​

Scenario 1: A fetch that fails with "protocol error" ​

bash
# 1. Capture the full wire-level conversation
GIT_TRACE_PACKET=1 GIT_TRACE=1 git fetch origin main 2>&1 > fetch-trace.log

# 2. Check for a ref mismatch — does the remote advertise the ref we're asking for?
grep "git< " fetch-trace.log | head -10
# If the remote doesn't advertise refs/heads/main, the server side lost the ref
# or the refspec is wrong.

# 3. Check for in-band errors — does the server send "ERR" packets?
grep "ERR" fetch-trace.log
# "ERR\n" mid-stream usually means a pre-receive hook rejected the push.

Scenario 2: Slow clone in CI ​

bash
# 1. Capture performance profile
GIT_TRACE_PERFORMANCE=1 git clone --depth=1 \
  https://github.com/org/big-repo.git ci-build 2>&1 > clone-performance.log

# 2. Aggregate timing by step
grep "performance: " clone-performance.log | \
  awk -F'trace: performance: ' '{print $2}' | \
  sort -t' ' -k1 -rn | head -5
# 3.895 s  (read_cache)
# 1.234 s  (traverse_commit_list)
# 0.987 s  (check_connected)

# If read_cache dominates, the index is large — consider sparse checkout.
# If check_connected dominates, object completeness check is scanning too many refs.

Scenario 3: Authentication failure in automated pipeline ​

bash
# 1. See the full HTTP auth negotiation
GIT_CURL_VERBOSE=1 git fetch origin main 2>&1 | grep -E '(Authorization|HTTP/|Auth|error)'

# 2. Check if credentials are being sent at all
# If the request never includes "Authorization: Basic" or "Authorization: Bearer",
# the credential helper didn't populate. Check GIT_ASKPASS or credential.helper config.

# 3. For token-based auth, verify the token isn't expired without throwing it in a log
# by checking the response status line
GIT_CURL_VERBOSE=1 git ls-remote https://git.local.sneaksquid.xyz/hermes/git-master.git 2>&1
# 200 OK = authenticated, 401 = no valid token

Scenario 4: Pipe Git traces to structured logging ​

For CI systems that aggregate structured JSON logs:

bash
# Convert trace output to JSON (using a wrapper)
gittrace2json() {
  local logfile="$1"
  while IFS= read -r line; do
    if [[ "$line" =~ ^([0-9:.]+)\ ([^ ]+)\ trace:\ (.*)$ ]]; then
      jq -n --arg ts "${BASH_REMATCH[1]}" \
             --arg file "${BASH_REMATCH[2]}" \
             --arg msg "${BASH_REMATCH[3]}" \
             '{timestamp: $ts, file: $file, message: $msg}'
    fi
  done < "$logfile"
}

Config-based trace (persistent) ​

For a CI runner where you want traces on every Git operation without wrapping every call:

bash
# Enable trace permanently for one repo
git config --local trace.perf 1         # GIT_TRACE_PERFORMANCE
git config --local trace.pack 1         # GIT_TRACE_PACKET
git config --local trace.setup 1        # GIT_TRACE_SETUP
git config --local trace.curl 1         # GIT_CURL_VERBOSE
git config --local http.curlVerbose 1   # also enables curl verbose via config

# Traces go to stderr by default. Redirect to a file:
git config --local trace.output /var/log/git-traces/runner-$(hostname).log

Run git config --local --list | grep trace afterward to verify which categories are active.


When not to use GIT_TRACE ​

  • In production pipelines without log redaction. Trace output can contain tokens, cookies, and Authorization headers. Pipe through grep -vE '(Authorization|access_token|password)' or redirect to a dedicated log collection service that strips secrets before indexing.
  • GIT_TRACE_PACK_ACCESS on repos with >50k objects. It generates gigabytes of output per operation. Use only when narrowed to a specific corrupt-object investigation.
  • GIT_TRACE_PACKET in performance benchmarks. The enormous output volume distorts timing measurements. Run production benchmarks with only GIT_TRACE_PERFORMANCE.
  • Detached CI steps with no log capture. If the CI runner doesn't collect stderr, 2>&1 redirect is mandatory. Without it, traces go to stderr and vanish.

Summary ​

VariableDebguish whenWhat it exposes
GIT_TRACEAny failed Git commandCall tree — which call failed, what arguments
GIT_TRACE_PACKETProtocol errors, negotiation failuresRaw wire-protocol frames between client and server
GIT_TRACE_SETUP"not a git repository"Directory discovery, config load order
GIT_TRACE_PERFORMANCESlow operationsWall-clock time per internal Git step
GIT_CURL_VERBOSEAuth, TLS, proxy, HTTP errorsFull HTTPS request/response chain
GIT_TRACE_SHALLOWShallow clone boundary errorsDepth negotiation and unshallow points
GIT_TRACE_PACK_ACCESSCorrupt pack hypothesisIndividual object access source and offset

The full diagnostics bundle — GIT_TRACE=1 GIT_TRACE_PACKET=1 GIT_TRACE_SETUP=1 GIT_TRACE_PERFORMANCE=1 GIT_TRACE_SHALLOW=1 GIT_CURL_VERBOSE=1 git <command> 2>&1 — is the universal first-response for opaque Git failures in CI.

Git Master — Git Reference for Every Skill Level