Appearance
Git debugging and diagnostics with GIT_TRACE
CI pipelines, Git hooks, and large-repo operations fail silently or with cryptic output ("fatal: protocol error", "bad object", "did not send all necessary objects"). The default Git error message tells you that something broke, not where or why. Git's built-in trace system exposes the internal state — protocol negotiation, pack transfer, credential exchange, ref resolution — across 9 environment variables.
This guide covers each variable, what it reveals, and the single-shot invocation that drops a complete diagnostics bundle for a failing CI job.
Quick capture: full diagnostics bundle
When a pipeline step fails with an opaque Git error, run the failing command under a blanket trace:
bash
# Collect every trace category in one session
GIT_TRACE=1 \
GIT_TRACE_PACKET=1 \
GIT_TRACE_SETUP=1 \
GIT_TRACE_PERFORMANCE=1 \
GIT_TRACE_SHALLOW=1 \
GIT_CURL_VERBOSE=1 \
git fetch origin main 2>&1 | tee git-trace-$(date +%s).logThe single git-trace-<ts>.log file contains the entire negotiation and transport story.
Trace variables reference
GIT_TRACE=1
Prints every invoked Git command, subcommand, and external helper call — the call tree. When a large CI script runs 50 git calls and one of them fails 20 steps in, GIT_TRACE shows you exactly which invocation returned the non-zero exit code and how long each call took to start.
bash
# All Git commands and their arguments
GIT_TRACE=1 git log --oneline -1
# Sample output:
# 19:30:01.123456 git.c:470 trace: built-in: git log '--oneline' '-1'
# 19:30:01.125432 read-cache.c:2188 trace: index uses .gitmodules, setup work-treeGIT_TRACE_PACKET=1
Dumps every packet sent and received over the Git wire protocol — the full conversation between client and server during fetch, push, clone, or ls-remote. Indispensable when the error is "protocol error", "expected old/new/ref", "did not receive packfile", or any server -- client mismatch.
bash
# Full wire protocol dump
GIT_TRACE_PACKET=1 git fetch origin main 2>&1 | head -40
# Sample:
# 19:30:01.456789 pkt-line.c:80 trace: packet: git< 0000000000000000000000000000000000000000 HEAD refs/heads/main
# 19:30:01.456790 pkt-line.c:80 trace: packet: git< 0000
# 19:30:01.458901 pkt-line.c:80 trace: packet: git> want 7b3c8a9e5f1d2c4e6f8a0b1c2d3e4f5a6b7c8d9e
# 19:30:01.458902 pkt-line.c:80 trace: packet: git> done
# 19:30:01.461234 pkt-line.c:80 trace: packet: git< ACK 7b3c8a9e5f1d2c4e6f8a0b1c2d3e4f5a6b7c8d9e common
# 19:30:01.461235 pkt-line.c:80 trace: packet: git< NAKA want line tells you exactly which commit the client requested — if it's an old SHA that doesn't exist on the remote, the negotiation was stale. NAK without ACK means the server has none of your advertised refs, hinting at a forced-push divergence or shallow boundary.
GIT_TRACE_SETUP=1
Shows Git's working-directory discovery — how setup_git_directory() figured out where .git/ is, which config files it loaded, and in what order. Debugs "not a git repository" errors when a CI runner's GIT_DIR or current directory is wrong.
bash
GIT_TRACE_SETUP=1 git status
# Sample:
# trace: setup: found gitdir: /builds/project/.git
# trace: setup: setup_git_directory: gitdir=/builds/project/.git (cwd=/builds/project/app)
# trace: setup: config /etc/gitconfig
# trace: setup: config /builds/project/.git/configGIT_TRACE_PERFORMANCE=1
Records wall-clock time for each step — reading the index, loading config, traversing the object graph, writing the pack. When a clone that should take 30 seconds takes 6 minutes, this tells you where the time went.
bash
GIT_TRACE_PERFORMANCE=1 git status 2>&1
# Sample:
# 19:30:01.481234 read-cache.c:1769 trace: performance: 0.002345 s (read_cache)
# 19:30:01.489012 diff-lib.c:1695 trace: performance: 0.005678 s (diff-index --cached)
# 19:30:01.491234 status.c:1067 trace: performance: 0.002123 s (run_diff_files)The cumulative column tells you the slowest step. In large monorepos, read_cache dominates when the index has 100k+ entries.
GIT_TRACE_SHALLOW=1
Debugs shallow clone boundary negotiation, unshallow commits, and graft resolution. Use when --depth=1 clones fail with "fatal: error in object" or the shallow boundary doesn't match the server's.
bash
GIT_TRACE_SHALLOW=1 git fetch --depth=1 origin main
# trace: shallow: new_depth = 1
# trace: shallow: sending shallows: abc123...
# trace: shallow: got unshallow points: xyz789...GIT_CURL_VERBOSE=1
Everything cURL sends and receives — DNS resolution, TLS handshake, HTTP headers, transfer speed, redirects, and error codes. For HTTPS remotes, this is the single most useful diagnostic when clones/fetches/pushes fail.
bash
GIT_CURL_VERBOSE=1 git fetch origin main 2>&1
# Sample:
# * Trying 192.168.1.100:443...
# * Connected to git.local.sneaksquid.xyz (192.168.1.100) port 443
# * TLS 1.3 handshake complete
# > GET /hermes/git-master.git/info/refs?service=git-upload-pack HTTP/1.1
# > Host: git.local.sneaksquid.xyz
# > Accept: */*
# < HTTP/1.1 401 Authorization Required
# < WWW-Authenticate: Basic realm="Git"
# * Authorization failedCommon patterns from GIT_CURL_VERBOSE:
| CURL output | Likely issue |
|---|---|
SSL certificate problem: self-signed certificate | Internal CA cert missing from CI container |
401 Authorization Required every time | Credential helper not configured, token expired, or no cached auth |
HTTP 417 Expectation Failed | Reverse proxy rejects Expect: 100-continue header |
Transfer closed with N bytes remaining | Proxy timeout or interrupted connection |
Empty reply from server | Server closed connection before finishing response |
schannel: next InitializeSecurityContext failed | Windows certificate store doesn't trust the CA |
GIT_TRACE_PACK_ACCESS=1
Records every object access with the source file and offset — loose object or packfile, byte position, delta base resolution. Extremely verbose. Use when suspecting a corrupt pack or slow delta resolution.
bash
GIT_TRACE_PACK_ACCESS=1 git rev-list --all --objects --countGIT_TRACE_CURL_NO_DATA=1
Suppresses the actual HTTP body (pack data) from GIT_CURL_VERBOSE output while keeping headers. When GIT_CURL_VERBOSE dumps too much binary pack data to scroll through useful header content.
bash
GIT_CURL_VERBOSE=1 GIT_TRACE_CURL_NO_DATA=1 git fetch originGIT_TRACE_REDIRECTS (Windows)
On Windows, traces path resolution for git.exe redirecting to git-wrapper.exe or using GIT_TEXT_DOMAIN_DIR. Rarely needed on Linux CI runners but important for mixed-platform devops.
Practical diagnostics scenarios
Scenario 1: A fetch that fails with "protocol error"
bash
# 1. Capture the full wire-level conversation
GIT_TRACE_PACKET=1 GIT_TRACE=1 git fetch origin main 2>&1 > fetch-trace.log
# 2. Check for a ref mismatch — does the remote advertise the ref we're asking for?
grep "git< " fetch-trace.log | head -10
# If the remote doesn't advertise refs/heads/main, the server side lost the ref
# or the refspec is wrong.
# 3. Check for in-band errors — does the server send "ERR" packets?
grep "ERR" fetch-trace.log
# "ERR\n" mid-stream usually means a pre-receive hook rejected the push.Scenario 2: Slow clone in CI
bash
# 1. Capture performance profile
GIT_TRACE_PERFORMANCE=1 git clone --depth=1 \
https://github.com/org/big-repo.git ci-build 2>&1 > clone-performance.log
# 2. Aggregate timing by step
grep "performance: " clone-performance.log | \
awk -F'trace: performance: ' '{print $2}' | \
sort -t' ' -k1 -rn | head -5
# 3.895 s (read_cache)
# 1.234 s (traverse_commit_list)
# 0.987 s (check_connected)
# If read_cache dominates, the index is large — consider sparse checkout.
# If check_connected dominates, object completeness check is scanning too many refs.Scenario 3: Authentication failure in automated pipeline
bash
# 1. See the full HTTP auth negotiation
GIT_CURL_VERBOSE=1 git fetch origin main 2>&1 | grep -E '(Authorization|HTTP/|Auth|error)'
# 2. Check if credentials are being sent at all
# If the request never includes "Authorization: Basic" or "Authorization: Bearer",
# the credential helper didn't populate. Check GIT_ASKPASS or credential.helper config.
# 3. For token-based auth, verify the token isn't expired without throwing it in a log
# by checking the response status line
GIT_CURL_VERBOSE=1 git ls-remote https://git.local.sneaksquid.xyz/hermes/git-master.git 2>&1
# 200 OK = authenticated, 401 = no valid tokenScenario 4: Pipe Git traces to structured logging
For CI systems that aggregate structured JSON logs:
bash
# Convert trace output to JSON (using a wrapper)
gittrace2json() {
local logfile="$1"
while IFS= read -r line; do
if [[ "$line" =~ ^([0-9:.]+)\ ([^ ]+)\ trace:\ (.*)$ ]]; then
jq -n --arg ts "${BASH_REMATCH[1]}" \
--arg file "${BASH_REMATCH[2]}" \
--arg msg "${BASH_REMATCH[3]}" \
'{timestamp: $ts, file: $file, message: $msg}'
fi
done < "$logfile"
}Config-based trace (persistent)
For a CI runner where you want traces on every Git operation without wrapping every call:
bash
# Enable trace permanently for one repo
git config --local trace.perf 1 # GIT_TRACE_PERFORMANCE
git config --local trace.pack 1 # GIT_TRACE_PACKET
git config --local trace.setup 1 # GIT_TRACE_SETUP
git config --local trace.curl 1 # GIT_CURL_VERBOSE
git config --local http.curlVerbose 1 # also enables curl verbose via config
# Traces go to stderr by default. Redirect to a file:
git config --local trace.output /var/log/git-traces/runner-$(hostname).logRun git config --local --list | grep trace afterward to verify which categories are active.
When not to use GIT_TRACE
- In production pipelines without log redaction. Trace output can contain tokens, cookies, and
Authorizationheaders. Pipe throughgrep -vE '(Authorization|access_token|password)'or redirect to a dedicated log collection service that strips secrets before indexing. GIT_TRACE_PACK_ACCESSon repos with >50k objects. It generates gigabytes of output per operation. Use only when narrowed to a specific corrupt-object investigation.GIT_TRACE_PACKETin performance benchmarks. The enormous output volume distorts timing measurements. Run production benchmarks with onlyGIT_TRACE_PERFORMANCE.- Detached CI steps with no log capture. If the CI runner doesn't collect stderr,
2>&1redirect is mandatory. Without it, traces go to stderr and vanish.
Summary
| Variable | Debguish when | What it exposes |
|---|---|---|
GIT_TRACE | Any failed Git command | Call tree — which call failed, what arguments |
GIT_TRACE_PACKET | Protocol errors, negotiation failures | Raw wire-protocol frames between client and server |
GIT_TRACE_SETUP | "not a git repository" | Directory discovery, config load order |
GIT_TRACE_PERFORMANCE | Slow operations | Wall-clock time per internal Git step |
GIT_CURL_VERBOSE | Auth, TLS, proxy, HTTP errors | Full HTTPS request/response chain |
GIT_TRACE_SHALLOW | Shallow clone boundary errors | Depth negotiation and unshallow points |
GIT_TRACE_PACK_ACCESS | Corrupt pack hypothesis | Individual object access source and offset |
The full diagnostics bundle — GIT_TRACE=1 GIT_TRACE_PACKET=1 GIT_TRACE_SETUP=1 GIT_TRACE_PERFORMANCE=1 GIT_TRACE_SHALLOW=1 GIT_CURL_VERBOSE=1 git <command> 2>&1 — is the universal first-response for opaque Git failures in CI.