Git object store forensics and emergency repository repair¶
Production systems, CI runners, and developer workstations occasionally suffer ungraceful shutdowns, disk space exhaustion, or interrupted processes. These events leave Git repositories in corrupted states with broken lock files, damaged object hashes, missing HEAD pointers, or unreadable pack indices.
This guide provides forensic techniques and step-by-step procedures to diagnose and repair damaged Git repositories without losing uncommitted work or commit history.
1. Common failure modes in production and CI runners¶
Git manages state across loose objects, packfiles, reference pointers, and index files. Different system interruptions damage distinct parts of .git/:
| Error message | Root cause | Impact |
|---|---|---|
fatal: Unable to create '.git/index.lock': File exists |
Process killed during index write | Blocks all staging, commit, and checkout actions |
fatal: bad index file sha1 signature |
Truncated or zeroed .git/index |
Prevents staging or status checks |
fatal: bad default revision 'HEAD' |
.git/HEAD empty or points to deleted ref |
Git cannot resolve the active branch |
error: inflate: data stream error (incorrect header check) |
Corrupt loose object file in .git/objects/ |
Git cannot read specific commit, tree, or blob |
error: packfile .git/objects/pack/pack-... is corrupted |
Damaged .pack or missing .idx file |
Access to packed commits or trees fails |
fatal: reference is not a tree: <sha> |
Loose ref points to missing tree object | Checkout or diff operations abort |
2. Diagnosing repository damage with git-fsck¶
Before modifying files inside .git/, identify the scope of corruption with git fsck.
2.1 Understanding fsck output¶
git fsck categorizes issues into distinct alerts:
dangling <type> <sha>: Valid objects that no branch or tag references. These are normal after rebases or discarded stashes.missing <type> <sha>: An existing commit or tree points to an object that does not exist in.git/objects/.corrupt <type> <sha>: An object exists on disk, but its content fails SHA verification or zlib decompression.zero-padded filemode: Non-standard file permissions recorded in a tree object.
3. Resolving orphaned lock files¶
When a CI agent is terminated by an out-of-memory killer or timeout, lock files remain behind in .git/. These prevent subsequent jobs from acquiring locks.
3.1 Identifying stale locks¶
Check for any .lock files in the repository metadata:
Common locks include:
- .git/index.lock: Created during staging, commits, or branch switches.
- .git/HEAD.lock: Created during branch checkouts or detached head switches.
- .git/refs/heads/<branch>.lock: Created during fast-forward updates or commit creation.
- .git/shallow.lock: Created when updating shallow clone depth.
3.2 Safe removal process¶
Before deleting a lock file, verify that no running Git process owns it:
# Check if any git process is active in the repository
pgrep -fl "git" || true
# Remove stale lock files safely
find .git -name "*.lock" -type f -delete
4. Rebuilding a corrupt index¶
A zero-byte or truncated .git/index causes fatal: index file corrupt or fatal: bad index file sha1 signature. The index is a cache of the working tree against the commit history; you can rebuild it from HEAD without losing working copy changes.
4.1 Rebuild procedure¶
# Step 1: Back up the damaged index
mv .git/index .git/index.corrupt
# Step 2: Reset the index to HEAD without modifying working files
git reset --mixed HEAD
# Step 3: Verify the new index matches the working tree state
git status
If git reset fails because HEAD is unreadable, read the tree directly into the index:
5. Repairing damaged HEAD and reference files¶
If .git/HEAD or branch files under .git/refs/heads/ contain zero bytes or corrupt text, Git cannot determine the current commit.
5.1 Restoring .git/HEAD¶
When .git/HEAD is blank or unreadable:
# Check the target branch
cat .git/HEAD
# Write the symbolic reference for master or main
echo "ref: refs/heads/master" > .git/HEAD
5.2 Recovering branch refs from the reflog¶
If .git/refs/heads/master is empty or damaged, find the most recent valid commit hash from the reflog:
# Method 1: Read the last line of the branch reflog
tail -n 1 .git/logs/refs/heads/master | awk '{print $2}'
# Method 2: Read the global HEAD reflog
tail -n 5 .git/logs/HEAD
# Restore the branch pointer
VALID_SHA=$(tail -n 1 .git/logs/HEAD | awk '{print $2}')
echo "$VALID_SHA" > .git/refs/heads/master
5.3 Recovering refs from packed-refs¶
If loose reference files are missing, Git checks .git/packed-refs:
6. Recovering corrupt loose objects¶
When Git fails with error: inflate: data stream error or fatal: loose object <sha> is corrupt, a file in .git/objects/xx/ contains corrupt bytes.
6.1 Locating the corrupt object file¶
A Git object with hash 4b825dc642cb6eb9a060e54bf8d69288fbee4904 lives at:
Check if the object file has zero bytes:
Remove zero-byte object files after recording their hash names:
6.2 Fetching the missing object from a remote¶
The fastest way to replace a damaged or missing object is fetching it from the upstream repository:
# Fetch all objects from origin
git fetch origin --tags
# If specific objects are missing, fetch the full commit history
git fetch origin '+refs/heads/*:refs/remotes/origin/*' --unshallow || git fetch origin '+refs/heads/*:refs/remotes/origin/*'
6.3 Extracting objects from backup packfiles¶
If you have an uncorrupted clone or another remote, copy the object file directly into .git/objects/:
# Copy loose object from a clean clone
CLEAN_REPO="/path/to/backup/clone"
OBJECT_PATH="4b/825dc642cb6eb9a060e54bf8d69288fbee4904"
mkdir -p ".git/objects/$(dirname "$OBJECT_PATH")"
cp "$CLEAN_REPO/.git/objects/$OBJECT_PATH" ".git/objects/$OBJECT_PATH"
7. Repairing damaged packfiles and pack indices¶
Git bundles objects into .pack files paired with .idx index files under .git/objects/pack/.
7.1 Rebuilding a missing or corrupted .idx file¶
If a .idx file is damaged but the corresponding .pack file is intact, generate a fresh index with git index-pack:
cd .git/objects/pack
# Verify existing packfiles
for pack in *.pack; do
echo "Checking $pack..."
git verify-pack -v "$pack" || echo "FAILED: $pack"
done
# Rebuild index for a specific packfile
git index-pack pack-abcdef1234567890.pack
7.2 Unpacking objects from damaged packfiles¶
If a packfile is partially unreadable, unpack intact objects into loose files:
# Move packfile to a temporary location
mv .git/objects/pack/pack-corrupt.* /tmp/
# Unpack all readable objects into loose object store
git unpack-objects < /tmp/pack-corrupt.pack
8. Plumbing reconstruction for missing trees and commits¶
When an unrecoverable object leaves a gap in history, use Git plumbing commands to reconstruct valid objects.
8.1 Recreating a missing empty tree¶
The empty tree hash is universal in all SHA-1 Git repositories:
8.2 Recreating a missing tree from the working directory¶
If a tree object is missing, write the current working directory structure to the object database:
# Stage current directory files
git add -A
# Write the new tree object and output its hash
NEW_TREE=$(git write-tree)
echo "Generated tree: $NEW_TREE"
8.3 Synthesizing a commit object¶
Stitch the reconstructed tree into the commit history:
PARENT_COMMIT=$(git rev-parse HEAD~1 2>/dev/null || echo "")
if [ -n "$PARENT_COMMIT" ]; then
NEW_COMMIT=$(echo "Emergency repair commit" | git commit-tree "$NEW_TREE" -p "$PARENT_COMMIT")
else
NEW_COMMIT=$(echo "Emergency root repair commit" | git commit-tree "$NEW_TREE")
fi
# Point the active branch to the newly synthesized commit
git update-ref refs/heads/master "$NEW_COMMIT"
9. Automated CI health checks and self-healing¶
In automated CI environments, add a repository validation step before critical build jobs: