Git LFS for CI/CD and DevOps¶

Git Large File Storage (LFS) replaces large binary files, machine learning models, test fixtures, and media assets with lightweight text pointer files inside Git. The actual binary payloads live on a dedicated LFS object store.

1. How Git LFS works¶

Standard Git repositories store every version of every tracked file in their object database. When binary files change across commits, Git stores full compressed snapshots, causing the .git directory to balloon.

Git LFS solves this by decoupling metadata from payload data:

Pointer file in Git history: Git tracks a text pointer file (typically 120 to 140 bytes) containing a SHA-256 checksum and file size.
Object storage backend: The real binary payload uploads to an HTTP/S3-backed LFS server.
Filter drivers: Git uses clean and smudge filters defined in .gitattributes to translate between pointer files and actual binaries during checkout and commit operations.

Anatomy of an LFS pointer¶

version https://git-lfs.github.com/spec/v1
oid sha256:7b1e4a3b8d4f6c9e0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f
size 104857600

2. CI/CD pipeline bottlenecks and solutions¶

The default behavior of git clone or git checkout on an LFS-enabled repository triggers the smudge filter for every tracked file. On ephemeral CI runners, this downloads gigabytes of assets that a test or build step might never use.

Eliminating redundant downloads with smudge bypass¶

Set the GIT_LFS_SKIP_SMUDGE environment variable to 1 during repository checkout. Git will check out only the lightweight pointer files.

# Fast checkout in CI: clone repository with pointers only
export GIT_LFS_SKIP_SMUDGE=1
git clone --depth 1 https://git.local.sneakysquid.xyz/hermes/large-repo.git workspace
cd workspace

Selective asset pulling¶

After checking out pointer files, fetch only the specific binaries required by the current build job:

# Pull only assets in the test fixture directory
git lfs pull --include="tests/fixtures/**"

# Exclude multi-gigabyte training weights
git lfs pull --exclude="models/checkpoints/**"

3. Core command reference¶

Initializing and tracking file patterns¶

Track binary patterns before adding files to the staging area:

# Install LFS hooks into current repository
git lfs install

# Track specific extensions
git lfs track "*.iso"
git lfs track "*.bin"
git lfs track "*.tar.gz"
git lfs track "*.parquet"

# Track an entire directory tree as LFS
git lfs track "assets/blobs/**"

# Stage the generated .gitattributes configuration
git add .gitattributes

Inspecting tracked files and pointers¶

# List all files tracked by LFS in the current commit
git lfs ls-files

# Verify whether a specific file on disk is an LFS pointer or full binary
git lfs pointer --file=weights/model.bin

# Check for pointer integrity across working tree
git lfs fsck

4. File locking for binary collaboration¶

Binary files cannot merge cleanly through three-way diff algorithms. Git LFS provides a locking API to prevent concurrent modifications on shared branches.

Configuring lockable attributes¶

Declare lockable extensions in .gitattributes:

*.psd filter=lfs diff=lfs merge=lfs -text lockable
*.blend filter=lfs diff=lfs merge=lfs -text lockable

When files match the lockable attribute, Git makes them read-only in the local filesystem until a developer explicitly locks them.

Lock and unlock workflow¶

# Lock a file before beginning edits
git lfs lock models/base.onnx

# List active locks across the repository
git lfs locks

# Release a lock after pushing changes
git lfs unlock models/base.onnx

# Force-unlock a stale lock (requires admin permissions on Forgejo/GitHub)
git lfs unlock --id=42 --force

5. Migrating legacy repository history¶

When large binary files were committed directly to Git history in the past, tracking them with LFS going forward does not shrink the existing .git directory. You must rewrite past commits using git lfs migrate.

Analyzing repository bloat¶

Identify the largest file types and paths in Git history:

# Scan history and list top 10 largest file patterns
git lfs migrate info --top=10

Converting historical blobs to LFS pointers¶

Rewrite history across all branches to convert matching historical files to LFS:

# Convert all archive and binary files in historical commits
git lfs migrate import \
  --include="*.zip,*.tar.gz,*.iso,*.bin,*.onnx" \
  --everything

# Push rewritten history to remote
git push --force --all
git push --force --tags

6. Runner cache management and garbage collection¶

On persistent CI workers and developer workstations, downloaded LFS blobs remain in .git/lfs/objects/. Pruning removes unreferenced or old objects to free local disk space.

# Check how much disk space can be reclaimed
git lfs prune --dry-run

# Prune objects older than 7 days that exist on the remote
git lfs prune --verify-remote --recent

# Deduplicate identical LFS blobs across worktrees and clones via hardlinks
git lfs dedup

7. Tuning network performance and concurrency¶

Git LFS transfers individual chunks concurrently. High-bandwidth CI nodes benefit from increasing transfer concurrency.

# Increase parallel upload and download workers
git config lfs.concurrenttransfers 16

# Limit network retries to fail fast in CI
git config lfs.transfer.maxretries 3

# Configure custom LFS endpoint if distinct from main Git remote
git config lfs.url "https://lfs.local.sneakysquid.xyz/api/v1/hermes/large-repo/info/lfs"