Appearance
Git LFS for CI/CD and DevOps
Git Large File Storage (LFS) replaces large binary files, machine learning models, test fixtures, and media assets with lightweight text pointer files inside Git. The actual binary payloads live on a dedicated LFS object store.
1. How Git LFS works
Standard Git repositories store every version of every tracked file in their object database. When binary files change across commits, Git stores full compressed snapshots, causing the .git directory to balloon.
Git LFS solves this by decoupling metadata from payload data:
- Pointer file in Git history: Git tracks a text pointer file (typically 120 to 140 bytes) containing a SHA-256 checksum and file size.
- Object storage backend: The real binary payload uploads to an HTTP/S3-backed LFS server.
- Filter drivers: Git uses clean and smudge filters defined in
.gitattributesto translate between pointer files and actual binaries during checkout and commit operations.
Anatomy of an LFS pointer
text
version https://git-lfs.github.com/spec/v1
oid sha256:7b1e4a3b8d4f6c9e0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f
size 1048576002. CI/CD pipeline bottlenecks and solutions
The default behavior of git clone or git checkout on an LFS-enabled repository triggers the smudge filter for every tracked file. On ephemeral CI runners, this downloads gigabytes of assets that a test or build step might never use.
Eliminating redundant downloads with smudge bypass
Set the GIT_LFS_SKIP_SMUDGE environment variable to 1 during repository checkout. Git will check out only the lightweight pointer files.
bash
# Fast checkout in CI: clone repository with pointers only
export GIT_LFS_SKIP_SMUDGE=1
git clone --depth 1 https://git.local.sneakysquid.xyz/hermes/large-repo.git workspace
cd workspaceSelective asset pulling
After checking out pointer files, fetch only the specific binaries required by the current build job:
bash
# Pull only assets in the test fixture directory
git lfs pull --include="tests/fixtures/**"
# Exclude multi-gigabyte training weights
git lfs pull --exclude="models/checkpoints/**"3. Core command reference
Initializing and tracking file patterns
Track binary patterns before adding files to the staging area:
bash
# Install LFS hooks into current repository
git lfs install
# Track specific extensions
git lfs track "*.iso"
git lfs track "*.bin"
git lfs track "*.tar.gz"
git lfs track "*.parquet"
# Track an entire directory tree as LFS
git lfs track "assets/blobs/**"
# Stage the generated .gitattributes configuration
git add .gitattributesInspecting tracked files and pointers
bash
# List all files tracked by LFS in the current commit
git lfs ls-files
# Verify whether a specific file on disk is an LFS pointer or full binary
git lfs pointer --file=weights/model.bin
# Check for pointer integrity across working tree
git lfs fsck4. File locking for binary collaboration
Binary files cannot merge cleanly through three-way diff algorithms. Git LFS provides a locking API to prevent concurrent modifications on shared branches.
Configuring lockable attributes
Declare lockable extensions in .gitattributes:
text
*.psd filter=lfs diff=lfs merge=lfs -text lockable
*.blend filter=lfs diff=lfs merge=lfs -text lockableWhen files match the lockable attribute, Git makes them read-only in the local filesystem until a developer explicitly locks them.
Lock and unlock workflow
bash
# Lock a file before beginning edits
git lfs lock models/base.onnx
# List active locks across the repository
git lfs locks
# Release a lock after pushing changes
git lfs unlock models/base.onnx
# Force-unlock a stale lock (requires admin permissions on Forgejo/GitHub)
git lfs unlock --id=42 --force5. Migrating legacy repository history
When large binary files were committed directly to Git history in the past, tracking them with LFS going forward does not shrink the existing .git directory. You must rewrite past commits using git lfs migrate.
Analyzing repository bloat
Identify the largest file types and paths in Git history:
bash
# Scan history and list top 10 largest file patterns
git lfs migrate info --top=10Converting historical blobs to LFS pointers
Rewrite history across all branches to convert matching historical files to LFS:
bash
# Convert all archive and binary files in historical commits
git lfs migrate import \
--include="*.zip,*.tar.gz,*.iso,*.bin,*.onnx" \
--everything
# Push rewritten history to remote
git push --force --all
git push --force --tags6. Runner cache management and garbage collection
On persistent CI workers and developer workstations, downloaded LFS blobs remain in .git/lfs/objects/. Pruning removes unreferenced or old objects to free local disk space.
bash
# Check how much disk space can be reclaimed
git lfs prune --dry-run
# Prune objects older than 7 days that exist on the remote
git lfs prune --verify-remote --recent
# Deduplicate identical LFS blobs across worktrees and clones via hardlinks
git lfs dedup7. Tuning network performance and concurrency
Git LFS transfers individual chunks concurrently. High-bandwidth CI nodes benefit from increasing transfer concurrency.
bash
# Increase parallel upload and download workers
git config lfs.concurrenttransfers 16
# Limit network retries to fail fast in CI
git config lfs.transfer.maxretries 3
# Configure custom LFS endpoint if distinct from main Git remote
git config lfs.url "https://lfs.local.sneakysquid.xyz/api/v1/hermes/large-repo/info/lfs"