Appearance
Git Bundle & Shallow / Partial Clone: Air-Gapped CI & Bandwidth-Constrained Pipelines
git bundle packs a repository (or subset of refs) into a single portable file — ideal for air-gapped transfers, offline CI runners, disaster recovery snapshots, and secure one-way pushes through restricted networks. git clone --depth and partial clone filters dramatically reduce bandwidth and disk usage for massive monorepos on CI.
Quick Reference
| Action | Command | When to Use |
|---|---|---|
| Bundle entire repo | git bundle create backup.bundle --all | Full offline mirror / DR snapshot |
| Bundle specific refs | git bundle create HEAD.bundle HEAD main develop | Selective push through restricted network |
| Verify a bundle | git bundle verify backup.bundle | Confirm bundle is self-contained before transfer |
| List refs in bundle | git bundle list-heads backup.bundle | Inspect what a bundle contains |
| Clone from bundle | git clone backup.bundle ./repo | Offline clone / restore from bundle |
| Pull from bundle | git pull bundle.bundle main | Merge bundle into existing repo |
| Shallow clone | git clone --depth 50 --branch main <url> | CI runner with limited bandwidth |
| Incremental shallow fetch | git fetch --depth 100 | Extend shallow history on CI |
| Full history | git fetch --unshallow | When a late pipeline stage needs full history |
| Partial clone (no blobs) | git clone --filter=blob:none <url> | Large monorepo where you only need source tree |
| Sparse checkout | git sparse-checkout set src/app build/ | Only needed directories in partial clone |
Part 1: git bundle — Offline Transfer and CI Backup
Why Bundle?
Most CI systems fetch directly from a Git server. But production environments with DMZ isolation, on-premise Forgejo/Gitea behind a VPN, or air-gapped datacenters cannot reach upstream. git bundle solves this by serialising Git objects and refs into a single file that can be transferred via SCP, USB, S3 pre-signed URL, or any file-based mechanism.
Common DevOps scenarios:
- DR snapshot: Bundle a production repo before a risky migration or major refactor; store the
.bundlefile in object storage. - One-way sync through a jump host:
git bundle createon the inside, transfer via SCP to the DMZ, thengit pull bundle.bundleon the air-gapped runner. - Offline code review: Bundle a feature branch and all dependent commits to share without exposing the full repo.
- CI artifact: In a pipeline that cannot reach the Git server (network policy), the previous job bundles
--all, publishes as a build artifact, and the next job clones from it.
Creating a Bundle
bash
# Full repository bundle (all branches + tags)
git bundle create /backups/repo-$(date +%Y%m%d).bundle --all
# Selective: only the refs you need for the CI job
git bundle create ci-input.bundle HEAD main release/2.4.1
# Incremental: bundle only commits since the last run
git bundle create incremental.bundle ^last-build-commit HEAD mainThe ^<commit> <refspec> syntax means "everything reachable from <refspec> but not from <commit>". Use git rev-parse last-build-commit to get the SHA from the previous pipeline run's env var or artifact.
Verifying and Inspecting a Bundle
Always verify before transferring — a corrupted or incomplete bundle wastes time on an air-gapped system.
bash
# Check the bundle is self-contained and covers the claimed refs
git bundle verify /path/to/repo.bundle
# Exit 0: bundle is complete and valid
# Exit 1: refs are missing or bundle is corrupt
# List all refs packed into the bundle
git bundle list-heads /path/to/repo.bundleCloning from / Pulling into a Bundle
bash
# Fresh clone from bundle
git clone /backups/repo-$(date +%Y%m%d).bundle ./workspace
# Clone to a specific branch (if bundle only contains one branch)
git clone -b main /backups/repo-$(date +%Y%m%d).bundle ./workspace
# Pull/merge bundle into an existing repo (common in CI chains)
cd existing-repo
git pull /path/to/ci-input.bundle mainCI Pipeline Example: Two-Job Air-Gapped Setup
Job 1 (has Git server access):
bash
#!/bin/bash
set -euo pipefail
# Create bundle of the trigger commit and main branch
GIT_COMMIT=$(git rev-parse HEAD)
git bundle create repo.bundle --tagged-refs \
HEAD "$GIT_COMMIT" main origin/main
# Upload as pipeline artifact (e.g., Azure Pipelines, GitHub Actions, etc.)
# In GitHub Actions:
echo "artifact_name=repo" >> $GITHUB_ENVJob 2 (air-gapped runner — no Git server access):
bash
#!/bin/bash
set -euo pipefail
# Download artifact: repo.bundle
# Then:
git clone repo.bundle ./app
cd app
git bundle verify repo.bundle # sanity check
npm ci
npm run buildBundle Size and Object Packing
Bundles are thin on first creation (they store all reachable objects). Use git repack before bundling for maximum portability:
bash
git bundle create clean.bundle --all
git clone clean.bundle /tmp/check
cd /tmp/check && git repack -ad
# Re-bundle for a smaller file:
git bundle create ../final.bundle --allPart 2: Shallow Clone — Speed for CI Runners
--depth: Fixed History Depth
bash
# Clone with last 50 commits on main
git clone --depth 50 --branch main https://git.internal/repo.git ./app
# Verify depth
cd ./app
git log --oneline | wc -l # should be ~50
# Check total object count saved
git count-objects -vH--single-branch (also --no-single-branch)
bash
# Only history needed for this branch (default with --depth)
git clone --depth 1 --branch main https://git.internal/repo.git ./app
# Clone a single branch without depth limit (no history merge commits)
git clone --single-branch --branch main https://git.internal/repo.git ./app--branch with Tags
bash
# Shallow clone a specific release tag (no all-branches history)
git clone --depth 1 --branch v2.4.1 https://git.internal/repo.git ./releaseExtending a Shallow Clone
bash
# Fetch more history (e.g., from 50 to 200 commits)
git fetch --depth 200
# Fetch all remaining history
git fetch --unshallowWarning: --unshallow on a shallow clone that contains merge commits may fail with fatal: refusing to merge unrelated histories. Resolve with git pull --allow-unrelated-histories or re-clone without --depth if full history is truly needed.
Shallow Clone in CI: Makefile / Script Pattern
bash
# In Dockerfile / CI script
GIT_DEPTH="${GIT_DEPTH:-50}"
git clone --depth "$GIT_DEPTH" --branch "${GIT_BRANCH:-main}" \
"$REPO_URL" /app
cd /app
# For pipelines that need full history only in certain stages:
# Stage 1: shallow (fast install/build)
# Stage N: fetch full history only when needed
if [ "$NEED_FULL_HISTORY" = "1" ]; then
git fetch --unshallow
fiPart 3: Partial Clone — Monorepo Without the Weight
For repos over 1 GB, shallow clone alone is insufficient. git clone --filter downloads only the Git object graph (commits, trees), leaving blobs (file content) to be fetched on demand.
--filter=blob:none — Tree-Only Clone
bash
git clone --filter=blob:none https://git.internal/monorepo.git ./app
# Downloads commit graph and directory structure; file content fetched lazily
# Typical size reduction: 60–90% for repos with large binary assets--filter=blob:limit=<n>[kmg] — Filter by Size
bash
# Exclude blobs larger than 10 MB from initial download
git clone --filter=blob:limit=10m https://git.internal/monorepo.git ./appSparse Checkout — Only the Directory You Need
Partial clone fetches the full tree. Sparse checkout limits the working directory further:
bash
# After partial clone:
cd ./monorepo
# Only check out src/ and build/ directories (no lib/, docs/, etc.)
git sparse-checkout init --cone
git sparse-checkout set src/app build/ terraform/
# Verify sparse checkout
git sparse-checkout list
# Output: src/app build terraformFor non-cone mode (glob patterns):
bash
git sparse-checkout set --no-cone 'src/**' 'build/**' '!tests/**'Combining Filters
bash
# Partial clone + shallow depth + sparse checkout = minimum CI footprint
git clone \
--filter=blob:none \
--depth 30 \
--branch main \
--sparse \
https://git.internal/monorepo.git ./app
cd ./app
git sparse-checkout set src/api tests/e2eLazy Blob Fetch
When a partial clone is checked out, Git fetches blobs on first access (e.g., opening a file in an editor). For CI build scripts that need specific files:
bash
# Force-fetch a specific directory's files (avoids on-demand stalls during build)
git fetch --filter=blob:none --no-recurse-submodules origin -- src/frontend
# Or use rev-list to determine what blobs you need and pre-fetch them:
git rev-list --objects HEAD -- src/frontend | \
git fetch-pack --stdin --quantize=0 origin | \
git index-pack --stdinPart 4: Performance Comparison and Decision Tree
| Strategy | Network Transfer | Disk Usage | History Available | Best For |
|---|---|---|---|---|
| Full clone | 100% of repo | Full | Yes | Local dev, release builds |
Shallow clone --depth N | ~N× avg commit size | Minimal | Last N commits | CI install/test steps |
Partial --filter=blob:none | Commit graph only | ~5–20% | Yes | Monorepo CI build |
| Partial + sparse checkout | Commit graph + subset | ~1–10% | Yes | Mega-monorepo (NxN builds) |
| Bundle (full) | ~full repo size (once) | Full | Yes | Offline/air-gap CI |
| Bundle (incremental) | Diff only | Full | Yes | Frequent air-gap sync |
Decision tree:
- Can the runner reach the Git server? → No → Use
git bundle - Is the repo > 500 MB? → Yes → Use
--filter=blob:none - Does the pipeline only need one subdirectory? → Yes → Add
git sparse-checkout set - Is build speed critical? → Yes → Add
--depth 50(adjust to minimum viable) - Does a later stage need full history? → Yes → Plan
--unshallowonly for that stage
Security Considerations
- Bundles are not encrypted. Transport via HTTPS, SCP with key-based auth, or encrypt with
gpg --encryptbefore transfer if the bundle contains secrets (e.g., if a dev accidentally bundled a repo with.envfiles). - Verify bundle signatures if bundled from an untrusted source:
git bundle verifyonly checks completeness, not authenticity — use GPG-signed tags alongside the bundle. - Shallow clones reduce attack surface — fewer old commits mean fewer potential references to leaked credentials in commit messages.
- Partial clones do not reduce risk from malicious blobs that may exist in history; use
git filter-repoto scrub history before creating bundles.
Cleanup
bash
# Remove bundle files after successful transfer / CI run
rm -f repo.bundle incremental.bundle
# Convert shallow clone to full (after confirming no more incremental fetches needed)
git fetch --unshallow
# Convert partial clone back to full (download all remaining blobs)
git fetch --filter=blob:none --refetch origin
# (refetch downloads all blobs not already present)