Skip to content

Git Bundle & Shallow / Partial Clone: Air-Gapped CI & Bandwidth-Constrained Pipelines ​

git bundle packs a repository (or subset of refs) into a single portable file — ideal for air-gapped transfers, offline CI runners, disaster recovery snapshots, and secure one-way pushes through restricted networks. git clone --depth and partial clone filters dramatically reduce bandwidth and disk usage for massive monorepos on CI.


Quick Reference ​

ActionCommandWhen to Use
Bundle entire repogit bundle create backup.bundle --allFull offline mirror / DR snapshot
Bundle specific refsgit bundle create HEAD.bundle HEAD main developSelective push through restricted network
Verify a bundlegit bundle verify backup.bundleConfirm bundle is self-contained before transfer
List refs in bundlegit bundle list-heads backup.bundleInspect what a bundle contains
Clone from bundlegit clone backup.bundle ./repoOffline clone / restore from bundle
Pull from bundlegit pull bundle.bundle mainMerge bundle into existing repo
Shallow clonegit clone --depth 50 --branch main <url>CI runner with limited bandwidth
Incremental shallow fetchgit fetch --depth 100Extend shallow history on CI
Full historygit fetch --unshallowWhen a late pipeline stage needs full history
Partial clone (no blobs)git clone --filter=blob:none <url>Large monorepo where you only need source tree
Sparse checkoutgit sparse-checkout set src/app build/Only needed directories in partial clone

Part 1: git bundle — Offline Transfer and CI Backup ​

Why Bundle? ​

Most CI systems fetch directly from a Git server. But production environments with DMZ isolation, on-premise Forgejo/Gitea behind a VPN, or air-gapped datacenters cannot reach upstream. git bundle solves this by serialising Git objects and refs into a single file that can be transferred via SCP, USB, S3 pre-signed URL, or any file-based mechanism.

Common DevOps scenarios:

  • DR snapshot: Bundle a production repo before a risky migration or major refactor; store the .bundle file in object storage.
  • One-way sync through a jump host: git bundle create on the inside, transfer via SCP to the DMZ, then git pull bundle.bundle on the air-gapped runner.
  • Offline code review: Bundle a feature branch and all dependent commits to share without exposing the full repo.
  • CI artifact: In a pipeline that cannot reach the Git server (network policy), the previous job bundles --all, publishes as a build artifact, and the next job clones from it.

Creating a Bundle ​

bash
# Full repository bundle (all branches + tags)
git bundle create /backups/repo-$(date +%Y%m%d).bundle --all

# Selective: only the refs you need for the CI job
git bundle create ci-input.bundle HEAD main release/2.4.1

# Incremental: bundle only commits since the last run
git bundle create incremental.bundle ^last-build-commit HEAD main

The ^<commit> <refspec> syntax means "everything reachable from <refspec> but not from <commit>". Use git rev-parse last-build-commit to get the SHA from the previous pipeline run's env var or artifact.

Verifying and Inspecting a Bundle ​

Always verify before transferring — a corrupted or incomplete bundle wastes time on an air-gapped system.

bash
# Check the bundle is self-contained and covers the claimed refs
git bundle verify /path/to/repo.bundle
# Exit 0: bundle is complete and valid
# Exit 1: refs are missing or bundle is corrupt

# List all refs packed into the bundle
git bundle list-heads /path/to/repo.bundle

Cloning from / Pulling into a Bundle ​

bash
# Fresh clone from bundle
git clone /backups/repo-$(date +%Y%m%d).bundle ./workspace

# Clone to a specific branch (if bundle only contains one branch)
git clone -b main /backups/repo-$(date +%Y%m%d).bundle ./workspace

# Pull/merge bundle into an existing repo (common in CI chains)
cd existing-repo
git pull /path/to/ci-input.bundle main

CI Pipeline Example: Two-Job Air-Gapped Setup ​

Job 1 (has Git server access):

bash
#!/bin/bash
set -euo pipefail
# Create bundle of the trigger commit and main branch
GIT_COMMIT=$(git rev-parse HEAD)
git bundle create repo.bundle --tagged-refs \
  HEAD "$GIT_COMMIT" main origin/main
# Upload as pipeline artifact (e.g., Azure Pipelines, GitHub Actions, etc.)
# In GitHub Actions:
echo "artifact_name=repo" >> $GITHUB_ENV

Job 2 (air-gapped runner — no Git server access):

bash
#!/bin/bash
set -euo pipefail
# Download artifact: repo.bundle
# Then:
git clone repo.bundle ./app
cd app
git bundle verify repo.bundle   # sanity check
npm ci
npm run build

Bundle Size and Object Packing ​

Bundles are thin on first creation (they store all reachable objects). Use git repack before bundling for maximum portability:

bash
git bundle create clean.bundle --all
git clone clean.bundle /tmp/check
cd /tmp/check && git repack -ad
# Re-bundle for a smaller file:
git bundle create ../final.bundle --all

Part 2: Shallow Clone — Speed for CI Runners ​

--depth: Fixed History Depth ​

bash
# Clone with last 50 commits on main
git clone --depth 50 --branch main https://git.internal/repo.git ./app

# Verify depth
cd ./app
git log --oneline | wc -l   # should be ~50

# Check total object count saved
git count-objects -vH

--single-branch (also --no-single-branch) ​

bash
# Only history needed for this branch (default with --depth)
git clone --depth 1 --branch main https://git.internal/repo.git ./app

# Clone a single branch without depth limit (no history merge commits)
git clone --single-branch --branch main https://git.internal/repo.git ./app

--branch with Tags ​

bash
# Shallow clone a specific release tag (no all-branches history)
git clone --depth 1 --branch v2.4.1 https://git.internal/repo.git ./release

Extending a Shallow Clone ​

bash
# Fetch more history (e.g., from 50 to 200 commits)
git fetch --depth 200

# Fetch all remaining history
git fetch --unshallow

Warning: --unshallow on a shallow clone that contains merge commits may fail with fatal: refusing to merge unrelated histories. Resolve with git pull --allow-unrelated-histories or re-clone without --depth if full history is truly needed.

Shallow Clone in CI: Makefile / Script Pattern ​

bash
# In Dockerfile / CI script
GIT_DEPTH="${GIT_DEPTH:-50}"
git clone --depth "$GIT_DEPTH" --branch "${GIT_BRANCH:-main}" \
  "$REPO_URL" /app
cd /app

# For pipelines that need full history only in certain stages:
# Stage 1: shallow (fast install/build)
# Stage N: fetch full history only when needed
if [ "$NEED_FULL_HISTORY" = "1" ]; then
  git fetch --unshallow
fi

Part 3: Partial Clone — Monorepo Without the Weight ​

For repos over 1 GB, shallow clone alone is insufficient. git clone --filter downloads only the Git object graph (commits, trees), leaving blobs (file content) to be fetched on demand.

--filter=blob:none — Tree-Only Clone ​

bash
git clone --filter=blob:none https://git.internal/monorepo.git ./app
# Downloads commit graph and directory structure; file content fetched lazily
# Typical size reduction: 60–90% for repos with large binary assets

--filter=blob:limit=<n>[kmg] — Filter by Size ​

bash
# Exclude blobs larger than 10 MB from initial download
git clone --filter=blob:limit=10m https://git.internal/monorepo.git ./app

Sparse Checkout — Only the Directory You Need ​

Partial clone fetches the full tree. Sparse checkout limits the working directory further:

bash
# After partial clone:
cd ./monorepo

# Only check out src/ and build/ directories (no lib/, docs/, etc.)
git sparse-checkout init --cone
git sparse-checkout set src/app build/ terraform/

# Verify sparse checkout
git sparse-checkout list
# Output: src/app build terraform

For non-cone mode (glob patterns):

bash
git sparse-checkout set --no-cone 'src/**' 'build/**' '!tests/**'

Combining Filters ​

bash
# Partial clone + shallow depth + sparse checkout = minimum CI footprint
git clone \
  --filter=blob:none \
  --depth 30 \
  --branch main \
  --sparse \
  https://git.internal/monorepo.git ./app

cd ./app
git sparse-checkout set src/api tests/e2e

Lazy Blob Fetch ​

When a partial clone is checked out, Git fetches blobs on first access (e.g., opening a file in an editor). For CI build scripts that need specific files:

bash
# Force-fetch a specific directory's files (avoids on-demand stalls during build)
git fetch --filter=blob:none --no-recurse-submodules origin -- src/frontend

# Or use rev-list to determine what blobs you need and pre-fetch them:
git rev-list --objects HEAD -- src/frontend | \
  git fetch-pack --stdin --quantize=0 origin | \
  git index-pack --stdin

Part 4: Performance Comparison and Decision Tree ​

StrategyNetwork TransferDisk UsageHistory AvailableBest For
Full clone100% of repoFullYesLocal dev, release builds
Shallow clone --depth N~N× avg commit sizeMinimalLast N commitsCI install/test steps
Partial --filter=blob:noneCommit graph only~5–20%YesMonorepo CI build
Partial + sparse checkoutCommit graph + subset~1–10%YesMega-monorepo (NxN builds)
Bundle (full)~full repo size (once)FullYesOffline/air-gap CI
Bundle (incremental)Diff onlyFullYesFrequent air-gap sync

Decision tree:

  1. Can the runner reach the Git server? → No → Use git bundle
  2. Is the repo > 500 MB? → Yes → Use --filter=blob:none
  3. Does the pipeline only need one subdirectory? → Yes → Add git sparse-checkout set
  4. Is build speed critical? → Yes → Add --depth 50 (adjust to minimum viable)
  5. Does a later stage need full history? → Yes → Plan --unshallow only for that stage

Security Considerations ​

  • Bundles are not encrypted. Transport via HTTPS, SCP with key-based auth, or encrypt with gpg --encrypt before transfer if the bundle contains secrets (e.g., if a dev accidentally bundled a repo with .env files).
  • Verify bundle signatures if bundled from an untrusted source: git bundle verify only checks completeness, not authenticity — use GPG-signed tags alongside the bundle.
  • Shallow clones reduce attack surface — fewer old commits mean fewer potential references to leaked credentials in commit messages.
  • Partial clones do not reduce risk from malicious blobs that may exist in history; use git filter-repo to scrub history before creating bundles.

Cleanup ​

bash
# Remove bundle files after successful transfer / CI run
rm -f repo.bundle incremental.bundle

# Convert shallow clone to full (after confirming no more incremental fetches needed)
git fetch --unshallow

# Convert partial clone back to full (download all remaining blobs)
git fetch --filter=blob:none --refetch origin
# (refetch downloads all blobs not already present)

Git Master — Git Reference for Every Skill Level