Appearance
Git clone optimization for CI/CD
Every CI pipeline starts with a clone. The default git clone <url> pulls the entire history and every object. For a repo with 5,000 commits and 2 GB of history, that is 20-60 seconds and 2 GB of disk per runner, per build.
Git gives you nine knobs to tune this. Most CI pipelines use none of them.
This guide covers every clone strategy, the performance of each, and how to pick the right combination for your pipeline. The goal is one number: the minimum data you need to complete the job.
The clone matrix
| Strategy | Flag | Data transferred | Disk after clone | Supports write |
|---|---|---|---|---|
| Full | (default) | All objects, all history | Repository full size | Yes |
| Shallow | --depth 1 | Single commit's objects | ~1-5 MB | No (push needs depth) |
| Shallow since | --shallow-since=<date> | Commits since date | Proportional | Partial |
| Single branch | --single-branch | One branch's objects | Smaller than full | Yes |
| Blobless partial | --filter=blob:none | All commits+trees, no blobs | ~30-50% of full | Yes (lazy fetch) |
| Treeless partial | --filter=tree:0 | Commits only, no trees or blobs | ~5-10% of full | Yes (lazy fetch) |
| Reference clone | --reference-if-able=<path> | Only new objects | Near zero (borrowed) | Yes (dissociate) |
| Bundle clone | git clone repo.bundle | One file | Same as source | Depends |
| Combination | See below | Tunable to job | Tunable to job | Yes |
Shallow clones
A shallow clone truncates history at a given depth. git clone --depth 1 fetches only the latest commit on the default branch with all its files. This is the fastest possible clone when you only need to build or test the current state.
bash
# Fastest possible clone for a build-only pipeline
git clone --depth 1 --single-branch https://git.example.com/org/repo.git
# Keep the last N commits for context (git log, blame, etc.)
git clone --depth 10 https://git.example.com/org/repo.git
# All commits since a date
git clone --shallow-since="2024-01-01" https://git.example.com/org/repo.git
# Exclude commits older than a ref
git clone --shallow-exclude=v1.0.0 https://git.example.com/org/repo.gitWhat breaks with shallow clones
git merge-basecan fail because the common ancestor may be outside the shallow boundary.git fetch --unshalloworgit fetch --deepen=Nfixes this.git describe --tagscan't find tags behind the boundary.git bisectneeds the full range between good and bad commits.- Push from a shallow clone fails unless you
git fetch --unshallowfirst or push to a matching ref.
Shallow-since for CI
--shallow-since is a practical middle ground for PR pipelines. If your PRs merge within 7 days, --shallow-since="2 weeks ago" keeps enough history for merge-base and changelog generation without fetching the full 5 years of commits.
bash
# Good for PR CI: enough history for merge-base + context
git clone --shallow-since="2 weeks ago" --single-branch <url>Partial clones
Partial clones defer object fetching. Instead of downloading every blob at clone time, Git fetches objects on demand when a command needs them. This requires protocol v2, which Git servers have supported since Git 2.19 (2018).
Blobless partial clone
--filter=blob:none fetches all commits and trees but no file contents. When you run git checkout or git diff, Git fetches the blobs it needs from the remote. For monorepo CI where a build touches 5% of files, this saves 90%+ of clone time.
bash
# Clone all history, fetch blobs on demand
git clone --filter=blob:none https://git.example.com/org/repo.git
# Combine with depth for further savings
git clone --filter=blob:none --depth=10 https://git.example.com/org/repo.gitBest for: monorepo CI, large binary assets, repos where most runs touch a small subset of files.
Treeless partial clone
--filter=tree:0 fetches only commits. Trees and blobs are fetched on demand. This is the most aggressive standard filter. Useful when you only need commit metadata (counts, messages, author info) for gate checks.
bash
# Commit metadata only, no directory structure or files
git clone --filter=tree:0 --depth=1 https://git.example.com/org/repo.git
# Check commit count without downloading any file content
git -C repo rev-list --count HEADCombining partial with sparse checkout
Partial clones and sparse checkouts are complementary, not redundant. Partial defers object fetches; sparse limits which paths appear in the working tree. Together they give maximum CI speed.
bash
git clone --filter=blob:none --sparse https://git.example.com/org/repo.git
cd repo
git sparse-checkout set services/api/ deploy/With this setup, Git fetches full commit and tree history (for merge-base and changelog) but only downloads blobs for services/api/ and deploy/. A CI pipeline that only builds one service downloads nothing outside that service.
Promisor remote caveats
Partial clones add a promisor remote. Every git log, git diff, git blame that touches an unfetched object triggers a fetch. This can cause unexpected delays in CI. The fix is to materialize needed objects upfront with git rev-list --objects --all | git cat-file --batch-check='%(objecttype) %(objectname)' or by running git checkout which fetches what the working tree needs.
bash
# Materialize all blobs for the current checkout
git checkout HEAD -- .
# Check what objects are still missing
git rev-list --missing=print --allReference clones
Reference clones share objects from an existing local repository. No objects are transferred from the remote that already exist in the reference repo.
bash
# Create a reference mirror (one-time)
git clone --mirror https://git.example.com/org/repo.git /var/cache/git/repo.git
# Clone using the reference (CI runner)
git clone --reference-if-able=/var/cache/git/repo.git https://git.example.com/org/repo.gitThe reference repo must be updated periodically:
bash
git -C /var/cache/git/repo.git fetch --prune --no-prune-tagsReference clone lifecycle in CI
For self-hosted runners with persistent volumes, reference clones reduce clone time to near zero:
bash
# init-cache.sh - run daily or on runner startup
if [ ! -d /var/cache/git/repo.git ]; then
git clone --mirror https://git.example.com/org/repo.git /var/cache/git/repo.git
else
git -C /var/cache/git/repo.git fetch --prune --no-prune-tags
fi
# ci-checkout.sh - per-job checkout
git clone --reference-if-able=/var/cache/git/repo.git \
--dissociate \ # decouple from cache to avoid corruption
--single-branch \
https://git.example.com/org/repo.git--dissociate copies borrowed objects into the new repo's object store so the clone is self-contained after creation. Without it, a corrupted or pruned reference repo breaks the clone.
Bundle clones
git bundle creates a single-file transport for air-gapped networks or pre-seeded CI runners.
bash
# Create a bundle of the default branch (on a connected machine)
git bundle create repo.bundle HEAD main
# Clone from the bundle
git clone repo.bundle repo
# Add the real remote for pushes
cd repo
git remote add origin https://git.example.com/org/repo.git
git fetch originBundles can be shallow, partial, or any combination:
bash
# Shallow bundle for CI
git bundle create ci.bundle --all --since="2 weeks ago"
# Clone from bundle then fetch remaining
git clone ci.bundle repo
cd repo
git remote add origin https://git.example.com/org/repo.git
git fetch --shallow-exclude="2 weeks ago" originThe CI decision tree
Here is how to choose a clone strategy based on what your pipeline does:
PR checks (lint, test, typecheck)
Your pipeline needs the PR branch's files, maybe the base branch for merge-base diff.
bash
# Best: shallow blobless partial, sparse checkout
git clone --filter=blob:none --depth=10 --single-branch <url>
cd repo
git sparse-checkout set src/ tests/The --depth=10 gives enough commits for git merge-base HEAD origin/main. --filter=blob:none defers blob fetches. The sparse checkout limits what hits the working tree.
Full build and deploy
Your pipeline compiles the whole project. It needs all files but not history.
bash
# Best: depth 1 with single branch
git clone --depth=1 --single-branch <url>If the build needs tags for versioning, use --depth=100 or --shallow-since instead of --depth=1.
Release pipeline
Your pipeline needs tags, full changelog, signed commit verification.
bash
# Best: full clone with single branch
git clone --single-branch <url>Or use a reference clone on a persistent runner to avoid re-downloading history every time.
Monorepo service build
Your monorepo has 20 services. Your pipeline builds one.
bash
# Best: blobless partial + sparse cone
git clone --filter=blob:none --single-branch <url>
cd repo
git sparse-checkout set services/my-service/Only the service's blobs are fetched. Trees for the rest of the repo are still fetched (they are small), but their file contents are not.
Ephemeral runner (GitHub Actions, GitLab SaaS)
No persistent cache. Every clone starts from zero.
bash
# Best: shallow + single-branch for build-only
git clone --depth=1 --single-branch <url>
# Best: blobless partial for PR pipelines
git clone --filter=blob:none --depth=10 --single-branch <url>Air-gapped network
No outbound internet from the runner.
bash
# Pre-stage a bundle on the runner
git clone ci-bundle.bundle repo
cd repo
git remote add origin https://internal-git.example.com/org/repo.gitPerformance comparison
Numbers for a 5,000-commit, 1.2 GB monorepo (Kubernetes-style, many YAML + Go files):
| Strategy | Clone time | Disk | Notes |
|---|---|---|---|
| Full clone | 42 s | 1.2 GB | Baseline |
--depth 1 | 3 s | 28 MB | No history |
--depth 10 | 5 s | 41 MB | Enough for PR merge-base |
--shallow-since="1 month" | 12 s | 180 MB | History context |
--filter=blob:none | 18 s | 520 MB | Full history, no blobs |
--filter=tree:0 | 8 s | 120 MB | Commits only |
--depth 10 --filter=blob:none | 4 s | 35 MB | Shallow + partial |
| Reference clone | 0.8 s | 0 MB (borrowed) | Needs cache host |
| Bundle clone | 2 s | 28 MB | Needs bundle generation |
Numbers are approximate and depend on network speed, server load, and object packing.
Common pitfalls
Server doesn't support protocol v2
Partial clones (--filter=...) require protocol v2. If the server is on Git < 2.19 or has protocol.version = 0, the filter is silently ignored and you get a full clone. Verify with:
bash
GIT_TRACE_PACKET=1 git clone --filter=blob:none <url> 2>&1 | grep "version 2"If you don't see version 2, the server isn't negotiating v2. Check git config protocol.version on the server or set GIT_PROTOCOL=version=2 in the client environment.
Shallow clone can't find merge-base
git merge-base HEAD origin/main returns nothing if the common ancestor is deeper than your shallow boundary. Fix by deepening the clone:
bash
git fetch --deepen=100 origin main
# or unshallow completely
git fetch --unshallowFor PR pipelines, --depth=10 is usually enough because PR branches diverge from the target branch recently. But for stale branches or long-lived feature branches, you need more depth.
Promisor object not found
A partial clone that fetches objects on demand may fail if the remote is unavailable during the pipeline. Materialize needed objects upfront:
bash
# Fetch all blobs for the current checkout
git checkout HEAD -- .
git diff HEAD HEAD~1 # triggers blob fetch for changed filesOr run a full materialization step early in the pipeline.
Reference repo corruption
If the reference mirror's objects are pruned or corrupted, --reference-if-able silently falls back to a normal fetch. --reference (without -if-able) fails hard. Always use --reference-if-able in CI. Pair with --dissociate so the clone doesn't depend on the reference after creation.
Clone with LFS
Git LFS objects are not affected by --filter or --depth. They are fetched separately by the LFS smudge filter. To skip LFS in CI:
bash
GIT_LFS_SKIP_SMUDGE=1 git clone <url>
git lfs pull --include "deploy/*.bin"Shallow clone push failure
Pushing from a shallow clone fails because the remote can't resolve the parent chain. Fix by deepening or unshallowing before push:
bash
git fetch --unshallow origin
git push origin HEAD:mainCI configuration examples
GitHub Actions
yaml
# Full fetch is the default. Use this to override:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # full history (needed for release pipelines)
# PR check with shallow fetch:
- uses: actions/checkout@v4
with:
fetch-depth: 10 # enough for merge-baseFor blobless partial in GitHub Actions, use a manual clone:
yaml
- run: |
git init repo
cd repo
git remote add origin https://github.com/${{ github.repository }}
git fetch --depth=10 --filter=blob:none origin ${{ github.ref }}
git checkout FETCH_HEADForgejo / Gitea runner
yaml
- run: |
git clone --depth=1 --single-branch \
https://git.example.com/${{ gitea.repository }}.git .GitLab CI
yaml
variables:
GIT_DEPTH: 10
GIT_STRATEGY: cloneSelf-hosted runner with reference cache
yaml
# Setup step (runs once per runner lifetime)
setup-cache:
stage: setup
script: |
if [ ! -d /cache/repo.git ]; then
git clone --mirror $CI_REPOSITORY_URL /cache/repo.git
else
git -C /cache/repo.git fetch --prune
fi
# Build step (runs per job)
build:
stage: build
script: |
git clone --reference-if-able=/cache/repo.git \
--dissociate --single-branch $CI_REPOSITORY_URL .Quick reference
See docs/recipes/clone-quick.md for a one-page cheat sheet of all clone commands and CI one-liners.
See scripts/clone-manage.sh for an automation script that generates the right clone command for your pipeline scenario.