Skip to content

Git Sparse Index — Performance at Monorepo Scale ​

The sparse index shrinks the Git index to only the entries inside your sparse-checkout cone. For monorepo CI runners and developer workstations with hundreds of thousands of files, this turns git status from a multi-second scan into a sub-millisecond lookup.


1. The Index Problem in Large Repos ​

A standard Git index (git ls-files --stage | wc -l) lists every tracked file in the repo. In a monorepo with 300k files across 50 services, the index is 300k entries. Every git status, git add, git commit, and git diff reads and parses the entire thing.

Even with a cone-mode sparse checkout that only populates the working tree with one service's files, the index still tracks all 300k entries. The sparse index solves this by only tracking entries inside the cone.

2. What the Sparse Index Is ​

The sparse index is a Git index (v4 format) that replaces entire directory subtrees outside the sparse cone with a single sparse directory entry. Instead of 200k individual file entries under services/analytics/, the index stores one entry:

040000 tree <sha> 0 services/analytics/

This directory entry (mode 040000 instead of the usual file mode 100644) tells Git "everything under here exists but is not expanded." Commands that only need to know about files inside the cone never expand it.

Enabled by Default in Cone Mode ​

Since Git 2.32, git sparse-checkout init --cone automatically enables the sparse index. In Git 2.37+, the sparse-index feature is the default for cone-mode checkouts.

# Verify sparse index is active
git config index.sparse
# -> true

Manual Control ​

git sparse-checkout init --cone --sparse-index   # enable (default since 2.37)
git sparse-checkout sparse-index disable         # revert to full index
git sparse-checkout sparse-index enable          # re-enable

The git sparse-checkout sparse-index subcommand (Git 2.37+) toggles the sparse index on an existing sparse checkout without re-initializing.

3. Performance Characteristics ​

The sparse index gives the biggest wins on index-heavy operations in large repos.

OperationFull Index (300k files)Sparse Index (500 files)Speedup
git status2.8 s0.02 s140x
git add .3.1 s0.04 s77x
git commit3.5 s0.06 s58x
git diff2.2 s0.01 s220x
git log --oneline0.4 s0.4 s1x (no index read)
git checkout (switch branch)4.0 s0.5 s8x

These numbers are from real monorepo benchmarks (Azure DevOps / Windows repo at ~300k tracked files). Your speedup depends on the ratio of total tracked files to cone-matched files.

Where the Savings Come From ​

  1. Index read. Loading 300k entries from disk and parsing them is a linear scan over megabytes of data. A sparse index with 500 entries is two orders of magnitude smaller.
  2. Index write. git add rewrites the index. A smaller index writes faster.
  3. Index comparison. git status diffs the index against HEAD and the working tree. A sparse index limits the diff to cone entries.
  4. Working tree scan. git status skips directories outside the cone entirely — it doesn't even stat() them.

4. Sparse Directory Entries ​

When the sparse index is active, git ls-files shows sparse directory entries:

git ls-files --stage | head -5
100644 <sha> 0 src/main.ts
100644 <sha> 0 src/utils.ts
040000 <sha> 0 services/analytics/    # ← sparse directory, not expanded
040000 <sha> 0 services/payments/     # ← sparse directory, not expanded

To see the full expanded index (including entries inside sparse directories):

git ls-files --sparse --stage

The --sparse flag tells Git to expand sparse directories in its output. Without it, git ls-files omits entries inside sparse directories.

Checking Sparse Index State ​

# Is the sparse index active?
git config index.sparse

# How many entries in the index?
git ls-files --stage | wc -l

# How many sparse directory entries?
git ls-files --stage | grep '^040000' | wc -l

# What directories are collapsed?
git ls-files --stage | grep '^040000'

5. Commands That Expand the Sparse Index ​

Some Git commands need to see the full index to work correctly. When you run them with a sparse index, Git automatically expands it to a full index, performs the operation, and (in recent versions) re-collapses it.

Commands That Auto-Expand ​

CommandWhat HappensGit Version with Re-collapse
git mergeExpands full index for three-way merge2.34+ re-collapses
git rebaseExpands during rebase operations2.35+ re-collapses
git cherry-pickExpands for patch application2.34+ re-collapses
git revertExpands during revert2.34+ re-collapses
git amExpands for patch series2.35+ re-collapses
git stashExpands for stash operations2.36+ re-collapses
git bisectExpands during checkout2.36+ re-collapses
git read-tree -mExpands for tree merge2.35+ re-collapses

Expansion and re-collapse is transparent — you don't need to do anything. But it has a cost: each expansion/re-collapse cycle reads and writes the full index. Running git merge on a sparse index is still faster than running it on a permanently full index (because the index was smaller before the command and gets re-collapsed after), but the first merge after a clone will be slower than subsequent merges because the re-collapse hasn't happened yet.

Detecting Expansion ​

# Before: sparse
git config index.sparse   # true

# Run a command that expands
git merge --no-ff feature-branch

# After: still sparse (Git re-collapsed)
git config index.sparse   # true

If you want to verify re-collapse happened:

git ls-files --stage | grep '^040000' | wc -l
# Should be > 0 — sparse directories are present

6. CI/CD Patterns ​

Pattern 1: Fast CI Checkout with Sparse Index ​

# Git 2.32+ — sparse index is the default
git clone --filter=blob:none --sparse <url>
git sparse-checkout set --cone services/my-service/

# Verify sparse index is active
if [ "$(git config index.sparse)" != "true" ]; then
  echo "warning: sparse index not enabled, falling back to full index"
fi

# Subsequent operations are fast
git status            # milliseconds, not seconds
git diff --name-only  # only checks cone files

Pattern 2: Benchmarking Sparse vs Full Index ​

Run this in a CI job to measure the difference:

#!/bin/bash
set -e

REPO_URL="$1"
SPARSE_PATH="$2"

# Full index timing
rm -rf full-clone
git clone --filter=blob:none "$REPO_URL" full-clone
cd full-clone
git sparse-checkout set --cone --no-sparse-index "$SPARSE_PATH"
time git status > /dev/null

# Sparse index timing
rm -rf sparse-clone
git clone --filter=blob:none "$REPO_URL" sparse-clone
cd sparse-clone
git sparse-checkout set --cone --sparse-index "$SPARSE_PATH"
time git status > /dev/null

Pattern 3: CI Jobs That Need Full Index ​

When a CI job needs git merge or git cherry-pick (e.g., a backport pipeline), the sparse index auto-expands. If the job does many operations that expand the index, consider running the merge step first (when the index is still full) to avoid repeated expansion/collapse:

# Bad: multiple expansion cycles
git checkout target-branch
git merge feature-a    # expands, re-collapses
git merge feature-b    # expands, re-collapses
git merge feature-c    # expands, re-collapses

# Better: batch merges before the index re-collapses
git checkout target-branch
git config index.sparse false           # disable sparse index for this batch
git merge feature-a feature-b feature-c # single merge
git sparse-checkout reapply             # re-enables sparse index

Pattern 4: Post-Clone Index Status ​

After a filtered clone with sparse checkout, check whether the sparse index is working:

git clone --filter=blob:none --sparse <url> repo
cd repo
git sparse-checkout set --cone src/

# Count index entries
total=$(git ls-files --stage | wc -l)
sparse=$(git ls-files --stage | grep '^040000' | wc -l)
echo "Index: $total entries ($sparse sparse directories)"
pct=$(echo "scale=1; 100 * $sparse / $total" | bc)
echo "Sparse directory ratio: $pct%"

if [ "$sparse" -eq 0 ]; then
  echo "WARNING: sparse index is not working — index is fully expanded"
  echo "Check Git version (need 2.32+) and index.sparse config"
fi

Pattern 5: Preventing Accidental Full Index ​

Set a CI lint check that fails if the index becomes fully expanded when it shouldn't be:

assert_sparse_index() {
  local sparse_count
  sparse_count=$(git ls-files --stage | grep '^040000' | wc -l)
  if [ "$sparse_count" -eq 0 ]; then
    echo "FAIL: Sparse index is fully expanded ($(git ls-files --stage | wc -l) entries)"
    echo "Run 'git sparse-checkout reapply' to restore sparse index"
    exit 1
  fi
  echo "OK: $sparse_count sparse directories, $(git ls-files --stage | wc -l) total entries"
}

7. When to Avoid the Sparse Index ​

The sparse index is nearly always beneficial, but there are edge cases:

Git Versions Before 2.32 ​

Older Git versions don't support the sparse index. git config index.sparse returns nothing (or false). Commands fall back to the full index transparently.

Repos With Few Tracked Files ​

If your repo has fewer than ~10,000 tracked files, the sparse index provides minimal benefit. The overhead of expansion/re-collapse during merges can actually make things slower for tiny repos.

Scripts That Parse git ls-files --stage Directly ​

Scripts that parse git ls-files --stage output and expect only regular file entries (mode 100644 or 100755) will break on sparse directory entries (mode 040000). Either pass --sparse to expand the output, or update the script to handle directory entries.

Shared Index Cache Across Repos ​

If you use GIT_INDEX_FILE to share an index across multiple repos (rare), the sparse index format may cause incompatibilities.

FeatureWhat It SolvesIndex Impact
Full indexEverything tracked300k entries
Cone-mode sparse checkoutWorking tree limited to cone300k entries (full index still)
Sparse indexIndex limited to cone~500 entries
Partial clone (blob:none)Objects fetched on demandNo index impact
Sparse index + partial cloneBoth index and objects limited~500 entries + lazy objects

The sparse index and partial clone are complementary — use them together for maximum performance.

9. Troubleshooting ​

Sparse Index Not Active After sparse-checkout init --cone ​

git config index.sparse
# -> true (if active)

If it's false or empty, your Git version may be older than 2.32, or --no-sparse-index was passed during init. Re-enable:

git sparse-checkout sparse-index enable

Commands Running Slowly Despite Sparse Checkout ​

Check if the sparse index is actually active:

git ls-files --stage | grep '^040000' | wc -l
# Should be > 0

If zero, the index is fully expanded. Common causes:

  • A git merge, git stash, or git rebase ran and the re-collapse failed (rare, but happens on Git <2.34).
  • A tool or script ran git read-tree without --no-sparse-index.
  • The git sparse-checkout reapply command was run with an older Git version.

Fix:

git sparse-checkout reapply   # rebuilds the sparse index

git ls-files Shows All Entries, No Sparse Directories ​

This is expected without --stage. The sparse index affects the on-disk index format and --stage output, but git ls-files (without flags) lists all files in the working tree, not the index entries. Use git ls-files --stage | grep '^040000' to see sparse directory entries.

10. Summary ​

ActionCommandGit Version
Enable sparse indexgit sparse-checkout init --cone --sparse-index2.32+
Toggle without re-initgit sparse-checkout sparse-index enable/disable2.37+
Check if activegit config index.sparse2.32+
Count sparse dirsgit ls-files --stage | grep '^040000' | wc -l2.32+
Force re-collapsegit sparse-checkout reapply2.32+
List expandedgit ls-files --sparse --stage2.32+
CI lint checkassert_sparse_index() (Pattern 5 above)—

The sparse index is the single highest-impact performance optimization available for Git monorepo workflows. Enable it. Verify it. Build your CI around it.

Git Master — Git Reference for Every Skill Level