Skip to content

Git fast-import/fast-export: Bulk Repo Migration and Stream Rewriting ​

Export and reimport repository history as a text stream — rewrite authors, split or merge repos, anonymize sensitive data, and generate synthetic test repos at speeds that make filter-branch weep.

When to reach for fast-import ​

  • Repo consolidation. Merge 5 repos into a monorepo with directory prefixes and no common ancestor.
  • Repo splitting. Carve one directory out of a monorepo into its own repo, preserving annotated tags and per-file history.
  • History anonymization. Strip names, emails, and file paths from a repo so you can share a reproduction case without leaking customer data.
  • Synthetic test repos. Generate a repo with a specific commit topology — parallel branches, merge storms, cherry-pick chains — to benchmark bisect, blame, or range-diff.
  • Big repo surgery. Pipe the export stream through awk, sed, or a Python script to rewrite author lines, remove blobs, or re-root paths, then reimport.
  • Incremental migration. Export a range of commits from a live source repo and import them into a target, run by run, with mark-based tracking so nothing is replayed.

How the stream works ​

git fast-export writes a text stream to stdout. git fast-import reads that stream from stdin and builds Git objects. The stream is line-oriented with a handful of command types.

blob
mark :1
data 42
<raw blob content, 42 bytes>

commit refs/heads/main
mark :2
author Alice <alice@example.com> 1700000000 +0000
committer Alice <alice@example.com> 1700000000 +0000
data 18
Initial commit
from :1
M 100644 :1 README.md

The mark system (:1, :2, ...) lets you reference objects from earlier in the stream. Reuse a blob mark from a different commit without re-sending the blob.

Stream commands ​

CommandPurpose
blobDeclare a raw file blob (content follows on data lines).
commitCreate a commit. References parent commits via from and merge with mark numbers.
tagCreate an annotated tag. Includes tagger, message, and the tagged mark.
resetMove a reference to a specific mark. Useful for rewriting branch starting points.
progressEmit progress messages to the receiving side.
checkpointForce an fsync and intermediate pack flush. Prevents memory growth on huge imports.
featureDeclare required capabilities: feature date-format=raw, feature done, feature export-marks, feature import-marks.
doneEnd-of-stream marker. Required when feature done is declared.

The data command declares the length of the payload on the next line. Always exact byte count, no terminator.

File modification commands inside a commit ​

Inside a commit block, file changes use M (modify), D (delete), C (copy), R (rename), or deleteall (clear the tree).

M 100644 :3 src/main.py
M 100755 :4 scripts/deploy.sh
D obsolete/file.txt
R old/path.py new/path.py
deleteall

Mode values: 100644 (normal), 100755 (executable), 120000 (symlink), 160000 (gitlink/submodule).

Export basics ​

bash
# Export everything
git fast-export --all > export.stream

# Export last 10 commits
git fast-export HEAD~10..HEAD > export.stream

# Export a single branch
git fast-export --branches main > export.stream

# Export with path filter (only docs/ and src/)
git fast-export --path docs/ --path src/ main > export.stream

# Export with refspec rewriting (rename branch in transit)
git fast-export --refspec refs/heads/oldname:refs/heads/newname

# Export with original commit IDs preserved in the stream
git fast-export --show-original-ids --all > export.stream

Import basics ​

bash
# Import into a fresh repo
mkdir target && cd target && git init
git fast-import < ../export.stream
git reset --hard HEAD

# Import into an existing repo with marks tracking
git fast-import --export-marks=marks.out --import-marks=marks.in < incremental.stream

# Import over SSH (remote target)
ssh user@server "cd /repo && git fast-import --export-marks=.git/marks" < export.stream

DevOps patterns ​

Monorepo consolidation: merge N repos under directory prefixes ​

The problem: five microservice repos need to live in one monorepo without colliding paths. fast-import handles this by rewriting the export stream on the fly.

bash
# For each source repo
git -C repo-A fast-export --all --path src/ |
  sed 's#^M 100644 \([0-9:]\+\) #M 100644 \1 services/A/' > /tmp/service-A.stream

git -C repo-B fast-export --all --path src/ |
  sed 's#^M 100644 \([0-9:]\+\) #M 100644 \1 services/B/' > /tmp/service-B.stream

Then import into the monorepo target, writing marks files so each stream's mark namespace stays separate:

bash
cd monorepo
git fast-import --import-marks=.git/marks-a --export-marks=.git/marks-a-new < /tmp/service-A.stream
mv .git/marks-a-new .git/marks-a

git fast-import --import-marks=.git/marks-b --export-marks=.git/marks-b-new < /tmp/service-B.stream
mv .git/marks-b-new .git/marks-b

git reset --hard HEAD

Each import uses its own marks file, so mark :1 in service-A is a different blob than mark :1 in service-B.

A more robust approach uses --prefix on export:

bash
git fast-export --all --prefix=services/A/ > service-A.stream

The --prefix flag prepends the given string to every file path in the export stream, which is exactly what you need for repo consolidation. This avoids fragile sed patterns on mode lines.

Repo splitting: extract one directory into its own repo ​

bash
git fast-export --path=services/A/ --refspec refs/heads/main:refs/heads/main --all > service-A.stream

mkdir service-a-only && cd service-a-only && git init
git fast-import < ../service-A.stream
git reset --hard HEAD

The --path filter discards everything outside services/A/. The refspec maps the branches as-is. The resulting repo has only the history that touched that directory.

History anonymization for debugging reproduction ​

When sharing a repo to reproduce a bug — without leaking employee names, email domains, or internal file names:

bash
git fast-export --anonymize --all > anonymized.stream

But --anonymize is all-or-nothing. For partial anonymization (keep file names, strip authors):

bash
git fast-export --all |
  sed -E '
    s/^author .* <[^>]+>/author Developer <dev@example.com>/
    s/^committer .* <[^>]+>/committer Developer <dev@example.com>/
    /^data [0-9]+/{
      :loop
      N
      /\n$/!bloop
      s/\n[^\n]*$/\nAnonymized commit message/
    }
  ' > anonymized.stream

This preserves the file tree and commit topology while replacing authorship. The data block requires careful length recomputation, so for production use a dedicated tool like git filter-repo or a Python stream processor that counts bytes.

Synthetic test repo generation ​

Create a repo with a specific commit graph for benchmarking:

bash
# Generate a stream that creates 10 linear commits with 1 file each
for i in $(seq 1 10); do
  echo "blob"
  echo "mark :$((i * 2 - 1))"
  echo "data 5"
  echo "v$i"
  echo ""
  echo "commit refs/heads/main"
  echo "mark :$((i * 2))"
  echo "author Test <test@example.com> $((1700000000 + i * 100)) +0000"
  echo "committer Test <test@example.com> $((1700000000 + i * 100)) +0000"
  echo "data 10"
  echo "commit v$i"
  if [ "$i" -eq 1 ]; then
    echo "feature date-format=raw"
  else
    echo "from :$((i * 2 - 2))"
  fi
  echo "M 100644 :$((i * 2 - 1)) file-$i.txt"
  echo ""
done
echo "done" | git fast-import

For more complex topologies — branches, merges, conflicts — generate the stream with a script. This is how you create test repos for CI performance benchmarking without waiting for real history to accumulate.

Incremental migration with marks ​

Migrate a repo in chunks while the source stays live:

bash
# Phase 1: initial full export
git -C source fast-export --export-marks=.git/marks --all > phase1.stream
cd target && git fast-import --export-marks=.git/marks < ../phase1.stream
git reset --hard HEAD

# Phase 2: next week, only new commits
git -C source fast-export --export-marks=.git/marks-new \
  --import-marks=.git/marks --all > phase2.stream
cd target && git fast-import \
  --export-marks=.git/marks --import-marks=.git/marks < ../phase2.stream
git reset --hard HEAD

The marks file tracks which objects were already exported. On the second run, --import-marks tells fast-export to skip objects already sent. This produces a compact incremental stream.

Stream manipulation with external scripts ​

The fastest way to rewrite massive history is to pipe the export stream through a script:

bash
# Remove all blobs larger than 10MB from the export stream
git fast-export --all |
  python3 -c '
import sys
lines = sys.stdin.readlines()
i = 0
skip = False
while i < len(lines):
    if lines[i].startswith("blob") and not skip:
        i += 1
        assert lines[i].startswith("mark"), lines[i]
        i += 1
        assert lines[i].startswith("data"), lines[i]
        size = int(lines[i].split()[1])
        if size > 10_000_000:
            skip = True
            i += 1
            # skip content
            while size > 0:
                consumed = len(lines[i])
                size -= consumed
                i += 1
            skip = False
            continue
    print(lines[i], end="")
    i += 1
' > filtered.stream

This avoids unpacking and repacking objects — it operates on the stream text, which is orders of magnitude faster than filter-branch.

Performance characteristics ​

fast-import is a streaming, incremental pack-builder. It never decompresses objects to the filesystem, never writes loose objects by default, and never walks ancestry more than once.

OperationSpeed (100k commits, 500 blobs)
git fast-export --all~2-5 seconds
git fast-import < stream~1-3 seconds
git filter-branch (equivalent)~5-15 minutes
git filter-repo (equivalent)~3-10 seconds

The stream format is compact: 100k commits with a few files each produce about 50-200 MB of stream text. fast-import compresses the resulting pack on the fly, so the final .git size is usually smaller than the original.

Limitations and gotchas ​

  • No merge commit metadata rewriting. fast-export preserves parent relationships faithfully, but if you need to drop a merge commit or change its parents, you must edit the stream manually. filter-repo handles this more cleanly.
  • Tag-of-filtered-object. If a tag points to a commit that was filtered out by --path, the tag becomes dangling. Use --tag-of-filtered-object=rewrite or --tag-of-filtered-object=drop to control the behavior.
  • Mark file collisions. When consolidating repos, each import stream uses independent mark numbers. Use separate marks files per stream (as shown in the monorepo consolidation pattern) or the marks will collide.
  • No signed-object preservation. fast-export strips GPG signatures on commits and tags. Re-sign after import.
  • Remote protocol limitations. fast-import over SSH works but TCP streaming has no built-in retry. For large migrations, export to file, checksum it, then pipe the file into the target.
  • Memory on large blobs. fast-import holds the entire stream in memory during a single checkpoint window. For repos with multi-GB blobs, set --max-pack-size=2g and insert checkpoint commands periodically to flush packfiles to disk.

Comparison: fast-import vs alternatives ​

Needfast-importfilter-repofilter-branchmanual git
SpeedFastestFastSlowN/A
Author rewritingVia stream scriptBuilt-inBuilt-inManual
Path filtering--path flagBuilt-inSupport removedManual
Repo consolidationExcellentGoodVery slowError-prone
Repo splittingExcellentGoodSlowManual
AnonymizationBuilt-in --anonymizeBuilt-in email/nameCustom scriptManual
Synthetic reposBest (stream gen)Not designed forNot designed forVery slow
Incremental migrationMarks-basedNot designed forNot designed forManual
GPG signature preservationLostLostPreservedManual re-sign
RFC 2822 timestampsHandledHandledHandledManual

Recipe: streaming a repo across hosts without intermediate files ​

bash
ssh build-server "cd /var/build/repo && git fast-export --all --anonymize" |
  cd /tmp/debug && git init && git fast-import && git reset --hard HEAD

This clones and anonymizes a repo across hosts in one pipe — no intermediate file, no credentials on the debug machine (the SSH session authenticates).

See also ​

  • docs/recipes/fast-import-quick.md — one-page cheat sheet
  • scripts/fast-import-manage.sh — automation utility
  • docs/devops-workflows/git-filter-repo.md — alternative for simpler history rewrites
  • docs/devops-workflows/git-plumbing-ci-automation.md — low-level object manipulation
  • docs/devops-workflows/git-reference-clones-alternates.md — accelerating clone operations

Git Master — Git Reference for Every Skill Level