Appearance
Git fast-import/fast-export: Bulk Repo Migration and Stream Rewriting
Export and reimport repository history as a text stream — rewrite authors, split or merge repos, anonymize sensitive data, and generate synthetic test repos at speeds that make filter-branch weep.
When to reach for fast-import
- Repo consolidation. Merge 5 repos into a monorepo with directory prefixes and no common ancestor.
- Repo splitting. Carve one directory out of a monorepo into its own repo, preserving annotated tags and per-file history.
- History anonymization. Strip names, emails, and file paths from a repo so you can share a reproduction case without leaking customer data.
- Synthetic test repos. Generate a repo with a specific commit topology — parallel branches, merge storms, cherry-pick chains — to benchmark bisect, blame, or range-diff.
- Big repo surgery. Pipe the export stream through awk, sed, or a Python script to rewrite author lines, remove blobs, or re-root paths, then reimport.
- Incremental migration. Export a range of commits from a live source repo and import them into a target, run by run, with mark-based tracking so nothing is replayed.
How the stream works
git fast-export writes a text stream to stdout. git fast-import reads that stream from stdin and builds Git objects. The stream is line-oriented with a handful of command types.
blob
mark :1
data 42
<raw blob content, 42 bytes>
commit refs/heads/main
mark :2
author Alice <alice@example.com> 1700000000 +0000
committer Alice <alice@example.com> 1700000000 +0000
data 18
Initial commit
from :1
M 100644 :1 README.mdThe mark system (:1, :2, ...) lets you reference objects from earlier in the stream. Reuse a blob mark from a different commit without re-sending the blob.
Stream commands
| Command | Purpose |
|---|---|
blob | Declare a raw file blob (content follows on data lines). |
commit | Create a commit. References parent commits via from and merge with mark numbers. |
tag | Create an annotated tag. Includes tagger, message, and the tagged mark. |
reset | Move a reference to a specific mark. Useful for rewriting branch starting points. |
progress | Emit progress messages to the receiving side. |
checkpoint | Force an fsync and intermediate pack flush. Prevents memory growth on huge imports. |
feature | Declare required capabilities: feature date-format=raw, feature done, feature export-marks, feature import-marks. |
done | End-of-stream marker. Required when feature done is declared. |
The data command declares the length of the payload on the next line. Always exact byte count, no terminator.
File modification commands inside a commit
Inside a commit block, file changes use M (modify), D (delete), C (copy), R (rename), or deleteall (clear the tree).
M 100644 :3 src/main.py
M 100755 :4 scripts/deploy.sh
D obsolete/file.txt
R old/path.py new/path.py
deleteallMode values: 100644 (normal), 100755 (executable), 120000 (symlink), 160000 (gitlink/submodule).
Export basics
bash
# Export everything
git fast-export --all > export.stream
# Export last 10 commits
git fast-export HEAD~10..HEAD > export.stream
# Export a single branch
git fast-export --branches main > export.stream
# Export with path filter (only docs/ and src/)
git fast-export --path docs/ --path src/ main > export.stream
# Export with refspec rewriting (rename branch in transit)
git fast-export --refspec refs/heads/oldname:refs/heads/newname
# Export with original commit IDs preserved in the stream
git fast-export --show-original-ids --all > export.streamImport basics
bash
# Import into a fresh repo
mkdir target && cd target && git init
git fast-import < ../export.stream
git reset --hard HEAD
# Import into an existing repo with marks tracking
git fast-import --export-marks=marks.out --import-marks=marks.in < incremental.stream
# Import over SSH (remote target)
ssh user@server "cd /repo && git fast-import --export-marks=.git/marks" < export.streamDevOps patterns
Monorepo consolidation: merge N repos under directory prefixes
The problem: five microservice repos need to live in one monorepo without colliding paths. fast-import handles this by rewriting the export stream on the fly.
bash
# For each source repo
git -C repo-A fast-export --all --path src/ |
sed 's#^M 100644 \([0-9:]\+\) #M 100644 \1 services/A/' > /tmp/service-A.stream
git -C repo-B fast-export --all --path src/ |
sed 's#^M 100644 \([0-9:]\+\) #M 100644 \1 services/B/' > /tmp/service-B.streamThen import into the monorepo target, writing marks files so each stream's mark namespace stays separate:
bash
cd monorepo
git fast-import --import-marks=.git/marks-a --export-marks=.git/marks-a-new < /tmp/service-A.stream
mv .git/marks-a-new .git/marks-a
git fast-import --import-marks=.git/marks-b --export-marks=.git/marks-b-new < /tmp/service-B.stream
mv .git/marks-b-new .git/marks-b
git reset --hard HEADEach import uses its own marks file, so mark :1 in service-A is a different blob than mark :1 in service-B.
A more robust approach uses --prefix on export:
bash
git fast-export --all --prefix=services/A/ > service-A.streamThe --prefix flag prepends the given string to every file path in the export stream, which is exactly what you need for repo consolidation. This avoids fragile sed patterns on mode lines.
Repo splitting: extract one directory into its own repo
bash
git fast-export --path=services/A/ --refspec refs/heads/main:refs/heads/main --all > service-A.stream
mkdir service-a-only && cd service-a-only && git init
git fast-import < ../service-A.stream
git reset --hard HEADThe --path filter discards everything outside services/A/. The refspec maps the branches as-is. The resulting repo has only the history that touched that directory.
History anonymization for debugging reproduction
When sharing a repo to reproduce a bug — without leaking employee names, email domains, or internal file names:
bash
git fast-export --anonymize --all > anonymized.streamBut --anonymize is all-or-nothing. For partial anonymization (keep file names, strip authors):
bash
git fast-export --all |
sed -E '
s/^author .* <[^>]+>/author Developer <dev@example.com>/
s/^committer .* <[^>]+>/committer Developer <dev@example.com>/
/^data [0-9]+/{
:loop
N
/\n$/!bloop
s/\n[^\n]*$/\nAnonymized commit message/
}
' > anonymized.streamThis preserves the file tree and commit topology while replacing authorship. The data block requires careful length recomputation, so for production use a dedicated tool like git filter-repo or a Python stream processor that counts bytes.
Synthetic test repo generation
Create a repo with a specific commit graph for benchmarking:
bash
# Generate a stream that creates 10 linear commits with 1 file each
for i in $(seq 1 10); do
echo "blob"
echo "mark :$((i * 2 - 1))"
echo "data 5"
echo "v$i"
echo ""
echo "commit refs/heads/main"
echo "mark :$((i * 2))"
echo "author Test <test@example.com> $((1700000000 + i * 100)) +0000"
echo "committer Test <test@example.com> $((1700000000 + i * 100)) +0000"
echo "data 10"
echo "commit v$i"
if [ "$i" -eq 1 ]; then
echo "feature date-format=raw"
else
echo "from :$((i * 2 - 2))"
fi
echo "M 100644 :$((i * 2 - 1)) file-$i.txt"
echo ""
done
echo "done" | git fast-importFor more complex topologies — branches, merges, conflicts — generate the stream with a script. This is how you create test repos for CI performance benchmarking without waiting for real history to accumulate.
Incremental migration with marks
Migrate a repo in chunks while the source stays live:
bash
# Phase 1: initial full export
git -C source fast-export --export-marks=.git/marks --all > phase1.stream
cd target && git fast-import --export-marks=.git/marks < ../phase1.stream
git reset --hard HEAD
# Phase 2: next week, only new commits
git -C source fast-export --export-marks=.git/marks-new \
--import-marks=.git/marks --all > phase2.stream
cd target && git fast-import \
--export-marks=.git/marks --import-marks=.git/marks < ../phase2.stream
git reset --hard HEADThe marks file tracks which objects were already exported. On the second run, --import-marks tells fast-export to skip objects already sent. This produces a compact incremental stream.
Stream manipulation with external scripts
The fastest way to rewrite massive history is to pipe the export stream through a script:
bash
# Remove all blobs larger than 10MB from the export stream
git fast-export --all |
python3 -c '
import sys
lines = sys.stdin.readlines()
i = 0
skip = False
while i < len(lines):
if lines[i].startswith("blob") and not skip:
i += 1
assert lines[i].startswith("mark"), lines[i]
i += 1
assert lines[i].startswith("data"), lines[i]
size = int(lines[i].split()[1])
if size > 10_000_000:
skip = True
i += 1
# skip content
while size > 0:
consumed = len(lines[i])
size -= consumed
i += 1
skip = False
continue
print(lines[i], end="")
i += 1
' > filtered.streamThis avoids unpacking and repacking objects — it operates on the stream text, which is orders of magnitude faster than filter-branch.
Performance characteristics
fast-import is a streaming, incremental pack-builder. It never decompresses objects to the filesystem, never writes loose objects by default, and never walks ancestry more than once.
| Operation | Speed (100k commits, 500 blobs) |
|---|---|
git fast-export --all | ~2-5 seconds |
git fast-import < stream | ~1-3 seconds |
git filter-branch (equivalent) | ~5-15 minutes |
git filter-repo (equivalent) | ~3-10 seconds |
The stream format is compact: 100k commits with a few files each produce about 50-200 MB of stream text. fast-import compresses the resulting pack on the fly, so the final .git size is usually smaller than the original.
Limitations and gotchas
- No merge commit metadata rewriting. fast-export preserves parent relationships faithfully, but if you need to drop a merge commit or change its parents, you must edit the stream manually. filter-repo handles this more cleanly.
- Tag-of-filtered-object. If a tag points to a commit that was filtered out by
--path, the tag becomes dangling. Use--tag-of-filtered-object=rewriteor--tag-of-filtered-object=dropto control the behavior. - Mark file collisions. When consolidating repos, each import stream uses independent mark numbers. Use separate marks files per stream (as shown in the monorepo consolidation pattern) or the marks will collide.
- No signed-object preservation. fast-export strips GPG signatures on commits and tags. Re-sign after import.
- Remote protocol limitations. fast-import over SSH works but TCP streaming has no built-in retry. For large migrations, export to file, checksum it, then pipe the file into the target.
- Memory on large blobs. fast-import holds the entire stream in memory during a single checkpoint window. For repos with multi-GB blobs, set
--max-pack-size=2gand insertcheckpointcommands periodically to flush packfiles to disk.
Comparison: fast-import vs alternatives
| Need | fast-import | filter-repo | filter-branch | manual git |
|---|---|---|---|---|
| Speed | Fastest | Fast | Slow | N/A |
| Author rewriting | Via stream script | Built-in | Built-in | Manual |
| Path filtering | --path flag | Built-in | Support removed | Manual |
| Repo consolidation | Excellent | Good | Very slow | Error-prone |
| Repo splitting | Excellent | Good | Slow | Manual |
| Anonymization | Built-in --anonymize | Built-in email/name | Custom script | Manual |
| Synthetic repos | Best (stream gen) | Not designed for | Not designed for | Very slow |
| Incremental migration | Marks-based | Not designed for | Not designed for | Manual |
| GPG signature preservation | Lost | Lost | Preserved | Manual re-sign |
| RFC 2822 timestamps | Handled | Handled | Handled | Manual |
Recipe: streaming a repo across hosts without intermediate files
bash
ssh build-server "cd /var/build/repo && git fast-export --all --anonymize" |
cd /tmp/debug && git init && git fast-import && git reset --hard HEADThis clones and anonymizes a repo across hosts in one pipe — no intermediate file, no credentials on the debug machine (the SSH session authenticates).
See also
docs/recipes/fast-import-quick.md— one-page cheat sheetscripts/fast-import-manage.sh— automation utilitydocs/devops-workflows/git-filter-repo.md— alternative for simpler history rewritesdocs/devops-workflows/git-plumbing-ci-automation.md— low-level object manipulationdocs/devops-workflows/git-reference-clones-alternates.md— accelerating clone operations