Git Submodule Management — DevOps Reference¶

Git submodules let a repository embed a fixed-pointer snapshot of another repo at a known commit. Unlike subtrees (which merge full history), submodules:

Store only a single commit SHA in the parent repo
Keep submodule history independent and pruned by default
Require an explicit git submodule update to advance the pointer
Work well for pinned dependencies, vendored libraries, or shared tooling

Quick Reference¶

# Add a submodule
git submodule add https://github.com/org/shared-lib.git libs/shared-lib

# Clone a repo WITH submodules
git clone --recurse-submodules https://github.com/org/monorepo.git
git clone                          # then:
git submodule update --init --recursive

# Update a submodule to latest on its default branch
git submodule update --remote libs/shared-lib

# Work inside a submodule (commit in submodule first)
cd libs/shared-lib
git checkout main && git pull
cd ..
git add libs/shared-lib
git commit -m "chore(submodule): advance shared-lib to $(git -C libs/shared-lib rev-parse --short HEAD)"

1. Adding a Submodule¶

# Add and track the default branch (usually main)
git submodule add https://github.com/org/shared-lib.git libs/shared-lib

# Pin to a specific branch
git submodule add -b release/v2 https://github.com/org/shared-lib.git libs/shared-lib

# Add with a specific commit (if already cloned locally)
git submodule add https://github.com/org/shared-lib.git libs/shared-lib
git -C libs/shared-lib checkout <specific-sha>
git add libs/shared-lib
git commit -m "chore(submodule): pin shared-lib at $(git -C libs/shared-lib rev-parse --short HEAD)"

What happens: 1. .gitmodules is created/updated with the submodule URL and path 2. The submodule directory contains a gitlink (index entry with mode 160000) instead of files 3. The first submodule commit is recorded in the parent

2. Cloning and Initialising¶

Scenario	Command
Clone + all submodules	`git clone --recurse-submodules URL`
Clone then init	`git clone URL && git submodule update --init`
Init non-recursive submodules	`git submodule update --init`
Init specific submodule	`git submodule update --init libs/shared-lib`
Init + checkout current branch in submodule	`git submodule update --init --remote`
Recursive (submodules of submodules)	`git submodule update --init --recursive`
Clone with depth (submodules get depth too)	`git clone --recurse-submodules --depth 1 URL`

3. Working in a Submodule¶

Three Modes of Submodule Update¶

--remote        → fetch + merge the tracked branch (like a normal pull)
(no flag)       → checkout the commit recorded in the index (detached HEAD)

# MODE A: Point to the recorded SHA (after parent git pull)
git submodule update

# MODE B: Fetch and fast-forward to the latest on the tracked branch
git submodule update --remote
git submodule update --remote --recursive          # all submodules

# MODE C: Checkout a specific branch inside the submodule (for active dev)
cd libs/shared-lib
git checkout feature/my-fix
# work, commit, push
cd ../..
git add libs/shared-lib
git commit  # records the new submodule SHA

DevOps tip: In CI, prefer --remote only when you explicitly want bleeding-edge. Pin to SHA for reproducible builds. Combine with --remote-branch to control which branch is pulled.

4. CI/CD Integration Patterns¶

4a. Shallow Submodule Clones (CI Speed)¶

# Clone parent shallowly + submodules with depth 1
git clone --depth 50 --recurse-submodules --shallow-submodules URL

# Azure Pipelines example
- bash: |
    git clone --depth 100 --recurse-submodules --shallow-submodules $(repo_url) .
    git submodule update --init --recursive
  displayName: Clone with shallow submodules

4b. Submodule Auth in CI (Private Dependencies)¶

# Option 1: Git credential store (token-based)
git config --global url."https://${GITHUB_TOKEN}@github.com/".insteadOf "https://github.com/"

# Option 2: SSH forwarding
git config --global url."git@github.com:".insteadOf "https://github.com/"

# Option 3: Token in .gitmodules URL (not recommended for public repos)
# Set before clone:
export GITmodules_URL="https://x-access-token:${CI_TOKEN}@github.com/org/repo.git"

4c. Submodule Status Check in CI¶

# Check if submodules are at recorded SHA vs behind
git submodule status

# CI gate: fail if any submodule is dirty or behind remote
if ! git submodule status | grep -q "^ "; then
  echo "ERROR: Submodule at unexpected commit"
  exit 1
fi

4d. Fetching Specific Submodule Branches in CI¶

# Allow submodule to fetch its tracking branch (needed for --remote)
git submodule foreach git fetch origin
git submodule update --remote --init

5. Converting Between Subtrees and Submodules¶

Submodule → Subtree (useful when you need tighter integration)¶

# 1. Ensure the submodule is at the commit you want to start the subtree history from
git -C libs/shared-lib log --oneline -1

# 2. Remove the submodule definition
git submodule deinit -f libs/shared-lib
git rm libs/shared-lib
rm -rf .git/modules/libs/shared-lib

# 3. Add as subtree
git subtree add --prefix=libs/shared-lib \
  https://github.com/org/shared-lib.git main --squash

# 4. Commit
git commit -m "chore: convert shared-lib submodule to subtree"

Subtree → Submodule (useful when you want independent versioning)¶

# 1. Extract the subtree path to a separate bare repo
git subtree split --prefix=libs/shared-lib -b subtree-work

# 2. Clone the subtree work as a new repo (or push to a new remote)
git push git@github.com:org/shared-lib.git subtree-work:main

# 3. Remove the subtree files from the parent
git rm -r libs/shared-lib

# 4. Add as a submodule pointing to the new repo
git submodule add https://github.com/org/shared-lib.git libs/shared-lib

# 5. Commit
git commit -m "chore: convert shared-lib subtree to submodule"

6. Advanced: Nested Submodules¶

# Add a submodule that itself has submodules
git submodule add --recursive https://github.com/org/nested-lib.git ext/nested-lib

# Update all levels
git submodule update --init --recursive

Caveat: Deep nesting makes CI cloning slower and credential management more complex. Consider flattening at 2 levels maximum.

7. Submodule Gitlinks and the Index¶

The submodule pointer in the parent is just a gitlink entry in the index:

# View the gitlink (shows the commit SHA the parent points to)
git ls-files --stage libs/shared-lib
# 160000 abc123... 0    libs/shared-lib

# The 160000 mode = commit gitlink

# See diff between recorded SHA and current HEAD in submodule
git diff libs/shared-lib
# Shows: -old SHA +new SHA

# See what changed inside the submodule (detached HEAD vs branch)
git -C libs/shared-lib log --oneline -3

8. Gitlinks as Artifacts in Pipelines¶

If your pipeline builds a binary and you want reproducible submodule pins:

# Export submodule gitlinks as a JSON artifact
git submodule status --recursive | \
  awk '{print $1, $2}' | \
  jq -Rn 'split("\n") | map(select(length > 0)) | map(split(" ")) | map({path: .[1], sha: .[0]})' \
  > submodule-pins.json

# Store submodule-pins.json as a pipeline artifact
# In downstream pipeline: restore submodules to exact pins
git submodule update --init --recursive

9. Removing a Submodule¶

# 1. Deinit all tracked files and the submodule root
git submodule deinit -f -- libs/shared-lib

# 2. Remove the working directory (if not needed)
git rm libs/shared-lib

# 3. Remove the .git/modules entry
rm -rf .git/modules/libs/shared-lib

# 4. Commit
git commit -m "chore: remove shared-lib submodule"

10. Troubleshooting¶

Symptom	Cause	Fix
`fatal: needed be singular`	Submodule not initialised	`git submodule update --init`
Submodule dir is empty after clone	`--recurse-submodules` not used	`git submodule update --init --recursive`
`Permission denied (publickey)`	Submodule uses SSH URL in CI	Set credential helper or rewrite URL
Submodule appears dirty after `git submodule update`	Local branch diverged from recorded SHA	`git submodule update --force`
`Authentication failed` in CI	Token not propagated to submodule fetch	Use `url.<base>.insteadOf` rewrite
Submodule stuck at old SHA	Parent repo was not committed after `git submodule update --remote`	Re-run update and `git add` the new SHA

Recovering a Lost Submodule Commit¶

# Find the commit in the reflog of the submodule's git dir
git -C .git/modules/libs/shared-lib reflog
# Find the SHA you need, then:
git -C libs/shared-lib checkout <recovered-sha>
git -C libs/shared-lib checkout -b recovery