Appearance
Git Hotfix Lifecycle: Branch, Patch, Tag, Deploy
A hotfix is an urgent production fix that cannot wait for the regular release cycle. The defining constraint is speed without sacrificing audit trail. Every step must be reproducible, every commit signed, and every deployment tagged.
For a DevOps engineer, the hotfix lifecycle is the highest-stakes git workflow you run. This guide covers the complete lifecycle from branch creation through back-merge and cleanup, with patterns for CI/CD integration, parallel release lines, and emergency compliance.
The hotfix workflow
1. Branch git checkout -b hotfix/CVE-2024-9999 main
2. Fix git commit -s -m "fix(auth): validate JWT expiry before session"
3. Verify CI runs on hotfix branch (targeted tests only)
4. Tag git tag -s v4.2.1 -m "v4.2.1 — CVE-2024-9999 backport"
5. Deploy git push origin hotfix/CVE-2024-9999 --follow-tags
6. Merge git checkout main && git merge --no-ff hotfix/CVE-2024-9999
7. Backport git cherry-pick <SHA> release/v4.1.x release/v4.0.x
8. Cleanup git branch -d hotfix/CVE-2024-9999
git push origin --delete hotfix/CVE-2024-9999Each step is described in detail below.
1. Branch from the right base
The hotfix branches from the latest stable tag or release branch, not from main. Main may have unreleased features that would be pulled into production.
bash
# From the latest stable tag
git checkout -b hotfix/CVE-2024-9999 v4.2.0
# From a release branch (if you maintain release lines)
git checkout -b hotfix/CVE-2024-9999 release/v4.2.xThe wt-hotfix.sh script in this repo automates this:
bash
./scripts/wt-hotfix.sh CVE-2024-9999 v4.2.0This creates a worktree at ../<repo>-CVE-2024-9999 on branch hotfix/CVE-2024-9999 based on the tag v4.2.0. A worktree isolates the hotfix from your current working changes — you can work on the fix without stashing or switching branches.
2. Commit with signature and trailer
Every hotfix commit must be signed (GPG or SSH) and include a Signed-off-by: trailer for DCO compliance. The commit message should reference the issue or CVE in the body.
bash
git commit -sS -m "fix(auth): validate JWT expiry before session lookup
CVE-2024-9999: missing expiry check allows replay of expired tokens.
Backport target: v4.2.x, v4.1.x, v4.0.x"The -s flag adds Signed-off-by:. The -S flag signs with GPG/SSH. If your repo uses trailers for CI gating, add them:
bash
git commit -sS -m "fix(auth): validate JWT expiry
CVE-2024-9999
Backport: v4.2.x
Deploy-Env: production
CI-Skip-Tests: integration, e2e
"3. Verify in CI
Push the hotfix branch to trigger CI. The pipeline should run a reduced test set focused on the fix area, not the full suite.
bash
git push origin hotfix/CVE-2024-9999CI should detect the hotfix/ prefix and adjust scope:
yaml
# .forgejo/workflows/ci.yml
jobs:
test:
if: startsWith(github.ref, 'refs/heads/hotfix/')
steps:
- name: Detect hotfix scope
run: |
CHANGED=$(git diff --name-only origin/main...HEAD)
echo "Changed files: $CHANGED"
- name: Run targeted tests
run: |
# Only run tests for changed modules
./scripts/test-scope.sh "$CHANGED"Forgejo/GitHub branch protection rules should allow bypass for hotfix branches with an override review.
4. Tag the release
Tagging is mandatory for hotfixes. The tag is the deployment artifact that enables rollback, audit, and cherry-pick tracking.
bash
# Annotated and signed
git tag -s v4.2.1 -m "v4.2.1 — CVE-2024-9999: JWT expiry validation"
# If your project uses SemVer with pre-release metadata:
git tag -s v4.2.1-hotfix.1 -m "v4.2.1-hotfix.1 — CVE-2024-9999"The tag body should list the commits included and the CVEs addressed:
bash
git tag -s v4.2.1 -m "$(cat <<EOF
v4.2.1 — Security hotfix
Fixes:
- CVE-2024-9999: JWT expiry not validated in session middleware
(commit a1b2c3d4)
Backport status:
- release/v4.2.x: included in this tag
- release/v4.1.x: pending cherry-pick
- release/v4.0.x: pending cherry-pick
EOF
)"5. Deploy
Push the tag. Deploy pipelines should trigger on tag pushes, not branch pushes.
bash
git push origin hotfix/CVE-2024-9999 --follow-tagsThe deploy pipeline watches for signed tags matching v*:
yaml
# .forgejo/workflows/deploy.yml
on:
push:
tags:
- 'v*'
jobs:
deploy:
steps:
- name: Verify tag signature
run: |
git tag -v "${{ github.ref_name }}"
- name: Deploy to production
run: |
./scripts/deploy.sh "${{ github.ref_name }}"6. Merge back to main
After the hotfix is deployed, merge it back to main. Use --no-ff to preserve the hotfix branch topology in the history.
bash
git checkout main
git merge --no-ff hotfix/CVE-2024-9999 -m "merge: CVE-2024-9999 hotfix into main"
git push origin mainNever squash-merge a hotfix into main. The individual commits must remain intact for cherry-pick tracking and audit.
7. Cherry-pick to release branches
If you maintain multiple release lines, cherry-pick the fix to each active release branch.
bash
# Get the commit SHA from the hotfix branch
HOTFIX_SHA=$(git rev-parse hotfix/CVE-2024-9999)
# Cherry-pick to each release line
for branch in release/v4.2.x release/v4.1.x release/v4.0.x; do
git checkout "$branch"
git cherry-pick -xs "$HOTFIX_SHA"
git tag -s "v${branch#release/v}-hotfix.1" -m "v${branch#release/v}-hotfix.1 — CVE-2024-9999"
git push origin "$branch" --follow-tags
doneIf a cherry-pick conflicts, the release branch may have drifted. Resolve the conflict, commit, and note the deviation in the tag message.
8. Clean up
Delete the hotfix branch locally and remotely once everything is merged and tagged.
bash
git branch -d hotfix/CVE-2024-9999
git push origin --delete hotfix/CVE-2024-9999
git worktree prune # if you used a worktreeCI/CD integration patterns
Pattern 1: Hotfix branch protection bypass
Normal branch protection requires passing CI. For hotfixes, allow an override with a review from two maintainers.
yaml
# .forgejo/forgejo.yml or branch protection rules
protected_branches:
main:
required_pull_request: false # hotfixes merge directly
bypass_allowed: true # override for hotfix branches
required_approvals: 2 # but require reviewPattern 2: Automatic tag-based deploy
Deploy pipelines should trigger on v* tags, not on hotfix/* branch pushes. This prevents half-finished hotfix branches from triggering production deploys.
yaml
on:
push:
tags:
- 'v*'
- '!v*-rc*'
- '!v*-alpha*'
- '!v*-beta*'Pattern 3: Backport status tracking
Use git notes to annotate each cherry-pick with its backport status:
bash
# After cherry-picking to release/v4.1.x
git notes add -m "backported-to: release/v4.1.x (v4.1.1-hotfix.1)" "$HOTFIX_SHA"
git push origin refs/notes/*Then audit outstanding backports:
bash
git notes list | while read -r sha; do
note=$(git notes show "$sha" 2>/dev/null || echo "")
if ! echo "$note" | grep -q "backported-to: release/v4.0.x"; then
echo "MISSING: $sha not backported to release/v4.0.x"
fi
donePattern 4: Hotfix rollback
If a hotfix causes a regression, the rollback is a revert of the merge commit on main plus a tag bump.
bash
# On main, find the hotfix merge commit
git log --merges --first-parent --oneline main | grep "CVE-2024-9999"
# Revert it
git revert -m 1 <merge-SHA> --no-edit
git tag -s v4.2.2 -m "v4.2.2 — revert CVE-2024-9999 hotfix (regression)"
git push origin main --follow-tagsThen fix the issue properly on a new hotfix branch and repeat the cycle.
Multiple release line strategy
When you maintain N active release lines, a hotfix must propagate through all of them. The order matters:
- Fix on the oldest supported release line first (narrowest diff, fewest conflicts).
- Cherry-pick forward to each newer line.
- Merge into main last.
bash
# Fix on the oldest release line
git checkout -b hotfix/CVE-2024-9999 v4.0.0
# ... fix, commit, tag, deploy ...
# Cherry-pick forward
git checkout release/v4.1.x
git cherry-pick -xs <SHA>
git tag -s v4.1.1 -m "v4.1.1 — CVE-2024-9999"
git push origin release/v4.1.x --follow-tags
git checkout release/v4.2.x
git cherry-pick -xs <SHA>
git tag -s v4.2.1 -m "v4.2.1 — CVE-2024-9999"
git push origin release/v4.2.x --follow-tags
# Finally main
git checkout main
git merge --no-ff release/v4.2.x -m "merge: CVE-2024-9999 hotfix"
git push origin mainThis forward-propagation approach minimizes conflicts because each step moves to a newer, more changed codebase. Doing it in reverse (fix on main, then backport) is the common but conflict-prone approach.
Decision table: hotfix vs regular release
| Factor | Hotfix | Regular release |
|---|---|---|
| Trigger | Production incident | Sprint cadence |
| Branch base | Stable tag or release branch | Main |
| CI scope | Targeted (changed modules) | Full suite |
| Sign-off | 2 maintainers (emergency override) | PR review + CI |
| Merge strategy | --no-ff (preserve commits) | Squash or rebase |
| Tag required | Yes, signed | Yes, annotated |
| Backport | Immediate, all active lines | Per-release schedule |
| Rollback plan | Required before deploy | Optional |
Pitfalls
- Branching from main. If main has unreleased features, the hotfix pulls them into production. Always branch from the last deployed tag.
- Squash merging the hotfix. Squash destroys the individual commit SHAs that cherry-pick and backport tracking depend on. Use
--no-ff. - Forgetting the tag. A hotfix without a tag cannot be rolled back to cleanly. The tag is the deployment artifact.
- Cherry-pick order. Cherry-picking from main backward (to older release lines) causes more conflicts. Fix on the oldest line and cherry-pick forward.
- Missing signed commits. A hotfix that bypasses signing requirements breaks supply-chain audit. Enforce signatures server-side with a pre-receive hook.
- Overlapping hotfixes. Two hotfixes in flight at the same time on the same release line create merge conflicts between them. Serialise them or use stacked hotfix branches (base the second on the first's tag).
- Hotfix branch leak. A hotfix branch pushed before the fix is complete may trigger deploy pipelines that pick up the tag. Use
--follow-tagsonly when ready, or push the tag separately from the branch.
Companion resources
- Script:
scripts/wt-hotfix.sh— create a hotfix worktree from a tag. - Script:
scripts/hotfix-manage.sh— automate the full lifecycle. - Recipe:
docs/recipes/hotfix-lifecycle-quick.md— one-page cheat sheet. - Related:
docs/devops-workflows/git-tagging-strategy.md— tag conventions. - Related:
docs/devops-workflows/git-cherry-pick.md— selective backport. - Related:
docs/devops-workflows/git-commit-signing.md— GPG/SSH signing.