Skip to content

Git attributes and filter drivers for DevOps

Git attributes define path-specific behavior across repository workflows. They control end-of-line normalization, custom diff engines, binary identification, and clean-smudge filter pipelines. In DevOps and continuous integration workflows, filter drivers automate secret sanitization, canonicalize structured configurations like JSON and YAML for clean diffs, and normalize file formats across diverse operating systems.

1. Core architecture

Git applies attributes defined in .gitattributes at the root or inside subdirectories. Attributes match file patterns and assign specific driver names.

Working tree file
       ▲               │
       │ smudge        │ clean
       │ (checkout)    │ (stage / commit)
       │               ▼
  Object database / Git index

Two distinct driver mechanisms operate on file contents:

  1. Filter drivers (filter.<name>): Transform file content when moving between the working tree and the Git object database.
  2. clean: Runs when staging files with git add. Converts working tree contents into the format stored in Git.
  3. smudge: Runs when checking out files with git checkout, git switch, or git restore. Converts stored repository contents into the working tree format.

  4. required: If set to true, a filter failure aborts the Git operation. If false, Git falls back to raw file contents on failure.

  5. Diff drivers (diff.<name>): Transform file content dynamically during git diff and git log -p without modifying stored blobs or working tree files.

  6. textconv: Converts binary or formatted text files into plain text before generating diffs.
  7. xfuncname: Defines regular expressions that identify the enclosing function or section header displayed on hunk headers (@@ ... @@).
  8. binary: Forces Git to treat matching files as binary data, preventing text diffs and merge conflicts.

2. Setting up clean and smudge filters

Secret sanitization before staging

A clean filter can strip local connection strings, development API tokens, or private endpoints before committing changes to Git. The smudge filter restores local developer values upon checkout.

Step 1. Define the attribute

Add the filter assignment to .gitattributes:

# .gitattributes
config/local.env filter=sanitize-env
deploy/*.conf filter=sanitize-conf

Step 2. Register the filter scripts

Configure the clean and smudge commands in local Git configuration (.git/config) or system configuration:

# Register clean filter: strips actual tokens and writes placeholders to the index
git config filter.sanitize-env.clean "sed -E 's/(API_KEY|SECRET_TOKEN)=.*/\1=REDACTED_BY_GIT_FILTER/'"

# Register smudge filter: reads local override file if present, else outputs raw repository stream
git config filter.sanitize-env.smudge "bash -c '
  if [ -f ~/.config/dev-secrets.env ]; then
    source ~/.config/dev-secrets.env
    sed -e \"s/API_KEY=REDACTED_BY_GIT_FILTER/API_KEY=\$LOCAL_DEV_API_KEY/\"
  else
    cat
  fi
'"

# Mark filter non-fatal so CI builds without local secrets succeed
git config filter.sanitize-env.required false

3. Custom diff drivers and textconv

Standard git diff struggles with structured data where key ordering varies between tool runs, as well as binary payloads such as PDF runbooks, compiled schemas, or Jupyter notebooks. Diff drivers solve this by converting data on the fly.

3.1 Canonical JSON and YAML diffs

When automated tools serialize JSON or YAML with random key ordering, git diff reports massive, unreadable line changes. A textconv filter sorts keys before diff computation.

Step 1. Assign diff drivers in .gitattributes

# .gitattributes
*.json diff=json-canonical
*.yaml diff=yaml-canonical
*.yml diff=yaml-canonical

Step 2. Configure textconv commands

# Canonical JSON diff via jq
git config diff.json-canonical.textconv "jq --sort-keys ."

# Canonical YAML diff via python or yq
git config diff.yaml-canonical.textconv "python3 -c 'import sys, yaml; print(yaml.dump(yaml.safe_load(sys.stdin.read()), sort_keys=True, indent=2))'"

When running git diff, Git runs the command on both versions and diffs the normalized output. Stored repository blobs remain untouched.

3.2 Jupyter notebook diffs

DevOps runbooks and data pipelines frequently track .ipynb files. Raw diffs contain noisy base64 plot outputs and execution counters.

*.ipynb diff=ipynb-clean

# Strip outputs and metadata for clean notebook diffs
git config diff.ipynb-clean.textconv "jq -M '{cells: [.cells[] | {cell_type, source, outputs: [.outputs[]? | select(.output_type != \"stream\")]}]}'"

3.3 Custom hunk context with xfuncname

The hunk header @@ -12,6 +12,8 @@ <context> displays the enclosing function or section. For infrastructure-as-code files, default C-style regex fails to show meaningful context.

Terraform / HCL hunk headers

*.tf diff=terraform
*.tfvars diff=terraform

# Capture resource, module, and data block headers
git config diff.terraform.xfuncname '^[[:space:]]*((resource|data|module|variable|output)[[:space:]]+"[^"]+"([[:space:]]+"[^"]+")?)'

Kubernetes manifests

*.k8s.yaml diff=k8s

# Capture kind and metadata.name
git config diff.k8s.xfuncname '^([A-Za-z]+:.*|  name:.*)'

4. End-of-line normalization and binary handling

Mixed line endings cause false diffs across Windows, macOS, and Linux CI runners. Enforce consistent rules at the repository root.

# Set default behavior to normalize text files to LF on checkout
* text=auto eol=lf

# Explicit text files
*.sh text eol=lf
*.py text eol=lf
*.md text
*.json text
*.yaml text
*.yml text

# Explicit binary files: disable CRLF translation and diff generation
*.png binary
*.jpg binary
*.tar.gz binary
*.tgz binary
*.zip binary
*.parquet binary
*.wasm binary
*.so binary
*.dylib binary

To normalize existing files across the entire history of the working tree:

# Renormalize all tracked files against updated .gitattributes
git add --renormalize .
git commit -m "chore: renormalize repository line endings to LF"

5. Verifying and debugging attributes

To determine which attributes Git resolves for a specific file path:

# Check all attributes assigned to a path
git check-attr -a -- deploy/values.yaml

# Check specific attribute
git check-attr diff -- config/schema.json
git check-attr filter -- config/local.env

# Inspect resolved rules across entire repository
git check-attr -a -- $(git ls-files)

6. Long-running filter process protocol

Single-file clean and smudge scripts incur process fork overhead for every tracked file during checkout. For high-throughput monorepos, Git provides a long-running stdio protocol configured with filter.<name>.process.

Git launches a single persistent daemon process that communicates with Git via packet-format lines over stdin and stdout. Tools like Git LFS use this mechanism to stream hundreds of large files without restarting processes.

# Example process filter configuration
git config filter.custom-lfs.process "custom-filter-agent --protocol=git-filter"
git config filter.custom-lfs.required true

7. CI/CD automation and pipeline setup

When CI runners clone repositories with custom attributes, runners must configure matching drivers or bypass non-critical filters.

Pipeline bootstrap pattern

In your CI setup step (e.g. GitHub Actions, Forgejo Actions, GitLab CI):

#!/usr/bin/env bash
# ci-setup-attributes.sh

set -euo pipefail

# 1. Register diff drivers for PR review and automated changelog generation
if command -v jq >/dev/null 2>&1; then
    git config diff.json-canonical.textconv "jq --sort-keys ."
fi

if command -v python3 >/dev/null 2>&1; then
    git config diff.yaml-canonical.textconv "python3 -c 'import sys, yaml; print(yaml.dump(yaml.safe_load(sys.stdin.read()), sort_keys=True, indent=2))'"
fi

# 2. Configure Terraform diff driver
git config diff.terraform.xfuncname '^[[:space:]]*((resource|data|module|variable|output)[[:space:]]+"[^"]+"([[:space:]]+"[^"]+")?)'

# 3. Ensure smudge filters do not break builds if dev secrets are absent
git config filter.sanitize-env.smudge "cat"
git config filter.sanitize-env.clean "cat"
git config filter.sanitize-env.required false

8. Common pitfalls and edge cases

  1. Non-idempotent clean and smudge filters: If smudge(clean(file)) != file, Git marks the file modified immediately after checkout, causing dirty working trees on fresh clones. Test filter symmetry before deploying.
  2. Missing filter binaries with required=true: Setting required = true on a filter whose command is missing breaks git checkout and git clone. Use required = false for developer conveniences.
  3. Diff caching: Git caches textconv results. If the conversion script changes, clear the cache with git update-ref -d refs/notes/textconv/diff-filter-name or run git diff --no-textconv.
  4. Security boundary: .gitattributes files are checked into version control, but the actual command executions are configured in .git/config. Git deliberately isolates driver definitions to prevent repository clones from running arbitrary shell commands without user consent.