implementing-secret-scanning-with-gitleaks

使用 Gitleaks 实现密钥扫描

适用场景

• 开发者可能会意外将 API 密钥、密码、令牌或私钥提交到代码仓库时
• 建立提交前检查，以防止密钥进入 git 历史记录时
• 扫描现有代码仓库历史，查找此前提交、需要轮换的密钥时
• 合规要求规定必须检测所有源代码仓库中的密钥时
• 从手动审计密钥迁移到自动化持续扫描时

请勿用于检测运行中的应用或内存中的密钥（请使用运行时密钥检测工具）、管理检测到的密钥（请使用 Vault 或 AWS Secrets Manager），或扫描容器镜像（请使用 Trivy 或 Grype）。

前提条件

• 已通过二进制文件、Go install 或 Docker 安装 Gitleaks v8.18+
• 已安装 pre-commit 框架，用于集成本地钩子
• 有待扫描历史记录的 Git 代码仓库
• 可访问 CI/CD 平台（GitHub Actions、GitLab CI 或同类平台）

工作流

步骤 1：安装并运行初始代码仓库扫描

对代码仓库进行基线扫描，以识别 git 历史记录中已有的所有密钥。

# Install Gitleaks
brew install gitleaks  # macOS
# or download binary from https://github.com/gitleaks/gitleaks/releases

# Scan entire git history for secrets
gitleaks detect --source . --report-format json --report-path gitleaks-report.json -v

# Scan only staged changes (for pre-commit)
gitleaks protect --staged --report-format json --report-path gitleaks-staged.json

# Scan specific commit range
gitleaks detect --source . --log-opts="HEAD~10..HEAD" --report-format json

# Scan without git history (filesystem only)
gitleaks detect --source . --no-git --report-format json

步骤 2：配置提交前钩子

将 Gitleaks 配置为提交前钩子，防止密钥被提交。

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.21.2
    hooks:
      - id: gitleaks
        name: gitleaks
        description: Detect hardcoded secrets using Gitleaks
        entry: gitleaks protect --staged --verbose --redact
        language: golang
        pass_filenames: false

# Install pre-commit framework
pip install pre-commit

# Install hooks defined in .pre-commit-config.yaml
pre-commit install

# Run against all files (not just staged)
pre-commit run gitleaks --all-files

# Test the hook with a deliberate secret
echo 'AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"' >> test.txt
git add test.txt
git commit -m "test"  # Should be blocked by gitleaks

步骤 3：集成到 GitHub Actions

# .github/workflows/secret-scanning.yml
name: Secret Scanning

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  gitleaks:
    name: Gitleaks Secret Scan
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Full history for comprehensive scanning

      - name: Run Gitleaks
        uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GITLEAKS_LICENSE: ${{ secrets.GITLEAKS_LICENSE }}  # Required for gitleaks-action v2

      # Alternative: Run Gitleaks directly
      - name: Install Gitleaks
        run: |
          wget -q https://github.com/gitleaks/gitleaks/releases/download/v8.21.2/gitleaks_8.21.2_linux_x64.tar.gz
          tar -xzf gitleaks_8.21.2_linux_x64.tar.gz
          chmod +x gitleaks

      - name: Scan for secrets
        run: |
          if [ "${{ github.event_name }}" == "pull_request" ]; then
            ./gitleaks detect \
              --source . \
              --log-opts="${{ github.event.pull_request.base.sha }}..${{ github.event.pull_request.head.sha }}" \
              --report-format sarif \
              --report-path gitleaks.sarif \
              --exit-code 1
          else
            ./gitleaks detect \
              --source . \
              --report-format sarif \
              --report-path gitleaks.sarif \
              --exit-code 1 \
              --baseline-path .gitleaks-baseline.json
          fi

      - name: Upload SARIF
        if: always()
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: gitleaks.sarif
          category: gitleaks

步骤 4：编写自定义检测规则

为组织内部的密钥模式创建专用规则。

# .gitleaks.toml
title = "Organization Gitleaks Configuration"

[extend]
useDefault = true  # Include all default rules

# Custom rule for internal API tokens
[[rules]]
id = "internal-api-token"
description = "Internal API token for service-to-service auth"
regex = '''(?i)x-internal-token["\s:=]+["\']?([a-zA-Z0-9_\-]{40,})["\']?'''
entropy = 3.5
keywords = ["x-internal-token"]
tags = ["internal", "api"]

[[rules]]
id = "database-connection-string"
description = "Database connection string with embedded credentials"
regex = '''(?i)(postgres|mysql|mongodb|redis)://[^:]+:[^@]+@[^/]+/\w+'''
keywords = ["postgres://", "mysql://", "mongodb://", "redis://"]
tags = ["database", "credentials"]

[[rules]]
id = "jwt-secret"
description = "JWT signing secret"
regex = '''(?i)(jwt[_-]?secret|jwt[_-]?key)["\s:=]+["\']?([a-zA-Z0-9/+_\-]{32,})["\']?'''
entropy = 3.0
keywords = ["jwt_secret", "jwt-secret", "jwt_key", "jwt-key"]

# Allowlist for test files and known safe patterns
[allowlist]
description = "Global allowlist"
paths = [
  '''(^|/)test(s)?/''',
  '''(^|/)spec/''',
  '''\.test\.(js|ts|py)$''',
  '''\.spec\.(js|ts|py)$''',
  '''__mocks__/''',
  '''fixtures/''',
  '''(^|/)vendor/''',
  '''node_modules/'''
]
regexes = [
  '''EXAMPLE''',
  '''example\.com''',
  '''test[-_]?(key|secret|token|password)''',
  '''(?i)placeholder''',
  '''000000+'''
]

步骤 5：管理现有代码仓库的基线

为已知发现项创建基线，以免在轮换历史密钥期间阻碍开发。

# Generate baseline from current state
gitleaks detect --source . --report-format json --report-path .gitleaks-baseline.json

# Subsequent scans compare against baseline (only new findings trigger failures)
gitleaks detect --source . --baseline-path .gitleaks-baseline.json --exit-code 1

# Review baseline periodically and remove entries as secrets are rotated
cat .gitleaks-baseline.json | python3 -m json.tool | head -50

步骤 6：修复已泄露的密钥

检测到密钥时，请遵循轮换和清理历史记录的流程。

# 1. Immediately rotate the exposed credential
#    - Revoke the old API key/token in the service provider
#    - Generate a new credential
#    - Store the new credential in a secrets manager

# 2. Remove secret from git history using git-filter-repo
pip install git-filter-repo

# Create expressions file for secrets to remove
cat > /tmp/expressions.txt << 'EOF'
regex:AKIA[0-9A-Z]{16}==>REDACTED_AWS_KEY
regex:(?i)password\s*=\s*"[^"]*"==>password="REDACTED"
EOF

git filter-repo --replace-text /tmp/expressions.txt --force

# 3. Force-push the cleaned history (coordinate with team)
# git push --force --all  # WARNING: Requires team coordination

# 4. Add the secret pattern to .gitleaks.toml rules
# 5. Update the baseline file to remove the resolved finding

核心概念

术语	定义
密钥	不应出现在源代码中的任何凭据、令牌、密钥或敏感字符串
提交前钩子	在创建提交之前运行的 Git 钩子，会阻止包含已检测密钥的提交
熵	字符串随机程度的度量；高熵字符串更可能是密钥
基线	现有发现项的快照，用于区分新密钥和此前已存在的密钥
允许列表	指定要从检测中排除的路径、模式或提交的配置
SARIF	静态分析结果交换格式，用于将发现项上传到安全仪表板
git-filter-repo	用于重写 git 历史记录、从所有提交中移除敏感数据的工具

工具与系统

• Gitleaks：开源密钥检测工具，支持提交前钩子、CI/CD 和历史记录扫描
• pre-commit：用于管理和维护多语言提交前钩子的框架
• git-filter-repo：用于重写历史记录、从 git 历史中移除密钥的工具
• TruffleHog：另一款密钥扫描工具，具备已验证密钥检测能力
• GitHub Secret Scanning：GitHub 的原生功能，可检测符合合作伙伴模式的密钥

常见场景

场景：在旧代码仓库中启用机密扫描

背景：一个已有 5 年历史的代码仓库从未接受过扫描。团队需要启用机密扫描，同时避免在轮换历史机密期间阻碍所有开发工作。

方法：

1. 对完整历史记录运行 gitleaks detect，并生成基线 JSON 文件
2. 对每项发现进行分类：有效（需要轮换）、无效（已轮换）或误报
3. 立即轮换所有有效机密，并更新使用这些机密的服务
4. 提交基线文件（排除已修复的有效机密）
5. 立即为新的开发工作启用提交前钩子
6. 在 CI/CD 中使用基线添加扫描，以便只检测新的机密
7. 随着历史机密逐步轮换，逐步缩减基线

注意事项：未进行分类就生成基线，意味着接受尚未轮换的机密所带来的风险。切勿未经服务提供方确认，就假定历史机密已失效。在未协调的情况下对共享代码仓库运行 git-filter-repo，会导致所有团队成员遇到变基冲突。

输出格式

Gitleaks Secret Scanning Report
=================================
Repository: org/web-application
Scan Type: Full History
Commits Scanned: 4,523
Date: 2026-02-23

FINDINGS:
  Total: 12
  New (not in baseline): 3
  Baseline (pre-existing): 9

NEW FINDINGS (blocking):
  [1] AWS Access Key ID
      Rule: aws-access-key-id
      File: src/config/aws.py:23
      Commit: a1b2c3d (2026-02-22, [email protected])
      Secret: AKIA...REDACTED
      Entropy: 3.8

  [2] GitHub Personal Access Token
      Rule: github-pat
      File: scripts/deploy.sh:15
      Commit: d4e5f6g (2026-02-21, [email protected])
      Secret: ghp_...REDACTED
      Entropy: 4.2

  [3] Internal API Token
      Rule: internal-api-token
      File: src/services/auth.py:89
      Commit: h7i8j9k (2026-02-20, [email protected])

QUALITY GATE: FAILED (3 new findings)
Action: Rotate exposed credentials immediately.