Gitleaks

Gitleaks

┌─○───┐
│ │╲  │
│ │ ○ │
│ ○ ░ │
└─░───┘

警告

Gitleaks 功能已完善。我不会再将新功能合并到 Gitleaks 中。未来版本将仅包含安全补丁。我将把精力转向 Betterleaks

Gitleaks 是一款工具，用于检测 git 仓库、文件以及通过 stdin 输入的任何内容中的密码、API 密钥和令牌等秘密。如果你想进一步了解检测引擎的工作原理，请查看这篇博客：正则表达式几乎就够用了。

➜  ~/code(master) gitleaks git -v

    ○
    │╲
    │ ○
    ○ ░
    ░    gitleaks


Finding:     "export BUNDLE_ENTERPRISE__CONTRIBSYS__COM=cafebabe:deadbeef",
Secret:      cafebabe:deadbeef
RuleID:      sidekiq-secret
Entropy:     2.609850
File:        cmd/generate/config/rules/sidekiq.go
Line:        23
Commit:      cd5226711335c68be1e720b318b7bc3135a30eb2
Author:      John
Email:       [email protected]
Date:        2022-08-03T12:31:40Z
Fingerprint: cd5226711335c68be1e720b318b7bc3135a30eb2:cmd/generate/config/rules/sidekiq.go:sidekiq-secret:23

快速开始

Gitleaks 可以通过 Homebrew、Docker 或 Go 安装。Gitleaks 也提供适用于许多热门平台和操作系统类型的二进制文件，可在发布页面获取。此外，还可以将 Gitleaks 配置为仓库中的 pre-commit 钩子，或通过 Gitleaks-Action 配置为 GitHub Action。

安装

# MacOS
brew install gitleaks

# Docker (DockerHub)
docker pull zricethezav/gitleaks:latest
docker run -v ${path_to_host_folder_to_scan}:/path zricethezav/gitleaks:latest [COMMAND] [OPTIONS] [SOURCE_PATH]

# Docker (ghcr.io)
docker pull ghcr.io/gitleaks/gitleaks:latest
docker run -v ${path_to_host_folder_to_scan}:/path ghcr.io/gitleaks/gitleaks:latest [COMMAND] [OPTIONS] [SOURCE_PATH]

# From Source (make sure `go` is installed)
git clone https://github.com/gitleaks/gitleaks.git
cd gitleaks
make build

Pre-Commit

1. 从 https://pre-commit.com/#install 安装 pre-commit

2. 在仓库根目录创建 .pre-commit-config.yaml 文件，并写入以下内容：

   repos:
     - repo: https://github.com/gitleaks/gitleaks
       rev: v8.24.2
       hooks:
         - id: gitleaks
   

   以原生方式运行 gitleaks，或使用 gitleaks-docker pre-commit ID，通过官方 Docker 镜像运行 gitleaks

3. 执行 pre-commit autoupdate，将配置自动更新到仓库的最新版本

4. 使用 pre-commit install 安装

5. 现在一切就绪！

➜ git commit -m "this commit contains a secret"
Detect hardcoded secrets.................................................Failed

注意：若要禁用 gitleaks pre-commit 钩子，可在提交命令前添加 SKIP=gitleaks 这样就会跳过运行 gitleaks

➜ SKIP=gitleaks git commit -m "skip gitleaks check"
Detect hardcoded secrets................................................Skipped

用法

Gitleaks scans code, past or present, for secrets

Usage:
  gitleaks [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  dir         scan directories or files for secrets
  git         scan git repositories for secrets
  help        Help about any command
  stdin       detect secrets from stdin
  version     display gitleaks version

Flags:
  -b, --baseline-path string          path to baseline with issues that can be ignored
  -c, --config string                 config file path
                                      order of precedence:
                                      1. --config/-c
                                      2. env var GITLEAKS_CONFIG
                                      3. env var GITLEAKS_CONFIG_TOML with the file content
                                      4. (target path)/.gitleaks.toml
                                      If none of the four options are used, then gitleaks will use the default config
      --diagnostics string            enable diagnostics (http OR comma-separated list: cpu,mem,trace). cpu=CPU prof, mem=memory prof, trace=exec tracing, http=serve via net/http/pprof
      --diagnostics-dir string        directory to store diagnostics output files when not using http mode (defaults to current directory)
      --enable-rule strings           only enable specific rules by id
      --exit-code int                 exit code when leaks have been encountered (default 1)
  -i, --gitleaks-ignore-path string   path to .gitleaksignore file or folder containing one (default ".")
  -h, --help                          help for gitleaks
      --ignore-gitleaks-allow         ignore gitleaks:allow comments
  -l, --log-level string              log level (trace, debug, info, warn, error, fatal) (default "info")
      --max-archive-depth int         allow scanning into nested archives up to this depth (default "0", no archive traversal is done)
      --max-decode-depth int          allow recursive decoding up to this depth (default "0", no decoding is done)
      --max-target-megabytes int      files larger than this will be skipped
      --no-banner                     suppress banner
      --no-color                      turn off color for verbose output
      --redact uint[=100]             redact secrets from logs and stdout. To redact only parts of the secret just apply a percent value from 0..100. For example --redact=20 (default 100%)
  -f, --report-format string          output format (json, csv, junit, sarif, template)
  -r, --report-path string            report file
      --report-template string        template file used to generate the report (implies --report-format=template)
      --timeout int                   set a timeout for gitleaks commands in seconds (default "0", no timeout is set)
  -v, --verbose                       show verbose output from scan
      --version                       version for gitleaks

Use "gitleaks [command] --help" for more information about a command.

命令

⚠️ v8.19.0 引入了一项变更，弃用了 detect 和 protect。这些命令仍可使用， 但不会显示在 --help 菜单中。请查看此 gist，了解如何轻松转换命令。 如果 v8.19.0 导致现有命令（detect/protect）无法使用，请提交 issue。

有三种扫描模式：git、dir 和 stdin。

Git

git 命令可用于扫描本地 git 仓库。在底层，gitleaks 使用 git log -p 命令扫描补丁。 你可以使用 log-opts 选项配置 git log -p 的行为。 例如，如果要对一段提交范围运行 gitleaks，可以使用以下命令：gitleaks git -v --log-opts="--all commitA..commitB" path_to_repo。更多信息请参阅 git log 文档。 如果未通过位置参数指定目标，gitleaks 会尝试将当前工作目录作为 git 仓库进行扫描。

Dir

dir 命令（别名包括 files、directory）可让你扫描目录和文件。例如：gitleaks dir -v path_to_directory_or_file。 如果未通过位置参数指定目标，gitleaks 将扫描当前工作目录。

Stdin

你也可以使用 stdin 命令将数据流式传输给 gitleaks。例如：cat some_file | gitleaks -v stdin

创建基线

扫描大型仓库或历史记录较长的仓库时，使用基线会很方便。使用基线时，gitleaks 会忽略基线中已有的旧发现。基线可以是任何 gitleaks 报告。若要创建 gitleaks 报告，请使用 --report-path 参数运行 gitleaks。

gitleaks git --report-path gitleaks-report.json # This will save the report in a file called gitleaks-report.json

创建基线后，可在再次运行 detect 命令时应用它：

gitleaks git --baseline-path gitleaks-report.json --report-path findings.json

使用 --baseline-path 参数运行 detect 命令后，报告输出（findings.json）将只包含新问题。

Pre-Commit hook

将示例 pre-commit.py 脚本复制到你的 .git/hooks/ 目录，即可将 Gitleaks 用作 pre-commit hook。

加载配置

优先级顺序如下：

1. --config/-c 选项：

   gitleaks git --config /home/dev/customgitleaks.toml .
   

2. 环境变量 GITLEAKS_CONFIG，其值为文件路径：

   export GITLEAKS_CONFIG="/home/dev/customgitleaks.toml"
   gitleaks git .
   

3. 环境变量 GITLEAKS_CONFIG_TOML，其值为文件内容：

   export GITLEAKS_CONFIG_TOML=`cat customgitleaks.toml`
   gitleaks git .
   

4. 目标路径中的 .gitleaks.toml 文件：

   gitleaks git .
   

如果未使用以上四种选项中的任何一种，gitleaks 将使用默认配置。

配置

Gitleaks 提供了一种配置格式，你可以遵循该格式编写自己的密钥检测规则：

# Title for the gitleaks configuration file.
title = "Custom Gitleaks configuration"

# You have basically two options for your custom configuration:
#
# 1. define your own configuration, default rules do not apply
#
#    use e.g., the default configuration as starting point:
#    https://github.com/gitleaks/gitleaks/blob/master/config/gitleaks.toml
#
# 2. extend a configuration, the rules are overwritten or extended
#
#    When you extend a configuration the extended rules take precedence over the
#    default rules. I.e., if there are duplicate rules in both the extended
#    configuration and the default configuration the extended rules or
#    attributes of them will override the default rules.
#    Another thing to know with extending configurations is you can chain
#    together multiple configuration files to a depth of 2. Allowlist arrays are
#    appended and can contain duplicates.

# useDefault and path can NOT be used at the same time. Choose one.
[extend]
# useDefault will extend the default gitleaks config built in to the binary
# the latest version is located at:
# https://github.com/gitleaks/gitleaks/blob/master/config/gitleaks.toml
useDefault = true
# or you can provide a path to a configuration to extend from.
# The path is relative to where gitleaks was invoked,
# not the location of the base config.
# path = "common_config.toml"
# If there are any rules you don't want to inherit, they can be specified here.
disabledRules = [ "generic-api-key"]

# An array of tables that contain information that define instructions
# on how to detect secrets
[[rules]]
# Unique identifier for this rule
id = "awesome-rule-1"

# Short human-readable description of the rule.
description = "awesome rule 1"

# Golang regular expression used to detect secrets. Note Golang's regex engine
# does not support lookaheads.
regex = '''one-go-style-regex-for-this-rule'''

# Int used to extract secret from regex match and used as the group that will have
# its entropy checked if `entropy` is set.
secretGroup = 3

# Float representing the minimum shannon entropy a regex group must have to be considered a secret.
entropy = 3.5

# Golang regular expression used to match paths. This can be used as a standalone rule or it can be used
# in conjunction with a valid `regex` entry.
path = '''a-file-path-regex'''

# Keywords are used for pre-regex check filtering. Rules that contain
# keywords will perform a quick string compare check to make sure the
# keyword(s) are in the content being scanned. Ideally these values should
# either be part of the identiifer or unique strings specific to the rule's regex
# (introduced in v8.6.0)
keywords = [
  "auth",
  "password",
  "token",
]

# Array of strings used for metadata and reporting purposes.
tags = ["tag","another tag"]

    # ⚠️ In v8.21.0 `[rules.allowlist]` was replaced with `[[rules.allowlists]]`.
    # This change was backwards-compatible: instances of `[rules.allowlist]` still  work.
    #
    # You can define multiple allowlists for a rule to reduce false positives.
    # A finding will be ignored if _ANY_ `[[rules.allowlists]]` matches.
    [[rules.allowlists]]
    description = "ignore commit A"
    # When multiple criteria are defined the default condition is "OR".
    # e.g., this can match on |commits| OR |paths| OR |stopwords|.
    condition = "OR"
    commits = [ "commit-A", "commit-B"]
    paths = [
      '''go\.mod''',
      '''go\.sum'''
    ]
    # note: stopwords targets the extracted secret, not the entire regex match
    # like 'regexes' does. (stopwords introduced in 8.8.0)
    stopwords = [
      '''client''',
      '''endpoint''',
    ]

    [[rules.allowlists]]
    # The "AND" condition can be used to make sure all criteria match.
    # e.g., this matches if |regexes| AND |paths| are satisfied.
    condition = "AND"
    # note: |regexes| defaults to check the _Secret_ in the finding.
    # Acceptable values for |regexTarget| are "secret" (default), "match", and "line".
    regexTarget = "match"
    regexes = [ '''(?i)parseur[il]''' ]
    paths = [ '''package-lock\.json''' ]

# You can extend a particular rule from the default config. e.g., gitlab-pat
# if you have defined a custom token prefix on your GitLab instance
[[rules]]
id = "gitlab-pat"
# all the other attributes from the default rule are inherited

    [[rules.allowlists]]
    regexTarget = "line"
    regexes = [ '''MY-glpat-''' ]


# ⚠️ In v8.25.0 `[allowlist]` was replaced with `[[allowlists]]`.
#
# Global allowlists have a higher order of precedence than rule-specific allowlists.
# If a commit listed in the `commits` field below is encountered then that commit will be skipped and no
# secrets will be detected for said commit. The same logic applies for regexes and paths.
[[allowlists]]
description = "global allow list"
commits = [ "commit-A", "commit-B", "commit-C"]
paths = [
  '''gitleaks\.toml''',
  '''(.*?)(jpg|gif|doc)'''
]
# note: (global) regexTarget defaults to check the _Secret_ in the finding.
# Acceptable values for regexTarget are "match" and "line"
regexTarget = "match"
regexes = [
  '''219-09-9999''',
  '''078-05-1120''',
  '''(9[0-9]{2}|666)-\d{2}-\d{4}''',
]
# note: stopwords targets the extracted secret, not the entire regex match
# like 'regexes' does. (stopwords introduced in 8.8.0)
stopwords = [
  '''client''',
  '''endpoint''',
]

# ⚠️ In v8.25.0, `[[allowlists]]` have a new field called |targetRules|.
#
# Common allowlists can be defined once and assigned to multiple rules using |targetRules|.
# This will only run on the specified rules, not globally.
[[allowlists]]
targetRules = ["awesome-rule-1", "awesome-rule-2"]
description = "Our test assets trigger false-positives in a couple rules."
paths = ['''tests/expected/._\.json$''']

如需了解示例，请参阅默认的 gitleaks 配置；如需为默认配置做出贡献，请遵循贡献指南。此外，你还可以查看这篇 gitleaks 博客文章，了解高级配置方案。

其他配置

复合规则（多部分或 required 规则）

在 v8.28.0 中，Gitleaks 引入了复合规则。复合规则由一个“主”规则和一个或多个辅助规则（或 required 规则）组成。要创建复合规则，请在主规则中添加一个 [[rules.required]] 表格，指定 id，并可选择设置 withinLines 和／或 withinColumns 邻近约束。片段是 Gitleaks 一次处理的一段内容（通常是一个文件、文件的一部分或 git diff），而邻近匹配会指示主规则：只有当辅助 required 规则也在片段的指定区域内找到匹配项时，才报告发现项。

**邻近匹配：**使用 withinLines 和 withinColumns 字段，可指示主规则：只有当辅助 required 规则也在指定邻近范围内找到匹配项时，才报告发现项。你可以设置：

• withinLines: N - required 发现项必须位于 N 行以内（垂直方向）
• withinColumns: N - required 发现项必须位于 N 个字符以内（水平方向）
• 两者都设置 - 创建矩形搜索区域（必须同时满足两个约束）
• 两者都不设置 - 按片段匹配（required 发现项可以位于同一片段中的任意位置）

以下图示说明了各类邻近匹配行为：

p = primary captured secret
a = auxiliary (required) captured secret
fragment = section of data gitleaks is looking at


    *Fragment-level proximity*
    Any required finding in the fragment
          ┌────────┐
   ┌──────┤fragment├─────┐
   │      └──────┬─┤     │ ┌───────┐
   │             │a│◀────┼─│✓ MATCH│
   │          ┌─┐└─┘     │ └───────┘
   │┌─┐       │p│        │
   ││a│    ┌─┐└─┘        │ ┌───────┐
   │└─┘    │a│◀──────────┼─│✓ MATCH│
   └─▲─────┴─┴───────────┘ └───────┘
     │    ┌───────┐
     └────│✓ MATCH│
          └───────┘


   *Column bounded proximity*
   `withinColumns = 3`
          ┌────────┐
   ┌────┬─┤fragment├─┬───┐
   │      └──────┬─┤     │ ┌───────────┐
   │    │        │a│◀┼───┼─│+1C ✓ MATCH│
   │          ┌─┐└─┘     │ └───────────┘
   │┌─┐ │     │p│    │   │
┌──▶│a│  ┌─┐  └─┘        │ ┌───────────┐
│  │└─┘ ││a│◀────────┼───┼─│-2C ✓ MATCH│
│  │       ┘             │ └───────────┘
│  └── -3C ───0C─── +3C ─┘
│  ┌─────────┐
│  │ -4C ✗ NO│
└──│  MATCH  │
   └─────────┘


   *Line bounded proximity*
   `withinLines = 4`
         ┌────────┐
   ┌─────┤fragment├─────┐
  +4L─ ─ ┴────────┘─ ─ ─│
   │                    │
   │              ┌─┐   │ ┌────────────┐
   │         ┌─┐  │a│◀──┼─│+1L ✓ MATCH │
   0L  ┌─┐   │p│  └─┘   │ ├────────────┤
   │   │a│◀──┴─┴────────┼─│-1L ✓ MATCH │
   │   └─┘              │ └────────────┘
   │                    │ ┌─────────┐
  -4L─ ─ ─ ─ ─ ─ ─ ─┌─┐─│ │-5L ✗ NO │
   │                │a│◀┼─│  MATCH  │
   └────────────────┴─┴─┘ └─────────┘


   *Line and column bounded proximity*
   `withinLines = 4`
   `withinColumns = 3`
         ┌────────┐
   ┌─────┤fragment├─────┐
  +4L   ┌└────────┴ ┐   │
   │            ┌─┐     │ ┌───────────────┐
   │    │       │a│◀┼───┼─│+2L/+1C ✓ MATCH│
   │         ┌─┐└─┘     │ └───────────────┘
   0L   │    │p│    │   │
   │         └─┘        │
   │    │           │   │ ┌────────────┐
  -4L    ─ ─ ─ ─ ─ ─┌─┐ │ │-5L/+3C ✗ NO│
   │                │a│◀┼─│   MATCH    │
   └───-3C────0L───+3C┴─┘ └────────────┘

关于复合规则的一些简短补充。

这是一项实验性功能！它可能会发生变化，所以别急着基于这项功能推出新的 B2B SaaS 功能。按扫描类型（git 与 dir）选择上下文的做法很有意思。我会持续关注。复合规则对 git 扫描可能用处不大，因为 gitleaks 只检查 git 历史中的新增内容。不过，扫描 git 历史中非新增内容的 `required` 规则或许会有用。哦，对了，这是 README，我现在闭嘴。

gitleaks:allow

如果你明知故犯地提交了一个会被 gitleaks 捕获的测试密钥，可以在该行添加 gitleaks:allow 注释，指示 gitleaks 忽略该密钥。例如：

class CustomClass:
    discord_client_secret = '8dyfuiRyq=vVc3RRr_edRk-fK__JItpZ'  #gitleaks:allow

.gitleaksignore

你可以在仓库根目录创建 .gitleaksignore 文件，以忽略特定发现项。在 v8.10.0 版本中，Gitleaks 为 Gitleaks 报告新增了 Fingerprint 值。每个泄漏项或发现项都有一个唯一标识密钥的指纹。将此指纹添加到 .gitleaksignore 文件中，即可忽略该特定密钥。请参阅 Gitleaks 的 .gitleaksignore 示例。注意：此功能为实验性功能，未来可能会发生变化。

解码

有时，密钥会以编码形式存在，仅靠正则表达式很难找到。 现在，你可以让 gitleaks 自动查找并解码编码文本。标志 --max-decode-depth 可启用此功能（默认值 "0" 表示默认禁用此功能）。

支持递归解码，因为解码后的文本也可能包含编码文本。标志 --max-decode-depth 用于设置递归上限。递归会在没有新的编码文本片段可供解码时停止，因此将最大深度设置得非常高，并不意味着它会执行那么多轮。它只会按需解码文本。总体而言，解码只会略微增加扫描时间。

编码文本的发现项与普通发现项在以下方面有所不同：

• 位置指向编码文本的边界
  • 如果规则在编码文本之外匹配，边界也会相应调整，将其纳入其中
• 匹配内容和密钥包含解码后的值
• 会添加两个标签：decoded:<encoding> 和 decode-depth:<depth>

当前支持的编码：

• percent - 任意可打印的 ASCII 百分号编码值
• hex - 长度至少为 32 个字符的任意可打印 ASCII 十六进制编码值
• base64 - 长度至少为 16 个字符的任意可打印 ASCII base64 编码值

归档文件扫描

有时，密钥会打包在 zip 文件或 tarball 等归档文件中， 因此难以发现。现在，你可以让 gitleaks 自动 提取并扫描归档文件的内容。--max-archive-depth 标志 可为 dir 和 git 扫描类型启用此功能。默认值 "0" 表示默认禁用此功能。

支持递归扫描，因为归档文件中也可能包含其他归档文件。 --max-archive-depth 标志用于设置递归限制。当 没有新的归档文件可供提取时，递归就会停止，因此将最大深度设置得很高， 只是允许递归到那么深。实际递归深度只会达到所需的程度。

归档文件中发现的密钥，其扫描结果会包含 归档文件内文件的路径。内部路径以 ! 分隔。

扫描结果示例（为简洁起见，已缩短）：

Finding:     DB_PASSWORD=8ae31cacf141669ddfb5da
...
File:        testdata/archives/nested.tar.gz!archives/files.tar!files/.env.prod
Line:        4
Commit:      6e6ee6596d337bb656496425fb98644eb62b4a82
...
Fingerprint: 6e6ee6596d337bb656496425fb98644eb62b4a82:testdata/archives/nested.tar.gz!archives/files.tar!files/.env.prod:generic-api-key:4
Link:        https://github.com/leaktk/gitleaks/blob/6e6ee6596d337bb656496425fb98644eb62b4a82/testdata/archives/nested.tar.gz

这表示在 archives/files.tar 中的 files/.env.prod. 第 4 行检测到了密钥， 而该文件位于 testdata/archives/nested.tar.gz 中。

当前支持的格式：

支持 mholt 的 archives 包 所支持的压缩格式 和归档格式。

报告

Gitleaks 内置支持多种报告格式：json、csv、junit 和 sarif。

如果这些格式都不符合你的需求，你可以使用 Go text/template .tmpl 文件和 --report-template 标志创建自定义报告格式。模板可以使用 Masterminds/sprig 模板库的扩展功能。

例如，以下模板会生成自定义的 JSON 输出：

# jsonextra.tmpl
[{{ $lastFinding := (sub (len . ) 1) }}
{{- range $i, $finding := . }}{{with $finding}}
    {
        "Description": {{ quote .Description }},
        "StartLine": {{ .StartLine }},
        "EndLine": {{ .EndLine }},
        "StartColumn": {{ .StartColumn }},
        "EndColumn": {{ .EndColumn }},
        "Line": {{ quote .Line }},
        "Match": {{ quote .Match }},
        "Secret": {{ quote .Secret }},
        "File": "{{ .File }}",
        "SymlinkFile": {{ quote .SymlinkFile }},
        "Commit": {{ quote .Commit }},
        "Entropy": {{ .Entropy }},
        "Author": {{ quote .Author }},
        "Email": {{ quote .Email }},
        "Date": {{ quote .Date }},
        "Message": {{ quote .Message }},
        "Tags": [{{ $lastTag := (sub (len .Tags ) 1) }}{{ range $j, $tag := .Tags }}{{ quote . }}{{ if ne $j $lastTag }},{{ end }}{{ end }}],
        "RuleID": {{ quote .RuleID }},
        "Fingerprint": {{ quote .Fingerprint }}
    }{{ if ne $i $lastFinding }},{{ end }}
{{- end}}{{ end }}
]

用法：

$ gitleaks dir ~/leaky-repo/ --report-path "report.json" --report-format template --report-template testdata/report/jsonextra.tmpl

赞助商

coderabbit.ai

退出代码

检测到泄漏时，你始终可以使用 --exit-code 标志设置退出代码。以下为默认退出代码：

0 - no leaks present
1 - leaks or error encountered
126 - unknown flag

加入 Discord！