py-test-quality

Python 测试质量分析

使用覆盖率分析和变异测试衡量测试覆盖率并验证测试套件的有效性。

目标

1. 全面衡量代码覆盖率
2. 找出未经测试的代码路径
3. 验证测试套件能否捕获错误（变异测试）
4. 通过高测试覆盖率实现安全重构
5. 跟踪覆盖率随时间的变化趋势

所需工具

添加到 [dependency-groups] dev："pytest"、"pytest-cov"、"mutmut"、"coverage" 可选："cosmic-ray"（高级变异测试）

• pytest-cov：代码覆盖率测量
• mutmut：变异测试——验证测试能否捕获错误
• cosmic-ray：高级变异测试（速度较慢）

权限：先运行 py-quality-setup，配置 .claude/settings.local.json，授予所有必需工具的权限。

覆盖率分析

测量覆盖率

# Run tests with coverage
pytest --cov=. --cov-report=term-missing
pytest --cov=. --cov-report=html  # Generate HTML report
pytest --cov=. --cov-report=term --cov-report=html  # Both

# Coverage with specific targets
pytest --cov=src --cov=lib tests/
pytest --cov=mypackage --cov-branch  # Include branch coverage

# Fail if coverage below threshold
pytest --cov=. --cov-fail-under=80

# Show only uncovered lines
pytest --cov=. --cov-report=term-missing:skip-covered

配置覆盖率

添加到 pyproject.toml（基础配置请参阅 py-quality-setup）：

[tool.coverage.run]
source = ["src"]  # Adjust to your source directory
omit = [
    "*/tests/*",
    "*/test_*.py",
    "*/__pycache__/*",
    "*/venv/*",
    "*/.venv/*",
]
branch = true  # Enable branch coverage

[tool.coverage.report]
precision = 2
show_missing = true
skip_covered = false
exclude_lines = [
    "pragma: no cover",
    "def __repr__",
    "raise AssertionError",
    "raise NotImplementedError",
    "if __name__ == .__main__.:",
    "if TYPE_CHECKING:",
    "@abstractmethod",
]

[tool.coverage.html]
directory = "htmlcov"

解读覆盖率报告

Name                 Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------------
src/auth.py            45      5     12      2    87%   23-25, 67
src/database.py        89      0     18      0   100%
src/handlers.py       123     35     28     12    68%   45-78, 99-110
----------------------------------------------------------------
TOTAL                 257     40     58     14    82%

关键指标：

• Stmts：语句总数
• Miss：未覆盖的语句数
• Branch：分支总数（if/else 等）
• BrPart：部分覆盖的分支（一条路径已测试，另一条未测试）
• Cover：覆盖率百分比
• Missing：未覆盖的行号

覆盖率目标：

• ≥80%：最低可接受水平
• ≥90%：覆盖率良好
• 100%：理想水平（并非所有代码都能实际达到）

变异测试

变异测试通过引入细微更改（变异）并检查测试是否失败，来验证测试能否真正捕获错误。

运行变异测试

# Using mutmut (easier, faster)
mutmut run                    # Run all mutations
mutmut run --paths-to-mutate=src/  # Specific directory
mutmut results                # Show summary
mutmut show <mutation-id>     # Show specific mutation
mutmut apply <mutation-id>    # Apply mutation to see code change

# Common workflow
mutmut run
mutmut results               # Shows: survived, killed, timeout
mutmut show 1                # Examine first surviving mutation

配置 Mutmut

添加到 setup.cfg 或 pyproject.toml：

[tool.mutmut]
paths_to_mutate = "src/"
backup = false
runner = "pytest -x --tb=short"
tests_dir = "tests/"

解读变异测试结果

Legend for output:
🎉 Killed mutants: Tests caught the bug (good!)
⏰ Timeout: Mutation created infinite loop (acceptable)
🤔 Suspicious: Needs investigation
🙁 Survived: Bug not caught by tests (bad!)

变异得分：killed / (killed + survived) * 100%

变异得分目标：

• ≥75%：测试质量良好
• ≥85%：测试质量优秀
• 100%：完美（很少能达到）

处理存活变异

# 1. Identify surviving mutations
mutmut results

# 2. Show specific mutation
mutmut show 5

# Example output:
# src/auth.py:23
# -    if user.age >= 18:
# +    if user.age > 18:

# 3. Write test to kill this mutation
def test_user_exactly_18_is_adult():
    user = User(age=18)
    assert is_adult(user) is True  # This would fail with > instead of >=

# 4. Re-run mutmut
mutmut run

# 5. Verify mutation now killed
mutmut results

以覆盖率指导重构

黄金法则：只重构经过充分测试的代码。如果覆盖率较低，先编写测试。

工作流

1. Run: pytest --cov=. --cov-report=html --cov-fail-under=80
2. If coverage < 80%:
   a. Open htmlcov/index.html
   b. Identify modules with low coverage (red/yellow)
   c. Write tests to increase coverage to ≥80%
   d. Re-run coverage to verify
3. Run: mutmut run (optional but recommended)
4. If mutation score < 75%:
   a. Review surviving mutations
   b. Write tests to kill mutations
   c. Re-run mutmut
5. NOW safe to refactor:
   a. Apply refactoring patterns
   b. Re-run: pytest --cov=. (ensure coverage maintained)
   c. Re-run: mutmut run (ensure mutation score maintained)

与重构集成

重构前检查清单

# 1. Measure baseline coverage
pytest --cov=src/module_to_refactor.py --cov-report=term-missing

# 2. If coverage < 80%, write tests first
# ... write tests ...

# 3. Verify tests are effective
mutmut run --paths-to-mutate=src/module_to_refactor.py

# 4. Now proceed with refactoring
# ... refactor ...

# 5. Verify coverage maintained
pytest --cov=src/module_to_refactor.py --cov-fail-under=80

# 6. Verify tests still effective
mutmut run --paths-to-mutate=src/module_to_refactor.py

CI/CD 集成

在 CI 中添加覆盖率强制检查：

# .github/workflows/test.yml
- name: Run tests with coverage
  run: |
    pytest --cov=. --cov-report=term --cov-report=xml --cov-fail-under=80

- name: Upload coverage to Codecov
  uses: codecov/codecov-action@v3
  with:
    file: ./coverage.xml

验证检查清单

• pytest --cov=. --cov-fail-under=80 通过
• 已查看覆盖率报告（htmlcov/index.html）
• 关键路径均有测试覆盖
• 已对关键模块运行变异测试（mutmut run）
• 关键代码的变异得分 ≥75%
• pyproject.toml 中已配置覆盖率

示例

示例：重构前提高覆盖率

1. Run: pytest --cov=src/handlers.py --cov-report=html
2. Coverage: 55% (too low to refactor safely)
3. Open htmlcov/handlers_py.html
4. Lines 45-78 not covered (error handling paths)
5. Write tests for error cases:
   - test_handler_invalid_input()
   - test_handler_database_error()
   - test_handler_missing_params()
6. Re-run: pytest --cov=src/handlers.py
7. Coverage now: 85% (safe to refactor)
8. Proceed with complexity reduction refactoring
9. After refactoring: pytest --cov=src/handlers.py --cov-fail-under=85
10. Verify coverage maintained at 85%

示例：使用变异测试改进测试

1. Run: mutmut run --paths-to-mutate=src/auth.py
2. Results: 15 killed, 3 survived (83% mutation score)
3. Run: mutmut show 5
4. Mutation: Changed >= to > in age check
5. Realize: Missing boundary test for age exactly 18
6. Write: test_user_exactly_18_is_adult()
7. Run: mutmut run --paths-to-mutate=src/auth.py
8. Results: 16 killed, 2 survived (89% mutation score)
9. Repeat for remaining survivors
10. Final: 18 killed, 0 survived (100% mutation score)

示例：以覆盖率指导重构的会话

1. Target: Refactor src/payment.py (complexity D, 150 lines)
2. Check coverage: pytest --cov=src/payment.py --cov-report=term-missing
3. Coverage: 92% (good! safe to refactor)
4. Check test quality: mutmut run --paths-to-mutate=src/payment.py
5. Mutation score: 78% (acceptable)
6. Proceed with refactoring:
   - Extract 4 smaller functions
   - Reduce complexity from D to A/B
7. After refactoring:
   - pytest --cov=src/payment.py --cov-fail-under=92 ✓
   - mutmut run --paths-to-mutate=src/payment.py
   - Mutation score: 80% (improved!)
8. Commit changes with confidence

示例：在 CI 中设置覆盖率跟踪

1. Add to pyproject.toml:
   [tool.coverage.run]
   source = ["src"]
   branch = true

2. Create .github/workflows/test.yml:
   - pytest --cov=. --cov-fail-under=80

3. First run fails: Coverage 67%
4. Write tests to reach 80%
5. CI now passes
6. All future PRs must maintain 80% coverage
7. Consider increasing threshold as coverage improves

相关技能

• 前置条件：py-quality-setup（在 pyproject.toml 中配置 pytest-cov）
• 支持技能：py-security、py-code-health、py-complexity（借助测试覆盖率安全重构）
• 强制执行：py-git-hooks（将覆盖率检查添加到 CI）