testing-best-practices

Testing Best Practices

This skill has four modes:

1. Write — add tests or test-first bug fixes/features.
2. Assess — review an existing suite for quality, not just coverage.
3. Upgrade — strengthen weak/flaky/over-mocked tests.
4. Detect — find skipped, sabotaged, fake-coverage, or gap-prone tests.

First 90 seconds

Before writing, assessing, or changing tests:

1. Detect language/framework from repo files and adjacent tests.
2. Read nearby tests for naming, fixtures, builders, assertion style, and runner commands.
3. Find the nearest validation command before broad full-suite commands.
4. Identify the risk boundary: pure logic, internal component boundary, external API, UI/user journey, security boundary, hostile-input parser, or type/schema invariant.
5. Classify the change and asset: behavior change vs. structure/tidying, reusable library vs. throwaway probe, customer-facing rule vs. internal helper.
6. Choose the smallest useful test tier from references/test-types.md.
7. Load only relevant references from the matrix below.

If the language is unsupported by a dedicated reference, follow project conventions plus the generic principles here.

Reference matrix

Always consider:

• Test type selection → references/test-types.md
• Anti-pattern detection → references/antipatterns.md

Language/framework references:

• Python / pytest / Hypothesis → references/python.md
• TypeScript/JavaScript / Vitest/Jest/fast-check/Playwright → references/typescript.md
• Go / testing / testing.F / Rapid → references/go.md
• Rust → references/rust.md

For property-based testing or coverage-guided fuzzing, also load the matching language reference: collection, discovery, and replay differ by engine. If the selected engine is not documented here, follow its project documentation rather than transferring another engine's behavior.

Topical references by trigger:

• Writing/reviewing property tests, choosing a property, generator design, stateful/model-based testing, seeds and replay → references/property-based-testing.md
• Coverage-guided targets or campaigns for exposed/custom parsers, native or unsafe decoders, crash-prone boundaries, or security tooling whose crash disables a check → references/fuzzing.md
• Legacy/refactor safety → references/characterization-testing.md
• Reimplementation, port, multi-language SDK, custom data structure with no reference, or approximate/probabilistic/ANN/ranking output → references/differential-testing.md
• Complex outputs, snapshots, transformation pipelines, save/load or migration roundtrips → references/golden-file-testing.md
• Time, timers, schedules, sleeps, flaky time tests, background threads/async work tests can only reach by sleeping → references/deterministic-time.md
• External APIs, recorded real responses, mock drift → references/vcr-cassettes.md
• CLI/plugin/docs registry sync, or verification claims in docs and agent instructions → references/doc-sync-testing.md
• Mutation testing, mutation scores/survivors, recurring mutation lanes, or high coverage but escaping bugs → references/mutation-testing.md. For recurrence, no completed target-CI baseline or demonstrated fault-class bite means no schedule; never copy an absolute score floor.
• Finite state spaces or expensive cross-products → references/exhaustive-testing.md
• Large configuration/feature/runtime matrices, pairwise replacement, or interacting factors → references/combinatorial-testing.md. Preserve fixed regressions and exact registry enrollment; use named variable-strength groups and shadow evidence before deletion.
• Arithmetic/domain operators/laws → references/mathematical-properties.md
• Fixtures/builders/assertion helpers → references/test-data-builders.md
• Same invariant checked across layers, type-vs-test decisions, invalid states → references/correctness-by-construction.md
• CI/hooks/test-config changes, gated or scheduled tiers, "is this check running and can it fail?", deploy verification → references/gate-integrity.md

Core principles

Test behavior through public contracts

Good tests describe observable behavior: outputs, state transitions, side effects, rendered UI, persisted data, API contracts, or invariants. Avoid tests that break under behavior-preserving refactors: private methods, incidental call order, exact SQL strings, or broad mock choreography.

When a user brings examples, tables, bug reports, spreadsheet rows, or support cases, treat them as communication artifacts first and assertions second. Preserve the business language, then run the examples against the narrowest honest seam: usually the domain/service/API contract, not a full UI path for every business-rule row. UI/E2E tests should prove the wiring and golden path; rule tables should exercise the domain layer directly.

Calibrate test investment to lifetime and change kind

Do not give every artifact the same test plan. Reusable libraries, parsers, payment/auth/security boundaries, migrations, and code that will be refactored deserve stronger regression/property/contract coverage. Throwaway probes, one-off research scripts, or generated exploration may need only a smoke check or no tests if the user accepts that lifecycle.

Separate behavior changes from structure changes. Behavior changes need tests that can fail for the new/changed behavior. Pure tidying/refactoring should ride an existing green suite; if no such suite exists, add characterization tests for the behavior you need to preserve before refactoring.

For a new service with no pipeline, start with a walking skeleton: the thinnest build/deploy/test slice. Track unfinished acceptance tests as visible in-progress items tied to a story/issue, not silent skips. Keep unit tests green; do not commit failing unit tests as proof.

Use red-green-refactor when feasible

For bug fixes and new behavior, default to:

1. Red — add a focused test that fails for the current bug/missing behavior.
2. Green — implement the smallest change that passes.
3. Refactor — clean up while tests stay green.

The evidence matters. Separate red evidence (exact command and failing output before the fix) from green evidence (passing command/result after the fix). Only claim completed TDD/red-green-refactor when both were observed in order. If only a passing/green run is available, say tests were added and now pass, but the red phase was not observed/unverified; do not backfill a TDD claim from a green-only log. For exploratory prototypes, generated code, or user-scoped “tests only” tasks, stay within scope and state the deviation.

When practicing TDD, keep a visible test list. Park new edge-case ideas there instead of chasing them mid-cycle. If stuck, downshift: write the assertion first, extract a smaller child test, use a learning test for third-party APIs, or use a crash-test-dummy fake to trigger hard error paths. Choose Fake It, Triangulate, or Obvious Implementation according to uncertainty; never leave fake constants as the final behavior.

Quality beats coverage

Coverage shows what ran; it does not prove bugs would be caught. Prefer branch coverage over line coverage and treat coverage as a map for finding untested paths, not a quality gate by itself.

Assertion count is a heuristic, not a law; never turn it into a quota. Example-based behavior tests often need multiple meaningful assertions to verify structure, state, and negative cases. But a property test, table row, or exception test may have one excellent oracle. Flag weak sole assertions such as “not empty,” toBeDefined(), toBeTruthy(), Assert.IsNotNull(result), or logging without assertions.

For sanitizers, validators, filters, auth/security checks, and transformations, verify both directions where applicable: dangerous/invalid content is rejected or removed, and safe/valid content is preserved.

Assert the fields the behavior under test is about. Full-object/whole-structure equality implicitly asserts every unrelated field, so the test breaks on unrelated changes and joins a change-detector treadmill (update the literal each time a field is added). Reserve whole-state comparison for tests where breadth is the contract — golden files and save/load roundtrips (references/golden-file-testing.md). Prefer assertion forms whose failure message alone can start the debugging (matchers/fluent asserts that print expected vs. actual, not bare booleans); when many tests keep breaking for innocent reasons like iteration order, fix the assertion vocabulary (order-insensitive/structural matchers), not each test. For coverage, gate new/changed code rather than one repo-wide number, and read what is not covered as the review signal.

Every custom oracle, linter, validator, fixer, or eval grader needs a known-good input that passes and a known-bad input that fails; keep them small, and don't rerun an expensive checker for every bad input. Goldens, approvals, or eval cases written by the same actor in the same change as the code are characterizations, not independent oracles.

Prefer real behavior over mocks

Prefer, in order:

1. Real in-memory/local objects, temp dirs, in-memory databases, real parsers.
2. Local emulation of the real runtime (workerd/vitest-pool-workers, Pyodide in Node, wrangler dev, SQLite loaded from migrations).
3. Purpose-built fakes that implement the same interface and can record history.
4. Deterministic stubs for controlled edge cases.
5. Framework mocks as a last resort.

Take a double's expected values from the real service (recorded or locally emulated), never from the mock author's belief; when a real engine can run in-process, delete the mock.

Prefer visible state over call choreography: assert the saved row, emitted event, response body, file on disk, or a small logging fake's recorded effects. Do not mock values; construct them. Mock roles you own, not third-party libraries directly; wrap provider SDKs behind an owned interface and add a contract/VCR check for the real provider shape. For interaction tests, allow queries and expect commands: getters can be called freely, side-effecting commands are where expectations earn their keep.

Use properties for broad input spaces

Use property-based testing when behavior spans a broader input or operation space than examples can cover economically. Choose a property that states meaningful behavior—roundtrip, idempotence, conservation, metamorphic relation, algebraic law, or agreement with a small model—not merely that execution did not crash.

Match the generator to the behavior being tested: arbitrary input for totality and documented errors, specification-valid input for semantic branches, stateful traces for lifecycle behavior, and corpus mutation for coverage-guided discovery. Use an oracle independent of the implementation. Property tools construct values and shrink failures; coverage-guided fuzzers mutate a corpus toward new control flow. Do not use the terms or tools interchangeably.

For small finite spaces, prefer exhaustive generation over sampling, after checking per-cell cost and whether a cheaper layer decides the property (references/exhaustive-testing.md). See references/property-based-testing.md for generator, oracle, execution, and replay guidance.

Test error-handling paths, not just invalid input

Most critical production failures are shallow: they live in error-handling code that only runs when a dependency already failed — a disk-write error, a dropped connection, a timeout, a partial response. Empirical studies of catastrophic distributed-systems failures find the majority were reachable by simple tests that exercised those paths. So "sad path" means more than rejecting bad arguments: inject the downstream failure (a fake/stub that raises, an injected I/O error, a side_effect exception) and assert the system degrades correctly.

Assert the failure behavior the code actually has, not one you wish it had. Read the spec or observe the code to learn what a failure should do — propagate, retry, wrap, fall back, roll back — and test that. Do not invent a retry budget, a custom error type, or rollback semantics the contract never promised; that tests a fictional contract and is scope creep. If the intended failure behavior is unspecified, characterize what the code does today and flag the gap as a follow-up rather than designing new production behavior inside a test.

Push internal invariants into types/schemas/contracts

Types and tests answer different questions:

• Types/schemas/contracts: “Can this invalid state exist?”
• Tests: “Does observable behavior match the spec?”

When an invariant is repeated across internal layers, consider lifting it to a boundary parser, smart constructor, schema, sealed enum, state machine, or database constraint. Before deleting checks or their tests, verify all of this:

• the context is internal and non-adversarial,
• the invariant is actually enforced by the type/schema/constructor,
• boundary tests cover hostile or malformed input,
• security/auth/external-failure layers defend different failure modes,
• production type changes are in scope or explicitly approved.

If user scope says “tests only” or “do not change production code,” add tests for the current API and list type/schema tightening as a follow-up.

Test concurrent code under a race detector, and pin the concurrency contract

For code meant to be used by multiple goroutines/threads, sequential tests prove almost nothing. Drive the shared state from many workers released together (a start barrier maximizes contention) and run under a race detector or thread sanitizer (go test -race, ThreadSanitizer / -fsanitize=thread, repeated runs via -count). Treat a green concurrent test as weaker evidence than a sequential one — races are probabilistic.

Assert the invariant the API actually promises — no lost updates, a monotonic/linearizable count, compute-at-most-once — instead of logging whatever count you happened to observe and moving on. If the implementation can violate that contract (e.g. it computes outside the lock, so two callers double-compute), that is a defect to surface with a failing test or an explicit flag, not a t.Logf that quietly bakes the race into the expected behavior.

Mode workflows

Write mode

1. Read adjacent tests/config and load the relevant language reference.
2. Choose test tier from references/test-types.md.
3. For bugs/new behavior, add the failing regression test first when feasible.
4. Convert user/customer examples into readable test cases without forcing every example through the UI.
5. Use builders/factories/helpers where setup noise hides intent; test custom helpers when they become a mini-DSL.
6. Prefer user-facing/public interfaces over internals.
7. Pin nondeterminism: time, randomness, network, filesystem, order.
8. Choose test values deliberately: include distinct, non-default values so at least one case would expose a dropped, defaulted, or swapped argument. Still cover zero, empty, and equal-value cases when they are boundaries or part of the contract. Derive expected results independently of the SUT: literals are clearest for simple examples, while properties or a genuinely independent reference model fit broader cases. Do not recompute the expectation with the SUT's own logic or constants, because the test can share its bug and stay green.
9. Run the nearest tests, then broader checks when practical.

For transformations or complex generated output, use golden files with explicit review discipline; see references/golden-file-testing.md. For external APIs, prefer recorded real fixtures/cassettes or contract checks over live CI calls; see references/vcr-cassettes.md.

Assess mode

Report evidence by severity and include positive observations. For generative tests, load the relevant language and topical references before recommending engine-specific changes. Check:

0. Is it running, and can it fail? For each tier: the job that runs it, its last result on the default branch, whether gated tests are reachable, and whether each gate can go red (|| true, continue-on-error, baseline updaters, skip-green, empty targets). A dead gate is the first finding. See references/gate-integrity.md.
1. Sabotage / false confidence: skipped/focused tests, no assertions, logging-not-asserting, commented-out assertions, always-true assertions.
2. Oracle strength: weak sole assertions, missing negative/error/state/structural assertions, tautologies. For property tests, a bare never-crashes oracle is weak when a semantic invariant is available.
3. Mock-reality drift: hardcoded mocks that would not notice real API/schema changes.
4. Tier integrity: unit tests hitting live network, integration tests mocking every boundary they claim to exercise, E2E tests that mock the system under test or drive it through a test facade, PR suites that target production, and whether the artifact under test is the artifact deployed.
5. Determinism: sleeps, wall-clock time, unseeded random, order dependence, global state leaks.
6. Coverage quality: branch coverage; for high-coverage suites with escaping bugs, seed a fault on the critical path and see whether any test fails, cheapest first (references/gate-integrity.md §5). A recurring mutation lane is not a default recommendation.
7. Invariant placement: repeated internal validation that should be a type/schema/contract.
8. Lifecycle fit: throwaway probes over-tested, reusable assets under-tested, or structure-only changes getting behavior-test theater.
9. Example quality: business examples buried in UI scripts, unreadable fixtures, or helpers that form a DSL but have no tests of their own.
10. Generative evidence: the configured runner reaches the production path, the generator reaches the claimed domain, the oracle is independent, and seed replay is not mislabeled as active discovery. Verify the exact CI configuration when multiple runners, projects, filters, or duplicated target lists make reachability uncertain; add a permanent drift guard only for recurring risk.
11. Test-code readability (DAMP): logic in test bodies (loops/branches/computed expectations), expectations derived from the SUT's own constants, shared fixtures mutated far from the assertions that depend on them. See references/antipatterns.md.
12. Suite shape: count tests per tier; name the inverted pyramid/ice-cream cone (mostly E2E over a thin unit base) and the hourglass (unit + E2E with an empty integration middle). Recommend the smallest-tier home for each E2E case that has one — business rules move below the UI — while keeping a small golden-path/critical-journey E2E core.

Upgrade mode

Prioritize by risk:

1. P0: security/auth/injection/sanitizer tests with weak or missing assertions.
2. P1: tests that only assert not-null/not-empty/truthy.
3. P2: flaky tests: sleeps, live network, wall-clock, order coupling. Before patching waits, ask whether a smaller SUT can host the test — size (dependencies, processes, RAM footprint) predicts flakiness better than tool choice — and treat auto-retry/quarantine as masking debt, not a fix.
4. P3: mock-heavy “integration” tests and implementation-coupled mocks.
5. P4: fixture noise, duplication, unclear names, missing builders.

Common upgrades:

• Replace sleep() with virtual time, injected clocks, or condition-based waits.
• Replace live API calls with recorded cassettes or committed fixtures.
• Replace broad mocks with fakes, contract tests, or assertions on public behavior.
• Replace repeated object literals with builders that keep behavior-specific fields explicit.

Never silently downgrade a tier: list removed or replaced tests by tier, and keep real-engine tests (SQLite, workerd, browser) unless an equal or stronger tier replaces them. A lesson or postmortem counts only when a check enforces it.

Detect mode

Use concrete search signals from references/antipatterns.md:

• @skip, skip, skipif without real condition, test.only, fit, xit, xdescribe.
• print, console.log, t.Log, Debug.WriteLine used instead of assertions.
• Assertion-free tests or only truthy/not-empty/not-null checks.
• Property tests with no independent oracle, generators filtered into one easy path, or small finite domains sampled when they can be enumerated.
• sleep, waitForTimeout, Thread.Sleep, Task.Delay for synchronization.
• Try/catch swallowing exceptions.
• Mock return values identical to assertions.
• Global state/env/registry mutations without cleanup.
• Snapshot/golden updates without diff review.
• CI steps with || true, continue-on-error, scan --baseline, or ::warning:: skips; coverage or mutation thresholds that never run.
• Mutation run as a program: calendar schedules over unchanged code, round-number floors, internals exported or pinned only to kill survivors.
• Assertions only inside loops over possibly-empty results; violations == [] with no non-empty check; tests that import nothing from the code under test.
• Source, CSS, or workflow text read and asserted as strings; helper-level retry loops; raised global timeouts or isCI ? a : b budgets.

Validation loop

After writing or changing tests:

1. Run the nearest relevant test command.
2. If bug-fix TDD was intended, report red evidence separately from green evidence; if the pre-fix failing run was not observed, call the red phase unverified instead of claiming completed TDD.
3. Scan the changed tests for weak sole assertions, skips/focus markers, logging-not-asserting, sleeps, live network, and implementation-detail coupling. Also check: expected results come from an oracle independent of the SUT, at least one test value would expose dropped/defaulted/swapped inputs, relevant boundary values are still covered, and assertions are narrow to the behavior's fields.
4. For security/transformation tests, verify both rejection/removal and preservation.
5. For invariant work, verify both tactics where relevant: property/invariant proof and invalid-state reachability.
6. For property/fuzz tests, confirm the claimed input domain, independent oracle, engine-native replay, and—where configuration makes it uncertain—collection or target selection with the exact CI command.
7. Decide pass/fail from exit status, not output text. Run the exact CI entrypoint rather than a sub-mode, and compile tag-gated files you touched with their tags.
8. When repairing CI, keep "update a stale selector" separate from "weaken an assertion or raise a timeout", and justify every weakening. Call a failure pre-existing only after reproducing it on a clean checkout of the base.
9. If validation is blocked, report the exact command, failure, and next-best check. Never claim tests passed without running them; say which environment a green claim came from, and check the pushed commit's CI when there is one.

Final report contract

End testing work with:

Tests changed/assessed:
Behavior covered:
Commands run:
Results:
CI status (ran / red / not running):
TDD evidence (if claimed):
Gaps / risks:
Follow-ups:

Keep it concise, but make validation and remaining risk auditable.