fable-verify

Fable Verify

Try to prove the implementation wrong, then report only the claims that survive fresh evidence.

Mission

Verification is not "run the test suite." It is coverage of changed risk with evidence that is both relevant and fresh.

A green command proves only what it actually exercised. Build success does not prove runtime behavior. Security success does not prove functional correctness. Unit success does not prove a packaging or integration path. Old evidence does not prove a newer mutation.

Activate When

• implementation or repair is ready for independent proof;
• a completion claim needs fresh evidence;
• a review/release gate requires test/build/runtime evidence;
• stale evidence must be refreshed after mutation;
• the suspected regression surface spans more than the focused TDD test.

Do Not Activate When

• writing the implementation (fable-execute/fable-tdd);
• deciding architecture (fable-plan);
• reviewing design/maintainability from the diff (fable-review);
• diagnosing repeated confusing failures (fable-recover).

Risk Classification

Map each changed surface to the evidence capable of falsifying it.

Changed surface	Typical evidence
pure behavior	focused unit/property + affected suite
cross-module contract	integration/contract test
public CLI/API	invocation/smoke + contract tests
build/export/package	build + package/clean-install smoke
persistence/migration	integration + migration/compatibility fixtures
async/concurrency	deterministic ordering/stress supplement
config/feature flag	tests under relevant config branches
browser/UI	component/integration/E2E as appropriate
security boundary	security-specific checks plus functional evidence
performance-sensitive path	targeted measurement when requirement exists

Verification Protocol

Stage 1 — Read the diff and execution packet

Do not choose commands from habit alone. Identify:

• behavior changed;
• files/contracts touched;
• tests added/changed;
• generated/package/config surfaces;
• residual risks from implementation;
• current mutation generation.

Stage 2 — Build a verification matrix

For each material risk, write:

• claim;
• failure mode;
• evidence/command that would catch it;
• whether evidence must be narrow, integration, runtime, build, E2E, security, or package-level.

Remove duplicate checks that prove the same narrow fact; add missing checks for untested surfaces.

Stage 3 — Run narrow, causal checks first

Start with the focused changed behavior and affected tests. Fast local evidence helps distinguish implementation failure from unrelated suite noise.

Stage 4 — Expand according to blast radius

Run broader typecheck/build/test/integration/E2E/package gates only where the change can affect them or where repository release policy requires them.

Do not skip required project-wide gates merely because narrow tests pass.

Stage 5 — Adversarial falsification

Actively probe the most plausible regression surfaces:

• boundary/empty/invalid inputs;
• error propagation;
• retries/idempotency;
• compatibility old/new formats;
• async ordering/concurrency;
• cleanup/resource lifecycle;
• package/export/runtime entry point;
• configuration branch.

Prefer tests/commands that can fail meaningfully over speculative prose.

Stage 6 — Detect nondeterminism and stale execution

If identical commands alternate outcomes, stop counting passes. Capture the variability and route to recovery.

If results do not reflect known source changes, check source-vs-build, cache, branch, env, process, and artifact path before accepting output.

Stage 7 — Record typed, fresh evidence

Evidence should include:

• kind (test, build, runtime, review, security where applicable);
• exact command/probe;
• exit/result and relevant counts;
• mutation generation/artifact SHA when available;
• scope/claim the evidence proves.

Stage 8 — State the verdict narrowly

Use:

• PASS: every required claim has fresh relevant evidence;
• FAIL: at least one required claim is falsified;
• INCOMPLETE: required evidence cannot be obtained or does not cover the risk.

Never convert INCOMPLETE to PASS because the remaining check is inconvenient.

Decision Rules

• Evidence generation older than current relevant mutation → stale, rerun.
• Test passes but does not execute changed path → irrelevant evidence, not PASS.
• Security-only evidence for functional bug → require functional proof.
• Build/typecheck-only evidence for runtime behavior → require runtime/test proof.
• Focused test green but integration contract changed → run integration/contract evidence.
• Package/export change → inspect/package/smoke the distributed artifact, not only source tests.
• Flaky alternating outcomes → route to fable-recover; do not cherry-pick a green run.
• New verification failure caused by a bounded obvious implementation defect → return one repair card; repeated/ambiguous failure → recover.
• If a command is unavailable in current environment, mark evidence incomplete and name the external gate instead of fabricating a result.

Invariants

• Verification is read-only with respect to product behavior; any repair starts a new mutation/evidence cycle.
• Every completion claim maps to fresh evidence.
• Evidence kinds are not interchangeable.
• The final verdict covers the current mutation generation/artifact.
• Required failures/warnings are not hidden by truncating output or rerunning until green.

Failure Taxonomy

Relevant test failure

Changed behavior is falsified. Return bounded repair or recover depending on clarity/repetition.

Unrelated suite failure

Prove it is unrelated before excluding it; do not dismiss merely because it predates the change.

Stale evidence/artifact

Output corresponds to older mutation/build. Refresh build/path/env before evaluating implementation.

Nondeterministic verification

Same state produces inconsistent result. Diagnose shared state/timing/environment before verdict.

Coverage gap

Available checks do not exercise material changed risk. Add/locate suitable evidence or mark incomplete.

Environment limitation

Required external service/browser/platform/release environment unavailable. Report exact missing evidence; do not emulate a pass.

Anti-Patterns

• tests passed as the entire verification report;
• running only tests the implementer just wrote;
• rerunning flaky tests until green;
• trusting old screenshots/logs after code changed;
• treating typecheck/build/security as functional proof;
• ignoring packaging/export paths;
• broad full-suite runs with no affected-risk reasoning;
• claiming pass when an external required gate was never run.

Verification Packet

Current mutation/artifact:
Changed risks:
Verification matrix:
- claim → evidence → result
Adversarial probes:
Stale/nondeterministic evidence detected:
Required external evidence not run:
Verdict: PASS | FAIL | INCOMPLETE
Repair/recovery recommendation:

Completion Criteria

Verification completes when:

• changed risks have relevant checks;
• required project gates are executed or explicitly marked unavailable;
• evidence is fresh for current mutation/artifact;
• likely edge/failure surfaces were actively probed;
• nondeterminism/stale output is not hidden;
• verdict is no broader than the evidence supports.

Progressive Resources

• Deep matrix guide: references/verification-matrix-and-evidence-strength.md
• Existing falsification heuristics: references/falsification-heuristics.md
• Evidence recording: references/evidence-recording.md
• Example: examples/falsification-session.md