openclaw-testing

OpenClaw Testing

Proof policy

• PR CI = default broad proof. Change builds + focused tests pass → open ready PR. CI runs broad suites; never delay PR for pre-PR suite runs.

• Before PR, prove narrowly: touched tests with pnpm test <file> --maxWorkers=1, targeted typecheck, lint, format. Prefer local proof for trusted, bounded work when dependencies and current host capacity are qualified. Local execution is optional: a focused command can still load a large compiler graph or expand to broad lanes.

• Choose remote proof deliberately. Use PR CI for broad proof; Crabbox/Testbox may cover expensive checks, a busy or unqualified local host, another OS/device, live providers/channels, long E2E, or heavy benchmarks. Acquire only when the command is ready and capacity is granted promptly. If queued, stop only the task-owned allocation; use PR CI or a safe local scope. Keep uncovered platform-specific proof pending. Never force heavy work onto a busy machine or chain providers. Use the smallest profile proven for the selected workload.

• Never rerun or re-push just for green. Fix real failures; read available CI failure classification before attributing red.

• Prove changed contract, complete required gates, finish. Broaden/repeat only for changed inputs, failures, or named unresolved risk.

• Follow scoped AGENTS.md; authoring/reviewing tests: use boundary guide and test-audit.

Select the proof

• Select proof for touched contract; reuse valid proof (rerun for changed inputs or missing coverage); finish workflow's required gates within user/host limits; report unrun checks and gaps. Prove user-visible behavior through real flow when feasible; external APIs need live contract proof; isolated mock-Gateway harness = valid channel boundary proof. Docs-only: docs sanity and git diff --check.

• Runtime defect: reproduce real entry point; rerun it + relevant siblings.

• Trusted source: pnpm changed:lanes --json, focused tests + targeted typecheck/lint/format; pnpm check:changed = optional smart local check.

• Public SDK/plugin contract: representative consumers, not all-plugin sweep.

• Build output, lazy imports, package boundaries: include pnpm build.

• Workflow: git diff --check + pnpm check:workflows.

• Docs only: relevant docs/link/format checks + git diff --check; no runtime tests.

• Read on demand: local commands, Crabbox, OpenClaw remote setup, package/Docker proof, release CI, plugin release matrix, Docker authoring, channel skill/ Control UI E2E.

• An isolated mock-Gateway harness is valid channel boundary proof; state live gaps. Prove user-visible behavior through the real flow when feasible; external APIs require live contract proof. Read the scoped channel/platform guide for live-channel or native-app work. Telegram claims require Test Server userbot proof with Convex-leased credentials. Platform claims require real-device evidence; macOS permission proof requires a stable signed app.

• Release proof: preserve candidate/Tooling SHAs; never grants publication authority.

Source and state boundaries

• Never execute untrusted contributor/fork code, wrappers, config locally. Use secretless fork CI or sanitized direct AWS through Crabbox; never credential-hydrated Testbox. Credentialed execution: maintainer approval after review required; never hydrate untrusted lease.
• Isolated state + free port. Never restart, edit, or test operator Gateway/real data without explicit per-task approval. Preserve unrelated processes and shared dependency installs.
• Concurrent test/check commands: separate OPENCLAW_VITEST_FS_MODULE_CACHE_PATH values or run serially; checks can run Vitest too. Repository wrappers only, never raw Vitest. Prepared linked worktrees: node scripts/run-vitest.mjs <file> --maxWorkers=1 avoids pnpm reconciliation; PR CI runs the broad checks.

CI failures

• PR CI fails → pnpm test:ci-failed <pr>; read pre-existing/new/unknown classification.
• Fix new failures; investigate unknown → pnpm test:ci-failed <pr> --run until selected tests pass locally → push once.
• --run enables OPENCLAW_NODE_TEST_PLAN_CONTINUE_ON_FAILURE=1: collect ordinary failures across selected configs, while infrastructure failures still stop; use the same opt-in for manual focused reruns.
• Pre-existing failures: failure-table landing rule. Matching signatures ≠ job-name-only attribution.
• Never gh run rerun to chase green; never local full suite for CI repair.
• Bind diagnosis to exact SHA + job. Fetch failed logs once; prefer exact run/job state over stale PR rollups.
• Distinguish product, harness, infrastructure, credentials, superseded-run cancellation. Snapshots pass macOS/fail CI: reproduce Linux/Node bytes before regenerating.

Baseline and expected-failure exceptions

Narrow baseline, snapshot, ignore, and expected-failure exceptions are implementer decisions only when each entry is verified against source and the PR records its reason (for example, tool-owned metadata in an upgrade snapshot while user data stays exact). Never mask a real regression or user-data change. Blanket or unverified exceptions need approval. Exact shrink-only ratchet updates are maintenance.

Test failure policy

• Failures = defects. Bounded reproduction attempt (same shard order first); identify owner; fix product, fixture, shared-state, ordering bugs.
• Proven repair: add regression; cite another owner's fix when applicable.
• Safe fix cannot be established/completed: record original failure, attempts, evidence, uncertainty in PR; continue under normal CI/review gates.
• Unresolved failure alone neither blocks landing nor requires extra approval. Passing replay ≠ proven fix.
• Never hide failures with retries, longer timeouts, weaker assertions, broader mocks, altered baselines.