mcpField
C
codex-readiness-integration-test

codex-readiness-integration-test

github.com/openaiTesting & quality28.4k60 installsUpdated 18w ago
Repository

About codex-readiness-integration-test

codex-readiness-integration-test is a skill that tests agentic execution in a repository using Codex CLI, followed by build/test commands and evaluation. It requires user approval of the task prompt before execution and produces review artifacts.

Key features

  1. 01Codex CLI agentic execution
  2. 02User-approved task prompt
  3. 03In-repository build and test runs
  4. 04Evidence and deterministic checks
  5. 05LLM evaluation and summaries

Use cases

  1. 01Testing an agent's repository change loop
  2. 02Validating build and test commands after edits
  3. 03Reviewing execution evidence and evaluation

MCP support

The skill does not exercise MCP servers or tools unless they are explicitly configured.

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/openai/skills --skill codex-readiness-integration-test

After installing, type/codex-readiness-integration-testin Claude Code to confirm it loaded.

Bundled files

References and scripts installed along with SKILL.md.

LICENSE.txt
references/
agentic_loop_eval.md
change_quality.md
checks.json
generate_prompt.md
json_fix.md
llm_eval_schema.json
scripts/
collect_evidence.py
deterministic_rules.py
run_agentic_loop.py
run_integration_test.py
run_llm_eval.py
run_plan.py
scoring.py
tests/
test_planning_signal_rule.py
test_verification_after_code_changes.py

Details

From the project README

LLM Codex Readiness Integration Test

This skill runs a multi-stage integration test to validate agentic execution quality. It always runs in execute mode (no read-only mode).

Outputs

Each run writes to .codex-readiness-integration-test/<timestamp>/ and updates .codex-readiness-integration-test/latest.json.

New outputs per run:

  • agentic_summary.json and logs/agentic.log (agentic loop execution)
  • llm_results.json (automatic LLM evaluation)
  • summary.txt (human-readable summary)

Pre-conditions (Required)

  • Authenticate with the Codex CLI using the repo-local HOME before running the test. Run these in your own terminal (not via the integration test): HOME=$PWD/.codex-home XDG_CACHE_HOME=$PWD/.codex-home/.cache codex login HOME=$PWD/.codex-home XDG_CACHE_HOME=$PWD/.codex-home/.cache codex login status
  • The integration test creates {repo_root}/.codex-home and {repo_root}/.codex-home/.cache/codex as its first step.

Workflow

  1. Ask the user how to source the task.
    • Offer two explicit options: (a) user provides a custom task/prompt, or (b) auto-generate a task.
    • Do not run the entry point until the user chooses one option.
  2. Generate or load {out_dir}/prompt.pending.json.
    • Use the integration test's expected prompt path, not prompt.json at the repo root.
    • With the default out dir, this path is .codex-readiness-integration-test/prompt.pending.json.
    • If --seed-task is provided, it is used as the starting task.
    • If not provided, generate a task with skills/codex-readiness-integration-test/references/generate_prompt.md and save the JSON to {out_dir}/prompt.pending.json.
    • The user must approve the prompt before execution (no auto-approve mode). Make sure to output a summary of the prompt when asking the user to approve.
  3. Execute the agentic loop via Codex CLI (uses AGENTS.md and change_prompt).
  4. Run build/test commands from the prompt plan via skills/codex-readiness-integration-test/scripts/run_plan.py.
  5. Collect evidence (evidence.json), deterministic checks, and run automatic LLM evals via Codex CLI.
  6. Score and write the report + summary output.

Configuration

Optional fields in {out_dir}/prompt.pending.json:

  • agentic_loop: configure Codex CLI invocation for the agentic loop.
  • llm_eval: configure Codex CLI invocation for automatic evals.

If these fields are omitted, defaults are used.

Requirements

  • The LLM evaluator must fail if evidence mentions the phrase Context compaction enabled.
  • Use qualitative context-usage evaluation (no strict thresholds).

What this test covers well

  • Runs Codex CLI against the real repo root, producing real filesystem edits and git diffs.
  • Executes the approved change prompt and then runs the build/test plan in-repo.
  • Captures evidence, deterministic checks, and LLM eval artifacts for review.

What this test does not represent

  • The agentic loop may use non-default flags (e.g., bypass approvals/sandbox), so interactive guardrails differ.
  • Uses a dedicated HOME (.codex-home), which can change auth/config/cache vs normal CLI use.
  • Auto-generated prompts and one-shot execution do not simulate interactive guidance.
  • MCP servers/tools are not exercised unless explicitly configured.

Notes

  • The prompts in skills/codex-readiness-integration-test/references/ expect strict JSON.
  • Use skills/codex-readiness-integration-test/references/json_fix.md to repair invalid JSON output.
  • This skill calls the codex CLI. Ensure it is installed and available on PATH, or override the command in {out_dir}/prompt.pending.json.
  • If the agentic loop detects sandbox-blocked tool access, it now writes requires_escalation: true to {run_dir}/agentic_summary.json and exits with code 3. Re-run the integration test with escalated permissions in that case.

Comments

Markdown supported