About skill-factory
skill-factory is a Codex CLI skill that guides a multi-provider pipeline for testing an implementation against a natural-language specification. It generates scenarios, runs holdout evaluation, scores satisfaction, and reports results.
Key features
- 01Eight-step factory execution contract
- 02NLSpec validation gate
- 03Scenario coverage review
- 04Holdout testing and scoring
- 05Factory report verification
Use cases
- 01Evaluating work against an NLSpec
- 02Checking edge-case scenario coverage
- 03Reviewing holdout results and satisfaction scores
Install
Copy the command and run it in a terminal.
npx skills add https:/ / github.com/ nyldn/ claude-octopus --skill skill-factoryAfter installing, type/skill-factoryin Claude Code to confirm it loaded.
Bundled filesagents/openai.yaml
Details
From the project READMEHost: Codex CLI — This skill was designed for Claude Code and adapted for Codex. Cross-reference commands use installed skill names in Codex rather than
/octo:*slash commands. Use the active Codex shell and subagent tools. Do not claim a provider, model, or host subagent is available until the current session exposes it. For host tool equivalents, seeskills/blocks/codex-host-adapter.md.
STOP - SKILL ALREADY LOADED
DO NOT call Skill() again. Execute directly.
EXECUTION CONTRACT (MANDATORY — 8-step sequence, CANNOT SKIP)
STEP 1: Clarifying Questions (MANDATORY)
Ask via AskUserQuestion BEFORE any other action:
- Spec location — path to NLSpec file, or paste inline
- Satisfaction target override (Use spec default / Custom 0.80-0.99)
- Cost approval — ~0.50-2.00 USD for ~20-30 agent calls (Approve / Approve --ci / Decline)
If spec path provided inline with the command, use it but still ask remaining questions. If user says "skip", use defaults and proceed. DO NOT proceed to Step 2 until answered.
STEP 2: Display Visual Indicators (MANDATORY — BLOCKING)
Check providers:
command -v codex &> /dev/null && codex_status="Available" || codex_status="Not installed"
command -v agy &> /dev/null && agy_status="Available" || agy_status="Not installed"
Display banner:
🐙 CLAUDE OCTOPUS ACTIVATED - Dark Factory Mode
Pipeline: Parse → Scenarios → Embrace → Holdout → Score → Report
Providers:
Codex CLI - <status> — Scenario generation + holdout evaluation
Antigravity CLI - <status> — Additional external-model challenge
Claude - Orchestration + satisfaction scoring
Spec: <path>
Estimated: 0.50-2.00 USD / 15-45 min
Validation: All external providers unavailable → continue with Claude-only (warn user about reduced diversity). At least one available → proceed normally.
STEP 3: Validate Spec File (MANDATORY — Validation Gate)
if [[ ! -f "<spec_path>" ]]; then
echo "ERROR: Spec file not found at <spec_path>"
exit 1
fi
# Check minimum content
word_count=$(wc -w < "<spec_path>")
if [[ $word_count -lt 20 ]]; then
echo "WARNING: Spec is very thin ($word_count words). Results may be limited."
fi
If spec file missing → STOP, ask user for correct path. If spec is thin (< 50 words) → WARN but proceed. DO NOT proceed past this gate if file does not exist.
STEP 4: Read Prior State (OPTIONAL)
"${HOME}/.claude-octopus/plugin/scripts/state-manager.sh" init_state 2>/dev/null || true
"${HOME}/.claude-octopus/plugin/scripts/state-manager.sh" set_current_workflow "factory" "factory" 2>/dev/null || true
Failure → continue without state, warn user.
STEP 4.5: Adversarial Scenario Coverage Gate (RECOMMENDED)
Before committing to the expensive embrace phase (~0.50-2.00 USD), verify that generated scenarios actually cover the spec's edge cases. A quick cross-provider challenge here can prevent wasting an entire factory run on incomplete scenario coverage.
After orchestrate.sh parses the spec (Phase 1-2) and before embrace execution (Phase 4), dispatch a scenario coverage review:
If a second provider is available, dispatch the challenge through Octopus routing:
# Read the spec to extract behaviors and constraints
SPEC_CONTENT=$(<"<spec_path>")
review_provider=""
command -v codex >/dev/null 2>&1 && review_provider="codex"
[[ -z "$review_provider" ]] && command -v agy >/dev/null 2>&1 && review_provider="agy"
if [[ -n "$review_provider" ]]; then
"${HOME}/.claude-octopus/plugin/scripts/orchestrate.sh" spawn "$review_provider" \
"You are a QA adversary. Given this specification and generated scenarios, identify coverage gaps.
SPECIFICATION:
${SPEC_CONTENT}
Answer:
1. Which spec BEHAVIORS have no scenario testing them?
2. Which EDGE CASES from the spec constraints are untested?
3. Which FAILURE MODES are not covered (network failures, invalid input, concurrent access)?
4. Rate overall coverage: SUFFICIENT / GAPS-FOUND / CRITICAL-GAPS
Be specific — cite the behavior or constraint ID that lacks coverage." 2>/dev/null || true
fi
After receiving the challenge response:
- If CRITICAL-GAPS found: warn the user and suggest refining the spec with
/octo:specbefore proceeding - If GAPS-FOUND: note the gaps but proceed — the holdout phase (Phase 5) will catch some of these
- If SUFFICIENT: proceed with confidence
This is a lightweight gate — it adds ~30 seconds but can save a 2.00 USD factory run on a spec with poor scenario coverage.
Skip with --fast or when user explicitly requests speed over thoroughness.
STEP 5: Execute orchestrate.sh factory (MANDATORY — Bash Tool)
${HOME}/.claude-octopus/plugin/scripts/orchestrate.sh factory --spec "<spec_path>"
With optional flags based on Step 1 answers:
--holdout-ratio <value>if custom ratio requested--max-retries <value>if custom retry count--ciif user approved non-interactive mode
STEP 6: Verify Factory Report (MANDATORY — Validation Gate)
REPORT_FILE=$(find .octo/factory -name "factory-report.md" -mmin -60 2>/dev/null | sort -r | head -1)
if [[ -z "$REPORT_FILE" ]]; then
echo "ERROR: Factory report not found"
exit 1
fi
cat "$REPORT_FILE"
If validation fails: report error, show logs from ~/.claude-octopus/logs/, DO NOT proceed, DO NOT substitute.
STEP 7: Read Scores and Present Results (MANDATORY)
SCORES_FILE=$(find .octo/factory -name "satisfaction-scores.json" -mmin -60 2>/dev/null | sort -r | head -1)
if [[ -n "$SCORES_FILE" ]]; then
cat "$SCORES_FILE"
fi
Present to user:
- Verdict with emoji (PASS/WARN/FAIL)
- Composite score vs satisfaction target
- Dimension breakdown (behavior, constraints, holdout, quality)
- Holdout highlights — which blind scenarios passed/failed
- Artifact directory path for deep review
STEP 8: Update State & Next Steps (MANDATORY)
"${HOME}/.claude-octopus/plugin/scripts/state-manager.sh" record_decision "factory" "Factory run completed: <verdict> (<score>/<target>)" 2>/dev/null || true
Present next-step suggestions based on verdict:
- PASS: Implementation meets spec. Review artifacts, run manual testing, ship.
- WARN: Close to target. Review holdout failures, consider targeted fixes.
- FAIL: Below target. Review
holdout-results.mdfor specific failures. Consider:- Refining the NLSpec with
/octo:spec - Manual fixes + re-run with
--max-retries 2 - Breaking spec into smaller, clearer pieces
- Refining the NLSpec with
Include attribution footer:
Dark Factory Mode powered by Claude Octopus v8.25.0
Pipeline: Spec → Scenarios → Embrace → Holdout → Score → Report
Providers: Codex | Antigravity | Claude
Error Handling (by step)
- Step 1 (Questions): If user declines all, proceed with defaults
- Step 2 (Providers): Both external unavailable → Claude-only mode (warn)
- Step 3 (Spec): File missing → STOP. Thin spec → WARN and proceed.
- Step 4 (State): Failure → continue without state
- Step 5 (orchestrate.sh): Show bash error, check logs — DO NOT substitute
- Step 6 (Report): Missing → show logs, DO NOT proceed
- Step 7 (Scores): Missing JSON → extract from report markdown
- Step 8 (State): Failure → skip state update, still present results
Prohibited Actions
- CANNOT skip orchestrate.sh factory execution
- CANNOT simulate or fake the factory pipeline
- CANNOT substitute direct Claude analysis for multi-provider scoring
- CANNOT skip spec validation gate
- CANNOT proceed past a failed validation gate
- CANNOT create working/progress files in plugin directory