About write-code-eval
A skill for implementing code evaluators for identified agent failure modes, intended for teams working in a project's language and evaluation framework. It guides rule definition, implementation, and testing against passing, failing, and borderline examples.
Key features
- 01Turn failure modes into checks
- 02Identify required trace fields or reference data
- 03Implement in the project's language and eval framework
- 04Test passing, failing, and borderline cases
- 05Inspect results and compare with human labels
Use cases
- 01Check output structure or required fields
- 02Match missing or forbidden text
- 03Validate citation sources or tool calls
- 04Compare output values with reference values
Install
Copy the command and run it in a terminal.
$
npx skills add https:/After installing, type/write-code-evalin Claude Code to confirm it loaded.
Bundled filesagents/openai.yaml
Details
From the project READMEWrite a code evaluator
Start with a failure mode found through error analysis. Write one check for that failure mode, much like a unit test that asserts what should hold for each trace.
- State the rule and identify the trace fields or reference data the check needs. If the rule requires interpretation, use
write-judge-prompt. - Implement the check in the project's language and eval framework. Return a result and a reason in the format that framework expects.
- Test known passes and failures, including borderline cases. Run the check on available traces and inspect mistakes. If the rule uses a proxy for human judgment, compare its results with human labels.
Examples
| Failure mode | Possible check |
|---|---|
| Invalid output structure | Parse the output and check required fields |
| Missing or forbidden text | Match a string or pattern |
| Citation not in retrieved documents | Compare cited IDs with retrieved IDs |
| Bad tool call | Check arguments against the tool schema or run the call in a safe test environment |
| Wrong value | Compare the output with a reference value |
Choose the check from the failure rule. For a failure with both objective and interpretive parts, check the objective part with code and use a judge for the rest.
Comments
Markdown supported