ai-feature-testing

AI Feature Testing

When to Use

• Use this skill when you need to verify product AI features for quality, safety, fairness, fallback behavior, and user experience.
• Use it to review an existing plan, result, or evidence set and produce actionable improvements.
• Use it when context is incomplete but a bounded first pass is still valuable.

Output Format Options

• Default to Markdown for review, execution, and incremental refinement.
• When the user requests tables, CSV, JSON, or ticket fields, preserve risk, evidence, priority, and boundary information.
• For machine-consumed output, confirm the schema, enums, and required fields first.

How to Use

1. Read and follow prompts/ai-feature-testing.md, including its input contract, execution rules, minimum coverage, and output order.
2. Add only context that changes the decision: scope, environment, version, constraints, evidence, and success criteria.
3. Audit the input, then separate confirmed facts, working assumptions, and open questions.
4. Rank by risk and evidence strength, and produce an artifact that can be executed or reviewed directly.
5. If information is missing, deliver a bounded first pass and state which conclusions remain unsupported.

Reference Files

• When calibrating domain outcomes, valid controls, or evidence gaps, read references/domain-decision-example.md for sanitized input, per-item oracles, and reproducible calculations.

• Always read prompts/ai-feature-testing.md; it is the complete execution specification for this skill.

• For evaluation or regression, read evals/eval.yaml and the relevant cases under evals/cases/.

• Load references/, examples/, scripts/, or output-formats.md only when those directories exist and the task needs them.

Domain Execution Essentials

1. Fix model, prompt, knowledge/evaluation data, product, and environment versions. Slice by product task×language×user group×risk, recording sample provenance/counts; aggregates cannot hide minority groups.
2. Give Human rubric dimensions observable anchors, labelers, disagreement handling, and reference provenance. Calibrate examples before project-defined scores/weights; undefined scoring stays open.
3. Separate task quality, safety/privacy, required refusal versus answerable controls, and fallback gates. Record source, denominator, and version for each gate; more refusal is not inherently safer.
4. Missing slice samples or thresholds yield UNKNOWN/open questions. Release recommendations retain unresolved gates and Human decisions; static plans do not establish passed execution.

Core Constraints

• do not test only ideal inputs
• distinguish model limitations from integration defects
• require refusal or human fallback for high-risk outputs
• Never invent system behavior, fields, data, metrics, or root causes absent from the evidence.
• Link important conclusions to evidence; mark unsupported conclusions as hypotheses with a verification method.
• Explain priority using business impact, likelihood, or detectability.

Delivery Checklist

• Covered: task quality, factuality, consistency, safety, fairness, privacy, latency and cost, fallback and human handoff.
• Separated facts, assumptions, gaps, and recommendations.
• Gave high-risk items a priority, evidence basis, owner or next action.
• Defined verifiable decision criteria instead of generic advice.
• Performed no unauthorized production writes or destructive actions.

Common Pitfalls

• Listing checks without preconditions, expected outcomes, or evidence.
• Marking everything high priority and avoiding tradeoffs.
• Substituting tool names or generic theory for domain reasoning.
• Refusing incomplete input, or pretending incomplete evidence supports certainty.

Best Practices

• Start with paths most likely to cause business loss, safety issues, or release blockage.
• Reduce uncertainty through the smallest verifiable experiment and record reproduction conditions.
• Make the artifact executable and independently reviewable by another engineer.