mcpField
F
failure-taxonomy

failure-taxonomy

github.com/owl-listenerTesting & quality191133 installsUpdated 20w ago
Repository

About failure-taxonomy

failure-taxonomy is a testing skill for classifying AI failures by type and severity. It helps teams log incidents, track trends, prioritise fixes, and investigate root causes.

Key features

  1. 01Classifies content, behavioral, technical, and safety failures
  2. 02Defines four severity levels
  3. 03Supports failure logging and trend tracking
  4. 04Guides prioritisation and root cause analysis
  5. 05Lists prevention artefacts

Use cases

  1. 01Label failures found during testing
  2. 02Prioritise fixes by severity and frequency
  3. 03Identify recurring causes across failure reports

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/owl-listener/ai-design-skills --skill failure-taxonomy

After installing, type/failure-taxonomyin Claude Code to confirm it loaded.

Details

From the project README

Failure Taxonomy

Not all AI failures are the same. A hallucination is different from a refusal, which is different from a tone mismatch. A failure taxonomy classifies failure types so teams can track, prioritise, and address them systematically.

Failure Categories

Content Failures:

  • Hallucination: The AI presents false information as fact
  • Inaccuracy: The AI gets details wrong (dates, numbers, names)
  • Incompleteness: The AI misses important information
  • Irrelevance: The AI's response doesn't address the user's actual question
  • Contradiction: The AI contradicts itself within or across responses Behavioral Failures:
  • Inappropriate refusal: The AI refuses a reasonable request
  • Missing refusal: The AI fulfils a request it should have declined
  • Tone mismatch: The AI's tone is wrong for the context
  • Persona break: The AI drops out of its defined persona
  • Over-generation: The AI produces far more than needed Technical Failures:
  • Latency: Response takes too long
  • Truncation: Response is cut off
  • Format errors: Output is in the wrong format or structure
  • Tool failures: The AI attempts to use a tool and fails
  • Context loss: The AI loses track of conversation history Safety Failures:
  • Harmful content: The AI generates content that could cause harm
  • Privacy violation: The AI reveals sensitive information
  • Bias manifestation: The AI's output shows bias against a group
  • Manipulation: The AI's output could be used to deceive or manipulate

Severity Levels

  • Critical: Causes harm or creates serious trust violation. Requires immediate fix.
  • High: Significantly degrades user experience or task success. Fix within days.
  • Medium: Noticeable quality issue that users can work around. Fix within weeks.
  • Low: Minor quality issue. Track and batch with other fixes.

Using the Taxonomy

  • Logging: Classify every detected failure by type and severity
  • Trending: Track failure type frequency over time
  • Prioritisation: Address highest-severity, highest-frequency failures first
  • Root cause analysis: Group failures by type to identify systemic causes
  • Prevention: Use failure patterns to inform guardrail design and prompt improvements

Design Artefacts

  • Failure taxonomy reference document
  • Failure logging templates
  • Severity classification rubric
  • Failure trend dashboards
  • Root cause analysis protocols

Comments

Markdown supported