mcpField
T
task-success-metrics

task-success-metrics

github.com/owl-listenerTesting & quality191131 installsUpdated 20w ago
Repository

About task-success-metrics

task-success-metrics is a skill for measuring whether AI-assisted users complete their actual tasks, not merely whether they receive a good response. It helps teams define observable outcomes and select direct, inferred, self-reported, or comparative measures.

Key features

  1. 01Task completion and timing measures
  2. 02Turns, first-attempt success, and intervention rates
  3. 03Abandonment tracking
  4. 04Direct, inferred, and self-reported measurement
  5. 05Comparison of task outcomes across versions

Use cases

  1. 01Defining success criteria for key user tasks
  2. 02Evaluating whether AI assistance improves completion
  3. 03Investigating gaps between response quality and task outcomes

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/owl-listener/ai-design-skills --skill task-success-metrics

After installing, type/task-success-metricsin Claude Code to confirm it loaded.

Details

From the project README

Task Success Metrics

Output quality doesn't guarantee task success. The AI might produce a beautiful response that doesn't actually help the user do what they came to do. Task success metrics measure the end-to-end outcome.

Defining Task Success

For each user task, define:

  • What does success look like? The user completed their goal (sent the email, found the information, finished the design)
  • What are the success criteria? Specific, observable conditions that indicate the task is done
  • What's the time expectation? How long should this task take with AI assistance vs. without?
  • What's the quality bar? Not just done, but done well enough

Task Success Metrics

  • Task completion rate: Percentage of users who complete the task (not just get a response)
  • Time to completion: How long from first input to task done
  • Turns to completion: How many back-and-forth exchanges needed
  • First-attempt success rate: Did the AI's first response accomplish the task, or did it require iteration?
  • Intervention rate: How often did the user need to correct, redirect, or override the AI?
  • Abandonment rate: How often did users give up before completing the task?

Measuring Task Success

  • Direct measurement: Track task completion through product analytics (user clicked "done", saved the output, moved to next step)
  • Inferred measurement: Infer success from proxy signals (session length, return rate, output edits)
  • Self-reported measurement: Ask users whether the AI helped them accomplish their goal
  • Comparative measurement: Compare task success with AI vs. without AI, or with version A vs. version B

Task Success vs. Output Quality

These can diverge:

  • High output quality, low task success: The AI's answer is well-written but doesn't address the real need
  • Low output quality, high task success: The AI's answer is rough but gives the user exactly what they needed
  • Both matter: Track both and investigate when they diverge

Design Artefacts

  • Task success definitions per key user task
  • Metrics framework with measurement methods
  • Success criteria specifications
  • Baseline measurements (before AI, or current version)
  • Task success dashboard specifications

Comments

Markdown supported