About data-pipeline-auditor
Data Pipeline Auditor is a skill for evaluating dataset and preprocessing risks in model experiments. It guides reviewers through leakage, label semantics, sampling, and train-test mismatch, then ranks risks and suggests targeted diagnostics.
Key features
- 01Map dataset creation and splits
- 02Check leakage, duplicates, and contamination
- 03Review labels and preprocessing
- 04Assess augmentation and sampling effects
- 05Rank risks by likelihood and impact
- 06Suggest targeted diagnostics and fixes
Use cases
- 01Audit a model's dataset and split design
- 02Investigate suspicious evaluation gains
- 03Check whether preprocessing changes label meaning
- 04Identify likely train-test mismatch
Install
Copy the command and run it in a terminal.
$
npx skills add https:/After installing, type/data-pipeline-auditorin Claude Code to confirm it loaded.
Details
From the project READMEData Pipeline Auditor
Assume the data pipeline can break the result.
Intake
Ask for or infer:
- dataset source and construction process;
- split logic and any filtering;
- preprocessing/tokenization/normalization;
- augmentation and sampling;
- label source and ambiguity;
- train/validation/test metrics and surprising patterns.
Workflow
- Inspect dataset creation and splits.
- Check leakage, duplicates, and contamination.
- Verify label semantics and preprocessing.
- Examine augmentation and sampling effects.
- Identify likely train-test mismatch.
- Rank suspected bugs by likelihood and impact.
- Recommend targeted fixes or diagnostics.
Rules
- Suspicious gains deserve extra leakage checks.
- Data bugs count as findings.
- Do not call a model brittle until the data path has been checked.
- Separate confirmed leakage from plausible leakage.
Output
Return:
- pipeline map;
- confirmed risks;
- likely risks needing checks;
- highest-value diagnostics;
- fixes that would change the paper's claim or evaluation.
Comments
Markdown supported