data-quality-auditor

データ品質監査ツール

CSV／Excel データセットの包括的なデータ品質評価を行います。

機能

• 完全性：欠損値の分析
• 一意性：重複の検出
• 妥当性：型の検証と制約の確認
• 一貫性：パターンと形式のチェック
• 品質スコア：データ品質の総合指標
• レポート：詳細な HTML／JSON レポート

クイックスタート

from data_quality_auditor import DataQualityAuditor

auditor = DataQualityAuditor()
auditor.load_csv("customers.csv")

# Run full audit
report = auditor.audit()
print(f"Quality Score: {report['quality_score']}/100")

# Check specific issues
missing = auditor.check_missing()
duplicates = auditor.check_duplicates()

CLI の使用方法

# Full audit
python data_quality_auditor.py --input data.csv

# Generate HTML report
python data_quality_auditor.py --input data.csv --report report.html

# Check specific aspects
python data_quality_auditor.py --input data.csv --missing
python data_quality_auditor.py --input data.csv --duplicates
python data_quality_auditor.py --input data.csv --types

# JSON output
python data_quality_auditor.py --input data.csv --json

# Validate against rules
python data_quality_auditor.py --input data.csv --rules rules.json

API リファレンス

DataQualityAuditor クラス

class DataQualityAuditor:
    def __init__(self)

    # Data loading
    def load_csv(self, filepath: str, **kwargs) -> 'DataQualityAuditor'
    def load_dataframe(self, df: pd.DataFrame) -> 'DataQualityAuditor'

    # Full audit
    def audit(self) -> dict
    def quality_score(self) -> float

    # Individual checks
    def check_missing(self) -> dict
    def check_duplicates(self, subset: list = None) -> dict
    def check_types(self) -> dict
    def check_uniqueness(self) -> dict
    def check_patterns(self, column: str, pattern: str) -> dict

    # Validation
    def validate_column(self, column: str, rules: dict) -> dict
    def validate_dataset(self, rules: dict) -> dict

    # Reports
    def generate_report(self, output: str, format: str = "html") -> str
    def summary(self) -> str

品質チェック

欠損値

missing = auditor.check_missing()
# Returns:
{
    "total_cells": 10000,
    "missing_cells": 150,
    "missing_percent": 1.5,
    "by_column": {
        "email": {"count": 50, "percent": 5.0},
        "phone": {"count": 100, "percent": 10.0}
    },
    "rows_with_missing": 120
}

重複

dups = auditor.check_duplicates()
# Returns:
{
    "total_rows": 1000,
    "duplicate_rows": 25,
    "duplicate_percent": 2.5,
    "duplicate_groups": [...],
    "by_columns": {
        "email": {"duplicates": 15},
        "phone": {"duplicates": 20}
    }
}

型の検証

types = auditor.check_types()
# Returns:
{
    "columns": {
        "age": {
            "detected_type": "int64",
            "unique_values": 75,
            "sample_values": [25, 30, 45],
            "issues": []
        },
        "date": {
            "detected_type": "object",
            "unique_values": 365,
            "sample_values": ["2023-01-01", "invalid"],
            "issues": ["Mixed date formats detected"]
        }
    }
}

検証ルール

カスタム検証ルールを定義します。

{
    "columns": {
        "email": {
            "required": true,
            "unique": true,
            "pattern": "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"
        },
        "age": {
            "type": "integer",
            "min": 0,
            "max": 120
        },
        "status": {
            "allowed_values": ["active", "inactive", "pending"]
        },
        "created_at": {
            "type": "date",
            "format": "%Y-%m-%d"
        }
    }
}

results = auditor.validate_dataset(rules)

品質スコア

品質スコア（0～100）は、次の項目に基づいて算出されます。

• 完全性（30%）：欠損値の割合
• 一意性（25%）：重複行の割合
• 妥当性（25%）：型と制約への準拠
• 一貫性（20%）：形式とパターンへの準拠

score = auditor.quality_score()
# 85.5

出力形式

監査レポート

{
    "file": "data.csv",
    "rows": 1000,
    "columns": 15,
    "quality_score": 85.5,
    "completeness": {
        "score": 92.0,
        "missing_cells": 800,
        "details": {...}
    },
    "uniqueness": {
        "score": 97.5,
        "duplicate_rows": 25,
        "details": {...}
    },
    "validity": {
        "score": 78.0,
        "type_issues": [...],
        "details": {...}
    },
    "consistency": {
        "score": 80.0,
        "pattern_issues": [...],
        "details": {...}
    },
    "recommendations": [
        "Column 'phone' has 10% missing values",
        "25 duplicate rows detected",
        "Column 'date' has inconsistent formats"
    ]
}

ワークフローの例

インポート前の検証

auditor = DataQualityAuditor()
auditor.load_csv("import_data.csv")

report = auditor.audit()
if report['quality_score'] < 80:
    print("Data quality below threshold!")
    for rec in report['recommendations']:
        print(f"  - {rec}")
    exit(1)

ETL パイプラインのチェック

auditor = DataQualityAuditor()
auditor.load_dataframe(transformed_df)

# Check critical columns
email_check = auditor.validate_column("email", {
    "required": True,
    "unique": True,
    "pattern": r"^[\w.+-]+@[\w-]+\.[\w.-]+$"
})

if email_check['issues']:
    raise ValueError(f"Email validation failed: {email_check['issues']}")

ドキュメントの生成

auditor = DataQualityAuditor()
auditor.load_csv("dataset.csv")

# Generate comprehensive report
auditor.generate_report("quality_report.html", format="html")

# Or get summary text
print(auditor.summary())

依存関係

• pandas>=2.0.0
• numpy>=1.24.0