data-validation

数据验证

验证不同系统中的数据质量、类型、架构和一致性。此技能通过全面的验证规则、架构验证、数据质量检查和异常检测，帮助确保数据的准确性、完整性和可靠性。

何时使用

在以下情况下使用此技能：

• 需要确保不同系统中的数据质量和一致性
• 正在构建数据管道，需要制定验证规则
• 正在不同系统之间迁移数据，需要验证数据准确性
• 需要实现数据质量监控和告警
• 正在设计数据架构，需要制定验证规则
• 需要检测数据异常和不一致
• 正在实施数据治理和质量标准
• 需要根据业务规则和约束验证数据

我能做什么

• 架构验证：根据架构验证数据（JSON Schema、Avro、Protobuf、XML Schema）
• 数据类型验证：验证数据类型（字符串、数字、日期、布尔值等）
• 数据质量检查：检查数据的完整性、准确性、一致性和及时性
• 业务规则验证：根据业务规则和约束验证数据
• 引用完整性验证：验证数据实体之间的关系
• 异常检测：检测数据异常值和离群值
• 数据分析：分析数据以了解其特征和质量
• 数据沿袭验证：验证数据沿袭和转换准确性
• 数据一致性验证：验证多个数据源之间的一致性
• 数据验证规则管理：管理验证规则并进行版本控制

示例

# Validate data against JSON schema
./scripts/analyze-data-validation.sh --schema schema.json --data data.json

# Perform data quality checks
./scripts/analyze-data-validation.sh --quality-checks --source database --rules rules.yaml

# Detect data anomalies
./scripts/analyze-data-validation.sh --anomaly-detection --data data.csv --threshold 3

# Validate data consistency across sources
./scripts/analyze-data-validation.sh --consistency --sources source1.csv source2.csv

# Generate data validation report
./scripts/analyze-data-validation.sh --report --output validation-report.json

输出格式

Data Validation Report
─────────────────────────────────────
Validation Date: 2025-01-15T10:30:00Z
Data Source: customer_data.csv
Total Records: 1,250,847
Validation Duration: 45s

SCHEMA VALIDATION RESULTS:
───────────────────────────
✅ Valid: 1,200,543 records (96%)
❌ Invalid: 50,304 records (4%)

Schema Violations:
• Missing required field "customer_id": 12,847 records
• Invalid email format: 18,542 records
• Invalid date format (birth_date): 8,925 records
• Invalid phone number format: 10,090 records

DATA QUALITY METRICS:
──────────────────────
Completeness: 97.8%
  • Customer name: 99.2% complete
  • Email address: 96.5% complete
  • Phone number: 95.8% complete
  • Address: 92.4% complete

Accuracy: 95.3%
  • Email deliverability: 94.8% valid
  • Phone number validity: 96.2% valid
  • Address validity: 95.0% valid

Consistency: 98.1%
  • Date formats consistent: 99.2%
  • Country codes consistent: 97.8%
  • Currency formats consistent: 97.3%

Timeliness: 99.5%
  • Data freshness: 99.8% updated within 24 hours
  • Update frequency: 99.2% updated daily

BUSINESS RULE VALIDATION:
─────────────────────────
✅ Valid: 1,245,120 records (99.5%)
❌ Invalid: 5,727 records (0.5%)

Business Rule Violations:
• Customer age < 18: 1,247 records
• Invalid order amount (negative): 842 records
• Duplicate customer records: 2,150 records
• Inconsistent region-country mapping: 1,488 records

REFERENTIAL INTEGRITY VALIDATION:
─────────────────────────────────
✅ Valid: 1,248,950 references (99.8%)
❌ Invalid: 1,897 references (0.2%)

Referential Integrity Issues:
• Orphaned order records (missing customer): 892 references
• Invalid product references: 1,005 references

ANOMALY DETECTION:
──────────────────
⚠️ Detected: 3,842 anomalies (0.3%)

Anomaly Types:
• Unusual customer age distribution: 1,250 anomalies
• Abnormal order amounts: 1,892 anomalies
• Unexpected geographic distribution: 700 anomalies

DATA PROFILING SUMMARY:
───────────────────────
Numeric Fields:
  • order_amount: Min=$1, Max=$15,250, Avg=$245.75, StdDev=$125.42
  • customer_age: Min=18, Max=95, Avg=42.3, StdDev=12.8

Categorical Fields:
  • country: 45 distinct values (US: 45%, UK: 12%, CA: 8%, ...)
  • product_category: 12 distinct values (Electronics: 35%, Clothing: 25%, ...)

Date Fields:
  • order_date: Range=2024-01-01 to 2025-01-15
  • customer_since: Range=2018-03-15 to 2025-01-15

DATA CONSISTENCY ACROSS SOURCES:
────────────────────────────────
• Customer database vs CRM: 98.7% consistent
• Product catalog vs inventory: 97.2% consistent
• Order system vs payment gateway: 99.1% consistent

RECOMMENDATIONS:
────────────────
1. IMMEDIATE ACTION:
   • Fix missing customer_id field (12,847 records)
   • Clean invalid email addresses (18,542 records)
   • Remove duplicate customer records (2,150 records)

2. SHORT TERM (1-2 weeks):
   • Implement data validation in ingestion pipeline
   • Set up data quality monitoring and alerting
   • Establish data quality SLA (target: 99.5% quality)

3. MEDIUM TERM (1-3 months):
   • Implement data profiling and anomaly detection
   • Establish data governance framework
   • Implement data lineage tracking

4. LONG TERM (3-12 months):
   • Implement master data management
   • Establish data quality improvement program
   • Implement predictive data quality monitoring

VALIDATION RULE COVERAGE:
─────────────────────────
• Schema validation: 100% coverage
• Data type validation: 100% coverage
• Business rule validation: 85% coverage
• Referential integrity: 90% coverage
• Anomaly detection: 75% coverage

Overall Data Quality Score: 92.8/100

备注

• 应尽可能在数据录入环节附近实施数据验证
• 应像管理代码一样对验证规则进行版本控制和管理
• 考虑数据验证对性能的影响，尤其是处理大型数据集时
• 对流式数据实施增量验证
• 使用统计方法进行异常检测，以减少误报
• 应持续跟踪数据质量指标，以识别趋势
• 验证失败时应触发相应的补救工作流
• 验证敏感数据时应考虑数据隐私和安全