outlier-detective

异常值侦探

使用多种方法检测数值数据中的异常和离群值。

功能

• 统计方法：Z-score、IQR、修正 Z-score
• 机器学习方法：Isolation Forest、LOF、DBSCAN
• 可视化：箱线图、散点图
• 多列分析：分析多个变量
• 报告：详细的异常值报告
• 灵活的阈值：可配置的灵敏度

快速开始

from outlier_detective import OutlierDetective

detective = OutlierDetective()
detective.load_csv("sales_data.csv")

# Detect outliers in a column
outliers = detective.detect("revenue", method="iqr")
print(f"Found {len(outliers)} outliers")

# Get full report
report = detective.analyze("revenue")
print(report)

CLI 用法

# Detect outliers using IQR method
python outlier_detective.py --input data.csv --column sales --method iqr

# Use Z-score with custom threshold
python outlier_detective.py --input data.csv --column price --method zscore --threshold 3

# Analyze all numeric columns
python outlier_detective.py --input data.csv --all

# Generate visualization
python outlier_detective.py --input data.csv --column revenue --plot boxplot.png

# Export outliers to CSV
python outlier_detective.py --input data.csv --column value --output outliers.csv

# Use Isolation Forest (ML)
python outlier_detective.py --input data.csv --method isolation_forest

API 参考

OutlierDetective 类

class OutlierDetective:
    def __init__(self)

    # Data loading
    def load_csv(self, filepath: str, **kwargs) -> 'OutlierDetective'
    def load_dataframe(self, df: pd.DataFrame) -> 'OutlierDetective'

    # Detection (single column)
    def detect(self, column: str, method: str = "iqr", **kwargs) -> pd.DataFrame
    def analyze(self, column: str) -> dict

    # Detection (multi-column)
    def detect_multivariate(self, columns: list = None, method: str = "isolation_forest") -> pd.DataFrame
    def analyze_all(self) -> dict

    # Visualization
    def plot_boxplot(self, column: str, output: str) -> str
    def plot_scatter(self, col1: str, col2: str, output: str) -> str
    def plot_distribution(self, column: str, output: str) -> str

    # Export
    def get_outliers(self, column: str, method: str = "iqr") -> pd.DataFrame
    def get_clean_data(self, column: str, method: str = "iqr") -> pd.DataFrame

检测方法

统计方法

IQR（四分位距）

• 默认且最稳健的方法
• 异常值：低于 Q1 - 1.5×IQR 或高于 Q3 + 1.5×IQR 的值
• 倍数可配置（默认值：1.5）

outliers = detective.detect("price", method="iqr", multiplier=1.5)

Z-Score

• 基于与均值的标准差
• 假设数据呈正态分布
• 阈值可配置（默认值：3）

outliers = detective.detect("price", method="zscore", threshold=3)

修正 Z-Score

• 使用中位数而非均值
• 对现有异常值的鲁棒性更强
• 基于 MAD（中位数绝对偏差）

outliers = detective.detect("price", method="modified_zscore", threshold=3.5)

机器学习方法

Isolation Forest

• 集成方法，适用于高维数据
• 污染率参数用于设置预期的异常值比例

outliers = detective.detect_multivariate(
    method="isolation_forest",
    contamination=0.1
)

局部离群因子（LOF）

• 基于密度的方法
• 将局部密度与邻近点进行比较

outliers = detective.detect_multivariate(
    method="lof",
    n_neighbors=20
)

输出格式

detect() 结果

# Returns DataFrame of outlier rows with additional columns:
#   - outlier_score: How extreme the value is
#   - outlier_reason: Description of why it's an outlier

   index  value  outlier_score  outlier_reason
0     15   5000          4.2    Above Q3 + 1.5×IQR
1     42  -1000         -3.8    Below Q1 - 1.5×IQR

analyze() 结果

{
    "column": "revenue",
    "total_rows": 1000,
    "outlier_count": 23,
    "outlier_percent": 2.3,
    "methods": {
        "iqr": {"count": 23, "indices": [...]},
        "zscore": {"count": 18, "indices": [...]},
        "modified_zscore": {"count": 20, "indices": [...]}
    },
    "stats": {
        "mean": 5432.10,
        "median": 4890.00,
        "std": 1234.56,
        "min": -1000.00,
        "max": 15000.00,
        "q1": 3500.00,
        "q3": 6200.00,
        "iqr": 2700.00
    },
    "bounds": {
        "lower": -550.00,
        "upper": 10250.00
    }
}

示例工作流

数据清理流程

detective = OutlierDetective()
detective.load_csv("raw_data.csv")

# Analyze and visualize
report = detective.analyze("price")
print(f"Found {report['outlier_count']} outliers ({report['outlier_percent']:.1f}%)")

# Get clean data
clean_data = detective.get_clean_data("price", method="iqr")
clean_data.to_csv("clean_data.csv")

欺诈检测

detective = OutlierDetective()
detective.load_csv("transactions.csv")

# Use multiple methods for consensus
iqr_outliers = set(detective.detect("amount", method="iqr").index)
zscore_outliers = set(detective.detect("amount", method="zscore").index)

# Transactions flagged by both methods
high_confidence = iqr_outliers & zscore_outliers
print(f"High-confidence anomalies: {len(high_confidence)}")

多变量分析

detective = OutlierDetective()
detective.load_csv("sensors.csv")

# Detect multivariate outliers
outliers = detective.detect_multivariate(
    columns=["temp", "pressure", "humidity"],
    method="isolation_forest",
    contamination=0.05
)
print(f"Anomalous readings: {len(outliers)}")

可视化示例

# Box plot with outliers highlighted
detective.plot_boxplot("revenue", "revenue_boxplot.png")

# Distribution with bounds
detective.plot_distribution("price", "price_dist.png")

# Scatter plot (2D outliers)
detective.plot_scatter("feature1", "feature2", "scatter.png")

依赖项

• pandas>=2.0.0
• numpy>=1.24.0
• scipy>=1.10.0
• scikit-learn>=1.3.0
• matplotlib>=3.7.0