recommendation-engine

推荐引擎

构建个性化内容和产品推荐系统。

推荐方法

方法	工作原理	优点	缺点
协同过滤	用户-物品交互	发现隐藏模式	冷启动
基于内容	物品特征	适用于新物品	发现范围有限
混合	结合两者	兼具两者优点	复杂

协同过滤

import numpy as np
from scipy.sparse import csr_matrix
from sklearn.metrics.pairwise import cosine_similarity

class CollaborativeFilter:
    def __init__(self):
        self.user_similarity = None
        self.item_similarity = None

    def fit(self, user_item_matrix):
        # User-based similarity
        self.user_similarity = cosine_similarity(user_item_matrix)
        # Item-based similarity
        self.item_similarity = cosine_similarity(user_item_matrix.T)

    def recommend_for_user(self, user_id, n=10):
        scores = self.user_similarity[user_id].dot(self.user_item_matrix)
        # Exclude already interacted items
        already_interacted = self.user_item_matrix[user_id].nonzero()[0]
        scores[already_interacted] = -np.inf
        return np.argsort(scores)[-n:][::-1]

矩阵分解（SVD）

from sklearn.decomposition import TruncatedSVD

class MatrixFactorization:
    def __init__(self, n_factors=50):
        self.svd = TruncatedSVD(n_components=n_factors)

    def fit(self, user_item_matrix):
        self.user_factors = self.svd.fit_transform(user_item_matrix)
        self.item_factors = self.svd.components_.T

    def predict(self, user_id, item_id):
        return np.dot(self.user_factors[user_id], self.item_factors[item_id])

混合推荐器

class HybridRecommender:
    def __init__(self, collab_weight=0.7, content_weight=0.3):
        self.collab = CollaborativeFilter()
        self.content = ContentBasedFilter()
        self.weights = (collab_weight, content_weight)

    def recommend(self, user_id, n=10):
        collab_scores = self.collab.score(user_id)
        content_scores = self.content.score(user_id)
        combined = self.weights[0] * collab_scores + self.weights[1] * content_scores
        return np.argsort(combined)[-n:][::-1]

评估指标

• Precision@K、Recall@K
• NDCG（排序质量）
• 覆盖率（目录多样性）
• A/B 测试转化率

冷启动解决方案

• 新用户：热门物品、引导偏好设置、基于人口统计信息
• 新物品：基于内容的引导、主动学习
• 探索策略：ε-greedy、Thompson sampling 多臂老虎机

快速开始：分 5 步构建推荐器

from scipy.sparse import csr_matrix
import numpy as np

# 1. Prepare user-item interaction matrix
# rows = users, cols = items, values = ratings/interactions
ratings_data = [(0, 5, 5), (0, 10, 4), (1, 5, 3), ...]  # (user, item, rating)
n_users, n_items = 1000, 5000

row_idx = [r[0] for r in ratings_data]
col_idx = [r[1] for r in ratings_data]
ratings = [r[2] for r in ratings_data]
user_item_matrix = csr_matrix((ratings, (row_idx, col_idx)), shape=(n_users, n_items))

# 2. Choose and train model
from recommendation_engine import ItemBasedCollaborativeFilter  # See references

model = ItemBasedCollaborativeFilter(similarity_metric='cosine', k_neighbors=20)
model.fit(user_item_matrix)

# 3. Generate recommendations
recommendations = model.recommend(user_id=42, n=10)
print(recommendations)  # [(item_id, score), ...]

# 4. Evaluate on test set
from evaluation_metrics import precision_at_k, recall_at_k

test_items = {42: {10, 25, 30}}  # True relevant items for user 42
rec_items = [item for item, score in recommendations]

precision = precision_at_k(rec_items, test_items[42], k=10)
recall = recall_at_k(rec_items, test_items[42], k=10)
print(f"Precision@10: {precision:.3f}, Recall@10: {recall:.3f}")

# 5. Handle cold start
from cold_start import PopularityRecommender

popularity_model = PopularityRecommender()
popularity_model.fit(interactions_with_timestamps)
new_user_recs = popularity_model.recommend(n=10)

已知问题预防

1. 热门度偏差

问题：只推荐热门物品，忽略长尾物品。这会降低多样性和意外发现的机会。

解决方案：平衡热门度与个性化，并通过重新排序提升多样性：

def diversify_recommendations(
    recommendations: List[Tuple[int, float]],
    item_features: np.ndarray,
    diversity_weight: float = 0.3
) -> List[Tuple[int, float]]:
    """Re-rank to increase diversity while maintaining relevance."""
    from sklearn.metrics.pairwise import cosine_distances

    selected = []
    candidates = recommendations.copy()

    while len(selected) < len(recommendations) and candidates:
        if not selected:
            # First item: highest score
            selected.append(candidates.pop(0))
            continue

        # Compute diversity scores
        selected_features = item_features[[item for item, _ in selected]]
        diversity_scores = []

        for item, relevance in candidates:
            item_feature = item_features[item].reshape(1, -1)
            # Average distance to already selected items
            avg_distance = cosine_distances(item_feature, selected_features).mean()
            # Combined score: relevance + diversity
            combined = (1 - diversity_weight) * relevance + diversity_weight * avg_distance
            diversity_scores.append((item, relevance, combined))

        # Select item with best combined score
        best = max(diversity_scores, key=lambda x: x[2])
        selected.append((best[0], best[1]))
        candidates = [(i, s) for i, s, _ in diversity_scores if i != best[0]]

    return selected

2. 数据稀疏（矩阵 >99% 为空）

问题：当大多数用户评分的物品不足 1% 时，协同过滤会失效。

解决方案：使用矩阵分解（SVD、ALS），而不是基于记忆的协同过滤：

# ❌ Bad: User-based CF on sparse data (fails to find similar users)
user_cf = UserBasedCollaborativeFilter()
user_cf.fit(sparse_matrix)  # Most users have <10 ratings

# ✅ Good: Matrix factorization handles sparsity
from sklearn.decomposition import TruncatedSVD

svd = TruncatedSVD(n_components=50)
user_factors = svd.fit_transform(sparse_matrix)
item_factors = svd.components_.T

# Predict rating: user_factors[u] @ item_factors[i]

3. 冷启动时没有回退方案

问题：对于新用户或新物品，推荐器崩溃或返回空结果。

解决方案：始终实现回退链：

def recommend_with_fallback(user_id, n=10):
    """Graceful degradation through fallback chain."""
    try:
        # Try personalized recommendations
        if has_sufficient_history(user_id, min_interactions=5):
            return collaborative_filter.recommend(user_id, n)
    except Exception as e:
        logger.warning(f"CF failed for user {user_id}: {e}")

    # Fallback 1: Demographic-based
    if user_demographics_available(user_id):
        return demographic_recommender.recommend(user_id, n)

    # Fallback 2: Popularity
    return popularity_recommender.recommend(n)

4. 未排除已有交互的物品

问题：推荐用户已购买或浏览过的物品，会浪费推荐位。

解决方案：始终过滤已交互的物品：

# ✅ Correct: Exclude interacted items
user_items = user_item_matrix[user_id].nonzero()[1]
scores[user_items] = -np.inf  # Ensure they don't appear in top-K
recommendations = np.argsort(scores)[-n:][::-1]

# ❌ Wrong: Forgetting to filter
recommendations = np.argsort(scores)[-n:][::-1]  # May include already purchased!

5. 忽略隐式反馈的置信度

问题：将所有点击和浏览一视同仁。浏览 1 次 ≠ 浏览 100 次。

解决方案：根据交互强度加权（浏览次数、观看时长等）：

# For implicit feedback, use confidence weighting
confidence_matrix = 1 + alpha * np.log(1 + interaction_counts)

# In ALS: C_ui * (P_ui - X_ui)²
# Higher confidence for items with more interactions

6. 未评估排序质量（仅使用准确率）

问题：较高的预测准确率（RMSE）并不意味着优质的 Top-K 推荐。

解决方案：使用排序指标（NDCG、MAP@K）：

# ❌ Bad: Only RMSE
from sklearn.metrics import mean_squared_error
rmse = np.sqrt(mean_squared_error(y_true, y_pred))

# ✅ Good: Ranking metrics for top-K evaluation
from evaluation_metrics import ndcg_at_k, mean_average_precision_at_k

# NDCG rewards putting highly relevant items first
ndcg = ndcg_at_k(recommendations, relevance_scores, k=10)

# MAP@K considers precision at each relevant item position
map_score = mean_average_precision_at_k(all_recommendations, ground_truth, k=10)

7. 信息茧房（缺乏探索）

问题：总是推荐相似物品会限制发现，并随着时间推移降低用户参与度。

解决方案：实现探索-利用策略：

class ExploreExploitRecommender:
    def __init__(self, base_model, epsilon=0.1):
        self.base_model = base_model
        self.epsilon = epsilon  # 10% exploration

    def recommend(self, user_id, n=10):
        # Exploit: Use trained model for most recommendations
        n_exploit = int(n * (1 - self.epsilon))
        exploitative_recs = self.base_model.recommend(user_id, n=n_exploit)

        # Explore: Add random diverse items
        n_explore = n - n_exploit
        explored_items = sample_diverse_items(n_explore)

        return exploitative_recs + explored_items

何时加载参考资料

需要详细实现时，请加载参考文件：

• 协同过滤：加载 references/collaborative-filtering-deep-dive.md，查看完整的基于用户和基于物品的协同过滤实现及相似度指标（余弦、Pearson、Jaccard）、可扩展性优化（稀疏矩阵、近似最近邻）和边缘情况处理（冷启动、稀疏性）

• 矩阵分解：加载 references/matrix-factorization-methods.md，查看 SVD、ALS 和 NMF 的实现，包括超参数调优、隐式反馈处理和高级技术（BPR、WARP）

• 评估指标：加载 references/evaluation-metrics-implementation.md，查看 Precision@K、Recall@K、NDCG、覆盖率、多样性指标、交叉验证策略和统计显著性检验（配对 t 检验、Bootstrap 置信区间）

• 冷启动解决方案：加载 references/cold-start-strategies.md，查看新用户和新物品策略（基于热门度、引导流程、人口统计信息、基于内容的引导、主动学习）、探索-利用方法（ε-greedy、Thompson sampling）和混合回退链