Recommendation System

推荐系统

概述

此技能使用矩阵分解技术实现协同过滤和基于内容的推荐系统，用于预测用户偏好、提升参与度，并通过个性化的商品推荐促进转化。

适用场景

• 开发推荐功能，以提升用户参与度和留存率
• 实现个性化商品推荐，以提高销量和转化率
• 构建结合协同过滤和基于内容的方法的混合推荐系统
• 分析并优化推荐覆盖率、多样性和准确性
• 处理稀疏的用户-商品交互矩阵和冷启动场景
• 运行 A/B 测试，衡量推荐算法对业务指标的影响

方法

• 协同过滤：与你相似的用户喜欢 X
• 基于内容：与你喜欢的内容相似的商品
• 混合方法：结合多种方法
• 矩阵分解：潜因子模型
• 深度学习：用于嵌入的神经网络

关键指标

• Precision@K：推荐结果中相关内容的百分比
• Recall@K：找到的相关商品的百分比
• NDCG：排序质量指标
• 覆盖率：获得推荐的商品百分比
• 多样性：推荐结果的丰富程度

使用 Python 实现

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import NMF
import seaborn as sns

# Create sample user-item interaction data
np.random.seed(42)
users = [f'user_{i}' for i in range(100)]
items = [f'item_{i}' for i in range(50)]

# Generate ratings (sparse matrix)
ratings_list = []
for user in users:
    n_items_rated = np.random.randint(5, 20)
    rated_items = np.random.choice(items, n_items_rated, replace=False)
    for item in rated_items:
        rating = np.random.randint(1, 6)
        ratings_list.append({'user': user, 'item': item, 'rating': rating})

ratings_df = pd.DataFrame(ratings_list)
print("Sample Ratings:")
print(ratings_df.head(10))

# Create user-item matrix
user_item_matrix = ratings_df.pivot_table(
    index='user', columns='item', values='rating', fill_value=0
)

print(f"\nUser-Item Matrix Shape: {user_item_matrix.shape}")
print(f"Sparsity: {1 - (user_item_matrix != 0).sum().sum() / (user_item_matrix.shape[0] * user_item_matrix.shape[1]):.2%}")

# 1. User-based Collaborative Filtering
user_similarity = cosine_similarity(user_item_matrix)
user_similarity_df = pd.DataFrame(
    user_similarity, index=user_item_matrix.index, columns=user_item_matrix.index
)

print("\n1. User Similarity Matrix (Sample):")
print(user_similarity_df.iloc[:5, :5])

# Get recommendations for a user
def get_user_based_recommendations(user_id, user_sim_matrix, user_item_mat, n=5):
    similar_users = user_sim_matrix[user_id].sort_values(ascending=False)[1:11]

    recommendations = {}
    for item in user_item_mat.columns:
        if user_item_mat.loc[user_id, item] == 0:  # Not yet rated
            score = (similar_users * user_item_mat.loc[similar_users.index, item]).sum()
            recommendations[item] = score

    top_recs = sorted(recommendations.items(), key=lambda x: x[1], reverse=True)[:n]
    return [rec[0] for rec in top_recs]

# Example: Get recommendations for user_0
user_recommendations = get_user_based_recommendations('user_0', user_similarity_df, user_item_matrix)
print(f"\nRecommendations for user_0: {user_recommendations}")

# 2. Item-based Collaborative Filtering
item_similarity = cosine_similarity(user_item_matrix.T)
item_similarity_df = pd.DataFrame(
    item_similarity, index=user_item_matrix.columns, columns=user_item_matrix.columns
)

print("\n2. Item Similarity Matrix (Sample):")
print(item_similarity_df.iloc[:5, :5])

# 3. Content-based Filtering
item_features = np.random.rand(len(items), 10)  # Simulate item features
item_feature_similarity = cosine_similarity(item_features)

fig, axes = plt.subplots(2, 2, figsize=(14, 10))

# User similarity heatmap
sns.heatmap(user_similarity_df.iloc[:10, :10], annot=True, fmt='.2f', cmap='coolwarm',
            ax=axes[0, 0], cbar_kws={'label': 'Similarity'})
axes[0, 0].set_title('User Similarity Matrix (Sample)')

# Item similarity heatmap
sns.heatmap(item_similarity_df.iloc[:10, :10], annot=True, fmt='.2f', cmap='coolwarm',
            ax=axes[0, 1], cbar_kws={'label': 'Similarity'})
axes[0, 1].set_title('Item Similarity Matrix (Sample)')

# Rating distribution
axes[1, 0].hist(ratings_df['rating'], bins=5, color='steelblue', edgecolor='black', alpha=0.7)
axes[1, 0].set_xlabel('Rating')
axes[1, 0].set_ylabel('Count')
axes[1, 0].set_title('Rating Distribution')
axes[1, 0].grid(True, alpha=0.3, axis='y')

# Sparsity by user
user_rating_counts = user_item_matrix.astype(bool).sum(axis=1)
axes[1, 1].hist(user_rating_counts, bins=20, color='lightcoral', edgecolor='black', alpha=0.7)
axes[1, 1].set_xlabel('Number of Rated Items')
axes[1, 1].set_ylabel('Number of Users')
axes[1, 1].set_title('User Activity Distribution')
axes[1, 1].grid(True, alpha=0.3, axis='y')

plt.tight_layout()
plt.show()

# 4. Matrix Factorization (NMF)
nmf = NMF(n_components=10, init='random', random_state=42, max_iter=200)
user_latent = nmf.fit_transform(user_item_matrix)
item_latent = nmf.components_.T

print(f"\n4. Matrix Factorization:")
print(f"User latent factors shape: {user_latent.shape}")
print(f"Item latent factors shape: {item_latent.shape}")

# Reconstruct ratings
reconstructed_ratings = user_latent @ item_latent.T
reconstructed_df = pd.DataFrame(
    reconstructed_ratings, index=user_item_matrix.index, columns=user_item_matrix.columns
)

# Calculate RMSE
original_ratings = user_item_matrix[user_item_matrix > 0]
predicted_ratings = reconstructed_df[user_item_matrix > 0]
rmse = np.sqrt(np.mean((original_ratings - predicted_ratings) ** 2))
print(f"Reconstruction RMSE: {rmse:.4f}")

# 5. Evaluation Metrics
def precision_at_k(actual, predicted, k=5):
    if len(actual) == 0:
        return 0
    return len(set(actual[:k]) & set(predicted)) / k

def recall_at_k(actual, predicted, k=5):
    if len(actual) == 0:
        return 0
    return len(set(actual[:k]) & set(predicted)) / len(actual)

# Simulate test set
test_user = 'user_0'
actual_items = ratings_df[ratings_df['user'] == test_user]['item'].values
predicted_items = get_user_based_recommendations(test_user, user_similarity_df, user_item_matrix, n=10)

p_at_5 = precision_at_k(predicted_items, actual_items, k=5)
r_at_5 = recall_at_k(predicted_items, actual_items, k=5)

print(f"\n5. Evaluation Metrics:")
print(f"Precision@5: {p_at_5:.2%}")
print(f"Recall@5: {r_at_5:.2%}")
print(f"F1@5: {2 * (p_at_5 * r_at_5) / (p_at_5 + r_at_5):.2%}")

# 6. Coverage and Diversity
recommended_items = set()
for user in user_item_matrix.index[:20]:
    recs = get_user_based_recommendations(user, user_similarity_df, user_item_matrix, n=5)
    recommended_items.update(recs)

coverage = len(recommended_items) / len(items)
print(f"\nCoverage: {coverage:.2%}")

# 7. Popularity Analysis
item_popularity = ratings_df['item'].value_counts()

fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# Top items
axes[0].barh(item_popularity.head(10).index, item_popularity.head(10).values,
             color='steelblue', edgecolor='black', alpha=0.7)
axes[0].set_xlabel('Number of Ratings')
axes[0].set_title('Top 10 Most Popular Items')
axes[0].grid(True, alpha=0.3, axis='x')

# Popularity distribution
axes[1].hist(item_popularity, bins=20, color='lightcoral', edgecolor='black', alpha=0.7)
axes[1].set_xlabel('Number of Ratings')
axes[1].set_ylabel('Number of Items')
axes[1].set_title('Item Popularity Distribution')
axes[1].grid(True, alpha=0.3, axis='y')

plt.tight_layout()
plt.show()

# 8. Cold Start Problem Analysis
new_user = 'new_user'
new_user_ratings = pd.DataFrame({
    'user': [new_user] * 2,
    'item': ['item_0', 'item_1'],
    'rating': [5, 4]
})

print(f"\n8. Cold Start Problem:")
print(f"New user has rated: {len(new_user_ratings)} items")
print(f"Recommendation challenge: Limited user history")

# 9. Recommendation accuracy over time
k_values = [1, 3, 5, 10]
metrics_over_k = []

for k in k_values:
    precision_scores = []
    for user in user_item_matrix.index[:10]:
        recs = get_user_based_recommendations(user, user_similarity_df, user_item_matrix, n=k)
        actual = ratings_df[ratings_df['user'] == user]['item'].values
        precision_scores.append(precision_at_k(recs, actual, k=k))

    metrics_over_k.append({
        'K': k,
        'Precision': np.mean(precision_scores),
        'Recall': np.mean([recall_at_k(get_user_based_recommendations(user, user_similarity_df, user_item_matrix, n=k),
                          ratings_df[ratings_df['user'] == user]['item'].values, k=k)
                          for user in user_item_matrix.index[:10]])
    })

metrics_df = pd.DataFrame(metrics_over_k)

fig, ax = plt.subplots(figsize=(10, 5))
ax.plot(metrics_df['K'], metrics_df['Precision'], marker='o', linewidth=2, label='Precision', markersize=8)
ax.plot(metrics_df['K'], metrics_df['Recall'], marker='s', linewidth=2, label='Recall', markersize=8)
ax.set_xlabel('K (Number of Recommendations)')
ax.set_ylabel('Score')
ax.set_title('Precision and Recall vs K')
ax.legend()
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

# 10. A/B Test Results (Simulated)
print("\n10. A/B Test Results (Simulated):")
print("Control (No recommendations): 5.2% Conversion Rate")
print("Treatment (Recommendations): 7.8% Conversion Rate")
print("Lift: 50% (Statistically Significant, p < 0.05)")

print("\nRecommendation system complete!")

算法比较

• 协同过滤：简单，无需内容信息
• 基于内容：适用于冷启动场景
• 矩阵分解：可扩展，能够发现潜在模式
• 深度学习：可捕捉复杂模式，但需要数据
• 混合方法：结合多种方法的优势

实现注意事项

• 处理冷启动问题（新用户/新商品）
• 确保大规模场景下的计算效率
• 解决稀疏性问题（大多数商品都没有评分）
• 权衡多样性与相关性
• 实时推荐与批量推荐

交付内容

• 用户-商品交互矩阵
• 相似度矩阵
• 面向示例用户的推荐结果
• 评估指标（precision、recall、NDCG）
• 覆盖率和多样性分析
• 结果可视化
• 生产环境实现代码