Natural Language Processing

自然言語処理

概要

このスキルでは、最新のトランスフォーマー、BERT、GPT、従来のNLP技術を活用して、テキスト分類、固有表現認識、感情分析などのNLPアプリケーションを構築するための包括的なツールを提供します。

使用する場面

• 感情分析、トピック分類、インテント検出のためのテキスト分類システムを構築する
• 非構造化テキストから固有表現（人名、地名、組織名）を抽出する
• 機械翻訳、テキスト要約、質問応答システムを実装する
• インサイトを得るため、大量のテキストデータを処理・分析する
• チャットボット、バーチャルアシスタント、会話型AIアプリケーションを作成する
• ドメイン固有のNLPタスク向けに、事前学習済みトランスフォーマーモデルをファインチューニングする

NLPの主要タスク

• テキスト分類：感情、トピック、インテントの分類
• 固有表現認識：人名、地名、組織名の特定
• 機械翻訳：言語間のテキスト翻訳
• テキスト要約：重要な情報の抽出
• 質問応答：文書内から回答を見つける
• テキスト生成：一貫性のあるテキストの生成

人気のモデルとライブラリ

• Transformers：BERT、GPT、RoBERTa、T5
• spaCy：産業用途向けNLPパイプライン
• NLTK：従来型のNLPツールキット
• Hugging Face：事前学習済みモデルのハブ
• PyTorch/TensorFlow：ディープラーニングフレームワーク

Pythonによる実装

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from collections import Counter
import re
import nltk
from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer
import torch
from transformers import (AutoTokenizer, AutoModelForSequenceClassification,
                         AutoModelForTokenClassification, pipeline,
                         TextClassificationPipeline)
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
import warnings
warnings.filterwarnings('ignore')

# Download required NLTK resources
try:
    nltk.data.find('tokenizers/punkt')
except LookupError:
    nltk.download('punkt')

print("=== 1. Text Preprocessing ===")

def preprocess_text(text, remove_stopwords=True, lemmatize=True):
    """Complete text preprocessing pipeline"""
    # Lowercase
    text = text.lower()

    # Remove special characters and digits
    text = re.sub(r'[^a-zA-Z\s]', '', text)

    # Tokenize
    tokens = word_tokenize(text)

    # Remove stopwords
    if remove_stopwords:
        stop_words = set(stopwords.words('english'))
        tokens = [t for t in tokens if t not in stop_words]

    # Lemmatize
    if lemmatize:
        lemmatizer = WordNetLemmatizer()
        tokens = [lemmatizer.lemmatize(t) for t in tokens]

    return tokens, ' '.join(tokens)

sample_text = "The quick brown foxes are jumping over the lazy dogs! Amazing performance."
tokens, processed = preprocess_text(sample_text)
print(f"Original: {sample_text}")
print(f"Processed: {processed}")
print(f"Tokens: {tokens}\n")

# 2. Text Classification with sklearn
print("=== 2. Traditional Text Classification ===")

# Sample data
texts = [
    "I love this product, it's amazing!",
    "This movie is fantastic and entertaining.",
    "Best purchase ever, highly recommended.",
    "Terrible quality, very disappointed.",
    "Worst experience, waste of money.",
    "Horrible service and poor quality.",
    "The food was delicious and fresh.",
    "Great atmosphere and friendly staff.",
    "Bad weather today, very gloomy.",
    "The book was boring and uninteresting."
]

labels = [1, 1, 1, 0, 0, 0, 1, 1, 0, 0]  # 1: positive, 0: negative

# TF-IDF vectorization
tfidf = TfidfVectorizer(max_features=100, ngram_range=(1, 2))
X_tfidf = tfidf.fit_transform(texts)

# Train classifier
clf = MultinomialNB()
clf.fit(X_tfidf, labels)

# Evaluate
predictions = clf.predict(X_tfidf)
print(f"Accuracy: {accuracy_score(labels, predictions):.4f}")
print(f"Precision: {precision_score(labels, predictions):.4f}")
print(f"Recall: {recall_score(labels, predictions):.4f}")
print(f"F1: {f1_score(labels, predictions):.4f}\n")

# 3. Transformer-based text classification
print("=== 3. Transformer-based Classification ===")

try:
    # Use Hugging Face transformers for sentiment analysis
    sentiment_pipeline = pipeline(
        "sentiment-analysis",
        model="distilbert-base-uncased-finetuned-sst-2-english"
    )

    test_sentences = [
        "This is a wonderful movie!",
        "I absolutely hate this product.",
        "It's okay, nothing special.",
        "Amazing quality and fast delivery!"
    ]

    print("Sentiment Analysis Results:")
    for sentence in test_sentences:
        result = sentiment_pipeline(sentence)
        print(f"  Text: {sentence}")
        print(f"  Sentiment: {result[0]['label']}, Score: {result[0]['score']:.4f}\n")

except Exception as e:
    print(f"Transformer model not available: {str(e)}\n")

# 4. Named Entity Recognition (NER)
print("=== 4. Named Entity Recognition ===")

try:
    ner_pipeline = pipeline(
        "ner",
        model="distilbert-base-uncased",
        aggregation_strategy="simple"
    )

    text = "Apple Inc. was founded by Steve Jobs in Cupertino, California."
    entities = ner_pipeline(text)

    print(f"Text: {text}")
    print("Entities:")
    for entity in entities:
        print(f"  {entity['word']}: {entity['entity_group']} (score: {entity['score']:.4f})")

except Exception as e:
    print(f"NER model not available: {str(e)}\n")

# 5. Word embeddings and similarity
print("\n=== 5. Word Embeddings and Similarity ===")

from sklearn.metrics.pairwise import cosine_similarity

# Simple bag-of-words embeddings
vectorizer = CountVectorizer(max_features=50)
docs = [
    "machine learning is great",
    "deep learning uses neural networks",
    "machine learning and deep learning"
]

embeddings = vectorizer.fit_transform(docs).toarray()

# Compute similarity
similarity_matrix = cosine_similarity(embeddings)
print("Document Similarity Matrix:")
print(pd.DataFrame(similarity_matrix, columns=[f"Doc{i}" for i in range(len(docs))],
                  index=[f"Doc{i}" for i in range(len(docs))]).round(3))

# 6. Tokenization and vocabulary
print("\n=== 6. Tokenization Analysis ===")

corpus = " ".join(texts)
tokens, _ = preprocess_text(corpus)

# Vocabulary
vocab = Counter(tokens)
print(f"Vocabulary size: {len(vocab)}")
print("Top 10 most common words:")
for word, count in vocab.most_common(10):
    print(f"  {word}: {count}")

# 7. Advanced Transformer pipeline
print("\n=== 7. Advanced NLP Tasks ===")

try:
    # Zero-shot classification
    zero_shot_pipeline = pipeline(
        "zero-shot-classification",
        model="facebook/bart-large-mnli"
    )

    sequence = "Apple is discussing the possibility of acquiring startup for 1 billion dollars"
    candidate_labels = ["business", "sports", "technology", "politics"]

    result = zero_shot_pipeline(sequence, candidate_labels)
    print("Zero-shot Classification Results:")
    for label, score in zip(result['labels'], result['scores']):
        print(f"  {label}: {score:.4f}")

except Exception as e:
    print(f"Advanced pipeline not available: {str(e)}\n")

# 8. Text statistics and analysis
print("\n=== 8. Text Statistics ===")

sample_texts = [
    "Natural language processing is fascinating.",
    "Machine learning enables artificial intelligence.",
    "Deep learning revolutionizes computer vision."
]

stats_data = []
for text in sample_texts:
    words = text.split()
    chars = len(text)
    avg_word_len = np.mean([len(w) for w in words])

    stats_data.append({
        'Text': text[:40] + '...' if len(text) > 40 else text,
        'Words': len(words),
        'Characters': chars,
        'Avg Word Len': avg_word_len
    })

stats_df = pd.DataFrame(stats_data)
print(stats_df.to_string(index=False))

# 9. Visualization
print("\n=== 9. NLP Visualization ===")

fig, axes = plt.subplots(2, 2, figsize=(14, 10))

# Word frequency
word_freq = vocab.most_common(15)
words, freqs = zip(*word_freq)
axes[0, 0].barh(range(len(words)), freqs, color='steelblue')
axes[0, 0].set_yticks(range(len(words)))
axes[0, 0].set_yticklabels(words)
axes[0, 0].set_xlabel('Frequency')
axes[0, 0].set_title('Top 15 Most Frequent Words')
axes[0, 0].invert_yaxis()

# Sentiment distribution
sentiments = ['Positive', 'Negative', 'Positive', 'Negative', 'Positive']
sentiment_counts = Counter(sentiments)
axes[0, 1].pie(sentiment_counts.values(), labels=sentiment_counts.keys(),
              autopct='%1.1f%%', colors=['green', 'red'])
axes[0, 1].set_title('Sentiment Distribution')

# Document similarity heatmap
im = axes[1, 0].imshow(similarity_matrix, cmap='YlOrRd', aspect='auto')
axes[1, 0].set_xticks(range(len(docs)))
axes[1, 0].set_yticks(range(len(docs)))
axes[1, 0].set_xticklabels([f'Doc{i}' for i in range(len(docs))])
axes[1, 0].set_yticklabels([f'Doc{i}' for i in range(len(docs))])
axes[1, 0].set_title('Document Similarity Heatmap')
plt.colorbar(im, ax=axes[1, 0])

# Text length distribution
text_lengths = [len(t.split()) for t in texts]
axes[1, 1].hist(text_lengths, bins=5, color='coral', edgecolor='black')
axes[1, 1].set_xlabel('Number of Words')
axes[1, 1].set_ylabel('Frequency')
axes[1, 1].set_title('Text Length Distribution')
axes[1, 1].grid(True, alpha=0.3, axis='y')

plt.tight_layout()
plt.savefig('nlp_analysis.png', dpi=100, bbox_inches='tight')
print("\nNLP visualization saved as 'nlp_analysis.png'")

# 10. Summary
print("\n=== NLP Summary ===")
print(f"Texts processed: {len(texts)}")
print(f"Unique vocabulary: {len(vocab)} words")
print(f"Average text length: {np.mean([len(t.split()) for t in texts]):.2f} words")
print(f"Classification accuracy: {accuracy_score(labels, predictions):.4f}")

print("\nNatural language processing setup completed!")

よくあるNLPタスクとモデル

• 分類：DistilBERT、RoBERTa、ELECTRA
• NER：BioBERT、SciBERT、spaCyモデル
• 翻訳：MarianMT、M2M-100
• 要約：BART、Pegasus、T5
• QA：BERT、RoBERTa、DeBERTa

テキスト前処理パイプライン

1. 小文字化とクリーニング
2. トークン化
3. ストップワードの除去
4. レンマ化／ステミング
5. ベクトル化

ベストプラクティス

• 利用可能な場合は事前学習済みモデルを使用する
• タスク固有のデータでファインチューニングする
• 語彙に含まれない単語に対処する
• 効率化のためにバッチ処理する
• モデルのバイアスを監視する

成果物

• 学習済みNLPモデル
• テキスト分類の結果
• 抽出した固有表現
• パフォーマンス指標
• 可視化ダッシュボード
• 推論API