karpathy-practice-environments

スキル 14：LLM 教科書 + 練習環境

出典：https://x.com/karpathy/status/1885026028428681698 | https://x.com/karpathy/status/1960803117689397543 「LLM を学校に連れていく」|「RL のための環境」に関する投稿

基本原則

LLM は学生のように学びます。教科書、解答例、練習の場を用意しましょう。

Karpathy の洞察：LLM のトレーニングに欠けている要素は、パラメーターを増やすことではなく、エージェントが試し、失敗し、フィードバックを見て、再び試せる練習環境です。理論を読むだけでなく、問題集にも取り組む必要がある学生と同じです。

エージェントにもこの考え方を適用しましょう。何もないところに出力を生成させるだけでなく、自動採点を行いながらスキルを練習できる環境を構築します。

3 種類のトレーニングデータ

優れた学習環境には、次の 3 種類すべてが必要です。

1. EXPOSITION (pretrain equivalent)
   → Background knowledge, concepts, context
   → The "textbook chapter" the agent reads before practicing
   
2. WORKED EXAMPLES (SFT equivalent)
   → Complete input-output pairs with reasoning shown
   → "Here's a solved problem — learn the pattern"
   
3. PRACTICE PROBLEMS with feedback (RL equivalent)
   → Problems with verifiable correct answers
   → Automatic scoring so agent knows how it did
   → Enough variety that memorization doesn't work

練習の場を構築する（汎用テンプレート）

エージェントに習得させたいスキルがある場合は、次のようにします。

#!/usr/bin/env python3
"""
Practice Gym for: [SKILL_NAME]
Karpathy-style RL environment for agent skill development
"""

import json
import random
from typing import Callable

class PracticeGym:
    """
    A practice environment where an agent can repeatedly attempt
    a task and receive automatic feedback.
    """
    
    def __init__(self, task_generator: Callable, scorer: Callable):
        self.task_generator = task_generator  # generates new practice problems
        self.scorer = scorer                   # returns 0.0-1.0 score for an attempt
        self.history = []
    
    def sample_task(self):
        """Generate a new practice problem."""
        return self.task_generator()
    
    def evaluate(self, task, attempt):
        """Score an agent's attempt. Returns dict with score + feedback."""
        score = self.scorer(task, attempt)
        result = {
            "task": task,
            "attempt": attempt,
            "score": score,
            "passed": score >= 0.8,
        }
        self.history.append(result)
        return result
    
    def summary(self):
        """Summarize performance across all attempts."""
        if not self.history:
            return "No attempts yet."
        scores = [r["score"] for r in self.history]
        return {
            "total_attempts": len(scores),
            "avg_score": sum(scores) / len(scores),
            "pass_rate": sum(1 for s in scores if s >= 0.8) / len(scores),
            "recent_trend": "improving" if scores[-1] > scores[0] else "declining"
        }

練習環境の設計プロンプト

どのようなスキルにも対応した練習の場を構築します。

Design a practice environment for training an LLM agent to [SKILL].

Skill description: [WHAT THE AGENT SHOULD GET BETTER AT]
Current performance: [HOW WELL IT DOES NOW]
Target performance: [WHAT GOOD LOOKS LIKE]

Design the gym with:

1. TASK GENERATOR
   - Input format: [what the task looks like]
   - Variation parameters: [what changes between tasks]
   - Difficulty levels: [easy / medium / hard criteria]
   - Example task: [concrete example]

2. SCORER  
   - What makes an attempt correct? (exact match / rubric / functional test)
   - Score breakdown: [what earns partial credit]
   - Automatic vs human evaluation: [which parts can be automated]
   - Example: good attempt vs bad attempt with scores

3. CURRICULUM
   - Start: [simplest tasks to build confidence]
   - Progress: [how to increase difficulty as performance improves]
   - Mastery criterion: [when is the skill "learned"?]

4. FEEDBACK FORMAT
   What should the agent receive after each attempt?
   - Score (0-1)
   - Specific error: [what went wrong]
   - Hint for next attempt: [one actionable tip]

クイック評価ループのプロンプト

練習問題のセットを使ってエージェントをすばやくテストします。

Run an evaluation loop on this agent capability.

Agent task: [WHAT THE AGENT IS SUPPOSED TO DO]

Test cases:
1. Input: [test input 1]
   Expected output: [expected 1]
   
2. Input: [test input 2]
   Expected output: [expected 2]
   
3. Input: [test input 3]
   Expected output: [expected 3]

For each test case:
1. Attempt the task
2. Compare to expected output
3. Score: PASS / PARTIAL / FAIL
4. Explain the error (if any) in one sentence

Final report:
- Pass rate: N/3
- Most common failure mode: [pattern in errors]
- Suggested improvement: [one specific fix]

カリキュラム設計テンプレート

段階的な学習の進行を構築します。

Design a learning curriculum for an agent to master [SKILL].

Like Karpathy's "take LLMs to school" approach — structure it as:

WEEK 1 — Foundations (exposition):
- Core concepts to internalize
- 3-5 worked examples with full reasoning traces
- Quiz: 5 basic problems with answers

WEEK 2 — Pattern Recognition (practice):
- 20 practice problems, graduated difficulty
- Automatic scoring criteria
- Common error analysis

WEEK 3 — Generalization (RL-style):
- Novel problems the agent hasn't seen
- Real-world variants
- Adversarial examples (edge cases designed to fail)

Mastery test: [describe the final eval that confirms the skill is learned]

一般的なスキルの採点基準

コード生成

Score 1.0: Correct output, handles edge cases, clean style
Score 0.8: Correct output, misses 1 edge case
Score 0.5: Core logic correct, fails on some inputs
Score 0.2: Wrong approach but partially useful
Score 0.0: Doesn't compile or clearly wrong

推論／分析

Score 1.0: Correct conclusion + correct reasoning chain + appropriate confidence
Score 0.8: Correct conclusion, minor reasoning gap
Score 0.5: Correct conclusion, wrong reasoning
Score 0.2: Wrong conclusion but shows relevant knowledge
Score 0.0: Wrong conclusion, no relevant reasoning

指示への準拠

Score 1.0: All instructions followed, output matches spec exactly
Score 0.8: Minor deviation from spec, intent preserved
Score 0.5: Core task done, 1-2 instructions ignored
Score 0.2: Significant departure from instructions
Score 0.0: Instructions ignored entirely

要点をまとめた環境仕様

Karpathy のパターン：誰でも練習の場を構築できるよう、実装ではなく環境仕様を公開します。

Write a Gist-style environment spec for [SKILL_GYM].

Format:
# [GYM_NAME] — Practice Environment Spec
## What this trains
## Task format
## Scoring criteria  
## Sample task + ideal response
## Curriculum (3 stages)
## How to evaluate mastery

This is a spec, not code. Anyone with an LLM should be able to implement it.

ワークフロー

ワークフローに属します：月次チェックアップ（第 3 ステップ）

位置	上流	下流
第 3 ステップ（練習）	karpathy-understanding-first（盲点の特定後）	karpathy-education-first（練習の成果を教材化）

全体の流れ：meta-reflection → understanding-first → practice-environments → education-first

プロンプトの契約

Build a practice environment for <SKILL_OR_CAPABILITY>. Include: 1) Exposition — what this skill is and why it matters (3-5 sentences), 2) Worked examples — 2-3 complete input→output demonstrations, 3) Practice tasks — 5 exercises with increasing difficulty, 4) Automatic grading — for each task define pass/fail criteria that can be checked without human judgment, 5) Retry loop — if failed, what feedback to give and how to adjust difficulty.

検証チェックリスト

• 練習環境には明確なスキルの定義がある（漠然と「上達する」ではない）
• 解答例が典型的なケースと境界ケースをカバーしている
• 採点基準に人の判断が不要である（自動化されている）
• 失敗したときに具体的なフィードバックがある（単に「間違いです」ではない）
• 難易度が段階的に上がる（いきなり最終段階に進まない）
• エージェントは合格するまで自律的に再試行できる