experiment-design

Experiment Design Methodology

You are helping a researcher design rigorous experiments. Follow this methodology systematically.

Step 1: Understand the Research Question

Before designing any experiment:

• Ask what specific hypothesis or claim the experiment should support
• Identify the dependent variable (metric) and independent variables (factors)
• Clarify the baseline: what is the current best result or default configuration?

Step 2: Single-Variable Isolation

Every ablation study must change exactly ONE variable at a time. For each factor:

1. Define the factor — what is being varied (e.g., loss function, learning rate, architecture component)
2. List levels — all values this factor will take (e.g., CE, focal, VAR)
3. Fix everything else — document what stays constant (seed, data split, epochs, hardware)
4. Predict outcome — before running, state what you expect and why

Template for each ablation row:

| Run ID | Factor | Value | Fixed Config | Expected Outcome |
|--------|--------|-------|-------------|-----------------|

Step 3: Experiment Matrix

For multi-factor studies, use a structured matrix:

1. Full factorial — if factors are few (≤3) and levels are few (≤3 each)
2. Sequential elimination — if factors are many: run single-factor ablations first, then combine winners
3. Latin square — if full factorial is too expensive: sample representative combinations

Always calculate total runs before committing:

Total runs = product of all factor levels
GPU hours = total runs × hours_per_run

Step 4: Resource Estimation

For each experiment plan, estimate:

• GPU hours: runs × time_per_run (check with user's hardware)
• API costs: if using external APIs (Gemini, OpenAI), estimate tokens × price
• Wall clock time: accounting for sequential dependencies and GPU availability
• Storage: checkpoint sizes × number of runs

Flag if total cost exceeds reasonable bounds and suggest prioritization.

Step 5: Config Stub Generation

Generate configuration stubs that match the user's existing config format. Read existing configs first to match:

• File format (YAML, JSON, TOML)
• Key naming conventions
• Directory structure for outputs
• Logging/tracking integration (wandb, neptune, tensorboard)

Step 6: Execution Plan

Create a concrete execution plan:

1. Order runs by dependency (baselines first, then ablations)
2. Identify which runs can be parallelized across GPUs
3. Create a shell script or batch runner matching the project's existing patterns
4. Include checkpointing strategy for long runs

Step 7: Analysis Plan

Before running, define how results will be analyzed:

• Which metrics to compare (primary + secondary)
• Statistical significance test if applicable (paired t-test, bootstrap CI)
• How to handle failed/crashed runs
• Visualization: what plots to generate (comparison tables, bar charts, learning curves)

Verification Checkpoints

Before finalizing the experiment plan:

• Each ablation changes exactly one variable
• Baseline is clearly defined and will be run with same setup
• Resource estimate is within budget
• Config stubs match existing project format
• Analysis plan is defined before execution begins
• Seeds are fixed for reproducibility

Output Format

Always produce:

1. Experiment matrix table — all runs with their configurations
2. Resource estimate — GPU hours, API costs, storage
3. Execution script — ready-to-run commands matching project conventions
4. Analysis plan — metrics, comparisons, visualizations