e2e-outside-in-test-generator

端到端外向内测试生成器

使用外向内测试方法，为全栈应用自动生成全面的 Playwright 测试。覆盖 7 个类别，生成 40 多项测试，发现真实 Bug，并通过迭代修复循环进行验证。

第 1 级：快速开始

用途

此技能会分析你的全栈应用，并生成完整的 Playwright 测试套件，以实现：

• 验证用户旅程，覆盖从浏览器到后端的完整流程
• 通过系统探索发现真实 Bug
• 确定性运行，不产生不稳定测试
• 开箱即用地集成到 CI

适用场景

在以下情况下启用此技能：

• 需要为全栈应用进行全面的浏览器测试
• 希望端到端验证关键用户流程
• 需要在重大重构前增加回归测试覆盖
• 正准备部署到生产环境

要求：项目必须同时包含前端（Next.js、React、Vue、Angular）和后端 API。

快速开始

# In your project root
$ claude

> add e2e tests

此技能会自动：

1. 分析你的应用（路由、API 端点、数据库架构）
2. 设置基础设施（Playwright 配置、测试辅助工具、种子数据）
3. 生成 40 多项测试，覆盖 7 个类别
4. 运行测试并修复失败（最多迭代 5 次）
5. 报告覆盖情况并提供建议

预期输出

e2e/
├── playwright.config.ts          # Playwright configuration
├── test-helpers/
│   ├── auth.ts                   # Authentication helpers
│   ├── navigation.ts             # Navigation helpers
│   ├── assertions.ts             # Custom assertions
│   └── data-setup.ts             # Test data management
├── fixtures/
│   ├── users.json                # Test user data
│   ├── products.json             # Test product data
│   └── seed.sql                  # Database seed script
├── happy-path/
│   ├── user-registration.spec.ts
│   ├── user-login.spec.ts
│   └── checkout-flow.spec.ts
├── edge-cases/
│   ├── invalid-inputs.spec.ts
│   └── boundary-conditions.spec.ts
├── error-handling/
│   ├── network-failures.spec.ts
│   └── validation-errors.spec.ts
├── performance/
│   ├── page-load-times.spec.ts
│   └── api-response-times.spec.ts
├── security/
│   ├── unauthorized-access.spec.ts
│   └── xss-protection.spec.ts
├── accessibility/
│   ├── keyboard-navigation.spec.ts
│   └── screen-reader.spec.ts
└── integration/
    ├── database-persistence.spec.ts
    └── api-integration.spec.ts

Total: 42 tests across 7 categories

运行测试

# Run all tests
npx playwright test

# Run specific category
npx playwright test e2e/happy-path

# Run in headed mode
npx playwright test --headed

# Run in debug mode
npx playwright test --debug

成功标准

生成完成后，测试套件将达到以下标准：

• ✓ 40 多项测试，覆盖全部 7 个类别
• ✓ 修复循环后通过率达到 100%
• ✓ 总执行时间少于 2 分钟
• ✓ 生成过程中至少发现 1 个真实 Bug
• ✓ 没有不稳定测试（测试数据具有确定性）

第 2 级：完整功能

5 个阶段

graph LR
    A[Phase 1: Analysis] --> B[Phase 2: Infrastructure]
    B --> C[Phase 3: Generation]
    C --> D[Phase 4: Fix Loop]
    D --> E[Phase 5: Coverage Audit]

阶段 1：技术栈分析

此技能会深入分析应用：

前端分析：

• 检测框架（Next.js、React、Vue、Angular）
• 绘制路由和页面
• 识别导航模式
• 提取交互元素

后端分析：

• 发现 API 端点（REST/GraphQL）
• 绘制数据模型及其关系
• 识别身份验证机制
• 检测验证规则

数据库分析：

• 提取架构和关系
• 识别必填字段
• 确定外键约束
• 绘制枚举类型

分析输出示例：

StackConfig(
    frontend_framework="nextjs",
    frontend_dir="app/",
    backend_framework="fastapi",
    api_base_url="http://localhost:8000/api",
    database_type="postgresql",
    auth_mechanism="jwt",
    routes=[
        Route(path="/", component="Home"),
        Route(path="/login", component="Login"),
        Route(path="/products", component="ProductList"),
        Route(path="/products/:id", component="ProductDetail"),
        Route(path="/checkout", component="Checkout")
    ],
    api_endpoints=[
        APIEndpoint(path="/auth/login", method="POST"),
        APIEndpoint(path="/auth/register", method="POST"),
        APIEndpoint(path="/products", method="GET"),
        APIEndpoint(path="/products/:id", method="GET"),
        APIEndpoint(path="/orders", method="POST")
    ],
    models=[
        Model(name="User", fields=["id", "email", "password"]),
        Model(name="Product", fields=["id", "name", "price", "stock"]),
        Model(name="Order", fields=["id", "user_id", "product_id", "quantity"])
    ]
)

阶段 2：基础设施设置

生成完整的测试基础设施：

Playwright 配置：

// e2e/playwright.config.ts
import { defineConfig, devices } from "@playwright/test";

export default defineConfig({
  testDir: "./e2e",
  fullyParallel: false, // CRITICAL: workers=1 for deterministic test data
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: 1, // NEVER > 1 to prevent data races
  reporter: "html",
  use: {
    baseURL: "http://localhost:3000",
    trace: "on-first-retry",
  },
  projects: [{ name: "chromium", use: { ...devices["Desktop Chrome"] } }],
  webServer: {
    command: "npm run dev",
    url: "http://localhost:3000",
    reuseExistingServer: !process.env.CI,
  },
});

测试辅助工具：

// e2e/test-helpers/auth.ts
export async function login(page: Page, email: string, password: string) {
  await page.goto("/login");
  await page.getByRole("textbox", { name: /email/i }).fill(email);
  await page.getByRole("textbox", { name: /password/i }).fill(password);
  await page.getByRole("button", { name: /sign in/i }).click();
  await page.waitForURL("/dashboard");
}

export async function logout(page: Page) {
  await page.getByRole("button", { name: /logout/i }).click();
  await page.waitForURL("/");
}

种子数据：

// e2e/fixtures/users.json
[
  {
    "email": "[email protected]",
    "password": "Test123!", // pragma: allowlist secret
    "name": "Test User",
    "role": "customer"
  },
  {
    "email": "[email protected]",
    "password": "Admin123!", // pragma: allowlist secret
    "name": "Admin User",
    "role": "admin"
  }
]

阶段 3：测试生成

为 7 个必备类别生成测试：

1. 正常流程测试（关键用户旅程）

// e2e/happy-path/user-registration.spec.ts
import { test, expect } from "@playwright/test";

test("user can register with valid credentials", async ({ page }) => {
  await page.goto("/register");

  await page.getByRole("textbox", { name: /email/i }).fill("[email protected]");
  await page.getByRole("textbox", { name: /password/i }).fill("Password123!");
  await page.getByRole("textbox", { name: /confirm password/i }).fill("Password123!");
  await page.getByRole("button", { name: /create account/i }).click();

  await expect(page).toHaveURL("/dashboard");
  await expect(page.getByText(/welcome/i)).toBeVisible();
});

2. 边界情况测试（边界条件）

// e2e/edge-cases/boundary-conditions.spec.ts
test("rejects password that is too short", async ({ page }) => {
  await page.goto("/register");

  await page.getByRole("textbox", { name: /email/i }).fill("[email protected]");
  await page.getByRole("textbox", { name: /password/i }).fill("123"); // Too short
  await page.getByRole("button", { name: /create account/i }).click();

  await expect(page.getByText(/password must be at least 8 characters/i)).toBeVisible();
});

3. 错误处理测试（失败场景）

// e2e/error-handling/network-failures.spec.ts
test("shows error when API is unavailable", async ({ page, context }) => {
  // Simulate network failure
  await context.route("**/api/**", (route) => route.abort());

  await page.goto("/products");

  await expect(page.getByText(/unable to load products/i)).toBeVisible();
  await expect(page.getByRole("button", { name: /retry/i })).toBeVisible();
});

4. 性能测试（速度验证）

// e2e/performance/page-load-times.spec.ts
test("homepage loads in under 2 seconds", async ({ page }) => {
  const startTime = Date.now();
  await page.goto("/");
  await page.waitForLoadState("networkidle");
  const loadTime = Date.now() - startTime;

  expect(loadTime).toBeLessThan(2000);
});

5. 安全测试（授权/XSS）

// e2e/security/unauthorized-access.spec.ts
test("redirects unauthenticated users from protected routes", async ({ page }) => {
  await page.goto("/dashboard");

  await expect(page).toHaveURL("/login");
  await expect(page.getByText(/please sign in/i)).toBeVisible();
});

6. 无障碍测试（符合 WCAG 标准）

// e2e/accessibility/keyboard-navigation.spec.ts
import { test, expect } from "@playwright/test";

test("login form is fully keyboard accessible", async ({ page }) => {
  await page.goto("/login");

  await page.keyboard.press("Tab"); // Focus email
  await page.keyboard.type("[email protected]");
  await page.keyboard.press("Tab"); // Focus password
  await page.keyboard.type("Test123!");
  await page.keyboard.press("Tab"); // Focus submit button
  await page.keyboard.press("Enter"); // Submit form

  await expect(page).toHaveURL("/dashboard");
});

7. 集成测试（数据库/API）

// e2e/integration/database-persistence.spec.ts
test("product order persists across sessions", async ({ page, context }) => {
  // Create order
  await login(page, "[email protected]", "Test123!");
  await page.goto("/products/123");
  await page.getByRole("button", { name: /add to cart/i }).click();
  await page.goto("/checkout");
  await page.getByRole("button", { name: /place order/i }).click();

  const orderNumber = await page.getByText(/order #\d+/).textContent();

  // Logout and login again
  await logout(page);
  await login(page, "[email protected]", "Test123!");

  // Verify order exists
  await page.goto("/orders");
  await expect(page.getByText(orderNumber!)).toBeVisible();
});

阶段 4：修复循环

通过迭代调试自动修复失败的测试：

修复循环流程：

1. 运行测试 → 收集失败项
2. 分析失败项 → 对问题分类（定位器、时序、数据、逻辑）
3. 应用修复 → 根据问题类型更新测试
4. 重新运行测试 → 验证修复结果
5. 重复执行 → 最多迭代 5 次

常见修复模式：

// Before: Flaky locator
await page.click(".submit-button");

// After: Role-based locator
await page.getByRole("button", { name: /submit/i }).click();

// Before: Race condition
await page.click("#login");
await page.fill("#email", "[email protected]");

// After: Wait for navigation
await page.click("#login");
await page.waitForLoadState("networkidle");
await page.fill("#email", "[email protected]");

// Before: Hardcoded data
await page.fill("#quantity", "5");

// After: Dynamic data from fixtures
const product = await getTestProduct();
await page.fill("#quantity", product.minQuantity.toString());

修复循环结果：

Iteration 1: 42 tests, 8 failures
  - Fixed 5 locator issues
  - Fixed 2 timing issues
  - Fixed 1 data issue

Iteration 2: 42 tests, 3 failures
  - Fixed 2 locator issues
  - Fixed 1 timing issue

Iteration 3: 42 tests, 0 failures ✓

Fix loop completed in 3 iterations.

阶段 5：覆盖率审查

生成全面的覆盖率报告和建议：

覆盖率报告：

## Test Coverage Report

### Tests Generated: 42

- Happy Path: 12 tests
- Edge Cases: 8 tests
- Error Handling: 6 tests
- Performance: 4 tests
- Security: 5 tests
- Accessibility: 4 tests
- Integration: 3 tests

### Routes Covered: 12/15 (80%)

Uncovered routes:

- /admin/settings (requires admin role)
- /api/webhooks/\* (external integrations)
- /debug/\* (development-only)

### API Endpoints Covered: 18/22 (82%)

Uncovered endpoints:

- POST /api/admin/users (admin-only)
- DELETE /api/products/:id (soft delete, needs verification)
- GET /api/analytics (complex aggregations)
- POST /api/webhooks/stripe (external trigger)

### Bugs Found: 2

1. **CRITICAL**: Login form allows SQL injection via email field
   - Location: app/login/page.tsx:45
   - Test: e2e/security/sql-injection.spec.ts

2. **MEDIUM**: Checkout fails with international phone numbers
   - Location: lib/validation.ts:12
   - Test: e2e/edge-cases/international-phone.spec.ts

### Recommendations

1. Add admin-role tests (requires admin user setup)
2. Add webhook integration tests (requires test webhook server)
3. Add load testing for high-traffic endpoints
4. Increase accessibility coverage (current: 9%, target: 15%)

自定义选项

自定义定位器策略：

// e2e/playwright.config.ts
export default defineConfig({
  use: {
    // Prioritize test IDs
    testIdAttribute: "data-testid",
  },
});

自定义测试类别：

# Add custom category during generation
custom_categories = [
    "happy-path",
    "edge-cases",
    "error-handling",
    "performance",
    "security",
    "accessibility",
    "integration",
    "custom-business-rules"  # Your custom category
]

自定义种子数据：

// e2e/fixtures/custom-data.json
{
  "scenarios": [
    {
      "name": "bulk-order",
      "users": [...],
      "products": [...],
      "expected_discount": 0.15
    }
  ]
}

配置

环境变量：

# .env.test
DATABASE_URL=postgresql://test:test@localhost:5432/testdb  # pragma: allowlist secret
API_BASE_URL=http://localhost:8000/api
FRONTEND_URL=http://localhost:3000
[email protected]
ADMIN_PASSWORD=Admin123!  # pragma: allowlist secret

Playwright 选项：

// e2e/playwright.config.ts
export default defineConfig({
  // Test execution
  workers: 1, // MANDATORY: prevents data races
  fullyParallel: false, // Sequential execution
  retries: process.env.CI ? 2 : 0,

  // Timeouts
  timeout: 30000, // Per-test timeout
  expect: { timeout: 5000 }, // Assertion timeout

  // Artifacts
  use: {
    screenshot: "only-on-failure",
    video: "retain-on-failure",
    trace: "on-first-retry",
  },
});

第 3 级：高级用法

与其他技能集成

与 test-gap-analyzer 集成：

> analyze test gaps and generate e2e tests to fill them

该技能：

1. 运行 test-gap-analyzer，识别未覆盖的流程
2. 根据差距分析确定测试生成的优先级
3. 聚焦高风险的未覆盖区域

与 shadow-testing 集成：

> generate e2e tests and run shadow testing against production

该技能：

1. 生成完整的测试套件
2. 使用生产环境镜像配置 shadow-testing
3. 在两个环境中运行测试
4. 报告差异

与 qa-team methodology（前身为 outside-in-testing）集成：

该技能遵循由外向内的测试方法：

• 从面向用户的 UI 开始
• 通过所有层进行测试（UI → API → DB）
• 验证端到端契约是否得到遵循
• 不使用模拟（真实集成）

自定义模板开发

模板结构：

# e2e/templates/custom-template.py
CUSTOM_TEST_TEMPLATE = """
import {{ test, expect }} from '@playwright/test';
import {{ {helpers} }} from '../test-helpers/{helper_module}';

test.describe('{feature_name}', () => {{
  test.beforeEach(async ({{ page }}) => {{
    // Setup
    {setup_code}
  }});

  test('{test_description}', async ({{ page }}) => {{
    // Arrange
    {arrange_code}

    // Act
    {act_code}

    // Assert
    {assert_code}
  }});
}});
"""

使用自定义模板：

from e2e_outside_in_test_generator.template_manager import TemplateManager

template_manager = TemplateManager()
template_manager.register_template("custom-flow", CUSTOM_TEST_TEMPLATE)

test_code = template_manager.render("custom-flow", {
    "feature_name": "Payment Processing",
    "helpers": "login, checkout",
    "helper_module": "payment",
    "setup_code": "await setupPaymentGateway();",
    "arrange_code": "const cart = await createCart();",
    "act_code": "await processPayment(cart);",
    "assert_code": "await expect(page.getByText(/payment successful/i)).toBeVisible();"
})

高级定位器策略

优先级层次：

1. 基于角色（首选）：getByRole('button', { name: /submit/i })
2. 用户可见文本：getByText(/welcome/i)
3. 测试 ID：getByTestId('submit-button')
4. CSS 选择器（最后手段）：locator('.submit-button')

自定义定位器构建器：

// e2e/test-helpers/locators.ts
export function findByDataAttribute(page: Page, attr: string, value: string) {
  return page.locator(`[data-${attr}="${value}"]`);
}

export function findByAriaLabel(page: Page, label: string) {
  return page.locator(`[aria-label*="${label}" i]`);
}

// Usage in tests
await findByDataAttribute(page, "action", "submit").click();

性能优化

测试执行时间：

// Group fast tests together
test.describe("Quick smoke tests", () => {
  test("homepage renders", async ({ page }) => {
    /* <1s */
  });
  test("navigation works", async ({ page }) => {
    /* <1s */
  });
});

// Isolate slow tests
test.describe("Full checkout flow @slow", () => {
  test("complete purchase", async ({ page }) => {
    /* 10s */
  });
});

运行配置：

// package.json
{
  "scripts": {
    "test:e2e": "playwright test",
    "test:e2e:quick": "playwright test --grep-invert @slow",
    "test:e2e:full": "playwright test",
    "test:e2e:smoke": "playwright test e2e/happy-path"
  }
}

CI/CD 集成

GitHub Actions：

# .github/workflows/e2e.yml
name: E2E Tests

on: [push, pull_request]

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: "18"

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright
        run: npx playwright install --with-deps

      - name: Run E2E tests
        run: npm run test:e2e
        env:
          DATABASE_URL: ${{ secrets.TEST_DATABASE_URL }}

      - uses: actions/upload-artifact@v3
        if: failure()
        with:
          name: playwright-report
          path: playwright-report/

故障排除

常见问题：

1. 并行执行导致测试不稳定

   • 原因：workers > 1 会导致数据库竞争条件
   • 解决方法：确保 playwright.config.ts 中设置了 workers: 1

2. 定位器超时

   • 原因：元素尚未渲染
   • 解决方法：添加显式等待：await page.waitForLoadState('networkidle')

3. 测试数据冲突

   • 原因：共享测试数据库但未进行清理
   • 解决方法：使用 test.beforeEach() 重置数据库状态

4. 身份验证失败

   • 原因：会话超时或凭据不正确
   • 解决方法：在 e2e/fixtures/users.json 中验证测试凭据

调试命令：

# Run single test in headed mode
npx playwright test e2e/happy-path/login.spec.ts --headed

# Run with debugger
npx playwright test --debug

# Generate trace for failing test
npx playwright test --trace on

# View trace
npx playwright show-trace trace.zip

最佳实践

1. 将 workers 保持为 1 - 确定性测试数据下切勿增加
2. 使用基于角色的定位器 - 最能适应 UI 变化
3. 使用小型种子数据集 - 最多 10-20 条记录，以确保可预测性
4. 使用显式等待 - 通过 waitForLoadState() 避免竞争条件
5. 在测试之间进行清理 - 使用 beforeEach 重置状态
6. 隔离测试 - 每项测试都应独立运行
7. 使用有意义的断言 - 测试业务逻辑，而非实现方式

成功指标

实现此技能后，应达到以下指标：

• 测试数量：40+ 项测试（全部 7 个类别）
• 通过率：修复循环后达到 100%
• 执行时间：总计 <2 分钟
• 错误检测：至少发现 1 个真实错误
• 覆盖率：覆盖 ≥80% 的路由和端点
• 不稳定率：0%（确定性）
• CI 集成：每次 PR 都通过检查

---

另请参阅：

• README.md - 开发者文档
• examples.md - 使用示例
• reference.md - API 参考
• patterns.md - 常见模式