netra-evaluation-setup

Netra Evaluation 设置 — 由规范驱动的数据集规划

此技能会读取 QA/开发规范文档，并生成一份完整、可编辑的评估计划——包括数据集条目、评估器、变量映射和通过标准——随后在用户批准后于 Netra 中创建所有内容。

工作流

按顺序执行以下阶段。完成当前阶段后再进入下一阶段。

阶段 0：验证提供方配置

在开始任何规划或创建工作之前，确认组织已配置默认提供方。

通过 MCP 调用 netra_get_default_llm_configuration。

• 如果返回有效配置（包含 provider、model 和 providerConfigurationId），则进入阶段 1。
• 如果返回 null 或错误，立即停止并告知用户：

⚠️ No default provider configured.

Evaluation and dataset creation require a default LLM provider in your organization.
Please configure a default provider in the Netra dashboard before proceeding:

  Settings → Providers → Set Default

Once configured, run this skill again.

在确认默认提供方有效之前，不要继续任何后续阶段。

阶段 0.5：检测项目语言

生成任何 SDK 代码（任务实现、钩子脚手架、执行示例）之前，确定项目使用的是 Python 还是 TypeScript/JavaScript。按以下顺序检查项目根目录：

信号文件	语言
pyproject.toml、setup.py、requirements.txt、Pipfile	Python
package.json、tsconfig.json、bun.lockb	TypeScript / JavaScript

如果两者都存在（monorepo），询问用户正在使用哪个子项目。如果两者都不存在，也询问用户。

从此处开始，只生成检测到的语言对应的 SDK 示例。切勿混用 Python 和 TypeScript 约定。

此技能的语言专属约定：

概念	Python	TypeScript
初始化	Netra.init(...)（同步）	await Netra.init(...)
模拟 API	Netra.simulation.run_simulation(...)	await Netra.simulation.runSimulation({...})
评估 API	Netra.evaluation.run_test_suite(...)	await Netra.evaluation.runTestSuite({...})
钩子字段	before_all、before_each、before、after、after_each、after_all、setup_context、session_id、dataset_item_id	beforeAll、beforeEach、before、after、afterEach、afterAll、setupContext、sessionId、datasetItemId
钩子对象	SimulationHooks(before_all=..., before_each=..., before={...})	const hooks: SimulationHooks = { beforeAll: ..., beforeEach: ..., before: {...} }
钩子描述	对每个钩子函数设置 fn.description = "..."（必需，≤ 200 个字符）	对每个钩子函数设置 (fn as any).description = "..."（必需，≤ 200 个字符）

阶段 1：摄取规范

1. 阅读用户提供的规范文档（文件路径、粘贴的文本或 URL）。
2. 提取每项可测试的要求。对于每项要求，确定：
   • 场景名称 — 简洁易懂地描述要验证的行为
   • 类别 — 正向路径、负向、边界情况、安全、性能
   • 输入 — 用于测试此场景的用户消息或触发条件
   • 预期行为 — 正确的 Agent 响应或操作

场景命名指南

根据所测试的行为，生成有意义的场景名称。

良好示例：

• 用户登录成功
• 使用有效邮箱重置密码
• 商品搜索返回相关结果
• 拒绝提示注入尝试
• 处理空用户查询
• 客户申请退款
• 检索账户余额
• 多步骤旅行规划对话

避免使用通用名称，例如：

• 场景 1
• 测试用例 1
• 项目 3
• 验证测试
• 顺利路径测试

3. 提出规范可能遗漏的其他测试用例：
   • 未明确列出的边界条件
   • 常见的对抗性输入（提示注入、越狱尝试）
   • 性能边界情况（超长输入、空输入、特殊字符）
   • 如果 Agent 支持多步推理，则添加多步推理场景
4. 展示完整列表，并请用户在继续之前确认。

阶段 2：了解 Agent Span 结构

构建一个 span 映射 — 即 Agent 在运行时生成的具名 span 列表。

主要方法 — 代码分析：

使用 Grep 和 Read 工具，在 Agent 代码库中搜索插桩模式。

Patterns to search for:

Python Netra decorators:
  @workflow, @agent, @task, @span          → span name from name= parameter
  Netra.start_span("name", ...)            → span name from first argument

Python tool definitions:
  @tool def func_name(...)                 → span name = function name
  class MyTool(BaseTool): name = "..."     → span name from class attribute

TypeScript Netra decorators:
  @Workflow(), @Agent(), @Task()           → span name from decorator argument
  netra.startSpan("name", ...)             → span name from first argument

Framework-specific:
  CrewAI Agent/Task definitions            → span name from name field
  LangGraph node definitions               → span name from node name
  Claude Agent SDK tool definitions        → span name from tool name

对于找到的每个 span，记录：

• name — span 在跟踪记录中显示的名称
• type — 工作流、Agent、任务、工具或生成
• output description — 函数返回的内容（根据代码推断）
• position in flow — 相对于其他 span 的调用顺序

备用方法 — 示例跟踪记录：

如果代码分析得出的结果不足（未找到装饰器、span 名称是动态生成的）：

1. 询问用户：“我在代码中没有找到足够的 span 定义。这个 Agent 是否至少运行过一次，且启用了 Netra 插桩？”
2. 如果是，使用 netra_query_traces 查找近期跟踪记录，然后使用 netra_get_trace_by_id 获取其 span。
3. 根据实际跟踪记录数据构建 span 映射，包括 span 名称、父子关系、输入和输出。

此阶段的输出：

Agent Flow:
  {root-span-name} ({type})
  ├── {child-span-1} ({type}) → {output description}
  ├── {child-span-2} ({type}) → {output description}
  │   └── {nested-span} ({type}) → {output description}
  └── {child-span-3} ({type}) → {output description}

展示给用户，并确认 span 名称是否准确。

阶段 3：选择评估器并配置映射

对于阶段 1 中的每个测试用例，确定要附加哪些评估器，以及如何连接它们的变量。

步骤 3a：获取可用的评估器

先尝试使用 MCP 工具。如果不可用，请使用内嵌参考。

Try:    netra_get_evaluator_library (all categories)
        netra_list_evaluators (custom evaluators in the project)

Fallback: Read references/evaluator-library.md for the full catalog

步骤 3b：将评估器与测试用例匹配

应用以下映射规则：

场景类别	推荐的评估器
正常路径 — 预期得到正确的回答	答案相关性、答案正确性或语义相似度
正常路径 — 预期执行正确的操作	目标准确性、工具正确性
负面场景 — 应妥善拒绝或处理错误	答案相关性、简洁性
边缘情况 — 边界行为	答案正确性、忠实度
安全 — 提示词注入、越狱	毒性、主题遵循度
性能 — 延迟／成本上限	延迟、成本、Token 用量
RAG — 检索质量至关重要	上下文相关性、上下文精确度、忠实度、幻觉

这些只是初步建议。务必考虑具体场景的含义——只有当评估器的用途与测试用例的验证目标一致时，才应使用该评估器。

步骤 3c：确定可配置变量

某些评估器需要用户提供的配置，无法根据对话上下文或元数据自动解析。这些变量在评估器库中定义为 configurableVariables。如果可配置变量留空，评估器将失败或给出不可靠的评分（例如，显示“无法评估”并降低评分）。

需要配置变量的评估器：

评估器	变量	类型	需要提供的内容
指南遵循度	assistant_instructions	string	提供给 AI Agent 的完整系统提示词或指令——它必须执行的任务、应遵循的行为、安全规则、升级处理指南
指南遵循度	assistant_constraints	string	AI Agent 必须遵守的约束——它绝不能做的事情、边界、禁止的操作。也可从 metadata.assistant_constraints 自动解析，但可配置值优先
事实准确性（多轮）	reference_facts	json	Agent 应准确传达的事实——产品详情、政策、价格、日期。也可从 metadata.reference_facts 自动解析，但可配置值优先

会话评估器的变量解析优先级：

1. evaluatorConfigs[].configuredValues（优先级最高 — 用户提供）
2. 会话元数据自动解析（例如 metadata.assistant_constraints、metadata.reference_facts）
3. 空字符串（后备值 — 会导致评估失败）

assistant_instructions 没有自动解析路径——必须通过 evaluatorConfigs 提供。

当推荐的评估器包含可配置变量时：

• 请用户提供这些变量的值
• 在计划中为每个评估器添加“可配置变量”部分，并列出这些变量
• 如果用户暂时无法提供这些值，请在计划中将其标记为 ⚠️ NEEDS CONFIGURATION

步骤 3d：配置变量映射

对于附加到测试用例的每个评估器，将所有必需变量映射到数据源。

response / actual_output 变量的决策逻辑：

IF the evaluator checks the final user-facing answer
  → source: taskOutput
  → expression: "taskOutput"

IF the evaluator checks a specific intermediate step
  → source: span output
  → find the span from Phase 2 whose output is relevant
  → expression: "spans[?name=='{spanName}'] | [0].output"

IF the evaluator checks tool usage
  → source: trace tools
  → expression: "trace.tools"

reference / expected 变量的决策逻辑：

IF ground truth is available in the dataset item
  → source: expectedOutput
  → expression: "expectedOutput"

IF reference comes from a specific span
  → expression: "spans[?name=='{spanName}'] | [0].output"

context 的决策逻辑（RAG 评估器）：

→ source: retriever span output
→ expression: "spans[?name=='{retriever-span}'] | [0].output"

性能评估器的决策逻辑：

actual_latency → expression: "trace.latency"
actual_cost    → expression: "trace.cost"
actual_tokens  → expression: "trace.tokens"
expected_*     → source: literal value from the spec

如果无法有把握地确定映射，请在计划中将其标记为 ⚠️ NEEDS CONFIGURATION，并说明原因。

阶段 3e：设置通过标准和模型

• 除非规范另有说明，否则使用评估器的默认通过标准。
• 如果用户指定了模型偏好，请将其应用于计划中列出的所有 LLM 评估器。

如果需要创建评估器，计划应显示将使用的模型配置。

提供商／模型解析在创建评估器时进行，而不是在推荐评估器时进行。

单项提供商配置

多轮数据集项需要包含 providerConfig，因为模拟会通过由模型支持的任务实现来执行。

{
  "provider_id": "{providerConfigurationId}",
  "model": "{model}"
}

使用 netra_get_default_llm_configuration 检索组织的默认提供商配置。

规则：

• 对于多轮数据集，每个数据集项都必须包含 providerConfig。
• 如果规范未定义模型，请使用组织的默认提供商配置。
• 如果场景明确要求使用其他模型，请相应设置该项的 providerConfig，以覆盖默认配置。
• 对于单轮数据集，providerConfig 是可选的，仅当某项需要使用不同于评估器默认模型时才应提供。

阶段 3f：确定执行策略

生成最终计划前，请确定规范描述的是单轮评估还是多轮模拟。

单轮评估

如果规范描述的是单次请求-响应交互：

• 将 Turn Type 设为 single
• 使用 Netra.evaluation.run_test_suite() 执行

评估执行时，绝不可直接调用 Agent。

原因：

• 自动捕获跟踪 ID
• 评估器可以访问跟踪和跨度输出
• 基于跨度的变量映射能够正常工作
• 评估结果会继续关联到执行跟踪

执行示例：

Netra.evaluation.run_test_suite(
    dataset_id="dataset-123",
    ...
)

多轮模拟

如果规范描述的是对话、工作流、支持交互、故障排查流程或任何多步骤交流：

• 将 Turn Type 设为 multi
• 将 Eval Type 设为 session——必须显式传递给 netra_create_evaluator
• 使用与语言对应的模拟 API：
  • Python：Netra.simulation.run_simulation()
  • TypeScript：await Netra.simulation.runSimulation({...})

模拟执行时，绝不可直接调用 Agent。

用户必须提供与 Netra Simulation 兼容的任务实现。

Python 示例：

class MyAgentTask(BaseTask):
    def __init__(self, agent):
        self.agent = agent

    def run(
        self,
        message: str,
        session_id: str | None = None,
        files: list[ProcessedFile] | None = None,
        setup_context: dict | None = None,  # populated by before_all / before hooks
    ) -> TaskResult:
        ctx = setup_context or {}
        response = self.agent.chat(
            message,
            session_id=session_id,
            files=files,
            # Pass any setup data from hooks (e.g. auth token, customer ID)
            auth_token=ctx.get("auth_token"),
        )

        return TaskResult(
            message=response.text,
            session_id=session_id or "default",
        )

result = Netra.simulation.run_simulation(
    name="Customer Support Simulation",
    dataset_id="dataset-123",
    task=MyAgentTask(my_agent),
    context={"environment": "staging"},
    max_concurrency=5,
    hooks=hooks,  # optional SimulationHooks from Phase 3g
)

TypeScript 示例：

import { BaseTask } from "netra-sdk";
import type { ProcessedFile, TaskResult } from "netra-sdk";

class MyAgentTask extends BaseTask {
  constructor(private agent: any) {
    super();
  }

  async run(
    message: string,
    sessionId?: string | null,
    files?: ProcessedFile[] | null,
    setupContext?: Record<string, any> | null, // populated by beforeAll / before hooks
  ): Promise<TaskResult> {
    const ctx = setupContext || {};
    const response = await this.agent.chat(message, {
      sessionId,
      files,
      authToken: ctx.authToken,
    });
    return {
      message: response.text,
      sessionId: sessionId || "default",
    };
  }
}

const result = await Netra.simulation.runSimulation({
  name: "Customer Support Simulation",
  datasetId: "dataset-123",
  task: new MyAgentTask(myAgent),
  context: { environment: "staging" },
  maxConcurrency: 5,
  hooks, // optional SimulationHooks from Phase 3g
});

这样可以确保：

• 对话状态得以保持
• 会话 ID 得到跟踪
• 生成模拟跟踪
• 评估器可以访问对话历史和跨度输出
• 相关场景可以安全地共享状态，而不依赖顺序执行

回合类型检测规则

根据规范确定执行模式：

规范模式	回合类型
单次请求 → 单次响应	单轮
一次性工具执行	单轮
QA 验证	单轮
客户支持对话	多轮
销售对话	多轮
故障排查工作流	多轮
多步骤助手交互	多轮
角色扮演模拟	多轮

如果无法确定，请在生成计划前询问用户。

阶段 3g：识别场景关联并提出钩子

仅对多轮数据集应用此阶段。

数据集项通常包含彼此相关的场景——它们共享状态，或假设先前场景已经运行。由于 Netra 会并行执行场景（并发数可配置），无法保证按顺序执行。钩子允许用户在场景前后运行设置和清理代码，无需合并数据集项，也不依赖执行顺序。

步骤 1：检测场景关联

阅读计划中的每个场景，并标记以下类别的关联：

模式	描述	所需钩子
共享资源	场景 B 操作由场景 A 创建的实体（员工、账户、记录）	使用 before_all 创建一次该实体
身份验证状态	场景需要已登录的会话或令牌	使用 before_each 为每个场景获取新令牌；使用 before 配置场景专属凭据
数据填充	所有场景都需要相同的种子数据（目录、配置、参考记录）	使用 before_all 填充数据
外部服务状态	场景需要第三方服务处于特定状态	使用 before_all / after_all 进行设置和重置
通用的逐项设置	每个场景都需要相同的隔离步骤（新令牌、空购物车）	使用 before_each / after_each
需要隔离	只有部分场景需要清理状态	使用 before[dataset_item_id]
需要清理	场景中创建的资源必须在之后删除	使用 after / after_each 或 after_all

如果未发现关联，则不需要钩子。请在计划中明确说明这一点。

第 2 步：设计钩子策略

对于每种检测到的关联模式，决定使用哪个钩子级别：

情况	推荐的钩子	原因
创建所有场景都会使用的资源	before_all	只运行一次；比按场景创建成本更低
每个场景都需要相同的设置（新令牌、空购物车）	before_each	避免为每个项目重复注册相同的 before
仅为特定场景设置	before[dataset_item_id]	针对特定项目；接收合并后的 before_all + before_each 上下文
为每个场景清理	after_each	无需为每个项目注册即可进行通用清理
清理特定场景的资源	after[dataset_item_id]	在 after_each 之前运行；针对特定项目进行清理
删除在 before_all 中创建的共享资源	after_all	所有场景完成后运行一次

需要向用户说明的失败语义：

• 如果 before_all 失败 → 整次运行标记为失败；不执行任何场景
• 如果某个场景的 before_each 或 before[dataset_item_id] 失败 → 该场景标记为 prescript_failed；其他场景继续运行
• 如果某个原本成功的场景中 after / after_each 失败 → 该场景标记为 postscript_failed（两者都会始终尝试运行；错误会合并）。如果场景已失败或状态为 prescript_failed，则保留现有状态
• 如果 after_all 失败 → 已成功完成的场景标记为 postscript_failed；已失败或状态为 prescript_failed 的项目保留原状态
• 如果 before 钩子失败，after / after_each 仍会运行，并接收成功构建到最远阶段的 setup_context（例如，如果项目的 before 失败，则接收 before_all + before_each）

prescript_failed 的终态行为：

• prescript_failed 是一种终态——进入该状态的项目即已完成。SDK 的轮询循环会正确地将 prescript_failed 项目计为终态，不会无限期等待。
• prescript_failed 项目的 evalStatus 也会立即设置为 NOT_AVAILABLE——对于 Agent 从未运行的项目，绝不会将评估器加入队列。
• 如果所有项目最终都处于 failed 或 prescript_failed 状态（没有成功完成的项目），则整次运行的总体评估状态会变为 NOT_AVAILABLE。
• prescript_failed 项目的状态是稳定的——后续的批量失败处理（例如运行超时时）绝不会覆盖该状态。

postscript_failed 的终态行为：

• postscript_failed 是一种终态——对话已完成，但清理失败。
• 不会禁止评估——已完成对话的评估器仍然有效。
• postscript_failed 不计入“全部失败 → NOT_AVAILABLE”的汇总（与 prescript_failed 不同）。
• 状态是稳定的——后续的批量失败处理绝不会覆盖该状态。

如何确定用于钩子映射的数据集项目 ID：

生成计划时，你会创建带有特定 ID 的数据集项目。这些 ID 由 netra_create_dataset_item MCP 调用返回。创建完所有项目后，指示用户：

1. 将返回的数据集项目 ID 值映射到相应的场景
2. 将这些 ID 用作 SimulationHooks.before / SimulationHooks.after 的键
   • Python 的键：dataset_item_id 字符串
   • TypeScript 的键：相同的 ID 字符串（SDK 类型字段名为 datasetItemId）

Python 映射示例：

# Step 1: Create dataset items and capture their IDs
refund_item = netra_create_dataset_item(...)  # returns {"id": "item-abc-123", ...}
balance_item = netra_create_dataset_item(...)  # returns {"id": "item-xyz-789", ...}

# Step 2: Map hooks using the actual dataset_item_id values
hooks = SimulationHooks(
    before_all=before_all,
    before_each=before_each,  # optional: runs for every scenario
    before={
        "item-abc-123": setup_refund_scenario,  # Use actual ID from step 1
        "item-xyz-789": setup_balance_scenario,
    },
    after={
        "item-abc-123": teardown_refund_scenario,
        "item-xyz-789": teardown_balance_scenario,
    },
    after_each=after_each,  # optional: runs for every scenario
    after_all=after_all,
)

TypeScript 映射示例：

// Step 1: Create dataset items and capture their IDs
const refundItem = /* netra_create_dataset_item(...) → { id: "item-abc-123", ... } */;
const balanceItem = /* netra_create_dataset_item(...) → { id: "item-xyz-789", ... } */;

// Step 2: Map hooks using the actual datasetItemId values
const hooks: SimulationHooks = {
  beforeAll,
  beforeEach, // optional: runs for every scenario
  before: {
    "item-abc-123": setupRefundScenario,
    "item-xyz-789": setupBalanceScenario,
  },
  after: {
    "item-abc-123": teardownRefundScenario,
    "item-xyz-789": teardownBalanceScenario,
  },
  afterEach, // optional: runs for every scenario
  afterAll,
};

步骤 3：生成钩子框架代码

对于每个已识别的钩子，请使用检测到的项目语言（阶段 0.5）生成框架代码，其中包括：

• 与所需钩子级别匹配的函数签名
• 对钩子功能的简短说明，通过每个钩子函数上的 .description 设置（≤ 200 个字符）。切勿省略此项，否则 Netra UI 中会显示 description: null。
  • Python：fn.description = "..."
  • TypeScript：(fn as any).description = "..."
• 实际实现的占位内容（标记为 # TODO / // TODO）
• SimulationHooks 连接配置
• 更新后的模拟运行调用，并传入 hooks

TypeScript 项目不得生成 Python 框架代码，反之亦然。 生成的钩子必须附带 .description。

Python 钩子函数签名：

# before_all: no args → returns dict (shared context) or None
def before_all() -> dict | None:
    ...
before_all.description = "One-line description for the Netra UI."

# before_each: receives shared_context → returns dict merged into setup_context (runs for every item)
def before_each(shared_context: dict | None) -> dict | None:
    ...
before_each.description = "One-line description for the Netra UI."

# before: dict keyed by dataset_item_id; each receives merged context (before_all + before_each)
def setup_scenario_a(shared_context: dict | None) -> dict | None:
    ...
setup_scenario_a.description = "One-line description for the Netra UI."

def setup_scenario_b(shared_context: dict | None) -> dict | None:
    ...
setup_scenario_b.description = "One-line description for the Netra UI."

# after: dict keyed by dataset_item_id; each receives result + setup_context
def teardown_scenario_a(result: dict, setup_context: dict | None) -> None:
    ...
teardown_scenario_a.description = "One-line description for the Netra UI."

# after_each: receives result + setup_context (runs for every item, after item-specific after)
def after_each(result: dict, setup_context: dict | None) -> None:
    ...
after_each.description = "One-line description for the Netra UI."

# after_all: receives aggregated results dict and shared_context (before_all only) → returns None
def after_all(results: dict, shared_context: dict | None) -> None:
    ...
after_all.description = "One-line description for the Netra UI."

如果用户的设置代码是异步的，所有 Python 钩子都可以是异步的（async def）。

**必需：**上述每个 Python 钩子都必须设置 .description。省略此项会导致 lifecycleHooks 中的 description: null。

TypeScript 钩子函数签名：

// beforeAll: no args → returns shared context object or null/void
async function beforeAll(): Promise<Record<string, any> | null | void> {
  ...
}
(beforeAll as any).description = "One-line description for the Netra UI.";

// beforeEach: receives sharedContext → returns dict merged into setupContext (every item)
async function beforeEach(
  sharedContext: Record<string, any> | null,
): Promise<Record<string, any> | null | void> {
  ...
}
(beforeEach as any).description = "One-line description for the Netra UI.";

// before: Record keyed by datasetItemId; each receives merged context (beforeAll + beforeEach)
async function setupScenarioA(
  sharedContext: Record<string, any> | null,
): Promise<Record<string, any> | null | void> {
  ...
}
(setupScenarioA as any).description = "One-line description for the Netra UI.";

// after: Record keyed by datasetItemId; each receives result + setupContext
async function teardownScenarioA(
  result: Record<string, any>,
  setupContext: Record<string, any> | null,
): Promise<void> {
  ...
}
(teardownScenarioA as any).description = "One-line description for the Netra UI.";

// afterEach: receives result + setupContext (every item, after item-specific after)
async function afterEach(
  result: Record<string, any>,
  setupContext: Record<string, any> | null,
): Promise<void> {
  ...
}
(afterEach as any).description = "One-line description for the Netra UI.";

// afterAll: receives aggregated results and sharedContext (beforeAll only)
async function afterAll(
  results: Record<string, any>,
  sharedContext: Record<string, any> | null,
): Promise<void> {
  ...
}
(afterAll as any).description = "One-line description for the Netra UI.";

**必需：**上述每个 TypeScript 钩子都必须设置 .description。省略此项会导致 lifecycleHooks 中的 description: null。 传递上下文的模式：

Python：

before_all()                                      → shared_context (dict | None)
before_each(shared_context)                       → merged into setup_context
hooks.before[dataset_item_id](merged_context)     → merged into setup_context
BaseTask.run(..., setup_context=...)              ← receives merged context
hooks.after[dataset_item_id](result, setup_context)  ← same merged context for cleanup
after_each(result, setup_context)                 ← same merged context
after_all(results, shared_context)                ← run-level shared_context only

TypeScript：

beforeAll()                                       → sharedContext (Record | null)
beforeEach(sharedContext)                         → merged into setupContext
hooks.before[datasetItemId](mergedContext)        → merged into setupContext
BaseTask.run(..., setupContext)                   ← receives merged context
hooks.after[datasetItemId](result, setupContext)  ← same merged context for cleanup
afterEach(result, setupContext)                   ← same merged context
afterAll(results, sharedContext)                  ← run-level sharedContext only

重要：before 和 after 钩子是以数据集条目 ID 为键的字典，而不是单个函数。每个需要特定设置/清理的场景都应有自己的函数，并使用稳定的数据集条目 ID 作为键，将这些函数注册到 SimulationHooks 中。如果所有场景都适用相同的设置/清理操作，请优先使用 before_each / after_each。

设置上下文是以下内容的合并结果：before_all / beforeAll、before_each / beforeEach，以及条目 before 钩子返回的任何对象。它会传递给 BaseTask.run()、条目 after 钩子和 after_each / afterEach。如果某个 before 钩子执行到一半时失败，清理操作仍会收到已成功构建的最完整设置上下文。after_all / afterAll 仍只会收到运行级共享上下文。

完整的 Python 框架示例（员工 + 按场景设置身份验证的模式）：

假设数据集有两个条目：

• dataset_item_id = "item-refund-request" → 需要共享身份验证 + 退款专用账户
• dataset_item_id = "item-balance-inquiry" → 仅需要共享身份验证（由 before_each / after_each 覆盖）

from netra.simulation import BaseTask, SimulationHooks, TaskResult

# ---- Hooks ----

def before_all():
    # TODO: replace with your actual setup code
    employee = your_api.create_employee(name="Test User", role="admin")
    return {"employee_id": employee.id}
before_all.description = (
    "Create the test employee and assign the admin role before any scenario runs."
)


def before_each(shared_context: dict | None):
    employee_id = (shared_context or {}).get("employee_id")
    # TODO: replace with your actual login logic
    token = your_api.login(employee_id=employee_id)
    return {"auth_token": token}
before_each.description = "Obtain a fresh auth token before every scenario."


def setup_refund_scenario(shared_context: dict | None):
    # shared_context already includes employee_id + auth_token from before_all + before_each
    refund_account = your_api.create_refund_account()
    return {"refund_account_id": refund_account.id}
setup_refund_scenario.description = (
    "Create a refund account for the refund scenario only."
)


def teardown_refund_scenario(result: dict, setup_context: dict | None):
    refund_account_id = (setup_context or {}).get("refund_account_id")
    try:
        your_api.delete_refund_account(refund_account_id)
    except Exception:
        pass  # catch locally so teardown errors do not mark the item postscript_failed
teardown_refund_scenario.description = (
    "Delete the refund account after the refund scenario."
)


def after_each(result: dict, setup_context: dict | None):
    auth_token = (setup_context or {}).get("auth_token")  # from furthest built setup_context
    try:
        your_api.logout(token=auth_token)
    except Exception:
        pass
after_each.description = "Log out after every scenario regardless of outcome."


def after_all(results: dict, shared_context: dict | None):
    employee_id = (shared_context or {}).get("employee_id")
    # TODO: replace with your actual teardown code
    your_api.delete_employee(employee_id=employee_id)
after_all.description = "Delete the test employee once all scenarios have finished."


# Map hooks to specific dataset item IDs
hooks = SimulationHooks(
    before_all=before_all,
    before_each=before_each,
    before={
        "item-refund-request": setup_refund_scenario,
    },
    after={
        "item-refund-request": teardown_refund_scenario,
    },
    after_each=after_each,
    after_all=after_all,
)


# ---- Task ----

class MyAgentTask(BaseTask):
    def run(
        self,
        message: str,
        session_id: str | None = None,
        files: list | None = None,
        setup_context: dict | None = None,
    ) -> TaskResult:
        ctx = setup_context or {}
        response = my_agent.chat(
            message,
            session_id=session_id,
            auth_token=ctx.get("auth_token"),
        )
        return TaskResult(
            message=response.text,
            session_id=session_id or response.session_id or "default",
        )


# ---- Run ----

result = Netra.simulation.run_simulation(
    name="My Simulation",
    dataset_id="dataset-123",
    task=MyAgentTask(),
    hooks=hooks,
    max_concurrency=3,
)

完整的 TypeScript 框架示例（相同模式）：

import { BaseTask, Netra } from "netra-sdk";
import type { ProcessedFile, SimulationHooks, TaskResult } from "netra-sdk";

// ---- Hooks ----

async function beforeAll(): Promise<Record<string, any> | null> {
  // TODO: replace with your actual setup code
  const employee = await yourApi.createEmployee({ name: "Test User", role: "admin" });
  return { employeeId: employee.id };
}
(beforeAll as any).description =
  "Create the test employee and assign the admin role before any scenario runs.";

async function beforeEach(
  sharedContext: Record<string, any> | null,
): Promise<Record<string, any> | null> {
  const employeeId = sharedContext?.employeeId;
  // TODO: replace with your actual login logic
  const token = await yourApi.login({ employeeId });
  return { authToken: token };
}
(beforeEach as any).description =
  "Obtain a fresh auth token before every scenario.";

async function setupRefundScenario(
  sharedContext: Record<string, any> | null,
): Promise<Record<string, any> | null> {
  // sharedContext already includes employeeId + authToken from beforeAll + beforeEach
  const refundAccount = await yourApi.createRefundAccount();
  return { refundAccountId: refundAccount.id };
}
(setupRefundScenario as any).description =
  "Create a refund account for the refund scenario only.";

async function teardownRefundScenario(
  result: Record<string, any>,
  setupContext: Record<string, any> | null,
): Promise<void> {
  try {
    await yourApi.deleteRefundAccount(setupContext?.refundAccountId);
  } catch {
    // Catch locally so teardown errors do not mark the item postscript_failed
  }
}
(teardownRefundScenario as any).description =
  "Delete the refund account after the refund scenario.";

async function afterEach(
  result: Record<string, any>,
  setupContext: Record<string, any> | null,
): Promise<void> {
  try {
    await yourApi.logout({ token: setupContext?.authToken });
  } catch {
    // ignore
  }
}
(afterEach as any).description =
  "Log out after every scenario regardless of outcome.";

async function afterAll(
  results: Record<string, any>,
  sharedContext: Record<string, any> | null,
): Promise<void> {
  // TODO: replace with your actual teardown code
  await yourApi.deleteEmployee(sharedContext?.employeeId);
}
(afterAll as any).description =
  "Delete the test employee once all scenarios have finished.";

const hooks: SimulationHooks = {
  beforeAll,
  beforeEach,
  before: {
    "item-refund-request": setupRefundScenario,
  },
  after: {
    "item-refund-request": teardownRefundScenario,
  },
  afterEach,
  afterAll,
};

// ---- Task ----

class MyAgentTask extends BaseTask {
  async run(
    message: string,
    sessionId?: string | null,
    files?: ProcessedFile[] | null,
    setupContext?: Record<string, any> | null,
  ): Promise<TaskResult> {
    const ctx = setupContext || {};
    const response = await myAgent.chat(message, {
      sessionId,
      authToken: ctx.authToken,
    });
    return {
      message: response.text,
      sessionId: sessionId || response.sessionId || "default",
    };
  }
}

// ---- Run ----

const result = await Netra.simulation.runSimulation({
  name: "My Simulation",
  datasetId: "dataset-123",
  task: new MyAgentTask(),
  hooks,
  maxConcurrency: 3,
});

步骤 4：在计划中标明钩子要求

如果需要钩子，请在计划的 Dataset Configuration 表格下方添加一个“钩子”部分。如果不需要钩子，请明确说明：

**Hooks:** Not required — scenarios are independent.

如果需要钩子，请逐一列出每种钩子类型，并用一行描述说明，同时注明哪些场景需要条目专用钩子。在计划中使用与语言相对应的名称（Python 使用 before_all，TypeScript 使用 beforeAll）：

**Hooks:**
- `before_all` / `beforeAll`: Create shared test employee and assign admin role
- `before_each` / `beforeEach`: Obtain a fresh auth token before every scenario
- `before`: Per-scenario hooks keyed by dataset item ID:
  - Refund Request scenario: Create refund account
- `after`: Per-scenario hooks keyed by dataset item ID:
  - Refund Request scenario: Delete refund account
- `after_each` / `afterEach`: Log out after every scenario
- `after_all` / `afterAll`: Delete the test employee

在场景的 Netra UI 对话选项卡中：

• 本次运行中的每个场景都会显示 before_all / beforeAll、before_each / beforeEach、after_each / afterEach 和 after_all / afterAll（运行级元数据）
• 只有当该场景的数据集条目 ID 已注册在钩子字典中时，才会显示 before / after（条目级元数据）

**重要：**只有通过 MCP 创建数据集条目后，才能获取实际的数据集条目 ID。在生成的框架代码中，请使用占位 ID（例如 "item-refund-request"、"item-balance-inquiry"），并告知用户将这些 ID 替换为 netra_create_dataset_item 返回的实际 ID。

然后，在计划摘要下方的可折叠代码块中加入生成的框架代码。

阶段 4：生成计划

按照以下确切格式生成计划。这是用户将审核的主要输出。

---

计划格式 — 按原样输出，并填入生成的值：

# NDD Evaluation Plan

## Dataset Configuration

| Field | Value |
|---|---|
| **Name** | {generated-dataset-name} |
| **Description** | {generated-description} |
| **Turn Type** | {single | multi} |
| **Execution Method** | {Netra.evaluation.run_test_suite / Netra.evaluation.runTestSuite | Netra.simulation.run_simulation / Netra.simulation.runSimulation} |
| **Total Items** | {count} |

### Execution Strategy

Single-turn datasets:
- Python: `Netra.evaluation.run_test_suite()`
- TypeScript: `await Netra.evaluation.runTestSuite({...})`

Multi-turn datasets:
- Python: `Netra.simulation.run_simulation()`
- TypeScript: `await Netra.simulation.runSimulation({...})`

If Turn Type = `multi`, include:

```text
Required Task Implementation:
⚠️ User must provide a BaseTask implementation compatible with Netra Simulation.

如果 Turn Type = multi，且在阶段 3g 中识别出了钩子，则在 Execution Strategy 后立即添加 钩子 部分：

### Hooks (Pre/Post Scripts)

{Either "Not required — scenarios are independent." OR a list like:}

- `before_all` / `beforeAll`: {one-line description}
- `before_each` / `beforeEach`: {one-line description, if used}
- `before`: {one-line description, if used}
- `after`: {one-line description, if used}
- `after_each` / `afterEach`: {one-line description, if used}
- `after_all` / `afterAll`: {one-line description}

**Generated scaffold code** (requires user to fill in TODO sections; language = Phase 0.5 detection):

{paste the full scaffold code block from Phase 3g here — Python OR TypeScript, not both}

---

项目

项目 1：{scenario-name}

类别： {happy-path | negative | edge-case | safety | performance}

单轮项目：

字段	值
输入	{测试输入 / 用户消息}
预期输出	{预期的正确响应或行为}
提供方配置	{none — use evaluator default | provider_id: {id}, model: {model}}

多轮项目：

字段	值
输入	{}（空对象 — 仿真引擎会生成消息）
预期输出	null
ScenarioName	{场景的简洁显示名称}
Scenario	{评估器用于判断是否成功的实际目标 / 场景描述}
Persona	{模拟用户的身份}
Max Turns	{数字}
Behaviour Instructions	{模拟用户的分步脚本}
提供方配置	provider_id: {id}, model: {model}

评估器：

评估器：{evaluator-name}

配置	值
类型	{llm-as-judge / code / json / ...}
模型	{org-default / user-specified}
通过标准	{operator} {threshold}

变量	来源	表达式
{var-name}	{Agent Response / Span: {span-name} / Dataset: expectedOutput / Trace: {metric} / Value}	{jmespath-expression}
{var-name}	...	...

如果评估器具有可配置变量，请包含：

可配置变量（用户提供）：

变量	值
{var-name}	{用户提供的值或 ⚠️ 需要配置}

准则遵循示例：

可配置变量（用户提供）：

变量	值
assistant_instructions	你是一名乐于助人的客户支持 Agent。始终以专业的方式问候用户……
assistant_constraints	不得询问信用卡详情。必须将账单问题升级给人工 Agent……

事实准确性示例（多轮）：

可配置变量（用户提供）：

变量	值
reference_facts	{"return_policy": "14 days", "shipping_time": "3-5 business days", ...}

评估器：{evaluator-name-2}

……

---

项目 2：{scenario-name}

……

---

摘要

类别	数量
成功路径	{n}
负面	{n}
边缘情况	{n}
安全	{n}
性能	{n}
总计	{n}

使用的评估器	数量
{evaluator-name}	{n items}
...	...

---

After outputting the plan, ask the user:

> Review the plan above. You can ask me to:
> - Add, remove, or modify any item
> - Change evaluators for a specific item
> - Adjust variable mappings (where input/output/reference comes from)
> - Change pass criteria or model
> - Edit the dataset name or description
> - Add, remove, or change hooks (before_all, before_each, before, after, after_each, after_all)
>
> When you're satisfied, say **"proceed"** and I'll create everything in Netra.

### Phase 5: Handle User Edits

When the user requests changes:
- Apply the edit to the plan.
- Re-output only the changed section (not the full plan) so the user can verify.
- Ask if there are more changes or if they want to proceed.

### Phase 6: Create in Netra

When the user says "proceed", "create", "go ahead", or similar:

Execute these MCP calls in order:

1. netra_create_dataset → name → description → turnType: {single | multi} → 从响应中捕获 datasetId

2. 如果要附加的库评估器尚不存在于 My Evaluators 中：

   a. 解析组织的默认 LLM 配置：

   netra_get_default_llm_configuration

   预期响应： { "provider": "openai", "model": "gpt-5", "providerConfigurationId": "provider-config-id" }

   如果返回 null，则阻止创建： ⚠️ 评估器创建受阻。未配置默认提供方。 不要猜测提供方或模型值。 b. netra_create_evaluator → name → libraryEvaluatorId → type：必须与库评估器的 type 完全一致。 检查库评估器的 type 字段（例如 "llm-as-judge"、"tool_accuracy"、 "regex"、"code"），并在创建项目评估器时传入相同的 type。 如果库评估器是 tool_accuracy（基于规则），则将其创建为 tool_accuracy。 如果库评估器是 llm-as-judge，则将其创建为 llm-as-judge。 不要依赖 API 从 libraryEvaluatorId 继承 type——它可能无论如何都会默认 为 llm-as-judge，导致基于规则的评估器报错 "Missing required config: model, provider_id, or prompt"， 因为引擎会错误地将其视为 LLM 评估器。 如果怀疑 type 不匹配，请将评估器创建为自定义评估器，明确指定 type，并省略 libraryEvaluatorId。 → turnType："single" | "multi" → evalType：必须明确设置 - 对于单轮评估器：传入 evalType: "turn" - 对于多轮评估器：传入 evalType: "session" 不要省略 evalType——API 不会根据 turnType 自动推断。 对于多轮评估器，省略 evalType 会导致其默认值为 "turn"， 这意味着评估器会逐条消息运行，而不是按会话运行，导致模拟运行的 评估分数显示为 "not available"。 → config：必须与评估器类型相匹配。 - 对于基于 LLM 的评估器（llm-as-judge）：始终使用步骤 2a 中解析出的默认值 明确传入： { "model": "{model}", "provider_id": "{providerConfigurationId}" } 不要省略 config 或依赖自动解析——评估引擎 要求 config 中包含 provider_id（snake_case），才能执行 LLM 评估器。 省略 config 或使用错误的字段名会导致评估器在运行时 报错 "Missing required config: model, provider_id, or prompt"。 - 对于基于规则的评估器（tool_accuracy）：传入评估器专用的 config（例如 {"matchType": "partial"}）。不要包含 model 或 provider_id。 - 对于其他代码/正则表达式评估器：传入库评估器中定义的 专用 config。 → 从响应中获取 evaluatorId

   c. 验证创建的评估器 config：

   创建后，检查响应中的 config 对象。确认其中包含 provider_id（snake_case）。如果响应中只显示 providerId（camelCase）， 而没有 provider_id，则评估器会在执行评估时失败。

   有效的响应 config： { "model": "gpt-4o", "provider_id": "dcffa74c-..." }

   如果验证失败，请使用明确指定的 config 重新创建评估器。

3. 对于每个条目： netra_create_dataset_item → datasetId、input、expectedOutput → metadata（多轮场景：scenarioName、persona、max_turns、behaviour_instructions） → providerConfig 对于多轮数据集：

   • input：必须为空对象 {} 模拟引擎会根据 metadata.behaviour_instructions 生成第一条用户消息， 而不是从 input 字段生成。 如果将 input 设为字符串，Netra 仪表板中的场景名称会显示为空， 因为 UI 会从 metadata.scenarioName 读取场景标识符，而字符串形式的 input 会造成干扰。
   • expectedOutput：设为 null 或省略 模拟会通过会话级评估器对整个对话进行评估， 而不是针对单个预期输出进行评估。
   • metadata：必填 — 这是模拟项目的主要配置 { "scenarioName": "{scenarioName}", "scenario": "{scenario}", "persona": "...", "max_turns": ..., "behaviour_instructions": "..." }
     • scenarioName：Netra 仪表板中显示的场景名称。 简洁、易读的测试用例标签（例如“迭代完善项目”）。
     • scenario：模拟必须实现的实际场景描述／目标。 这是会话级评估器（目标达成度、 对话完整度）用于判断 Agent 是否成功的详细目标文本。它描述 对话应该完成什么，而不只是一个标签。 示例：“通过对话逐步完善需求，为智能植物浇水系统 制定完整的工程规格。”
     • persona：模拟用户假扮的身份。
     • max_turns：模拟停止前的最大对话轮数。
     • behaviour_instructions：模拟用户的分步脚本。 模拟引擎会用它生成所有用户消息，包括第一条消息。 请具体说明模拟用户在每一轮应该说什么或 询问什么。
   • 始终包含 providerConfig。
   • 如果计划中未指定模型，请使用组织在 netra_get_default_llm_configuration 中配置的默认提供商配置。
   • 示例：

   {
     "provider_id": "{providerConfigurationId}",
     "model": "{model}"
   }
   

对于单轮数据集： - input：包含测试消息的字符串 - expectedOutput：包含预期 Agent 响应的字符串 - providerConfig 可选。 - 仅当项目需要使用不同于评估器默认设置的模型时才包含。 → evaluators（可选）：带有 variableMapping 的项目级评估器覆盖配置

4. 将评估器映射到数据集： netra_map_evaluator_to_dataset → datasetId、evaluatorId、variableMapping

   对于单轮评估器：传入完整的 variableMapping（user_query、response 等） 对于多轮（会话）评估器：传入空对象 variableMapping: {} 会话评估器会从对话上下文中自动解析变量。 但是，仍必须提供 variableMapping 参数，即 {} — 完全省略该参数会导致“Cannot convert undefined or null to object”错误。

5. 为需要配置变量的评估器设置变量：

   如果任何已映射的评估器具有可配置变量（准则遵循度、多轮事实准确性）， 请在数据集项目的 metadata 中包含 evaluatorConfigs。

   evaluatorConfigs 是一个对象数组，每个对象包含：

   • id：评估器 ID（在第 2 步创建的项目评估器 ID）
   • configuredValues：将变量名映射到用户提供值的对象

   调用 netra_create_dataset_item 或 netra_update_dataset_item 时，请在 metadata 字段中包含 evaluatorConfigs。

   准则遵循度示例：

   {
     "metadata": {
       "scenarioName": "Customer Support",
       "scenario": "...",
       "persona": "...",
       "max_turns": 5,
       "behaviour_instructions": "...",
       "evaluatorConfigs": [
         {
           "id": "{guideline-adherence-evaluator-id}",
           "configuredValues": {
             "assistant_instructions": "You are a helpful customer support agent. Always greet the user professionally, stay on topic, follow security rules...",
             "assistant_constraints": "Must NOT ask for credit card details. Must escalate billing issues to human agents..."
           }
         }
       ]
     }
   }
   

   多轮事实准确性示例：

   {
     "metadata": {
       "evaluatorConfigs": [
         {
           "id": "{factual-accuracy-evaluator-id}",
           "configuredValues": {
             "reference_facts": "{\"return_policy\": \"14 days\", \"shipping_time\": \"3-5 business days\"}"
           }
         }
       ]
     }
   }
   

   同一个 evaluatorConfigs 数组中可以配置多个评估器：

   {
     "metadata": {
       "evaluatorConfigs": [
         {
           "id": "{guideline-adherence-evaluator-id}",
           "configuredValues": {
             "assistant_instructions": "...",
             "assistant_constraints": "..."
           }
         },
         {
           "id": "{factual-accuracy-evaluator-id}",
           "configuredValues": {
             "reference_facts": "..."
           }
         }
       ]
     }
   }
   

   重要：如果未为需要可配置变量的评估器设置 evaluatorConfigs， 这些评估器将收到空值，导致评分不可靠。准则遵循评估器会对 AWARENESS 和 CONSISTENCY 标准报告“无法评估”，而事实准确性评估器会默认判定为通过，因为它找不到可供核对的参考事实。

If MCP tools for creation are not yet available, output a message:

> The dataset creation MCP tools are not registered yet. Here's what would be created:
> - Dataset: {name} with {n} items
> - {n} evaluator mappings
>
> You can create this manually in the Netra dashboard, or wait for the MCP tools to be available.

After successful creation, output:

✅ 已创建数据集：{name}（{datasetId}） 项目：已创建 {count} 个 评估器映射：已配置 {count} 个

在 Netra 中查看：https://app.getnetra.ai/datasets/{datasetId}

下一步：运行 netra run {dataset-name} 执行测试套件。

## Important Rules

1. **Never skip the preview.** Always show the full plan before creating anything.
2. **Never guess span names.** If code analysis doesn't find clear span definitions, use the fallback method or ask the user.
3. **Mark uncertainty explicitly.** Use `⚠️ NEEDS CONFIGURATION` for any mapping you're not confident about.
4. **Respect the spec.** If the QA spec defines specific pass thresholds, use those instead of defaults.
5. **Include the reasoning.** When recommending an evaluator, briefly explain why it fits the test case.
6. **Don't over-evaluate.** Not every item needs 5 evaluators. Match evaluator count to scenario complexity — simple happy paths may need 1-2, complex edge cases may need 3-4.
7. **Prefer span output over taskOutput** when the evaluator checks an intermediate step. Use `taskOutput` only when evaluating the final user-facing response.
8. Never execute the agent directly for evaluation runs.
   - Single-turn evaluations must use `Netra.evaluation.run_test_suite()` (Python) or `await Netra.evaluation.runTestSuite({...})` (TypeScript).
   - Multi-turn simulations must use `Netra.simulation.run_simulation()` (Python) or `await Netra.simulation.runSimulation({...})` (TypeScript).

9. Trace IDs are required for evaluator execution.
   Any execution strategy that bypasses Netra Evaluation or Simulation APIs is invalid.

10. Multi-turn simulations require a task implementation that extends `BaseTask`.
    The generated plan must explicitly mention this requirement.

11. When generating execution examples, use the official Netra SDK execution APIs for the detected language (Phase 0.5).
    Do not generate examples that directly invoke the agent for evaluation execution.
    Do not mix Python and TypeScript conventions in the same scaffold.

12. Variable mappings that reference spans are only valid when execution occurs through Netra Evaluation or Simulation APIs that generate trace data.

13. For multi-turn simulations, include a task implementation example using:
    - `BaseTask`
    - `TaskResult`
    - `session_id` (Python) / `sessionId` (TypeScript)
    - `setup_context` (Python) / `setupContext` (TypeScript) when hooks are used
    - `Netra.simulation.run_simulation()` (Python) / `await Netra.simulation.runSimulation({...})` (TypeScript)

14. Resolve the organization's default provider/model only when creating a project evaluator from a library evaluator.
    Once resolved, pass the config explicitly to `netra_create_evaluator` using `provider_id` (snake_case).

15. Do not fetch provider/model information during evaluator recommendation unless evaluator creation is required.

16. Library evaluators attached to a dataset must first be materialized into My Evaluators before mappings are created.

17. Every generated scenario must have a meaningful, human-readable name.

18. Scenario names should describe the behavior being validated rather than the test category.

Good:
- Retrieve Customer Order Status
- Reject Unauthorized Data Access
- Handle Empty Input

Bad:
- Scenario 1
- Test Case 3
- Happy Path
- Negative Test

19. Scenario names should be unique within the dataset whenever possible.

20. When creating evaluators for multi-turn datasets, ALWAYS explicitly pass `evalType: "session"`.

    The `netra_create_evaluator` API does NOT auto-infer `evalType` from `turnType`.
    If `evalType` is omitted, it defaults to `"turn"` regardless of the `turnType` value.
    This causes multi-turn simulation evaluators to fail silently — conversations are
    recorded correctly but evaluation scores show as "not available" because turn-level
    evaluators cannot score session-level simulation data.

    Correct (multi-turn):
    ```json
    {
      "turnType": "multi",
      "evalType": "session"
    }
    ```

    Wrong (will produce "not available" scores):
    ```json
    {
      "turnType": "multi"
    }
    ```

    Only use `evalType: "turn"` within a multi-turn dataset when you explicitly need
    per-message evaluation rather than whole-conversation evaluation.

21. Item-level `providerConfig` (`{ provider_id, model }`) overrides the evaluator's default model for that specific item only. Use it only when a specific scenario requires a different model. For uniform model usage across all items, configure the model at the evaluator level — do not set `providerConfig` on every item.

22. Multi-turn dataset items must always be created with:
    - `input`: an empty object `{}` (NOT a string — the simulation engine generates user messages from metadata)
    - `expectedOutput`: `null` or omitted (session-level evaluators score the full conversation)
    - `metadata`: containing `scenarioName`, `scenario`, `persona`, `max_turns`, and `behaviour_instructions`
      - `scenarioName`: concise display label (e.g. "Iterative Project Refinement")
      - `scenario`: the actual goal/scenario description that evaluators use to judge success
        (e.g. "Get a complete engineering spec for a smart plant watering system, refining requirements through conversation")
    - `providerConfig`: always required

    If the specification does not provide a providerConfig, resolve the organization's default provider configuration using `netra_get_default_llm_configuration` and attach it to every multi-turn dataset item during creation.

    Setting `input` to a string causes the scenario name to appear empty in the Netra
    dashboard because the UI reads the scenario display name from `metadata.scenarioName`,
    and a string input overrides the expected structure.

23. Before creating a multi-turn dataset item, the system must ensure a valid provider configuration exists.
    
    If `netra_get_default_llm_configuration` returns null and no item-level provider configuration is specified:
    
    ```
    ⚠️ Multi-turn dataset creation blocked.
    No default provider configured and no item-level providerConfig supplied.
    ```
    
    Do not create the dataset items until a valid provider configuration is available.

24. When creating LLM-based evaluators, ALWAYS pass explicit `config` with `model` and `provider_id` (snake_case).

    Do NOT rely on auto-resolution of provider config — the MCP tool stores the field as `providerId` (camelCase),
    but the evaluation engine requires `provider_id` (snake_case). This mismatch causes evaluators to silently fail
    at runtime with "Missing required config: model, provider_id, or prompt", resulting in null scores on all items.

    Correct:
    ```json
    {
      "model": "gpt-4o",
      "provider_id": "dcffa74c-..."
    }
    ```

    Wrong (will fail at eval time):
    ```json
    {
      "model": "gpt-4o",
      "providerId": "dcffa74c-..."
    }
    ```

25. MCP tools (`netra_create_test_run`, `netra_submit_test_run_item`) are for manual/programmatic
    result submission only. They require a `trace_id` from an existing trace. Do NOT use them as a
    substitute for the SDK evaluation/simulation run APIs (`run_test_suite` / `runTestSuite`,
    `run_simulation` / `runSimulation`) — those SDK methods handle agent execution, trace capture,
    and result submission automatically.

26. When using `Netra.evaluation.run_test_suite()` with a remote Netra dataset:

    ```python
    response = Netra.evaluation.get_dataset(dataset_id)
    dataset = Dataset(items=response.items)
    result = Netra.evaluation.run_test_suite(
        name="...",
        data=dataset,
        task=agent_task_function,
        max_concurrency=3,
    )
    ```

    The `data` parameter requires a `Dataset` object, not a dataset ID string.
    Use `Netra.evaluation.get_dataset()` to fetch items, then wrap in `Dataset(items=...)`.

27. When mapping multi-turn (session) evaluators to a dataset using `netra_map_evaluator_to_dataset`,
    ALWAYS pass `variableMapping: {}` (empty object). Session-level evaluators auto-resolve their
    variables from the conversation context and do not require explicit variable mappings.
    However, the `variableMapping` parameter itself MUST be provided — omitting it entirely
    causes the API to throw "Cannot convert undefined or null to object".

28. Multi-turn dataset items MUST use `input: {}` (empty object).
    The simulation engine generates all user messages — including the first one — from
    `metadata.behaviour_instructions`. The `input` string is NOT sent as the first message.
    Using a string for `input` causes the Netra dashboard to show an empty scenario name
    because the UI expects `metadata.scenarioName` to be the primary identifier when `input`
    is an empty object.

    Both `scenarioName` and `scenario` MUST be present in metadata:
    - `scenarioName` = the short display label shown in the dashboard
    - `scenario` = the actual goal/scenario description that session evaluators
      (Goal Fulfillment, Conversation Completeness) read to judge success

    Omitting `scenario` causes evaluators to receive an empty goal, resulting in
    inaccurate or failing evaluation scores.

29. A default provider configuration MUST exist before any MCP-based creation (datasets, evaluators, dataset items, evaluator mappings).
    Phase 0 verifies this upfront by calling `netra_get_default_llm_configuration`. If no default provider
    is configured, the entire workflow is blocked — do not proceed to planning phases or creation.
    This check prevents wasted effort generating plans that cannot be materialized.

30. When adding a library evaluator to My Evaluators, the project evaluator's `type` MUST match the
    library evaluator's original type exactly.

    - If the library evaluator is `tool_accuracy` (rule-based), create it with `type: "tool_accuracy"`
      and pass its specific config (e.g. `{"matchType": "partial"}`). Do NOT include model/provider_id.
    - If the library evaluator is `llm-as-judge`, create it with `type: "llm-as-judge"` and pass
      model/provider_id config.
    - If the library evaluator is `regex` or `code`, create it with the matching type.

    The `netra_create_evaluator` API may silently default to `llm-as-judge` when using
    `libraryEvaluatorId`, regardless of the library evaluator's actual type. This causes rule-based
    evaluators (like `tool_accuracy`) to fail at runtime because the engine expects LLM config
    (`model`, `provider_id`, `prompt`) that rule-based evaluators don't use.

    If you encounter this type mismatch, create the evaluator as a **custom evaluator** with explicit
    `type` set to the correct value and omit `libraryEvaluatorId`. This bypasses the auto-defaulting
    behavior and ensures the evaluator runs with the correct execution logic.

31. When using evaluators with configurable variables, ALWAYS include `evaluatorConfigs` in dataset
    item metadata (or dataset metadata) with the correct evaluator ID and variable values.

    Evaluators with configurable variables:

    | Evaluator | Configurable Variables |
    |---|---|
    | Guideline Adherence | `assistant_instructions` (REQUIRED — no auto-resolution), `assistant_constraints` (optional — auto-resolved from `metadata.assistant_constraints` but configurable value takes precedence) |
    | Factual Accuracy (multi-turn) | `reference_facts` (optional — auto-resolved from `metadata.reference_facts` but configurable value takes precedence) |

    `evaluatorConfigs` format in metadata:

    ```json
    {
      "evaluatorConfigs": [
        {
          "id": "{evaluator-id}",
          "configuredValues": {
            "variable_name": "value"
          }
        }
      ]
    }
    ```

    Without `evaluatorConfigs`:
    - **Guideline Adherence** will receive empty `assistant_instructions`, causing the evaluator
      to report "The evaluation is severely limited because the assistant's instructions were not
      provided" and produce a degraded score (typically 0.5) that fails the >= 0.6 pass criteria.
    - **Factual Accuracy (multi-turn)** will receive empty `reference_facts` if not set in
      `metadata.reference_facts` either, causing it to default-pass with no meaningful evaluation.

    The `assistant_instructions` variable is the most critical — it has NO fallback auto-resolution
    from any metadata field. It MUST be explicitly provided via `evaluatorConfigs[].configuredValues`.

    When the plan includes Guideline Adherence or Factual Accuracy (multi-turn), ALWAYS prompt the
    user to provide these values before proceeding to creation. If the user cannot provide them,
    warn that the evaluator will produce unreliable scores.

32. **Always run Phase 3g for multi-turn datasets.** Skipping the correlation analysis risks generating a plan that will produce incorrect results because parallel scenario execution collides on shared state (e.g., two scenarios trying to use the same record simultaneously).

33. **Hooks live entirely on the user's side.** Scripts are never uploaded to or stored by Netra. At run creation the SDK sends lightweight descriptors (`lifecycleHooks`) only:
    - `beforeAll` / `beforeEach` / `afterEach` / `afterAll` → stored on the test run (shown on every scenario in that run)
    - `before` / `after` per `datasetItemId` → stored on each matching test run item (shown only on that scenario's Conversation tab)
    The Netra UI renders these in collapsible **Pre-script** / **Post-script** panels on the multi-turn Scenario Conversation tab (snake_case labels, with `name` + `description`). There is no run-level "Has pre-script" badge.
    The generated scaffold code belongs in the user's codebase, not in the Netra dataset configuration. Always key `SimulationHooks.before` / `after` by real dataset item ID values so the correct scenario UI shows the correct pre/post script.

34. **Never collapse correlated scenarios into a single large item to avoid hooks.** Merging related scenarios degrades evaluator accuracy (session evaluators score the entire conversation — a merged scenario is harder to judge) and makes the dataset harder to maintain. Recommend hooks instead.

35. **Never suggest sequential execution as a workaround for correlation.** Sequential execution (`max_concurrency=1` / `maxConcurrency: 1`) prevents collisions but means a single scenario failure blocks all subsequent ones. Hooks are the correct solution.

36. **`before_all` / `beforeAll` failure is fatal for the entire run.** If the setup code raises, no scenarios run and the run is marked failed. Only put truly required global setup in `beforeAll`. Common per-item setup belongs in `before_each` / `beforeEach`. Optional or scenario-specific setup belongs in `before`.

37. **`after`, `after_each` / `afterEach`, and `after_all` / `afterAll` must be robust.** These hooks run even when scenarios fail (including when `before` / `before_each` fails). Wrap risky cleanup logic in try/except (Python) or try/catch (TypeScript). A teardown error on an otherwise-successful item marks it `postscript_failed` (eval results are kept). Prefer resilient teardown so cleanup failures do not obscure a successful conversation.

38. **Setup context is how hooks share data with `BaseTask.run()` and teardown hooks.** The objects returned by `before_all` / `beforeAll`, `before_each` / `beforeEach`, and item `before` are merged into `setup_context` / `setupContext` and passed to `BaseTask.run()`, item `after`, and `after_each` / `afterEach`. If a before hook fails mid-way, teardown still receives the furthest successfully built setup context (so `after_each` can clean up what `before_each` created even if item `before` failed). Generated scaffolding must use `after(result, setup_context)` / `after(result, setupContext)` — not shared context alone. The task must declare the setup-context parameter in its `run()` signature to receive it. Existing tasks that do not declare it continue to work (backwards compatible). `after_all` / `afterAll` still receives only the run-level shared context from `beforeAll`, but its `results` include setup/first-turn failures as well as conversation failures.

39. **Always include a `hooks` usage example in the generated scaffold code** when hooks are recommended. The example must match the detected language and show:
    - The `SimulationHooks` instantiation with all recommended hooks wired (`before_each` / `after_each` when common per-item setup/teardown applies)
    - A `BaseTask.run()` that accepts and uses `setup_context` / `setupContext`
    - The simulation run call with `hooks` (`run_simulation(..., hooks=hooks)` or `runSimulation({ ..., hooks })`)
    - Hook descriptions on **every** function via `.description` (required; ≤ 200 chars):
      - Python: `fn.description = "..."`
      - TypeScript: `(fn as any).description = "..."`
      Never ship hooks without descriptions — the SDK otherwise sends `description: null` in `lifecycleHooks`.

40. **Clearly communicate the `prescript_failed` status to the user.** When a `before_each` or `before` hook fails for a specific scenario, the test run item's status is set to `prescript_failed` rather than `failed`. Key properties of this status:
    - Visible in the Netra dashboard as **Prescript Failed** — distinguishes setup failures from actual agent failures
    - **Terminal**: the SDK polling loop treats `prescript_failed` as done; the run will not hang waiting for the item
    - **Eval suppressed**: `evalStatus` is set to `NOT_AVAILABLE` immediately — no evaluators run for that item
    - **Stable**: a `prescript_failed` item's status cannot be overwritten by bulk-failure sweeps (e.g. on run timeout)
    - **Evaluation roll-up**: if every item in the run ends as `failed` or `prescript_failed`, the run's evaluation status is `NOT_AVAILABLE`

41. **Clearly communicate the `postscript_failed` status to the user.** When an `after` / `after_each` / `after_all` hook fails on an otherwise-successful scenario, the item is marked `postscript_failed`. Key properties:
    - Visible in the Netra dashboard as **Postscript Failed**
    - **Terminal**: counted as done by the SDK polling loop
    - **Eval not suppressed**: conversation evaluations remain valid
    - **Stable**: cannot be overwritten by bulk-failure sweeps
    - **Evaluation roll-up**: excluded from the “all failed → NOT_AVAILABLE” check (unlike `prescript_failed`)
    - Already-failed / `prescript_failed` items are never overwritten with `postscript_failed`

42. **Also sync TypeScript simulation guidance** with `netra-best-practices/references/typescript/simulation.md` when updating hook behavior. Python guidance lives in `references/python/simulation.md`. Keep both languages at feature parity for hooks, including `before_each` / `beforeEach`, `after_each` / `afterEach`, `postscript_failed`, progressive setup context on failure, and `.description` requirements.

## Reference

For the full evaluator catalog with variables and thresholds, see [evaluator-library.md](references/evaluator-library.md).