tavus-cvi-interactions

Tavus CVI 交互协议

通过 WebRTC 数据通道以编程方式控制实时对话。

设置：Daily.js 客户端

<script src="https://unpkg.com/@daily-co/daily-js"></script>
<script>
const call = window.Daily.createFrame();

// Listen for events from CVI
call.on('app-message', (event) => {
  console.log('CVI event:', event.data);
});

// Join the conversation
call.join({ url: 'YOUR_CONVERSATION_URL' });

// Send interaction
function send(interaction) {
  call.sendAppMessage(interaction, '*');
}
</script>

可发送的交互

Echo：让 Replica 朗读文本

绕过大模型，让 Replica 原样朗读你提供的内容：

send({
  "message_type": "conversation",
  "event_type": "conversation.echo",
  "conversation_id": "YOUR_CONVERSATION_ID",
  "properties": {
    "modality": "text",
    "text": "Hello! Let me tell you about our product."
  }
});

对于流式音频（base64）：

send({
  "message_type": "conversation",
  "event_type": "conversation.echo",
  "conversation_id": "YOUR_CONVERSATION_ID",
  "properties": {
    "modality": "audio",
    "audio": "BASE64_ENCODED_AUDIO",
    "sample_rate": 24000,
    "inference_id": "unique-id",
    "done": "true"
  }
});

Respond：注入用户输入

将文本视为用户说出的内容（会经过大模型处理）：

send({
  "message_type": "conversation",
  "event_type": "conversation.respond",
  "conversation_id": "YOUR_CONVERSATION_ID",
  "properties": {
    "text": "What are your pricing plans?"
  }
});

Interrupt：停止 Replica 说话

send({
  "message_type": "conversation",
  "event_type": "conversation.interrupt",
  "conversation_id": "YOUR_CONVERSATION_ID"
});

覆盖上下文

替换整个对话上下文：

send({
  "message_type": "conversation",
  "event_type": "conversation.overwrite_context",
  "conversation_id": "YOUR_CONVERSATION_ID",
  "properties": {
    "context": "User is now asking about enterprise features."
  }
});

追加上下文

在现有上下文中添加内容，而不进行替换：

send({
  "message_type": "conversation",
  "event_type": "conversation.append_context",
  "conversation_id": "YOUR_CONVERSATION_ID",
  "properties": {
    "context": "User mentioned they have a team of 50 people."
  }
});

调整灵敏度

在对话过程中更改轮流发言的灵敏度：

send({
  "message_type": "conversation",
  "event_type": "conversation.sensitivity",
  "conversation_id": "YOUR_CONVERSATION_ID",
  "properties": {
    "participant_pause_sensitivity": "high",
    "participant_interrupt_sensitivity": "low"
  }
});

取值：low、medium、high

可接收的事件

话语（说了什么）

{
  "event_type": "conversation.utterance",
  "properties": {
    "role": "user",
    "content": "Tell me about your product"
  }
}

角色：user 或 replica

Replica 开始／停止说话

{
  "event_type": "conversation.replica.started_speaking",
  "properties": {
    "inference_id": "inf-123"
  }
}

{
  "event_type": "conversation.replica.stopped_speaking",
  "properties": {
    "inference_id": "inf-123",
    "duration": 4.5
  }
}

用户开始／停止说话

{
  "event_type": "conversation.user.started_speaking"
}

工具调用（函数调用）

当大模型调用工具时：

{
  "event_type": "conversation.tool_call",
  "properties": {
    "tool_name": "get_weather",
    "arguments": {
      "location": "San Francisco, CA"
    },
    "inference_id": "inf-123"
  }
}

处理后，通过 echo 或 respond 交互进行响应。

感知分析

当 Raven 分析用户时：

{
  "event_type": "conversation.perception_analysis",
  "properties": {
    "analysis": "User appears engaged, smiling, looking at camera"
  }
}

Replica 被打断

当 Replica 被打断时触发：

{
  "event_type": "conversation.replica.interrupted",
  "properties": {
    "inference_id": "inf-123"
  }
}

Python 客户端（Daily-Python）

from daily import Daily, CallClient

Daily.init()
client = CallClient()

def on_app_message(message, sender):
    print(f"Received: {message}")

client.set_user_name("bot")
client.join(meeting_url, completion=on_join)

# Send interaction
client.send_app_message({
    "message_type": "conversation",
    "event_type": "conversation.echo",
    "conversation_id": "xxx",
    "properties": {"text": "Hello!"}
})

常见模式

Echo 模式与手动控制

1. 使用 pipeline_mode: "echo" 创建 persona
2. 使用 Daily 客户端加入对话
3. 发送 conversation.echo 事件来控制语音
4. 发送 conversation.interrupt 以停止语音
5. 监听事件以跟踪状态

混合模式：大模型 + 手动注入

1. 使用 pipeline_mode: "full" 进行常规对话
2. 使用 conversation.append_context 注入上下文
3. 需要时使用 conversation.echo 覆盖内容
4. 使用 conversation.interrupt + conversation.echo 立即接管