azure-ai-voicelive-ts

@azure/ai-voicelive（JavaScript/TypeScript）

用于在 Node.js 和浏览器环境中，借助 Azure AI 构建双向语音助手的实时语音 AI SDK。

安装

npm install @azure/ai-voicelive @azure/identity
# TypeScript users
npm install @types/node

历史示例基线：1.0.0-beta.3。使用前请固定版本并验证预览版 API 的兼容性。

支持的环境：

• Node.js LTS 版本（20+）
• 现代浏览器（Chrome、Firefox、Safari、Edge）

环境变量

AZURE_VOICELIVE_ENDPOINT=https://<resource>.cognitiveservices.azure.com
# Optional: API key if not using Entra ID
AZURE_VOICELIVE_API_KEY=<your-api-key>
# Optional: Logging
AZURE_LOG_LEVEL=info

身份验证

Microsoft Entra ID（推荐）

import { DefaultAzureCredential } from "@azure/identity";
import { VoiceLiveClient } from "@azure/ai-voicelive";

const credential = new DefaultAzureCredential();
const endpoint = "https://your-resource.cognitiveservices.azure.com";

const client = new VoiceLiveClient(endpoint, credential);

API 密钥

import { AzureKeyCredential } from "@azure/core-auth";
import { VoiceLiveClient } from "@azure/ai-voicelive";

const endpoint = "https://your-resource.cognitiveservices.azure.com";
const credential = new AzureKeyCredential("your-api-key");

const client = new VoiceLiveClient(endpoint, credential);

客户端层级

VoiceLiveClient
└── VoiceLiveSession (WebSocket connection)
    ├── updateSession()      → Configure session options
    ├── subscribe()          → Event handlers (Azure SDK pattern)
    ├── sendAudio()          → Stream audio input
    ├── addConversationItem() → Add messages/function outputs
    └── sendEvent()          → Send raw protocol events

快速入门

import { DefaultAzureCredential } from "@azure/identity";
import { VoiceLiveClient } from "@azure/ai-voicelive";

const credential = new DefaultAzureCredential();
const endpoint = process.env.AZURE_VOICELIVE_ENDPOINT!;

// Create client and start session
const client = new VoiceLiveClient(endpoint, credential);
const session = await client.startSession("gpt-4o-mini-realtime-preview");

// Configure session
await session.updateSession({
  modalities: ["text", "audio"],
  instructions: "You are a helpful AI assistant. Respond naturally.",
  voice: {
    type: "azure-standard",
    name: "en-US-AvaNeural",
  },
  turnDetection: {
    type: "server_vad",
    threshold: 0.5,
    prefixPaddingMs: 300,
    silenceDurationMs: 500,
  },
  inputAudioFormat: "pcm16",
  outputAudioFormat: "pcm16",
});

// Subscribe to events
const subscription = session.subscribe({
  onResponseAudioDelta: async (event, context) => {
    // Handle streaming audio output
    const audioData = event.delta;
    playAudioChunk(audioData);
  },
  onResponseTextDelta: async (event, context) => {
    // Handle streaming text
    process.stdout.write(event.delta);
  },
  onConversationItemInputAudioTranscriptionCompleted: async (event, context) => {
    console.log("User said:", event.transcript);
  },
});

// Send audio from microphone
function sendAudioChunk(audioBuffer: ArrayBuffer) {
  return session.sendAudio(audioBuffer);
}

会话配置

await session.updateSession({
  // Modalities
  modalities: ["audio", "text"],
  
  // System instructions
  instructions: "You are a customer service representative.",
  
  // Voice selection
  voice: {
    type: "azure-standard",  // or "azure-custom", "openai"
    name: "en-US-AvaNeural",
  },
  
  // Turn detection (VAD)
  turnDetection: {
    type: "server_vad",      // or "azure_semantic_vad"
    threshold: 0.5,
    prefixPaddingMs: 300,
    silenceDurationMs: 500,
  },
  
  // Audio formats
  inputAudioFormat: "pcm16",
  outputAudioFormat: "pcm16",
  
  // Tools (function calling)
  tools: [
    {
      type: "function",
      name: "get_weather",
      description: "Get current weather",
      parameters: {
        type: "object",
        properties: {
          location: { type: "string" }
        },
        required: ["location"]
      }
    }
  ],
  toolChoice: "auto",
});

事件处理（Azure SDK 模式）

该 SDK 使用基于订阅的事件处理模式：

const subscription = session.subscribe({
  // Connection lifecycle
  onConnected: async (args, context) => {
    console.log("Connected:", args.connectionId);
  },
  onDisconnected: async (args, context) => {
    console.log("Disconnected:", args.code, args.reason);
  },
  onError: async (args, context) => {
    console.error("Error:", args.error.message);
  },
  
  // Session events
  onSessionCreated: async (event, context) => {
    console.log("Session created:", context.sessionId);
  },
  onSessionUpdated: async (event, context) => {
    console.log("Session updated");
  },
  
  // Audio input events (VAD)
  onInputAudioBufferSpeechStarted: async (event, context) => {
    console.log("Speech started at:", event.audioStartMs);
  },
  onInputAudioBufferSpeechStopped: async (event, context) => {
    console.log("Speech stopped at:", event.audioEndMs);
  },
  
  // Transcription events
  onConversationItemInputAudioTranscriptionCompleted: async (event, context) => {
    console.log("User said:", event.transcript);
  },
  onConversationItemInputAudioTranscriptionDelta: async (event, context) => {
    process.stdout.write(event.delta);
  },
  
  // Response events
  onResponseCreated: async (event, context) => {
    console.log("Response started");
  },
  onResponseDone: async (event, context) => {
    console.log("Response complete");
  },
  
  // Streaming text
  onResponseTextDelta: async (event, context) => {
    process.stdout.write(event.delta);
  },
  onResponseTextDone: async (event, context) => {
    console.log("\n--- Text complete ---");
  },
  
  // Streaming audio
  onResponseAudioDelta: async (event, context) => {
    const audioData = event.delta;
    playAudioChunk(audioData);
  },
  onResponseAudioDone: async (event, context) => {
    console.log("Audio complete");
  },
  
  // Audio transcript (what assistant said)
  onResponseAudioTranscriptDelta: async (event, context) => {
    process.stdout.write(event.delta);
  },
  
  // Function calling
  onResponseFunctionCallArgumentsDone: async (event, context) => {
    if (event.name === "get_weather") {
      const args = JSON.parse(event.arguments);
      const result = await getWeather(args.location);
      
      await session.addConversationItem({
        type: "function_call_output",
        callId: event.callId,
        output: JSON.stringify(result),
      });
      
      await session.sendEvent({ type: "response.create" });
    }
  },
  
  // Catch-all for debugging
  onServerEvent: async (event, context) => {
    console.log("Event:", event.type);
  },
});

// Clean up when done
await subscription.close();

函数调用

// Define tools in session config
await session.updateSession({
  modalities: ["audio", "text"],
  instructions: "Help users with weather information.",
  tools: [
    {
      type: "function",
      name: "get_weather",
      description: "Get current weather for a location",
      parameters: {
        type: "object",
        properties: {
          location: {
            type: "string",
            description: "City and state or country",
          },
        },
        required: ["location"],
      },
    },
  ],
  toolChoice: "auto",
});

// Handle function calls
const subscription = session.subscribe({
  onResponseFunctionCallArgumentsDone: async (event, context) => {
    if (event.name === "get_weather") {
      const args = JSON.parse(event.arguments);
      const weatherData = await fetchWeather(args.location);
      
      // Send function result
      await session.addConversationItem({
        type: "function_call_output",
        callId: event.callId,
        output: JSON.stringify(weatherData),
      });
      
      // Trigger response generation
      await session.sendEvent({ type: "response.create" });
    }
  },
});

语音选项

语音类型	配置	示例
Azure 标准	{ type: "azure-standard", name: "..." }	"en-US-AvaNeural"
Azure 自定义	{ type: "azure-custom", name: "...", endpointId: "..." }	自定义语音终结点
Azure 个人语音	{ type: "azure-personal", speakerProfileId: "..." }	个人语音克隆
OpenAI	{ type: "openai", name: "..." }	"alloy"、"echo"、"shimmer"

支持的模型

模型	描述	使用场景
gpt-4o-realtime-preview	支持实时音频的 GPT-4o	高质量对话式 AI
gpt-4o-mini-realtime-preview	轻量版 GPT-4o	快速、高效的交互
phi4-mm-realtime	Phi 多模态模型	经济高效的应用

轮次检测选项

// Server VAD (default)
turnDetection: {
  type: "server_vad",
  threshold: 0.5,
  prefixPaddingMs: 300,
  silenceDurationMs: 500,
}

// Azure Semantic VAD (smarter detection)
turnDetection: {
  type: "azure_semantic_vad",
}

// Azure Semantic VAD (English optimized)
turnDetection: {
  type: "azure_semantic_vad_en",
}

// Azure Semantic VAD (Multilingual)
turnDetection: {
  type: "azure_semantic_vad_multilingual",
}

音频格式

格式	采样率	使用场景
pcm16	24kHz	默认，高质量
pcm16-8000hz	8kHz	电话通信
pcm16-16000hz	16kHz	语音助手
g711_ulaw	8kHz	电话通信（美国）
g711_alaw	8kHz	电话通信（欧洲）

关键类型参考

类型	用途
VoiceLiveClient	用于创建会话的主客户端
VoiceLiveSession	活动的 WebSocket 会话
VoiceLiveSessionHandlers	事件处理程序接口
VoiceLiveSubscription	活动事件订阅
ConnectionContext	连接事件的上下文
SessionContext	会话事件的上下文
ServerEventUnion	所有服务器事件的联合类型

错误处理

import {
  VoiceLiveError,
  VoiceLiveConnectionError,
  VoiceLiveAuthenticationError,
  VoiceLiveProtocolError,
} from "@azure/ai-voicelive";

const subscription = session.subscribe({
  onError: async (args, context) => {
    const { error } = args;
    
    if (error instanceof VoiceLiveConnectionError) {
      console.error("Connection error:", error.message);
    } else if (error instanceof VoiceLiveAuthenticationError) {
      console.error("Auth error:", error.message);
    } else if (error instanceof VoiceLiveProtocolError) {
      console.error("Protocol error:", error.message);
    }
  },
  
  onServerError: async (event, context) => {
    console.error("Server error:", event.error?.message);
  },
});

日志记录

import { setLogLevel } from "@azure/logger";

// Enable verbose logging
setLogLevel("info");

// Or via environment variable
// AZURE_LOG_LEVEL=info

浏览器用法

// Browser requires bundler (Vite, webpack, etc.)
import { VoiceLiveClient } from "@azure/ai-voicelive";
import { InteractiveBrowserCredential } from "@azure/identity";

// Use browser-compatible credential
const credential = new InteractiveBrowserCredential({
  clientId: "your-client-id",
  tenantId: "your-tenant-id",
});

const client = new VoiceLiveClient(endpoint, credential);

// Request microphone access
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const audioContext = new AudioContext({ sampleRate: 24000 });

// Process audio and send to session
// ... (see samples for full implementation)

最佳实践

1. 使用适合环境的凭据 — DefaultAzureCredential 是服务器端模式；请使用受支持的交互式浏览器身份验证，并且切勿暴露服务密钥
2. 设置两种模态 — 为语音助手添加 ["text", "audio"]
3. 评估 VAD — 根据语言、延迟和噪声条件选择并测试受支持的模式
4. 处理所有错误类型 — 连接、身份验证和协议错误
5. 清理订阅 — 完成后调用 subscription.close()
6. 使用合适的音频格式 — 为获得最佳质量，请使用 24kHz 的 PCM16

参考链接

资源	URL
npm 包	https://www.npmjs.com/package/@azure/ai-voicelive
GitHub 源码	https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/ai/ai-voicelive
示例	https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/ai/ai-voicelive/samples
API 参考	https://learn.microsoft.com/javascript/api/@azure/ai-voicelive

适用场景

此技能适用于执行概述中所述的工作流或操作。

局限性

• 仅当任务明确符合上述范围时，才使用此技能。
• 不要将输出视为可替代特定环境的验证、测试或专家审查。
• 如果缺少必要的输入、权限、安全边界或成功标准，请停止并请求澄清。

代码片段是彼此独立、未经编译的示例，其中省略了应用辅助函数。模型、语音和格式支持因地区和 API 版本而异。必要时，请先订阅再发送输入，以免错过事件。请验证解析后的工具参数，并独立授权操作；模型输出不可信。录制麦克风音频或克隆语音前，请先获得许可，并明确音频和转录文本的保留政策。无论通过何种路径退出，都要停止音轨、关闭音频上下文、取消订阅并关闭底层会话。仅指定 AudioContext 采样率，并不能自动将捕获的浏览器音频编码为 PCM16；请使用固定版本对应的 SDK 示例进行重采样和编码，并限制缓冲区大小及背压。