azure-ai-contentsafety-ts

用于 TypeScript 的 Azure AI Content Safety REST SDK

使用可自定义的阻止列表分析文本和图像中的有害内容。

安装

npm install @azure-rest/ai-content-safety @azure/identity @azure/core-auth

环境变量

CONTENT_SAFETY_ENDPOINT=https://<resource>.cognitiveservices.azure.com
CONTENT_SAFETY_KEY=<api-key>

身份验证

重要提示：这是一个 REST 客户端。ContentSafetyClient 是一个函数，不是类。

API 密钥

import ContentSafetyClient from "@azure-rest/ai-content-safety";
import { AzureKeyCredential } from "@azure/core-auth";

const client = ContentSafetyClient(
  process.env.CONTENT_SAFETY_ENDPOINT!,
  new AzureKeyCredential(process.env.CONTENT_SAFETY_KEY!)
);

DefaultAzureCredential

import ContentSafetyClient from "@azure-rest/ai-content-safety";
import { DefaultAzureCredential } from "@azure/identity";

const client = ContentSafetyClient(
  process.env.CONTENT_SAFETY_ENDPOINT!,
  new DefaultAzureCredential()
);

分析文本

import ContentSafetyClient, { isUnexpected } from "@azure-rest/ai-content-safety";

const result = await client.path("/text:analyze").post({
  body: {
    text: "Text content to analyze",
    categories: ["Hate", "Sexual", "Violence", "SelfHarm"],
    outputType: "FourSeverityLevels"  // or "EightSeverityLevels"
  }
});

if (isUnexpected(result)) {
  throw result.body;
}

for (const analysis of result.body.categoriesAnalysis) {
  console.log(`${analysis.category}: severity ${analysis.severity}`);
}

分析图像

Base64 内容

import { readFileSync } from "node:fs";

const imageBuffer = readFileSync("./image.png");
const base64Image = imageBuffer.toString("base64");

const result = await client.path("/image:analyze").post({
  body: {
    image: { content: base64Image }
  }
});

if (isUnexpected(result)) {
  throw result.body;
}

for (const analysis of result.body.categoriesAnalysis) {
  console.log(`${analysis.category}: severity ${analysis.severity}`);
}

Blob URL

const result = await client.path("/image:analyze").post({
  body: {
    image: { blobUrl: "https://storage.blob.core.windows.net/container/image.png" }
  }
});

阻止列表管理

创建阻止列表

const result = await client
  .path("/text/blocklists/{blocklistName}", "my-blocklist")
  .patch({
    contentType: "application/merge-patch+json",
    body: {
      description: "Custom blocklist for prohibited terms"
    }
  });

if (isUnexpected(result)) {
  throw result.body;
}

console.log(`Created: ${result.body.blocklistName}`);

向阻止列表添加项

const result = await client
  .path("/text/blocklists/{blocklistName}:addOrUpdateBlocklistItems", "my-blocklist")
  .post({
    body: {
      blocklistItems: [
        { text: "prohibited-term-1", description: "First blocked term" },
        { text: "prohibited-term-2", description: "Second blocked term" }
      ]
    }
  });

if (isUnexpected(result)) {
  throw result.body;
}

for (const item of result.body.blocklistItems ?? []) {
  console.log(`Added: ${item.blocklistItemId}`);
}

使用阻止列表进行分析

const result = await client.path("/text:analyze").post({
  body: {
    text: "Text that might contain blocked terms",
    blocklistNames: ["my-blocklist"],
    haltOnBlocklistHit: false
  }
});

if (isUnexpected(result)) {
  throw result.body;
}

// Check blocklist matches
if (result.body.blocklistsMatch) {
  for (const match of result.body.blocklistsMatch) {
    console.log(`Blocked: "${match.blocklistItemText}" from ${match.blocklistName}`);
  }
}

列出阻止列表

const result = await client.path("/text/blocklists").get();

if (isUnexpected(result)) {
  throw result.body;
}

for (const blocklist of result.body.value ?? []) {
  console.log(`${blocklist.blocklistName}: ${blocklist.description}`);
}

删除阻止列表

await client.path("/text/blocklists/{blocklistName}", "my-blocklist").delete();

有害类别

类别	API 术语	说明
仇恨与公平性	Hate	针对身份群体的歧视性语言
色情	Sexual	色情内容、裸露、色情制品
暴力	Violence	身体伤害、武器、恐怖主义
自残	SelfHarm	自伤、自杀、饮食失调

严重程度级别

级别	风险	建议操作
0	安全	允许
2	低	审核或发出警告后允许
4	中	阻止或要求人工审核
6	高	立即阻止

输出类型：

• FourSeverityLevels（默认）：返回 0、2、4、6
• EightSeverityLevels：返回 0-7

内容审核辅助工具

import ContentSafetyClient, { 
  isUnexpected, 
  TextCategoriesAnalysisOutput 
} from "@azure-rest/ai-content-safety";

interface ModerationResult {
  isAllowed: boolean;
  flaggedCategories: string[];
  maxSeverity: number;
  blocklistMatches: string[];
}

async function moderateContent(
  client: ReturnType<typeof ContentSafetyClient>,
  text: string,
  maxAllowedSeverity = 2,
  blocklistNames: string[] = []
): Promise<ModerationResult> {
  const result = await client.path("/text:analyze").post({
    body: { text, blocklistNames, haltOnBlocklistHit: false }
  });

  if (isUnexpected(result)) {
    throw result.body;
  }

  const analyses = result.body.categoriesAnalysis;
  const expected = ["Hate", "SelfHarm", "Sexual", "Violence"];
  if (!Array.isArray(analyses) || expected.some(category =>
    !analyses.some(c => c.category === category && Number.isInteger(c.severity) && c.severity! >= 0 && c.severity! <= 7))) {
    throw new Error("Incomplete moderation assessment");
  }

  const flaggedCategories = analyses
    .filter(c => c.severity! > maxAllowedSeverity)
    .map(c => c.category!);

  const maxSeverity = Math.max(
    ...analyses.map(c => c.severity!)
  );

  const blocklistMatches = (result.body.blocklistsMatch ?? [])
    .map(m => m.blocklistItemText!);

  return {
    isAllowed: flaggedCategories.length === 0 && blocklistMatches.length === 0,
    flaggedCategories,
    maxSeverity,
    blocklistMatches
  };
}

API 终结点

操作	方法	路径
分析文本	POST	/text:analyze
分析图像	POST	/image:analyze
创建/更新阻止列表	PATCH	/text/blocklists/{blocklistName}
列出阻止列表	GET	/text/blocklists
删除阻止列表	DELETE	/text/blocklists/{blocklistName}
添加阻止列表项	POST	/text/blocklists/{blocklistName}:addOrUpdateBlocklistItems
列出阻止列表项	GET	/text/blocklists/{blocklistName}/blocklistItems
移除阻止列表项	POST	/text/blocklists/{blocklistName}:removeBlocklistItems

关键类型

import ContentSafetyClient, {
  isUnexpected,
  AnalyzeTextParameters,
  AnalyzeImageParameters,
  TextCategoriesAnalysisOutput,
  ImageCategoriesAnalysisOutput,
  TextBlocklist,
  TextBlocklistItem
} from "@azure-rest/ai-content-safety";

最佳实践

1. 始终使用 isUnexpected() - 用于错误处理的类型守卫
2. 设置适当的阈值 - 不同类别可能需要不同的严重程度阈值
3. 将阻止列表用于特定领域的术语 - 使用自定义规则补充 AI 检测
4. 记录审核决策 - 保留审计记录以满足合规要求
5. 处理边界情况 - 空文本、超长文本、不受支持的图像格式

适用场景

当任务与上述概述所述的工作流或操作相符时，可使用此技能。

限制

• 仅当任务明确符合上述范围时，才使用此技能。
• 请勿将输出视为环境特定验证、测试或专家审核的替代品。
• 如果缺少必需的输入、权限、安全边界或成功标准，请暂停并请求澄清。

严重程度为零是模型的结果，并不能证明内容安全；请针对应用校准阈值和升级处理流程。未完成的分析和服务故障仍属于未经评估。请先授权将私密内容传输到云端，并且不要在公开日志中暴露原始错误正文或阻止列表中的文本。