vectordb

向量数据库技能

为 Golden Armada AI Agent Fleet Platform 提供全面的向量数据库功能。

何时使用此技能

在处理以下任务时启用此技能：

• 语义搜索实现
• RAG（检索增强生成）
• 嵌入的存储与检索
• 相似度搜索
• 向量索引管理

嵌入生成

```python import openai from anthropic import Anthropic

OpenAI 嵌入

def get_openai_embedding(text: str) -> list[float]: response = openai.embeddings.create( model="text-embedding-3-small", input=text ) return response.data[0].embedding

批量嵌入

def get_batch_embeddings(texts: list[str]) -> list[list[float]]: response = openai.embeddings.create( model="text-embedding-3-small", input=texts ) return [item.embedding for item in response.data] ```

Pinecone

```python from pinecone import Pinecone, ServerlessSpec

初始化

pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])

创建索引

pc.create_index( name="agents", dimension=1536, metric="cosine", spec=ServerlessSpec( cloud="aws", region="us-west-2" ) )

获取索引

index = pc.Index("agents")

写入向量

index.upsert( vectors=[ { "id": "agent-1", "values": embedding, "metadata": { "name": "Claude Agent", "type": "claude", "description": "General assistant" } } ], namespace="production" )

查询

results = index.query( vector=query_embedding, top_k=10, include_metadata=True, namespace="production", filter={"type": {"$eq": "claude"}} )

for match in results.matches: print(f"{match.id}: {match.score} - {match.metadata}")

删除

index.delete(ids=["agent-1"], namespace="production") ```

Chroma

```python import chromadb from chromadb.config import Settings

初始化

client = chromadb.PersistentClient(path="./chroma_db")

创建集合

collection = client.get_or_create_collection( name="agents", metadata={"hnsw:space": "cosine"} )

添加文档

collection.add( ids=["agent-1", "agent-2"], embeddings=[embedding1, embedding2], documents=["Document 1 text", "Document 2 text"], metadatas=[ {"type": "claude", "version": "3"}, {"type": "gpt", "version": "4"} ] )

查询

results = collection.query( query_embeddings=[query_embedding], n_results=10, where={"type": "claude"}, include=["documents", "metadatas", "distances"] )

更新

collection.update( ids=["agent-1"], metadatas=[{"type": "claude", "version": "3.5"}] )

删除

collection.delete(ids=["agent-1"]) ```

pgvector（PostgreSQL）

```sql -- 启用扩展 CREATE EXTENSION vector;

-- 创建表 CREATE TABLE documents ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), content TEXT NOT NULL, embedding vector(1536), metadata JSONB DEFAULT '{}' );

-- 创建索引（适用于较大数据集的 IVFFlat） CREATE INDEX ON documents USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);

-- 或使用 HNSW 以获得更高的召回率 CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops);

-- 插入 INSERT INTO documents (content, embedding, metadata) VALUES ('Document content', '[0.1, 0.2, ...]', '{"type": "manual"}');

-- 相似度搜索 SELECT id, content, 1 - (embedding <=> $1) AS similarity FROM documents ORDER BY embedding <=> $1 LIMIT 10;

-- 添加筛选条件 SELECT id, content, 1 - (embedding <=> $1) AS similarity FROM documents WHERE metadata->>'type' = 'manual' ORDER BY embedding <=> $1 LIMIT 10; ```

使用 pgvector 的 Python

from pgvector.sqlalchemy import Vector
from sqlalchemy import Column, String, JSON
from sqlalchemy.dialects.postgresql import UUID

class Document(Base):
    __tablename__ = 'documents'

    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
    content = Column(String, nullable=False)
    embedding = Column(Vector(1536))
    metadata = Column(JSON, default={})

# 插入
doc = Document(
    content="Document content",
    embedding=embedding,
    metadata={"type": "manual"}
)
session.add(doc)
session.commit()

# 查询
from sqlalchemy import select
from pgvector.sqlalchemy import cosine_distance

results = session.execute(
    select(Document)
    .order_by(cosine_distance(Document.embedding, query_embedding))
    .limit(10)
).scalars().all()

RAG 实现

class RAGService:
    def __init__(self, vector_store, llm_client):
        self.vector_store = vector_store
        self.llm = llm_client

    async def query(self, question: str, top_k: int = 5) -> str:
        # 1. 生成查询嵌入
        query_embedding = await self.get_embedding(question)

        # 2. 检索相关文档
        docs = await self.vector_store.search(
            embedding=query_embedding,
            top_k=top_k
        )

        # 3. 构建上下文
        context = "\n\n".join([
            f"Document {i+1}:\n{doc.content}"
            for i, doc in enumerate(docs)
        ])

        # 4. 生成回复
        prompt = f"""Answer the question based on the following context.

Context:
{context}

Question: {question}

Answer:"""

        response = await self.llm.generate(prompt)
        return response

    async def add_document(self, content: str, metadata: dict = None):
        # 对文档分块
        chunks = self.chunk_text(content)

        # 生成嵌入
        embeddings = await self.get_batch_embeddings(chunks)

        # 存储
        for i, (chunk, embedding) in enumerate(zip(chunks, embeddings)):
            await self.vector_store.upsert(
                id=f"{metadata.get('doc_id', 'doc')}-{i}",
                embedding=embedding,
                content=chunk,
                metadata=metadata
            )

    def chunk_text(self, text: str, chunk_size: int = 1000, overlap: int = 200) -> list[str]:
        chunks = []
        start = 0
        while start < len(text):
            end = start + chunk_size
            chunk = text[start:end]
            chunks.append(chunk)
            start += chunk_size - overlap
        return chunks

最佳实践

1. 选择合适的索引类型（追求召回率时使用 HNSW，追求规模时使用 IVFFlat）
2. 合理切分文档（通常为 500-1000 个 token）
3. 在分块之间设置重叠（10-20%）
4. 存储元数据以便筛选
5. 使用命名空间/集合来组织数据
6. 监控查询延迟和索引性能
7. 批量执行操作以便批量插入