Unstructured API MCP Server for Research Paper Data Processing

用于研究论文数据处理的 Unstructured API MCP 服务

借助 Unstructured API，此服务可轻松访问一组强大的工具，从研究论文中提取有意义的信息，随后可将这些信息用于微调大模型（LLM），从而缩短研究人员的文献综述时间。

点击此处查看博客：

• 如需详细了解黑客马拉松和项目，并跟进相关内容，请查看我发布在 Dev.to 上的博客文章：Unstructured Model Context Protocol Hackathon。

目录：

1. 设置
2. 要求
3. 项目流程
4. 可用工具
5. 跟着操作
6. Claude Desktop 集成
7. 调试工具
8. 在本地运行最简客户端并通过 HTTP 和 SSE 访问本地 MCP 服务

设置

安装依赖项：

• uv add "mcp[cli]"
• uv pip install --upgrade unstructured-client python-dotenv

或使用 uv sync。

要求

开始使用 UNS_MCP 项目前，请确保已完成以下设置：

1. UNSTRUCTURED_API_KEY

   • 从 Unstructured 平台获取 API 密钥，以访问其文档处理 API。

2. GOOGLEDRIVE_SERVICE_ACCOUNT_KEY

   • 设置 Google Cloud 项目并创建服务账号，以便访问 Google Drive 并读取 PDF。请在此处查看设置流程。
   • 保存服务账号的 JSON 凭据，并使用它设置 GOOGLEDRIVE_SERVICE_ACCOUNT_KEY。

3. MONGO_DB_CONNECTION_STRING

   • 设置 MongoDB 数据库（云端），并获取用于连接数据库的连接字符串。请在此处查看设置流程。

4. .env.template

   • .env.template 文件包含所有必需的环境变量。将此文件复制为 .env，并为上述密钥设置必要的值。

   .env 文件示例：

   UNSTRUCTURED_API_KEY="<key-here>"
   MONGO_DB_CONNECTION_STRING="<CONNECTION_STRING>"
   GOOGLEDRIVE_SERVICE_ACCOUNT_KEY="<converted string>"
   
   

Project Flow

1. User Query to MCP Client

2. Claude Interacts with UNS_MCP Server

   • Claude forwards the user's query to the custom MCP server named UNS_MCP.

3. MCP Tool Executes Unstructured API

   • UNS_MCP interacts with the Unstructured API to process the research paper PDF, extract relevant information, and convert it into structured JSON data.

4. Structured Data (JSON) Output is stored in the destination source

   • The result from the Unstructured API is transformed into JSON format, which can then be further utilized to fine-tune LLMs, helping researchers quickly find the relevant information without manually reading the entire paper.

Available Tools

Tool	Description
list_sources	Lists available sources from the Unstructured API.
get_source_info	Get detailed information about a specific source connector.
create_gdrive_source	Create a google drive source connector.
update_gdrive_source	Update an existing google source connector by params.
delete_gdrive_source	Delete a source connector by source id.
list_destinations	Lists available destinations from the Unstructured API.
get_destination_info	Get detailed info about a specific destination connector. Currently, we have s3/weaviate/astra/neo4j/mongo DB (more to come!)
create_mongodb_destination	Create a mongodb destination connector by params.
update_mongodb_destination	Update an existing mongodb destination connector by destination id.
delete_mongodb_destination	Delete a mongodb destination connector by destination id.
list_workflows	Lists workflows from the Unstructured API.
get_workflow_info	Get detailed information about a specific workflow.
create_workflow	Create a new workflow with source, destination id, etc.
run_workflow	Run a specific workflow with workflow id
update_workflow	Update an existing workflow by params.
delete_workflow	Delete a specific workflow by id.
list_jobs	Lists jobs for a specific workflow from the Unstructured API.
get_job_info	Get detailed information about a specific job by job id.
cancel_job	Delete a specific job by id.

Follow Along

1. Set Up Required Connectors

Google Drive Source Connector:

• Create a Google Drive Source Connector to connect your service account with Google Drive and retrieve PDFs.
• Test the connection to ensure accessibility.

MongoDB Destination Connector:

• Set up the MongoDB Destination Connector to store processed data.
• Test the connection to ensure accessibility.

---

2. Develop the Workflow

1. Define Connectors: Set up the Google Drive source and MongoDB destination connectors.

2. Partitioning: Use Auto partitioning for optimal document splitting.

3. Chunking: Apply by-page chunking for manageable text segments.

4. Enrichment: Use NER to extract entities and table enrichment for any tables.

5. Embedding: Convert text into embeddings for querying or analysis.

Note: Tweak the Flow: Adjust any step (partitioning, chunking, enrichment, embedding) as needed.

---

3. Set Up Claude Desktop

1. Install Claude Desktop and integrate it with the UNS_MCP server by following steps given below.
2. Restart Claude to link with the MCP server and ensure workflow functionality.

---

4. Query and Run the Workflow

• Use Claude to interact with the system and execute queries to list, create, edit, delete and run the workflow. You can perform many such tasks, go through Available Tools given above.

5. Results

Claude Desktop Integration

To install in Claude Desktop:

1. Go to claude_desktop_config.json by running the below command.

# For macOS or Linux:
code ~/Library/Application\ Support/Claude/claude_desktop_config.json

# For Windows:
code $env:AppData\Claude\claude_desktop_config.json

2. 在该文件中添加：

{
    "mcpServers":
    {
        "UNS_MCP":
        {
            "command": "ABSOLUTE/PATH/TO/.local/bin/uv",
            "args":
            [
                "--directory",
                "ABSOLUTE/PATH/TO/YOUR-UNS-MCP-REPO/uns_mcp",
                "run",
                "server.py"
            ],
            "env":
            [
            "UNSTRUCTURED_API_KEY":"<your key>"
            ],
            "disabled": false
        }
    }
}

3. 重启 Claude Desktop。

4. Claude Desktop 中遇到的问题示例。

   • 查询目标连接器列表时，你会看到 No destinations found。请检查 .env 或配置 JSON 中的 API 密钥，它必须是你在 https://platform.unstructured.io/app/account/api-keys 上的个人密钥。

调试工具

Anthropic 提供了 MCP Inspector 工具，用于调试/测试你的 MCP 服务。运行以下命令即可启动调试界面。在界面左侧面板中，你可以添加环境变量（指向本地环境）。请在此处将个人 API 密钥设为环境变量。进入 tools，即可测试你添加到 MCP 服务中的功能。

mcp dev uns_mcp/server.py

如果需要记录发送给 UnstructuredClient 的请求调用参数，请设置环境变量 DEBUG_API_REQUESTS=false。 日志保存在文件中，文件名格式为 unstructured-client-{date}.log，你可以查看这些日志，以调试对 UnstructuredClient 函数的请求调用参数。

本地运行最简客户端，通过 HTTP + SSE 访问本地 MCP 服务

主要区别在于，开发期间更容易在服务端设置断点——客户端和服务端是解耦的。

# in one terminal, run the server:
uv run python uns_mcp/server.py --host 127.0.0.1 --port 8080

or
make sse-server

# in another terminal, run the client:
uv run python minimal_client/client.py "http://127.0.0.1:8080/sse"
or
make sse-client

提示：先按 ctrl+c 退出客户端，再退出服务端。否则服务端看起来会卡住。