GenPark Bloom Filter URL Dedup Crawler
LocalAbout GenPark Bloom Filter URL Dedup Crawler
GenPark Bloom Filter URL Dedup Crawler uses an in-memory Bloom filter to identify duplicate URLs and document hashes. It is intended for web crawler ingestion workflows that need to avoid processing duplicates.
Key features
- 01In-memory Bloom filter
- 02URL deduplication
- 03Document-hash deduplication
- 04Python standard-library runtime
Use cases
- 01Filtering duplicate URLs during crawling
- 02Avoiding repeated document ingestion
MCP support
It runs as an MCP server and can be used by MCP clients. The README also describes direct library use.
Configuration
Local server config is the same for every client. Copy it into your client’s MCP config file.
We haven’t collected a package or remote endpoint for this server. Follow the instructions in the project README.
Tools
Local servers expose tools only once running on your machine, and the maintainer has not provided a tool list.
Details
From the project READMEGenPark Bloom Filter URL Dedup Crawler
In-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler ingestion.
Part of the GenPark AI Agent Skills Ecosystem.
🌟 Key Features
- 100% Python Standard Library: Zero third-party runtime dependencies.
- MCP Native: Plugs directly into Claude Desktop, Cursor, LobeChat, and autonomous agent DAGs.
- Production Resilience: Strict schema conformance, zero data loss, enterprise hardened.
🚀 Installation & Usage
Run as an MCP server:
python mcp_server.py
Use directly as a library:
from client import *
See example_usage.py for comprehensive integration demonstrations.
📄 License
MIT License. Maintained by GenPark AI.
Comments
New to MCP?
From what the protocol is to installing your first server, these answers cover most first questions.
An open protocol that defines how AI applications connect to external tools and data in a uniform way. Clients talk to the model; MCP servers provide tools, resources and prompt templates. They exchange standard messages, so one server can be reused by many clients.