mcpField
D
dedupe-rank

dedupe-rank

github.com/willoscarData & databases51744 installsUpdated 3w ago
Repository

About dedupe-rank

dedupe-rank is a deterministic research skill that turns raw paper candidates into deduplicated records and a stable core set. It is for research workflows that need a repeatable shortlist for later organization, not retrieval or screening.

Key features

  1. 01Deduplicate by title and year
  2. 02Rank candidates deterministically
  3. 03Generate stable paper IDs
  4. 04Write deduplicated JSONL and core-set CSV

Use cases

  1. 01Prepare a paper set for taxonomy or outline work
  2. 02Re-run ranking consistently on the same input
  3. 03Convert raw candidate records into a manageable core set

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/willoscar/research-units-pipeline-skills --skill dedupe-rank

After installing, type/dedupe-rankin Claude Code to confirm it loaded.

Bundled files

References and scripts installed along with SKILL.md.

assets/domain_packs/
embodied_ai.json
llm_agents.json
references/
domain_pack_overview.md
scripts/
run.py

Details

From the project README

Dedupe + Rank

Triggers & routing

  • Trigger: dedupe, rank, core set, 去重, 排序, 精选论文, 核心集合.
  • Use when: 检索后需要把广覆盖集合收敛成可管理的 core set（用于 taxonomy/outline/mapping）。
  • Skip if: 已经有人手工整理了稳定的 papers/core_set.csv（无需再次 churn）。
  • Network: none.
  • Guardrail: 偏 deterministic；输出应可重复（稳定 paper_id、字段规范）。

Turns a raw candidate pool into a deduped pool and a stable core set.

Input

  • papers/papers_raw.jsonl

Outputs

  • papers/papers_dedup.jsonl
  • papers/core_set.csv

Script boundary

scripts/run.py should own only:

  • title/year deduplication
  • deterministic ranking
  • stable paper_id generation

Use shared domain packs or pipeline contract metadata for topic-specific or product-specific behavior.

Contract-driven behavior

The script should prefer pipeline contract metadata over profile-name branching.

Current important field:

  • quality_contract.candidate_pool_policy.keep_full_deduped_pool
  • quality_contract.candidate_pool_policy.include_domain_pins
  • quality_contract.candidate_pool_policy.minimum_domain_surveys
  • quality_contract.candidate_pool_policy.survey_title_bonus

If true, the script keeps the full deduped pool in papers/core_set.csv unless the user explicitly overrides core size. Focused briefing Workflows can disable unconditional domain pins, reserve a smaller survey floor, and remove the generic survey-title bonus without changing broad Survey defaults.

Acceptance

  • deduped JSONL exists
  • core-set CSV exists
  • reruns are stable for the same inputs

Non-goals

  • retrieval
  • screening
  • manual topic authoring inside the script

Comments

Markdown supported