mcpField
A
agent-survey-corpus

agent-survey-corpus

github.com/willoscarDocuments51742 installsUpdated 3w ago
Repository

About agent-survey-corpus

An optional toolkit for building a local reference corpus of agent surveys from arXiv PDFs. It helps researchers and writing pipelines study survey structure, comparison practices, and style using downloaded papers and text extracts.

Key features

  1. 01Download arXiv survey PDFs
  2. 02Extract text from selected pages
  3. 03Review section and subsection structure
  4. 04Generate a style report

Use cases

  1. 01Study paper-like section organization
  2. 02Compare subsection size and transitions
  3. 03Identify evidence-backed comparison patterns

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/willoscar/research-units-pipeline-skills --skill agent-survey-corpus

After installing, type/agent-survey-corpusin Claude Code to confirm it loaded.

Bundled filesscripts/run.py

Details

From the project README

Agent Survey Corpus (arXiv PDFs → text extracts)

Triggers & routing

  • Trigger: agent survey corpus, ref corpus, download surveys, 学习综述写法, 下载 survey.
  • Use when: you want to study how real agent surveys structure sections (6–8 H2), size subsections, and write evidence-backed comparisons.

Goal: create a small, local reference library so you can learn from real agent surveys when refining:

  • C2 outline structure (paper-like sectioning)
  • C4 tables/claims organization
  • C5 writing style and density

This is intentionally not part of the pipeline; it is an optional, repo-level toolkit.

Inputs

  • ref/agent-surveys/arxiv_ids.txt

Outputs

  • ref/agent-surveys/pdfs/
  • ref/agent-surveys/text/
  • ref/agent-surveys/STYLE_REPORT.md (tracked; auto-generated summary)

Workflow

  1. Edit ref/agent-surveys/arxiv_ids.txt (one arXiv id per line).
  2. Run the downloader to fetch PDFs and extract the first N pages to text.
  3. Skim the extracted text under ref/agent-surveys/text/:
    • look at section counts (H2), subsection granularity (H3), and how they transition between chapters.
    • identify repeated rhetorical patterns you want the pipeline writer to imitate.

Script

Quick Start
  • uv run python .codex/skills/agent-survey-corpus/scripts/run.py --help
  • uv run python .codex/skills/agent-survey-corpus/scripts/run.py --workspace . --max-pages 20
All Options
  • --workspace <dir> (use . to write into repo root)
  • --inputs <semicolon-separated> (default: ref/agent-surveys/arxiv_ids.txt)
  • --max-pages <N> (default: 20)
  • --sleep <seconds> (default: 1.0)
  • --overwrite (re-download + re-extract)
Examples
  • Download/extract into repo root ref/:
    • uv run python .codex/skills/agent-survey-corpus/scripts/run.py --workspace . --max-pages 20
  • Download/extract into a specific folder (treated as workspace root):
    • uv run python .codex/skills/agent-survey-corpus/scripts/run.py --workspace /tmp/surveys --max-pages 30

Troubleshooting

  • Download fails / timeout: rerun with a larger --sleep, or try fewer ids.
  • Text extract is empty: the PDF may be scanned; try another survey or increase --max-pages.
  • Files showing up in git status: PDFs/text are ignored via .gitignore (ref/**/pdfs/, ref/**/text/).

Comments

Markdown supported