mcpField
D
doctrine-batch-processing

doctrine-batch-processing

github.com/dev-toolingsData & databases22788 installsUpdated 3w ago
Repository

About doctrine-batch-processing

A Symfony skill for processing large datasets with Doctrine while controlling memory use. It covers streaming or ID-based reads, periodic flushes and clears, and choosing ORM, DQL, or DBAL for writes.

Key features

  1. 01Stream reads with toIterable()
  2. 02Paginate large tables by ID
  3. 03Flush and clear in batches
  4. 04Choose ORM, DQL, or DBAL for writes
  5. 05Track memory and resumable progress

Use cases

  1. 01Import or export datasets larger than memory
  2. 02Prevent memory exhaustion in commands or workers
  3. 03Perform bulk updates, deletes, or inserts

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/dev-toolings/superpowers-symfony --skill doctrine-batch-processing

After installing, type/doctrine-batch-processingin Claude Code to confirm it loaded.

Bundled filesreference.md

Details

From the project README

Doctrine Batch Processing (Symfony)

Use when

  • Processing more rows than fit in memory: imports, exports, backfills.
  • A command or worker runs out of memory iterating over entities.
  • Updating or deleting many rows at once.
  • Inserting large volumes through Doctrine or DBAL.

Default workflow

  1. Measure the dataset size and the memory budget of the process.
  2. Stream reads with toIterable(), or paginate by id (id > :lastId) for very large tables.
  3. Flush and clear() every N items, and re-fetch anything needed after a clear.
  4. Use DQL UPDATE/DELETE or DBAL for bulk writes that need no hydration.
  5. Log memory usage and progress, and make the run resumable.

Guardrails

  • Never findAll() on a table that grows.
  • Entities loaded before a clear() are detached: do not modify or persist them.
  • Bulk DQL and DBAL skip lifecycle events and listeners. Check what relies on them.
  • On ORM 3, Query#iterate() and setSQLLogger() are gone: use toIterable() and a logging middleware.
  • Paginate by id rather than by offset: offsets get slower and skip rows when data moves.

Progressive disclosure

  • Use this file for execution posture and risk controls.
  • Open references when deep implementation details are needed.

Output contract

  • A batch loop with an explicit batch size and clear points.
  • The choice between ORM, DQL bulk and DBAL for each write, with the reason.
  • Memory measurements before and after on a realistic volume.

References

  • reference.md

Comments

Markdown supported