mcpField
D
django-queryset-batch-processing

django-queryset-batch-processing

github.com/lvtd-llcDevelopment161 installsUpdated 7w ago
Repository

About django-queryset-batch-processing

A skill for Django developers processing large numbers of database rows. It guides read and write patterns, batch sizing, and safety checks to limit unnecessary model loading and queryset caching while preserving required behavior.

Key features

  1. 01Choose scalar or iterator reads
  2. 02Use set-based bulk writes
  3. 03Control batch and transaction size
  4. 04Account for skipped hooks and signals
  5. 05Measure throughput, memory, and correctness

Use cases

  1. 01Building exports or reports over many rows
  2. 02Updating many records efficiently
  3. 03Running production jobs with bounded batches

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/lvtd-llc/skills --skill django-queryset-batch-processing

After installing, type/django-queryset-batch-processingin Claude Code to confirm it loaded.

Bundled filesreferences/batch-patterns.md

Details

From the project README

Django QuerySet Batch Processing

Use this skill when Django code processes many rows. The goal is to avoid loading unnecessary model instances, avoid queryset result-cache blowups, and move writes into set-based database operations when behavior allows.

Workflow

  1. Identify the per-row work.

    • Is it read-only export/reporting?
    • Does it need model methods, validation, or signals?
    • Can the database compute or update the value directly?

  2. Choose the read pattern.

    • Use values() or values_list() for scalar exports and reports.
    • Use iterator(chunk_size=...) when model instances are needed but queryset caching is not.
    • Keep ordering deliberate; unnecessary ordering costs work.

  3. Choose the write pattern.

    • Use QuerySet.update() with F() or expressions for uniform updates.
    • Use bulk_update() when each object has a different value.
    • Use bulk_create() for inserts, with conflict options only when the project supports their semantics.
    • Fall back to per-instance save() only when hooks, validation, side effects, or signals are required.

  4. Control batch size.

    • Keep transactions bounded.
    • Avoid huge IN lists and oversized CASE updates.
    • Monitor locks, replication lag, and memory for production jobs.

See batch-patterns.md for examples and caveats.

Safety Notes

  • Bulk update/delete operations do not call each model instance's save() or delete() methods.
  • Bulk operations can skip application-level side effects and signals.
  • Long transactions can hold locks and delay vacuum or replication.

Verification

Measure rows processed per second, query count, memory, transaction duration, and correctness on a representative batch.

Comments

Markdown supported