Django QuerySet Batch Processing

Use this skill when Django code processes many rows. The goal is to avoid loading unnecessary model instances, avoid queryset result-cache blowups, and move writes into set-based database operations when behavior allows.

Workflow

1. Identify the per-row work.

- Is it read-only export/reporting? - Does it need model methods, validation, or signals? - Can the database compute or update the value directly?

1. Choose the read pattern.

- Use values() or valueslist() for scalar exports and reports. - Use iterator(chunksize=...) when model instances are needed but queryset caching is not. - Keep ordering deliberate; unnecessary ordering costs work.

1. Choose the write pattern.

- Use QuerySet.update() with F() or expressions for uniform updates. - Use bulkupdate() when each object has a different value. - Use bulkcreate() for inserts, with conflict options only when the project supports their semantics. - Fall back to per-instance save() only when hooks, validation, side effects, or signals are required.

1. Control batch size.

- Keep transactions bounded. - Avoid huge IN lists and oversized CASE updates. - Monitor locks, replication lag, and memory for production jobs.

See [batch-patterns.md](references/batch-patterns.md) for examples and caveats.

Safety Notes

• Bulk update/delete operations do not call each model instance's save() or delete() methods.
• Bulk operations can skip application-level side effects and signals.
• Long transactions can hold locks and delay vacuum or replication.

Verification

Measure rows processed per second, query count, memory, transaction duration, and correctness on a representative batch.