Friday, May 17, 2024
1 change
Resolved issues and error corrections
This fix addresses memory errors that occur when updating large numbers of records in the system, which commonly happens during software upgrades. The change optimizes how the system processes and stores temporary data during these updates, reducing memory consumption from tens of megabytes to just 1-2 megabytes, allowing upgrades to complete successfully without running out of memory.
Original PR description
Motivation: MemoryError exceptions when a large number of records on the same model have dirty fields. Such often happens during upgrades. In the current implementation, the cached data is re-arranged in multiple steps using local data structures. The most problematic is `id_vals[record.id][field.name]`, because it creates a dictionary with a potentially long field name (think studio fields) as key for each dirty record. For thousands of records, this quickly accumulates to 10s of MiB in RAM. The idea of this patch is: 1. collect all dirty ids for all dirty fields on the model. This does not cost additional memory, since the list of ids per field will be pop()'ed from the cache. 2. Walk over fields and ids collecting all fields and values of each id in the same loop, directly building the `updates` dictionary, without creating the intermediate data structure.