Sunday, May 19, 2024
1 change
Resolved issues and error corrections
This fix optimizes how Odoo handles memory when updating large numbers of records, reducing memory consumption by approximately 95%. The improvement prevents MemoryError exceptions that commonly occur during system upgrades when processing thousands of records simultaneously, making upgrades more stable and reliable.
Original PR description
This is re-targeting https://github.com/odoo/odoo/pull/162442 to 16.0 Motivation: MemoryError exceptions when a large number of records on the same model have dirty fields. Such often happens during…
This is re-targeting https://github.com/odoo/odoo/pull/162442 to 16.0 Motivation: MemoryError exceptions when a large number of records on the same model have dirty fields. Such often happens during upgrades. In the current implementation, the cached data is re-arranged in multiple steps using local data structures. The most problematic is `id_vals[record.id][field.name]`, because it creates a dictionary with a potentially long field name (think studio fields) as key for each dirty record. For thousands of records, this quickly accumulates to 10s or even 100s of MiB in RAM. The idea of this patch is: 1. collect all dirty ids for all dirty fields on the model. This does not cost additional memory, since the ist of ids per field will be pop()'ed from the cache. 2. Walk over fields and ids collecting all fields and values of each id in the same loop, carefully consuming objects, while directly building the `updates` dictionary, without creating the intermediate data structures. This way, the _flush method only consumes a marginal amount of memory compared to the memory already consumed by the cache. Careful profiling (using `memray` for memory profiling, the odoo profiler for runtime profiling) of a testcase that flushes 80k records in 8 writes has shown this new version to consume roughly 95% less memory while running slightly faster. Forward-Port-Of: odoo/odoo#165393