Monday, November 29, 2021
1 change · master
Resolved issues and error corrections
Improves performance when creating very large batches of records that include certain calculated fields. This reduces long processing times for bulk imports or automated data creation, making large operations complete much faster.
Original PR description
Issue: ------ The `create` method (models.py) doesn't scale correctly (with huge number of values) **when compute store no-readonly field(s) are in values** (at most one). It is due to the method…
Issue: ------ The `create` method (models.py) doesn't scale correctly (with huge number of values) **when compute store no-readonly field(s) are in values** (at most one). It is due to the method `protecting` (api.py) which has a bad complexity in the `create` situation (list of (field, records) pairs as argument): O(r²) with `r` = number of record containing the protected field: List of `len(r)` send to `protecting`, loop of this list (`r` factor), loop on fields (constant factor), create a new frozen set with the previous one (`r` factor). Fix: ---- Decrease the complexity to O(r) by creating a map of set of ids by protected field which allows avoiding recreating a new frozenset for each record (update with tuple of one inside => O(1)). Performance gain: ---------- For a very simple model with only one compute store no-readonly field, and all `create` `values` contains this field. ``` +--------------+---------------+---------------+---------------+----------------+ | Batch -> | 1000 | 10000 | 30000 | 80000 | +--------------+---------------+---------------+---------------+----------------+ | Before (sec) | 0.083 ± 0.006 | 1.433 ± 0.091 | 8.764 ± 0.609 | 94.428 ± 3.021 | | After (sec) | 0.069 ± 0.003 | 0.706 ± 0.006 | 2.188 ± 0.076 | 5.875 ± 0.104 | +--------------+---------------+---------------+---------------+----------------+ ```