Wednesday, April 19, 2023
2 changes · master
Code cleanup and technical improvements
This update rebuilds how Odoo calculates grouped totals and summaries behind the scenes, making reports and dashboards easier for developers to optimize and maintain. It also aligns these calculations with standard security rules, improving consistency across areas such as accounting, CRM, calendar, and fleet.
Original PR description
## Rationale The method `read_group()` was designed to be used by the web client to efficiently compute aggregations grouped by one or more fields. However, more and more developers have been using…
## Rationale
The method `read_group()` was designed to be used by the web client to efficiently compute aggregations grouped by one or more fields. However, more and more developers have been using it from the backend to make computations more efficient (avoid doing the aggregation in Python). Unfortunately, the API was designed for the web client, which adds a lot of boilerplate when used in the Python (list of dict with misleading key names).
Method `_read_group()` was created to improve the performance of `read_group()` for backend use (4ef0c00b4b37d75b78c5cd30dba004ffd0f8b0ac), but didn't change the API and based the implementation on `read_group()` itself.
This rewrites `_read_group()` from scratch with a new API to make it easier to use from the backend. Also, split the method to make it easy to override and add custom behavior.
## New API of `_read_group()`
The purpose of the method is unchanged: get field aggregations specified by `aggregates` grouped by the given `groupby` fields where records are filtered by the `domain`.
### Parameters
- Rename `fields` into `aggregates`; parameter `aggregates` is a list of _explicit_ aggregate specifications (no implicit aggregator coming from `group_operator`).
- Parameter `groupby` becomes the second parameter (just before `aggregates`). It is much more readable for the returning value. Also the groupby specification for date/datetime fields should be always explicit (no default 'month').
- New parameter `having`, which is a domain-like list to filter the result depending of aggregate values. It allows to simplify some code and avoid to fetch unnecessary data from the database.
- Parameter `orderby` is renamed into `order`, to be consistent with method `search()`. Order terms should be an explicit aggregate specification or groupby specification (+ ASC/DESC).
- Remove parameter `lazy`, which was only used for the webclient and doesn't have any sense to be used in backend.
### Returned value
The returned value is a list of tuples containing, in order, the group values and aggregate values (flatten). For instance, if you group by `['foo', 'bar']` and aggregate `['baz:sum', 'quux:max']`, the result will contain tuples like `(foo_value, bar_value, baz_sum_value, quux_max_value)`.
If a group is relational field, its corresponding value will be a recordset (with a correct prefetch set that includes all returned values). For other fields, the returned values are similar to the ones you get from a record: no labels anymore, date/datetime values are not transformed strings, NULL values are returned as `False`, etc.
## Performance
Minor performance improvements:
- The aggregate `__count` generates `COUNT(*)` instead `COUNT(<table>.id)` (which generates useless null checks from PostgreSQL). Moreover, `COUNT(*)` is only added when we explicitly ask for `__count`.
- Remove the implicit `min(<table>.id)` aggregate added previously to the query at each read_group.
- The recordsets with correct prefetch set are returned for relational group values: it adds a very small cost of instantiation, but avoids manual browsing and bad field prefetching in business code.
- Don't include extra values generated for the webclient: `__domain`, `__range`, `__fold`, `__context`.
- A falsy domain simply doesn't make any query at all, just like `search`. It is very convenient with domains like `[(..., 'in', self.ids)]` where `self.ids` can be empty (it happens a lot in case of onchange methods).
Performance regression:
- The values of aggregation `array_agg` are now ordered by `id`, which is less efficient but it ensures a deterministic result.
### Performance Testing
Performance tests show that the new version/usage is faster in general.
With a populated database (`--size=medium` + some tweaks) and with all these changes,
the performance of the `_read_group`/`read_group` calls, for 2280 calls, is in average +- 9.17 % faster.
- 388 (on 2280) cases are statically slower (mean = -7.66 %, median = -5.17%) than before for several reasons:
- more restricted access due to the commit named "simplify security of read_group", which in some case generates extra queries.
- small overhead to instantiate recordsets;
- `array_agg` is now deterministic;
- noise.
- 862 cases are statically faster (mean = +181.08 %, median = +15.82%) than before.
## Next step
The second part of this task will be focused on the API of public method `read_group()`:
- Make it more consistent with this new `_read_group`.
- Simplify the parsing of data (always return raw values aside the labelled ones).
- Avoid useless aggregations.
Enterprise: https://github.com/odoo/enterprise/pull/38639This update converts the remaining Odoo JavaScript modules to a newer standard format. It supports future simplification of the platform's startup code and helps keep the web experience maintainable without changing day-to-day user workflows.
Original PR description
This commit converts odoo modules that haven't been converted with commit https://github.com/odoo/odoo/pull/117305/commits/e10b45c69e72f09128e49eb46e42834b1ef515d7. The goal is to deprecate odoo.define in favor of native module and then simplify boot.js by removing the regexp that finds module dependencies. task id: 3162300