Tuesday, February 20, 2024
3 changes · 17.0
Enhancements to existing features
This update dramatically speeds up the process of generating referral links in the HR Referral module, reducing generation time from about 1 second to under 10 milliseconds. The improvement comes from removing unnecessary page rendering that was slowing down the link creation process, making the referral feature much more responsive for users.
Original PR description
**Performance Improvement on Referral Link Generation** --------------------------------------------------------------------------------------- Current State ------------------- Currently in the…
**Performance Improvement on Referral Link Generation**
---------------------------------------------------------------------------------------
Current State
-------------------
Currently in the `hr.referral` module when you want to generate a new referral link for a user it takes more or less a second.
State After this commit
---------------------------------
The generation of the link is in average way under the 10ms mark
Tests ordered by performance uplift ascending
-------------------------------------------------------------------
Note: All the tests have been made 3times and I took the less advantageous one for this commit every time.
Note2: As explained after the result are probably underestimated since with the current behavior we litterally make a request and get the full web page of the job position when generating a referral link which is extremely dependent on the load of the server and the db. (We can even have timeout) The time it takes currently is also directly dependant of the size of the job page so it could be theoretically speaking arbitrary long to get the job page.
Note3: All the tests are realized on 15.0
**Full response (+- 18x)**
**On runbot including the time of response before**

**On runbot including the time of response after**

We are in the golden bracket of 50-150ms latency
**Create (+- 100x)**
**On runbot before**
Note: Don't hesitate to click on the photos to zoom in it
cropped

full

**On runbot after**
cropped

full

**Note for master**
In master the `create` method is taking most of the time 3.8ms to execute with no real changes for the `search_or_create` so you'll get an uplift of arround 250x with this commit
Explaination
------------------
Currently when you generate a referral link in the referral app you'll call the the `search_or_create` method of the `link.tracker` model.
``` python
@api.model
def search_or_create(self, vals):
if 'url' not in vals:
raise ValueError(_('Creating a Link Tracker without URL is not possible'))
if vals['url'].startswith(('?', '#')):
raise UserError(_("%r is not a valid link, links cannot redirect to the current page.", vals['url']))
vals['url'] = tools.validate_url(vals['url'])
search_domain = [
(fname, '=', value)
for fname, value in vals.items()
if fname in ['url', 'campaign_id', 'medium_id', 'source_id']
]
result = self.search(search_domain, limit=1)
if result:
return result
return self.create(vals)
```
And if we don't have a hit during the search we will create the `link.tracker` record that is created this way:
```python
@api.model_create_multi
def create(self, vals_list):
vals_list = [vals.copy() for vals in vals_list]
for vals in vals_list:
if 'url' not in vals:
raise ValueError(_('Creating a Link Tracker without URL is not possible'))
if vals['url'].startswith(('?', '#')):
raise UserError(_("%r is not a valid link, links cannot redirect to the current page.", vals['url']))
vals['url'] = tools.validate_url(vals['url'])
if not vals.get('title'):
vals['title'] = self._get_title_from_url(vals['url'])
# Prevent the UTMs to be set by the values of UTM cookies
for (__, fname, __) in self.env['utm.mixin'].tracking_fields():
if fname not in vals:
vals[fname] = False
```
As we can see here if no **title** is given to the `search_or_create` method we will call the `_get_title_from_url` method of `link.tracker`
```python
@api.model
@api.depends('url')
def _get_title_from_url(self, url):
preview = link_preview.get_link_preview_from_url(url)
if preview and preview.get('og_title'):
return preview['og_title']
return url
```
If you time this method it takes around .9 seconds to do it's job. (In local it represent more than 99 percent of the time that take the `create` method)
On the runbot

If we go to `odoo.addons.mail.tools.link_preview` we can clearly see why it takes time
```python
def get_link_preview_from_url(url, request_session=None):
# Some websites are blocking non browser user agent.
user_agent = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0'}
try:
if request_session:
response = request_session.get(url, timeout=3, headers=user_agent, allow_redirects=True, stream=True)
else:
response = requests.get(url, timeout=3, headers=user_agent, allow_redirects=True, stream=True)
except requests.exceptions.RequestException:
return False
if not response.ok or not response.headers.get('Content-Type'):
return False
# Content-Type header can return a charset, but we just need the
# mimetype (eg: image/jpeg;charset=ISO-8859-1)
content_type = response.headers['Content-Type'].split(';')
if response.headers['Content-Type'].startswith('image/'):
return {
'image_mimetype': content_type[0],
'og_image': url, # If the url mimetype is already an image type, set url as preview image
'source_url': url,
}
elif response.headers['Content-Type'].startswith('text/html'):
return get_link_preview_from_html(url, response)
return False
def get_link_preview_from_html(url, response):
content = b""
for chunk in response.iter_content(chunk_size=8192):
content += chunk
pos = content.find(b'</head>', -8196 * 2)
# Stop reading once all the <head> data is found
if pos != -1:
content = content[:pos + 7]
break
if not content:
return False
tree = html.fromstring(content)
og_title = tree.xpath('//meta[@property="og:title"]/@content')
if og_title:
og_title = og_title[0]
elif tree.find('.//title') is not None:
# Fallback on the <title> tag if it exists
og_title = tree.find('.//title').text
else:
return False
og_description = tree.xpath('//meta[@property="og:description"]/@content')
og_type = tree.xpath('//meta[@property="og:type"]/@content')
og_site_name = tree.xpath('//meta[@property="og:site_name"]/@content')
og_image = tree.xpath('//meta[@property="og:image"]/@content')
og_mimetype = tree.xpath('//meta[@property="og:image:type"]/@content')
return {
'og_description': og_description[0] if og_description else None,
'og_image': og_image[0] if og_image else None,
'og_mimetype': og_mimetype[0] if og_mimetype else None,
'og_title': og_title,
'og_type': og_type[0] if og_type else None,
'og_site_name': og_site_name[0] if og_site_name else None,
'source_url': url,
}
```
As we can see it clearly makes a request to the server (in this case itself) that will take in most cases something like a second. (It can be longer and **timeout after 3 seconds and thus not even giving us a title at all and then give you the url as a title**)
[BTW In this case, doing that is probably sub optimal because we could probably just used some internal methods to accelerate the process instead of using `get`]
So we make a request to the server that will need to actually render the view with all the fields and then analyzing it to retrieve infos.
**In conclusion**
By giving a title to the link that we want to generate it will be way faster and since the title is not taken into account during the search part of the `search_or_create`, we basically have no change in behavior and even getting more consistency in the titles that will be displayed on base_url/r.
Other benefits
----------------------
1. Before the links title where generated using the meta of the web page. This can lead to basically random titles because if no one has visited the page before, the title will be the link (same if the request timeout) and if it has been visited it will be the title that you can see in your tabs.
2. You can basically reduce the number of requests to the server which in this case is even better because the more user you have the more likely you will overload it and increase the probability of a request timeout.
3. Before the multi db on read mode all the activity on website are tracked and since you literally visit the web page of the job title you'll get some bias on the website activity. This commit solve this issue as well because we'll not visit the job url anymore.
4. Consistency in the time needed to generated a link.
Performance impact for big companies like odoo
----------------------------------------------------------------------
**Today**
we have 35 different jobs opened for referral
We are 4k employees
if you want to generate all referral links even when taking advantage of batch it will take you more or less 35hours of computations
(in the near future we will include the possibility to generate all the links and send them to all employees and with this it's possible to do it in less than a minute by taking advantage of the batch create)
For the near future
----------------------------
2 tasks are including to generate a bunch of referral links at the same time (with a company of the size of odoo)
- 3418434 (more than a minute just to open the jobs page)
- 3607159 (with current state more than 2hours to send the mails)
task-3707478
Forward-Port-Of: odoo/enterprise#56907
Forward-Port-Of: odoo/enterprise#55349Resolved issues and error corrections
This fixes tax calculations for complex combinations such as multiple included taxes, fixed fees combined with included VAT, and division-style taxes used in several localizations. Businesses should see more reliable invoice totals and tax amounts in affected accounting flows, reducing rounding discrepancies and compliance risks.
Original PR description
REAL TAX CASES TO COVER 1) INDIAN CASE: 6% incl + 6% incl + 3% excl - Both 6% incl must always have the same tax amounts (not working in master but fixed as well in this task). - The 3% must be based…
REAL TAX CASES TO COVER 1) INDIAN CASE: 6% incl + 6% incl + 3% excl - Both 6% incl must always have the same tax amounts (not working in master but fixed as well in this task). - The 3% must be based on 12% (working thanks to the is_base_affected checkbox). 2) BELGIUM CASE: fixed tax + 21% incl (recupel case) That's for this case we allow to mix price-excluded with price-included taxes. 3) BRAZILIAN CASE: 5 taxes having the 'division' type: 5% 3%, 0.65%, 9% and 15%. This case is tricky because it's based on the price-included amount and the whole computation was made only from the price-excluded amount. With a base of 48.0, the base amount of the 15% tax is computed as 48.0 * (1 - 0.15) = 40.8 so a tax amount of 48.0 - 40.8 = 7.2. So the respective <base, tax_amount> of each taxes are: 45.6, 2.4 46.56, 1.44 47.69, 0.31 43.68, 4.32 40.8, 7.2 ...and the price-excluded amount is 40.8. PROBLEMS TO SOLVE 1) INDIAN CASE: Suppose a base of 100 with 2 x 6% incl taxes. The behavior in master: a - Find the price-excluded base: 100 / 1.12 ~= 89.29 b - Compute the first 6% incl tax amount: 89.29 * 0.06 = 5.36 c - Compute the second 6% incl tax amount. Since it's the last one before the "cached base amount of 100", it's computed as 100 - 89.29 - 5.36 = 5.35 => 5.35 != 5.36 The behavior in the current task: a - Compute the base amount for the computation of the 2 x 6% incl taxes: 100 / 1.12 = 89.2857 b - Compute the tax amount for the 2 taxes: 89.2857 * 0.06 = 5.357142 ~= 5.36 c - Compute the base amount of the 2 x 6% incl taxes: 100 - 5.36 - 5.36 = 89.28. => Problem solved 2) BELGIUM CASE: Suppose a base of 120.90 with 0.10 fixed tax (must be include_base_amount), then 21% incl tax. The behavior in master: a - Find the price-excluded base: 120.90 / 1.21 = 99.92 b - Compute the percentage tax: (99.92 + 0.10) * 0.21 = 21.0 => total tax is 21.0 + 0.10 = 21.10 but the base is 99.92 so the total of the invoice will be 121.02. The results is supposed to be the same as 2 lines: line1: 120.90 with 21% incl tax line2: 0.10 with 21% incl tax ...giving a price total of 121, a price subtotal of 100 and a tax amount of 21. The behavior in the current task: a - First ascending computation: Compute first the tax amounts of the fixed taxes: 0.10. b - Descending computation: Compute the base and tax amounts for the 21% tax: (120.90 + 0.10) / 1.21 = 100 then 100 * 0.21 = 21.0. c - Second ascending computation: Compute the base of 0.10 being 120.90. 3) BRAZILIAN CASE: As said before, from 40.8, it's impossible to recompute the correct tax amounts. Suppose a base of 48.0 with 5% 3%, 0.65%, 9% and 15%, all division price included taxes. The behavior in master: a - Find the price-excluded base: 48.0 * (1 - 0.3265) ~= 32.33 b - Wrongly compute the tax amounts price-excluded for 5%, 3%, 0.65%, 9%: tax of 5%: 32.33 / 0.95 - 32.33 = 1.7 tax of 3%: 32.33 / 0.97 - 32.33 = 1.0 tax of 0.65%: 32.33 / 0.9935 - 32.33 = 0.21 tax of 9%: 32.33 / 0.91 - 32.33 = 3.2 c - the tax of 15% takes the remaining amount: 48.0 - 32.33 - 1.7 - 1.0 - 0.21 - 3.2 = 9.56 => Nothing works at all... The behavior in the current task: a - Descending computation: Compute the base and tax amounts for all taxes: 48.0 * 0.95 = 45.6; 48.0 - 45.6 = 2.4 48.0 * 0.97 = 46.56; 48.0 - 46.56 = 1.44 48.0 * 0.65 = 47.69; 48.0 - 47.69 = 0.31 48.0 * 0.91 = 43.68; 48.0 - 43.68 = 4.32 48.0 * 0.85 = 40.8; 48.0 - 40.8 = 7.2 REMAINING PROBLEMS Even the taxes computation will be fixed by this commit, some issues remain: - rounding issues on POS global discount/loyalties - bad computation of combo product with complex taxes - python taxes not working on the POS - perf of the tax details queries - round globally not working due to the accounting grouping key - ... For all those reasons, this commit also adds new cool features: The taxes computation is splitted in 2 parts: a - prepare_taxes_computation that gives a formula to compute each tax independently. b - evaluate the taxes computation given by (a). This will help a lot to change the tax details query later by: a - pre-compile the taxes combinations first python-side. b - compute the tax details in SQL. The taxes computation is now completely reversible if you don't have any rounding in the process and thus, would help to solve the global discount/loyalties/product combo taxes computation. Also, the fiscal position mapping is now more accurate and is able to manage division taxes as well. The method are splitted in a way is will be quite easy to fix the round globally: Instead of: - For each line, create a tax detail per repartition line - Sum the tax details per repartition line and create tax lines => It gives a sum of rounded amounts that could be far from the expected amount: round(base * percentage). Do: - For each line, create a tax detail per tax. - Sum the tax details per tax and round them if round_per_line. - Spread the amounts onto the repartition lines. => It will give exactly the tax amount expected by the user: round(base * percentage). opw: 3443703 --- I confirm I have signed the CLA and read the PR guidelines at www.odoo.com/submit-pr
This fixes tax calculations for invoices and point-of-sale scenarios that combine included, excluded, fixed, and division taxes. Businesses in countries such as India, Belgium, and Brazil should see more consistent totals, fewer rounding discrepancies, and better compliance with local tax expectations.
Original PR description
REAL TAX CASES TO COVER INDIAN CASE: 6% incl + 6% incl + 3% excl Both 6% incl must always have the same tax amounts (not working in master but fixed as well in this task). The 3% must be based on 12%…
REAL TAX CASES TO COVER INDIAN CASE: 6% incl + 6% incl + 3% excl Both 6% incl must always have the same tax amounts (not working in master but fixed as well in this task). The 3% must be based on 12% (working thanks to the is_base_affected checkbox). BELGIUM CASE: fixed tax + 21% incl (recupel case) That's for this case we allow to mix price-excluded with price-included taxes. BRAZILIAN CASE: 5 taxes having the 'division' type: 5% 3%, 0.65%, 9% and 15%. This case is tricky because it's based on the price-included amount and the whole computation was made only from the price-excluded amount. With a base of 48.0, the base amount of the 15% tax is computed as 48.0 * (1 - 0.15) = 40.8 so a tax amount of 48.0 - 40.8 = 7.2. So the respective <base, tax_amount> of each taxes are: 45.6, 2.4 46.56, 1.44 47.69, 0.31 43.68, 4.32 40.8, 7.2 ...and the price-excluded amount is 40.8. PROBLEMS TO SOLVE INDIAN CASE: Suppose a base of 100 with 2 x 6% incl taxes. The behavior in master: a - Find the price-excluded base: 100 / 1.12 ~= 89.29 b - Compute the first 6% incl tax amount: 89.29 * 0.06 = 5.36 c - Compute the second 6% incl tax amount. Since it's the last one before the "cached base amount of 100", it's computed as 100 - 89.29 - 5.36 = 5.35 => 5.35 != 5.36 The behavior in the current task: a - Compute the base amount for the computation of the 2 x 6% incl taxes: 100 / 1.12 = 89.2857 b - Compute the tax amount for the 2 taxes: 89.2857 * 0.06 = 5.357142 ~= 5.36 c - Compute the base amount of the 2 x 6% incl taxes: 100 - 5.36 - 5.36 = 89.28. => Problem solved BELGIUM CASE: Suppose a base of 120.90 with 0.10 fixed tax (must be include_base_amount), then 21% incl tax. The behavior in master: a - Find the price-excluded base: 120.90 / 1.21 = 99.92 b - Compute the percentage tax: (99.92 + 0.10) * 0.21 = 21.0 => total tax is 21.0 + 0.10 = 21.10 but the base is 99.92 so the total of the invoice will be 121.02. The results is supposed to be the same as 2 lines: line1: 120.90 with 21% incl tax line2: 0.10 with 21% incl tax ...giving a price total of 121, a price subtotal of 100 and a tax amount of 21. The behavior in the current task: a - First ascending computation: Compute first the tax amounts of the fixed taxes: 0.10. b - Descending computation: Compute the base and tax amounts for the 21% tax: (120.90 + 0.10) / 1.21 = 100 then 100 * 0.21 = 21.0. c - Second ascending computation: Compute the base of 0.10 being 120.90. BRAZILIAN CASE: As said before, from 40.8, it's impossible to recompute the correct tax amounts. Suppose a base of 48.0 with 5% 3%, 0.65%, 9% and 15%, all division price included taxes. The behavior in master: a - Find the price-excluded base: 48.0 * (1 - 0.3265) ~= 32.33 b - Wrongly compute the tax amounts price-excluded for 5%, 3%, 0.65%, 9%: tax of 5%: 32.33 / 0.95 - 32.33 = 1.7 tax of 3%: 32.33 / 0.97 - 32.33 = 1.0 tax of 0.65%: 32.33 / 0.9935 - 32.33 = 0.21 tax of 9%: 32.33 / 0.91 - 32.33 = 3.2 c - the tax of 15% takes the remaining amount: 48.0 - 32.33 - 1.7 - 1.0 - 0.21 - 3.2 = 9.56 => Nothing works at all... The behavior in the current task: a - Descending computation: Compute the base and tax amounts for all taxes: 48.0 * 0.95 = 45.6; 48.0 - 45.6 = 2.4 48.0 * 0.97 = 46.56; 48.0 - 46.56 = 1.44 48.0 * 0.65 = 47.69; 48.0 - 47.69 = 0.31 48.0 * 0.91 = 43.68; 48.0 - 43.68 = 4.32 48.0 * 0.85 = 40.8; 48.0 - 40.8 = 7.2 REMAINING PROBLEMS Even the taxes computation will be fixed by this commit, some issues remain: -rounding issues on POS global discount/loyalties -bad computation of combo product with complex taxes -python taxes not working on the POS -perf of the tax details queries -round globally not working due to the accounting grouping key -... For all those reasons, this commit also adds new cool features: The taxes computation is splitted in 2 parts: a - prepare_taxes_computation that gives a formula to compute each tax independently. b - evaluate the taxes computation given by (a). This will help a lot to change the tax details query later by: a - pre-compile the taxes combinations first python-side. b - compute the tax details in SQL. The taxes computation is now completely reversible if you don't have any rounding in the process and thus, would help to solve the global discount/loyalties/product combo taxes computation. Also, the fiscal position mapping is now more accurate and is able to manage division taxes as well. The method are splitted in a way is will be quite easy to fix the round globally: Instead of: For each line, create a tax detail per repartition line Sum the tax details per repartition line and create tax lines => It gives a sum of rounded amounts that could be far from the expected amount: round(base * percentage). Do: For each line, create a tax detail per tax. Sum the tax details per tax and round them if round_per_line. Spread the amounts onto the repartition lines. => It will give exactly the tax amount expected by the user: round(base * percentage). opw: 3443703