Tuesday, February 20, 2024
1 change
Enhancements to existing features
This update dramatically speeds up the process of generating referral links in the HR Referral module, reducing generation time from about 1 second to under 10 milliseconds. The improvement comes from removing unnecessary page rendering that was slowing down the link creation process, making the referral feature much more responsive for users.
Original PR description
**Performance Improvement on Referral Link Generation** --------------------------------------------------------------------------------------- Current State ------------------- Currently in the…
**Performance Improvement on Referral Link Generation**
---------------------------------------------------------------------------------------
Current State
-------------------
Currently in the `hr.referral` module when you want to generate a new referral link for a user it takes more or less a second.
State After this commit
---------------------------------
The generation of the link is in average way under the 10ms mark
Tests ordered by performance uplift ascending
-------------------------------------------------------------------
Note: All the tests have been made 3times and I took the less advantageous one for this commit every time.
Note2: As explained after the result are probably underestimated since with the current behavior we litterally make a request and get the full web page of the job position when generating a referral link which is extremely dependent on the load of the server and the db. (We can even have timeout) The time it takes currently is also directly dependant of the size of the job page so it could be theoretically speaking arbitrary long to get the job page.
Note3: All the tests are realized on 15.0
**Full response (+- 18x)**
**On runbot including the time of response before**
**On runbot including the time of response after**
We are in the golden bracket of 50-150ms latency
**Create (+- 100x)**
**On runbot before**
Note: Don't hesitate to click on the photos to zoom in it
cropped
full
**On runbot after**
cropped
full
**Note for master**
In master the `create` method is taking most of the time 3.8ms to execute with no real changes for the `search_or_create` so you'll get an uplift of arround 250x with this commit
Explaination
------------------
Currently when you generate a referral link in the referral app you'll call the the `search_or_create` method of the `link.tracker` model.
``` python
@api.model
def search_or_create(self, vals):
if 'url' not in vals:
raise ValueError(_('Creating a Link Tracker without URL is not possible'))
if vals['url'].startswith(('?', '#')):
raise UserError(_("%r is not a valid link, links cannot redirect to the current page.", vals['url']))
vals['url'] = tools.validate_url(vals['url'])
search_domain = [
(fname, '=', value)
for fname, value in vals.items()
if fname in ['url', 'campaign_id', 'medium_id', 'source_id']
]
result = self.search(search_domain, limit=1)
if result:
return result
return self.create(vals)
```
And if we don't have a hit during the search we will create the `link.tracker` record that is created this way:
```python
@api.model_create_multi
def create(self, vals_list):
vals_list = [vals.copy() for vals in vals_list]
for vals in vals_list:
if 'url' not in vals:
raise ValueError(_('Creating a Link Tracker without URL is not possible'))
if vals['url'].startswith(('?', '#')):
raise UserError(_("%r is not a valid link, links cannot redirect to the current page.", vals['url']))
vals['url'] = tools.validate_url(vals['url'])
if not vals.get('title'):
vals['title'] = self._get_title_from_url(vals['url'])
# Prevent the UTMs to be set by the values of UTM cookies
for (__, fname, __) in self.env['utm.mixin'].tracking_fields():
if fname not in vals:
vals[fname] = False
```
As we can see here if no **title** is given to the `search_or_create` method we will call the `_get_title_from_url` method of `link.tracker`
```python
@api.model
@api.depends('url')
def _get_title_from_url(self, url):
preview = link_preview.get_link_preview_from_url(url)
if preview and preview.get('og_title'):
return preview['og_title']
return url
```
If you time this method it takes around .9 seconds to do it's job. (In local it represent more than 99 percent of the time that take the `create` method)
On the runbot
If we go to `odoo.addons.mail.tools.link_preview` we can clearly see why it takes time
```python
def get_link_preview_from_url(url, request_session=None):
# Some websites are blocking non browser user agent.
user_agent = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0'}
try:
if request_session:
response = request_session.get(url, timeout=3, headers=user_agent, allow_redirects=True, stream=True)
else:
response = requests.get(url, timeout=3, headers=user_agent, allow_redirects=True, stream=True)
except requests.exceptions.RequestException:
return False
if not response.ok or not response.headers.get('Content-Type'):
return False
# Content-Type header can return a charset, but we just need the
# mimetype (eg: image/jpeg;charset=ISO-8859-1)
content_type = response.headers['Content-Type'].split(';')
if response.headers['Content-Type'].startswith('image/'):
return {
'image_mimetype': content_type[0],
'og_image': url, # If the url mimetype is already an image type, set url as preview image
'source_url': url,
}
elif response.headers['Content-Type'].startswith('text/html'):
return get_link_preview_from_html(url, response)
return False
def get_link_preview_from_html(url, response):
content = b""
for chunk in response.iter_content(chunk_size=8192):
content += chunk
pos = content.find(b'</head>', -8196 * 2)
# Stop reading once all the <head> data is found
if pos != -1:
content = content[:pos + 7]
break
if not content:
return False
tree = html.fromstring(content)
og_title = tree.xpath('//meta[@property="og:title"]/@content')
if og_title:
og_title = og_title[0]
elif tree.find('.//title') is not None:
# Fallback on the <title> tag if it exists
og_title = tree.find('.//title').text
else:
return False
og_description = tree.xpath('//meta[@property="og:description"]/@content')
og_type = tree.xpath('//meta[@property="og:type"]/@content')
og_site_name = tree.xpath('//meta[@property="og:site_name"]/@content')
og_image = tree.xpath('//meta[@property="og:image"]/@content')
og_mimetype = tree.xpath('//meta[@property="og:image:type"]/@content')
return {
'og_description': og_description[0] if og_description else None,
'og_image': og_image[0] if og_image else None,
'og_mimetype': og_mimetype[0] if og_mimetype else None,
'og_title': og_title,
'og_type': og_type[0] if og_type else None,
'og_site_name': og_site_name[0] if og_site_name else None,
'source_url': url,
}
```
As we can see it clearly makes a request to the server (in this case itself) that will take in most cases something like a second. (It can be longer and **timeout after 3 seconds and thus not even giving us a title at all and then give you the url as a title**)
[BTW In this case, doing that is probably sub optimal because we could probably just used some internal methods to accelerate the process instead of using `get`]
So we make a request to the server that will need to actually render the view with all the fields and then analyzing it to retrieve infos.
**In conclusion**
By giving a title to the link that we want to generate it will be way faster and since the title is not taken into account during the search part of the `search_or_create`, we basically have no change in behavior and even getting more consistency in the titles that will be displayed on base_url/r.
Other benefits
----------------------
1. Before the links title where generated using the meta of the web page. This can lead to basically random titles because if no one has visited the page before, the title will be the link (same if the request timeout) and if it has been visited it will be the title that you can see in your tabs.
2. You can basically reduce the number of requests to the server which in this case is even better because the more user you have the more likely you will overload it and increase the probability of a request timeout.
3. Before the multi db on read mode all the activity on website are tracked and since you literally visit the web page of the job title you'll get some bias on the website activity. This commit solve this issue as well because we'll not visit the job url anymore.
4. Consistency in the time needed to generated a link.
Performance impact for big companies like odoo
----------------------------------------------------------------------
**Today**
we have 35 different jobs opened for referral
We are 4k employees
if you want to generate all referral links even when taking advantage of batch it will take you more or less 35hours of computations
(in the near future we will include the possibility to generate all the links and send them to all employees and with this it's possible to do it in less than a minute by taking advantage of the batch create)
For the near future
----------------------------
2 tasks are including to generate a bunch of referral links at the same time (with a company of the size of odoo)
- 3418434 (more than a minute just to open the jobs page)
- 3607159 (with current state more than 2hours to send the mails)
task-3707478
Forward-Port-Of: odoo/enterprise#56907
Forward-Port-Of: odoo/enterprise#55349