Daily updates from Odoo
Wednesday, January 7, 2026
1 change · 19.0
Enhancements to existing features
This update ensures Odoo's internal crawler, Odoobot, follows website guidelines outlined in robots.txt. By checking if our crawler is permitted to access specific URLs, we prevent unauthorized data fetching and improve the reliability of our data collection process. This enhances our data integrity and avoids potential issues with website owners.
Original PR description
### Purpose Add `robots.txt` compliance to the `_fetch_url` helper. Before fetching any URL, we now check the site's `robots.txt` and ensure that our user agent (`Odoobot/1.0`) is allowed to access the requested path. ### Changes - Added `robots.txt` validation using `urllib.robotparser` - Respect disallow rules for the `Odoobot/1.0` user agent ### Why This ensures that our crawler behaves politely, avoids fetching disallowed resources, and provides more transparent failure reporting. --- **Task-ID:** 4779962