Wednesday, October 23, 2024
1 change · master
Enhancements to existing features
Large spreadsheets now load much faster by avoiding unnecessary server-side processing of already prepared data. This reduces waiting time for users opening spreadsheet documents and lowers server resource usage, especially for very large files.
Original PR description
Server-side load time for a 16.7Mb spreadsheet: **before: 606ms** (`join_spreadsheet_session`) **after: 83ms** (-86%!) (new `/spreadsheet/data/<...>` controller) Previously,…
Server-side load time for a 16.7Mb spreadsheet: **before: 606ms** (`join_spreadsheet_session`) **after: 83ms** (-86%!) (new `/spreadsheet/data/<...>` controller) Previously, `join_spreadsheet_session` returned a combination of: - The full spreadsheet snapshot (JSON file on the filestore) - Pending revision data (also serialized JSON, stored in a Char field) - Additional metadata These were combined into a Python dict and re-serialized into JSON for the response. This process is inefficient as it involved unnecessary parsing and re-serialization of already serialized JSON data. With this commit, we handcraft the json response body, avoiding the need for parsing and re-serializing large json objects, resulting in significant performance improvements in both CPU and memory. To have control of the response body, we now have to use an http controller instead of an RPC method. We could also have sent the raw/serialized data from `join_spreadsheet_session` but then it would be serialized json over json which has several drawbacks: - reduces client-side debuggability - introduce a lot of additional escape characters (e.g. `\"`) - the client would need to explicitly parse the nested json With this solution, the client receives a clean, valid json content type Alternative idea: we could stream the snapshot file directly from the file store to be even more efficient. But then we would need to load the additional metadata and revisions with another http request. As those two requests would not be executed as part of the same transaction, it would come with its own share of issues.