Summary
A change introduced in SDK 25.14.0, specifically in the python/looker_sdk/rtl/serialize.py file, has introduced a major performance bottleneck. The SDK now re-registers a cattrs hook factory on every single API response deserialization, leading to slowness in multithreaded applications. We would also like to know why was this change introduced as there is no proper documentation also behind this change.
The Problem
In the versions >25.14.0, the deserialize.py includes this line:
converter.register_structure_hook_factory(is_sequence, list_structure_factory)
Due to this the multithreading in Python does not work as fast as it is supposed to.
Impact
When running high-concurrency scripts (e.g., managing 5,000+ user attributes across multiple threads), the time taken is nearly an hour. Without this hook registration, in higher versions, the time take is less than 10 minutes.
Steps to Reproduce
- Use Python SDK 25.16.0 or 26.2.0.
- Execute a high volume (1,000+) of metadata calls (e.g., user_attribute_user_values) using a ThreadPoolExecutor.
- Observe high CPU usage and note the speed.
- Remove the
converter.register_structure_hook_factory(is_sequence, list_structure_factory) line from deserialize.py and then observe the CPU and the speed taken
Proposed Fix
- Maybe instead of inside in
deserialize(), it could be placed more globally on the module level so individual API calls are not affected/
Summary
A change introduced in SDK 25.14.0, specifically in the python/looker_sdk/rtl/serialize.py file, has introduced a major performance bottleneck. The SDK now re-registers a cattrs hook factory on every single API response deserialization, leading to slowness in multithreaded applications. We would also like to know why was this change introduced as there is no proper documentation also behind this change.
The Problem
In the versions >25.14.0, the deserialize.py includes this line:
converter.register_structure_hook_factory(is_sequence, list_structure_factory)Due to this the multithreading in Python does not work as fast as it is supposed to.
Impact
When running high-concurrency scripts (e.g., managing 5,000+ user attributes across multiple threads), the time taken is nearly an hour. Without this hook registration, in higher versions, the time take is less than 10 minutes.
Steps to Reproduce
converter.register_structure_hook_factory(is_sequence, list_structure_factory)line fromdeserialize.pyand then observe the CPU and the speed takenProposed Fix
deserialize(), it could be placed more globally on the module level so individual API calls are not affected/