We could add caching of corrections so that they are not parsed a second time. Users will often re-read the same correction multiple time or the same correction when processing different chunks on the same worker without the python interpreter exiting. We can save time and avoid reparsing the jsons by simply adding a cache on all the loading functions. It can be done at the python level, like a simple lru cache.
We could add caching of corrections so that they are not parsed a second time. Users will often re-read the same correction multiple time or the same correction when processing different chunks on the same worker without the python interpreter exiting. We can save time and avoid reparsing the jsons by simply adding a cache on all the loading functions. It can be done at the python level, like a simple lru cache.