Cloudpickle is set as the default pickle_library in 2.65.0, where the previous default was dill. See https://s.apache.org/beam-cloudpickle-next-steps for background.
This can cause breakages in cases where the behavior of dill and cloudpickle diverge.
cloudpickle_pickler_test tests demonstrates the behavior of cloudpickle in various cases. Notable behavior includes:
- Globals defined in
__main__ module are pickled by value
- Globals defined in importable modules are pickled by reference
- Module aliased globals are pickled by value
- All functions and classes defined in
__main__ module are pickled by value
- All closures and dynamic types are pickled by value.
Known issues include:
- Unittests that rely on globals will fail. Cloudpickle assumes the
__main__ module is not available in the unpickling environment and therefore redefines globals. To fix tests that rely on globals use the apache_beam.utils.shared module as shown in
|
def test_globals_shared_are_pickled_by_reference(self): |
- Closures and dynamic classes that reference unpicklable objects fail. This can be fixed by defining functions at the top level and binding arguments with
functools.partial when necessary
- When encountering types not picklable by cloudpickle, rather define these types in an importable module in which case they will be pickled by reference.
Please report any new issues on this tracking bug. For any breakages that require reverting back to dill specify the --pickle_library=dill pipeline option.
Issue Priority
Priority: 2 (default / most normal work should be filed as P2)
Issue Components
Cloudpickle is set as the default
pickle_libraryin 2.65.0, where the previous default was dill. See https://s.apache.org/beam-cloudpickle-next-steps for background.This can cause breakages in cases where the behavior of dill and cloudpickle diverge.
cloudpickle_pickler_test tests demonstrates the behavior of cloudpickle in various cases. Notable behavior includes:
__main__module are pickled by value__main__module are pickled by valueKnown issues include:
__main__module is not available in the unpickling environment and therefore redefines globals. To fix tests that rely on globals use the apache_beam.utils.shared module as shown inbeam/sdks/python/apache_beam/internal/cloudpickle_pickler_test.py
Line 54 in b9fa49a
functools.partialwhen necessaryPlease report any new issues on this tracking bug. For any breakages that require reverting back to dill specify the
--pickle_library=dillpipeline option.Issue Priority
Priority: 2 (default / most normal work should be filed as P2)
Issue Components