PEP 810: Update decisions on with blocks and __dict__ reification (#4656)

Author: pablogsal

peps/pep-0810.rst

@@ -246,7 +246,7 @@ Syntax restrictions
 ~~~~~~~~~~~~~~~~~~~
 
 The soft keyword is only allowed at the global (module) level, **not** inside
-functions, class bodies, with ``try``/``with`` blocks, or ``import *``. Import
+functions, class bodies, ``try`` blocks, or ``import *``. Import
 statements that use the soft keyword are *potentially lazy*. Imports that
 can't be lazy are unaffected by the global lazy imports flag, and instead are
 always eager. Additionally, ``from __future__ import`` statements cannot be
@@ -270,10 +270,6 @@ Examples of syntax errors:
   except ImportError:
       pass
 
-  # SyntaxError: lazy import not allowed inside with blocks
-  with suppress(ImportError):
-      lazy import json
-
   # SyntaxError: lazy from ... import * is not allowed
   lazy from json import *
 
@@ -465,54 +461,37 @@ immediately resolve all lazy objects (e.g. ``lazy from`` statements) that
 referenced the module. It **only** resolves the lazy object being accessed.
 
 Accessing a lazy object (from a global variable or a module attribute) reifies
-the object. Accessing a module's ``__dict__`` reifies **all** lazy objects in
-that module. Calling ``dir()`` at the global scope will not reify the globals
-and calling ``dir(mod)`` will be special cased in ``mod.__dir__`` avoid
-reification as well.
-
-Example using ``__dict__`` from external code:
-
-.. code-block:: python
-
-  # my_module.py
-  import sys
-  lazy import json
+the object.
 
-  print('json' in sys.modules)  # False - still lazy
-
-  # main.py
-  import sys
-  import my_module
-
-  # Accessing __dict__ from external code DOES reify all lazy imports
-  d = my_module.__dict__
-
-  print('json' in sys.modules)  # True - reified by __dict__ access
-  print(type(d['json']))  # <class 'module'>
+However, calling ``globals()`` or accessing a module's ``__dict__`` does
+**not** trigger reification -- they return the module's dictionary, and
+accessing lazy objects through that dictionary still returns lazy proxy
+objects that need to be manually reified upon use. A lazy object can be
+resolved explicitly by calling the ``resolve`` method. Calling ``dir()`` at
+the global scope will not reify the globals, nor will calling ``dir(mod)``
+(through special-casing in ``mod.__dir__``.) Other, more indirect ways of
+accessing arbitrary globals (e.g. inspecting ``frame.f_globals``) also do
+**not** reify all the objects.
 
-However, calling ``globals()`` does **not** trigger reification -- it returns
-the module's dictionary, and accessing lazy objects through that dictionary
-still returns lazy proxy objects that need to be manually reified upon use. A
-lazy object can be resolved explicitly by calling the ``resolve`` method.
-Other, more indirect ways of accessing arbitrary globals (e.g. inspecting
-``frame.f_globals``) also do **not** reify all the objects.
-
-Example using ``globals()``:
+Example using ``globals()`` and ``__dict__``:
 
 .. code-block:: python
 
+  # my_module.py
   import sys
   lazy import json
 
   # Calling globals() does NOT trigger reification
   g = globals()
-
   print('json' in sys.modules)  # False - still lazy
   print(type(g['json']))  # <class 'LazyImport'>
 
+  # Accessing __dict__ also does NOT trigger reification
+  d = __dict__
+  print(type(d['json']))  # <class 'LazyImport'>
+
   # Explicitly reify using the resolve() method
   resolved = g['json'].resolve()
-
   print(type(resolved))  # <class 'module'>
   print('json' in sys.modules)  # True - now loaded
 
@@ -707,15 +686,15 @@ Where ``<mode>`` can be:
 
 * ``"normal"`` (or unset): Only explicitly marked lazy imports are lazy
 
-* ``"all"``: All module-level imports (except in ``try``  or ``with``
+* ``"all"``: All module-level imports (except in ``try``
   blocks and ``import *``) become *potentially lazy*
 
 * ``"none"``: No imports are lazy, even those explicitly marked with
   ``lazy`` keyword
 
 When the global flag is set to ``"all"``, all imports at the global level
-of all modules are *potentially lazy* **except** for those inside a ``try`` or
-``with`` block or any wild card (``from ... import *``) import.
+of all modules are *potentially lazy* **except** for those inside a ``try``
+block or any wild card (``from ... import *``) import.
 
 If the global lazy imports flag is set to ``"none"``, no *potentially
 lazy* import is ever imported lazily, the import filter is never called, and
@@ -1251,36 +1230,9 @@ either the lazy proxy or the final resolved object.
 Can I force reification of a lazy import without using it?
 ----------------------------------------------------------
 
-Yes, accessing a module's ``__dict__`` will reify all lazy objects in that
-module. Individual lazy objects can be resolved by calling their ``resolve()``
+Yes, individual lazy objects can be resolved by calling their ``resolve()``
 method.
 
-What's the difference between ``globals()`` and ``mod.__dict__`` for lazy imports?
-----------------------------------------------------------------------------------
-
-Calling ``globals()`` returns the module's dictionary without reifying lazy
-imports -- you'll see lazy proxy objects when accessing them through the
-returned dictionary. However, accessing ``mod.__dict__`` from external code
-reifies all lazy imports in that module first. This design ensures:
-
-.. code-block:: python
-
-  # In your module:
-  lazy import json
-
-  g = globals()
-  print(type(g['json']))  # <class 'LazyImport'> - your problem
-
-  # From external code:
-  import sys
-  mod = sys.modules['your_module']
-  d = mod.__dict__
-  print(type(d['json']))  # <class 'module'> - reified for external access
-
-This distinction means adding lazy imports and calling ``globals()`` is your
-responsibility to manage, while external code accessing ``mod.__dict__``
-always sees fully loaded modules.
-
 Why not use ``importlib.util.LazyLoader`` instead?
 --------------------------------------------------
 
@@ -1664,6 +1616,52 @@ From the discussion on :pep:`690` it is clear that this is a fairly
 contentious idea, although perhaps once we have wide-spread use of lazy
 imports this can be reconsidered.
 
+Disallowing lazy imports inside ``with`` blocks
+------------------------------------------------
+
+An earlier version of this PEP proposed disallowing ``lazy import`` statements
+inside ``with`` blocks, similar to the restriction on ``try`` blocks. The
+concern was that certain context managers (like ``contextlib.suppress(ImportError)``)
+could suppress import errors in confusing ways when combined with lazy imports.
+
+However, this restriction was rejected because ``with`` statements have much
+broader semantics than ``try/except`` blocks. While ``try/except`` is explicitly
+about catching exceptions, ``with`` blocks are commonly used for resource
+management, temporary state changes, or scoping -- contexts where lazy imports
+work perfectly fine. The ``lazy import`` syntax is explicit enough that
+developers who write it inside a ``with`` block are making an intentional choice,
+aligning with Python's "consenting adults" philosophy. For genuinely problematic
+cases like ``with suppress(ImportError): lazy import foo``, static analysis
+tools and linters are better suited to catch these patterns than hard language
+restrictions.
+
+Forcing eager imports in ``with`` blocks under the global flag
+---------------------------------------------------------------
+
+Another rejected idea was to make imports inside ``with`` blocks remain eager
+even when the global lazy imports flag is set to ``"all"``. The rationale was
+to be conservative: since ``with`` statements can affect how imports behave
+(e.g., by modifying ``sys.path`` or suppressing exceptions), forcing imports to
+remain eager could prevent subtle bugs. However, this would create inconsistent
+behavior where ``lazy import`` is allowed explicitly in ``with`` blocks, but
+normal imports remain eager when the global flag is enabled. This inconsistency
+between explicit and implicit laziness is confusing and hard to explain.
+
+The simpler, more consistent rule is that the global flag affects imports
+everywhere that explicit ``lazy import`` syntax is allowed. This avoids having
+three different sets of rules (explicit syntax, global flag behavior, and filter
+mechanism) and instead provides two: explicit syntax rules match what the global
+flag affects, and the filter mechanism provides escape hatches for edge cases.
+For users who need fine-grained control, the filter mechanism
+(``sys.set_lazy_imports_filter()``) already provides a way to exclude specific
+imports or patterns. Additionally, there's no inverse operation: if the global
+flag forces imports eager in ``with`` blocks but a user wants them lazy, there's
+no way to override it, creating an asymmetry.
+
+In summary: imports in ``with`` blocks behave consistently whether marked
+explicitly with ``lazy import`` or implicitly via the global flag, creating a
+simple rule that's easy to explain and reason about.
+
 Modification of the dict object
 -------------------------------
 
@@ -1868,55 +1866,38 @@ from a real dict in almost all cases, which is extremely difficult to achieve
 correctly. Any deviation from true dict behavior would be a source of subtle
 bugs.
 
-Reifying lazy imports when ``globals()`` is called
----------------------------------------------------
+Automatically reifying on ``__dict__`` or ``globals()`` access
+--------------------------------------------------------------
 
-Calling ``globals()`` returns the module's namespace dictionary without
-triggering reification of lazy imports. Accessing lazy objects through the
-returned dictionary yields the lazy proxy objects themselves. This is an
-intentional design decision for several reasons:
-
-**The key distinction**: Adding a lazy import and calling ``globals()`` is the
-module author's concern and under their control. However, accessing
-``mod.__dict__`` from external code is a different scenario -- it crosses
-module boundaries and affects someone else's code. Therefore, ``mod.__dict__``
-access reifies all lazy imports to ensure external code sees fully realized
-modules, while ``globals()`` preserves lazy objects for the module's own
-introspection needs.
-
-**Technical challenges**: It is impossible to safely reify on-demand when
-``globals()`` is called because we cannot return a proxy dictionary -- this
-would break common usages like passing the result to ``exec()`` or other
-built-ins that expect a real dictionary. The only alternative would be to
-eagerly reify all lazy imports whenever ``globals()`` is called, but this
-behavior would be surprising and potentially expensive.
-
-**Performance concerns**: It is impractical to cache whether a reification
-scan has been performed with just the globals dictionary reference, whereas
-module attribute access (the primary use case) can efficiently cache
-reification state in the module object itself.
-
-**Use case rationale**: The chosen design makes sense precisely because of
-this distinction: adding a lazy import and calling ``globals()`` is your
-problem to manage, while having lazy imports visible in ``mod.__dict__``
-becomes someone else's problem. By reifying on ``__dict__`` access but not on
-``globals()``, we ensure external code always sees fully loaded modules while
-giving module authors control over their own introspection.
-
-Note that three options were considered:
+Three options were considered for how ``globals()`` and ``mod.__dict__`` should
+behave with lazy imports:
 
 1. Calling ``globals()`` or ``mod.__dict__`` traverses and resolves all lazy
    objects before returning.
 2. Calling ``globals()`` or ``mod.__dict__`` returns the dictionary with lazy
-   objects present.
+   objects present (chosen).
 3. Calling ``globals()`` returns the dictionary with lazy objects, but
    ``mod.__dict__`` reifies everything.
 
-We chose the third option because it properly delineates responsibility: if
-you add lazy imports to your module and call ``globals()``, you're responsible
-for handling the lazy objects. But external code accessing your module's
-``__dict__`` shouldn't need to know about your lazy imports -- it gets fully
-resolved modules.
+We chose option 2: both ``globals()`` and ``__dict__`` return the raw
+namespace dictionary without triggering reification. This provides a clean,
+predictable model where low-level introspection APIs don't trigger side
+effects.
+
+Having ``globals()`` and ``__dict__`` behave identically creates symmetry and
+a simple mental model: both expose the raw namespace view. Low-level
+introspection APIs should not automatically trigger imports, which would be
+surprising and potentially expensive. Real-world experience implementing lazy
+imports in the standard library (such as the traceback module) showed that
+automatic reification on ``__dict__`` access was cumbersome and forced
+introspection code to load modules it was only examining.
+
+Option 1 (always reifying) was rejected because it would make ``globals()``
+and ``__dict__`` access surprisingly expensive and prevent introspecting the
+lazy state of a module. Option 3 was initially considered to "protect" external
+code from seeing lazy objects, but real-world usage showed this created more
+problems than it solved, particularly for stdlib code that needs to introspect
+modules without triggering side effects.
 
 Acknowledgements
 ================