Support for lazy imports

[mauvilsa]

A topic closely related to optional imports are lazy imports. When a package is optional, it commonly happens that even though it is installed, it is not used for some process execution. In these cases, always importing all packages is a waste of resources. Implementing lazy imports can provide "startup time improvements up to 70% and memory-use reductions up to 40% on real-world Python CLIs".

Eventually lazy imports could become a native feature of python, but for the moment there are only long discussions and a rejected PEP 690. Thus, a package that provides it would certainly be welcome.

From my understanding adding lazy imports to generalimport would be relatively easy given what it already does. So I propose to add this as a feature.

Possible implementations

1. Add a new function lazy_imports() to explicitly make imports lazy.
2. Modify the existing generalimport() so that imports are always lazy.

Independent from this, from generalimport import generalimport does not look very nice or readable. Might be nicer if it were from generalimport import optional_import.

[mauvilsa]

@Mandera any comment regarding this?

[Mandera]

Lazy imports would be very interesting if it could be implemented cleanly!

I'm a little offended by your distaste of from generalimport import generalimport 😅 But I'm afraid you might have a point.
My reasoning was that for "flagship" methods it makes sense to use the same name, i.e. from pprint import pprint. It's easier to remember

[mauvilsa]

Good to hear that this would be a welcome feature!

What would be your choice for a user facing design? I mentioned two. One could be a totally new function, e.g. lazy_imports, which would mean that people need to use this new function or change their code if they would benefit. The other option would be that the existing generalimport would become lazy. My main question regarding both options are if there are use cases in which someone would want an optional dependency but not be lazy. In my case all the packages that I have developed that have optional dependencies makes sense for them to be lazy.

If lazy is added to generalimport it could also be with a transition period. When first released could be off by default and have a way to enable it. If it makes sense, in some other release make it enabled by default.

Regarding the actual implementation, I haven't really looked at how generalimport works. From what I understood, which a package is marked as optional via generalimport, an error of not being installed is delayed up to the moment that the package is attempted to be used. So I guess, currently if the package is installed, then the import statement imports it immediately. The required changes would be mainly two:

• When an optional package is installed, don't import it immediately. I assume there is already logic that intercepts import statements, so it is a modification of existing logic.
• When the optional package is first used, do the actual import at that moment. Again a modification of existing logic since currently that is where a not installed error is raised.

Is my understanding correct?

[Mandera]

I like this one:

If lazy is added to generalimport it could also be with a transition period. When first released could be off by default and have a way to enable it. If it makes sense, in some other release make it enabled by default.

I'm afraid I have my doubts about a clean implementation for lazy imports though.
The current way the "nice error messages" are produced is by trying to catch all dunder methods.
I'd argue it's impossible to have 100% coverage for all use-cases, but this is currently fine because the consequence of a use-case slipping by is simply a generic error message.

I'm afraid we'd want 100% coverage if functional code were to go through this system

Is my understanding correct?

I think so yeah!

[Mandera]

Right, there's also one remaining issue for a use-case we can't handle:

• Unhandled - X.foo = "bar" #8