[FEATURE REQUEST] Python Stubs for Type Checking and Completion

[KelSolaar]

With the newly released Pypi wheels, thanks you so much for this btw, it would be awesome to see Python stubs for type checking and completion. I need to pepper the code with # pyright: ignore pragma and thus, the code is not checked:

from OpenImageIO import DOUBLE  # pyright: ignore
from OpenImageIO import FLOAT  # pyright: ignore
from OpenImageIO import HALF  # pyright: ignore
from OpenImageIO import UINT8  # pyright: ignore
from OpenImageIO import UINT16  # pyright: ignore
from OpenImageIO import ImageInput  # pyright: ignore
from OpenImageIO import ImageOutput  # pyright: ignore
from OpenImageIO import ImageSpec  # pyright: ignore

Not an issue but this would be a great improvement.

@zachlewis for 👀

[JGoldstone]

Should the required Python version for OpenImageIO's bindings be listed somewhere in the online docs? I don't see it.

Contrairwise (or is it conversely?) what is the most recent version of Python that could be used in the OIIO bindings that Colour would support? I was using the 3.12 type syntax in camdkit but had to back it down to 3.11 when it came to submitting the PR.

[lgritz]

Should the required Python version for OpenImageIO's bindings be listed somewhere in the online docs? I don't see it.

https://github.com/AcademySoftwareFoundation/OpenImageIO/blob/main/INSTALL.md?plain=1#L38

[lgritz]

I'm not enough of a Python master to know what we actually need to do on our side to provide the stubs. But if somebody knows, please do make a PR.

@chadrik I seem to remember you corresponding with me about stubs a few months back? Is this something you already have?

[chadrik]

Hi all, here's the PR that adds OIIO stubs to my cg-stubs project: LumaPictures/cg-stubs#19

Stub file is here: https://github.com/LumaPictures/cg-stubs/pull/19/files#diff-b7f5854d5e69b5d0c516cd491877919e614d1db9e23ec918b3e083b0018d045c

Larry, I copied some info from our previous email conversation into the PR above as a refresher for why I had to maintain a set of manual corrections. I think we both agree it would be good to reduce these as much as possible.

The majority of the issues that I had to correct are due to functions with might return None / py:none but which pybind11 reports as returning object. For example:

       .def(
            "imagespec",
            [](TextureSystemWrap& ts, const std::string& filename,
               int subimage) -> py::object {
                py::gil_scoped_release gil;
                const ImageSpec* spec
                    = ts.m_texsys->imagespec(ustring(filename), subimage);
                if (!spec) {
                    return py::none();
                }
                return py::object(py::cast(*spec));
            },
            "filename"_a, "subimage"_a = 0)

The specified return type of py:object obscures the python result type, which is ImageSpec | None. I'm not a C++ expert so forgive me if this is a dumb idea, but would it be possible to use std::optional<ImageSpec> as the result type here? pybind11 claims to support this.

Here's how you would confirm that this is producing the right results via pybind11:

>>> import OpenImageIO
>>> print(OpenImageIO.TextureSystem.imagespec.__doc__)
imagespec(self: OpenImageIO.OpenImageIO.TextureSystem, filename: str, subimage: int = 0) -> object

Instead of -> object you should see Optional[ImageSpec], Union[ImageSpec, None] or ImageSpec | None.

---

The primary question here is this: Larry, are you ok with me publishing these stubs to pypi, as I've done with PySide, OpenUSD, OCIO, etc. All support questions for the stubs would be redirected to https://github.com/LumaPictures/cg-stubs.

If so, I can get these released today. If not, I can work on a PR for this repo to add the stub generation to the build process, but I have no idea how long this will take me to complete. I'm happy to do either one.

[lgritz]

If it works and pybind11 just does the right thing, I don't see why we couldn't change the python implementation so it looks like that lambda returns a std::optional<ImageSpec>, definitely.

I'm fine with you publishing the stubs, but is it something we should instead fold into OIIO itself so it's all maintained in one place and they can't get out of sync?

[lgritz]

I should add (mostly for myself): It was only OIIO 3.0 (released officially this past fall) that raised our minimum to C++17, thus making std::optional available. Before that, we would have been unable to take advantage of any pybind11 direct support of optional, and so we probably do a number of awkward things as a result that are no longer necessary, but we haven't yet returned to the python bindings to simplify based on C++17 being reliably available to us.

[lgritz]

I'm all for finding more uses for std::optional, it's only because of the relatively recent need to keep C++14 compatibility that you don't see it in the OIIO code base.

[chadrik]

I'm fine with you publishing the stubs, but is it something we should instead fold into OIIO itself so it's all maintained in one place and they can't get out of sync?

Yes, I think that's a worthy goal. I'll see how far I can get on this next week.

Here are some early thoughts:

• I have an internal python library in my cg-stubs project with utilities that are shared across many of the stub-gen projects within that repo. I'll need to publish that library so that it can be used here, and I'll need to actually adhere to a semantic version release methodology so that I don't accidentally break things. This will be a bit of a nuisance for me, as currently I can move fast since it's all internal.
  • Longer term, we can aim for this project to get to a state where the stub generation process doesn't need any corrections, which would eliminate the need for using my stubgen utility library. This would be achieved by A) pushing some of my stubgen corrections upstream to mypy, and B) modifying code here to provide more accurate info to pybind11 (such as using std:optional and providing inner types for tuples). That said, I think it could take awhile for all the fixes to make their way upstream, and it's likely that some problems will simply continue to require a very specific rule to correct, so I wouldn't hold my breath. If we decide to move the stubgen process here, it's likely to require my stubgen utility library for quite some time.
• I think that the build process here should produce two separate PyPI distributions: the current OpenImageIO distribution, and a new types-OpenImageIO distribution, with just the stubs. I think it'll be beneficial for users to be able to install the stubs without the library, and thereby keep their environment super light when all they need is type analysis.

If this ends up being more than both of us can pull off in our limited time, I can always publish my current stubs as types-OpenImageIO now, with little effort, and this project can take over ownership of that distribution at some future date when it's been integrated here. Because we'll hand off ownership of the same types-OpenImageIO distribution, it can be a smooth transition of ownership, transparent for early adopters, without any dead projects out in the wild causing confusion.

[zachlewis]

I think that the build process here should produce two separate PyPI distributions: the current OpenImageIO distribution, and a new types-OpenImageIO distribution, with just the stubs. I think it'll be beneficial for users to be able to install the stubs without the library, and thereby keep their environment super light when all they need is type analysis.

This seems pretty wise. I think eventually it would be cool if we were able to build type stubs into the main OpenImageIO distribution itself (too), but only if that doesn't cause conflicts or problems for people.

I can always publish my current stubs as types-OpenImageIO now, with little effort, and this project can take over ownership of that distribution at some future date when it's been integrated here. Because we'll hand off ownership of the same types-OpenImageIO distribution, it can be a smooth transition of ownership, transparent for early adopters, without any dead projects out in the wild causing confusion.

Sounds safe and smart.

So, for the time being, you'd be manually generating new type stubs each time there's an OIIO release?

[chadrik]

So, for the time being, you'd be manually generating new type stubs each time there's an OIIO release?

Well.... I'd make stubs for each major release. Someone may need to nudge me if there's been a breaking change that requires updating the stubs. This is why we all want to see this integrated into the build process in this repo.

[chadrik]

I had a go at adding this to the build process: #4692