Use ABI3 for wheels and drop cp314t until we have tests

[ocefpaf]

At the moment we are building +30 wheels! The free-threaded Python ones are not tests, right? The other can be simplified significantly by using the Python stable ABI (ABI3), if we are OK to support Python >=3.11.

What do you think @ChrisBarker-NOAA?

[ChrisBarker-NOAA]

OK to support Python >=3.11

Yeah, that's fine. So ABI3 should be good.

I forgot about free-threading Pygthon -- yeah we shouldn't do that without tests.

It looks like to test, we'll need:

[tool.pixi.dependencies]
python-freethreading = "3.14"

But what I'm not sure of is how to make sure that we are actually testing multi-threading -- I suppose we'll need to write some multi-threaded tests?

Or maybe there's a pytest plugin that will do that for you? Or some other third party package?

This is all new -- I haven't really been paying attention to what folks are doing ....

[ocefpaf]

But what I'm not sure of is how to make sure that we are actually testing multi-threading -- I suppose we'll need to write some multi-threaded tests?

That is the tricky part. I heard about some libraries that should help with creating those tests. I'll do some research.

[ChrisBarker-NOAA]

Thanks -- meanwhile, let's get it built without free threading :-)

[ocefpaf]

Thanks -- meanwhile, let's get it built without free threading :-)

For cp314t, free-threading, we should follow some of the guidelines in https://py-free-threading.github.io/testing/.

[ChrisBarker-NOAA]

Hmm -- this is going to take some work :-(

For this particular project, it's probably not too bad, but you never know!

Maybe:
https://github.com/tonybaloney/pytest-freethreaded

Can help. That, at least, looks like it's worth a shot.

On another note -- I'm pretty sure libnetCDF is not thread safe -- so that will affect a lot of stuff in the IOOS world.

Is there a place to capture notes, etc on that? Maybe a discussion in the IOOS org?

[ocefpaf]

Is there a place to capture notes, etc on that? Maybe a discussion in the IOOS org?

We can use that, but it is not common. Most people end up writing to the ioos_tech mailing list.

For this particular project, it's probably not too bad, but you never know!

I never tried that to be honest. I need to read more about it. My approach on this is to just disable cp314t at the moment.

On another note -- I'm pretty sure libnetCDF is not thread safe

Note that people are publishing cp314t regardless and warning users to... Not use thread!? I confess I don't see the point, but I'm not an expert here.

Unidata/netcdf4-python#1454 and Unidata/netcdf4-python#1459

[ocefpaf]

Maybe: https://github.com/tonybaloney/pytest-freethreaded

I have no idea which one is better, but https://github.com/Quansight-Labs/pytest-run-parallel seems to have more contributors and is active. The last commit in pytest-freethreaded was 2 years ago.

[ngoldbaum]

Hi! Happened to stumble on this discussion. Happy to help out.

but https://github.com/Quansight-Labs/pytest-run-parallel seems to have more contributors and is active. The last commit in pytest-freethreaded was 2 years ago.

Yes, pytest-run-parallel is the one you want to look at.

Pytest-run-parallel is only useful for finding issues due to use of global state in the implementation of a Python module. It doesn't test shared mutation of data structures, for example. NumPy doesn't support shared mutation of ndarrays and leave it up to users to handle synchronization: https://numpy.org/devdocs/reference/thread_safety.html#thread-safety. You could follow that approach.

I would also add test for multithreaded workflows that you think make sense to do with the library. Are there any workflows where you might parallelize something with multiprocessing? You could add a test that does a toy version with multithreading.

Happy to answer more questions.

[ChrisBarker-NOAA]

Thanks @ngoldbaum: that was very helpful.

All good so far :-)

I've added a test_threading job to the pixi.toml, and enabled it in the gitHub workflows -- at least for py 3.14

I've got it running with py313 and py314. (just 'cause)

Now t figure out how to test with 3.14t ....

[ChrisBarker-NOAA]

And success!
.
.
.
Of the testing, which is now failing on free-threaded Python, thanks to:

Check for compiled extension modules that do not use the 'Py_mod_gil' slot or the 'PyUnstable_Module_SetGIL' API.

So we need figure out how tup update to use the no GIL api -- and then see if the tests pass.

Closing this Issue and starting one for the "real" work.