MAST: Adding read_product function to mast.observations by AlexReedy · Pull Request #3610 · astropy/astroquery

AlexReedy · 2026-06-12T18:47:19Z

A function of the Mast.Observations class to read product from S3 to memory.

Observations.read_product() takes both FITS and ASDF file URIs and stores them using astropy.io.fits.open and asdf.open respectively.

Connection to S3 is done using fsspec

Changes made

Mast/observations - added read_product() function and import checks for asdf and fsspec
Mast/test - testing suite for read_product that mocks S3 connection and data and checks functionality (test_mast.py) as well as tests live connection to S3 and confirms product type return.

codecov · 2026-06-12T18:51:30Z

Codecov Report

❌ Patch coverage is 78.37838% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.18%. Comparing base (70c49c0) to head (f9a9f45).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
astroquery/mast/observations.py	78.37%	8 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3610   +/-   ##
=======================================
  Coverage   73.17%   73.18%           
=======================================
  Files         227      227           
  Lines       21078    21115   +37     
=======================================
+ Hits        15423    15452   +29     
- Misses       5655     5663    +8

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

snbianco

I had a few comments, but this looks good!

snbianco · 2026-07-01T17:59:15Z

+
+
+@pytest.fixture
+def mock_asdf_open(mocker):


If you add:

pytest.importorskip("asdf") pytest.importorskip("fsspec")

tests that use this fixture will automatically skip if asdf or fsspec are not installed. I think this would be preferable to mocking mock_fsspec_open.

snbianco · 2026-07-01T18:01:39Z

+        Observations.read_product("file.fits")
+
+
+def test_observations_read_product_asdf_missing(monkeypatch):


Could you combine this test with the one before it using parametrization?

snbianco · 2026-07-01T18:02:44Z

+        # Forces the path to be lowercase for the extension checks. This is only used for the checks
+        path = str(product_path).lower()
+
+        # Checks users enviornment for fsspec, required for both fits and asdf


Suggested change

# Checks users enviornment for fsspec, required for both fits and asdf

# Checks users environment for fsspec, required for both fits and asdf

snbianco · 2026-07-01T18:04:41Z

+                data_product = fits.open(product_path, fsspec_kwargs={"anon": True}, **kwargs)
+                log.info(f"Loaded: {product_path}")
+                return data_product
+            except Exception as e:


Can you be more specific about the exception here? Or if you want to catch generally, add a comment explaining why.

snbianco · 2026-07-01T18:05:43Z

+
+        # Logic for reading ASDF files
+        elif path.endswith(".asdf"):
+            if fsspec is None:


Since this package is needed for both FITS and ASDF, can you check that it exists in the environment outside of this block so that it's not there twice?

snbianco · 2026-07-01T18:06:42Z

+                raise ImportError('The "asdf" package is required to read ASDF files. Please install it with '
+                                  '`pip install asdf` or install astroquery with optional dependencies using '
+                                  '`pip install astroquery[all]`.')
+


I'd also like to raise warnings if the optional packages are not found. Something like this:

for package in ["gwcs", "lz4", "roman_datamodels"]: if importlib.util.find_spec(package) is None: warnings.warn(f'The "{package}" package is encouraged when reading ASDF files from the ' 'Roman Space Telescope mission. Please install it with `pip install {package}` ' 'or install astroquery with optional dependencies using ' '`pip install astroquery[all]`.', ImportWarning)

snbianco · 2026-07-01T18:07:33Z

+                data_product = asdf.open(f, ignore_unrecognized_tag=ignore_unrecognized)
+                log.info(f"Loaded: {product_path}")
+                return data_product
+            except Exception as e:


Use a more specific exception here if you can

snbianco · 2026-07-01T18:08:33Z

+.. doctest-remote-data::
+
+   >>> from astroquery.mast import Observations
+   >>> fits_product = Observations.read_product(product_path="s3://stpubdata/hst/public/u9o4/u9o40504m/u9o40504m_c3m.fits")


Can you add a fits_product.info() and asdf_product.info() just to demonstrate that actual objects with data are being returned?

…reviewer feedback

…t is package is not present

bsipocz

please rebase, and once I have a passing review from Sam I'll come back and do a final review.

bsipocz · 2026-07-19T19:43:11Z

    oldestdeps-alldeps: regions==0.5
    oldestdeps-alldeps: astropy-healpix==0.7
+    oldestdeps-alldeps: roman_datamodels==0.11.0
+    oldestdeps-alldeps: gwcs==0.18


gwcs is not listed as a dependency, so including it here makes little sense to me?

But also, if we add all the 3 packages as optional dependencies we will need to make sure they don't introduce some indirect limitations on supported versions.

bsipocz · 2026-07-19T19:44:06Z

+    @pytest.mark.parametrize(
+        "product_path, expected_type",
+        [
+            (
+                "s3://stpubdata/hst/public/u24r/u24r0102t/u24r0102t_c1f.fits",
+                fits.HDUList,
+            )
+        ],
+    )
+    def test_observations_read_product(self, product_path, expected_type):


we allow long lines in astroquery, please reformat it back to something less verbose and whitespace-y

bsipocz · 2026-07-19T19:53:53Z

+    import asdf
+except ImportError:
+    asdf = None
+


You already do skipifs, so is this really necessary? or maybe you can use @pytest.mark.skipif instead of the importorskip, so it's more consistent

Adding read_product function to mast.observations

f00645a

Updating CHANGES.rst and docs/mast/mast_obsquery

69f719b

AlexReedy changed the title ~~Adding read_product function to mast.observations~~ MAST: Adding read_product function to mast.observations Jun 12, 2026

Alex Reedy added 6 commits June 12, 2026 15:33

Fixing Docs and CHANGES.rst

8e4240d

Updating coverage

ee3d10a

Updating docs

6c1c2d8

adding asdf to docs env

7cf4e22

Merge branch 'main' into read_product_function

6925301

Fixing docs and merging changes from main

aa5026e

AlexReedy marked this pull request as ready for review June 23, 2026 15:10

snbianco self-requested a review July 1, 2026 17:55

snbianco added the mast label Jul 1, 2026

snbianco reviewed Jul 1, 2026

View reviewed changes

AlexReedy and others added 7 commits July 6, 2026 10:22

Merge branch 'astropy:main' into read_product_function

1afbc52

Merge branch 'astropy:main' into read_product_function

436ab8b

Adding tests for missing packages and updating read_product based on …

31790cc

…reviewer feedback

Aligning test methods with MastMissions.read_product to skip asdf tes…

aaf14d4

…t is package is not present

Removing dependencies that are no longer required

37c0716

Adding skip to test based on changes requested to test env packages

3311c53

Adjusting remote test

f9a9f45

bsipocz reviewed Jul 19, 2026

View reviewed changes

		Observations.read_product("file.fits")


		def test_observations_read_product_asdf_missing(monkeypatch):

	# Checks users enviornment for fsspec, required for both fits and asdf
	# Checks users environment for fsspec, required for both fits and asdf

Uh oh!

Uh oh!

Conversation

AlexReedy commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

snbianco left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bsipocz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AlexReedy commented Jun 12, 2026 •

edited

Loading

codecov Bot commented Jun 12, 2026 •

edited

Loading