Refactor integration testing system to build a dashboard and run in CI by astrofrog · Pull Request #44 · astropy/astropy-integration-testing

astrofrog · 2026-05-11T09:27:04Z

Work in progress!

astrofrog · 2026-05-11T14:16:02Z

Preview of Generated dashboard in CI

Once this is merged, future PRs will have a link posted automatically

bsipocz · 2026-05-11T17:35:32Z

fyi, I get auth error for the dashboard link above.

astrofrog · 2026-05-11T17:47:08Z

Yes sorry I need to fix the link

…sions

astrofrog · 2026-05-13T12:05:16Z

Here's the latest version of the dashboard:

https://github.com/astropy/astropy-integration-testing/suites/68789126288/artifacts/6969081003

As noted now, the CI in PRs is just testing that the infrastructure is all working and packages are working at a basic level but only checks the first 10 tests. This is to save computing/waiting time in PRs. The main published version of the dashboard will be authoritative and include all tests.

astrofrog · 2026-05-13T12:07:30Z

As a disclaimer, this was written in collaboration with an LLM - I thought this would be acceptable since it can produce much nicer dashboards than I ever could, and this is not going to be code running on users' machines.

Cadair · 2026-05-13T12:45:31Z

+            return f"{pkg['pypi_name']}{_extras_suffix(pkg)} @ git+{repo}", None
+
+    else:
+        include_pre = variant == "pre"


Does it always run the pre job even if the versions are the same? If not can we hide it from the report?

Yes I'll check

So technically pre applies to all packages including in wider non astropy ecosystem so could be useful to find out issues

bsipocz

Looks great!

Does this also means that we don't run into the duplicated conftest.py requirement any more, or tests are only passing for astroquery as you capped them at the first 10?

bsipocz · 2026-05-13T16:00:00Z

+          # PR previews limit each package to the first 10 tests so the
+          # matrix finishes in minutes; conftest.py at the repo root reads
+          # this env var and truncates the collected items.
+          PYTEST_LIMIT_N: ${{ github.event_name == 'pull_request' && '10' || '' }}


shouldn't we make sure to run the whole test suite, at least for the coordinated packages?

we discussed it off-github: this is not a concern of mine any more, having the limit on PRs is fine.

bsipocz

Merge this away; we discussed my concern off-band and it was made clear they are not concerns.

pllim · 2026-05-13T16:21:44Z

+  pull_request:
+
+permissions:
+  contents: write   # to push gh-pages on main / schedule / dispatch


Should we add pull-requests: none just to be sure or it does not matter?

pllim

Thank you for this awesome upgrade! Some comments:

Can we please pin GHA workflows to release hashes?
Please add a cooldown to dependabot.yml while you are at it: https://docs.github.com/en/code-security/reference/supply-chain-security/dependabot-options-reference#cooldown-
As mentioned in other review, the dashboard preview link needs to be fixed

pllim · 2026-05-13T16:24:56Z

+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'


Shouldn't this use Python version set up in matrix above?

No, this is just the version for the orchestrator, uv sets up the Python version it needs at the point of setting up the virtual environment to run the tests in.

Comment added to clarify this

pllim · 2026-05-13T16:46:22Z

+ASTROPY_NIGHTLY_INDEX = "https://pypi.anaconda.org/astropy/simple"
+LIBERFA_NIGHTLY_INDEX = (


You mentioned interest to share this infrastructure with non-Astropy projects. If that is to happen, I suspect things like these need to be made customizable.

Agree, though I'd rather keep this as a separate PR to make it easier to review, so I've opened a new issue here: #48

pllim · 2026-05-13T16:53:00Z

+    pytest_args: ["-c", "sunpy_pytest.ini"]
+
+  # --- Affiliated packages auto-imported from
+  # https://github.com/astropy/astropy.github.com/blob/main/affiliated/registry.json


What about newer ones in https://www.astropy.org/affiliated/index.html#affiliated-packages-registry (from pyOpenSci cross-listing)?

pllim · 2026-05-13T16:54:51Z

I think it would be clearer to future maintainers if you name the jobs explicitly rather than a vague "a fourth job" and so on.

astrofrog · 2026-05-13T18:26:47Z

The dashboard preview link will be fixed once this is merged and will work in subsequent PRs

Co-authored-by: P. L. Lim <2090236+pllim@users.noreply.github.com>

Add seven pyOpenSci-reviewed affiliated packages (astrodata, astromartini, autogalaxy, GALFITools, petrofit, stingray, zodipy) under a new pyopensci tier, ranked after affiliated in install and dashboard order. Set an explicit pull-requests: none workflow permission, document why setup-python uses a fixed version, and name the CI jobs explicitly in the README.

Pin every action in the integration and preview-link workflows to a full release commit SHA with a version comment. Add a 7-day cooldown to the github-actions Dependabot entry so a release has time to settle before a bump is proposed.

astrofrog · 2026-05-14T09:06:11Z

Can we please pin GHA workflows to release hashes?

Done

Please add a cooldown to dependabot.yml while you are at it: https://docs.github.com/en/code-security/reference/supply-chain-security/dependabot-options-reference#cooldown-

Done

As mentioned in other review, the dashboard preview link needs to be fixed

Will be fixed after this PR is merged

astrofrog · 2026-05-14T09:06:47Z

@pllim - think I've addressed all your comments?

pllim

✨ Main character energy. No cap! ✨

Given the preview link is currently broken, I will just have to trust your screenshot. We can fix it up later if your screenshot is no longer accurate.

p.s. Looks like this repo is no longer in-tox-icated. 😏

p.p.s. Webring is still on the table, right?

astrofrog force-pushed the refactor branch from 1306ef2 to 7dc1755 Compare May 11, 2026 13:03

neutrinoceros mentioned this pull request May 12, 2026

TST: making lowest-deps envs reproducible #46

Open

astrofrog added 14 commits May 13, 2026 10:13

Refactor integration testing system to build a dashboard and run in CI

856c60f

Add ability to preview page in PRs

d7e7c81

Fixes to CI

1f73091

Simplify CI setup

cb3eb58

Fix CI issues

2e4f0fb

Simplify dashboard set-up

6de8c25

Improvements, including making it possible to run multiple Python ver…

c1c85cc

…sions

Remove old infrastructure

f10df45

Add affiliated packages and make improvements to infrastructure

10dfa60

Fix more issues and only run first 10 tests in PRs

7a5a9c9

Tidy up into package

1e11a23

Improve uv cache handling

0ba2e0b

Cleanup and added pre-commit

e0a7992

More cleanup

22594f3

astrofrog force-pushed the refactor branch from 8719d88 to 22594f3 Compare May 13, 2026 11:15

astrofrog marked this pull request as ready for review May 13, 2026 11:45

astrofrog changed the title ~~WIP: Refactor integration testing system to build a dashboard and run in CI~~ Refactor integration testing system to build a dashboard and run in CI May 13, 2026

astrofrog requested a review from pllim May 13, 2026 12:08

Cadair reviewed May 13, 2026

View reviewed changes

bsipocz reviewed May 13, 2026

View reviewed changes

bsipocz approved these changes May 13, 2026

View reviewed changes

pllim reviewed May 13, 2026

View reviewed changes

astrofrog and others added 3 commits May 14, 2026 09:40

Apply suggestions from code review

3ed80ec

Co-authored-by: P. L. Lim <2090236+pllim@users.noreply.github.com>

pllim approved these changes May 14, 2026

View reviewed changes

pllim merged commit b89a1e7 into astropy:main May 14, 2026
7 checks passed

neutrinoceros mentioned this pull request May 27, 2026

DEP: declare uv as a python dependency instead of relying on a external install #57

Open

		ASTROPY_NIGHTLY_INDEX = "https://pypi.anaconda.org/astropy/simple"
		LIBERFA_NIGHTLY_INDEX = (

Uh oh!

Uh oh!

Conversation

astrofrog commented May 11, 2026

Uh oh!

astrofrog commented May 11, 2026

Uh oh!

bsipocz commented May 11, 2026

Uh oh!

astrofrog commented May 11, 2026

Uh oh!

astrofrog commented May 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

astrofrog commented May 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bsipocz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bsipocz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pllim left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

astrofrog commented May 13, 2026

Uh oh!

astrofrog commented May 14, 2026

Uh oh!

astrofrog commented May 14, 2026

Uh oh!

pllim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

astrofrog commented May 13, 2026 •

edited

Loading