ci: add caching to speed up tutor-based CI workflow by Copilot · Pull Request #802 · mitodl/open-edx-plugins

Copilot · 2026-05-19T15:56:03Z

What are the relevant tickets?

N/A

Description (What does it do?)

Adds several caching layers to .github/workflows/ci.yml to avoid rebuilding the Tutor environment from scratch on every push. The biggest time sinks were the Docker image build (~15–20 min), the edx-platform clone (~5–10 min), and Tutor config generation (~9–15 min).

Changes:

Resolve edx-platform tip SHA early — a cheap git ls-remote call (no download) gives the current HEAD SHA for the branch; used as cache key across all subsequent caches.
Cache built dist/ packages — keyed by a hash of all files under src/**, pyproject.toml, and uv.lock (including static assets and templates); skips uv build --all-packages on hits.
Cache pip packages — caches ~/.cache/pip keyed by branch + OS; speeds up repeated pip install tutor>=… calls.
Shallow-clone edx-platform + cache the clone dir — replaces the full two-step clone+checkout with a single git clone --depth=1 --branch=… and caches the directory keyed by branch+SHA; skips the clone entirely on hits.
Docker image cache via ghcr.io — before building, tries to pull ghcr.io/mitodl/openedx-dev-cache:<branch-sha>. On hit, retags the pulled image and skips tutor images build openedx-dev. On miss, builds normally then pushes to ghcr.io for future runs using tutor config printvalue DOCKER_IMAGE_OPENEDX_DEV to resolve the correct image name. Push is conditioned on push events only (never on pull_request runs). The GHCR namespace is hardcoded via env.GHCR_CACHE_OWNER: mitodl so fork PRs always pull from the upstream org's cache.
Cache Tutor config directory — caches ~/.local/share/tutor and ~/.local/share/tutor-main keyed by Tutor version + branch; skips tutor config save entirely on hits (the docker-compose env files are already present from the cache).
Generate edx-platform egg-info on the host runner — runs pip install --no-deps -e /path/to/edx-platform after the edx-platform is cloned/restored. When edx-platform is bind-mounted into the Tutor container, the image's Open_edX.egg-info/ directory is overwritten by the host checkout (which has no egg-info). Without the egg-info, pkg_resources cannot read the edx-platform's lms.djangoapp entry points, so get_plugin_apps(ProjectType.LMS) fails to discover apps like content_libraries, causing a RuntimeError at Django startup. Previously, tutor dev init (via mounted-directories.sh) regenerated this egg-info inside a running container; this step reproduces that regeneration in ~5 seconds on the host without starting any containers.

Estimated savings per run on cache hit:

Step	Before	After
Clone edx-platform	~5–10 min (full)	< 1 min (shallow) or skip
Build Docker image	~15–20 min	skip
Generate Tutor config	~9–15 min	~5 s or skip
Tutor pip install	~1–2 min	< 30 s

How can this be tested?

Open a PR and observe the CI run: the first run will populate all caches.
Push another commit to the same PR without changing edx-platform or the plugin source — subsequent runs should skip the Docker build, the edx-platform clone, and tutor config save.
Verify the ghcr.io/mitodl/openedx-dev-cache package is created in the org's packages after the first run on the main branch.

Additional Context

The packages: write permission is declared at the job level (GitHub Actions does not support conditional job-level permissions). However, it is never exercised during pull_request runs: both the GHCR login step and the Docker push step are conditioned on github.event_name == 'push'. PRs pull from GHCR using anonymous access (the cache package is public-readable).
tutor dev launch -I --skip-build was replaced by tutor config save because the former starts Docker containers (MongoDB, MySQL, etc.) that write data files owned by root. Those root-owned files caused tar: Permission denied errors in the post-cache step, preventing the Tutor config cache from ever being saved. tutor config save generates all needed docker-compose files in ~5 seconds without starting any containers.
The edx-platform's Open_edX.egg-info/ is generated on the host runner rather than inside a container. The egg-info directory is a portable text-format metadata directory; pkg_resources inside the container reads it correctly via the bind mount regardless of which Python generated it. The egg-info is also captured in the edx-platform directory cache, so on cache hits it is already present.
The Tutor config cache key omits the edx-platform SHA (unlike the Docker image and edx-platform clone caches). The generated config files depend only on the Tutor version and branch, not on the specific edx-platform commit, so tying the key to the SHA would cause unnecessary invalidations on every new master commit.
The Docker image pushed to ghcr.io will accumulate over time. Consider configuring a package retention policy in GitHub settings to clean up old image tags automatically.
actions/cache@v4 (SHA-pinned) and docker/login-action@v3 (SHA-pinned) are the only new actions added.

- Resolve edx-platform tip SHA early (full 40-char SHA used as cache key) - Cache built dist/ packages keyed by source hash - Cache pip packages for faster Tutor installation - Shallow-clone edx-platform (--depth=1) and cache the clone dir by branch+SHA - Add permissions: packages: write for ghcr.io push - Login to ghcr.io conditioned on non-fork PRs and pushes - Pull openedx-dev Docker image from ghcr.io cache; skip build on hit - Push freshly-built Docker image to ghcr.io (non-fork only) - Cache Tutor config directory; skip tutor dev launch entirely on hit Agent-Logs-Url: https://github.com/mitodl/open-edx-plugins/sessions/383405df-dd1b-48c8-b7d7-82aaa58a1fb8 Co-authored-by: asadali145 <52656433+asadali145@users.noreply.github.com>

- Improve SHA resolution error message with more detail - Add uv.lock to dist/ cache key hash - Validate DOCKER_IMAGE_OPENEDX is non-empty before using it Agent-Logs-Url: https://github.com/mitodl/open-edx-plugins/sessions/383405df-dd1b-48c8-b7d7-82aaa58a1fb8 Co-authored-by: asadali145 <52656433+asadali145@users.noreply.github.com>

for more information, see https://pre-commit.ci

Copilot

Pull request overview

This PR speeds up the Tutor-based integration test workflow by adding multiple caches (Python build artifacts, pip cache, edx-platform checkout, Tutor config, and a Docker image cache in GHCR) so repeated CI runs don’t rebuild everything from scratch.

Changes:

Resolve the edx-platform branch tip SHA up front and use it to key subsequent caches.
Cache dist/ artifacts, pip downloads, and the edx-platform checkout to reduce repeated network/build work.
Add a GHCR-backed Docker image cache and cache Tutor config dirs to potentially skip image builds and tutor dev launch.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

asadali145 · 2026-05-19T21:37:01Z

@copilot fix the failing integration tests in https://github.com/mitodl/open-edx-plugins/actions/runs/26108947537/job/76780542597, look for the reason and fix it.

On fork PRs, github.repository_owner resolves to the fork owner's namespace, so the cache image tag pointed at an empty registry. Introduce a workflow-level GHCR_CACHE_OWNER env var hardcoded to the upstream org (mitodl) so the cache tag always resolves correctly regardless of who opens the PR. Agent-Logs-Url: https://github.com/mitodl/open-edx-plugins/sessions/c0dd75d2-1aba-4631-9f2a-71b8b496d86c Co-authored-by: asadali145 <52656433+asadali145@users.noreply.github.com>

…g key The previous code constructed the dev image name by appending "-dev" to DOCKER_IMAGE_OPENEDX (which gives e.g. "overhangio/openedx:21.0.6-main-dev"), but Tutor actually builds the dev image under a different name ("openedx-dev:21.0.6-main"). Switch to tutor's canonical DOCKER_IMAGE_OPENEDX_DEV config key so the docker tag and docker push steps reference the image that Tutor actually created, avoiding the "No such image" error seen in CI. Agent-Logs-Url: https://github.com/mitodl/open-edx-plugins/sessions/219f72d7-6a13-4fa9-8fb7-2c8b1023be1d Co-authored-by: asadali145 <52656433+asadali145@users.noreply.github.com>

Copilot · 2026-05-19T21:40:17Z

@copilot fix the failing integration tests in https://github.com/mitodl/open-edx-plugins/actions/runs/26108947537/job/76780542597, look for the reason and fix it.

Fixed in 2895b45. The failure was in the "Push Docker image to registry cache" step:

Error response from daemon: No such image: overhangio/openedx:21.0.6-main-dev

The workflow was constructing the dev image name by appending -dev to DOCKER_IMAGE_OPENEDX (overhangio/openedx:21.0.6-main → overhangio/openedx:21.0.6-main-dev), but Tutor actually builds and names the dev image as openedx-dev:21.0.6-main — a completely different naming scheme. The fix replaces the manual construction with tutor config printvalue DOCKER_IMAGE_OPENEDX_DEV, which returns the correct local image name that Tutor actually created.

asadali145 · 2026-05-20T07:52:32Z

@copilot CI checks are still taking about 25 minutes on master branch. This includes about 15m for the tutor launch step. Looks like there is still some issue? Look at the latest CI run and see what can be fixed/improved.

…launch The Tutor config cache was never hitting because: 1. tutor dev launch -I --skip-build starts Docker containers (MongoDB, MySQL) which write data files owned by root. When the post-cache step tries to tar ~/.local/share/tutor[-main]/data/, it gets "Permission denied" errors and the cache is never saved, causing tutor dev launch to re-run every time (9-15 minutes). 2. The cache key included the edx-platform tip SHA, which changes many times per day for the master branch. Even if the cache had saved correctly, it would miss on every new master commit. Fix: - Replace `tutor dev launch -I --skip-build; tutor dev stop` with `tutor config save`, which generates the docker-compose files needed by the test runner in ~5 seconds without starting any containers (so no root-owned data files are created and the cache saves cleanly). - Remove the edx-platform SHA from the Tutor config cache key. The generated config files (docker-compose templates, nginx configs, etc.) don't depend on the specific edx-platform SHA content, only on the Tutor version and branch. This makes the cache persistent across edx-platform tip changes. Agent-Logs-Url: https://github.com/mitodl/open-edx-plugins/sessions/0fb67622-1262-4825-b046-161b9238b734 Co-authored-by: asadali145 <52656433+asadali145@users.noreply.github.com>

Copilot · 2026-05-20T08:03:04Z

@copilot CI checks are still taking about 25 minutes on master branch. This includes about 15m for the tutor launch step. Looks like there is still some issue? Look at the latest CI run and see what c...

Investigated and fixed in 396604e. Two root causes were found:

Root cause 1 — Tutor config cache never saved due to permission errors:
tutor dev launch -I --skip-build starts Docker containers (MongoDB, MySQL, etc.) which write data files owned by root into ~/.local/share/tutor[-main]/data/. The Post-cache step's tar couldn't read those files (Permission denied), so the cache was never stored and tutor dev launch re-ran every time (9–15 min).

Root cause 2 — Cache key included edx-platform SHA:
The master branch gets new commits many times per day, changing the SHA and invalidating the cache key even if the save had worked.

Fixes:

Replaced tutor dev launch -I --skip-build; tutor dev stop with tutor config save — generates the docker-compose files the test runner needs in ~5 seconds, without starting any containers (so no root-owned data files, cache saves cleanly).
Removed the edx-platform SHA from the Tutor config cache key. The generated config files (docker-compose templates, etc.) don't depend on edx-platform's specific commit — only on the Tutor version and branch. The cache now persists across edx-platform tip changes.

On the next run after a cache miss, "Generate Tutor config" will take ~5 seconds and save cleanly. Subsequent runs will skip it entirely via cache hit.

asadali145 · 2026-05-20T08:17:38Z

@copilot fix the pre-commit checks

- Quote all $GITHUB_OUTPUT redirections to satisfy shellcheck SC2086 - Quote $DIRECTORY, $DEV, and $PWD variables in the docker compose command to satisfy shellcheck SC2086 - Expand docker compose command to multi-line for readability - Accept yamlfmt reformatting of the long cache key line Agent-Logs-Url: https://github.com/mitodl/open-edx-plugins/sessions/e3c0d127-a6fc-49de-9bdd-c9d74a37e82f Co-authored-by: asadali145 <52656433+asadali145@users.noreply.github.com>

Copilot · 2026-05-20T08:22:12Z

@copilot fix the pre-commit checks

Fixed in b2f5f29. Two pre-commit hooks were failing:

yamlfmt — reformatted the long tutor-config cache key line (auto-fixed by the hook itself).
actionlint (shellcheck SC2086) — several unquoted shell variables:
- All >> $GITHUB_OUTPUT redirections → >> "$GITHUB_OUTPUT"
- The docker compose command's $DIRECTORY, $DEV, and $PWD variables → properly quoted; also expanded that long single-line command to multi-line for readability.

asadali145 · 2026-05-20T10:08:53Z

@copilot now tests are failing with missing apps, I think tutor launch cannot be replaced with tutor config save. R&D on how tutor works with open edx and then make required changes

… entry points The mounted-directories.sh init script (run by tutor dev init) creates Open_edX.egg-info/ in the bind-mounted edx-platform. Without this, get_plugin_apps(ProjectType.LMS) cannot find content_libraries, causing the INSTALLED_APPS error. This step runs pip install --no-deps -e on the host runner to create the egg-info without needing container startup. Agent-Logs-Url: https://github.com/mitodl/open-edx-plugins/sessions/1271879d-dd58-49da-b1bf-7433da499873 Co-authored-by: asadali145 <52656433+asadali145@users.noreply.github.com>

Copilot · 2026-05-20T10:34:23Z

@copilot now tests are failing with missing apps, I think tutor launch cannot be replaced with tutor config save. R&D on how tutor works with open edx and then make required changes

Fixed in 276723d.

Root cause: tutor dev launch runs tutor dev init, which executes the mounted-directories.sh script inside the LMS container. This script detects that edx-platform is bind-mounted (by checking for the missing bindmount-canary file) and runs pip install -e . to regenerate Open_edX.egg-info/ in the mounted directory.

Without this egg-info, pkg_resources cannot find the edx-platform's entry points. The edx-platform registers content_libraries (and many other apps) as LMS Django apps via the lms.djangoapp entry point group. The get_plugin_apps(ProjectType.LMS) call in lms/envs/common.py reads from these entry points to populate INSTALLED_APPS. Without the egg-info, content_libraries is never added to INSTALLED_APPS, causing the RuntimeError.

Fix: Added a step after Add edx-platform tutor mounts that runs pip install --no-deps -e /path/to/edx-platform on the host runner. This creates the Open_edX.egg-info/ directory in the edx-platform directory, which is then available inside the container via the bind mount. No container startup needed — this is equivalent to what mounted-directories.sh does but runs in ~5 seconds on the host instead of the ~9 minutes that tutor dev launch took.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

asadali145 · 2026-05-20T12:46:30Z

    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write


@copilot incorporate the above feedback.

Fixed in 3509aa6.

GitHub Actions does not support conditional job-level permissions (permissions must be static YAML), so a full job split would require saving and uploading the Docker image as a tarball artifact (several GB per matrix combination) then downloading and loading it in a separate job — adding significant overhead.

Instead, the feedback is addressed by ensuring packages: write is never exercised during PR runs:

The GHCR login step condition changed from github.event_name != 'pull_request' || same-repo-pr → github.event_name == 'push': the GITHUB_TOKEN with write scope is never used during PRs (PRs pull from GHCR anonymously, which works for public packages)

The Docker push step condition simplified to github.event_name == 'push': push never runs during PRs

A comment on packages: write documents this design and the GitHub Actions limitation

Also fixed the Tutor config cache key that was split across lines (reviewer comment at line 166-167).

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

for more information, see https://pre-commit.ci

…matting - Change GHCR login condition from 'not PR or same-repo PR' to 'push only' so the GITHUB_TOKEN with write scope is never used during pull_request runs - Change Docker push condition to 'push only' (same reasoning) - Add a comment on packages:write explaining the GitHub Actions limitation (job-level permissions can't be conditional) and confirming the permission is never exercised during PR runs - Fix Tutor config cache key that was split across lines Agent-Logs-Url: https://github.com/mitodl/open-edx-plugins/sessions/e7bee42e-7e80-4928-b5d0-06d5cc77f3c4 Co-authored-by: asadali145 <52656433+asadali145@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

.github/workflows/ci.yml:167

The cache key for the Tutor config cache is split across multiple lines. YAML will treat this as a multi-line plain scalar (folded), which is easy to accidentally break and makes it harder to reason about the exact cache key being generated. Keeping the full ${{ ... }} expression on a single line would be more robust and consistent with the other cache keys.

        key: tutor-config-${{ steps.tutor-version.outputs.version }}-${{ matrix.edx_branch
          }}

+    - name: Add edx-platform tutor mounts
+      run: |
+        cd ${{ github.workspace }}/../edx-platform
        tutor mounts add .



+          exit 1
+        fi
+        echo "openedx_dev_image=$OPENEDX_DEV_IMAGE" >> "$GITHUB_OUTPUT"
+        CACHE_IMAGE_TAG="ghcr.io/${{ env.GHCR_CACHE_OWNER }}/openedx-dev-cache:${{ steps.edx-sha.outputs.branch_slug }}-${{ steps.edx-sha.outputs.sha }}"


Copilot AI and others added 2 commits May 19, 2026 15:54

Copilot AI assigned Copilot and asadali145 May 19, 2026

[pre-commit.ci] auto fixes from pre-commit.com hooks

df3ac99

for more information, see https://pre-commit.ci

Copilot created this pull request from a session on behalf of asadali145 May 19, 2026 15:56 View session

Copilot finished work on behalf of asadali145 May 19, 2026 15:56

Copilot AI requested a review from asadali145 May 19, 2026 15:56

asadali145 requested a review from Copilot May 19, 2026 15:57

Copilot started reviewing on behalf of asadali145 May 19, 2026 15:58 View session

Copilot AI reviewed May 19, 2026

View reviewed changes

Comment thread .github/workflows/ci.yml Outdated

Comment thread .github/workflows/ci.yml Outdated

Comment thread .github/workflows/ci.yml Outdated

asadali145 and others added 2 commits May 20, 2026 02:35

Potential fix for pull request finding

c3a98b6

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

1080b6f

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot started work on behalf of asadali145 May 19, 2026 21:36 View session

Copilot finished work on behalf of asadali145 May 19, 2026 21:37

Copilot started work on behalf of asadali145 May 19, 2026 21:37 View session

Copilot finished work on behalf of asadali145 May 19, 2026 21:40

Copilot started work on behalf of asadali145 May 20, 2026 07:52 View session

Copilot finished work on behalf of asadali145 May 20, 2026 08:04

Copilot started work on behalf of asadali145 May 20, 2026 08:17 View session

Copilot finished work on behalf of asadali145 May 20, 2026 08:22

Copilot started work on behalf of asadali145 May 20, 2026 10:09 View session

Copilot finished work on behalf of asadali145 May 20, 2026 10:35

asadali145 requested a review from Copilot May 20, 2026 12:38

Copilot started reviewing on behalf of asadali145 May 20, 2026 12:39 View session

Copilot AI reviewed May 20, 2026

View reviewed changes

Copilot started work on behalf of asadali145 May 20, 2026 12:46 View session

asadali145 and others added 4 commits May 20, 2026 17:46

Potential fix for pull request finding

591b8d9

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

1255ce0

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

99fac6a

for more information, see https://pre-commit.ci

Copilot finished work on behalf of asadali145 May 20, 2026 12:55

asadali145 requested a review from Copilot May 21, 2026 18:50

Copilot started reviewing on behalf of asadali145 May 21, 2026 18:51 View session

Copilot AI reviewed May 21, 2026

View reviewed changes

Conversation

Copilot AI commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What are the relevant tickets?

Description (What does it do?)

How can this be tested?

Additional Context

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asadali145 commented May 19, 2026

Uh oh!

Copilot AI commented May 19, 2026

Uh oh!

asadali145 commented May 20, 2026

Uh oh!

Copilot AI commented May 20, 2026

Uh oh!

asadali145 commented May 20, 2026

Uh oh!

Copilot AI commented May 20, 2026

Uh oh!

asadali145 commented May 20, 2026

Uh oh!

Copilot AI commented May 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

asadali145 May 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented May 19, 2026 •

edited

Loading