Add GitHub token authentication to Makefile downloads by stuggi · Pull Request #1541 · openstack-k8s-operators/openstack-operator

stuggi · 2025-07-23T13:16:35Z

Fixes GitHub rate limiting issues in CI by automatically using GITHUB_TOKEN when available for downloads from GitHub (raw.githubusercontent.com and github.com/releases). Falls back gracefully to unauthenticated downloads when no token is present, maintaining compatibility with local development.

E.g seen in https://github.com/openstack-k8s-operators/openstack-operator/actions/runs/15896190926/job/44828209996, force-bump actions can fail with github rate limit:

Run make bindata
  make bindata
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
test -s /home/runner/work/openstack-operator/openstack-operator/bin/kustomize || { curl -Ss "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash -s -- 5.5.0  /home/runner/work/openstack-operator/openstack-operator/bin; }
Github rate-limiter failed the request. Either authenticate or wait a couple of minutes.
make: *** [Makefile:319: /home/runner/work/openstack-operator/openstack-operator/bin/kustomize] Error 1

Updated targets:

kustomize (install script)
yq (binary release)
kuttl (binary release)
operator-sdk (binary release)
opm (binary release)
golangci-lint (install script)

Usage in CI: export GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}
Local dev: no changes required

Jira: OSPRH-17779

openshift-ci · 2025-07-23T13:16:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: stuggi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [stuggi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

softwarefactory-project-zuul · 2025-07-23T13:30:05Z

This change depends on a change that failed to merge.

Change openstack-k8s-operators/openstack-k8s-operators-ci#137 is needed.

stuggi · 2025-07-23T14:49:24Z

/retest

stuggi · 2025-07-23T14:49:30Z

recheck

softwarefactory-project-zuul · 2025-07-23T14:58:06Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/15cfbf7a78af45b6bc7dfac4cbb2de3f

❌ openstack-k8s-operators-content-provider FAILURE in 6m 03s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ openstack-operator-tempest-multinode SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

ratailor · 2025-07-24T06:14:32Z

-	test -s $(LOCALBIN)/golangci-lint || curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s v1.59.1
+	test -s $(LOCALBIN)/golangci-lint || { \
+		CURL_AUTH_HEADER=""; \
+		if [ -n "$$GITHUB_TOKEN" ]; then \


shouldn't we use $ instead of $$ if we want to access variable value. IIUC the later would return PID of running process.

because its a Makefile to access environment variables we have to use $$. To pass a literal $ to the shell, you must escape it from make's interpretation by doubling it.

stuggi · 2025-07-24T07:15:24Z

/retest

stuggi · 2025-07-24T07:15:33Z

recheck

softwarefactory-project-zuul · 2025-07-24T07:25:42Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/337bf767273d4552a4e1bfc90e84c9a3

❌ openstack-k8s-operators-content-provider FAILURE in 6m 02s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ openstack-operator-tempest-multinode SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

softwarefactory-project-zuul · 2025-07-24T08:41:35Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/c61e973987134767af4a48a2b5fdfc5f

❌ openstack-k8s-operators-content-provider FAILURE in 5m 59s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ openstack-operator-tempest-multinode SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

softwarefactory-project-zuul · 2025-07-24T09:38:23Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/1fbc4d5c57184b84ac0dd0a06ce0d7b8

❌ openstack-k8s-operators-content-provider FAILURE in 7m 46s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ openstack-operator-tempest-multinode SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

Fixes GitHub rate limiting issues in CI by automatically using GITHUB_TOKEN when available for downloads from GitHub (raw.githubusercontent.com and github.com/releases). Falls back gracefully to unauthenticated downloads when no token is present, maintaining compatibility with local development. Updated targets: - kustomize (install script) - yq (binary release) - kuttl (binary release) - operator-sdk (binary release) - opm (binary release) - golangci-lint (install script) Usage in CI: export GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }} Local dev: no changes required Jira: OSPRH-17779 Signed-off-by: Martin Schuppert <mschuppert@redhat.com>

softwarefactory-project-zuul · 2025-07-24T10:10:08Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/8f8f1f56e538465bb972663e0dcd2f97

❌ openstack-k8s-operators-content-provider FAILURE in 5m 54s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ openstack-operator-tempest-multinode SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

stuggi · 2025-07-24T10:15:01Z

looks like ci-framework is calling the make command in the wrong dir openstack-k8s-operators-ci?
https://logserver.rdoproject.org/df4/rdoproject.org/df4015a7b8cc4522ad7bcfa68f44d478/ci-framework-data/logs/ci_script_001_openstack_k8s_operators_ci.log

~/src/github.com/openstack-k8s-operators/openstack-k8s-operators-ci ~/ci-framework-data/artifacts
make: *** No rule to make target 'manifests'.  Stop.

comparing to a different job there it is
https://logserver.rdoproject.org/867/rdoproject.org/86758a08ecee4c7cab5b999d32c3ea5e/ci-framework-data/logs/ci_script_001_openstack_operator_call.log

~/src/github.com/openstack-k8s-operators/openstack-operator ~/ci-framework-data/artifacts
test -f go.work || GOTOOLCHAIN=go1.21.0 go work init
go: downloading go1.21.0 (linux/amd64)

stuggi · 2025-07-24T10:22:37Z

2025-07-24 05:37:54,168 p=26016 u=zuul n=ansible | TASK [operator_build : openstack-k8s-operators-ci - Call manifests dry_run={{ cifmw_operator_build_dryrun|bool }}, chdir={{ operator.src }}, output_dir={{ cifmw_operator_build_basedir }}/artifacts, script=make manifests] ***
2025-07-24 05:37:54,168 p=26016 u=zuul n=ansible | Thursday 24 July 2025  05:37:54 -0400 (0:00:00.036)       0:02:37.880 ********* 
2025-07-24 05:37:54,231 p=26016 u=zuul n=ansible | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_001_openstack_k8s_operators_ci.log
2025-07-24 05:37:54,340 p=26016 u=zuul n=ansible | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: NoneType: None
2025-07-24 05:37:54,341 p=26016 u=zuul n=ansible | fatal: [localhost]: FAILED! => 
    changed: true
    msg: non-zero return code
    rc: 2
    stderr: ''
    stderr_lines: []
    stdout: |
      ~/src/github.com/openstack-k8s-operators/openstack-k8s-operators-ci ~/ci-framework-data/artifacts
      make: *** No rule to make target 'manifests'.  Stop.
    stdout_lines:
    - ~/src/github.com/openstack-k8s-operators/openstack-k8s-operators-ci ~/ci-framework-data/artifacts
    - 'make: *** No rule to make target ''manifests''.  Stop.'

stuggi · 2025-07-24T10:23:47Z

I have a depends-on on the openstack-k8s-operators-ci repo ... lets remove that.

Depends-On: https://github.com/openstack-k8s-operators/openstack-k8s-operators-ci/pull/137

stuggi · 2025-07-24T10:24:29Z

recheck

stuggi · 2025-07-24T11:59:37Z

/retest

openshift-ci · 2025-07-24T12:59:59Z

@stuggi: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/openstack-operator-build-deploy-kuttl	`7d679d0`	link	true	`/test openstack-operator-build-deploy-kuttl`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

dprince

Instead of automating it this way would it be better to just have the CI base container automatically add these dependencies via a Dockerfile? That way they won't be added repeatedly in each job. Also, the Makefile's for service operators would be simpler.

stuggi · 2025-07-28T13:47:34Z

Instead of automating it this way would it be better to just have the CI base container automatically add these dependencies via a Dockerfile? That way they won't be added repeatedly in each job. Also, the Makefile's for service operators would be simpler.

@dprince I have submitted PR [1] with the alternative approach to use a pre-build image with tools installed. One point I could see is that we need multiple images if we bump tool version, like golang, when we do the bump one service repo after each other.

will add hold on this until we decided which way to go

[1] openstack-k8s-operators/openstack-k8s-operators-ci#138

stuggi · 2025-08-01T12:48:38Z

closing in preference of openstack-k8s-operators/openstack-k8s-operators-ci#138

openshift-ci Bot requested review from dprince and slagle July 23, 2025 13:16

openshift-ci Bot added the approved label Jul 23, 2025

stuggi requested review from abays and olliewalsh July 23, 2025 14:50

ratailor reviewed Jul 24, 2025

View reviewed changes

stuggi force-pushed the OSPRH-17779 branch from d7bef92 to 956e244 Compare July 24, 2025 08:34

stuggi added the do-not-merge/hold label Jul 24, 2025

stuggi force-pushed the OSPRH-17779 branch from 956e244 to a8fca1b Compare July 24, 2025 09:29

stuggi force-pushed the OSPRH-17779 branch from a8fca1b to 7d679d0 Compare July 24, 2025 10:02

stuggi removed the do-not-merge/hold label Jul 24, 2025

dprince reviewed Jul 25, 2025

View reviewed changes

stuggi added the do-not-merge/hold label Jul 28, 2025

stuggi mentioned this pull request Jul 28, 2025

Add CI build tools container with tool coverage openstack-k8s-operators/openstack-k8s-operators-ci#138

Merged

stuggi closed this Aug 1, 2025

Conversation

stuggi commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci Bot commented Jul 23, 2025

Uh oh!

softwarefactory-project-zuul Bot commented Jul 23, 2025

Uh oh!

stuggi commented Jul 23, 2025

Uh oh!

stuggi commented Jul 23, 2025

Uh oh!

softwarefactory-project-zuul Bot commented Jul 23, 2025

Uh oh!

ratailor Jul 24, 2025

Choose a reason for hiding this comment

Uh oh!

stuggi Jul 24, 2025

Choose a reason for hiding this comment

Uh oh!

stuggi commented Jul 24, 2025

Uh oh!

stuggi commented Jul 24, 2025

Uh oh!

softwarefactory-project-zuul Bot commented Jul 24, 2025

Uh oh!

softwarefactory-project-zuul Bot commented Jul 24, 2025

Uh oh!

softwarefactory-project-zuul Bot commented Jul 24, 2025

Uh oh!

softwarefactory-project-zuul Bot commented Jul 24, 2025

Uh oh!

stuggi commented Jul 24, 2025

Uh oh!

stuggi commented Jul 24, 2025

Uh oh!

stuggi commented Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stuggi commented Jul 24, 2025

Uh oh!

stuggi commented Jul 24, 2025

Uh oh!

openshift-ci Bot commented Jul 24, 2025

Uh oh!

dprince left a comment

Choose a reason for hiding this comment

Uh oh!

stuggi commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stuggi commented Aug 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

stuggi commented Jul 23, 2025 •

edited

Loading

stuggi commented Jul 24, 2025 •

edited

Loading

stuggi commented Jul 28, 2025 •

edited

Loading