Bug | Transient chart dependency build errors: Original error swallowed + retry mechanism

[TBeijen]

Description

When running ~250 applications through the repo-server-api render method, we quite frequently hit chart download errors, making the entire CI run fail. These errors are not consistent, not limited to specific chart repos, so hard to reproduce.

Two things at play:

• The transient errors are returned as a permissionDenied error. It looks like the original error is replaced because no ProjectRepos are sent to repo-server. (See analysis below, I'll create a PR for this -> Fix for swallowing helm build errors #417 )
• Might be worth considering a retry mechanism, possibly controled by cli arg. (If not already present, haven't investigated yet)

PermissionError

This is triggered by logic in argocd itself, but can probably be avoided by a small change in this project. See: https://github.com/argoproj/argo-cd/blob/master/reposerver/repository/repository.go#L1363

Sequence:

1. helm template fails with a missing dependency error
2. runHelmBuild() is called to fetch dependencies via helm dependency build
3. helm dep build fails transiently (CDN timeout: "cannot be reached" or "could not download")
4. Repo-server iterates Helm repos, checking each against ProjectSourceRepos via isSourcePermitted()
5. Since argocd-diff-preview sends empty ProjectSourceRepos, every repo fails the permission check
6. Because of that, PermissionDenied error is returned: "helm repos X are not permitted in project ''"
7. This error is returned instead of the original error

Fix is probably as simple as sending * as project sourceRepos.

To reproduce, see: https://github.com/TBeijen/argocd-diff-testcases

examples/local-chart-with-deps/app.yaml contains a subchart dependency with an incorrect url. This results in a seemingly unrelated permission error:

❌ failed to render app local-chart-with-deps [t|examples/local-chart-with-deps/app.yaml]: repo server returned error for content source 0: failed to receive manifest response: rpc error: code = PermissionDenied desc = helm repos https://kubernetes.github.oopsie/ingress-nginx are not permitted in project ''
🕵️ Run with '--debug' for more details

Retry mechanism

(not investigated yet, will do after first surfacing the original error)

[TBeijen]

Running this in our CI with the fix from the PR now surfaced the error we see quite often (it's quite often the grafana repo):

❌ failed to render app oncall [b|platform/lgtm/helm/templates/oncall-app.yaml]: repo server returned error for content source 0: failed to receive manifest response: rpc error: code = Unknown desc = error building helm chart dependencies: failed to build helm dependencies: failed to build dependencies: failed running helm: `helm dependency build` failed exit status 1: Error: no cached repository for https:--grafana.github.io-helm-charts found. (try 'helm repo update'): error loading /helm-working-dir/repository/https:--grafana.github.io-helm-charts-index.yaml: empty index.yaml file

Not sure what's happening inside repo-server. I assume in a permanent setup it would have index.yaml from previous succesful pull, but in the ephemeral setup more easily shows intermittent failures in the grafana chart host.

Could retry on specific errors (e.g. this one, but not auth errors) but it looks like it's been a deliberate choice to only retry on transport errors, so I might not have the full picture here.

Ref: https://github.com/dag-andersen/argocd-diff-preview/blob/main/pkg/reposerver/client.go#L252

[dag-andersen]

@TBeijen Quick comment - this old issue might help on the index error 👀 #48 (comment)

[TBeijen]

Ah, interesting.

Configured that in our setup now, let's see how it goes. Based on analysis (by my token-consuming friend, see below) I have confidence this works.

Could be worth considering making this config setting the default?
(But might be first spot where argocd install helm chart defaults would be overriden, so that might complicate things.)

---

(🤖 / 👨)

When repoServer.useEphemeralHelmWorkingDir: true (the default), the argo-cd Helm chart sets HELM_CACHE_HOME, HELM_CONFIG_HOME, and HELM_DATA_HOME to /helm-working-dir, an emptyDir volume. This means:

• Every pod start begins with an empty Helm cache
• The repo index cache (/helm-working-dir/repository/<repo>-index.yaml) does not exist yet
• When multiple concurrent helm dep build calls target the same chart repo, they all race to fetch and write the same index.yaml file
• Concurrent writes corrupt the index, resulting in empty index.yaml file or unexpected end of stream errors

This is the concurrency bug described in argoproj/argo-cd#12902.

Setting useEphemeralHelmWorkingDir: false removes the emptyDir mount and env var overrides. Helm falls back to its default paths on the container's writable layer. The index is populated once on first fetch and reused for subsequent requests within the same pod lifecycle. This does not add persistence across pod restarts (the container layer is equally ephemeral), but it avoids the race condition because the repo-server's manifestGenerateLock serializes access per chart path.

For a permanent ArgoCD instance (long-lived pod, warm cache) this matters less. For the ephemeral setup it is a meaningful improvement. Set useEphemeralHelmWorkingDir: false in the argocd-values.yaml passed to the tool.

[TBeijen]

Similar to comment you linked, setting seEphemeralHelmWorkingDir: false fixed the issue for us.

Looks like the better default for argocd in KinD, so created #431

Also removes the need for bolt-on retry mechanisms (is complexity), so looks like this issue can be resolved if the PR is ok.