Improved perfromance helm execution

[fm1ck3y]

Summary

Hi ArgoCD Team!

This issue desccussion to:

• Concurrent issue with helm dependency build in argocd-repo-server #12902
• Concurrent issue with `helm dependency build` in argocd-repo-server #12877
• Helm concurrency Issue in Helm v3.7.2 helm/helm#10735
• index concurrency issue helm/helm#30983

We are implementing ArgoCD as a deployer in our company. We chose helm http repository instead of OCI because::

• OCI does not support wildcard in dependencies
• OCI does not support alias in dependencies
• OCI use only direct link

We have 2 regions, and one proxy link that redirects to the nearest region.
When executing helm dependency build & helm pull, we encounter 2 problems:

1. Network Issue

Description:

• We have 100 clusters, each running ArgoCD
• Each ArgoCD instance has between 300 to 800 applications
• Using reposerver.parallelism.limit: '30'
• Each ArgoCD instance uses a common helm http repository
• index.yaml file size is 40MB

Based on these parameters, we can derive the following formula for maximum network load:

Max Network Load = (Number of Clusters) × (index.yaml size) × (Number of Parallel Requests)

Where:
- Number of Clusters = 100
- Max Applications per Cluster = 800
- index.yaml size = 40MB
- Number of Parallel Requests = min(reposerver.parallelism.limit, Max Applications per Cluster) = min(30, 800) = 30

Therefore:
Max Network Load = 100 × 40MB × 30 = 120,000MB ~= 118GB

2. High RAM Usage Issue

Description:

• Using reposerver.parallelism.limit: '30'
• Through experimental testing, it was found that helm pull consumes ~900MB for a repository with an index.yaml file of 40MB.

We encountered a performance issue with the repo-server when the helm repository (http) is too large.
Helm is required to download the index for the helm pull command. With an index.yaml size of 40 MB in the helm repository, a single helm pull consumes ~900mb.

There are related tickets in Helm:

• Helm runs out of memory parsing large index.yaml files helm/helm#9931
• index.json - necessary for large projects helm/helm#10542

They mention that helm will consume less if the index is stored in .json format. In any case, when the helm repository is large enough, it significantly loads the repo-server.

Based on these parameters, we can derive a formula for the load on argocd-repo-server purely from helm execution:

Max RAM Usage Repo Server = (Helm RAM Usage) * (Number of Parallel Requests) + (argocd-repo-server RAM usage)

Where 
 - Number of Parallel Requests = 30
 - Helm RAM Usage = 900MB
 - argocd-repo-server RAM usage = ~500MB

Therefore: 
Max Ram Usage Repo Server = 30 * 900 + 500 = 27,500 MB ~= 27GB

Motivation

settings of argocd-cmd-params-cm:

reposerver.parallelism.limit: '30'
reposerver.exec.timeout: '180s'
controller.operation.processors: '10'
controller.status.processors: '20'

Number of replicas: in this case, it doesn't matter since we're looking at the maximum load per pod.

Example of helm pull execution for a chart:

time="2025-05-22T05:21:48Z" level=info msg=Trace args="[helm pull --destination /tmp/67aa0bad-e931-4c59-8b58-715bfdeac363 --version 0.1.0 --repo https://<registry>/helm.charts my-app]" dir= operation_name="exec helm" time_ms=90285.762594

From the logs, we can see that the operation takes 90s, which is quite long for a helm pull.

Proposal

1. Direct Helm Pull

We have an idea to add the ability to enable helm pull via direct URL, bypassing the index.yaml file, as a parameter in the argocd-cmd-params-cm ConfigMap and name it, for example: reposerver.helm.pull.direct.

helm pull --destination /tmp/67aa0bad-e931-4c59-8b58-715bfdeac363 --version 0.1.0 --repo https://<registry>/helm.charts my-app

->

helm pull --destination /tmp/67aa0bad-e931-4c59-8b58-715bfdeac363 https://<registry>/helm.charts/my-app-0.1.0.tgz

Such downloading takes only 1s and consumes almost no resources. At the same time, helm dependency build & helm repo update will work normally if needed.

Conclusions

• This will reduce the load on the network because helm will not download the index.yaml file
• This will help with RAM usage because helm will not search & load index.yaml into memory

2. Using Local Cache

Local Cache Within Container

In argocd-repo-server already use shared helm cache between containers. There is another problem associated with this:

• index concurrency issue helm/helm#30983
• Concurrent issue with helm dependency build in argocd-repo-server #12902

This mode assumes a separate thread that makes a helm repo update every n seconds. In this case helm dependency build will be executed with --skip-refresh flag.
In this case the load on argocd-repo-server will be reduced many times, but there will be more pronounced helm concurrency problem.
This can be solved by using a lock on any helm dependency build --skip-refresh when a helm repo update is performed.

It is suggested to make an additional parameter to enable such a mode in argocd-cmd-params-cm ConfigMap and name it, for example: reposerver.helm.background.sync.index.

Risks:

• The problem occurs with helm concurrency , when a lot of application tries to use cache and at the same time cache from another process is updated.

Local cache using local helm repository

For example: use a chartmuseum as a proxy server that will keep the main load from the main repository, and give a minified index.yaml file to argocd-repo-server

Conclusions

• This will greatly reduce the load on the network because the download index.yaml will be done per cluster instead of per ArgoCD
• This will reduce the load on RAM, because the index.yaml file will be much smaller

[crenshaw-dev]

The direct pull option seems too good to be true... are there downsides? Like, do you get a worse error message if the version is invalid or something like that?

[fm1ck3y]

The direct pull option seems too good to be true... are there downsides? Like, do you get a worse error message if the version is invalid or something like that?

Yes, there is actually a big disadvantage that I haven't described: the inability to support the wildcard version
Since the link is from a chart and version, the ability to support wildcard becomes impossible (at least specifically for the case of performance enhancement)
Also, as seen in PR version becomes a must for this type of pull. I think this should be described separately in the documentation

[crenshaw-dev]

Do we rely on Helm to process the wildcard, or does Argo CD do that?

Either way, is there a way to reliably detect globs and then fall back to the index.yaml behavior?

[fm1ck3y]

I think we should always rely on Helm for wildcard handling, there is no point in implementing the logic in ArgoCD.
We can check if the version is a wildcard, if it is we can use standard pull, otherwise direct. Let me try to implement this logic in PR.

But if you mean to try standard pull in case of failed pull, then I'm afraid it will lead to high load in case the wildcard is not found.

[agaudreault]

What if multiple registry are necessary? Currently, the proposal seems to only support one configured via reposerver.helm.pull.direct/

[fm1ck3y]

What if multiple registry are necessary? Currently, the proposal seems to only support one configured via reposerver.helm.pull.direct/

Sorry, I may have misunderstood your question. What do you mean by saying it won't work for multiple registries?
reposerver.helm.pull.direct is the switch that enables this graph pull mode.
Any repository definition occurs in Application CR or as a helm repository entity in ArgoCD. We're just changing the current behavioral pull

[fm1ck3y]

Another limitation of the current direct pull proposal is that it doesn't work for Helm repositories that store their charts externally — e.g., using GitHub Releases. In such cases, the actual .tgz URL is only discoverable via index.yaml.
Maybe that's what he was talking about @agaudreault
A good example is the argo-cd chart from Argo Helm:

https://argoproj.github.io/argo-helm/index.yaml

argo-cd:  
- annotations:
      artifacthub.io/changes: |
        - kind: changed
          description: Bump argo-cd to v3.0.6
      artifacthub.io/signKey: |
        fingerprint: 2B8F22F57260EFA67BE1C5824B11F800CD9D2252
        url: https://argoproj.github.io/argo-helm/pgp_keys.asc
    apiVersion: v2
    appVersion: v3.0.6
    created: "2025-06-10T09:15:38.932846603Z"
    dependencies:
    - condition: redis-ha.enabled
      name: redis-ha
      repository: https://dandydeveloper.github.io/charts/
      version: 4.33.7
    description: A Helm chart for Argo CD, a declarative, GitOps continuous delivery
      tool for Kubernetes.
    digest: 98636496ff072876aec71e91e34f79907c5faa2bec72910e1b84cf366ebec5c4
    home: https://github.com/argoproj/argo-helm
    icon: https://argo-cd.readthedocs.io/en/stable/assets/logo.png
    keywords:
    - argoproj
    - argocd
    - gitops
    kubeVersion: '>=1.25.0-0'
    maintainers:
    - name: argoproj
      url: https://argoproj.github.io/
    name: argo-cd
    sources:
    - https://github.com/argoproj/argo-helm/tree/main/charts/argo-cd
    - https://github.com/argoproj/argo-cd
    urls:
    - https://github.com/argoproj/argo-helm/releases/download/argo-cd-8.0.17/argo-cd-8.0.17.tgz
    version: 8.0.17

Do you think we could add a modified helm chart format as URL in the Application CR? Or is that too much?

[agaudreault]

I was more talking about the configuration flag being global for all Helm charts and not configurable "per registry". Maybe within one instance, I used a registry with a large index, but maybe I also use another registry that has a relatively small index (and maybe globs are used for the version in that one). In this case, I dont want to attempt a direct download.

But what you just found is valuable too. In that case, I would not want the "argoproj.github.io/argo-helm" registry to use direct downloads.

[fm1ck3y]

I have two ideas that might improve the flexibility of direct pull support:

1. Per-repository configuration (like with Helm OCI)
   We could introduce a setting at the repository configuration level, similar to how OCI support is toggled. This way, users can opt in or out of direct pull for specific Helm repositories — depending on their structure, index size, or compatibility.

2. Optional override in the Application CR
   We could introduce a second way to enable direct chart access via an explicit chartURL field inside the Application CR.

source:
  chartURL: https://bitnami-labs.github.io/sealed-secrets/sealed-secrets-1.16.1.tgz
  helm:
    releaseName: sealed-secrets

This way, users could completely bypass the registry/index resolution and reference the chart archive directly.

[ashish-b-choudhary-db]

Direct pull support is much needed. In our organization we have index file upwards of 200 MB and the memory requirements of helm pull are ridiculously high.

[agaudreault]

@fm1ck3y I think let's implement 1) first, and if there is really a usecase for 2), we can implement it in a second PR.

Probably option 2 would be more of a feature flag, but let's see if there are any use cases that 1) does not address.

source:
  chart: sealed-secrets
  repoURL: https://bitnami-labs.github.io/sealed-secrets
  targetRevision: 1.16.1
  helm:
    directPull: true

[fm1ck3y]

Hi @agaudreault
I have created 2 PRs:

1. Direct Pull Per-repository configuration: feat(helm): add helm direct pull for reposerver #23403
2. Direct Pull as option in Application CR: feat(helm): add helm direct pull as option in Application CR #23871

it seems to me that the second option is more flexible, as there can be different types of charts in the same repository:

1. that are stored at the expected URL
2. that are stored as specified in sources

Switch in Application CR, will allow you to manage a specific source rather than the entire repository..
but this is just a thought