Add helmForceUpdate config option

[krzysztofczyz-da]

Fixes #4940

Added a helmForceUpdate config option (true by default).
Tested on scratchneta.

Test

1. Manually add an annotation to a deamonset (fluentbit)
2. run cncluster pulumi cluster up
3. The annotation disappears
4. readd the annotation
5. change the helmForceUpdate setting to false
6. run cncluster pulumi cluster up
7. the annotation stays on the resource

[martinflorian-da]

Thanks!

--force should not be used normally (as replace can nuke your apps' availablity, particulary damaging if done for a slow starting app)

Hmm so my understanding is that since we chain this after Pulumi, a Helm upgrade is (still) only triggered is something actually changed in the Pulumi state. And then you'd get a restart, true, but AFAICT we almost always get a restart when we change things even without force... I'd say let's just test it by merging and if it causes problems we can revert before it reaches MainNet. I.e., default the config thing to true.

I assume most use cases will be applied from localhost.

So the main need is actually the long-running clusters (CILR, DevNet, TestNet, MainNet). These are managed by the operator, so AFAIU we'd need to deploy the operator once to enable this "feature"... and then it's enabled for all help operations.

env var

We're trying to reduce the number of weird env vars; can we add this to the config.yaml schema instead?

And let's do a CI scratchnet test before merging, maybe an upgrade test as this actually changes many things that are already deployed.

[martinflorian-da]

I actually don't think we need that extra cncluster feature tbh; assuming you add this to config.yaml, the "edit config.yaml and rerun" is a quite established flow by now.

[martinflorian-da]

I also suggest doing a basic "does it do what it should" test if you haven't already. If you're quick, you could just kubectl edit some deployment while the CI upgrade test is running (but before the upgrade step) to add some extra env var, and then confirm that the operator removed your manual changes again on the upgrade step.

[krzysztofczyz-da]

Hmm so my understanding is that since we chain this after Pulumi, a Helm upgrade is (still) only triggered is something actually changed in the Pulumi state. And then you'd get a restart, true, but AFAICT we almost always get a restart when we change things even without force...

Restart != replace - i.e. imagine you have a change which cannot get applied to a statefulset. a replace would kill it and create a new one (which may or may not work). A restart would roll out the change pod-by-pod.
Note that a replace would only happen if necessary (kinda rare tbh.).

Or to quote helm upgrade --help:

      --force                                      force resource updates through a replacement strategy

So the main need is actually the long-running clusters (CILR, DevNet, TestNet, MainNet). These are managed by the operator, so AFAIU we'd need to deploy the operator once to enable this "feature"... and then it's enabled for all help operations.

This changes things - we won't need the cncluster thing then - removed 🫡

We're trying to reduce the number of weird env vars; can we add this to the config.yaml schema instead?

For sure, happily changed 😄

And let's do a CI scratchnet test before merging, maybe an upgrade test as this actually changes many things that are already deployed.

Lemme trigger that

I also suggest doing a basic "does it do what it should" test if you haven't already. If you're quick, you could just kubectl edit some deployment while the CI upgrade test is running (but before the upgrade step) to add some extra env var, and then confirm that the operator removed your manual changes again on the upgrade step.

Done - added to description 👌

[krzysztofczyz-da]

/upgrade_test

[github-actions]

Deploy upgrade pipeline triggered for Commit 01f150020a369b675958353ca6bea52aceb39b54 in , please contact a Contributor to approve it in CircleCI: https://app.circleci.com/pipelines/github/DACH-NY/canton-network-internal/72616

[krzysztofczyz-da]

Upgrade unfortunately failed due to GCP quota 😿

[martinflorian-da]

@krzysztofczyz-da AFAIU we should be in an OK place again w.r.t. scaling up scratchnet clusters... want to give this another try?