fix(switchover): recreate primary when WAL-archiver sidecar is missing (cloudnative-pg/cloudnative-pg#11032)

Enabling a WAL-archiver plugin on an existing cluster leaves the primary
Pod without the injected sidecar, so it cannot archive WAL. When
`primaryUpdateMethod` is set to `switchover`, the operator rolls the
primary out by demoting it, but a clean demotion archives all pending
WAL before pg_rewind, and that archiving needs the very sidecar that is
still missing. The switchover never completes, the primary is marked
unhealthy and dropped from the -rw service, and the cluster dead-locks.

Detect this case in `updatePrimaryPod` and recreate the primary Pod in
place instead of switching over. It comes back as primary with the
sidecar, archiving resumes, and no demotion or rewind is involved.

Detecting the missing sidecar from instance status during reconciliation
replaces the previous mechanism, where the instance manager exited with
a dedicated code and the controller deleted the Pod, so that code is
removed.

Closes cloudnative-pg/cloudnative-pg#10981
Closes cloudnative-pg/plugin-barman-cloud#901

Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
Co-authored-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>

(cherry picked from commit cloudnative-pg/cloudnative-pg@6b275c3)

Author: mnencia

pkg/api/v1/cluster_types.go

@@ -119,11 +119,6 @@ const (
 	// MissingWALDiskSpaceExitCode is the exit code the instance manager
 	// will use to signal that there's no more WAL disk space
 	MissingWALDiskSpaceExitCode = 4
-
-	// MissingWALArchivePlugin is the exit code used by the instance manager
-	// to indicate that it started successfully, but the configured WAL
-	// archiving plugin is not available.
-	MissingWALArchivePlugin = 5
 )
 
 // SnapshotOwnerReference defines the reference type for the owner of the snapshot.