Merge branch 'add-version-1.16' into 'main'

Add version 1.16

See merge request ongresinc/stackgres!1631

Author: teoincontatto

.gitlab/issue_templates/Beta Release.md

@@ -50,7 +50,7 @@ sh stackgres-k8s/ci/utils/generate-release-template.sh $VERSION
      ```
      # Remember to create a key pair called gitlab-runner in the target AWS region
      AMI="$(aws ssm get-parameters --names /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-arm64-gp2 --query "Parameters[].Value" --output text)"
-     curl -s https://gitlab.com/snippets/1985684/raw | bash -s -- -r "" -t m6gd.4xlarge -i "" -d 14400 -df internal -dp /dev/nvme1n1 -rn army-builder -tl 'docker-junit-extension-runner, oci-image, ongresinc, stackgres-maven-runner, stackgres-native-build-runner, stackgres-quarkus-test-runner, stackgres-runner-v2, linux-arm64, stackgres-e2e-runner'
+     curl -s https://gitlab.com/snippets/1985684/raw | bash -s -- -r "glpat-CrykSN35Lymqo1UCJDUx" -t m6gd.4xlarge -i "" -d 14400 -df internal -dp /dev/nvme1n1 -rn army-builder -tl 'docker-junit-extension-runner, oci-image, ongresinc, stackgres-maven-runner, stackgres-native-build-runner, stackgres-quarkus-test-runner, stackgres-runner-v2, linux-arm64, stackgres-e2e-runner'
      ```
 
      Now we can push `release-1.15.0-beta1` branch and wait for the pipeline to complete:

.gitlab/issue_templates/GA Release.md

@@ -50,7 +50,7 @@ sh stackgres-k8s/ci/utils/generate-release-template.sh $VERSION
      ```
      # Remember to create a key pair called gitlab-runner in the target AWS region
      AMI="$(aws ssm get-parameters --names /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-arm64-gp2 --query "Parameters[].Value" --output text)"
-     curl -s https://gitlab.com/snippets/1985684/raw | bash -s -- -r "" -t m6gd.4xlarge -i "" -d 14400 -df internal -dp /dev/nvme1n1 -rn army-builder -tl 'docker-junit-extension-runner, oci-image, ongresinc, stackgres-maven-runner, stackgres-native-build-runner, stackgres-quarkus-test-runner, stackgres-runner-v2, linux-arm64, stackgres-e2e-runner'
+     curl -s https://gitlab.com/snippets/1985684/raw | bash -s -- -r "glpat-CrykSN35Lymqo1UCJDUx" -t m6gd.4xlarge -i "" -d 14400 -df internal -dp /dev/nvme1n1 -rn army-builder -tl 'docker-junit-extension-runner, oci-image, ongresinc, stackgres-maven-runner, stackgres-native-build-runner, stackgres-quarkus-test-runner, stackgres-runner-v2, linux-arm64, stackgres-e2e-runner'
      ```
 
      Now we can push `release-1.15.0` branch and wait for the pipeline to complete:

.gitlab/issue_templates/RC Release.md

@@ -1,3 +1,63 @@
+# :rocket: Release 1.15.0-rc1 (2024-12-04)
+
+## :notepad_spiral: NOTES
+
+StackGres 1.15.0-rc1 is (finally) out! :confetti_ball: :champagne: 
+
+This release brings support for Patroni 4 that include some cool features like "Quorum-based failover" or "Register Citus secondaries in pg_dist_node" but also many bugfixes.
+
+Another important change is the new SGDistributedLogs that after the upgrade will generate an SGCluster. This change allow to use SGDbOps in order to perform restart, security upgrade, minor version and major version upgrades operations.
+
+Performance and responsiveness have been improved by allowing to reconcile StackGres custom resources in parallel with a queue that gives priority to reconciliation of StackGres custom resources that receive changes. This should improve the use cases with many StackGres custom resources.
+
+So, what you are waiting for to try this release and have a look to the future of StackGres! 
+
+## :sparkles: NEW FEATURES AND CHANGES
+
+* Support for Patroni 4.0.4
+* Support for Postgres 17.2, 16.6, 15.10, 14.15, 13.18 and 12.22
+* Envoy 1.32.1
+* Prometheus Postgres Exporter 0.16.0
+* Fluent-bit 3.2.1
+* OTEL Collector 0.114.0
+* Make SGDistributedLogs generate an SGCluster
+* Support for headless service
+* Improve performance and responsiveness
+* Allow to disable Envoy
+
+## Web Console
+
+Nothing new here! :eyes:
+
+## :bug: FIXES
+
+* Make int-or-string fields real int-or-string and not just string fields
+* Allow to rollback upgrade when CHECKPOINT fail
+* SGScript and SGCluster script entry version are resetted to 0 if part of a reconciliation of another CR
+
+## Web Console
+
+Nothing new here! :eyes:
+
+## :construction: KNOWN ISSUES
+
+* Backups may be restored with inconsistencies when performed with a Postgres instance running on a different architecture ([#1539](https://gitlab.com/ongresinc/stackgres/-/issues/1539))
+
+## :up: UPGRADE
+
+To upgrade from a previous installation of the StackGres operator's helm chart you will have to upgrade the helm chart release.
+ For more detailed information please refer to [our documentation](https://stackgres.io/doc/latest/install/helm/upgrade/#upgrade-operator).
+
+To upgrade StackGres operator's (upgrade only works starting from 1.1 version or above) helm chart issue the following commands (replace namespace and release name if you used something different):
+
+`helm upgrade -n "stackgres" "stackgres-operator" https://stackgres.io/downloads/stackgres-k8s/stackgres/1.15.0-rc1/helm/stackgres-operator.tgz`
+
+> IMPORTANT: This release is incompatible with previous `alpha` or `beta` versions. Upgrading from those versions will require uninstalling completely StackGres including all clusters and StackGres CRDs (those in `stackgres.io` group) first.
+
+Thank you for all the issues created, ideas, and code contributions by the StackGres Community!
+
+## :twisted_rightwards_arrows: [FULL LIST OF COMMITS](https://gitlab.com/ongresinc/stackgres/-/commits/1.15.0-rc1)
+
 # :rocket: Release 1.14.1 (2024-11-25)
 
 ## :notepad_spiral: NOTES

CHANGELOG.md

@@ -8,6 +8,30 @@ showToc: true
 
 =======
 
+## StackGres 1.15.0-rc1
+
+| Component | Versions |
+| ------ | ----------- |
+| PostgreSQL 17 | 17.2, 17.0 |
+| PostgreSQL 16 | 16.6, 16.4, 16.3, 16.2, 16.1, 16.0 |
+| PostgreSQL 15 | 15.10, 15.8, 15.7, 15.6, 15.5, 15.4, 15.3, 15.2, 15.1, 15.0 |
+| PostgreSQL 14 | 14.15, 14.13, 14.12, 14.11, 14.10, 14.9, 14.8, 14.7, 14.6, 14.5, 14.4, ~14.2, 14.1, 14.0~ |
+| PostgreSQL 13 | 13.18, 13.16 13.15, 13.14, 13.13, 13.12, 13.11, 13.10, 13.9, 13.8, 13.7, 13.6, 13.5, 13.4, 13.3, 13.2, 13.1, 13.0 |
+| PostgreSQL 12 | 12.22, 12.20, 12.19, 12.18, 12.17, 12.16, 12.15, 12.14, 12.13, 12.12, 12.11, 12.10, 12.9, 12.8, 12.7, 12.6, 12.5, 12.4, 12.3, 12.2 and 12.1 |
+| PostgreSQL for Babelfish 16 | 16.4, 16.2, 16.1 |
+| PostgreSQL for Babelfish 15 | 15.8, 15.5 |
+| PostgreSQL for Babelfish 14 | 14.11, 14.10, 14.3 |
+| PostgreSQL for Babelfish 13 | 13.9, 13.6 |
+| Patroni | 4.0.4, 3.3.3, 3.3.2, 3.3.0, 3.2.2, 3.0.2, 2.1.4 |
+| WAL-G | 3.0.3, 3.0.2, 3.0.1, 2.0.1, 2.0.0, 1.1 |
+| PgBouncer | 1.23.1 |
+| Postgres Exporter | 0.16.0 |
+| Envoy | 1.32.1 |
+| Fluent-Bit | 3.2.1 |
+| Fluentd | 1.17.1 |
+| Babelfish Compass | 2024.10b |
+| OTEL Collector | 0.114.0 |
+
 ## StackGres 1.14.1
 
 | Component | Versions |

doc/content/en/01-introduction/06-versions/_index.md

@@ -1,6 +1,7 @@
 #!/bin/sh
 PROJECT_PATH="$(realpath "$(dirname "$0")/../../..")"
 CONTAINER_NAME="${CONTAINER_NAME:-buildw-$(printf '%x' "$(date +%s)")}"
+export PATH=/bin:/usr/bin:/usr/local/bin
 env > "/tmp/$CONTAINER_NAME.env"
 IMAGE="${IMAGE:-registry.gitlab.com/ongresinc/stackgres/$(yq -r '.variables.SG_CI_IMAGE_NAME' "$PROJECT_PATH"/.gitlab-ci.yml 2>/dev/null || sed -n 's/ *SG_CI_IMAGE_NAME: "\([^"]\+\)" */\1/p' .gitlab-ci.yml)-$(uname -m | grep -qxF aarch64 && echo arm64 || echo amd64)}"
 # shellcheck disable=SC2046

stackgres-k8s/ci/build/ciw

@@ -1,4 +1,4 @@
 #!/bin/sh
 
 mkdir -p ~/.config/glab-cli
-docker run -u "$(id -u):$(id -g)" -v /etc/passwd:/etc/passwd:ro -v "$HOME":"$HOME":ro -e PROMPT_COMMAND= --rm -ti -v "$(pwd):$(pwd)" -w "$(pwd)" -v "$HOME/.config/glab-cli:$HOME/.config/glab-cli" registry.gitlab.com/gitlab-org/cli glab "$@"
+docker run -u "$(id -u):$(id -g)" -v /etc/passwd:/etc/passwd:ro -v "$HOME":"$HOME":ro -e PROMPT_COMMAND= --rm $([ -t 0 ] && printf %s -t) -i -v "$(pwd):$(pwd)" -w "$(pwd)" -v "$HOME/.config/glab-cli:$HOME/.config/glab-cli" registry.gitlab.com/gitlab-org/cli glab "$@"

stackgres-k8s/ci/utils/glabw

@@ -59,13 +59,19 @@ e2e_test_install() {
   PREVIOUS_OLDEST_PATRONI_IMAGE_POSTGRES_VERSION_WITH_BUILD_VERSION="${PREVIOUS_OLDEST_PATRONI_IMAGE##*-pg}"
   PREVIOUS_OLDEST_PATRONI_IMAGE_POSTGRES_VERSION="${PREVIOUS_OLDEST_PATRONI_IMAGE_POSTGRES_VERSION_WITH_BUILD_VERSION%%-build-*}"
   PREVIOUS_OLDEST_PATRONI_IMAGE_POSTGRES_MAJOR_VERSION="${PREVIOUS_OLDEST_PATRONI_IMAGE_POSTGRES_VERSION_WITH_BUILD_VERSION%%.*}"
+  PREVIOUS_SHARDED_PATRONI_IMAGE="$(printf '%s' "$PREVIOUS_PATRONI_IMAGE" | grep '/patroni\(-ext\)\?:v[0-9.]\+-pg16\.' | head -n 1)"
+  PREVIOUS_SHARDED_PATRONI_IMAGE_POSTGRES_VERSION_WITH_BUILD_VERSION="${PREVIOUS_SHARDED_PATRONI_IMAGE##*-pg}"
+  PREVIOUS_SHARDED_PATRONI_IMAGE_POSTGRES_VERSION="${PREVIOUS_SHARDED_PATRONI_IMAGE_POSTGRES_VERSION_WITH_BUILD_VERSION%%-build-*}"
+  PREVIOUS_SHARDED_PATRONI_IMAGE_POSTGRES_MAJOR_VERSION="${PREVIOUS_SHARDED_PATRONI_IMAGE_POSTGRES_VERSION_WITH_BUILD_VERSION%%.*}"
   PREVIOUS_PATRONI_IMAGE="$(printf '%s' "$PREVIOUS_PATRONI_IMAGE" | tail -n 1)"
   PREVIOUS_PATRONI_IMAGE_POSTGRES_VERSION_WITH_BUILD_VERSION="${PREVIOUS_PATRONI_IMAGE##*-pg}"
   PREVIOUS_PATRONI_IMAGE_POSTGRES_VERSION="${PREVIOUS_PATRONI_IMAGE_POSTGRES_VERSION_WITH_BUILD_VERSION%%-build-*}"
   PREVIOUS_PATRONI_IMAGE_POSTGRES_MAJOR_VERSION="${PREVIOUS_PATRONI_IMAGE_POSTGRES_VERSION_WITH_BUILD_VERSION%%.*}"
   PATRONI_IMAGE="$(get_component_images "$STACKGRES_VERSION")"
   OLDEST_PATRONI_IMAGE="$(printf %s "$PATRONI_IMAGE" | grep '/patroni\(-ext\)\?:v[0-9.]\+-pg'"$PREVIOUS_OLDEST_PATRONI_IMAGE_POSTGRES_VERSION-")"
   OLDEST_PATRONI_IMAGE="$(printf '%s' "$OLDEST_PATRONI_IMAGE" | tail -n 1)"
+  SHARDED_PATRONI_IMAGE="$(printf %s "$PATRONI_IMAGE" | grep '/patroni\(-ext\)\?:v[0-9.]\+-pg'"$PREVIOUS_SHARDED_PATRONI_IMAGE_POSTGRES_VERSION-")"
+  SHARDED_PATRONI_IMAGE="$(printf '%s' "$SHARDED_PATRONI_IMAGE" | tail -n 1)"
   if [ "$PREVIOUS_VERSION_AS_NUMBER" -ge "$VERSION_AS_NUMBER_1_15_0" ]
   then
     DISTRIBUTEDLOGS_PATRONI_IMAGE="$(printf '%s' "$PATRONI_IMAGE" | tail -n 1)"
@@ -154,11 +160,13 @@ e2e_test_install() {
 
     create_or_replace_sharded_cluster_for_version "$STACKGRES_PREVIOUS_VERSION" \
       "$CLUSTER_3_NAME" "$CLUSTER_NAMESPACE" 3 1 \
-      --set configurations.create=false --set instanceProfiles=false \
+      --set configurations.create=true --set instanceProfiles=false \
+      --set configurations.postgresconfig.create=true \
+      --set configurations.poolingconfig.create=false \
       --set-string cluster.sgInstanceProfile=size-s \
-      --set-string cluster.configurations.sgPostgresConfig=postgresconf-"${PREVIOUS_PATRONI_IMAGE_POSTGRES_VERSION%.*}" \
-      --set-string shardedCluster.shards.configurations.sgPostgresConfig=postgresconf-"${PREVIOUS_PATRONI_IMAGE_POSTGRES_VERSION%.*}" \
-      --set-string cluster.postgres.version="$PREVIOUS_PATRONI_IMAGE_POSTGRES_VERSION" \
+      --set-string cluster.configurations.sgPostgresConfig=postgresconf-"${PREVIOUS_SHARDED_PATRONI_IMAGE_POSTGRES_VERSION%.*}" \
+      --set-string shardedCluster.shards.configurations.sgPostgresConfig=postgresconf-"${PREVIOUS_SHARDED_PATRONI_IMAGE_POSTGRES_VERSION%.*}" \
+      --set-string cluster.postgres.version="$PREVIOUS_SHARDED_PATRONI_IMAGE_POSTGRES_VERSION" \
       --set cluster.configurations.observability.prometheusAutobind=true \
       --set configurations.objectstorage.create=false \
       --set-string cluster.configurations.backups.sgObjectStorage=backupconf \
@@ -584,6 +592,8 @@ check_cluster_2_security_upgrade_start() {
 }
 
 check_cluster_3_security_upgrade_start() {
+  local PREVIOUS_PATRONI_IMAGE="$PREVIOUS_SHARDED_PATRONI_IMAGE"
+
   check_sharded_cluster_security_upgrade_start \
     "$CLUSTER_3_NAME" "$DBOPS_3_NAME" "InPlace" \
     "$PREVIOUS_PATRONI_3_CONFIG_PATH" "$PREVIOUS_PGBOUNCER_3_CONFIG_PATH"
@@ -609,6 +619,8 @@ check_cluster_2_security_upgrade() {
 }
 
 check_cluster_3_security_upgrade() {
+  local PATRONI_IMAGE="$SHARDED_PATRONI_IMAGE"
+
   check_sharded_cluster_security_upgrade \
     "$CLUSTER_3_NAME" "$DBOPS_3_NAME" "InPlace"
 }

stackgres-k8s/e2e/spec/dbops-security-upgrade

@@ -42,12 +42,16 @@ e2e_load_images() {
   cat << EOF > "$LOG_PATH/operator-bundle.Dockerfile"
 FROM "$BUNDLE_IMAGE_NAME" AS bundle
 
-FROM alpine AS overwrites
+FROM debian AS overwrites
   COPY --from=bundle / /bundle
   RUN sed -i 's|^  name: .*$|  name: stackgres.v$BUNDLE_STACKGRES_VERSION|' /bundle/manifests/stackgres.clusterserviceversion.yaml
   RUN sed -i 's|^  version: .*$|  version: $BUNDLE_STACKGRES_VERSION|' /bundle/manifests/stackgres.clusterserviceversion.yaml
   RUN sed -i 's|^    containerImage: quay.io/stackgres/operator:.*$|    containerImage: $OPERATOR_IMAGE_NAME|' /bundle/manifests/stackgres.clusterserviceversion.yaml
   RUN sed -i 's|^                    image: quay.io/stackgres/operator:.*$|                    image: $OPERATOR_IMAGE_NAME|' /bundle/manifests/stackgres.clusterserviceversion.yaml
+  RUN sed -i '/name: OPERATOR_VERSION$/{\$!{N;s/name: OPERATOR_VERSION\n\( *\)value: .*$/name: OPERATOR_VERSION\n\1value: "$STACKGRES_VERSION"/;ty;P;D;:y}}' /bundle/manifests/stackgres.clusterserviceversion.yaml
+  RUN sed -i '/name: OPERATOR_IMAGE_VERSION$/{\$!{N;s/name: OPERATOR_IMAGE_VERSION\n\( *\)value: .*$/name: OPERATOR_IMAGE_VERSION\n\1value: "$IMAGE_TAG"/;ty;P;D;:y}}' /bundle/manifests/stackgres.clusterserviceversion.yaml
+  RUN sed -i '/name: OPERATOR_JVM_IMAGE_VERSION$/{\$!{N;s/name: OPERATOR_JVM_IMAGE_VERSION\n\( *\)value: .*$/name: OPERATOR_JVM_IMAGE_VERSION\n\1value: "$NATIVE_IMAGE_TAG-jvm"/;ty;P;D;:y}}' /bundle/manifests/stackgres.clusterserviceversion.yaml
+  RUN sed -i '/name: OPERATOR_NATIVE_IMAGE_VERSION$/{\$!{N;s/name: OPERATOR_NATIVE_IMAGE_VERSION\n\( *\)value: .*$/name: OPERATOR_NATIVE_IMAGE_VERSION\n\1value: "$NATIVE_IMAGE_TAG"/;ty;P;D;:y}}' /bundle/manifests/stackgres.clusterserviceversion.yaml
 
 FROM scratch
   COPY --from=overwrites /bundle /
@@ -57,7 +61,7 @@ $(
       | sed 's/^/  /'
 )
 EOF
-  BUNDLE_IMAGE_NAME="${BUNDLE_IMAGE_NAME}-overridden"
+  BUNDLE_IMAGE_NAME="${BUNDLE_IMAGE_NAME}-overridden-$(date +%s)-$RANDOM"
   docker build -t "$BUNDLE_IMAGE_NAME" -f "$LOG_PATH/operator-bundle.Dockerfile" "$LOG_PATH/operator-bundle" 
   docker push "$BUNDLE_IMAGE_NAME"
 }

stackgres-k8s/e2e/spec/dbops-security-upgrade-olm

@@ -24,10 +24,10 @@ e2e_test() {
   run_test "Check that operator can not be upgrade from an alpha version" check_operator_upgrade_alpha
   run_test "Check that operator can not be upgrade from a beta version" check_operator_upgrade_beta
   run_test "Check that operator can not be upgrade from version 0.9.5" check_operator_upgrade_0_9_5
-  run_test "Check that operator can be upgrade from version 1.13.0" check_operator_upgrade_1_13_0
   run_test "Check that operator can be upgrade from version 1.14.0" check_operator_upgrade_1_14_0
+  run_test "Check that operator can be upgrade from version 1.15.0" check_operator_upgrade_1_15_0
   run_test "Check that operator can not be upgrade with an SGCluster that uses version 0.9.5" check_operator_upgrade_with_cluster_using_0_9_5
-  run_test "Check that operator can be upgrade with an SGCluster that uses version 1.13.0 and 1.14.0" check_operator_upgrade_with_cluster_using_1_13_0_and_1_14_0
+  run_test "Check that operator can be upgrade with an SGCluster that uses version 1.14.0 and 1.15.0" check_operator_upgrade_with_cluster_using_1_14_0_and_1_15_0
   run_test "Check that operator can be deleted" check_operator_delete
   run_test "Check that operator can be installed from outside" check_operator_install_outside
   run_test "Check that operator can be installed with load balancer" check_operator_install_load_balancer
@@ -163,17 +163,17 @@ check_operator_upgrade_0_9_5() {
   fi
 }
 
-check_operator_upgrade_1_13_0() {
-  local PREVIOUS_VERSION=1.13.0
+check_operator_upgrade_1_15_0() {
+  local PREVIOUS_VERSION=1.15.0
   kubectl label deployment -n "$OPERATOR_NAMESPACE" stackgres-operator \
     --overwrite "version=$PREVIOUS_VERSION"
   local EXIT_CODE RESULT
   try_function upgrade_operator
   if "$RESULT"
   then
-    success "Upgrade did not failed when previous version was 1.13.0"
+    success "Upgrade did not failed when previous version was 1.15.0"
   else
-    fail "Upgrade failed when previous version was 1.13.0"
+    fail "Upgrade failed when previous version was 1.15.0"
   fi
 }
 
@@ -214,13 +214,13 @@ check_operator_upgrade_with_cluster_using_0_9_5() {
   remove_cluster "$CLUSTER_NAME" "$CLUSTER_NAMESPACE"
 }
 
-check_operator_upgrade_with_cluster_using_1_13_0_and_1_14_0() {
+check_operator_upgrade_with_cluster_using_1_14_0_and_1_15_0() {
   CLUSTER_1_NAME="$(get_sgcluster_name "$SPEC_NAME-1")"
   CLUSTER_2_NAME="$(get_sgcluster_name "$SPEC_NAME-2")"
   NODE_LABEL_KEY="$(random_string)"
   NODE_LABEL_VALUE="$(random_string)"
-  local PREVIOUS_VERSION_1=1.13.0
-  local PREVIOUS_VERSION_2=1.14.0
+  local PREVIOUS_VERSION_1=1.14.0
+  local PREVIOUS_VERSION_2=1.15.0
   create_or_replace_cluster "$CLUSTER_NAME" "$CLUSTER_NAMESPACE" 1 \
     --set cluster.create=false \
     --set-string cluster.postgres.version=13.9
@@ -246,9 +246,9 @@ check_operator_upgrade_with_cluster_using_1_13_0_and_1_14_0() {
   try_function upgrade_operator
   if "$RESULT"
   then
-    success "Upgrade did not failed with an SGCluster that uses version 1.13.0 and 1.14.0"
+    success "Upgrade did not failed with an SGCluster that uses version 1.14.0 and 1.15.0"
   else
-    fail "Upgrade failed with an SGCluster that uses version 1.13.0 and 1.14.0"
+    fail "Upgrade failed with an SGCluster that uses version 1.14.0 and 1.15.0"
   fi
   remove_cluster "$CLUSTER_1_NAME" "$CLUSTER_NAMESPACE"
   remove_cluster "$CLUSTER_2_NAME" "$CLUSTER_NAMESPACE"