Takeover e2e tests#1141
Open
faizanahmad055 wants to merge 30 commits into
Open
Conversation
…or pod annotations
Signed-off-by: faizanahmad055 <faizan.ahmad55@outlook.com>
Signed-off-by: faizanahmad055 <faizan.ahmad55@outlook.com>
Signed-off-by: faizanahmad055 <faizan.ahmad55@outlook.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
This PR takes over and introduces a revamped end-to-end (e2e) testing setup centered around Ginkgo v2, adds supporting utilities (watch-based waiting, workload adapters, test environment/Helm helpers), and wires e2e execution into the CI workflow alongside several small refactors/cleanups in production code.
Changes:
- Added a comprehensive Ginkgo v2 e2e test suite with shared utilities (watch helpers, workload adapter registry, Helm deploy/cleanup, cluster capability detection).
- Added Makefile targets and scripts to set up/run/clean up e2e infrastructure (Kind + Argo Rollouts + CSI driver + Vault), and updated PR CI workflow to run e2e.
- Refactored/cleaned up various internal packages (controller initialization guards, handler config extraction, import ordering, small naming tweaks) and introduced golangci-lint configuration.
Reviewed changes
Copilot reviewed 84 out of 90 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| test/loadtest/internal/cmd/run.go | Import ordering / whitespace cleanup in loadtest command. |
| test/loadtest/internal/cmd/report.go | Formatting/alignment tweaks in report structs. |
| test/e2e/utils/workload_statefulset.go | New StatefulSet workload adapter for e2e suite. |
| test/e2e/utils/workload_openshift.go | New OpenShift DeploymentConfig adapter for e2e suite. |
| test/e2e/utils/workload_job.go | New Job adapter including recreation (UID) watch support. |
| test/e2e/utils/workload_deployment.go | New Deployment adapter including pause/unpause watchers. |
| test/e2e/utils/workload_daemonset.go | New DaemonSet adapter for e2e suite. |
| test/e2e/utils/workload_cronjob.go | New CronJob adapter including triggered Job detection. |
| test/e2e/utils/workload_argo.go | New Argo Rollout adapter including restartAt watch support. |
| test/e2e/utils/workload_adapter.go | New cross-workload adapter interfaces + registry. |
| test/e2e/utils/watch.go | New generic watch-based waiting primitives for e2e. |
| test/e2e/utils/utils.go | New command execution + project dir + kubeconfig helpers. |
| test/e2e/utils/testenv.go | New shared test environment setup/cleanup and deploy helpers. |
| test/e2e/utils/test_helpers.go | New annotation merge helper. |
| test/e2e/utils/test_helpers_test.go | Unit tests for annotation merge helper. |
| test/e2e/utils/rand.go | New random name utilities for test resource uniqueness. |
| test/e2e/utils/rand_test.go | Unit tests for random name utilities. |
| test/e2e/utils/openshift.go | New OpenShift capability detection helper. |
| test/e2e/utils/helm.go | New Helm deploy/undeploy helpers used by e2e suites. |
| test/e2e/utils/helm_test.go | Unit tests for Helm helper parsing/utilities. |
| test/e2e/utils/conditions.go | New reusable watch conditions (annotations, env vars, UID, SPCPS). |
| test/e2e/utils/argo.go | New Argo Rollout creation helpers and install checks. |
| test/e2e/utils/annotations.go | New constants/builders for Reloader annotations used by tests. |
| test/e2e/utils/accessors.go | New typed accessors to support generic watch conditions. |
| test/e2e/README.md | New e2e documentation (usage, labels, structure, examples). |
| test/e2e/flags/watch_globally_test.go | New e2e tests for watchGlobally flag behavior. |
| test/e2e/flags/resource_selector_test.go | New e2e tests for resourceLabelSelector behavior. |
| test/e2e/flags/reload_on_delete_test.go | New e2e tests for reloadOnDelete behavior. |
| test/e2e/flags/reload_on_create_test.go | New e2e tests for reloadOnCreate behavior. |
| test/e2e/flags/namespace_selector_test.go | New e2e tests for namespaceSelector behavior. |
| test/e2e/flags/namespace_ignore_test.go | New e2e tests for ignoreNamespaces behavior. |
| test/e2e/flags/ignored_workloads_test.go | New e2e tests for ignoreCronJobs/ignoreJobs behavior. |
| test/e2e/flags/ignore_resources_test.go | New e2e tests for ignoreSecrets/ignoreConfigMaps behavior. |
| test/e2e/flags/flags_suite_test.go | New Ginkgo suite bootstrap for flags tests. |
| test/e2e/flags/auto_reload_all_test.go | New e2e tests for autoReloadAll flag behavior. |
| test/e2e/csi/csi_suite_test.go | New CSI suite bootstrap with cluster capability checks. |
| test/e2e/core/core_suite_test.go | New core suite bootstrap + dynamic adapter registration. |
| test/e2e/argo/rollout_test.go | New Argo-specific e2e tests for rollout-strategy behaviors. |
| test/e2e/argo/argo_suite_test.go | New Argo suite bootstrap with install gating. |
| test/e2e/annotations/search_match_test.go | New e2e tests for search/match annotations (incl. pending cases). |
| test/e2e/annotations/resource_ignore_test.go | New e2e tests for ignore annotation on resources. |
| test/e2e/annotations/pause_period_test.go | New e2e tests for pause-period behavior (incl. pending case). |
| test/e2e/annotations/annotations_suite_test.go | New annotations suite bootstrap + dynamic adapter registration. |
| test/e2e/advanced/regex_test.go | New e2e tests for regex pattern behaviors. |
| test/e2e/advanced/multi_container_test.go | New e2e tests for multi-container and init-container CSI cases. |
| test/e2e/advanced/advanced_suite_test.go | New advanced suite bootstrap. |
| scripts/e2e-cluster-cleanup.sh | New cleanup script for Kind-installed e2e dependencies/resources. |
| pkg/kube/client.go | Comment spacing cleanup in kube config selection. |
| pkg/common/config.go | Import order/grouping cleanup. |
| pkg/common/common.go | Refactor ShouldReload param naming + import grouping. |
| Makefile | Added e2e targets; adjusted lint/test; added fmt and docker-build targets. |
| internal/pkg/util/util.go | Import order/grouping cleanup. |
| internal/pkg/util/util_test.go | Import order/grouping cleanup. |
| internal/pkg/util/interface.go | Adjusted type assertions in ToObjectMeta/ParseBool. |
| internal/pkg/testutil/kube.go | Import regrouping. |
| internal/pkg/leadership/leadership.go | Minor loop/goroutine setup change in controller runner. |
| internal/pkg/leadership/leadership_test.go | Import spacing tweak. |
| internal/pkg/handler/upgrade.go | Import regrouping + minor variable/comment cleanup. |
| internal/pkg/handler/update.go | Import order/grouping cleanup. |
| internal/pkg/handler/pause_deployment.go | Import regrouping. |
| internal/pkg/handler/pause_deployment_test.go | Import regrouping + error wrapping fix. |
| internal/pkg/handler/handlers_test.go | New unit tests for handler config derivation/handling. |
| internal/pkg/handler/delete.go | Small refactor in resource type checks + comment spacing. |
| internal/pkg/handler/create.go | Small refactor in resource type checks. |
| internal/pkg/controller/controller.go | Atomic init guards + safer handler type assertion; import regrouping. |
| internal/pkg/app/app.go | Renamed local var to avoid package name shadowing. |
| .golangci.yml | New golangci-lint configuration (linters + formatters). |
| .gitignore | Ignore *.test artifacts. |
| .github/workflows/pull_request.yaml | Bumped Kind version; added e2e dependency setup + e2e run steps. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Signed-off-by: faizanahmad055 <faizan.ahmad55@outlook.com>
Comment on lines
+28
to
+32
| { | ||
| name: "image with digest (not fully supported)", | ||
| image: "nginx@sha256:abc123", | ||
| expected: "nginx@sha256", | ||
| }, |
Comment on lines
+45
to
51
| // controllerInitialized flags guard against processing Add/Delete events before | ||
| // the worker goroutines have started. Written by runWorker (in a goroutine) and | ||
| // read by the informer event handlers, so they must be atomic. | ||
| var secretControllerInitialized atomic.Bool | ||
| var configmapControllerInitialized atomic.Bool | ||
| var selectedNamespacesCache []string | ||
|
|
Comment on lines
+53
to
+54
| func NewController(client kubernetes.Interface, resource string, namespace string, ignoredNamespaces []string, namespaceLabelSelector string, resourceLabelSelector string, collectors metrics.Collectors) (*Controller, | ||
| error) { |
Comment on lines
+42
to
+59
| // HasPodTemplateAnnotationChanged returns a condition that checks the pod template annotation | ||
| // is present AND its value differs from priorValue. If priorValue is empty, any non-empty value | ||
| // satisfies the condition (equivalent to HasPodTemplateAnnotation). | ||
| // Use this in WaitReloaded to correctly detect a reload after a prior reload has already set the annotation. | ||
| func HasPodTemplateAnnotationChanged[T any](accessor PodTemplateAccessor[T], key, priorValue string) Condition[T] { | ||
| return func(obj T) bool { | ||
| template := accessor(obj) | ||
| if template == nil || template.Annotations == nil { | ||
| return false | ||
| } | ||
| v, ok := template.Annotations[key] | ||
| if !ok { | ||
| return false | ||
| } | ||
| if priorValue == "" { | ||
| return true | ||
| } | ||
| return v != priorValue |
Comment on lines
+102
to
+105
| By("Verifying Deployment was NOT reloaded (different namespace with watchGlobally=false)") | ||
| time.Sleep(utils.NegativeTestWait) | ||
| reloaded, err := adapter.WaitReloaded(ctx, otherNS, deploymentName, | ||
| utils.AnnotationLastReloadedFrom, utils.ShortTimeout) |
Signed-off-by: faizanahmad055 <faizan.ahmad55@outlook.com>
| err := resourceHandler.(handler.ResourceHandler).Handle() | ||
| rh, ok := resourceHandler.(handler.ResourceHandler) | ||
| if !ok { | ||
| logrus.Errorf("Invalid resource handler type: %T", resourceHandler) |
Comment on lines
+165
to
+173
| $(CONTAINER_RUNTIME) build -t $(E2E_IMG) -f Dockerfile . | ||
| ifeq ($(notdir $(CONTAINER_RUNTIME)),podman) | ||
| $(CONTAINER_RUNTIME) save $(E2E_IMG) -o /tmp/reloader-e2e.tar | ||
| kind load image-archive /tmp/reloader-e2e.tar --name $(KIND_CLUSTER) | ||
| rm -f /tmp/reloader-e2e.tar | ||
| else | ||
| kind load docker-image $(E2E_IMG) --name $(KIND_CLUSTER) | ||
| endif | ||
| SKIP_BUILD=true RELOADER_IMAGE=$(E2E_IMG) "$(GOCMD)" tool ginkgo --keep-going -v --timeout=$(E2E_TIMEOUT) ./test/e2e/... |
| ./scripts/e2e-cluster-setup.sh | ||
|
|
||
| .PHONY: e2e | ||
| e2e: ## Run e2e tests (builds image, loads to Kind, runs tests in parallel) |
✅ Load Test Results (quick)✅ Load Test Results: ALL TESTS PASSED
3/3 passed (100%)
Artifacts: Download |
Signed-off-by: faizanahmad055 <faizan.ahmad55@outlook.com>
Signed-off-by: faizanahmad055 <faizan.ahmad55@outlook.com>
Comment on lines
27
to
34
| func ParseBool(value interface{}) bool { | ||
| if reflect.Bool == reflect.TypeOf(value).Kind() { | ||
| return value.(bool) | ||
| b, _ := value.(bool) | ||
| return b | ||
| } else if reflect.String == reflect.TypeOf(value).Kind() { | ||
| result, _ := strconv.ParseBool(value.(string)) | ||
| s, _ := value.(string) | ||
| result, _ := strconv.ParseBool(s) | ||
| return result |
Comment on lines
+16
to
+19
| ignoreNS string | ||
| cronJobAdapter *utils.CronJobAdapter | ||
| deploymentAdater *utils.DeploymentAdapter | ||
| ) |
Comment on lines
+102
to
+106
| By("Verifying Deployment was NOT reloaded (different namespace with watchGlobally=false)") | ||
| time.Sleep(utils.NegativeTestWait) | ||
| reloaded, err := adapter.WaitReloaded(ctx, otherNS, deploymentName, | ||
| utils.AnnotationLastReloadedFrom, utils.ShortTimeout) | ||
| Expect(err).NotTo(HaveOccurred()) |
Comment on lines
+45
to
+50
| | `RELOADER_IMAGE` | `ghcr.io/stakater/reloader:test` | Image to test | | ||
| | `SKIP_BUILD` | `false` | Skip the container image build and Kind load steps; requires `RELOADER_IMAGE` to point to an already-loaded image | | ||
| | `KIND_CLUSTER` | `reloader-e2e` | Kind cluster name | | ||
| | `E2E_TIMEOUT` | `45m` | Test timeout | | ||
| | `GINKGO_PROCS` | `4` | Number of parallel Ginkgo worker processes | | ||
|
|
Signed-off-by: faizanahmad055 <faizan.ahmad55@outlook.com>
Signed-off-by: faizanahmad055 <faizan.ahmad55@outlook.com>
✅ Load Test Results (quick)✅ Load Test Results: ALL TESTS PASSED
3/3 passed (100%)
Artifacts: Download |
✅ Load Test Results (quick)✅ Load Test Results: ALL TESTS PASSED
3/3 passed (100%)
Artifacts: Download |
Signed-off-by: faizanahmad055 <faizan.ahmad55@outlook.com>
✅ Load Test Results (quick)❌ Load Test Results: 1 TEST(S) FAILED
2/3 passed (66%)
Artifacts: Download |
Signed-off-by: faizanahmad055 <faizan.ahmad55@outlook.com>
Signed-off-by: faizanahmad055 <faizan.ahmad55@outlook.com>
Signed-off-by: faizanahmad055 <faizan.ahmad55@outlook.com>
Comment on lines
+25
to
+33
| // ErrWatchTimeout is returned when a watch times out waiting for condition. | ||
| var ErrWatchTimeout = errors.New("watch timeout waiting for condition") | ||
|
|
||
| // ErrWatchError is returned when the watch receives an error event from the API server. | ||
| var ErrWatchError = errors.New("watch received error event from API server") | ||
|
|
||
| // ErrUnsupportedOperation is returned when an operation is not supported for a workload type. | ||
| var ErrUnsupportedOperation = errors.New("operation not supported for this workload type") | ||
|
|
Comment on lines
+42
to
+60
| // HasPodTemplateAnnotationChanged returns a condition that checks the pod template annotation | ||
| // is present AND its value differs from priorValue. If priorValue is empty, any non-empty value | ||
| // satisfies the condition (equivalent to HasPodTemplateAnnotation). | ||
| // Use this in WaitReloaded to correctly detect a reload after a prior reload has already set the annotation. | ||
| func HasPodTemplateAnnotationChanged[T any](accessor PodTemplateAccessor[T], key, priorValue string) Condition[T] { | ||
| return func(obj T) bool { | ||
| template := accessor(obj) | ||
| if template == nil || template.Annotations == nil { | ||
| return false | ||
| } | ||
| v, ok := template.Annotations[key] | ||
| if !ok { | ||
| return false | ||
| } | ||
| if priorValue == "" { | ||
| return true | ||
| } | ||
| return v != priorValue | ||
| } |
| kind load docker-image $(E2E_IMG) --name $(KIND_CLUSTER) | ||
| endif | ||
| endif | ||
| RELOADER_IMAGE=$(E2E_IMG) "$(GOCMD)" tool ginkgo --keep-going -v --procs=$(GINKGO_PROCS) --timeout=$(E2E_TIMEOUT) ./test/e2e/... |
Comment on lines
+43
to
+50
| | Variable | Default | Description | | ||
| |----------|----------------------------------|-------------| | ||
| | `RELOADER_IMAGE` | `ghcr.io/stakater/reloader:test` | Image to test | | ||
| | `SKIP_BUILD` | `false` | Skip the container image build and Kind load steps; requires `RELOADER_IMAGE` to point to an already-loaded image | | ||
| | `KIND_CLUSTER` | `reloader-e2e` | Kind cluster name | | ||
| | `E2E_TIMEOUT` | `45m` | Test timeout | | ||
| | `GINKGO_PROCS` | `1` | Number of parallel Ginkgo worker processes | | ||
|
|
✅ Load Test Results (quick)✅ Load Test Results: ALL TESTS PASSED
3/3 passed (100%)
Artifacts: Download |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Takeover from 1081
This PR introduces a complete overhaul of the e2e testing infrastructure, replacing the previous test approach with a modern, maintainable, and faster test suite built on Ginkgo v2
Key Changes
Test Framework Migration
core,annotations,flags,advanced,csi,argoRuntime Image Building
make e2e)Real Infrastructure Integration
Watch-Based Waiting Infrastructure
time.Sleep()and polling-based waits with Kubernetes watch APIWatchUntil[T runtime.Object]function for type-safe resource watchingDeploymentReady,DaemonSetReady, etc.) for clean assertionsWorkload Adapter Pattern
WorkloadAdapterinterface for all workload typesPausable,Recreatable,JobTriggerer) for workload-specific featuresTest Coverage
auto=true, typed auto (configmap.auto,secret.auto,secretproviderclass.auto), search/match, exclude, pause-period--watch-globally,--namespaces-to-ignore,--resources-to-ignore,--reload-on-create,--reload-on-delete,--auto-reload-allSTAKATER_*environment variables)Infrastructure Scripts
scripts/e2e-cluster-setup.sh- Installs Argo Rollouts, CSI Secrets Store Driver, Vault (with Kubernetes auth configured)scripts/e2e-cluster-cleanup.sh- Cleans up test resourcesmake e2e-setup,make e2e,make e2e-cleanup,make e2e-ciCode Quality
.golangci.ymlwith comprehensive linter configuration (errcheck, govet, staticcheck, ginkgolinter, etc.)List of all tests
advanced/job_reload_test.go
JobJobJobJobJobJobadvanced/multi_container_test.go
advanced/regex_test.go
annotations/auto_reload_test.go
DeploymentDeploymentDeploymentDeploymentDeploymentannotations/combination_test.go
annotations/exclude_test.go
Deployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigannotations/pause_period_test.go
DeploymentDeploymentDeploymentannotations/resource_ignore_test.go
annotations/search_match_test.go
DeploymentDeploymentDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigargo/rollout_test.go
ArgoRolloutArgoRolloutcore/reference_methods_test.go
Deployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigcore/workloads_test.go
Deployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSetDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigCronJobCronJobCronJobDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSetDeployment, DaemonSet, StatefulSetDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigDeployment, DaemonSet, StatefulSet, ArgoRollout, DeploymentConfigcsi/csi_test.go
flags/auto_reload_all_test.go
flags/ignored_workloads_test.go
CronJobCronJobCronJobflags/ignore_resources_test.go
flags/namespace_ignore_test.go
flags/namespace_selector_test.go
flags/reload_on_create_test.go
flags/reload_on_delete_test.go
flags/resource_selector_test.go
flags/watch_globally_test.go