feat: add collect-on-failure to copy files from pods/containers before cleanup (#141)

## Summary

- Adds `cleanup.collect` config section to copy files from inside pods (Kind) or containers (Compose) when e2e tests fail, before the environment is destroyed
- Collection runs independently of `cleanup.on` — has its own `collect.on` condition (always/failure/never)
- Fires on any phase failure (setup/trigger/verify), tolerates unreachable targets with explicit logging
- Adds standalone `e2e collect` command for manual debugging with `e2e setup`
- Also auto-collects `kubectl describe` (Kind) / `docker inspect` (Compose) alongside files

### Config example

```yaml
cleanup:
  on: always
  collect:
    on: failure
    output-dir: /tmp/e2e-collect
    items:
      # Kind
      - namespace: default
        label-selector: app=oap
        paths:
          - /skywalking/logs/
      # Compose
      - service: oap-service
        paths:
          - /skywalking/logs/
```

### Output structure

Files are organized by full source path to avoid collisions:
- Kind: `output-dir/<namespace>/<pod-name>/<source-path>`
- Compose: `output-dir/<service-name>/<source-path>`

Author: wu-sheng

.github/workflows/e2e-test.yaml

@@ -60,9 +60,14 @@ jobs:
         with:
           e2e-file: ./test/e2e/e2e.yaml
 
-      - uses: engineerd/setup-kind@v0.6.2
+      - name: Install KinD
+        uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # v1.14.0
         with:
           version: "v0.27.0"
+          install_only: true
 
       - name: Run KinD E2E Test
         run: make e2e-test-kind
+
+      - name: Run KinD Collect-on-Failure E2E Test
+        run: make e2e-test-kind-collect

CLAUDE.md

@@ -0,0 +1,155 @@
+# CLAUDE.md - Project Guide for skywalking-infra-e2e
+
+## Project Overview
+
+Apache SkyWalking Infra E2E is a CLI tool for end-to-end testing. It orchestrates test environments
+(Kubernetes/Kind or Docker Compose), generates traffic, verifies results, and cleans up.
+
+## Build & Test Commands
+
+```bash
+make all              # clean + lint + test + build
+make test             # run unit tests with coverage
+make lint             # run golangci-lint (auto-installs if missing)
+make build            # build for windows/linux/darwin
+make darwin           # build for macOS only (use your current OS target)
+make e2e-test         # run e2e test with Docker Compose (test/e2e/e2e.yaml)
+make e2e-test-kind    # run e2e test with Kind (test/e2e/kind/e2e.yaml)
+```
+
+- Go module: `github.com/apache/skywalking-infra-e2e`
+- Entry point: `cmd/e2e/main.go`
+- Binary output: `bin/<os>/e2e`
+- Version injected via ldflags at build time
+
+## Architecture
+
+### CLI Commands (Cobra)
+
+| Command       | File                          | Purpose                        |
+|---------------|-------------------------------|--------------------------------|
+| `e2e run`     | `commands/run/run.go`         | Full lifecycle orchestration   |
+| `e2e setup`   | `commands/setup/setup.go`     | Setup env only (debug mode)    |
+| `e2e trigger` | `commands/trigger/trigger.go` | Run trigger only               |
+| `e2e verify`  | `commands/verify/verify.go`   | Run verification only          |
+| `e2e cleanup` | `commands/cleanup/cleanup.go` | Run cleanup only               |
+
+Global flags defined in `commands/root.go`:
+- `-c, --config` (default: `e2e.yaml`)
+- `-v, --verbosity` (debug/info/warn/error)
+- `-w, --work-dir` (default: `~/.skywalking-infra-e2e`)
+- `-l, --log-dir` (default: `~/.skywalking-infra-e2e/logs`)
+
+### Lifecycle (`e2e run`)
+
+```
+setup → trigger → verify → cleanup (deferred)
+```
+
+Cleanup runs via Go `defer` and is controlled by `cleanup.on`:
+- `always` / `success` / `failure` / `never`
+- Default: `success` locally, `always` in CI (`CI=true` env var)
+- Constants in `internal/constant/cleanup.go`
+
+### Environment Modes
+
+Determined by `setup.env` in e2e.yaml (`"kind"` or `"compose"`).
+
+Constants: `constant.Kind` and `constant.Compose` in `internal/constant/`.
+
+**Kind mode** (`internal/components/setup/kind.go`):
+- Creates Kind cluster, loads Docker images, applies K8s manifests
+- Pod log streaming via K8s client-go
+- Port forwarding via SPDY
+- Cleanup: `kind delete cluster` with retry (up to 5x)
+
+**Compose mode** (`internal/components/setup/compose.go`):
+- Uses testcontainers-go for Docker Compose
+- Container log streaming
+- Cleanup: `docker-compose down`
+
+### Configuration
+
+Config struct: `internal/config/e2eConfig.go` → `E2EConfig`
+
+```yaml
+setup:
+  env: kind|compose
+  file: path/to/kind-config.yaml  # or docker-compose.yml
+  kubeconfig: path                # alternative to file (use existing cluster)
+  timeout: 20m
+  steps:
+    - name: step-name
+      path: manifest.yaml         # or command: "shell cmd"
+      wait:
+        - namespace: default
+          resource: pod
+          label-selector: app=foo
+          for: condition=Ready
+  kind:
+    import-images: [image:tag]
+    expose-ports:
+      - namespace: default
+        resource: pod/name
+        port: "8080"
+
+cleanup:
+  on: always|success|failure|never
+
+trigger:
+  action: http
+  interval: 3s
+  times: 5
+  url: http://...
+  method: GET
+
+verify:
+  retry: { count: 10, interval: 10s }
+  fail-fast: true
+  concurrency: false
+  cases:
+    - name: case-name
+      query: "shell command"    # or actual: path/to/file
+      expected: path/to/expected.yaml
+```
+
+### Key Packages
+
+| Package                              | Role                                      |
+|--------------------------------------|-------------------------------------------|
+| `internal/config/`                   | YAML config parsing, global config state  |
+| `internal/components/setup/`         | Kind & Compose setup implementations      |
+| `internal/components/trigger/`       | HTTP trigger action                       |
+| `internal/components/verifier/`      | Test case verification with retry         |
+| `internal/components/cleanup/`       | Kind & Compose cleanup implementations    |
+| `internal/util/`                     | K8s client, Docker helpers, env/log utils |
+| `internal/constant/`                 | Constants for both modes and cleanup       |
+| `internal/logger/`                   | Logrus-based logging                      |
+| `pkg/output/`                        | Test result formatting (YAML/summary)     |
+| `third-party/go/template/`          | Extended Go template functions for verify |
+
+### Test Structure
+
+**Unit tests** (6 files):
+- `internal/config/e2eConfig_test.go`
+- `internal/util/config_test.go`, `utils_test.go`
+- `commands/verify/verify_test.go`
+- `internal/components/verifier/verifier_test.go`
+- `third-party/go/template/funcs_test.go`
+
+**E2E tests** (`test/e2e/`):
+- `e2e.yaml` — Compose-based e2e test
+- `kind/e2e.yaml` — Kind-based e2e test
+- Verify scenarios under `concurrency/` and `non-concurrency/` dirs
+
+### Log Collection
+
+- Logs streamed during setup to `LogDir/namespace/podName.log` (Kind) or `LogDir/serviceName/std.log` (Compose)
+- `internal/util/env_log.go` — `ResourceLogFollower` manages log writers
+- GitHub Actions: log dir defaults to `${runner.temp}/skywalking-infra-e2e/logs`
+- **No existing mechanism to copy arbitrary files from containers on failure**
+
+### GitHub Actions Integration
+
+- `action.yaml` at project root defines the composite action
+- Inputs: e2e-file, log-dir, plus matrix vars for log isolation

Makefile

@@ -118,3 +118,19 @@ e2e-test-kind:
 	$(MAKE) $(GOOS)
 	docker pull busybox:latest
 	./bin/$(GOOS)/$(PROJECT) run -c ./test/e2e/kind/e2e.yaml
+
+.PHONY: e2e-test-kind-collect
+# Run E2E test with KinD to verify collect-on-failure functionality
+# This test intentionally fails verification to trigger file collection.
+# It succeeds if the collected files exist at the expected output directory.
+e2e-test-kind-collect:
+	$(MAKE) $(GOOS)
+	docker pull busybox:latest
+	rm -rf /tmp/e2e-collect-test
+	./bin/$(GOOS)/$(PROJECT) run -c ./test/e2e/kind/collect-on-failure/e2e.yaml || true
+	@echo "Verifying collected files..."
+	@test -d /tmp/e2e-collect-test/default && echo "PASS: output directory created" || (echo "FAIL: output directory not found" && exit 1)
+	@ls /tmp/e2e-collect-test/default/collect-test/describe.txt > /dev/null 2>&1 && echo "PASS: describe.txt collected" || (echo "FAIL: describe.txt not found" && exit 1)
+	@ls /tmp/e2e-collect-test/default/collect-test/test-data/logs > /dev/null 2>&1 && echo "PASS: logs/ collected" || (echo "FAIL: logs/ not found" && exit 1)
+	@ls /tmp/e2e-collect-test/default/collect-test/test-data/debug.txt > /dev/null 2>&1 && echo "PASS: debug.txt collected" || (echo "FAIL: debug.txt not found" && exit 1)
+	@echo "All collect-on-failure checks passed."

commands/collect/collect.go

@@ -0,0 +1,42 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package collect
+
+import (
+	"fmt"
+
+	"github.com/spf13/cobra"
+
+	"github.com/apache/skywalking-infra-e2e/internal/components/collector"
+	"github.com/apache/skywalking-infra-e2e/internal/config"
+)
+
+var Collect = &cobra.Command{
+	Use:   "collect",
+	Short: "Collect files from pods/containers for debugging",
+	RunE: func(cmd *cobra.Command, args []string) error {
+		if config.GlobalConfig.Error != nil {
+			return config.GlobalConfig.Error
+		}
+		err := collector.DoCollect(&config.GlobalConfig.E2EConfig)
+		if err != nil {
+			return fmt.Errorf("[Collect] %s", err)
+		}
+		return nil
+	},
+}

commands/root.go

@@ -24,6 +24,7 @@ import (
 	"github.com/spf13/cobra"
 
 	"github.com/apache/skywalking-infra-e2e/commands/cleanup"
+	"github.com/apache/skywalking-infra-e2e/commands/collect"
 	"github.com/apache/skywalking-infra-e2e/commands/run"
 	"github.com/apache/skywalking-infra-e2e/commands/setup"
 	"github.com/apache/skywalking-infra-e2e/commands/trigger"
@@ -66,6 +67,10 @@ var Root = &cobra.Command{
 			return err
 		}
 
+		// Finalize collect config after LogDir is expanded, so any paths that
+		// depend on the log directory see the fully resolved value.
+		config.GlobalConfig.E2EConfig.Cleanup.Collect.Finalize()
+
 		return nil
 	},
 }
@@ -88,6 +93,7 @@ func Execute() error {
 	Root.AddCommand(trigger.Trigger)
 	Root.AddCommand(verify.Verify)
 	Root.AddCommand(cleanup.Cleanup)
+	Root.AddCommand(collect.Collect)
 
 	Root.PersistentFlags().StringVarP(&verbosity, "verbosity", "v", logrus.InfoLevel.String(), "log level (debug, info, warn, error, fatal, panic")
 	Root.PersistentFlags().StringVarP(&util.WorkDir, "work-dir", "w", "~/.skywalking-infra-e2e", "the working directory for skywalking-infra-e2e")

commands/run/run.go

@@ -22,6 +22,7 @@ import (
 	"github.com/apache/skywalking-infra-e2e/commands/setup"
 	"github.com/apache/skywalking-infra-e2e/commands/trigger"
 	"github.com/apache/skywalking-infra-e2e/commands/verify"
+	"github.com/apache/skywalking-infra-e2e/internal/components/collector"
 	t "github.com/apache/skywalking-infra-e2e/internal/components/trigger"
 	"github.com/apache/skywalking-infra-e2e/internal/config"
 	"github.com/apache/skywalking-infra-e2e/internal/constant"
@@ -43,7 +44,7 @@ var Run = &cobra.Command{
 	},
 }
 
-func runAccordingE2E() error {
+func runAccordingE2E() (err error) {
 	if config.GlobalConfig.Error != nil {
 		return config.GlobalConfig.Error
 	}
@@ -54,14 +55,24 @@ func runAccordingE2E() error {
 			action.Stop()
 		}
 	}
+
 	// If cleanup.on == Always and there is error in setup step, we should defer cleanup step right now.
 	cleanupOnCondition := config.GlobalConfig.E2EConfig.Cleanup.On
 	if cleanupOnCondition == constant.CleanUpAlways {
 		defer doCleanup(stopAction)
 	}
 
+	// Collection runs independently of cleanup — it has its own collect.on condition.
+	// Registered before setup so it fires even on partial setup failures.
+	// Registered AFTER the Always-cleanup defer so it runs BEFORE cleanup (LIFO),
+	// ensuring files are collected while pods/containers are still alive.
+	// Errors are tolerated — partial setup may leave some pods unreachable.
+	defer func() {
+		doCollect(err)
+	}()
+
 	// setup part
-	err := setup.DoSetupAccordingE2E()
+	err = setup.DoSetupAccordingE2E()
 	if err != nil {
 		return err
 	}
@@ -106,6 +117,22 @@ func runAccordingE2E() error {
 	return nil
 }
 
+// doCollect runs file collection independently of cleanup.
+// It evaluates collect.on against the run error to decide whether to collect.
+func doCollect(runErr error) {
+	collectCfg := config.GlobalConfig.E2EConfig.Cleanup.Collect
+	shouldCollect := (collectCfg.On == constant.CollectAlways) ||
+		(collectCfg.On == constant.CollectOnFailure && runErr != nil)
+	if !shouldCollect || len(collectCfg.Items) == 0 {
+		return
+	}
+	if err := collector.DoCollect(&config.GlobalConfig.E2EConfig); err != nil {
+		logger.Log.Warnf("collect files error: %s", err)
+	} else {
+		logger.Log.Infof("collect files finished successfully")
+	}
+}
+
 func doCleanup(stopAction func()) {
 	if stopAction != nil {
 		stopAction()

docs/en/setup/Configuration-File.md

@@ -261,13 +261,48 @@ After the E2E finished, how to clean up the environment.
 ```yaml
 cleanup:
    on: always     # Clean up strategy
+   collect:       # Collect files from pods/containers for debugging
+     on: failure  # When to collect: always|failure|never, default: failure
+     output-dir: /tmp/collect # Where to save the collected files (required)
+     items:
+       # For Kind environment
+       - namespace: default      # Pod namespace
+         label-selector: app=oap # Label selector to find pods
+         resource: pod/oap-0     # Specific pod resource (optional, instead of label-selector)
+         container: oap          # Container name (optional, defaults to first container)
+         paths:                  # Paths in the container to collect
+           - /skywalking/logs/
+       # For Compose environment
+       - service: ui-service     # Compose service name
+         paths:
+           - /var/log/nginx/
 ```
 
 If the `on` option under `cleanup` is not set, it will be automatically set to `always` if there is environment
 variable `CI=true`, which is present on many popular CI services, such as GitHub Actions, CircleCI, etc., otherwise it
 will be set to `success`, so the testing environment can be preserved when tests failed in your local machine.
 
-All available strategies:
+### Collect
+
+The `collect` section allows you to copy files from containers to the local machine before the environment is destroyed. This is useful for debugging failures in CI.
+
+* `on`: When to trigger collection.
+    * `always`: Always collect.
+    * `failure`: Only collect when any step (setup, trigger, verify) fails. Collection also attempts on setup failures — unreachable pods/containers are tolerated and logged.
+    * `never`: Never collect.
+* `output-dir`: **Required.** The local directory to save files. Supports environment variable expansion (e.g. `$SW_INFRA_E2E_LOG_DIR/collect`).
+* `items`: A list of collection tasks.
+    * For **Kind**: Specify `namespace` and either `label-selector` or `resource`. `container` is optional.
+    * For **Compose**: Specify `service`.
+    * `paths`: A list of file or directory paths inside the container.
+
+Collected files are organized by the full source path to avoid collisions:
+* Kind: `output-dir/<namespace>/<pod-name>/<source-path>`
+* Compose: `output-dir/<service-name>/<source-path>`
+
+Additionally, `kubectl describe` (for Kind) or `docker inspect` (for Compose) output is saved automatically alongside collected files.
+
+All available strategies for `cleanup.on`:
 1. `always`: No matter the execution result is success or failure, cleanup will be performed.
 1. `success`: Only when the execution succeeds.
 1. `failure`: Only when the execution failed.