Skip to content

test: save trace in job CI artifacts directory when ui e2e tests fails#1204

Merged
rsoaresd merged 5 commits intocodeready-toolchain:masterfrom
rsoaresd:add_trace
Oct 1, 2025
Merged

test: save trace in job CI artifacts directory when ui e2e tests fails#1204
rsoaresd merged 5 commits intocodeready-toolchain:masterfrom
rsoaresd:add_trace

Conversation

@rsoaresd
Copy link
Copy Markdown
Contributor

@rsoaresd rsoaresd commented Sep 24, 2025

Description

We need to save trace in the job CI artifacts directory when ui e2e tests fail, to have the prints of the ui to debug the failed test

Summary by CodeRabbit

  • Tests
    • CI runs now save sandbox UI trace artifacts to the job’s artifact directory using predictable, test-specific ZIP filenames for easier discovery and post-run debugging.
    • Containerized and local run trace locations remain unchanged.
    • Error handling, stop/save, and cleanup behavior are preserved.

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Sep 24, 2025

Walkthrough

Adds a CI-specific override in testsupport/sandbox-ui/trace.go to set tracePath to $ARTIFACT_DIR/sandbox-ui/trace/trace-<test>.zip when CI=="true". Keeps existing branches for RUNNING_IN_CONTAINER (now formatted without a leading slash) and the default filesystem trace path (../../trace/trace-.zip). Stop/save and cleanup behavior remain unchanged.

Changes

Cohort / File(s) Summary
Trace path selection logic
testsupport/sandbox-ui/trace.go
Added conditional to set tracePath to $ARTIFACT_DIR/sandbox-ui/trace/trace-<test>.zip when CI=="true". Preserved prior branches for RUNNING_IN_CONTAINER (container path formatted without a leading /) and the default path (../../trace/trace-<test>.zip). No changes to error handling, stop/save, or cleanup flows.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant Test as Test Runner
  participant Env as Environment
  participant Trace as trace.go

  Test->>Trace: Start trace for <test>
  Trace->>Env: Read CI, ARTIFACT_DIR, RUNNING_IN_CONTAINER
  alt CI == "true"
    Note over Trace: CI override (new)
    Trace->>Trace: tracePath = ARTIFACT_DIR/sandbox-ui/trace/trace-<test>.zip
  else RUNNING_IN_CONTAINER is set
    Note over Trace: Container path (no leading slash)
    Trace->>Trace: tracePath = trace/trace-<test>.zip
  else
    Note over Trace: Default filesystem path
    Trace->>Trace: tracePath = ../../trace/trace-<test>.zip
  end
  Test->>Trace: Stop/Save trace
  Trace-->>Test: Uses computed tracePath (unchanged flow)
Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

A rabbit finds a CI-lit nook,
"Put traces in the artifact book!"
Hop—no slash in container lane,
Artifacts gather, neat and plain.
Zipped trails hop off to show—
Tests leave footprints in a row. 🐇📦

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check ✅ Passed The title succinctly summarizes the primary change by indicating that test traces will be saved into the CI job artifacts directory when UI end-to-end tests fail, making it clear what behavior is being modified.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
✨ Finishing touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d043215 and 4a4a31c.

📒 Files selected for processing (1)
  • testsupport/sandbox-ui/trace.go (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • testsupport/sandbox-ui/trace.go
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
  • GitHub Check: Unit Tests
  • GitHub Check: Build & push Developer Sandbox UI image for UI e2e tests
  • GitHub Check: Build & push operator bundles for e2e tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🧪 Early access (Sonnet 4.5): enabled

We are currently testing the Sonnet 4.5 model, which is expected to improve code review quality. However, this model may lead to increased noise levels in the review comments. Please disable the early access features if the noise level causes any inconvenience.

Note:

  • Public repositories are always opted into early access features.
  • You can enable or disable early access features from the CodeRabbit UI or by updating the CodeRabbit configuration file.

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
testsupport/sandbox-ui/trace.go (1)

35-43: Create the trace directory before saving the trace to avoid failures.

Tracing().Stop(tracePath) will fail if the parent dir doesn't exist. Create it proactively.

Apply:

 	t.Cleanup(func() {
 		if t.Failed() {
+			// ensure target dir exists
+			if mkErr := os.MkdirAll(filepath.Dir(tracePath), 0o755); mkErr != nil {
+				t.Logf("failed to create trace dir: %v", mkErr)
+			}
 			if err := context.Tracing().Stop(tracePath); err != nil {
 				t.Logf("failed to save trace: %v", err)
 			} else {
 				t.Logf("saved trace to %s", tracePath)
 			}
 		}
 	})
🧹 Nitpick comments (1)
testsupport/sandbox-ui/trace.go (1)

27-29: Same absolute-path bug in container path resolution.

fmt.Sprintf("/trace/…") makes the path absolute and ignores E2E_REPO_PATH. Align with the CI fix.

Suggested change (outside the changed hunk):

traceDir := filepath.Join(os.Getenv("E2E_REPO_PATH"), "trace")
tracePath = filepath.Join(traceDir, fmt.Sprintf("trace-%s.zip", testName))
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d156db6 and f29c18a.

📒 Files selected for processing (3)
  • make/test.mk (1 hunks)
  • test/e2e/sandbox-ui/activities_page_test.go (1 hunks)
  • testsupport/sandbox-ui/trace.go (1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
  • GitHub Check: Unit Tests
  • GitHub Check: GolangCI Lint
  • GitHub Check: Build & push Developer Sandbox UI image for UI e2e tests
  • GitHub Check: Build & push operator bundles for e2e tests
🔇 Additional comments (1)
make/test.mk (1)

45-45: test-ui-e2e target exists — no action required.
Defined and marked .PHONY in make/sandbox-ui.mk (around lines 194–195).

Comment thread test/e2e/sandbox-ui/activities_page_test.go Outdated
Comment thread testsupport/sandbox-ui/trace.go
@rsoaresd rsoaresd changed the title experiment test: save trace in job CI artifacts directory when ui e2e tests fails Sep 24, 2025
Comment thread testsupport/sandbox-ui/trace.go Outdated
@sonarqubecloud
Copy link
Copy Markdown

@openshift-ci
Copy link
Copy Markdown

openshift-ci Bot commented Sep 30, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alexeykazakov, fbm3307, MatousJobanek, rajivnathan, rsoaresd, xcoulon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [MatousJobanek,alexeykazakov,fbm3307,rajivnathan,rsoaresd,xcoulon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@rsoaresd
Copy link
Copy Markdown
Contributor Author

rsoaresd commented Sep 30, 2025

/retest

similar to flakiness that we discussed on https://redhat-internal.slack.com/archives/CHK0J6HT6/p1758276534724369

host.go:2005: failed to find Space with matching criteria:
        ----
        actual:
        metadata:
          creationTimestamp: "2025-09-30T14:05:00Z"
          finalizers:
          - finalizer.toolchain.dev.openshift.com
          generation: 1
          labels:
            toolchain.dev.openshift.com/state: pending
          name: space-waitinglist1
          namespace: toolchain-host-30133728
          resourceVersion: "94211"
          uid: 1d99c060-7931-4664-8d4c-1af04ffd1ac6
        spec:
          tierName: appstudio
        status:
          conditions:
          - lastTransitionTime: "2025-09-30T14:05:00Z"
            message: unspecified target member cluster
            reason: ProvisioningPending
            status: "False"
            type: Ready
        
        ----
        diffs:
        expected target clusters not to be empty. Actual Space resource:
        &{{ } {space-waitinglist1  toolchain-host-30133728  1d99c060-7931-4664-8d4c-1af04ffd1ac6 94211 1 2025-09-30 14:05:00 +0000 UTC <nil> <nil> map[toolchain.dev.openshift.com/state:pending] map[] [] [finalizer.toolchain.dev.openshift.com] [{e2e.test Update toolchain.dev.openshift.com/v1alpha1 2025-09-30 14:05:00 +0000 UTC FieldsV1 {"f:spec":{".":{},"f:tierName":{}}} } {host-operator Update toolchain.dev.openshift.com/v1alpha1 2025-09-30 14:05:00 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"finalizer.toolchain.dev.openshift.com\"":{}},"f:labels":{".":{},"f:toolchain.dev.openshift.com/state":{}}}} } {host-operator Update toolchain.dev.openshift.com/v1alpha1 2025-09-30 14:05:00 +0000 UTC FieldsV1 {"f:status":{".":{},"f:conditions":{".":{},"k:{\"type\":\"Ready\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status}]} { [] appstudio  false} { [] [{Ready False 2025-09-30 14:05:00 +0000 UTC ProvisioningPending unspecified target member cluster <nil>}]}}
        
    space.go:108: 
        	Error Trace:	/go/src/github.com/codeready-toolchain/toolchain-e2e/testsupport/space/space.go:108
        	            				/go/src/github.com/codeready-toolchain/toolchain-e2e/test/e2e/space_autocompletion_test.go:73
        	Error:      	Received unexpected error:
        	            	context deadline exceeded
        	Test:       	TestAutomaticClusterAssignment/set_low_max_number_of_spaces_and_expect_that_space_won't_be_provisioned_but_added_on_waiting_list/increment_the_max_number_of_spaces_and_expect_that_first_space_will_be_provisioned.

@rsoaresd rsoaresd merged commit 3f847aa into codeready-toolchain:master Oct 1, 2025
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants