RHOAIENG-69862: RHOAI 3.3: fix lifecycled tests

[pawelpaszki]

Issue link

https://redhat.atlassian.net/browse/RHOAIENG-69862

What changes have been made

test fixes

Verification steps

Before fix

tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py::TestRayJobLifecycledCluster::test_lifecycled_kueue_managed creating Kueue resources ...
'test-resource-flavor-5rj66' created!
'test-cluster-queue-7s1w7' created
'test-local-queue-890b4' created in namespace 'test-ns-cdbyp'
✓ Secret kueue-lifecycled-files verified with proper owner reference
FAILED
'test-cluster-queue-7s1w7' cluster-queue deleted
'test-resource-flavor-5rj66' resource-flavor deleted

tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py::TestRayJobLifecycledCluster::test_lifecycled_kueue_resource_queueing Creating limited Kueue resources for preemption testing...
'limited-flavor-pescp' created!
✓ Created limited ClusterQueue: limited-cq-cld5b
'limited-lq-den7g' created in namespace 'test-ns-9atqo'
✓ Limited Kueue resources created successfully
FAILED
'limited-cq-cld5b' cluster-queue deleted
'limited-flavor-pescp' resource-flavor deleted


======================================================================================================================= FAILURES =======================================================================================================================
______________________________________________________________________________________________ TestRayJobLifecycledCluster.test_lifecycled_kueue_managed _______________________________________________________________________________________________
tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py:83: in test_lifecycled_kueue_managed
    assert self.job_api.wait_until_job_running(
E   AssertionError: assert False
E    +  where False = <bound method RayjobApi.wait_until_job_running of <codeflare_sdk.vendored.python_client.kuberay_job_api.RayjobApi object at 0x7fffdc9bdd30>>(name='kueue-lifecycled', k8s_namespace='test-ns-cdbyp', timeout=600)
E    +    where <bound method RayjobApi.wait_until_job_running of <codeflare_sdk.vendored.python_client.kuberay_job_api.RayjobApi object at 0x7fffdc9bdd30>> = <codeflare_sdk.vendored.python_client.kuberay_job_api.RayjobApi object at 0x7fffdc9bdd30>.wait_until_job_running
E    +      where <codeflare_sdk.vendored.python_client.kuberay_job_api.RayjobApi object at 0x7fffdc9bdd30> = <rayjob.rayjob_lifecycled_cluster_test.TestRayJobLifecycledCluster object at 0x7fffdd773260>.job_api
E    +    and   'kueue-lifecycled' = <codeflare_sdk.ray.rayjobs.rayjob.RayJob object at 0x7fffdca744d0>.name
E    +    and   'test-ns-cdbyp' = <codeflare_sdk.ray.rayjobs.rayjob.RayJob object at 0x7fffdca744d0>.namespace
_________________________________________________________________________________________ TestRayJobLifecycledCluster.test_lifecycled_kueue_resource_queueing __________________________________________________________________________________________
tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py:135: in test_lifecycled_kueue_resource_queueing
    assert self.job_api.wait_until_job_running(
E   AssertionError: assert False
E    +  where False = <bound method RayjobApi.wait_until_job_running of <codeflare_sdk.vendored.python_client.kuberay_job_api.RayjobApi object at 0x7fffdc886ba0>>(name='holder', k8s_namespace='test-ns-9atqo', timeout=60)
E    +    where <bound method RayjobApi.wait_until_job_running of <codeflare_sdk.vendored.python_client.kuberay_job_api.RayjobApi object at 0x7fffdc886ba0>> = <codeflare_sdk.vendored.python_client.kuberay_job_api.RayjobApi object at 0x7fffdc886ba0>.wait_until_job_running
E    +      where <codeflare_sdk.vendored.python_client.kuberay_job_api.RayjobApi object at 0x7fffdc886ba0> = <rayjob.rayjob_lifecycled_cluster_test.TestRayJobLifecycledCluster object at 0x7fffdd773350>.job_api
E    +    and   'holder' = <codeflare_sdk.ray.rayjobs.rayjob.RayJob object at 0x7fffdfd90a40>.name
E    +    and   'test-ns-9atqo' = <codeflare_sdk.ray.rayjobs.rayjob.RayJob object at 0x7fffdfd90a40>.namespace

After fix

tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py::TestRayJobLifecycledCluster::test_lifecycled_kueue_managed creating Kueue resources ...
'test-resource-flavor-fshlf' created!
'test-cluster-queue-vb0ao' created
'test-local-queue-jvpnm' created in namespace 'test-ns-uxgvf'
✓ RayJob kueue-lifecycled submitted successfully
✓ Secret kueue-lifecycled-files verified with proper owner reference
Waiting for Kueue admission of job 'kueue-lifecycled'...
✓ Job 'kueue-lifecycled' admitted by Kueue (no longer suspended)
Waiting for RayJob kueue-lifecycled to reach running state (attempt 1/3)
Waiting for Kueue admission of job 'kueue-lifecycled'...
✓ Job 'kueue-lifecycled' admitted by Kueue (no longer suspended)
✓ RayJob kueue-lifecycled reached running state on attempt 1
PASSED
'test-cluster-queue-vb0ao' cluster-queue deleted
'test-resource-flavor-fshlf' resource-flavor deleted

tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py::TestRayJobLifecycledCluster::test_lifecycled_kueue_resource_queueing Creating limited Kueue resources for preemption testing...
'limited-flavor-07h11' created!
✓ Created limited ClusterQueue: limited-cq-r793k
'limited-lq-tvjxm' created in namespace 'test-ns-afxbg'
✓ Limited Kueue resources created successfully
Waiting for Kueue admission of job 'holder'...
✓ Job 'holder' admitted by Kueue (no longer suspended)
Waiting for RayJob holder to reach running state (attempt 1/2)
Waiting for Kueue admission of job 'holder'...
✓ Job 'holder' admitted by Kueue (no longer suspended)
✓ RayJob holder reached running state on attempt 1
Waiting for Kueue admission of job 'waiter'...
✓ Job 'waiter' admitted by Kueue (no longer suspended)
PASSED

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

[openshift-ci-robot]

@pawelpaszki: This pull request references RHOAIENG-69862 which is a valid jira issue.

Details

In response to this:

Issue link

https://redhat.atlassian.net/browse/RHOAIENG-69862

What changes have been made

test fixes

Verification steps

Before fix

tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py::TestRayJobLifecycledCluster::test_lifecycled_kueue_managed creating Kueue resources ...
'test-resource-flavor-5rj66' created!
'test-cluster-queue-7s1w7' created
'test-local-queue-890b4' created in namespace 'test-ns-cdbyp'
✓ Secret kueue-lifecycled-files verified with proper owner reference
FAILED
'test-cluster-queue-7s1w7' cluster-queue deleted
'test-resource-flavor-5rj66' resource-flavor deleted

tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py::TestRayJobLifecycledCluster::test_lifecycled_kueue_resource_queueing Creating limited Kueue resources for preemption testing...
'limited-flavor-pescp' created!
✓ Created limited ClusterQueue: limited-cq-cld5b
'limited-lq-den7g' created in namespace 'test-ns-9atqo'
✓ Limited Kueue resources created successfully
FAILED
'limited-cq-cld5b' cluster-queue deleted
'limited-flavor-pescp' resource-flavor deleted


======================================================================================================================= FAILURES =======================================================================================================================
______________________________________________________________________________________________ TestRayJobLifecycledCluster.test_lifecycled_kueue_managed _______________________________________________________________________________________________
tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py:83: in test_lifecycled_kueue_managed
   assert self.job_api.wait_until_job_running(
E   AssertionError: assert False
E    +  where False = <bound method RayjobApi.wait_until_job_running of <codeflare_sdk.vendored.python_client.kuberay_job_api.RayjobApi object at 0x7fffdc9bdd30>>(name='kueue-lifecycled', k8s_namespace='test-ns-cdbyp', timeout=600)
E    +    where <bound method RayjobApi.wait_until_job_running of <codeflare_sdk.vendored.python_client.kuberay_job_api.RayjobApi object at 0x7fffdc9bdd30>> = <codeflare_sdk.vendored.python_client.kuberay_job_api.RayjobApi object at 0x7fffdc9bdd30>.wait_until_job_running
E    +      where <codeflare_sdk.vendored.python_client.kuberay_job_api.RayjobApi object at 0x7fffdc9bdd30> = <rayjob.rayjob_lifecycled_cluster_test.TestRayJobLifecycledCluster object at 0x7fffdd773260>.job_api
E    +    and   'kueue-lifecycled' = <codeflare_sdk.ray.rayjobs.rayjob.RayJob object at 0x7fffdca744d0>.name
E    +    and   'test-ns-cdbyp' = <codeflare_sdk.ray.rayjobs.rayjob.RayJob object at 0x7fffdca744d0>.namespace
_________________________________________________________________________________________ TestRayJobLifecycledCluster.test_lifecycled_kueue_resource_queueing __________________________________________________________________________________________
tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py:135: in test_lifecycled_kueue_resource_queueing
   assert self.job_api.wait_until_job_running(
E   AssertionError: assert False
E    +  where False = <bound method RayjobApi.wait_until_job_running of <codeflare_sdk.vendored.python_client.kuberay_job_api.RayjobApi object at 0x7fffdc886ba0>>(name='holder', k8s_namespace='test-ns-9atqo', timeout=60)
E    +    where <bound method RayjobApi.wait_until_job_running of <codeflare_sdk.vendored.python_client.kuberay_job_api.RayjobApi object at 0x7fffdc886ba0>> = <codeflare_sdk.vendored.python_client.kuberay_job_api.RayjobApi object at 0x7fffdc886ba0>.wait_until_job_running
E    +      where <codeflare_sdk.vendored.python_client.kuberay_job_api.RayjobApi object at 0x7fffdc886ba0> = <rayjob.rayjob_lifecycled_cluster_test.TestRayJobLifecycledCluster object at 0x7fffdd773350>.job_api
E    +    and   'holder' = <codeflare_sdk.ray.rayjobs.rayjob.RayJob object at 0x7fffdfd90a40>.name
E    +    and   'test-ns-9atqo' = <codeflare_sdk.ray.rayjobs.rayjob.RayJob object at 0x7fffdfd90a40>.namespace

After fix

tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py::TestRayJobLifecycledCluster::test_lifecycled_kueue_managed creating Kueue resources ...
'test-resource-flavor-fshlf' created!
'test-cluster-queue-vb0ao' created
'test-local-queue-jvpnm' created in namespace 'test-ns-uxgvf'
✓ RayJob kueue-lifecycled submitted successfully
✓ Secret kueue-lifecycled-files verified with proper owner reference
Waiting for Kueue admission of job 'kueue-lifecycled'...
✓ Job 'kueue-lifecycled' admitted by Kueue (no longer suspended)
Waiting for RayJob kueue-lifecycled to reach running state (attempt 1/3)
Waiting for Kueue admission of job 'kueue-lifecycled'...
✓ Job 'kueue-lifecycled' admitted by Kueue (no longer suspended)
✓ RayJob kueue-lifecycled reached running state on attempt 1
PASSED
'test-cluster-queue-vb0ao' cluster-queue deleted
'test-resource-flavor-fshlf' resource-flavor deleted

tests/e2e/rayjob/rayjob_lifecycled_cluster_test.py::TestRayJobLifecycledCluster::test_lifecycled_kueue_resource_queueing Creating limited Kueue resources for preemption testing...
'limited-flavor-07h11' created!
✓ Created limited ClusterQueue: limited-cq-r793k
'limited-lq-tvjxm' created in namespace 'test-ns-afxbg'
✓ Limited Kueue resources created successfully
Waiting for Kueue admission of job 'holder'...
✓ Job 'holder' admitted by Kueue (no longer suspended)
Waiting for RayJob holder to reach running state (attempt 1/2)
Waiting for Kueue admission of job 'holder'...
✓ Job 'holder' admitted by Kueue (no longer suspended)
✓ RayJob holder reached running state on attempt 1
Waiting for Kueue admission of job 'waiter'...
✓ Job 'waiter' admitted by Kueue (no longer suspended)
PASSED

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chipspeak

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [chipspeak]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment