engine-orchestration,vmware: hypervisor migration during start vm migration by shwstppr · Pull Request #7444 · apache/cloudstack

shwstppr · 2023-04-18T11:41:23Z

Description

This PR enables the hypervisor strategy for VM migration when a VM is to be started in a different cluster for VMware.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

codecov · 2023-04-18T14:30:42Z

Codecov Report

Merging #7444 (d02c769) into 4.18 (dabefca) will decrease coverage by 0.01%.
The diff coverage is 0.00%.

❗ Current head d02c769 differs from pull request most recent head d7cfefb. Consider uploading reports for the commit d7cfefb to get more accurate results

@@             Coverage Diff              @@
##               4.18    #7444      +/-   ##
============================================
- Coverage     12.70%   12.70%   -0.01%     
  Complexity     8673     8673              
============================================
  Files          2717     2717              
  Lines        256186   256207      +21     
  Branches      39929    39932       +3     
============================================
  Hits          32541    32541              
- Misses       219507   219528      +21     
  Partials       4138     4138

Impacted Files	Coverage Δ
.../main/java/com/cloud/vm/VirtualMachineManager.java	`92.30% <ø> (ø)`
...n/java/com/cloud/vm/VirtualMachineManagerImpl.java	`6.19% <0.00%> (-0.04%)`	⬇️
...ain/java/com/cloud/hypervisor/guru/VMwareGuru.java	`1.06% <0.00%> (ø)`

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

sonarqubecloud · 2023-04-19T13:46:14Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

32.1% Coverage
0.0% Duplication

shwstppr · 2023-05-08T08:33:41Z

@blueorangutan package

blueorangutan · 2023-05-08T08:36:03Z

@shwstppr a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-05-08T09:28:27Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6041

shwstppr · 2023-05-08T10:10:02Z

@blueorangutan test matrix

blueorangutan · 2023-05-08T10:12:03Z

@shwstppr a Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

DaanHoogland

clgtm, can you advice on test approach @shwstppr ? I mean in addition to your added unit tests?

shwstppr · 2023-05-08T10:29:01Z

@DaanHoogland A multi-cluster env would be needed. Then the following could be tested:

Deploy a test vm in cluster-1 and stop it
Fill cluster-1 to its capacity. Make sure the capacity reservation for the test vm is expired.
When no more new VMs can be deployed in the cluster-1 then try to start the test vm. This would trigger inter-cluster migration for the VM
Observe on the hypervisor side and in the logs that the migration is carried by the hypervisor (vMotion in case VMware) and not using the SSVM and export snapshot technique.

yadvr

LGTM

blueorangutan · 2023-05-08T21:46:35Z

Trillian test result (tid-6512)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40175 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7444-t6512-xenserver-71.zip
Smoke tests completed. 108 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2023-05-08T22:23:10Z

Trillian test result (tid-6514)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 42310 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7444-t6514-kvm-centos7.zip
Smoke tests completed. 108 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2023-05-09T02:35:40Z

Trillian test result (tid-6513)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 57520 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7444-t6513-vmware-67u3.zip
Smoke tests completed. 106 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestSnapshotRootDisk>:setup	`Error`	0.00	test_snapshots.py
test_01_deploy_vm_on_specific_host	`Error`	21.81	test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster	`Error`	3606.59	test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod	`Error`	15.72	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	9.57	test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod	`Error`	31.12	test_vm_deployment_planner.py

shwstppr · 2023-05-09T09:03:01Z

Need to investigate the VMware failures. Though they seem unrelated as we are seeing them on other PRs as well, eg: #7404

DaanHoogland · 2023-05-09T15:09:58Z

Need to investigate the VMware failures. Though they seem unrelated as we are seeing them on other PRs as well, eg: #7404

indeed, but the test_snapshot failure is extra! I think the test_vm_deployment_planner should get there own ticket (if they persist)

shwstppr · 2023-05-10T12:04:08Z

Tested with latest nightly and it seems error with test_vm_deployment_planner.py were intermittent or environment related,

[root@ref-trl-4867-v-M7-abhishek-kumar-marvin marvin]# nosetests --with-xunit --xunit-file=results.xml --with-marvin --marvin-config=./ref-trl-4867-v-M7-abhishek-kumar-advanced-cfg -s -a tags=advanced --hypervisor=Vmware tests/smoke/test_vm_deployment_planner.py 
/usr/local/lib/python3.6/site-packages/paramiko/transport.py:32: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6.
  from cryptography.hazmat.backends import default_backend

==== Marvin Init Started ====

=== Marvin Parse Config Successful ===

=== Marvin Setting TestData Successful===

==== Log Folder Path: /marvin/MarvinLogs/May_10_2023_11_53_53_U8PK9N All logs will be available here ====

=== Marvin Init Logging Successful===

==== Marvin Init Successful ====
=== TestName: test_01_deploy_vm_on_specific_host | Status : SUCCESS ===

=== TestName: test_02_deploy_vm_on_specific_cluster | Status : SUCCESS ===

=== TestName: test_03_deploy_vm_on_specific_pod | Status : SUCCESS ===

=== TestName: test_04_deploy_vm_on_host_override_pod_and_cluster | Status : SUCCESS ===

=== TestName: test_05_deploy_vm_on_cluster_override_pod | Status : SUCCESS ===

=== Final results are now copied to: /marvin//MarvinLogs/test_vm_deployment_planner_0VX1NC ===

test_snapshots.py failed while getting template in the ^^ VMware run,

CRITICAL: EXCEPTION: None: ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python3.6/site-packages/nose/suite.py", line 210, in run\n self.setUp()\n', ' File "/usr/local/lib/python3.6/site-packages/nose/suite.py", line 293, in setUp\n self.setupContext(ancestor)\n', ' File "/usr/local/lib/python3.6/site-packages/nose/suite.py", line 316, in setupContext\n try_run(context, names)\n', ' File "/usr/local/lib/python3.6/site-packages/nose/util.py", line 471, in try_run\n return func()\n', ' File "/marvin/tests/smoke/test_snapshots.py", line 67, in setUpClass\n assert False, "get_test_template() failed to return template"\n', 'AssertionError: get_test_template() failed to return template\n']
``

Will run VMware tests again

@blueorangutan package

shwstppr · 2023-05-10T12:04:25Z

@blueorangutan package

blueorangutan · 2023-05-10T12:06:06Z

@shwstppr a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-05-10T13:04:19Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6054

shwstppr · 2023-05-10T13:56:52Z

@blueorangutan test rocky8 vmware-67u3

shwstppr · 2023-05-18T08:14:11Z

@blueorangutan package

blueorangutan · 2023-05-18T08:16:04Z

@shwstppr a [LL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-05-18T09:43:29Z

Packaging result [LL]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6036

yadvr · 2023-05-22T06:29:26Z

@blueorangutan test rocky8 vmware-67u3

blueorangutan · 2023-05-22T06:30:03Z

@rohityadavcloud a [LL] Trillian-Jenkins test job (rocky8 mgmt + vmware-67u3) has been kicked to run smoke tests

blueorangutan · 2023-05-22T08:19:20Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6096

blueorangutan · 2023-05-23T01:11:17Z

Trillian test result (tid-6538)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 58577 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7444-t6538-vmware-70u3.zip
Smoke tests completed. 107 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_deploy_vm_on_specific_host	`Error`	3604.18	test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster	`Error`	1.27	test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod	`Error`	15.68	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	3604.61	test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod	`Error`	15.64	test_vm_deployment_planner.py

blueorangutan · 2023-05-23T10:25:50Z

Trillian test result (tid-6531)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 98749 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7444-t6531-vmware-67u3.zip
Smoke tests completed. 106 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_redundant_vpc_site2site_vpn	`Failure`	3609.70	test_vpc_vpn.py
test_01_redundant_vpc_site2site_vpn	`Error`	3609.78	test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options	`Error`	14778.33	test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options	`Error`	14778.68	test_vpc_vpn.py
test_01_vpc_remote_access_vpn	`Failure`	3610.74	test_vpc_vpn.py
test_01_vpc_site2site_vpn	`Failure`	3609.52	test_vpc_vpn.py
test_01_vpc_site2site_vpn	`Error`	3609.60	test_vpc_vpn.py
test_01_cancel_host_maintenace_with_no_migration_jobs	`Error`	1806.97	test_host_maintenance.py
test_02_cancel_host_maintenace_with_migration_jobs	`Error`	2003.78	test_host_maintenance.py
test_03_cancel_host_maintenace_with_migration_jobs_failure	`Error`	1906.18	test_host_maintenance.py

shwstppr · 2023-06-01T19:16:04Z

@blueorangutan package

blueorangutan · 2023-06-01T19:18:03Z

@shwstppr a [LL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-06-01T20:07:39Z

Packaging result [LL]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6069

shwstppr · 2023-06-05T08:15:27Z

@blueorangutan package

blueorangutan · 2023-06-05T08:16:02Z

@shwstppr a [LL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-06-05T10:19:39Z

Packaging result [LL]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6076

shwstppr · 2023-06-06T04:29:22Z

@blueorangutan test matrix

blueorangutan · 2023-06-06T04:30:03Z

@shwstppr a [SF] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2023-06-06T05:13:20Z

[SF] Trillian Build Failed (tid-6685)

blueorangutan · 2023-06-06T15:44:21Z

[SF] Trillian test result (tid-6686)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 38961 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7444-t6686-kvm-centos7.zip
Smoke tests completed. 108 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2023-06-06T16:12:42Z

[SF] Trillian test result (tid-6684)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40782 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7444-t6684-xenserver-71.zip
Smoke tests completed. 108 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

shwstppr · 2023-06-08T11:08:35Z

@blueorangutan test rocky8 vmware-67u3

blueorangutan · 2023-06-08T11:10:05Z

@shwstppr a [SF] Trillian-Jenkins test job (rocky8 mgmt + vmware-67u3) has been kicked to run smoke tests

blueorangutan · 2023-06-09T03:14:41Z

[SF] Trillian test result (tid-6706)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 56501 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7444-t6706-vmware-67u3.zip
Smoke tests completed. 106 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_deploy_vm_on_specific_host	`Error`	15.61	test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster	`Error`	3603.37	test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod	`Error`	4.37	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	4.37	test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod	`Error`	4.36	test_vm_deployment_planner.py
test_09_expunge_vm	`Failure`	427.75	test_vm_life_cycle.py

DaanHoogland · 2023-06-09T09:15:42Z

@blueorangutan test rocky8 vmware-67u3

blueorangutan · 2023-06-09T09:16:03Z

@DaanHoogland a [SF] Trillian-Jenkins test job (rocky8 mgmt + vmware-67u3) has been kicked to run smoke tests

blueorangutan · 2023-06-10T00:27:03Z

[SF] Trillian test result (tid-6712)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 53221 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7444-t6712-vmware-67u3.zip
Smoke tests completed. 106 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_04_autoscale_kubernetes_cluster	`Failure`	156.64	test_kubernetes_clusters.py
test_CreateTemplateWithDuplicateName	`Error`	1801.16	test_templates.py

vladimirpetrov

LGTM based on manual testing (tested with VMWare 7.0.3).

shwstppr added 2 commits April 18, 2023 17:07

wip

707ce7f

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

changes for target cluster host

3feea78

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

boring-cyborg bot added component:compute component:orchestration component:vmware labels Apr 18, 2023

test fix

d02c769

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr added 2 commits April 19, 2023 16:42

add test

e026e29

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix

d7cfefb

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr marked this pull request as ready for review May 8, 2023 08:33

weizhouapache added this to the 4.18.1.0 milestone May 8, 2023

DaanHoogland approved these changes May 8, 2023

View reviewed changes

yadvr approved these changes May 8, 2023

View reviewed changes

yadvr added the status:needs-testing label May 25, 2023

vladimirpetrov approved these changes Jun 20, 2023

View reviewed changes

DaanHoogland merged commit 3748f32 into apache:4.18 Jun 20, 2023

yadvr deleted the fix-startvm-interclustermig-vmotion branch March 5, 2024 11:41

DaanHoogland removed the status:needs-testing label Jan 28, 2026

Conversation

shwstppr commented Apr 18, 2023

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

codecov bot commented Apr 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sonarqubecloud bot commented Apr 19, 2023

Uh oh!

shwstppr commented May 8, 2023

Uh oh!

blueorangutan commented May 8, 2023

Uh oh!

blueorangutan commented May 8, 2023

Uh oh!

shwstppr commented May 8, 2023

Uh oh!

blueorangutan commented May 8, 2023

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

shwstppr commented May 8, 2023

Uh oh!

yadvr left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented May 8, 2023

Uh oh!

blueorangutan commented May 8, 2023

Uh oh!

blueorangutan commented May 9, 2023

Uh oh!

shwstppr commented May 9, 2023

Uh oh!

DaanHoogland commented May 9, 2023

Uh oh!

shwstppr commented May 10, 2023

Uh oh!

shwstppr commented May 10, 2023

Uh oh!

blueorangutan commented May 10, 2023

Uh oh!

blueorangutan commented May 10, 2023

Uh oh!

shwstppr commented May 10, 2023

Uh oh!

shwstppr commented May 18, 2023

Uh oh!

blueorangutan commented May 18, 2023

Uh oh!

blueorangutan commented May 18, 2023

Uh oh!

yadvr commented May 22, 2023

Uh oh!

blueorangutan commented May 22, 2023

Uh oh!

blueorangutan commented May 22, 2023

Uh oh!

blueorangutan commented May 23, 2023

Uh oh!

blueorangutan commented May 23, 2023

Uh oh!

shwstppr commented Jun 1, 2023

Uh oh!

blueorangutan commented Jun 1, 2023

Uh oh!

blueorangutan commented Jun 1, 2023

Uh oh!

shwstppr commented Jun 5, 2023

Uh oh!

blueorangutan commented Jun 5, 2023

Uh oh!

codecov bot commented Apr 18, 2023 •

edited

Loading