Don't allow inadvertent deletion of hidden details via API

[mlsorensen]

Description

This PR fixes the case where someone updates the VM details via API and the non-displayable details that cloudstack uses for managing config internally get deleted. An example of a non-displayable detail is configDriveLocation. If a user is using host based config drive, then configDriveLocation = "HOST". If any detail is updated/changed, then this non-displayable detail is wiped. It was never visible via the API, so there's no chance someone could preserve it during a detail update.

In the case that the VM is stopped, a detail is updated, and then the VM delete is attempted, VM deletion can fail because it can't find the proper config drive location. A workaround for this specific situation is to start the VM, which will rewrite the missing detail. However, this surfaced the issue that these hidden details aren't preserved if details are updated.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Tested as user and as admin manually.

Provided a Marvin test for updateVirtualMachine to cover common situations.

[DaanHoogland]

code looks good. one remark about the integration test

[weizhouapache]

code lgtm

need testing

[weizhouapache]

@blueorangutan package

[blueorangutan]

@weizhouapache a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[weizhouapache]

@blueorangutan package

[blueorangutan]

@weizhouapache a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6843

[weizhouapache]

@blueorangutan test

[blueorangutan]

@weizhouapache a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-7497)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 47267 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7880-t7497-kvm-centos7.zip
Smoke tests completed. 107 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_08_upgrade_kubernetes_ha_cluster	Failure	691.55	test_kubernetes_clusters.py

[mlsorensen]

@blueorangutan test

[blueorangutan]

@mlsorensen a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[codecov]

Codecov Report

Merging #7880 (8148309) into 4.18 (f7345e8) will increase coverage by 0.02%.
Report is 9 commits behind head on 4.18.
The diff coverage is 57.14%.

@@             Coverage Diff              @@
##               4.18    #7880      +/-   ##
============================================
+ Coverage     13.04%   13.06%   +0.02%     
- Complexity     9067     9089      +22     
============================================
  Files          2720     2720              
  Lines        257236   257299      +63     
  Branches      40103    40120      +17     
============================================
+ Hits          33552    33620      +68     
+ Misses       219474   219456      -18     
- Partials       4210     4223      +13     

Files Changed	Coverage Δ
.../src/main/java/com/cloud/vm/UserVmManagerImpl.java	7.47% <57.14%> (+0.14%)	⬆️

... and 10 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[blueorangutan]

[SF] Trillian test result (tid-7519)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40018 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7880-t7519-kvm-centos7.zip
Smoke tests completed. 108 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[weizhouapache]

tested on a xcpng environment, the hidden vm setting is still there

before vm setting change

mysql> select * from user_vm_details where vm_id=22;                                 
+-----+-------+------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------+
| id  | vm_id | name                               | value                                                                                                                     | display |
+-----+-------+------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------+
| 425 |    22 | configDriveLocation                | SECONDARY                                                                                                                 |       0 |
| 427 |    22 | cpuOvercommitRatio                 | 2.0                                                                                                                       |       1 |
| 426 |    22 | hypervisortoolsversion             | xenserver56                                                                                                               |       1 |
| 428 |    22 | memoryOvercommitRatio              | 1.0                                                                                                                       |       1 |
| 430 |    22 | Message.ReservedCapacityFreed.Flag | false                                                                                                                     |       1 |
| 429 |    22 | platform                           | device-model:qemu-upstream-compat;apic:true;viridian:true;timeoffset:1;pae:true;acpi:1;hpet:true;secureboot:false;nx:true |       1 |
+-----+-------+------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------+
6 rows in set (0.00 sec)

after

mysql> select * from user_vm_details where vm_id=22;
+-----+-------+------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------+
| id  | vm_id | name                               | value                                                                                                                     | display |
+-----+-------+------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------+
| 434 |    22 | configDriveLocation                | SECONDARY                                                                                                                 |       0 |
| 435 |    22 | cpuOvercommitRatio                 | 2.0                                                                                                                       |       1 |
| 433 |    22 | hypervisortoolsversion             | xenserver56                                                                                                               |       1 |
| 431 |    22 | memoryOvercommitRatio              | 11                                                                                                                        |       1 |
| 432 |    22 | Message.ReservedCapacityFreed.Flag | false                                                                                                                     |       1 |
| 436 |    22 | platform                           | device-model:qemu-upstream-compat;apic:true;viridian:true;timeoffset:1;pae:true;acpi:1;hpet:true;secureboot:false;nx:true |       1 |
+-----+-------+------------------------------------+---------------------------------------------------------------------------------------------------------------------------+---------+
6 rows in set (0.00 sec)