First VM deployment from VMware template fails with VMDK locked; second deployment succeeds

[midhunpjos]

problem

When deploying a VM from a VMware template, the first VM creation always fails,
but the second attempt succeeds.

versions

• CloudStack version: 4.21.0
• VMware version: 8.0.3 / 24853646
• Primary Storage: VMFS 6

The steps to reproduce the bug

1. Create a VMware template and register it in CloudStack.
2. Deploy a VM using this template.
3. First deployment fails.
4. Deploy again using the same template.
5. Second deployment succeeds.

Logs:
2025-11-06 08:51:08,334 INFO [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-1:[ctx-cc6702ac, job-34/job-35, ctx-fc2cf834]) (logid:74e2243d) releasing lock for VMTemplateStoragePool 2
2025-11-06 08:51:08,341 ERROR [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-1:[ctx-cc6702ac, job-34/job-35, ctx-fc2cf834]) (logid:74e2243d) Unable to create volume [{"name":"ROOT-3","uuid":"d91479c1-b9dd-4034-b394-26939dad4eff"}] due to [CopyCommand failed due to [Exception: java.lang.RuntimeException
Message: Unable to access file /vmfs/volumes/683fea5e-bc390134-295d-d4f5ef3ec3e8/8d93164511d13929aac98eec33062d49/8d93164511d13929aac98eec33062d49-000001.vmdk since it is locked
].].
2025-11-06 08:51:08,344 WARN [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-1:[ctx-cc6702ac, job-34/job-35, ctx-fc2cf834]) (logid:74e2243d) Unable to contact resource. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:4] is unreachable: Unable to create volume [{"name":"ROOT-3","uuid":"d91479c1-b9dd-4034-b394-26939dad4eff"}] due to [CopyCommand failed due to [Exception: java.lang.RuntimeException
Message: Unable to access file /vmfs/volumes/683fea5e-bc390134-295d-d4f5ef3ec3e8/8d93164511d13929aac98eec33062d49/8d93164511d13929aac98eec33062d49-000001.vmdk since it is locked
].].
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1872)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:2019)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy257.prepare(Unknown Source)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1482)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5827)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5951)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:689)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:637)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

2025-11-06 08:51:08,345 DEBUG [c.c.d.D.ExcludeList] (Work-Job-Executor-1:[ctx-cc6702ac, job-34/job-35, ctx-fc2cf834]) (logid:74e2243d) Adding StoragePool [4] to the avoid set due to [Resource [StoragePool:4] is unreachable: Unable to create volume [{"name":"ROOT-3","uuid":"d91479c1-b9dd-4034-b394-26939dad4eff"}] due to [CopyCommand failed due to [Exception: java.lang.RuntimeException
2025-11-06 08:51:08,345 DEBUG [c.c.d.D.ExcludeList] (Work-Job-Executor-1:[ctx-cc6702ac, job-34/job-35, ctx-fc2cf834]) (logid:74e2243d) Adding StoragePool [4] to the avoid set due to [Resource [StoragePool:4] is unreachable: Unable to create volume [{"name":"ROOT-3","uuid":"d91479c1-b9dd-4034-b394-26939dad4eff"}] due to [CopyCommand failed due to [Exception: java.lang.RuntimeException
Message: Unable to access file /vmfs/volumes/683fea5e-bc390134-295d-d4f5ef3ec3e8/8d93164511d13929aac98eec33062d49/8d93164511d13929aac98eec33062d49-000001.vmdk since it is locked

What to do about it?

No response

[weizhouapache]

I've noticed the same issue when I worked on vmware 8.0.1 support #7380
therefore I added a try-catch-retry logic when power on a VM
the issue was also reported in #7572

I think it is caused by some changes in vmware 8.0.1 as the issue does not exist with vmware 7.x and 8.0.0
however I cannot tell what changes and how to solve it permanently.

[github-actions]

This issue is stale because it has been open for 120 days with no activity. It may be removed by administrators of this project at any time. Remove the stale label or comment to request for removal of it to prevent this.