Repository navigation
Uploaded Volume cannot be attached to VM with local storage #2730
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
API
CLOUDSTACK VERSION
4.11.1
CONFIGURATION
4.11.1, KVM+Local Storage, NFS Secondary Storage
OS / ENVIRONMENT
Ubuntu 16.04
SUMMARY
When a volume is uploaded to CloudStack it cannot be attached to a VM with local storage with a error. Checking agent node, I don't even see a specific mount for that NFS volume where uploaded volume is located. Template mount points exist, so no problems with NFS for sure.
STEPS TO REPRODUCE
- Upload a new volume to CloudStack.
- Create a local storage VM from template
- Try to attach the volume to the VM
EXPECTED RESULTS
Volume is copied to local storage where VM run and become attached.
ACTUAL RESULTS
Exception
Failed to update state:com.cloud.utils.exception.CloudRuntimeException: Failed to transit volume: 9964, due to: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Uploaded via OperationFailed
2018-07-05 10:31:26,763 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-39:ctx-49dbeb5a job-179272) (logid:5c3207f7) Unexpected exception while executing org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd
com.cloud.utils.exception.CloudRuntimeException: Failed to update state:com.cloud.utils.exception.CloudRuntimeException: Failed to transit volume: 9964, due to: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Uploaded via OperationFailed
at org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:332)
at org.apache.cloudstack.storage.volume.VolumeServiceImpl.copyVolumeFromImageToPrimary(VolumeServiceImpl.java:1325)
at org.apache.cloudstack.storage.volume.VolumeServiceImpl.copyVolume(VolumeServiceImpl.java:1413)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.copyVolumeFromSecToPrimary(VolumeOrchestrator.java:492)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.copyVolume(VolumeOrchestrator.java:797)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.createVolumeOnPrimaryStorage(VolumeOrchestrator.java:817)
at com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:1428)
at com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:3145)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:3178)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy201.handleVmWorkJob(Unknown Source)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-07-05 10:31:26,764 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-39:ctx-49dbeb5a job-179272) (logid:5c3207f7) Complete async job-179272, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to update state:com.cloud.utils.exception.CloudRuntimeException: Failed to transit volume: 9964, due to: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Uploaded via OperationFailed"}
Activity
[-]Uploaded Volume Can not be attached to VM with local storage[/-][+]Uploaded Volume cannot be attached to VM with local storage[/+]PaulAngus commentedon Aug 16, 2018
I've replicated the issue.
Trace log attached.
CloudStack DB queries look for storage scoped to CLUSTER and ZONE level but not HOST
therefore do not find local storage pools
failure log.txt
I suspect that the regression was caused by #2425
DaanHoogland commentedon Aug 16, 2018
@bwsw it seems to work as expected but maybe not with as clear an error message as you would want/expect. You probably did not grant the volume a disk-offering with local-storage enabled. If you create a disk offering with local storage and assign that to your uploaded volume it should work.
@PaulAngus maybe we should call in the troops to discuss further; like @rafaelweingartner and maybe @wido ?
rafaelweingartner commentedon Aug 16, 2018
Great idea, I already added this to my list. I will test this in the next coming days
bwsw commentedon Aug 16, 2018
@DaanHoogland may be you are right. Try to do that way. You know there are some problems with local data volumes which are not addressed: e.g. one can create data volume without specifying VMID and if that volume is created on another local storage than where BM resides, then it can not be attached, because local storage data vols are not migrated. But, it indeed,impossible to specify VMID in native UI when creating local data volume, so it doesn't work as expected. It seems not so much people run local storages, though.
PaulAngus commentedon Aug 16, 2018
@bwsw - I'll close this ticket. Can you create new tickets for each of the other issues that you have discovered.