Linstor VM/Volume snapshots #9360

Activity

1. UAnton commented on Jul 9, 2024

   

   UAnton

   on Jul 9, 2024

   Author

   More actions

   2024-07-09 10:18:15,953 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-4:ctx-753b7118 job-52 ctx-7d1d55c9) (logid:feb78585) Failed to create snapshot due to an internal error creating snapshot for volume 12bf6956-591f-4f53-a1a1-36cb932752e0
   java.lang.RuntimeException: Unexpected exception
           at com.cloud.storage.VolumeApiServiceImpl.takeSnapshotInternal(VolumeApiServiceImpl.java:3528)
           at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:3445)
           at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
           at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           at java.base/java.lang.reflect.Method.invoke(Method.java:566)
           at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
           at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
           at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
           at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
           at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
           at com.sun.proxy.$Proxy236.takeSnapshot(Unknown Source)
           at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:228)
           at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
           at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
           at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
           at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
           at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
           at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
           at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
           at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
           at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
           at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
           at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
           at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
           at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
           at java.base/java.lang.Thread.run(Thread.java:829)
   Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to unhide zfs snapshot device.
           at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:382)
           at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:193)
           at com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSecondary(SnapshotManagerImpl.java:1439)
           ... 45 more
   2024-07-09 10:18:15,955 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-4:ctx-753b7118 job-52) (logid:feb78585) Complete async job-52, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 12bf6956-591f-4f53-a1a1-36cb932752e0"}
   
2. DaanHoogland

   added

   has-3rdparty-components

   3rd-party:linstor

   on Jul 9, 2024
3. DaanHoogland

   added this to Apache CloudStack BugFest - Issueson Jul 9, 2024
4. DaanHoogland commented on Jul 12, 2024

   

   DaanHoogland

   on Jul 12, 2024

   Last edited by DaanHoogland

   Contributor

   More actions

   I see I made some assumption that might not be true. Is this a linstor environment @UAnton ? what storage type(s) did you try this with?

   (I was looking at the title, so maybe not so wrong after all)

   can you have a look as well @rp- ?
5. DaanHoogland

   removed

   has-3rdparty-components

   3rd-party:linstor

   on Jul 12, 2024
6. DaanHoogland

   added this to the 4.19.2.0 milestone on Jul 12, 2024
7. DaanHoogland

   added

   has-3rdparty-components

   3rd-party:linstor

   on Jul 12, 2024
8. UAnton commented on Jul 12, 2024

   

   UAnton

   on Jul 12, 2024

   Author

   More actions

   @DaanHoogland I use Linstor. When I try to take a snapshot, I get an error in the UI and don’t see the snapshot, but the snapshot is created in the Linstor storage
9. rp- commented on Jul 15, 2024

   

   rp-

   on Jul 15, 2024

   Contributor

   More actions

   Well the exception kinda tells what the problem is:
   Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to unhide zfs snapshot device.

   We copy the data to secondary storage, from the zfs snapshot. But zfs snapshots are not directly accessible by default
   and you have to unhide the block device. For some reason the unhide fails on your setup.
   Can you try to manually unhide such a snapshot to see the error? e.g.:
   zfs set snapdev=visible <zfs-snapshotname>

   Or we need the agent debug output of the failed request.
10. sureshanaparti

    added this to deprecated: Apache CloudStack - Issues Aug'2024on Aug 9, 2024
11. DaanHoogland

    moved this to Todo in Apache CloudStack BugFest - Issueson Aug 13, 2024
12. DaanHoogland

    moved this to Todo in deprecated: Apache CloudStack - Issues Aug'2024on Aug 13, 2024
13. DaanHoogland

    moved this from Todo to No status in deprecated: Apache CloudStack - Issues Aug'2024on Aug 14, 2024
14. DaanHoogland

    removed this from deprecated: Apache CloudStack - Issues Aug'2024on Nov 8, 2024
15. DaanHoogland

    assigned

    rp-

    and

    UAnton

    on Jan 2, 2025
16. rp-

    linked a pull request that will close this issuelinstor: Fix ZFS snapshot backup #10219on Jan 20, 2025
17. harikrishna-patnala commented on Jan 21, 2025

    

    harikrishna-patnala

    on Jan 21, 2025

    Member

    More actions

    #10219 related PR is merged.
18. harikrishna-patnala

    closed this as completedon Jan 21, 2025
19. github-project-automation

    moved this from Todo to Done in Apache CloudStack BugFest - Issueson Jan 21, 2025
20. DaanHoogland

    removed this from Apache CloudStack BugFest - Issueson Aug 22, 2025