Repository navigation
KVM: createVolume / createTemplate from incremental snapshot fails randomly, managesnapshot.sh skips mkdir #14485
Description
problem
With kvm.incremental.snapshot=true and snapshot.backup.to.secondary=false, createVolume or createTemplate from an incremental snapshot fails randomly with:
errorcode 530
Failed to copy snapshot: java.lang.RuntimeException: InvocationTargetException when invoking RPC callback for command: copySnapshotAsyncCallback
The agent log only says Failed to backup <uuid> ... to <secondary>/snapshots/<account>/<volume>. The real error is hidden because managesnapshot.sh sends qemu-img's output to /dev/null. I traced it on the host:
qemu-img convert ... : Could not create '<secondary>/snapshots/2/3/<uuid>': No such file or directory
What happens:
- Both calls copy the snapshot into
snapshots/<account>/<volume>on secondary. After the call, the SSVM deletes the copy and also removes the folder if it's empty (NfsSecondaryStorageResource.java#L2116-L2120). - On the next copy,
backup_snapshot()only creates the folder if[ ! -d $destPath ](managesnapshot.sh#L237-L245). - If the host has the same secondary export mounted somewhere else (configdrive, template), the NFS client can still see the folder the SSVM just removed. mkdir is skipped and qemu-img fails (managesnapshot.sh#L285).
Every failed copy I traced had no mkdir before qemu-img. Every good one had it.
versions
- CloudStack 4.22.1.1, same code on main (0a5bf30)
- KVM: Ubuntu 26.04, QEMU 10.2.1, libvirt 12.0.0
- NFS 4.2 primary and secondary (Linux NFS server)
The steps to reproduce the bug
- KVM zone with NFS secondary. Set
kvm.incremental.snapshot=trueandsnapshot.backup.to.secondary=false, restart the management server. - On a KVM host, keep a subfolder of the same secondary export mounted, like a configdrive mount does:
mount -t nfs -o vers=4.2 <secstore>:/<export>/configdrive /mnt/keep - Deploy a VM on that host and take a full snapshot of its ROOT volume.
- Loop: createSnapshot (incremental), then createVolume from it, then createTemplate from it.
Without step 2 I got 0 failures in 8 rounds. With it, 6 of 12 rounds failed, all with the error above.
Same failure shows in the UI ("Failed to create volume"):
ui-repro.webm
What to do about it?
- Run
mkdir -p "$destPath"every time inbackup_snapshot(). It's safe if the folder exists. - Keep qemu-img's error in the message sent back, so this shows up in the logs.
- Maybe don't remove
snapshots/<account>/<volume>when the temp copy is deleted.
Activity
github-actions commentedon Oct 30, 2026
Warning
threat detection engine error
The threat detection engine encountered an error and could not complete analysis. This is a tooling failure, not a security finding.
What happened
The threat detection engine failed to produce results.
Review the workflow run logs for details.
🎯 Triage report
With
kvm.incremental.snapshot=trueandsnapshot.backup.to.secondary=false, createVolume or createTemplate from an incremental snapshot fails intermittently with error 530. The reporter traced it tomanagesnapshot.sh:backup_snapshot()creates the destination folder only when it is missing, and the SSVM can remove that folder after a copy. On the host,qemu-img convertthen fails with "No such file or directory". The reporter reproduced it with a second NFS mount of the same export on the host (for example a configdrive mount) and saw 6 of 12 rounds fail. Without the extra mount there were 0 failures in 8 rounds.📊 Assessment
managesnapshot.shand KVM snapshot pathmkdir -p "$destPath"unconditionally, pass qemu-img stderr back, and stop removing the account/volume folder🔗 Similar issues
KvmFileBasedStorageVmSnapshotStrategy💡 Notes and suggestions
Add this agentic workflow to your repo
To install this agentic workflow, run