Repository navigation
Backup related upgrade failure from 4.20.1/4.20.2 to 4.21.0/4.22.0 #12769
Description
problem
When upgrading from either 4.20.1 or 4.20.2 to 4.21.0 or 4.22.0, upgrade could fail when backup framework is in use.
An error like the following is seen,
2026-03-09 01:56:55,403 DEBUG [c.c.u.d.ScriptRunner] (main:[]) (logid:) -- Add diskOfferingId, deviceId, minIops and maxIops to backed_volumes in backups table
2026-03-09 01:56:55,403 DEBUG [c.c.u.d.ScriptRunner] (main:[]) (logid:) UPDATE `cloud`.`backups` b INNER JOIN `cloud`.`vm_instance` vm ON b.vm_id = vm.id SET b.backed_volumes = ( SELECT CONCAT("[", GROUP_CONCAT( CONCAT( "{\"uuid\":\"", v.uuid, "\",", "\"type\":\"", v.volume_type, "\",", "\"size\":", v.`size`, ",", "\"path\":\"", IFNULL(v.path, 'null'), "\",", "\"deviceId\":", IFNULL(v.device_id, 'null'), ",", "\"diskOfferingId\":\"", doff.uuid, "\",", "\"minIops\":", IFNULL(v.min_iops, 'null'), ",", "\"maxIops\":", IFNULL(v.max_iops, 'null'), "}" ) SEPARATOR "," ), "]") FROM `cloud`.`volumes` v LEFT JOIN `cloud`.`disk_offering` doff ON v.disk_offering_id = doff.id WHERE v.instance_id = vm.id )
2026-03-09 01:56:55,458 ERROR [c.c.u.d.ScriptRunner] (main:[]) (logid:) Error executing: UPDATE `cloud`.`backups` b INNER JOIN `cloud`.`vm_instance` vm ON b.vm_id = vm.id SET b.backed_volumes = ( SELECT CONCAT("[", GROUP_CONCAT( CONCAT( "{\"uuid\":\"", v.uuid, "\",", "\"type\":\"", v.volume_type, "\",", "\"size\":", v.`size`, ",", "\"path\":\"", IFNULL(v.path, 'null'), "\",", "\"deviceId\":", IFNULL(v.device_id, 'null'), ",", "\"diskOfferingId\":\"", doff.uuid, "\",", "\"minIops\":", IFNULL(v.min_iops, 'null'), ",", "\"maxIops\":", IFNULL(v.max_iops, 'null'), "}" ) SEPARATOR "," ), "]") FROM `cloud`.`volumes` v LEFT JOIN `cloud`.`disk_offering` doff ON v.disk_offering_id = doff.id WHERE v.instance_id = vm.id )
2026-03-09 01:56:55,475 ERROR [c.c.u.d.ScriptRunner] (main:[]) (logid:) java.sql.SQLException: Row 19 was cut by GROUP_CONCAT()
2026-03-09 01:56:55,495 ERROR [c.c.u.DatabaseUpgradeChecker] (main:[]) (logid:) Unable to execute upgrade script java.sql.SQLException: Row 19 was cut by GROUP_CONCAT()
at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:186)
at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:88)
This is due to the following DB changes in the 4.20.1to4.21.0 upgrade path,
https://github.com/apache/cloudstack/blob/main/engine/schema/src/main/resources/META-INF/db/schema-42010to42100.sql#L686-L709
The error is due to the length limit for group_concat_max_len in MySQL and it could be seen if there are backups for VMs having multiple volumes (possible 3 or more).
Workaround is setting value for group_concat_max_len:
SET GLOBAL group_concat_max_len = 1000000;
versions
Upgrade 4.20.x to 4.21/4.22
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
- Status
Activity
rajujith commentedon Apr 9, 2026
@blueorangutan package
DaanHoogland commentedon Apr 9, 2026
did this on the linked PR @rajujith ;)
daviftorres commentedon Apr 9, 2026
Hey Team,
Here is our Dev environment:
And this is our Prod environment:
Is it safe so say that our Prod went out of tree for upgrade paths?
How could we upgrade to
4.22.0.0? Is there any way we can craft an upgrade path to get out of the4.20.2.0, back to the rolling path of upgrades?Regards,
rajujith commentedon Apr 10, 2026
@daviftorres I would suggest upgrading to 4.22.1 once it is available, expecting to release in 1-2 weeks.
sureshanaparti commentedon Apr 10, 2026
Fixed in #12777
daviftorres commentedon May 11, 2026
Dear @rajujith ,
I see that a new release happened a few days ago, 4.22.0.1.
In the release notes I found this:
Does it also created the upgrade path from 4.20.2.0?
Regards,
abh1sar commentedon May 13, 2026
@daviftorres 4.22.0.1 was a security release. Although released recently, It doesn't contain the fix for the upgrade issue.
The upgrade fix is present in the 4.22.1.0 LTS release which should be out in a few weeks.
abh1sar commentedon May 13, 2026
@daviftorres Please use the workaround if you want to upgrade to 4.22.0.1
daviftorres commentedon May 13, 2026
Thank you. The workaround worked in a test environment and we are planning to upgrade PROD in the next 2 or 3 weeks.
Nicolas shared with me the instructions to upgrade our Non-Prod environments (Dev/Test/Stg/PoC) to 4.22.1.0-RC1 (https://download.cloudstack.org/testing/4.22.1.0-RC1/), and we are already testing it too.
Thank you!