Skip to content

Upgrade from 4.14 to 4.15 fails when using XenServer 7.0/7.1/7.4 #4502

Closed
Closed
Upgrade from 4.14 to 4.15 fails when using XenServer 7.0/7.1/7.4#4502
Assignees
vladimirpetrov
Labels
Severity:CriticalCritical bugcomponent:XenServercomponent:upgradetype:bug
Milestone
4.15.0.0

Description

@vladimirpetrov
vladimirpetrov
opened on Nov 24, 2020
ISSUE TYPE
  • Bug Report
COMPONENT NAME
Upgrade
CLOUDSTACK VERSION
Upgrade from 4.14 to 4.15
CONFIGURATION

2 hosts using XenServer 7.0/7.1/7.4

OS / ENVIRONMENT

XenServer 7.0(7.1, 7.4), two hypervisor hosts, CentOS 7.6 as management server OS.

SUMMARY

After upgrading from 4.14 to 4.15 and destroying the system VMs, they are not able to start again.

STEPS TO REPRODUCE
1. Install CloudStack 4.14 with XenServer 7.0 (7.1, 7.4), create some test objects - VMs, snapshots, networks, etc.
2. Follow the standard upgrade procedure
3. After the upgrade, destroy the system VMs (as described in the documentation)
EXPECTED RESULTS
Successful upgrade.
ACTUAL RESULTS
The new system VMs are not able to start, here is what is shown in the log:

2020-11-18 21:48:00,312 ERROR [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-8:ctx-5c32b189 job-71/job-72 ctx-c90dd860) (logid:84ab8c0f) Failed to start instance VM[DomainRouter|r-14-VM]
com.cloud.utils.exception.ExecutionException: Unable to start  VM:4acfa807-b78e-48f0-b615-177c9a9c954d due to error in finalizeStart, not retrying
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1266)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5461)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5628)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

After some investigations with @rhtyd , it turned out the reason is that the systemvm.iso file is not detected by the host VM.

Activity

  1. DaanHoogland
    added this to the 4.15.0.0 milestone on Nov 25, 2020
  2. DaanHoogland
    added
    component:upgrade
    component:XenServer
    Severity:Major
    type:bug
    on Nov 25, 2020
  3. vladimirpetrov
    changed the title [-]Upgrade from 4.14 to 4.15 fails when using XenServer 7.0/7.1[/-] [+]Upgrade from 4.14 to 4.15 fails when using XenServer 7.0/7.1/7.4[/+] on Nov 25, 2020

  4. vladimirpetrov commented on Nov 25, 2020

    @vladimirpetrov
    vladimirpetrov
    on Nov 25, 2020
    ContributorAuthor

    I just tested that the issue is reproducible also with XenServer 7.4.

  5. yadvr commented on Nov 25, 2020

    @yadvr
    yadvr
    on Nov 25, 2020
    Member

    @vladimirpetrov to confirm regression can you do an upgrade of a XenServer 7.1 env from 4.11 or 4.13 to 4.14 and see how the system behaves? If you get the same issue we can lower the severity and address workaround in upgrade docs.

  6. vladimirpetrov commented on Nov 27, 2020

    @vladimirpetrov
    vladimirpetrov
    on Nov 27, 2020
    ContributorAuthor

    ACS 4.11 -> 4.14
    ACS 4.13 -> 4.14
    ACS 4.11 -> 4.15 and
    ACS 4.13 -> 4.15 (all with XenServer 7.1) upgrades passed successfully.

  7. andrijapanicsb commented on Nov 27, 2020

    @andrijapanicsb
    andrijapanicsb
    on Nov 27, 2020
    Contributor

    Can you share more logs/info to let others also understand the issue as well.

    Also, I see " CentOS 7.0 on the hosts." in the description, but you are saying it's the XS setup - do you have mixed hypervisors or why is there " CentOS 7.0 on the hosts." ?

  8. yadvr
    assigned
    DaanHoogland
    and
    PaulAngus
    on Nov 30, 2020

  9. DaanHoogland commented on Nov 30, 2020

    @DaanHoogland
    DaanHoogland
    on Nov 30, 2020
    Contributor

    ACS 4.11 -> 4.14
    ACS 4.13 -> 4.14
    ACS 4.11 -> 4.15 and
    ACS 4.13 -> 4.15 (all with XenServer 7.1) upgrades passed successfully.

    given that all these succeed and 4.14 -> 4.15 fail, there is likely to be something wrong in the upgrade path. Can you share the log of the initial start after upgrade @vladimirpetrov ,
    tnx

  10. yadvr commented on Dec 1, 2020

    @yadvr
    yadvr
    on Dec 1, 2020
    Member

    My suspicion is that 4.14 systemvms (ssvm, cpvm and VRs) don't eject or fail to eject the systemvm.iso file. Can we confirm that @vladimirpetrov ?

  11. DaanHoogland commented on Dec 2, 2020

    @DaanHoogland
    DaanHoogland
    on Dec 2, 2020
    Contributor

    ah, if you ar right, that would than pertain to higher upgrade paths as well (4.15 -> future), making this slightly more pressing

  12. 5 remaining items

  13. andrijapanicsb commented on Dec 7, 2020

    @andrijapanicsb
    andrijapanicsb
    on Dec 7, 2020
    Contributor

    This need be fixed before 4.15 can go out.

  14. yadvr commented on Dec 7, 2020

    @yadvr
    yadvr
    on Dec 7, 2020
    Member

    I setup 4.14 + XS71 and did an upgrade test and was unable to reproduce this. I suspect in the upgraded env the old ssvm/cpvm were killed and new ssvm/cpvm got started before the 4.15 systemvm.iso/files/scripts were patched on the XS71 hosts which could explain the race conditions around old 4.14 systemvm.iso.

  15. yadvr
    added
    Severity:CriticalCritical bug
    and removed
    Severity:BLOCKER
    on Dec 7, 2020

  16. yadvr commented on Dec 7, 2020

    @yadvr
    yadvr
    on Dec 7, 2020
    Member

    Lowered from blocker as we don't consistently reproduce this, let's wait for @vladimirpetrov @andrijapanicsb and others if they can confirm it works or doesn't work. Thanks.

  17. PaulAngus commented on Dec 7, 2020

    @PaulAngus
    PaulAngus
    on Dec 7, 2020 via email
    Member

  18. andrijapanicsb commented on Dec 7, 2020

    @andrijapanicsb
    andrijapanicsb
    on Dec 7, 2020 via email
    Contributor

  19. yadvr commented on Dec 8, 2020

    @yadvr
    yadvr
    on Dec 8, 2020
    Member

    I did an upgrade test and discussion with Vladi, the original issue was due to not waiting enough after an upgrade for the management server to copy all the systemvm.iso/files/scripts to the XS host post-upgrade.

    In the latest upgrade env created by Vladi, from 4.14->4.15 with XS71, we couldn't reproduce the issue. However we had to restart the systemvms a few times due to a certificate setup issue likely caused due to:

    • CPU pressure - XS hosts are run inside VM, some cpu entropy is required to create keystore and certificates - this applies to only to SSVM and CPVM
    • Once SSVM and CPVM were up and connected, we were able to deploy a new VM on a new network that came up OK
    • We found that VRs failed on XS hosts with already too many VMs - VRs appeared to run slow on these hosts and had errors in running the get_template_version domr script, this didnot happen on XS hosts with less num. of VMs

    From experience I've seen VMs runs slow on XS65, 71; they appear to run faster on more recent XS version (it could also be a hvm setting).

    @vladimirpetrov can you add your notes/findings and confirmation.

  20. yadvr
    assigned
    vladimirpetrov
    and unassigned
    DaanHoogland
    and
    PaulAngus
    on Dec 9, 2020

  21. yadvr commented on Dec 9, 2020

    @yadvr
    yadvr
    on Dec 9, 2020
    Member

    PR fix #4525 pl test and review @vladimirpetrov thanks

  22. vladimirpetrov commented on Dec 9, 2020

    @vladimirpetrov
    vladimirpetrov
    on Dec 9, 2020
    ContributorAuthor

    The fix worked - well done, @rhtyd !

  23. yadvr commented on Dec 9, 2020

    @yadvr
    yadvr
    on Dec 9, 2020
    Member

    Fixed in #4525, kicking smoketests on health check PR #4131

  24. yadvr
    closed this as completedon Dec 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Severity:CriticalCritical bugcomponent:XenServercomponent:upgradetype:bug

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions