Skip to content

VM stopped while Migration with Maintenance Mode (KVM) #3720

Closed
Closed
VM stopped while Migration with Maintenance Mode (KVM)#3720
Labels
component:kvmstatus:needs-functional-definitiontype:bug
Milestone
unplanned

Description

@svenvogel
svenvogel
opened on Nov 26, 2019
ISSUE TYPE
  • Bug Report
COMPONENT NAME
Management Server
CLOUDSTACK VERSION
Master but maybe 4.8 upwards
CONFIGURATION
OS / ENVIRONMENT

KVM with QEMU 2.12 and libwirtd 5.0.0, i think other other versions are also affected because not always there is the problem.

SUMMARY
STEPS TO REPRODUCE
  1. Set a Host in Maintenance Mode
  2. VM Migration is started
  3. If a VM cant migrate it will hard stopped

after @andrijapanicsb started a discussion on mailing list "VMs crashing/stopped during live migration?" i opened this issue request to track down the problems.

@ustcweizhou gave some interesting hints to this problem.

i picked up specially your point three. i think its most interesting to track this down.

  1. vm is stopped due to some cloudstack bugs. for example, when we put a
    host to maintenance, the vm will be stopped if (1) no other host is Up in
    same cluster, or (2) vm migration fails and no retry in cloudstack, or (3)
    multiple vms are migrated to same destination at the same time but there is
    no enough memory on the destination.
EXPECTED RESULTS
The expected result should be to stop the maintenance mode but dont switch off the virtual machines
ACTUAL RESULTS

virtual machines will be hard stopped

Activity

  1. svenvogel
    added
    component:kvm
    type:bug
    on Nov 26, 2019

  2. svenvogel commented on Nov 26, 2019

    @svenvogel
    svenvogel
    on Nov 26, 2019
    ContributorAuthor

    @ustcweizhou do you have some idea how we can these part 3?

  3. andrijapanicsb commented on Nov 26, 2019

    @andrijapanicsb
    andrijapanicsb
    on Nov 26, 2019 via email
    Contributor

  4. weizhouapache commented on Nov 27, 2019

    @weizhouapache
    weizhouapache
    on Nov 27, 2019
    Member

    @svenvogel @andrijapanicsb

    I think @andrijapanicsb 's suggestion is good. Suppose we have a new parameter forceMaintenance in the command

    (1) if there are vms running on the host. if there is no any other server in Up status and Enabled state in the cluster
    (1.1) if forceMaintenance = true, stop all vms
    (1.2) if forceMaintenance = false, throw exception, host is put to ErrorInMaintenance or Up status.

    (2) if there are other servers in the cluster, all vms will be migrated away from the host
    (2.1) if all vm migrations succeed, then host is put to Maintenance.
    (2.2) if vm migration failed, then try to migrate vm to another host . If vm migration failed after 5 times (this can be configured)), then
    (2.2.1) if forceMaintenance = true, schedule a task to stop/start the vm.
    (2.2.2) if forceMaintenance = false, throw exception, host is put to ErrorInMaintenance.

    any suggestion ?

  5. svenvogel commented on Nov 27, 2019

    @svenvogel
    svenvogel
    on Nov 27, 2019
    ContributorAuthor

    @ustcweizhou @andrijapanicsb i think this sounds good.

    one question.

    (2.2.1) if forceMaintenance = true, schedule a task to stop/start the vm.

    @ustcweizhou why we should stop and start them?

    additional we encounter the following problem.

    1. iso could not be mounted or storage pool on destination host, i dont know why this is a problem, manually mounting was no problem. maybe anyone has information about that?
    2. if QEMU or Host Version version are different machine type will change the flags <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type>
      i think this should be on view of the infrastructure admin.

  6. andrijapanicsb commented on Nov 27, 2019

    @andrijapanicsb
    andrijapanicsb
    on Nov 27, 2019
    Contributor

    @weizhouapache that sounds OK to me - I assume you handle capacity issues on a per VM basis.

    @svenvogel 2.2.1 - if VM can't be live migrated - then stop it and start on another host to avoid permanent downtime - makes sense perfectly (if forceMaint=true)

  7. andrijapanicsb commented on Nov 27, 2019

    @andrijapanicsb
    andrijapanicsb
    on Nov 27, 2019
    Contributor

    CC @anuragaw since you were aware of the issue and want to review any code produced...

  8. andrijapanicsb commented on Nov 27, 2019

    @andrijapanicsb
    andrijapanicsb
    on Nov 27, 2019
    Contributor

    CC @rhtyd @PaulAngus @nvazquez

  9. weizhouapache commented on Nov 27, 2019

    @weizhouapache
    weizhouapache
    on Nov 27, 2019
    Member

    @ustcweizhou @andrijapanicsb i think this sounds good.

    one question.

    (2.2.1) if forceMaintenance = true, schedule a task to stop/start the vm.

    @ustcweizhou why we should stop and start them?

    @svenvogel if vm cannot be migrated after 5 times, to continue with the maintenance (because forceMaintenance is true), we need to stop it and start it on another server. make sense ?

    additional we encounter the following problem.

    1. iso could not be mounted or storage pool on destination host, i dont know why this is a problem, manually mounting was no problem. maybe anyone has information about that?

    yes, it is one root cause of failed migration.

    1. if QEMU or Host Version version are different machine type will change the flags <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type>
      i think this should be on view of the infrastructure admin.

    If hosts have different cpu model or qemu version, they should use custom cpu or cpu flags. It is not an issue in my point of view.

  10. andrijapanicsb commented on Nov 27, 2019

    @andrijapanicsb
    andrijapanicsb
    on Nov 27, 2019
    Contributor

    Agree on that last point of QEMU or HOST versions - not an issue and has been like that forever (nor I think would be possible since that would change what the OS sees and might crash it).

  11. svenvogel commented on Nov 27, 2019

    @svenvogel
    svenvogel
    on Nov 27, 2019
    ContributorAuthor

    @andrijapanicsb @ustcweizhou yes make sense. Agree. now i understand it.

  12. GabrielBrascher commented on Nov 27, 2019

    @GabrielBrascher
    GabrielBrascher
    on Nov 27, 2019
    Member

    @andrijapanicsb “forceMaintenance” is something that could help indeed; in some specific corner cases it would be interesting to have such a workaround.

    I believe the biggest problem I saw was VM crashing due to qemu bugs/issue (i.e. recently saw this one “qemu-kvm general protection error in libtcmalloc.so.4.4.5”) – can’t speak for other issues, since I don’t recall having them with ACS 4.5/4.8 with Ubuntu 14.04 (qemu 2.0 initially, later upgraded to 2.5) If not mistaken, there is indeed a behavior that if there is no enough capacities in the cluster (test with a single KVM host in the cluster), the VMs will be stopped – and that is a thing to change in my opinion. Maintenance should fail and host should be in ErrorInMaintenance mode. If you want to do maintenance of that host, put the storage into the maintenance mode and you are good to go. Even better, make an additional parameter like “forceMaintenance” which will indeed stop VMs in order to reach Maintenance mode. Othewise will simply be in ErrorInMaintenance.

  13. anuragaw commented on Nov 27, 2019

    @anuragaw
    anuragaw
    on Nov 27, 2019
    Contributor

    My 2 cents - in case of single host adding forceMaintenance is really going to complicate code unnecessarily. Marking host to maintenance and migration jobs for VMs are separate background tasks. Putting a host into maintenance is not going to be a frequent operation inmho worth the code complexity.

    Perhaps sticking to the general theory - If Migrations cannot be completed and host cannot be put to maintenance safely, we put the host to ErrorInMaintenance state. At the same time we log/throw exception for informing the admins.

    Admins can put the VMs off or snapshot or whatever works best - then they can manually shutdown all VMs on a host as needed and retry maintenance transition.

    Thoughts @andrijapanicsb , @svenvogel , @GabrielBrascher , @ustcweizhou?

  14. weizhouapache commented on Nov 27, 2019

    @weizhouapache
    weizhouapache
    on Nov 27, 2019
    Member

    let's wait @anuragaw 's PR #3425 to be done.

  15. 3 remaining items

  16. weizhouapache commented on Dec 3, 2019

    @weizhouapache
    weizhouapache
    on Dec 3, 2019
    Member

    @svenvogel @andrijapanicsb

    vm is stopped due to some cloudstack bugs. for example, when we put a
host to maintenance, the vm will be stopped if (1) no other host is Up in
same cluster, or (2) vm migration fails and no retry in cloudstack, or (3)
multiple vms are migrated to same destination at the same time but there is
no enough memory on the destination.

    (1) has been tracked in #3707 and discussed by some comments in this ticket.
    (2) is changed in #3425 but not done.
    (3) is fixed by #3727 and #3728. which caused crashed vm in host maintenance on our platforms.

  17. yadvr commented on Mar 3, 2020

    @yadvr
    yadvr
    on Mar 3, 2020
    Member

    Is this fixed/addressed @svenvogel @weizhouapache @andrijapanicsb ?

  18. svenvogel commented on Mar 6, 2020

    @svenvogel
    svenvogel
    on Mar 6, 2020
    ContributorAuthor

    @rhtyd we will test this in the next week. i lost this in our track.

  19. svenvogel
    changed the title [-]VM stopped/crashing while Migration with Maintenance Mode (KVM)[/-] [+]VM stopped while Migration with Maintenance Mode (KVM)[/+] on Mar 6, 2020

  20. svenvogel commented on Mar 6, 2020

    @svenvogel
    svenvogel
    on Mar 6, 2020
    ContributorAuthor

    @andrijapanicsb changed. yes it stopped the VM thats right.

  21. DaanHoogland commented on Dec 11, 2020

    @DaanHoogland
    DaanHoogland
    on Dec 11, 2020
    Contributor

    this is on hold it seems, any volunteers to specify this and #3707 to sum up the desired outcome ? it seems to me there are a lot of ideas here but no consensus on functionality, @svenvogel @andrijapanicsb @weizhouapache @rhtyd @borisstoyanov @GabrielBrascher @anuragaw .

  22. GabrielBrascher commented on Dec 15, 2020

    @GabrielBrascher
    GabrielBrascher
    on Dec 15, 2020
    Member

    My 2 cents on this one would be to add a force maintenance parameter or global settings (I would prefer a force parameter though).

    Here follows a draft of a proposal for this issue:

    1. call maintenance of a host with force parameter
    2. normal flow where CloudStack tries to migrate all VMs
    3. if not possible to migrate a VM, stop it
    4. place the host in maintenance state once all VMs are migrated or stopped

    PR #4212 is focused on migrating or stopping local storage VMs of a host put in maintenance. However, it holds some similarities with this issue.

  23. DaanHoogland
    added
    status:needs-functional-definition
    on Feb 1, 2021
  24. DaanHoogland
    added this to the unplanned milestone on Feb 1, 2021

  25. yadvr commented on Jun 30, 2022

    @yadvr
    yadvr
    on Jun 30, 2022
    Member

    is this still an issue @GabrielBrascher @svenvogel and others?

  26. GabrielBrascher commented on Jun 30, 2022

    @GabrielBrascher
    GabrielBrascher
    on Jun 30, 2022
    Member

    @rohityadavcloud @svenvogel should be fixed by PR #4212 which gives autonomy for admins to choose a strategy for such VMs, via global settings name host.maintenance.local.storage.strategy.

    With that PR Admins can choose (wisely and knowing the eventual risks of each option) between:

    1. error in maintenance when there are local storage VMs
    2. stop local storage VMs and put in maintenance
    3. or. migrate local storage VMs and put it in maintenance

    Might require some upgrades on CloudStack documentation thought.

    I would be +1 on closing this issue. Any thoughts @svenvogel?

  27. yadvr commented on Aug 17, 2022

    @yadvr
    yadvr
    on Aug 17, 2022
    Member

    As per comment on PR, it appears fixed by #4534 and #4212 is also merged
    Pl re-open if we got that wrong, thanks.

  28. yadvr
    closed this as completedon Aug 17, 2022
  29. yadvr
    removed this from the unplanned milestone on Aug 17, 2022
  30. DaanHoogland
    added this to the unplanned milestone on Sep 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    component:kvmstatus:needs-functional-definitiontype:bug

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions