Repository navigation
cloud-early-config detects unknown hypervisor type "xen-domU" #2561
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
systemVM image 4.11 for XenServer
CLOUDSTACK VERSION
4.11.0 as well as 4.11.1 git from today.
SystemVM: 4.11.0 Sun Jan 14 15:37:28 UTC 2018
CONFIGURATION
XenServer 6.5 SP1
OS / ENVIRONMENT
Management Server on CentOS 7.
SystemVMs running as "Debian 9.3" in HVM mode.
SUMMARY
Newly created system VMs are not configured, cloud.log reports "systemvm type=" (nothing else).
the /opt/cloud/bin/setup/cloud-early-config script tries to detect the hypervisor type, there are cases for "xen-pv" and "xen-hvm". But it uses "virt-what", which returns the unknown type "xen-domU".
After applying these fixes...
- add "xen-domU" to the "xen-hvm" case
- changing the path to xenstore-read from /usr/sbin to /usr/bin
and rebooting, the SystemVM comes up correctly.
Activity
yadvr commentedon Apr 11, 2018
@MartinEmrich the 4.11.0.0 systemvmtemplate cannot be used with 4.11.1.0-SNAPSHOT. There was a commit from @khos2ow that changed the xenserver pv vs hvm detection, this happens in cloud-early-config script that cannot be simply patched. I'll update 4.11.1.0 systemvmtemplate in next few weeks.
@khos2ow can you comment on the /usr/bin, /usr/sbin path change?
MartinEmrich commentedon Apr 11, 2018
@rhtyd Thanks for the hint, a new systemvm image is building, too, right now (No KVM in VirtualBox, so I removed "accelerator":"kvm" from the packer config and it will probably take ages...
MartinEmrich commentedon Apr 11, 2018
I built the new systemvm, but I have issues installing it:
During a previous update from 4.9.2.0 to 4.11.0, I installed the new systemvm via UI as "systemvm-xenserver-4.11". This maps to /secondary/templates/tmpl/2/204.
Using
cloud-install-sys-tmpltinstalls it into /secondary/templates/1/204.Newly created system vms use "systemvm-xenserver-4.11" (still the old one), which cannot be deleted. What's the safe way to replace a current systemvm template with the same version?
yadvr commentedon Apr 11, 2018
@MartinEmrich You register a template like you normally do, give it a name and change bunch of global settings, such as
router.template.xenserverto the new template's name, in the db in the vm_template details set type=SYSTEM. Btw I've the new 4.11 templates here: https://lab.yadav.cloud/systemvmtemplates/4.11/rafaelweingartner commentedon Apr 11, 2018
@rhtyd does it work for SSVM and console proxy? I was under the impression that this would only change the template for routers.
There is this PR to enable customizing system VMs template here: #2275
khos2ow commentedon Apr 11, 2018
@rhtyd this specific change doesn't require a new systemvm template and will be deployed on systemvms using
systemvm.iso.@MartinEmrich the file
/opt/cloud/bin/setup/cloud-early-configis the source file and you need to check if it's actually correctly updated in/etc/init.d/cloud-early-config.I'll have to look to see what's going on with
/usr/binand/usr/sbin.khos2ow commentedon Apr 11, 2018
@MartinEmrich afaik you should never use
cloud-install-sys-tmpltafter the first installation was done, it will mess up ssh key inside systemvm template and result in management server cannot talk to system vms.@rafaelweingartner yes it does, SSVM, CPVM work as is (with the process @rhtyd mentioned above) and VR gets to use the same new template through
router.template.xenserversetting.rafaelweingartner commentedon Apr 11, 2018
I am pretty sure it does not.... they get the systemvm.iso injected (if this is the only thing that matter to you), but I think the execution flow is different when it comes on selecting the template.
khos2ow commentedon Apr 11, 2018
Well, that's how we upgrade our systemvm templates when required.
typetoSYSTEM(in db)router.template.xenserverto use this new template (in db)rafaelweingartner commentedon Apr 11, 2018
@khos2ow ah... that might work because you change the type to system in the database. However, that might not be a sure thing. If you take a look into
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(long)andorg.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.startNew(long, Role). You will see that they do not look into that router parameter.Those methods simply use the method
com.cloud.storage.dao.VMTemplateDaoImpl.findSystemVMReadyTemplate(long, HypervisorType), which only looks for the first template it find.That is one of the reason for PR: #2275
yadvr commentedon Apr 11, 2018
@rafaelweingartner yes, as others has pointed out we use the
router.xxxto specify the systemvmtemplate name which gets used for all the systemvms (cpvm, ssvm, routers etc)khos2ow commentedon Apr 11, 2018
I've found a rather interesting, potential problem, on this. Hypervisor detection has been changed in bb96564,
virt-whatexecuted prior to check from/proc/xenor in my change at 535e615 to read fromdmesg.I just tested executing virt-what on both PV and HVM vms on my environment (Xen 7.1.1 and 7.1.0) and I'm getting conflicting results from
virt-what(needs more investigation) but I believe we need to read from/proc/xenanddmesgprior tovirt-what.MartinEmrich commentedon Apr 12, 2018
@rhtyd Thanks for the updated systemvm templates, I'd like to try them out.
But @khos2ow pointed out that cloud-install-sys-tmplt is not a good idea. But I cannot download it via UI either, as there's no working SSVM now.
So there's a chicken-and-egg problem...
I did the ugly thing, as this is just a test system: disabled all hosts and the zone, deleted all system VMs, removed all references from
vm_instance,template_store_ref,template_spool_refand the entry fromvm_template.I then installed your template via cloud-install-sys-tmplt, it installed to /1/1 this time.
after restarting and reenabling everything, two CPVM/SSVM pairs came up, only one was referenced in the DB. I deleted the orphaned pair from XenCenter.
The remaining pair is Debian 7, ACS 4.6.0 (the original installed during the first ACS setup with 4.9.2.0).
Looks like it was cached somewhere?
2 remaining items
yadvr commentedon Apr 20, 2018
this will be blocker if systemvms don't start/run properly on xenserver, therefore let's aim to fix this in 4.11.1.0
MartinEmrich commentedon Apr 20, 2018
@borisstoyanov that would be awesome, as we're stuck on 4.9 with XenServer <= 6.5 now.
Just trigger me when there's something new to test, maybe I can squeeze in an hour or two to try out new versions...
yadvr commentedon May 2, 2018
I've found the issue and testing the fix as part of #2616 /cc @MartinEmrich @khos2ow
yadvr commentedon May 2, 2018
Tested and validated the fix. @MartinEmrich you may help with testing the issue using #2616, I'll publish newer 4.11.1 systemvmtemplate later this week. /cc @khos2ow
I'll mark this issue as closed.
systemvm: Fixes apache#2561 patching on XenServer
systemvm: Fixes apache#2561 patching on XenServer
khos2ow commentedon May 2, 2018
Thanks @rhtyd about the fix, but I believe it's not complete as I mentioned in #2616 (comment).
systemvm: Fixes apache#2561 patching on XenServer
systemvm: Fixes apache#2561 patching on XenServer
systemvm: Fixes #2561 patching on XenServer
MartinEmrich commentedon May 7, 2018
@rhtyd Sorry for the delay... Lots of other stuff going on, and had issues with a circuit breaker on my Cloudstack test rig... Building git 4.11 now, I hope to get around to do some testing this week.
MartinEmrich commentedon May 15, 2018
Finally got my automated test bench working, and it is a success:
Installed 4.9.2.0, and upgraded to self-built 4.11.1 snapshot including system VM. the upgrade runs smooth, and my VMs still have connectivity with the 4.11.1 systemvm.
yadvr commentedon May 15, 2018
Thanks for sharing your results @MartinEmrich
Merge branch 'log-cpu-2561' into '4.20.0.0-scclouds'