Skip to content

microcode-20200609 Release, at least 06-4e-03, hangs user's system #31

Open

Description

@vicamo

Per reports from https://bugs.launchpad.net/ubuntu/+source/intel-microcode/+bug/1882890, sig 0x000406e3, pf_mask 0xc0, 2020-04-27, rev 0x00dc, size 104448 hangs user's system in a similar way as #24 does but to different cpu. It's also reported the same file of a previous revision microcode: sig=0x406e3, pf=0x80, revision=0xd6 works just fine.

Activity

  1. andyhhp commented on Jun 10, 2020

    @andyhhp

    The differences between 0xd6 and 0xdc were quite minimal, to work around an SGX issue.

    Does disabling SGX in the BIOS work around the hang using 0xdc?

  2. eguaj commented on Jun 10, 2020

    @eguaj

    As requested by @vicamo, here is the output from lscpu:

    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                4
    On-line CPU(s) list:   0-3
    Thread(s) per core:    2
    Core(s) per socket:    2
    Socket(s):             1
    NUMA node(s):          1
    Vendor ID:             GenuineIntel
    CPU family:            6
    Model:                 78
    Model name:            Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
    Stepping:              3
    CPU MHz:               700.042
    CPU max MHz:           2800.0000
    CPU min MHz:           400.0000
    BogoMIPS:              4800.00
    Virtualization:        VT-x
    L1d cache:             32K
    L1i cache:             32K
    L2 cache:              256K
    L3 cache:              3072K
    NUMA node0 CPU(s):     0-3
    Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
    

    It's on a Lenovo T460 laptop.

  3. eguaj commented on Jun 10, 2020

    @eguaj

    The differences between 0xd6 and 0xdc were quite minimal, to work around an SGX issue.

    Does disabling SGX in the BIOS work around the hang using 0xdc?

    Disabling SGX does not seems to help: SGX was initially on "Software controlled", so changed it to "Disabled", but laptop still hangs when booting the new initrd image.

  4. andyhhp commented on Jun 10, 2020

    @andyhhp

    Thanks for testing. I think this needs Intel to investigate now

  5. pjssilva commented on Jun 10, 2020

    @pjssilva

    Same thing on my system with a similar CPU. I had to uninstall the intel-microcode package to boot. here is the output of lscpu:

    Arquitetura:                     x86_64
    Modo(s) operacional da CPU:      32-bit, 64-bit
    Ordem dos bytes:                 Little Endian
    Address sizes:                   39 bits physical, 48 bits virtual
    CPU(s):                          4
    Lista de CPU(s) on-line:         0-3
    Thread(s) per núcleo:            2
    Núcleo(s) por soquete:           2
    Soquete(s):                      1
    Nó(s) de NUMA:                   1
    ID de fornecedor:                GenuineIntel
    Família da CPU:                  6
    Modelo:                          78
    Nome do modelo:                  Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz
    Step:                            3
    CPU MHz:                         500.006
    CPU MHz máx.:                    3000,0000
    CPU MHz mín.:                    400,0000
    BogoMIPS:                        4999.90
    Virtualização:                   VT-x
    cache de L1d:                    64 KiB
    cache de L1i:                    64 KiB
    cache de L2:                     512 KiB
    cache de L3:                     3 MiB
    CPU(s) de nó0 NUMA:              0-3
    Vulnerability Itlb multihit:     KVM: Mitigation: Split huge pages
    Vulnerability L1tf:              Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
    Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT vulnerable
    Vulnerability Meltdown:          Mitigation; PTI
    Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
    Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
    Vulnerability Spectre v2:        Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
    Vulnerability Srbds:             Vulnerable: No microcode
    Vulnerability Tsx async abort:   Mitigation; Clear CPU buffers; SMT vulnerable
    Opções:                          fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pn
                                     i pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb sti
                                     bp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp m
                                     d_clear flush_l1d
    ``
  6. esyr-rh commented on Jun 10, 2020

    @esyr-rh
    Contributor
  7. pjssilva commented on Jun 10, 2020

    @pjssilva

    I used the Ubuntu package intel-microcode (3.20200609.0ubuntu0.20.04.0). I don't have it in my system anymore to check out. From the description of the package in https://launchpad.net/ubuntu/+source/intel-microcode/3.20200609.0ubuntu0.20.04.0 I guess 0x00d6. Do you want me to reinstall it and check?

  8. esyr-rh commented on Jun 10, 2020

    @esyr-rh
    Contributor
  9. pjssilva commented on Jun 10, 2020

    @pjssilva

    So it is 0xdc (I found the reference in the middle of the description). As I said I can test what ever you want. Just point me to the file and instructions on how to apply it without an Ubuntu package. Thanks!

  10. hmh commented on Jun 10, 2020

    @hmh

    Did you also update the Linux kernel? It would be nice to know if the regression is related to the microcode update itself, or to the combination of the two...

  11. pjssilva commented on Jun 10, 2020

    @pjssilva

    It looks from my /var/log/apt/history.log that it also update the linux-image package to 5.4.0-37. This is the kernel I am using right now, it did not need to uninstall it. I only uninstalled intel-microcode and its dependencies (meta packages that install the latest kernel and friends). Both were unattended upgrades (because they were marked as a security update).

  12. hmh commented on Jun 10, 2020

    @hmh

    Would it trouble you too much to test the intel-microcode package with an older kernel? With an old kernel, the microcode enables SRDBS mitigation in transparent mode, it is supposed to "just work" even if it might end up being enabled in a configuration that does not need it...

    This ensures the issue is not being caused by some bug in the new kernel code.

  13. hmh commented on Jun 10, 2020

    @hmh

    If you do test with an older kernel, please look in /proc/cpuinfo to ensure you are indeed running microcode revision 0xdc, i.e. the new one for your processor.

  14. pjssilva commented on Jun 10, 2020

    @pjssilva

    You mean try to use a mainline kernel? What version? I can use mainline PPA to do it easily. How do I install the Intel microcode without a deb package. Just point me to the docs, I can manage myself.

  15. esyr-rh commented on Jun 10, 2020

    @esyr-rh
    Contributor
  16. 76 remaining items

  17. hmh commented on Jan 18, 2021

    @hmh

    Problem Statement:
    Intel has identified an issue when performing OS patch loading of MCU version 0xE2 on SKL R0 (506e3) and SKL D0 (406e3) systems when the existing Microcode Update (MCU) version is earlier than 0x80

    Issue description:
    When an OS loads the latest MCU patches on SKL R0 (506e3) and SKL D0 (406e3), may lead to unexpected failures in the following conditions:
    • The system has a BIOS containing MCU version earlier than 0x80 (MCU < 0x80)
    • The user tries to load via OS load mechanism a new MCU version 0xD8 or greater

    Workaround:
    Update affected systems to a BIOS containing MCU version 0x80 or greater.

    @whpenner: thanks, however we do need some extra information to act upon:

    1. Will Intel somehow be able to work around this issue in a future public update of this microcode ? If so, do you have any idea of when it will be out?
    2. If the answer to (1) above is "no, it is not possible", will Intel handle the required patches for the Linux microcode update driver to blacklist the unsupported updates, or do we need to do it ourselves? Because, well, a lot of vendors are not going to issue updates and a lot of users will not apply them anyway. So we need to avoid the lockups and we need to tell the user they need a firmware update if they require the fixes in the new microcode.

    If the answer to (1) above is "yes, now that Intel know where the problem is, there is a way to avoid it and Intel will update the microcode to avoid the issue", then we (distros) can remove the problematic update meanwhile, without the need for kernel patches, etc...

  18. dbischof90 commented on Jan 19, 2021

    @dbischof90

    So a recommended fix would be to update BIOS if possible and then attempt to use the new microcode package?

  19. hmh commented on Jan 19, 2021

    @hmh
  20. naitoshedo commented on Jan 21, 2021

    @naitoshedo

    If a bios update is not available from the motherboard manufacturer, I would highly recommend looking into UBU and patch the microcode into a bios update directly... Good luck expecting support from motherboard manufacturers... Last bios update for my server board was 4 years ago at this point (Thanks SuperMicro)

  21. ZerBea commented on Jan 21, 2021

    @ZerBea

    @naitoshedo thanks for pointing me in that direction. Now everything's fine:

    $ grep 'stepping\|model\|microcode' /proc/cpuinfo
    model		: 78
    model name	: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
    stepping	: 3
    microcode	: 0xe2
    model		: 78
    model name	: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
    stepping	: 3
    microcode	: 0xe2
    model		: 78
    model name	: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
    stepping	: 3
    microcode	: 0xe2
    model		: 78
    model name	: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
    stepping	: 3
    microcode	: 0xe2
    
  22. esyr-rh commented on Jun 8, 2021

    @esyr-rh
    Contributor

    New revision 0xea of 06-4e-03/06-5e-03 microcode files has been published as part of microcode-20210608 release, it may be worth trying out.

  23. Shados commented on Jun 9, 2021

    @Shados

    The 0xea revision in microcode-20210608 boots OK on my sig=0x406e3 device (an Asus UX305CA with an m3-6Y30).

  24. whpenner commented on Jun 9, 2021

    @whpenner

    The above should be resolved with MCU revision ids of 0xe2 or higher for CPUID 406e3 (06-4e-03) and 506e3 (06-5e-03). MCUs released yesterday included revision id 0xea for these products.

  25. fdutheil commented on Jun 15, 2021

    @fdutheil

    As already stated, the issue was fixed on a previous MCU version for my hardware (see previous comments), this 0xea version boots fine too here:

    $ dmesg |grep micro
    [    0.000000] microcode: microcode updated early to revision 0xea, date = 2021-01-25
    [    1.822001] microcode: sig=0x406e3, pf=0x80, revision=0xea
    [    1.822896] microcode: Microcode Update Driver: v2.2.
    
  26. JanZerebecki commented on Jun 15, 2021

    @JanZerebecki

    Like in #31 (comment) the latest release also fails in the same way (no change to the bios):
    Linux debian 5.10.0-7-amd64 #1 SMP Debian 5.10.40-1
    intel-microcode_3.20210608.1_amd64.deb
    according to release notes that is version=0xea

  27. esyr-rh commented on Jun 15, 2021

    @esyr-rh
    Contributor

    @JanZerebecki what microcode revision is reported when the kernel is loaded with dis_ucode_ldr parameter? Does early update to 0xd6 plus late update to 0xea work?

  28. esyr-rh commented on Jun 17, 2021

    @esyr-rh
    Contributor

    @jirireischig may I ask to check if 0xe2/0xea microcode revision helps you (in case you still have the problematic (<0x80) microcode revision applied by firmware/BIOS), by chance?

  29. hmh commented on Jun 23, 2021

    @hmh

    @whpenner, could you confirm if the latest revision (0xea) is supposed to work around the problem stated in #31 (comment) ?

    It would be really nice to be able to ship the microcode update without worrying about severely outdated BIOSes that cannot be fixed due to the usual motherboard-vendor-won't-issue-updated-firmware + EOL issues.

  30. whpenner commented on Jul 8, 2021

    @whpenner

    Yes, this should resolve the issue called out in #31. This MCU was updated to work with either the older MCU (older than revision ID 0x80) and the newer MCUs. I would not recommend an early load of an affected MCU and then load of the latest MCU. The results may be unpredictable. Updating from a BIOS loaded MCU to the latest MCU directly is the recommended method.

  31. hmh commented on Jul 8, 2021

    @hmh

    Thank you. Most Linux distros do exactly that: update directly from BIOS to latest MCU (via an early update), that's why we request a reboot to apply the microcode updates.

    It is important data for anyone doing late loads, though. Thank you for this information

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions