[int3]

Internal review before these two patches go upstream to linux-pm. This PR never merges: once it's reviewed, the patches go to the mailing list with b4, and review tags (Reviewed-by, Tested-by) get added to the commits from approvals here.

The base branch hfi-kexec-base is linux-pm's linux-next at 70f331d, which is where the HFI driver changes land.

The bug

HFI hardware writes updates to a table at a physical address the kernel programs. Nothing turns HFI off on shutdown. Since b33f3d2, a kernel with no thermal netlink subscribers never touches HFI. So after a kexec, hardware keeps writing into the old kernel's table, which the new kernel treats as free memory. On dual-socket Sapphire Rapids hosts that kexec from a 6.1 kernel into 7.0, the 6.1 kernel's tables ended up under the 7.0 kernel's inode hash table, and the 7.0 kernel crashed.

The patches

1. Disable HFI on shutdown and kexec. A syscore shutdown callback turns HFI off on every package. This is the real fix, but it only helps when the old kernel has it. It skips packages where HFI is already off, so it doesn't wait 2 ms for an acknowledgement on each one. After it runs, nothing can turn HFI back on: on kexec, user space keeps running until machine_shutdown() and CPU hotplug is re-enabled, so a new thermal netlink subscriber or a CPU coming online would otherwise enable it again.
2. Disable HFI left enabled by a previous kernel. When a package's first CPU comes online, turn HFI off if it's already on. This covers unpatched old kernels, but hardware can still write until that CPU comes online, which was about 30 s into boot on these hosts.

Testing

On one of those hosts (Xeon Platinum 8462Y+, HPE Cray XD670), with a kernel built from this branch.

Re-enable after shutdown (forced). A test-only commit made the shutdown callback sleep for 5 s, and a process waiting on /dev/kmsg for that point subscribed to thermal netlink events. Each run kexecs into a kernel with patch 2, which reports whether HFI was left enabled.

• Patch 1 without the hfi_shutdown_done check: the next kernel logged HFI left enabled for both packages, and HW_FEEDBACK_PTR had changed to the old kernel's new table address.
• Patch 1 as in this PR: no HFI left enabled lines, and HW_FEEDBACK_PTR was unchanged.

Kexec loop. Run on patch 1 with the hfi_shutdown_done check, before hfi_disable_if_enabled() was factored out (which doesn't change behavior). The previous revision, without the check, also passed. 100 kexecs alternating between this kernel and stock 7.0, via kexec_file_load + reboot(LINUX_REBOOT_CMD_KEXEC). Before each kexec, a thermal netlink event subscriber turned HFI on (CONFIG=0x1 on both packages). All 100 passed.

• Patch 1: after each of the 50 kexecs out of this kernel, stock 7.0 read CONFIG=0x0 on both packages. Stock 7.0 has neither patch, so only the shutdown callback can have turned HFI off.
• Patch 2: after each of the 50 kexecs into this kernel, it logged intel-hfi: HFI left enabled on package N, disabled for both packages and read CONFIG=0x0.

Before sending upstream

• Add Signed-off-by: Jez Ng. Kernel rules forbid AI tools from adding it, so the commits have none yet.
• Add a Reviewed-by tag for each person who approves this PR, to the patches they reviewed.
• The cover letter discloses that the work was drafted with Claude, as the kernel's Documentation/process/generated-content.rst asks.

Cover letter

This is the cover letter b4 will send, verbatim.

Subject: [PATCH 0/2] thermal: intel: hfi: Stop hardware writing to freed memory after kexec

HFI hardware writes updates to a table at a physical address the kernel
programs. Nothing disables HFI on shutdown, and since commit b33f3d2677b8
("thermal: intel: hfi: Enable HFI only when required") a kernel that
starts without thermal netlink clients does not touch HFI at all. So
after a kexec, hardware keeps writing into the previous kernel's table,
which the new kernel treats as free memory.

We hit this on dual-socket Sapphire Rapids hosts (Xeon Platinum 8462Y+)
that boot a Debian 6.1 kernel and kexec into a 7.0 kernel. The new
kernel crashed in find_inode_fast() when the inode hash table landed on
one of the old table pages. Reading MSR 0x17d0 on the running 7.0
kernel showed both packages still pointing at the 6.1 kernel's tables,
and the corruption had the HFI table layout: a timestamp, a changed
flag, then one row per CPU.

Patch 1 disables HFI in a syscore shutdown callback, which covers
reboot, power off and kexec, and keeps it disabled afterwards: on kexec,
user space runs until machine_shutdown(), so a new thermal netlink
client could otherwise enable HFI again. Patch 2 makes a starting kernel
disable HFI if the previous kernel left it enabled. Patch 2 can only
narrow the window: hardware can still write between kernel start and
the moment the package's first CPU reaches the thermal online callback,
which was about 30 seconds into boot on these hosts.

This is the kexec counterpart of the hibernation corruption fixed by
commit 97566d09fd02 ("thermal: intel: hfi: Add syscore callbacks for
system-wide PM"), which disables HFI for suspend and hibernation but not
for shutdown.

Tested on one of the hosts above with both patches applied to this base,
alternating 100 kexecs (kexec_file_load + reboot(LINUX_REBOOT_CMD_KEXEC))
between the patched kernel and an unpatched 7.0 kernel. Before each
kexec, a thermal netlink subscriber enabled HFI on both packages. All
100 passed:

 - Patch 1: after each of the 50 kexecs out of the patched kernel, the
   unpatched kernel read HW_FEEDBACK_CONFIG = 0 on both packages.

 - Patch 2: after each of the 50 kexecs into the patched kernel, it
   logged "HFI left enabled on package N, disabled" for both packages
   and read HW_FEEDBACK_CONFIG = 0.

A test-only 5 second sleep after the shutdown callback, with a thermal
netlink client subscribing during it, showed that patch 1 keeps HFI
disabled. Without the hfi_shutdown_done check, the next kernel found HFI
enabled again.

Thanks to Andrew James and John Ranzie for reviewing an earlier version.

The investigation, patches and changelogs were drafted with an AI coding
assistant (Claude, from Anthropic). I reviewed and tested all of it.

🤖 Generated with Claude Code

[semanticdiff-com]

Review changes with  

[ajames]

Great find; nice patch! I've pointed out a couple of things upstream may hassle you about. I'm flexible about those and willing to sign off on patch #1.

[jaranzie]

I think there are 2 races, one might already be present:

The first would be to make sure that it can't be re-enabled by the current kernel after shutting down.

The second (which may already be present) is to make sure hardware ack's the enablement before trying to disable.

The SDM manual says:

When the OS clears bit 0, hardware sets the IA32_PACKAGE_THERM_STATUS[bit 26] to 1 to acknowledge
disabling of the interface. The OS should wait for this bit to be set to 1 to reclaim the memory of the Intel Thread
Director structure, as by setting IA32_PACKAGE_THERM_STATUS[bit 26] hardware guarantees not to write into
the Intel Thread Director structure anymore.
The OS may clear bit 0 only after receiving an indication from the hardware that the structure initialization is
complete via the same IA32_PACKAGE_THERM_STATUS[bit 26], following enabling of HFI/Intel Thread Director,
thus avoiding a race condition between OS and hardware.

[int3]

The first would be to make sure that it can't be re-enabled by the current kernel after shutting down.

oh yeah that's a good catch... fixed it with a hfi_shutdown_done gate

the second one as you noted is already present in the other disable paths. I think we can skip it for now, at least for this patch set.

[int3]

thanks for all the comments, lots of good catches! I think I've addressed all of them. I also fiddled a bunch with the commit messages and added the cover letter that I plan to use.