aboutsummaryrefslogtreecommitdiff
path: root/arch/x86/include
AgeCommit message (Collapse)AuthorFilesLines
2024-02-07Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvmLinus Torvalds2-2/+5
Pull kvm fixes from Paolo Bonzini: "x86 guest: - Avoid false positive for check that only matters on AMD processors x86: - Give a hint when Win2016 might fail to boot due to XSAVES && !XSAVEC configuration - Do not allow creating an in-kernel PIT unless an IOAPIC already exists RISC-V: - Allow ISA extensions that were enabled for bare metal in 6.8 (Zbc, scalar and vector crypto, Zfh[min], Zihintntl, Zvfh[min], Zfa) S390: - fix CC for successful PQAP instruction - fix a race when creating a shadow page" * tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: x86/coco: Define cc_vendor without CONFIG_ARCH_HAS_CC_PLATFORM x86/kvm: Fix SEV check in sev_map_percpu_data() KVM: x86: Give a hint when Win2016 might fail to boot due to XSAVES erratum KVM: x86: Check irqchip mode before create PIT KVM: riscv: selftests: Add Zfa extension to get-reg-list test RISC-V: KVM: Allow Zfa extension for Guest/VM KVM: riscv: selftests: Add Zvfh[min] extensions to get-reg-list test RISC-V: KVM: Allow Zvfh[min] extensions for Guest/VM KVM: riscv: selftests: Add Zihintntl extension to get-reg-list test RISC-V: KVM: Allow Zihintntl extension for Guest/VM KVM: riscv: selftests: Add Zfh[min] extensions to get-reg-list test RISC-V: KVM: Allow Zfh[min] extensions for Guest/VM KVM: riscv: selftests: Add vector crypto extensions to get-reg-list test RISC-V: KVM: Allow vector crypto extensions for Guest/VM KVM: riscv: selftests: Add scaler crypto extensions to get-reg-list test RISC-V: KVM: Allow scalar crypto extensions for Guest/VM KVM: riscv: selftests: Add Zbc extension to get-reg-list test RISC-V: KVM: Allow Zbc extension for Guest/VM KVM: s390: fix cc for successful PQAP KVM: s390: vsie: fix race during shadow creation
2024-02-07kvmclock: Unexport kvmclock clocksourcePeter Hilber1-2/+0
The KVM PTP driver now refers to the clocksource ID CSID_X86_KVM_CLK, not to the clocksource itself any more. There are no remaining users of the clocksource export. Therefore, make the clocksource static again. Signed-off-by: Peter Hilber <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-02-06x86/kvm: Use separate percpu variable to track the enabling of asyncpfXiaoyao Li1-1/+0
Refer to commit fd10cde9294f ("KVM paravirt: Add async PF initialization to PV guest") and commit 344d9588a9df ("KVM: Add PV MSR to enable asynchronous page faults delivery"). It turns out that at the time when asyncpf was introduced, the purpose was defining the shared PV data 'struct kvm_vcpu_pv_apf_data' with the size of 64 bytes. However, it made a mistake and defined the size to 68 bytes, which failed to make fit in a cache line and made the code inconsistent with the documentation. Below justification quoted from Sean[*] KVM (the host side) has *never* read kvm_vcpu_pv_apf_data.enabled, and the documentation clearly states that enabling is based solely on the bit in the synthetic MSR. So rather than update the documentation, fix the goof by removing the enabled filed and use the separate percpu variable instread. KVM-as-a-host obviously doesn't enforce anything or consume the size, and changing the header will only affect guests that are rebuilt against the new header, so there's no chance of ABI breakage between KVM and its guests. The only possible breakage is if some other hypervisor is emulating KVM's async #PF (LOL) and relies on the guest to set kvm_vcpu_pv_apf_data.enabled. But (a) I highly doubt such a hypervisor exists, (b) that would arguably be a violation of KVM's "spec", and (c) the worst case scenario is that the guest would simply lose async #PF functionality. [*] https://lore.kernel.org/all/[email protected]/T/#u Suggested-by: Sean Christopherson <[email protected]> Signed-off-by: Xiaoyao Li <[email protected]> Link: https://lore.kernel.org/r/[email protected] [sean: use true/false instead of 1/0 for booleans] Signed-off-by: Sean Christopherson <[email protected]>
2024-02-06x86/sev: Fix position dependent variable references in startup codeArd Biesheuvel3-7/+30
The early startup code executes from a 1:1 mapping of memory, which differs from the mapping that the code was linked and/or relocated to run at. The latter mapping is not active yet at this point, and so symbol references that rely on it will fault. Given that the core kernel is built without -fPIC, symbol references are typically emitted as absolute, and so any such references occuring in the early startup code will therefore crash the kernel. While an attempt was made to work around this for the early SEV/SME startup code, by forcing RIP-relative addressing for certain global SEV/SME variables via inline assembly (see snp_cpuid_get_table() for example), RIP-relative addressing must be pervasively enforced for SEV/SME global variables when accessed prior to page table fixups. __startup_64() already handles this issue for select non-SEV/SME global variables using fixup_pointer(), which adjusts the pointer relative to a `physaddr` argument. To avoid having to pass around this `physaddr` argument across all functions needing to apply pointer fixups, introduce a macro RIP_RELATIVE_REF() which generates a RIP-relative reference to a given global variable. It is used where necessary to force RIP-relative accesses to global variables. For backporting purposes, this patch makes no attempt at cleaning up other occurrences of this pattern, involving either inline asm or fixup_pointer(). Those will be addressed later. [ bp: Call it "rip_rel_ref" everywhere like other code shortens "rIP-relative reference" and make the asm wrapper __always_inline. ] Co-developed-by: Kevin Loughlin <[email protected]> Signed-off-by: Kevin Loughlin <[email protected]> Signed-off-by: Ard Biesheuvel <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Cc: <[email protected]> Link: https://lore.kernel.org/all/[email protected]
2024-02-06x86/coco: Define cc_vendor without CONFIG_ARCH_HAS_CC_PLATFORMNathan Chancellor1-2/+3
After commit a9ef277488cf ("x86/kvm: Fix SEV check in sev_map_percpu_data()"), there is a build error when building x86_64_defconfig with GCOV using LLVM: ld.lld: error: undefined symbol: cc_vendor >>> referenced by kvm.c >>> arch/x86/kernel/kvm.o:(kvm_smp_prepare_boot_cpu) in archive vmlinux.a which corresponds to if (cc_vendor != CC_VENDOR_AMD || !cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) return; Without GCOV, clang is able to eliminate the use of cc_vendor because cc_platform_has() evaluates to false when CONFIG_ARCH_HAS_CC_PLATFORM is not set, meaning that if statement will be true no matter what value cc_vendor has. With GCOV, the instrumentation keeps the use of cc_vendor around for code coverage purposes but cc_vendor is only declared, not defined, without CONFIG_ARCH_HAS_CC_PLATFORM, leading to the build error above. Provide a macro definition of cc_vendor when CONFIG_ARCH_HAS_CC_PLATFORM is not set with a value of CC_VENDOR_NONE, so that the first condition can always be evaluated/eliminated at compile time, avoiding the build error altogether. This is very similar to the situation prior to commit da86eb961184 ("x86/coco: Get rid of accessor functions"). Signed-off-by: Nathan Chancellor <[email protected]> Acked-by: Borislav Petkov (AMD) <[email protected]> Message-Id: <20240202-provide-cc_vendor-without-arch_has_cc_platform-v1-1-09ad5f2a3099@kernel.org> Fixes: a9ef277488cf ("x86/kvm: Fix SEV check in sev_map_percpu_data()", 2024-01-31) Signed-off-by: Paolo Bonzini <[email protected]>
2024-02-02x86/fred: Fix a build warning with allmodconfig due to 'inline' failing to ↵Xin Li (Intel)1-1/+1
inline properly Change array_index_mask_nospec() to __always_inline because "inline" is broken as https://www.kernel.org/doc/local/inline.html. Fixes: 6786137bf8fd ("x86/fred: FRED entry/exit and dispatch code") Reported-by: Stephen Rothwell <[email protected]> Signed-off-by: Xin Li (Intel) <[email protected]> Signed-off-by: Ingo Molnar <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-02-01kernel.h: removed REPEAT_BYTE from kernel.hTanzir Hasan1-1/+2
This patch creates wordpart.h and includes it in asm/word-at-a-time.h for all architectures. WORD_AT_A_TIME_CONSTANTS depends on kernel.h because of REPEAT_BYTE. Moving this to another header and including it where necessary allows us to not include the bloated kernel.h. Making this implicit dependency on REPEAT_BYTE explicit allows for later improvements in the lib/string.c inclusion list. Suggested-by: Al Viro <[email protected]> Suggested-by: Andy Shevchenko <[email protected]> Signed-off-by: Tanzir Hasan <[email protected]> Reviewed-by: Andy Shevchenko <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Kees Cook <[email protected]>
2024-02-01KVM: x86/pmu: Snapshot and clear reprogramming bitmap before reprogrammingSean Christopherson1-0/+1
Refactor the handling of the reprogramming bitmap to snapshot and clear to-be-processed bits before doing the reprogramming, and then explicitly set bits for PMCs that need to be reprogrammed (again). This will allow adding a macro to iterate over all valid PMCs without having to add special handling for the reprogramming bit, which (a) can have bits set for non-existent PMCs and (b) needs to clear such bits to avoid wasting cycles in perpetuity. Note, the existing behavior of clearing bits after reprogramming does NOT have a race with kvm_vm_ioctl_set_pmu_event_filter(). Setting a new PMU filter synchronizes SRCU _before_ setting the bitmap, i.e. guarantees that the vCPU isn't in the middle of reprogramming with a stale filter prior to setting the bitmap. Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Sean Christopherson <[email protected]>
2024-02-01KVM: x86/pmu: Move pmc_idx => pmc translation helper to common codeSean Christopherson1-1/+0
Add a common helper for *internal* PMC lookups, and delete the ops hook and Intel's implementation. Keep AMD's implementation, but rename it to amd_pmu_get_pmc() to make it somewhat more obvious that it's suited for both KVM-internal and guest-initiated lookups. Because KVM tracks all counters in a single bitmap, getting a counter when iterating over a bitmap, e.g. of all valid PMCs, requires a small amount of math, that while simple, isn't super obvious and doesn't use the same semantics as PMC lookups from RDPMC! Although AMD doesn't support fixed counters, the common PMU code still behaves as if there a split, the high half of which just happens to always be empty. Opportunstically add a comment to explain both what is going on, and why KVM uses a single bitmap, e.g. the boilerplate for iterating over separate bitmaps could be done via macros, so it's not (just) about deduplicating code. Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Sean Christopherson <[email protected]>
2024-01-31KVM: x86: Give a hint when Win2016 might fail to boot due to XSAVES erratumMaciej S. Szmigiero1-0/+2
Since commit b0563468eeac ("x86/CPU/AMD: Disable XSAVES on AMD family 0x17") kernel unconditionally clears the XSAVES CPU feature bit on Zen1/2 CPUs. Because KVM CPU caps are initialized from the kernel boot CPU features this makes the XSAVES feature also unavailable for KVM guests in this case. At the same time the XSAVEC feature is left enabled. Unfortunately, having XSAVEC but no XSAVES in CPUID breaks Hyper-V enabled Windows Server 2016 VMs that have more than one vCPU. Let's at least give users hint in the kernel log what could be wrong since these VMs currently simply hang at boot with a black screen - giving no clue what suddenly broke them and how to make them work again. Trigger the kernel message hint based on the particular guest ID written to the Guest OS Identity Hyper-V MSR implemented by KVM. Defer this check to when the L1 Hyper-V hypervisor enables SVM in EFER since we want to limit this message to Hyper-V enabled Windows guests only (Windows session running nested as L2) but the actual Guest OS Identity MSR write is done by L1 and happens before it enables SVM. Fixes: b0563468eeac ("x86/CPU/AMD: Disable XSAVES on AMD family 0x17") Signed-off-by: Maciej S. Szmigiero <[email protected]> Message-Id: <b83ab45c5e239e5d148b0ae7750133a67ac9575c.1706127425.git.maciej.szmigiero@oracle.com> [Move some checks before mutex_lock(), rename function. - Paolo] Signed-off-by: Paolo Bonzini <[email protected]>
2024-01-31x86/fred: Add FRED initialization functionsH. Peter Anvin (Intel)1-0/+5
Add cpu_init_fred_exceptions() to: - Set FRED entrypoints for events happening in ring 0 and 3. - Specify the stack level for IRQs occurred ring 0. - Specify dedicated event stacks for #DB/NMI/#MCE/#DF. - Enable FRED and invalidtes IDT. - Force 32-bit system calls to use "int $0x80" only. Add fred_complete_exception_setup() to: - Initialize system_vectors as done for IDT systems. - Set unused sysvec_table entries to fred_handle_spurious_interrupt(). Co-developed-by: Xin Li <[email protected]> Signed-off-by: H. Peter Anvin (Intel) <[email protected]> Signed-off-by: Xin Li <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Tested-by: Shan Kang <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-31x86/entry: Add fred_entry_from_kvm() for VMX to handle IRQ/NMIXin Li1-0/+18
In IRQ/NMI induced VM exits, KVM VMX needs to execute the respective handlers, which requires the software to create a FRED stack frame, and use it to invoke the handlers. Add fred_irq_entry_from_kvm() for this job. Export fred_entry_from_kvm() because VMX can be compiled as a module. Suggested-by: Sean Christopherson <[email protected]> Suggested-by: Thomas Gleixner <[email protected]> Signed-off-by: Xin Li <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Tested-by: Shan Kang <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-31x86/fred: Fixup fault on ERETU by jumping to fred_entrypoint_userXin Li1-1/+3
If the stack frame contains an invalid user context (e.g. due to invalid SS, a non-canonical RIP, etc.) the ERETU instruction will trap (#SS or #GP). From a Linux point of view, this really should be considered a user space failure, so use the standard fault fixup mechanism to intercept the fault, fix up the exception frame, and redirect execution to fred_entrypoint_user. The end result is that it appears just as if the hardware had taken the exception immediately after completing the transition to user space. Suggested-by: H. Peter Anvin (Intel) <[email protected]> Signed-off-by: Xin Li <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Tested-by: Shan Kang <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-31x86/traps: Add sysvec_install() to install a system interrupt handlerXin Li2-2/+15
Add sysvec_install() to install a system interrupt handler into the IDT or the FRED system interrupt handler table. Signed-off-by: Xin Li <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Tested-by: Shan Kang <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-31x86/fred: FRED entry/exit and dispatch codeH. Peter Anvin (Intel)3-2/+9
The code to actually handle kernel and event entry/exit using FRED. It is split up into two files thus: - entry_64_fred.S contains the actual entrypoints and exit code, and saves and restores registers. - entry_fred.c contains the two-level event dispatch code for FRED. The first-level dispatch is on the event type, and the second-level is on the event vector. [ bp: Fold in an allmodconfig clang build fix: https://lore.kernel.org/r/[email protected] and a CONFIG_IA32_EMULATION=n build fix: https://lore.kernel.org/r/[email protected]] Suggested-by: Thomas Gleixner <[email protected]> Originally-by: Megha Dey <[email protected]> Co-developed-by: Xin Li <[email protected]> Signed-off-by: H. Peter Anvin (Intel) <[email protected]> Signed-off-by: Xin Li <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Tested-by: Shan Kang <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-31x86/idtentry: Incorporate definitions/declarations of the FRED entriesXin Li1-8/+63
FRED and IDT can share most of the definitions and declarations so that in the majority of cases the actual handler implementation is the same. The differences are the exceptions where FRED stores exception related information on the stack and the sysvec implementations as FRED can handle irqentry/exit() in the dispatcher instead of having it in each handler. Also add stub defines for vectors which are not used due to Kconfig decisions to spare the ifdeffery in the actual FRED dispatch code. Suggested-by: Thomas Gleixner <[email protected]> Signed-off-by: Xin Li <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Tested-by: Shan Kang <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-31x86/fred: Update MSR_IA32_FRED_RSP0 during task switchH. Peter Anvin (Intel)1-2/+6
MSR_IA32_FRED_RSP0 is used during ring 3 event delivery, and needs to be updated to point to the top of next task stack during task switch. Signed-off-by: H. Peter Anvin (Intel) <[email protected]> Signed-off-by: Xin Li <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Tested-by: Shan Kang <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-31x86/fred: Reserve space for the FRED stack frameH. Peter Anvin (Intel)1-3/+9
When using FRED, reserve space at the top of the stack frame, just like i386 does. Signed-off-by: H. Peter Anvin (Intel) <[email protected]> Signed-off-by: Xin Li <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Tested-by: Shan Kang <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-31x86/fred: Add a new header file for FRED definitionsH. Peter Anvin (Intel)1-0/+68
Add a header file for FRED prototypes and definitions. Signed-off-by: H. Peter Anvin (Intel) <[email protected]> Signed-off-by: Xin Li <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Tested-by: Shan Kang <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-31x86/ptrace: Add FRED additional information to the pt_regs structureXin Li1-5/+61
FRED defines additional information in the upper 48 bits of cs/ss fields. Therefore add the information definitions into the pt_regs structure. Specifically introduce a new structure fred_ss to denote the FRED flags above SS selector, which avoids FRED_SSX_ macros and makes the code simpler and easier to read. Suggested-by: Thomas Gleixner <[email protected]> Originally-by: H. Peter Anvin (Intel) <[email protected]> Signed-off-by: Xin Li <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Tested-by: Shan Kang <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-31x86/ptrace: Cleanup the definition of the pt_regs structureXin Li1-13/+35
struct pt_regs is hard to read because the member or section related comments are not aligned with the members. The 'cs' and 'ss' members of pt_regs are type of 'unsigned long' while in reality they are only 16-bit wide. This works so far as the remaining space is unused, but FRED will use the remaining bits for other purposes. To prepare for FRED: - Cleanup the formatting - Convert 'cs' and 'ss' to u16 and embed them into an union with a u64 - Fixup the related printk() format strings Suggested-by: Thomas Gleixner <[email protected]> Originally-by: H. Peter Anvin (Intel) <[email protected]> Signed-off-by: Xin Li <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Tested-by: Shan Kang <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-31x86/cpu: Add MSR numbers for FRED configurationH. Peter Anvin (Intel)1-1/+12
Add MSR numbers for the FRED configuration registers per FRED spec 5.0. Originally-by: Megha Dey <[email protected]> Signed-off-by: H. Peter Anvin (Intel) <[email protected]> Signed-off-by: Xin Li <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Tested-by: Shan Kang <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-31x86/cpu: Add X86_CR4_FRED macroH. Peter Anvin (Intel)1-0/+7
Add X86_CR4_FRED macro for the FRED bit in %cr4. This bit must not be changed after initialization, so add it to the pinned CR4 bits. Signed-off-by: H. Peter Anvin (Intel) <[email protected]> Signed-off-by: Xin Li <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Tested-by: Shan Kang <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-30KVM: x86/pmu: Prioritize VMX interception over #GP on RDPMC due to bad indexSean Christopherson1-1/+1
Apply the pre-intercepts RDPMC validity check only to AMD, and rename all relevant functions to make it as clear as possible that the check is not a standard PMC index check. On Intel, the basic rule is that only invalid opcodes and privilege/permission/mode checks have priority over VM-Exit, i.e. RDPMC with an invalid index should VM-Exit, not #GP. While the SDM doesn't explicitly call out RDPMC, it _does_ explicitly use RDMSR of a non-existent MSR as an example where VM-Exit has priority over #GP, and RDPMC is effectively just a variation of RDMSR. Manually testing on various Intel CPUs confirms this behavior, and the inverted priority was introduced for SVM compatibility, i.e. was not an intentional change for Intel PMUs. On AMD, *all* exceptions on RDPMC have priority over VM-Exit. Check for a NULL kvm_pmu_ops.check_rdpmc_early instead of using a RET0 static call so as to provide a convenient location to document the difference between Intel and AMD, and to again try to make it as obvious as possible that the early check is a one-off thing, not a generic "is this PMC valid?" helper. Fixes: 8061252ee0d2 ("KVM: SVM: Add intercept checks for remaining twobyte instructions") Cc: Jim Mattson <[email protected]> Tested-by: Dapeng Mi <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Sean Christopherson <[email protected]>
2024-01-30KVM: x86/pmu: Allow programming events that match unsupported arch eventsSean Christopherson1-1/+0
Remove KVM's bogus restriction that the guest can't program an event whose encoding matches an unsupported architectural event. The enumeration of an architectural event only says that if a CPU supports an architectural event, then the event can be programmed using the architectural encoding. The enumeration does NOT say anything about the encoding when the CPU doesn't report support the architectural event. Preventing the guest from counting events whose encoding happens to match an architectural event breaks existing functionality whenever Intel adds an architectural encoding that was *ever* used for a CPU that doesn't enumerate support for the architectural event, even if the encoding is for the exact same event! E.g. the architectural encoding for Top-Down Slots is 0x01a4. Broadwell CPUs, which do not support the Top-Down Slots architectural event, 0x01a4 is a valid, model-specific event. Denying guest usage of 0x01a4 if/when KVM adds support for Top-Down slots would break any Broadwell-based guest. Reported-by: Kan Liang <[email protected]> Closes: https://lore.kernel.org/all/[email protected] Fixes: a21864486f7e ("KVM: x86/pmu: Fix available_event_types check for REF_CPU_CYCLES event") Reviewed-by: Dapeng Mi <[email protected]> Reviewed-by: Jim Mattson <[email protected]> Reviewed-by: Kan Liang <[email protected]> Tested-by: Dapeng Mi <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Sean Christopherson <[email protected]>
2024-01-30x86: Do not include <asm/bootparam.h> in several filesThomas Zimmermann4-5/+3
Remove the include statement for <asm/bootparam.h> from several files that don't require it and limit the exposure of those definitions within the Linux kernel code. [ bp: Massage commit message. ] Signed-off-by: Thomas Zimmermann <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Reviewed-by: Ard Biesheuvel <[email protected]> Acked-by: Ard Biesheuvel <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-30x86/efi: Implement arch_ima_efi_boot_mode() in source fileThomas Zimmermann1-2/+3
The x86 implementation of arch_ima_efi_boot_mode() uses the global boot_param state. Move it into a source file to clean up the header. Avoid potential rebuilds of unrelated source files if boot_params changes. Suggested-by: Ard Biesheuvel <[email protected]> Signed-off-by: Thomas Zimmermann <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Reviewed-by: Ard Biesheuvel <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-30x86/setup: Move internal setup_data structures into setup_data.hThomas Zimmermann3-22/+32
Move struct_efi_setup_data in order to unify duplicated definition of the data structure in a single place. Also silence clang's warnings about GNU extensions in real-mode code which might occur from the changed includes. [ bp: Massage commit message. ] Signed-off-by: Thomas Zimmermann <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Reviewed-by: Ard Biesheuvel <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-30x86/setup: Move UAPI setup structures into setup_data.hThomas Zimmermann3-72/+85
The type definition of struct pci_setup_rom in <asm/pci.h> requires struct setup_data from <asm/bootparam.h>. Many drivers include <linux/pci.h>, but do not use boot parameters. Changes to bootparam.h or its included header files could easily trigger a large, unnecessary rebuild of the kernel. Moving struct setup_data and related code into its own header file avoids including <asm/bootparam.h> in <asm/pci.h>. Instead include the new header <asm/screen_data.h> and remove the include statement for x86_init.h, which is unnecessary but pulls in bootparams.h. Suggested-by: Ard Biesheuvel <[email protected]> Signed-off-by: Thomas Zimmermann <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Reviewed-by: Ard Biesheuvel <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-29Merge tag 'mm-hotfixes-stable-2024-01-28-23-21' of ↵Linus Torvalds1-1/+16
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Pull misc fixes from Andrew Morton: "22 hotfixes. 11 are cc:stable and the remainder address post-6.7 issues or aren't considered appropriate for backporting" * tag 'mm-hotfixes-stable-2024-01-28-23-21' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (22 commits) mm: thp_get_unmapped_area must honour topdown preference mm: huge_memory: don't force huge page alignment on 32 bit userfaultfd: fix mmap_changing checking in mfill_atomic_hugetlb selftests/mm: ksm_tests should only MADV_HUGEPAGE valid memory scs: add CONFIG_MMU dependency for vfree_atomic() mm/memory: fix folio_set_dirty() vs. folio_mark_dirty() in zap_pte_range() mm/huge_memory: fix folio_set_dirty() vs. folio_mark_dirty() selftests/mm: Update va_high_addr_switch.sh to check CPU for la57 flag selftests: mm: fix map_hugetlb failure on 64K page size systems MAINTAINERS: supplement of zswap maintainers update stackdepot: make fast paths lock-less again stackdepot: add stats counters exported via debugfs mm, kmsan: fix infinite recursion due to RCU critical section mm/writeback: fix possible divide-by-zero in wb_dirty_limits(), again selftests/mm: switch to bash from sh MAINTAINERS: add man-pages git trees mm: memcontrol: don't throttle dying tasks on memory.high mm: mmap: map MAP_STACK to VM_NOHUGEPAGE uprobes: use pagesize-aligned virtual address when replacing pages selftests/mm: mremap_test: fix build warning ...
2024-01-29x86/cpufeatures: Enable/unmask SEV-SNP CPU featureMichael Roth1-0/+4
With all the required host changes in place, it should now be possible to initialize SNP-related MSR bits, set up RMP table enforcement, and initialize SNP support in firmware while maintaining legacy support for SEV/SEV-ES guests. Go ahead and enable the SNP feature now. Signed-off-by: Michael Roth <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-29KVM: SEV: Make AVIC backing, VMSA and VMCB memory allocation SNP safeBrijesh Singh2-0/+2
Implement a workaround for an SNP erratum where the CPU will incorrectly signal an RMP violation #PF if a hugepage (2MB or 1GB) collides with the RMP entry of a VMCB, VMSA or AVIC backing page. When SEV-SNP is globally enabled, the CPU marks the VMCB, VMSA, and AVIC backing pages as "in-use" via a reserved bit in the corresponding RMP entry after a successful VMRUN. This is done for _all_ VMs, not just SNP-Active VMs. If the hypervisor accesses an in-use page through a writable translation, the CPU will throw an RMP violation #PF. On early SNP hardware, if an in-use page is 2MB-aligned and software accesses any part of the associated 2MB region with a hugepage, the CPU will incorrectly treat the entire 2MB region as in-use and signal a an RMP violation #PF. To avoid this, the recommendation is to not use a 2MB-aligned page for the VMCB, VMSA or AVIC pages. Add a generic allocator that will ensure that the page returned is not 2MB-aligned and is safe to be used when SEV-SNP is enabled. Also implement similar handling for the VMCB/VMSA pages of nested guests. [ mdr: Squash in nested guest handling from Ashish, commit msg fixups. ] Reported-by: Alper Gun <[email protected]> # for nested VMSA case Signed-off-by: Brijesh Singh <[email protected]> Co-developed-by: Marc Orr <[email protected]> Signed-off-by: Marc Orr <[email protected]> Co-developed-by: Ashish Kalra <[email protected]> Signed-off-by: Ashish Kalra <[email protected]> Signed-off-by: Michael Roth <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Acked-by: Vlastimil Babka <[email protected]> Acked-by: Paolo Bonzini <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-29crypto: ccp: Add panic notifier for SEV/SNP firmware shutdown on kdumpAshish Kalra1-0/+2
Add a kdump safe version of sev_firmware_shutdown() and register it as a crash_kexec_post_notifier so it will be invoked during panic/crash to do SEV/SNP shutdown. This is required for transitioning all IOMMU pages to reclaim/hypervisor state, otherwise re-init of IOMMU pages during crashdump kernel boot fails and panics the crashdump kernel. This panic notifier runs in atomic context, hence it ensures not to acquire any locks/mutexes and polls for PSP command completion instead of depending on PSP command completion interrupt. [ mdr: Remove use of "we" in comments. ] Signed-off-by: Ashish Kalra <[email protected]> Signed-off-by: Michael Roth <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-29x86/sev: Introduce an SNP leaked pages listAshish Kalra1-0/+2
Pages are unsafe to be released back to the page-allocator if they have been transitioned to firmware/guest state and can't be reclaimed or transitioned back to hypervisor/shared state. In this case, add them to an internal leaked pages list to ensure that they are not freed or touched/accessed to cause fatal page faults. [ mdr: Relocate to arch/x86/virt/svm/sev.c ] Suggested-by: Vlastimil Babka <[email protected]> Signed-off-by: Ashish Kalra <[email protected]> Signed-off-by: Michael Roth <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Reviewed-by: Vlastimil Babka <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-29x86/sev: Add helper functions for RMPUPDATE and PSMASH instructionBrijesh Singh1-0/+23
The RMPUPDATE instruction updates the access restrictions for a page via its corresponding entry in the RMP Table. The hypervisor will use the instruction to enforce various access restrictions on pages used for confidential guests and other specialized functionality. See APM3 for details on the instruction operations. The PSMASH instruction expands a 2MB RMP entry in the RMP table into a corresponding set of contiguous 4KB RMP entries while retaining the state of the validated bit from the original 2MB RMP entry. The hypervisor will use this instruction in cases where it needs to re-map a page as 4K rather than 2MB in a guest's nested page table. Add helpers to make use of these instructions. [ mdr: add RMPUPDATE retry logic for transient FAIL_OVERLAP errors. ] Signed-off-by: Brijesh Singh <[email protected]> Signed-off-by: Ashish Kalra <[email protected]> Signed-off-by: Michael Roth <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Reviewed-by: Liam Merwick <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-29x86/traps: Define RMP violation #PF error codeBrijesh Singh1-8/+12
Bit 31 in the page fault-error bit will be set when processor encounters an RMP violation. While at it, use the BIT() macro. Signed-off-by: Brijesh Singh <[email protected]> Signed-off-by: Michael Roth <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Acked-by: Dave Hansen <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-29x86/fault: Add helper for dumping RMP entriesBrijesh Singh1-0/+2
This information will be useful for debugging things like page faults due to RMP access violations and RMPUPDATE failures. [ mdr: move helper to standalone patch, rework dump logic as suggested by Boris. ] Signed-off-by: Brijesh Singh <[email protected]> Signed-off-by: Ashish Kalra <[email protected]> Signed-off-by: Michael Roth <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-29x86/sev: Add RMP entry lookup helpersBrijesh Singh1-0/+3
Add a helper that can be used to access information contained in the RMP entry corresponding to a particular PFN. This will be needed to make decisions on how to handle setting up mappings in the NPT in response to guest page-faults and handling things like cleaning up pages and setting them back to the default hypervisor-owned state when they are no longer being used for private data. [ mdr: separate 'assigned' indicator from return code, and simplify function signatures for various helpers. ] Signed-off-by: Brijesh Singh <[email protected]> Co-developed-by: Ashish Kalra <[email protected]> Signed-off-by: Ashish Kalra <[email protected]> Signed-off-by: Michael Roth <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-29x86/sev: Add SEV-SNP host initialization supportBrijesh Singh2-1/+16
The memory integrity guarantees of SEV-SNP are enforced through a new structure called the Reverse Map Table (RMP). The RMP is a single data structure shared across the system that contains one entry for every 4K page of DRAM that may be used by SEV-SNP VMs. The APM Volume 2 section on Secure Nested Paging (SEV-SNP) details a number of steps needed to detect/enable SEV-SNP and RMP table support on the host: - Detect SEV-SNP support based on CPUID bit - Initialize the RMP table memory reported by the RMP base/end MSR registers and configure IOMMU to be compatible with RMP access restrictions - Set the MtrrFixDramModEn bit in SYSCFG MSR - Set the SecureNestedPagingEn and VMPLEn bits in the SYSCFG MSR - Configure IOMMU RMP table entry format is non-architectural and it can vary by processor. It is defined by the PPR document for each respective CPU family. Restrict SNP support to CPU models/families which are compatible with the current RMP table entry format to guard against any undefined behavior when running on other system types. Future models/support will handle this through an architectural mechanism to allow for broader compatibility. SNP host code depends on CONFIG_KVM_AMD_SEV config flag which may be enabled even when CONFIG_AMD_MEM_ENCRYPT isn't set, so update the SNP-specific IOMMU helpers used here to rely on CONFIG_KVM_AMD_SEV instead of CONFIG_AMD_MEM_ENCRYPT. Signed-off-by: Brijesh Singh <[email protected]> Co-developed-by: Ashish Kalra <[email protected]> Signed-off-by: Ashish Kalra <[email protected]> Co-developed-by: Tom Lendacky <[email protected]> Signed-off-by: Tom Lendacky <[email protected]> Co-developed-by: Borislav Petkov (AMD) <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Co-developed-by: Michael Roth <[email protected]> Signed-off-by: Michael Roth <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-29iommu/amd: Don't rely on external callers to enable IOMMU SNP supportAshish Kalra1-0/+1
Currently, the expectation is that the kernel will call amd_iommu_snp_enable() to perform various checks and set the amd_iommu_snp_en flag that the IOMMU uses to adjust its setup routines to account for additional requirements on hosts where SNP is enabled. This is somewhat fragile as it relies on this call being done prior to IOMMU setup. It is more robust to just do this automatically as part of IOMMU initialization, so rework the code accordingly. There is still a need to export information about whether or not the IOMMU is configured in a manner compatible with SNP, so relocate the existing amd_iommu_snp_en flag so it can be used to convey that information in place of the return code that was previously provided by calls to amd_iommu_snp_enable(). While here, also adjust the kernel messages related to IOMMU SNP enablement for consistency/grammar/clarity. Suggested-by: Borislav Petkov (AMD) <[email protected]> Signed-off-by: Ashish Kalra <[email protected]> Co-developed-by: Michael Roth <[email protected]> Signed-off-by: Michael Roth <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Acked-by: Joerg Roedel <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-29x86/cpufeatures: Add SEV-SNP CPU featureBrijesh Singh2-1/+4
Add CPU feature detection for Secure Encrypted Virtualization with Secure Nested Paging. This feature adds a strong memory integrity protection to help prevent malicious hypervisor-based attacks like data replay, memory re-mapping, and more. Since enabling the SNP CPU feature imposes a number of additional requirements on host initialization and handling legacy firmware APIs for SEV/SEV-ES guests, only introduce the CPU feature bit so that the relevant handling can be added, but leave it disabled via a disabled-features mask. Once all the necessary changes needed to maintain legacy SEV/SEV-ES support are introduced in subsequent patches, the SNP feature bit will be unmasked/enabled. Signed-off-by: Brijesh Singh <[email protected]> Signed-off-by: Jarkko Sakkinen <[email protected]> Signed-off-by: Ashish Kalra <[email protected]> Signed-off-by: Michael Roth <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-26Merge branch 'linus' into x86/mm, to refresh the branch and pick up fixesIngo Molnar48-525/+502
Signed-off-by: Ingo Molnar <[email protected]>
2024-01-25mm, kmsan: fix infinite recursion due to RCU critical sectionMarco Elver1-1/+16
Alexander Potapenko writes in [1]: "For every memory access in the code instrumented by KMSAN we call kmsan_get_metadata() to obtain the metadata for the memory being accessed. For virtual memory the metadata pointers are stored in the corresponding `struct page`, therefore we need to call virt_to_page() to get them. According to the comment in arch/x86/include/asm/page.h, virt_to_page(kaddr) returns a valid pointer iff virt_addr_valid(kaddr) is true, so KMSAN needs to call virt_addr_valid() as well. To avoid recursion, kmsan_get_metadata() must not call instrumented code, therefore ./arch/x86/include/asm/kmsan.h forks parts of arch/x86/mm/physaddr.c to check whether a virtual address is valid or not. But the introduction of rcu_read_lock() to pfn_valid() added instrumented RCU API calls to virt_to_page_or_null(), which is called by kmsan_get_metadata(), so there is an infinite recursion now. I do not think it is correct to stop that recursion by doing kmsan_enter_runtime()/kmsan_exit_runtime() in kmsan_get_metadata(): that would prevent instrumented functions called from within the runtime from tracking the shadow values, which might introduce false positives." Fix the issue by switching pfn_valid() to the _sched() variant of rcu_read_lock/unlock(), which does not require calling into RCU. Given the critical section in pfn_valid() is very small, this is a reasonable trade-off (with preemptible RCU). KMSAN further needs to be careful to suppress calls into the scheduler, which would be another source of recursion. This can be done by wrapping the call to pfn_valid() into preempt_disable/enable_no_resched(). The downside is that this sacrifices breaking scheduling guarantees; however, a kernel compiled with KMSAN has already given up any performance guarantees due to being heavily instrumented. Note, KMSAN code already disables tracing via Makefile, and since mmzone.h is included, it is not necessary to use the notrace variant, which is generally preferred in all other cases. Link: https://lkml.kernel.org/r/[email protected] [1] Link: https://lkml.kernel.org/r/[email protected] Fixes: 5ec8e8ea8b77 ("mm/sparsemem: fix race in accessing memory_section->usage") Signed-off-by: Marco Elver <[email protected]> Reported-by: Alexander Potapenko <[email protected]> Reported-by: [email protected] Reviewed-by: Alexander Potapenko <[email protected]> Tested-by: Alexander Potapenko <[email protected]> Cc: Charan Teja Kalla <[email protected]> Cc: Borislav Petkov (AMD) <[email protected]> Cc: Dave Hansen <[email protected]> Cc: Dmitry Vyukov <[email protected]> Cc: "H. Peter Anvin" <[email protected]> Cc: Ingo Molnar <[email protected]> Cc: Thomas Gleixner <[email protected]> Signed-off-by: Andrew Morton <[email protected]>
2024-01-25x86/fred: Disable FRED support if CONFIG_X86_FRED is disabledH. Peter Anvin (Intel)1-1/+7
Add CONFIG_X86_FRED to <asm/disabled-features.h> to make cpu_feature_enabled() work correctly with FRED. Originally-by: Megha Dey <[email protected]> Signed-off-by: H. Peter Anvin (Intel) <[email protected]> Signed-off-by: Xin Li <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Tested-by: Shan Kang <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-25x86/cpufeatures: Add the CPU feature bit for FREDH. Peter Anvin (Intel)1-0/+1
Any FRED enabled CPU will always have the following features as its baseline: 1) LKGS, load attributes of the GS segment but the base address into the IA32_KERNEL_GS_BASE MSR instead of the GS segment’s descriptor cache. 2) WRMSRNS, non-serializing WRMSR for faster MSR writes. Signed-off-by: H. Peter Anvin (Intel) <[email protected]> Signed-off-by: Xin Li <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Tested-by: Shan Kang <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-25x86/trapnr: Add event type macros to <asm/trapnr.h>Xin Li2-8/+21
Intel VT-x classifies events into eight different types, which is inherited by FRED for event identification. As such, event types becomes a common x86 concept, and should be defined in a common x86 header. Add event type macros to <asm/trapnr.h>, and use them in <asm/vmx.h>. Suggested-by: H. Peter Anvin (Intel) <[email protected]> Signed-off-by: Xin Li <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Tested-by: Shan Kang <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-25x86/entry: Remove idtentry_sysvec from entry_{32,64}.SXin Li1-1/+1
idtentry_sysvec is really just DECLARE_IDTENTRY defined in <asm/idtentry.h>, no need to define it separately. Signed-off-by: Xin Li <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Tested-by: Shan Kang <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-25x86/cpufeatures,opcode,msr: Add the WRMSRNS instruction supportXin Li2-0/+19
WRMSRNS is an instruction that behaves exactly like WRMSR, with the only difference being that it is not a serializing instruction by default. Under certain conditions, WRMSRNS may replace WRMSR to improve performance. Add its CPU feature bit, opcode to the x86 opcode map, and an always inline API __wrmsrns() to embed WRMSRNS into the code. Signed-off-by: Xin Li <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Tested-by: Shan Kang <[email protected]> Acked-by: Masami Hiramatsu (Google) <[email protected]> Acked-by: Borislav Petkov (AMD) <[email protected]> Link: https://lore.kernel.org/r/[email protected]
2024-01-24x86/entry/ia32: Ensure s32 is sign extended to s64Richard Palethorpe1-4/+21
Presently ia32 registers stored in ptregs are unconditionally cast to unsigned int by the ia32 stub. They are then cast to long when passed to __se_sys*, but will not be sign extended. This takes the sign of the syscall argument into account in the ia32 stub. It still casts to unsigned int to avoid implementation specific behavior. However then casts to int or unsigned int as necessary. So that the following cast to long sign extends the value. This fixes the io_pgetevents02 LTP test when compiled with -m32. Presently the systemcall io_pgetevents_time64() unexpectedly accepts -1 for the maximum number of events. It doesn't appear other systemcalls with signed arguments are effected because they all have compat variants defined and wired up. Fixes: ebeb8c82ffaf ("syscalls/x86: Use 'struct pt_regs' based syscall calling for IA32_EMULATION and x32") Suggested-by: Arnd Bergmann <[email protected]> Signed-off-by: Richard Palethorpe <[email protected]> Signed-off-by: Nikolay Borisov <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Reviewed-by: Arnd Bergmann <[email protected]> Cc: [email protected] Link: https://lore.kernel.org/r/[email protected] Link: https://lore.kernel.org/ltp/[email protected]/
2024-01-23x86/cpu: Add model number for Intel Clearwater Forest processorTony Luck1-0/+2
Server product based on the Atom Darkmont core. Signed-off-by: Tony Luck <[email protected]> Signed-off-by: Borislav Petkov (AMD) <[email protected]> Link: https://lore.kernel.org/r/[email protected]
Powered by cgit, Gruvboxed by Blaster4385.