aboutsummaryrefslogtreecommitdiff
path: root/arch/riscv/kernel/patch.c
AgeCommit message (Collapse)AuthorFilesLines
2024-08-06riscv: Re-introduce global icache flush in patch_text_XXX()Alexandre Ghiti1-0/+4
commit edf2d546bfd6 ("riscv: patch: Flush the icache right after patching to avoid illegal insns") mistakenly removed the global icache flush in patch_text_nosync() and patch_text_set_nosync() functions, so reintroduce them. Fixes: edf2d546bfd6 ("riscv: patch: Flush the icache right after patching to avoid illegal insns") Reported-by: Samuel Holland <[email protected]> Closes: https://lore.kernel.org/linux-riscv/[email protected]/ Signed-off-by: Alexandre Ghiti <[email protected]> Reviewed-by: Samuel Holland <[email protected]> Reviewed-by: Charlie Jenkins <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Palmer Dabbelt <[email protected]>
2024-07-20Merge tag 'riscv-for-linus-6.11-mw1' of ↵Linus Torvalds1-32/+31
git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux Pull RISC-V updates from Palmer Dabbelt: - Support for various new ISA extensions: * The Zve32[xf] and Zve64[xfd] sub-extensios of the vector extension * Zimop and Zcmop for may-be-operations * The Zca, Zcf, Zcd and Zcb sub-extensions of the C extension * Zawrs - riscv,cpu-intc is now dtschema - A handful of performance improvements and cleanups to text patching - Support for memory hot{,un}plug - The highest user-allocatable virtual address is now visible in hwprobe * tag 'riscv-for-linus-6.11-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux: (58 commits) riscv: lib: relax assembly constraints in hweight riscv: set trap vector earlier KVM: riscv: selftests: Add Zawrs extension to get-reg-list test KVM: riscv: Support guest wrs.nto riscv: hwprobe: export Zawrs ISA extension riscv: Add Zawrs support for spinlocks dt-bindings: riscv: Add Zawrs ISA extension description riscv: Provide a definition for 'pause' riscv: hwprobe: export highest virtual userspace address riscv: Improve sbi_ecall() code generation by reordering arguments riscv: Add tracepoints for SBI calls and returns riscv: Optimize crc32 with Zbc extension riscv: Enable DAX VMEMMAP optimization riscv: mm: Add support for ZONE_DEVICE virtio-mem: Enable virtio-mem for RISC-V riscv: Enable memory hotplugging for RISC-V riscv: mm: Take memory hotplug read-lock during kernel page table dump riscv: mm: Add memory hotplugging support riscv: mm: Add pfn_to_kaddr() implementation riscv: mm: Refactor create_linear_mapping_range() for memory hot add ...
2024-06-26riscv: patch: Flush the icache right after patching to avoid illegal insnsAlexandre Ghiti1-8/+18
We cannot delay the icache flush after patching some functions as we may have patched a function that will get called before the icache flush. The only way to completely avoid such scenario is by flushing the icache as soon as we patch a function. This will probably be costly as we don't batch the icache maintenance anymore. Fixes: 6ca445d8af0e ("riscv: Fix early ftrace nop patching") Reported-by: Conor Dooley <[email protected]> Closes: https://lore.kernel.org/linux-riscv/20240613-lubricant-breath-061192a9489a@wendy/ Signed-off-by: Alexandre Ghiti <[email protected]> Reviewed-by: Andy Chiu <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Palmer Dabbelt <[email protected]>
2024-06-26riscv: Remove extra variable in patch_text_nosync()Samuel Holland1-8/+4
This cast is superfluous, and is incorrect anyway if compressed instructions may be present. Reviewed-by: Björn Töpel <[email protected]> Signed-off-by: Samuel Holland <[email protected]> Reviewed-by: Conor Dooley <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Palmer Dabbelt <[email protected]>
2024-06-26riscv: Use offset_in_page() in text patching functionsSamuel Holland1-3/+3
This is a bit easier to parse than the equivalent bit manipulation. Reviewed-by: Björn Töpel <[email protected]> Signed-off-by: Samuel Holland <[email protected]> Reviewed-by: Conor Dooley <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Palmer Dabbelt <[email protected]>
2024-06-26riscv: Pass patch_text() the length in bytesSamuel Holland1-9/+5
patch_text_nosync() already handles an arbitrary length of code, so this removes a superfluous loop and reduces the number of icache flushes. Reviewed-by: Björn Töpel <[email protected]> Signed-off-by: Samuel Holland <[email protected]> Reviewed-by: Conor Dooley <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Palmer Dabbelt <[email protected]>
2024-06-26riscv: Simplify text patching loopsSamuel Holland1-16/+21
This reduces the number of variables and makes the code easier to parse. Signed-off-by: Samuel Holland <[email protected]> Reviewed-by: Björn Töpel <[email protected]> Reviewed-by: Conor Dooley <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Palmer Dabbelt <[email protected]>
2024-05-22Merge tag 'riscv-for-linus-6.10-mw1' of ↵Linus Torvalds1-5/+12
git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux Pull RISC-V updates from Palmer Dabbelt: - Add byte/half-word compare-and-exchange, emulated via LR/SC loops - Support for Rust - Support for Zihintpause in hwprobe - Add PR_RISCV_SET_ICACHE_FLUSH_CTX prctl() - Support lockless lockrefs * tag 'riscv-for-linus-6.10-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux: (42 commits) riscv: defconfig: Enable CONFIG_CLK_SOPHGO_CV1800 riscv: select ARCH_HAS_FAST_MULTIPLIER riscv: mm: still create swiotlb buffer for kmalloc() bouncing if required riscv: Annotate pgtable_l{4,5}_enabled with __ro_after_init riscv: Remove redundant CONFIG_64BIT from pgtable_l{4,5}_enabled riscv: mm: Always use an ASID to flush mm contexts riscv: mm: Preserve global TLB entries when switching contexts riscv: mm: Make asid_bits a local variable riscv: mm: Use a fixed layout for the MM context ID riscv: mm: Introduce cntx2asid/cntx2version helper macros riscv: Avoid TLB flush loops when affected by SiFive CIP-1200 riscv: Apply SiFive CIP-1200 workaround to single-ASID sfence.vma riscv: mm: Combine the SMP and UP TLB flush code riscv: Only send remote fences when some other CPU is online riscv: mm: Broadcast kernel TLB flushes only when needed riscv: Use IPIs for remote cache/TLB flushes by default riscv: Factor out page table TLB synchronization riscv: Flush the instruction cache during SMP bringup riscv: hwprobe: export Zihintpause ISA extension riscv: misaligned: remove CONFIG_RISCV_M_MODE specific code ...
2024-04-16riscv: Fix text patching when IPI are usedAlexandre Ghiti1-4/+12
For now, we use stop_machine() to patch the text and when we use IPIs for remote icache flushes (which is emitted in patch_text_nosync()), the system hangs. So instead, make sure every CPU executes the stop_machine() patching function and emit a local icache flush there. Co-developed-by: Björn Töpel <[email protected]> Signed-off-by: Björn Töpel <[email protected]> Signed-off-by: Alexandre Ghiti <[email protected]> Reviewed-by: Andrea Parri <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Palmer Dabbelt <[email protected]>
2024-04-16riscv: Remove superfluous smp_mb()Alexandre Ghiti1-1/+0
This memory barrier is not needed and not documented so simply remove it. Suggested-by: Andrea Parri <[email protected]> Signed-off-by: Alexandre Ghiti <[email protected]> Reviewed-by: Andrea Parri <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Palmer Dabbelt <[email protected]>
2024-04-04riscv: Disable preemption when using patch_map()Alexandre Ghiti1-0/+8
patch_map() uses fixmap mappings to circumvent the non-writability of the kernel text mapping. The __set_fixmap() function only flushes the current cpu tlb, it does not emit an IPI so we must make sure that while we use a fixmap mapping, the current task is not migrated on another cpu which could miss the newly introduced fixmap mapping. So in order to avoid any task migration, disable the preemption. Reported-by: Andrea Parri <[email protected]> Closes: https://lore.kernel.org/all/ZcS+GAaM25LXsBOl@andrea/ Reported-by: Andy Chiu <[email protected]> Closes: https://lore.kernel.org/linux-riscv/CABgGipUMz3Sffu-CkmeUB1dKVwVQ73+7=sgC45-m0AE9RCjOZg@mail.gmail.com/ Fixes: cad539baa48f ("riscv: implement a memset like function for text") Fixes: 0ff7c3b33127 ("riscv: Use text_mutex instead of patch_lock") Co-developed-by: Andy Chiu <[email protected]> Signed-off-by: Andy Chiu <[email protected]> Signed-off-by: Alexandre Ghiti <[email protected]> Acked-by: Puranjay Mohan <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Palmer Dabbelt <[email protected]>
2024-01-09riscv: Check if the code to patch lies in the exit sectionAlexandre Ghiti1-1/+10
Otherwise we fall through to vmalloc_to_page() which panics since the address does not lie in the vmalloc region. Fixes: 043cb41a85de ("riscv: introduce interfaces to patch kernel code") Signed-off-by: Alexandre Ghiti <[email protected]> Reviewed-by: Charlie Jenkins <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Palmer Dabbelt <[email protected]>
2023-09-06riscv: implement a memset like function for textPuranjay Mohan1-0/+77
The BPF JIT needs to write invalid instructions to RX regions of memory to invalidate removed BPF programs. This needs a function like memset() that can work with RX memory. Implement patch_text_set_nosync() which is similar to text_poke_set() of x86. Signed-off-by: Puranjay Mohan <[email protected]> Reviewed-by: Pu Lehui <[email protected]> Acked-by: Björn Töpel <[email protected]> Tested-by: Björn Töpel <[email protected]> Acked-by: Daniel Borkmann <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Palmer Dabbelt <[email protected]>
2023-09-06riscv: extend patch_text_nosync() for multiple pagesPuranjay Mohan1-5/+32
The patch_insn_write() function currently doesn't work for multiple pages of instructions, therefore patch_text_nosync() will fail with a page fault if called with lengths spanning multiple pages. This commit extends the patch_insn_write() function to support multiple pages by copying at max 2 pages at a time in a loop. This implementation is similar to text_poke_copy() function of x86. Signed-off-by: Puranjay Mohan <[email protected]> Reviewed-by: Pu Lehui <[email protected]> Reviewed-by: Björn Töpel <[email protected]> Tested-by: Björn Töpel <[email protected]> Acked-by: Daniel Borkmann <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Palmer Dabbelt <[email protected]>
2023-03-09RISC-V: Don't check text_mutex during stop_machineConor Dooley1-3/+25
We're currently using stop_machine() to update ftrace & kprobes, which means that the thread that takes text_mutex during may not be the same as the thread that eventually patches the code. This isn't actually a race because the lock is still held (preventing any other concurrent accesses) and there is only one thread running during stop_machine(), but it does trigger a lockdep failure. This patch just elides the lockdep check during stop_machine. Fixes: c15ac4fd60d5 ("riscv/ftrace: Add dynamic function tracer support") Suggested-by: Steven Rostedt <[email protected]> Reported-by: Changbin Du <[email protected]> Signed-off-by: Palmer Dabbelt <[email protected]> Signed-off-by: Conor Dooley <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Palmer Dabbelt <[email protected]>
2023-02-17riscv: Extend patch_text for multiple instructionsPu Lehui1-7/+12
Extend patch_text for multiple instructions. This is the preparaiton for multiple instructions text patching in riscv BPF trampoline, and may be useful for other scenario. Signed-off-by: Pu Lehui <[email protected]> Signed-off-by: Daniel Borkmann <[email protected]> Tested-by: Björn Töpel <[email protected]> Reviewed-by: Conor Dooley <[email protected]> Acked-by: Björn Töpel <[email protected]> Link: https://lore.kernel.org/bpf/[email protected]
2022-04-22riscv: patch_text: Fixup last cpu should be masterGuo Ren1-1/+1
These patch_text implementations are using stop_machine_cpuslocked infrastructure with atomic cpu_count. The original idea: When the master CPU patch_text, the others should wait for it. But current implementation is using the first CPU as master, which couldn't guarantee the remaining CPUs are waiting. This patch changes the last CPU as the master to solve the potential risk. Signed-off-by: Guo Ren <[email protected]> Signed-off-by: Guo Ren <[email protected]> Acked-by: Palmer Dabbelt <[email protected]> Reviewed-by: Masami Hiramatsu <[email protected]> Fixes: 043cb41a85de ("riscv: introduce interfaces to patch kernel code") Cc: [email protected] Signed-off-by: Palmer Dabbelt <[email protected]>
2021-01-14riscv: Fixup compile error BUILD_BUG_ON failedGuo Ren1-2/+6
Unfortunately, the current code couldn't be compiled: CC arch/riscv/kernel/patch.o In file included from ./include/linux/kernel.h:11, from ./include/linux/list.h:9, from ./include/linux/preempt.h:11, from ./include/linux/spinlock.h:51, from arch/riscv/kernel/patch.c:6: In function ‘fix_to_virt’, inlined from ‘patch_map’ at arch/riscv/kernel/patch.c:37:17: ./include/linux/compiler.h:392:38: error: call to ‘__compiletime_assert_205’ declared with attribute error: BUILD_BUG_ON failed: idx >= __end_of_fixed_addresses _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__) ^ ./include/linux/compiler.h:373:4: note: in definition of macro ‘__compiletime_assert’ prefix ## suffix(); \ ^~~~~~ ./include/linux/compiler.h:392:2: note: in expansion of macro ‘_compiletime_assert’ _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__) ^~~~~~~~~~~~~~~~~~~ ./include/linux/build_bug.h:39:37: note: in expansion of macro ‘compiletime_assert’ #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg) ^~~~~~~~~~~~~~~~~~ ./include/linux/build_bug.h:50:2: note: in expansion of macro ‘BUILD_BUG_ON_MSG’ BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition) ^~~~~~~~~~~~~~~~ ./include/asm-generic/fixmap.h:32:2: note: in expansion of macro ‘BUILD_BUG_ON’ BUILD_BUG_ON(idx >= __end_of_fixed_addresses); ^~~~~~~~~~~~ Because fix_to_virt(, idx) needs a const value, not a dynamic variable of reg-a0 or BUILD_BUG_ON failed with "idx >= __end_of_fixed_addresses". Signed-off-by: Guo Ren <[email protected]> Reviewed-by: Masami Hiramatsu <[email protected]> Reviewed-by: Pekka Enberg <[email protected]> Signed-off-by: Palmer Dabbelt <[email protected]>
2020-06-17maccess: rename probe_kernel_{read,write} to copy_{from,to}_kernel_nofaultChristoph Hellwig1-2/+2
Better describe what these functions do. Suggested-by: Linus Torvalds <[email protected]> Signed-off-by: Christoph Hellwig <[email protected]> Signed-off-by: Linus Torvalds <[email protected]>
2020-06-09riscv: fix build warning of missing prototypesZong Li1-0/+1
Add the missing header in file, it was lost in original implementation. The warning message as follows: - no previous prototype for 'patch_text_nosync' [-Wmissing-prototypes] - no previous prototype for 'patch_text' [-Wmissing-prototypes] Changed in v2: - Correct the typo of commit message. Signed-off-by: Zong Li <[email protected]> Reported-by: kbuild test robot <[email protected]> Reviewed-by: Atish Patra <[email protected]> Signed-off-by: Palmer Dabbelt <[email protected]>
2020-05-18riscv: Use text_mutex instead of patch_lockZong Li1-6/+7
We don't need the additional lock protection when patching the text. There are two patching interfaces here: - patch_text: patch code and always synchronize with stop_machine() - patch_text_nosync: patch code without synchronization, it's caller's responsibility to synchronize all CPUs if needed. For the first one, stop_machine() is protected by its own mutex, and also the irq is already disabled here. For the second one, in risc-v real case now, it would be used to ftrace patching the mcount function, since it already running under kstop_machine(), no other thread will run, so we could use text_mutex on ftrace side. Signed-off-by: Zong Li <[email protected]> Reviewed-by: Masami Hiramatsu <[email protected]> Reviewed-by: Palmer Dabbelt <[email protected]> Signed-off-by: Palmer Dabbelt <[email protected]>
2020-05-18riscv: Use NOKPROBE_SYMBOL() instead of __krpobes annotationZong Li1-7/+14
The __kprobes annotation is old style, so change it to NOKPROBE_SYMBOL(). Signed-off-by: Zong Li <[email protected]> Reviewed-by: Masami Hiramatsu <[email protected]> Reviewed-by: Palmer Dabbelt <[email protected]> Signed-off-by: Palmer Dabbelt <[email protected]>
2020-05-18riscv: Remove the 'riscv_' prefix of function nameZong Li1-11/+11
Refactor the function name by removing the 'riscv_' prefix, it would be better unless it could mix up with arch-independent functions. Signed-off-by: Zong Li <[email protected]> Reviewed-by: Masami Hiramatsu <[email protected]> Reviewed-by: Palmer Dabbelt <[email protected]> Signed-off-by: Palmer Dabbelt <[email protected]>
2020-03-26riscv: introduce interfaces to patch kernel codeZong Li1-0/+120
On strict kernel memory permission, we couldn't patch code without writable permission. Preserve two holes in fixmap area, so we can map the kernel code temporarily to fixmap area, then patch the instructions. We need two pages here because we support the compressed instruction, so the instruction might be align to 2 bytes. When patching the 32-bit length instruction which is 2 bytes alignment, it will across two pages. Introduce two interfaces to patch kernel code: riscv_patch_text_nosync: - patch code without synchronization, it's caller's responsibility to synchronize all CPUs if needed. riscv_patch_text: - patch code and always synchronize with stop_machine() Signed-off-by: Zong Li <[email protected]> Signed-off-by: Palmer Dabbelt <[email protected]>
Powered by cgit, Gruvboxed by Blaster4385.