| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
* kvm-arm64/tlbi-range:
: .
: FEAT_TLBIRANGE support, courtesy of Raghavendra Rao Ananta.
: From the cover letter:
:
: "In certain code paths, KVM/ARM currently invalidates the entire VM's
: page-tables instead of just invalidating a necessary range. For example,
: when collapsing a table PTE to a block PTE, instead of iterating over
: each PTE and flushing them, KVM uses 'vmalls12e1is' TLBI operation to
: flush all the entries. This is inefficient since the guest would have
: to refill the TLBs again, even for the addresses that aren't covered
: by the table entry. The performance impact would scale poorly if many
: addresses in the VM is going through this remapping.
:
: For architectures that implement FEAT_TLBIRANGE, KVM can replace such
: inefficient paths by performing the invalidations only on the range of
: addresses that are in scope. This series tries to achieve the same in
: the areas of stage-2 map, unmap and write-protecting the pages."
: .
KVM: arm64: Use TLBI range-based instructions for unmap
KVM: arm64: Invalidate the table entries upon a range
KVM: arm64: Flush only the memslot after write-protect
KVM: arm64: Implement kvm_arch_flush_remote_tlbs_range()
KVM: arm64: Define kvm_tlb_flush_vmid_range()
KVM: arm64: Implement __kvm_tlb_flush_vmid_range()
arm64: tlb: Implement __flush_s2_tlb_range_op()
arm64: tlb: Refactor the core flush algorithm of __flush_tlb_range
KVM: Move kvm_arch_flush_remote_tlbs_memslot() to common code
KVM: Allow range-based TLB invalidation from common code
KVM: Remove CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL
KVM: arm64: Use kvm_arch_flush_remote_tlbs()
KVM: Declare kvm_arch_flush_remote_tlbs() globally
KVM: Rename kvm_arch_flush_remote_tlb() to kvm_arch_flush_remote_tlbs()
Signed-off-by: Marc Zyngier <[email protected]>
|
|
* kvm-arm64/nv-trap-forwarding: (30 commits)
: .
: This implements the so called "trap forwarding" infrastructure, which
: gets used when we take a trap from an L2 guest and that the L1 guest
: wants to see the trap for itself.
: .
KVM: arm64: nv: Add trap description for SPSR_EL2 and ELR_EL2
KVM: arm64: nv: Select XARRAY_MULTI to fix build error
KVM: arm64: nv: Add support for HCRX_EL2
KVM: arm64: Move HCRX_EL2 switch to load/put on VHE systems
KVM: arm64: nv: Expose FGT to nested guests
KVM: arm64: nv: Add switching support for HFGxTR/HDFGxTR
KVM: arm64: nv: Expand ERET trap forwarding to handle FGT
KVM: arm64: nv: Add SVC trap forwarding
KVM: arm64: nv: Add trap forwarding for HDFGxTR_EL2
KVM: arm64: nv: Add trap forwarding for HFGITR_EL2
KVM: arm64: nv: Add trap forwarding for HFGxTR_EL2
KVM: arm64: nv: Add fine grained trap forwarding infrastructure
KVM: arm64: nv: Add trap forwarding for CNTHCTL_EL2
KVM: arm64: nv: Add trap forwarding for MDCR_EL2
KVM: arm64: nv: Expose FEAT_EVT to nested guests
KVM: arm64: nv: Add trap forwarding for HCR_EL2
KVM: arm64: nv: Add trap forwarding infrastructure
KVM: arm64: Restructure FGT register switching
KVM: arm64: nv: Add FGT registers
KVM: arm64: Add missing HCR_EL2 trap bits
...
Signed-off-by: Marc Zyngier <[email protected]>
|
|
commit f922c13e778d ("KVM: arm64: Introduce
pkvm_alloc_private_va_range()") and commit 92abe0f81e13 ("KVM: arm64:
Introduce hyp_alloc_private_va_range()") added an alignment for the
start address of any allocation into the nVHE hypervisor private VA
range.
This alignment (order of the size of the allocation) intends to enable
efficient stack verification (if the PAGE_SHIFT bit is zero, the stack
pointer is on the guard page and a stack overflow occurred).
But this is only necessary for stack allocation and can waste a lot of
VA space. So instead make stack-specific functions, handling the guard
page requirements, while other users (e.g. fixmap) will only get page
alignment.
Reviewed-by: Kalesh Singh <[email protected]>
Signed-off-by: Vincent Donnefort <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next
Daniel Borkmann says:
====================
pull-request: bpf-next 2023-08-25
We've added 87 non-merge commits during the last 8 day(s) which contain
a total of 104 files changed, 3719 insertions(+), 4212 deletions(-).
The main changes are:
1) Add multi uprobe BPF links for attaching multiple uprobes
and usdt probes, which is significantly faster and saves extra fds,
from Jiri Olsa.
2) Add support BPF cpu v4 instructions for arm64 JIT compiler,
from Xu Kuohai.
3) Add support BPF cpu v4 instructions for riscv64 JIT compiler,
from Pu Lehui.
4) Fix LWT BPF xmit hooks wrt their return values where propagating
the result from skb_do_redirect() would trigger a use-after-free,
from Yan Zhai.
5) Fix a BPF verifier issue related to bpf_kptr_xchg() with local kptr
where the map's value kptr type and locally allocated obj type
mismatch, from Yonghong Song.
6) Fix BPF verifier's check_func_arg_reg_off() function wrt graph
root/node which bypassed reg->off == 0 enforcement,
from Kumar Kartikeya Dwivedi.
7) Lift BPF verifier restriction in networking BPF programs to treat
comparison of packet pointers not as a pointer leak,
from Yafang Shao.
8) Remove unmaintained XDP BPF samples as they are maintained
in xdp-tools repository out of tree, from Toke Høiland-Jørgensen.
9) Batch of fixes for the tracing programs from BPF samples in order
to make them more libbpf-aware, from Daniel T. Lee.
10) Fix a libbpf signedness determination bug in the CO-RE relocation
handling logic, from Andrii Nakryiko.
11) Extend libbpf to support CO-RE kfunc relocations. Also follow-up
fixes for bpf_refcount shared ownership implementation,
both from Dave Marchevsky.
12) Add a new bpf_object__unpin() API function to libbpf,
from Daniel Xu.
13) Fix a memory leak in libbpf to also free btf_vmlinux
when the bpf_object gets closed, from Hao Luo.
14) Small error output improvements to test_bpf module, from Helge Deller.
* tag 'for-netdev' of https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (87 commits)
selftests/bpf: Add tests for rbtree API interaction in sleepable progs
bpf: Allow bpf_spin_{lock,unlock} in sleepable progs
bpf: Consider non-owning refs to refcounted nodes RCU protected
bpf: Reenable bpf_refcount_acquire
bpf: Use bpf_mem_free_rcu when bpf_obj_dropping refcounted nodes
bpf: Consider non-owning refs trusted
bpf: Ensure kptr_struct_meta is non-NULL for collection insert and refcount_acquire
selftests/bpf: Enable cpu v4 tests for RV64
riscv, bpf: Support unconditional bswap insn
riscv, bpf: Support signed div/mod insns
riscv, bpf: Support 32-bit offset jmp insn
riscv, bpf: Support sign-extension mov insns
riscv, bpf: Support sign-extension load insns
riscv, bpf: Fix missing exception handling and redundant zext for LDX_B/H/W
samples/bpf: Add note to README about the XDP utilities moved to xdp-tools
samples/bpf: Cleanup .gitignore
samples/bpf: Remove the xdp_sample_pkts utility
samples/bpf: Remove the xdp1 and xdp2 utilities
samples/bpf: Remove the xdp_rxq_info utility
samples/bpf: Remove the xdp_redirect* utilities
...
====================
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Jakub Kicinski <[email protected]>
|
|
* for-next/perf:
drivers/perf: hisi: Update HiSilicon PMU maintainers
arm_pmu: acpi: Add a representative platform device for TRBE
arm_pmu: acpi: Refactor arm_spe_acpi_register_device()
hw_breakpoint: fix single-stepping when using bpf_overflow_handler
perf/imx_ddr: don't enable counter0 if none of 4 counters are used
perf/imx_ddr: speed up overflow frequency of cycle
drivers/perf: hisi: Schedule perf session according to locality
perf/arm-dmc620: Fix dmc620_pmu_irqs_lock/cpu_hotplug_lock circular lock dependency
perf/smmuv3: Add MODULE_ALIAS for module auto loading
perf/smmuv3: Enable HiSilicon Erratum 162001900 quirk for HIP08/09
perf: pmuv3: Remove comments from armv8pmu_[enable|disable]_event()
perf/arm-cmn: Add CMN-700 r3 support
perf/arm-cmn: Refactor HN-F event selector macros
perf/arm-cmn: Remove spurious event aliases
drivers/perf: Explicitly include correct DT includes
perf: pmuv3: Add Cortex A520, A715, A720, X3 and X4 PMUs
dt-bindings: arm: pmu: Add Cortex A520, A715, A720, X3, and X4
perf/smmuv3: Remove build dependency on ACPI
perf: xgene_pmu: Convert to devm_platform_ioremap_resource()
driver/perf: Add identifier sysfs file for Yitian 710 DDR
|
|
* for-next/mm:
arm64: fix build warning for ARM64_MEMSTART_SHIFT
arm64: Remove unsued extern declaration init_mem_pgprot()
arm64/mm: Set only the PTE_DIRTY bit while preserving the HW dirty state
arm64/mm: Add pte_rdonly() helper
arm64/mm: Directly use ID_AA64MMFR2_EL1_VARange_MASK
arm64/mm: Replace an open coding with ID_AA64MMFR1_EL1_HAFDBS_MASK
|
|
* for-next/misc:
arm64/sysreg: refactor deprecated strncpy
arm64: sysreg: Generate C compiler warnings on {read,write}_sysreg_s arguments
arm64: sdei: abort running SDEI handlers during crash
arm64: Explicitly include correct DT includes
arm64/Kconfig: Sort the RCpc feature under the ARMv8.3 features menu
arm64: vdso: remove two .altinstructions related symbols
arm64/ptrace: Clean up error handling path in sve_set_common()
|
|
Add set_ptes(), update_mmu_cache_range() and flush_dcache_folio(). Change
the PG_dcache_clean flag from being per-page to per-folio.
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Matthew Wilcox (Oracle) <[email protected]>
Reviewed-by: Catalin Marinas <[email protected]>
Acked-by: Mike Rapoport (IBM) <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
|
|
Tell the page table check how many PTEs & PFNs we want it to check.
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Matthew Wilcox (Oracle) <[email protected]>
Reviewed-by: Mike Rapoport (IBM) <[email protected]>
Acked-by: Pasha Tatashin <[email protected]>
Reviewed-by: Anshuman Khandual <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
|
|
Huang Shijie reports that, when profiling a guest from the host
with a number of events that exceeds the number of available
counters, the reported counts are wildly inaccurate. Without
the counter oversubscription, the reported counts are correct.
Their investigation indicates that upon counter rotation (which
takes place on the back of a timer interrupt), we fail to
re-apply the guest EL0 enabling, leading to the counting of host
events instead of guest events.
In order to solve this, add yet another hook between the host PMU
driver and KVM, re-applying the guest EL0 configuration if the
right conditions apply (the host is VHE, we are in interrupt
context, and we interrupted a running vcpu). This triggers a new
vcpu request which will apply the correct configuration on guest
reentry.
With this, we have the correct counts, even when the counters are
oversubscribed.
Reported-by: Huang Shijie <[email protected]>
Suggested-by: Oliver Upton <[email protected]>
Tested_by: Huang Shijie <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Cc: Leo Yan <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Will Deacon <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Acked-by: Mark Rutland <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
PG_dcache_clean is used in asm/hugetlb.h but defined in asm/cacheflush.h:
builds rely on an accident of that being included via linux/mempolicy.h,
but better include it directly (like arch/sh/include/asm/hugetlb.h does).
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Hugh Dickins <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Mike Kravetz <[email protected]>
Cc: Palmer Dabbelt <[email protected]>
Cc: Russell King <[email protected]>
Cc: Will Deacon <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
|
|
As part of the conversions to replace pgtable constructor/destructors with
ptdesc equivalents, convert various page table functions to use ptdescs.
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Vishal Moola (Oracle) <[email protected]>
Acked-by: Mike Rapoport (IBM) <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Christophe Leroy <[email protected]>
Cc: Claudio Imbrenda <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: "David S. Miller" <[email protected]>
Cc: Dinh Nguyen <[email protected]>
Cc: Geert Uytterhoeven <[email protected]>
Cc: Geert Uytterhoeven <[email protected]>
Cc: Guo Ren <[email protected]>
Cc: Huacai Chen <[email protected]>
Cc: Hugh Dickins <[email protected]>
Cc: John Paul Adrian Glaubitz <[email protected]>
Cc: Jonas Bonn <[email protected]>
Cc: Matthew Wilcox <[email protected]>
Cc: Palmer Dabbelt <[email protected]>
Cc: Paul Walmsley <[email protected]>
Cc: Richard Weinberger <[email protected]>
Cc: Thomas Bogendoerfer <[email protected]>
Cc: Yoshinori Sato <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
|
|
It is better to use huge page size instead of PAGE_SIZE for stride when
flush hugepage, which reduces the loop in __flush_tlb_range().
Let's support arch's flush_hugetlb_tlb_range(), which is used in
hugetlb_unshare_all_pmds(), move_hugetlb_page_tables() and
hugetlb_change_protection() for now.
Note,: for hugepages based on contiguous bit, it has to be invalidated
individually since the contiguous PTE bit is just a hint, the hardware may
or may not take it into account.
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Kefeng Wang <[email protected]>
Reviewed-by: Muchun Song <[email protected]>
Reviewed-by: Catalin Marinas <[email protected]>
Cc: Barry Song <[email protected]>
Cc: Joel Fernandes (Google) <[email protected]>
Cc: Kalesh Singh <[email protected]>
Cc: "Kirill A. Shutemov" <[email protected]>
Cc: Mike Kravetz <[email protected]>
Cc: Mina Almasry <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: William Kucharski <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
|
|
Add comments for arch_flush_tlb_batched_pending() and
arch_tlbbatch_flush() to illustrate why only a DSB is needed.
Link: https://lkml.kernel.org/r/[email protected]
Cc: Catalin Marinas <[email protected]>
Signed-off-by: Yicong Yang <[email protected]>
Reviewed-by: Alistair Popple <[email protected]>
Reviewed-by: Catalin Marinas <[email protected]>
Cc: Barry Song <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
|
|
Only the arch_efi_call_virt() macro that some architectures override
needs to be a macro, given that it is variadic and encapsulates calls
via function pointers that have different prototypes.
The associated setup and teardown code are not special in this regard,
and don't need to be instantiated at each call site. So turn them into
ordinary C functions and move them out of line.
Acked-by: Catalin Marinas <[email protected]>
Signed-off-by: Ard Biesheuvel <[email protected]>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
Pull arm64 fixes from Catalin Marinas:
"Two more SME fixes related to ptrace(): ensure that the SME is
properly set up for the target thread and that the thread sees
the ZT registers set via ptrace"
* tag 'arm64-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux:
arm64/ptrace: Ensure that the task sees ZT writes on first use
arm64/ptrace: Ensure that SME is set up for target when writing SSVE state
|
|
There are two main use cases for mmu notifiers. One is by KVM which uses
mmu_notifier_invalidate_range_start()/end() to manage a software TLB.
The other is to manage hardware TLBs which need to use the
invalidate_range() callback because HW can establish new TLB entries at
any time. Hence using start/end() can lead to memory corruption as these
callbacks happen too soon/late during page unmap.
mmu notifier users should therefore either use the start()/end() callbacks
or the invalidate_range() callbacks. To make this usage clearer rename
the invalidate_range() callback to arch_invalidate_secondary_tlbs() and
update documention.
Link: https://lkml.kernel.org/r/6f77248cd25545c8020a54b4e567e8b72be4dca1.1690292440.git-series.apopple@nvidia.com
Signed-off-by: Alistair Popple <[email protected]>
Suggested-by: Jason Gunthorpe <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Reviewed-by: Jason Gunthorpe <[email protected]>
Cc: Andrew Donnellan <[email protected]>
Cc: Chaitanya Kumar Borah <[email protected]>
Cc: Frederic Barrat <[email protected]>
Cc: Jason Gunthorpe <[email protected]>
Cc: John Hubbard <[email protected]>
Cc: Kevin Tian <[email protected]>
Cc: Michael Ellerman <[email protected]>
Cc: Nicholas Piggin <[email protected]>
Cc: Nicolin Chen <[email protected]>
Cc: Robin Murphy <[email protected]>
Cc: Sean Christopherson <[email protected]>
Cc: SeongJae Park <[email protected]>
Cc: Tvrtko Ursulin <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Zhi Wang <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
|
|
The invalidate_range() is going to become an architecture specific mmu
notifier used to keep the TLB of secondary MMUs such as an IOMMU in sync
with the CPU page tables. Currently it is called from separate code paths
to the main CPU TLB invalidations. This can lead to a secondary TLB not
getting invalidated when required and makes it hard to reason about when
exactly the secondary TLB is invalidated.
To fix this move the notifier call to the architecture specific TLB
maintenance functions for architectures that have secondary MMUs requiring
explicit software invalidations.
This fixes a SMMU bug on ARM64. On ARM64 PTE permission upgrades require
a TLB invalidation. This invalidation is done by the architecture
specific ptep_set_access_flags() which calls flush_tlb_page() if required.
However this doesn't call the notifier resulting in infinite faults being
generated by devices using the SMMU if it has previously cached a
read-only PTE in it's TLB.
Moving the invalidations into the TLB invalidation functions ensures all
invalidations happen at the same time as the CPU invalidation. The
architecture specific flush_tlb_all() routines do not call the notifier as
none of the IOMMUs require this.
Link: https://lkml.kernel.org/r/0287ae32d91393a582897d6c4db6f7456b1001f2.1690292440.git-series.apopple@nvidia.com
Signed-off-by: Alistair Popple <[email protected]>
Suggested-by: Jason Gunthorpe <[email protected]>
Tested-by: SeongJae Park <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Reviewed-by: Jason Gunthorpe <[email protected]>
Tested-by: Luis Chamberlain <[email protected]>
Cc: Andrew Donnellan <[email protected]>
Cc: Chaitanya Kumar Borah <[email protected]>
Cc: Frederic Barrat <[email protected]>
Cc: John Hubbard <[email protected]>
Cc: Kevin Tian <[email protected]>
Cc: Michael Ellerman <[email protected]>
Cc: Nicholas Piggin <[email protected]>
Cc: Nicolin Chen <[email protected]>
Cc: Robin Murphy <[email protected]>
Cc: Sean Christopherson <[email protected]>
Cc: Tvrtko Ursulin <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Zhi Wang <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
|
|
On x86, batched and deferred tlb shootdown has lead to 90% performance
increase on tlb shootdown. on arm64, HW can do tlb shootdown without
software IPI. But sync tlbi is still quite expensive.
Even running a simplest program which requires swapout can
prove this is true,
#include <sys/types.h>
#include <unistd.h>
#include <sys/mman.h>
#include <string.h>
int main()
{
#define SIZE (1 * 1024 * 1024)
volatile unsigned char *p = mmap(NULL, SIZE, PROT_READ | PROT_WRITE,
MAP_SHARED | MAP_ANONYMOUS, -1, 0);
memset(p, 0x88, SIZE);
for (int k = 0; k < 10000; k++) {
/* swap in */
for (int i = 0; i < SIZE; i += 4096) {
(void)p[i];
}
/* swap out */
madvise(p, SIZE, MADV_PAGEOUT);
}
}
Perf result on snapdragon 888 with 8 cores by using zRAM
as the swap block device.
~ # perf record taskset -c 4 ./a.out
[ perf record: Woken up 10 times to write data ]
[ perf record: Captured and wrote 2.297 MB perf.data (60084 samples) ]
~ # perf report
# To display the perf.data header info, please use --header/--header-only options.
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 60K of event 'cycles'
# Event count (approx.): 35706225414
#
# Overhead Command Shared Object Symbol
# ........ ....... ................. ......
#
21.07% a.out [kernel.kallsyms] [k] _raw_spin_unlock_irq
8.23% a.out [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore
6.67% a.out [kernel.kallsyms] [k] filemap_map_pages
6.16% a.out [kernel.kallsyms] [k] __zram_bvec_write
5.36% a.out [kernel.kallsyms] [k] ptep_clear_flush
3.71% a.out [kernel.kallsyms] [k] _raw_spin_lock
3.49% a.out [kernel.kallsyms] [k] memset64
1.63% a.out [kernel.kallsyms] [k] clear_page
1.42% a.out [kernel.kallsyms] [k] _raw_spin_unlock
1.26% a.out [kernel.kallsyms] [k] mod_zone_state.llvm.8525150236079521930
1.23% a.out [kernel.kallsyms] [k] xas_load
1.15% a.out [kernel.kallsyms] [k] zram_slot_lock
ptep_clear_flush() takes 5.36% CPU in the micro-benchmark swapping in/out
a page mapped by only one process. If the page is mapped by multiple
processes, typically, like more than 100 on a phone, the overhead would be
much higher as we have to run tlb flush 100 times for one single page.
Plus, tlb flush overhead will increase with the number of CPU cores due to
the bad scalability of tlb shootdown in HW, so those ARM64 servers should
expect much higher overhead.
Further perf annonate shows 95% cpu time of ptep_clear_flush is actually
used by the final dsb() to wait for the completion of tlb flush. This
provides us a very good chance to leverage the existing batched tlb in
kernel. The minimum modification is that we only send async tlbi in the
first stage and we send dsb while we have to sync in the second stage.
With the above simplest micro benchmark, collapsed time to finish the
program decreases around 5%.
Typical collapsed time w/o patch:
~ # time taskset -c 4 ./a.out
0.21user 14.34system 0:14.69elapsed
w/ patch:
~ # time taskset -c 4 ./a.out
0.22user 13.45system 0:13.80elapsed
Also tested with benchmark in the commit on Kunpeng920 arm64 server
and observed an improvement around 12.5% with command
`time ./swap_bench`.
w/o w/
real 0m13.460s 0m11.771s
user 0m0.248s 0m0.279s
sys 0m12.039s 0m11.458s
Originally it's noticed a 16.99% overhead of ptep_clear_flush()
which has been eliminated by this patch:
[root@localhost yang]# perf record -- ./swap_bench && perf report
[...]
16.99% swap_bench [kernel.kallsyms] [k] ptep_clear_flush
It is tested on 4,8,128 CPU platforms and shows to be beneficial on
large systems but may not have improvement on small systems like on
a 4 CPU platform.
Also this patch improve the performance of page migration. Using pmbench
and tries to migrate the pages of pmbench between node 0 and node 1 for
100 times for 1G memory, this patch decrease the time used around 20%
(prev 18.338318910 sec after 13.981866350 sec) and saved the time used
by ptep_clear_flush().
Link: https://lkml.kernel.org/r/[email protected]
Tested-by: Yicong Yang <[email protected]>
Tested-by: Xin Hao <[email protected]>
Tested-by: Punit Agrawal <[email protected]>
Signed-off-by: Barry Song <[email protected]>
Signed-off-by: Yicong Yang <[email protected]>
Reviewed-by: Kefeng Wang <[email protected]>
Reviewed-by: Xin Hao <[email protected]>
Reviewed-by: Anshuman Khandual <[email protected]>
Reviewed-by: Catalin Marinas <[email protected]>
Cc: Anshuman Khandual <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Cc: Nadav Amit <[email protected]>
Cc: Mel Gorman <[email protected]>
Cc: Anshuman Khandual <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Barry Song <[email protected]>
Cc: Darren Hart <[email protected]>
Cc: Jonathan Cameron <[email protected]>
Cc: lipeifeng <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Ryan Roberts <[email protected]>
Cc: Steven Miao <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Zeng Tao <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
|
|
Since hook functions ioremap_allowed() and iounmap_allowed() will be
obsoleted, add wrapper function ioremap_prot() to contain the the specific
handling in addition to generic_ioremap_prot() invocation.
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Baoquan He <[email protected]>
Reviewed-by: Christoph Hellwig <[email protected]>
Reviewed-by: Kefeng Wang <[email protected]>
Reviewed-by: Mike Rapoport (IBM) <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Alexander Gordeev <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Brian Cain <[email protected]>
Cc: Christian Borntraeger <[email protected]>
Cc: Christophe Leroy <[email protected]>
Cc: Chris Zankel <[email protected]>
Cc: David Laight <[email protected]>
Cc: Geert Uytterhoeven <[email protected]>
Cc: Gerald Schaefer <[email protected]>
Cc: Heiko Carstens <[email protected]>
Cc: Helge Deller <[email protected]>
Cc: "James E.J. Bottomley" <[email protected]>
Cc: John Paul Adrian Glaubitz <[email protected]>
Cc: Jonas Bonn <[email protected]>
Cc: Matthew Wilcox <[email protected]>
Cc: Max Filippov <[email protected]>
Cc: Michael Ellerman <[email protected]>
Cc: Nathan Chancellor <[email protected]>
Cc: Nicholas Piggin <[email protected]>
Cc: Niklas Schnelle <[email protected]>
Cc: Rich Felker <[email protected]>
Cc: Stafford Horne <[email protected]>
Cc: Stefan Kristiansson <[email protected]>
Cc: Sven Schnelle <[email protected]>
Cc: Vasily Gorbik <[email protected]>
Cc: Vineet Gupta <[email protected]>
Cc: Yoshinori Sato <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
|
|
Remove unused addr in __page_table_check_pud_set and
page_table_check_pud_set.
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Kemeng Shi <[email protected]>
Cc: Pavel Tatashin <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
|
|
Remove unused addr in __page_table_check_pmd_set and
page_table_check_pmd_set.
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Kemeng Shi <[email protected]>
Cc: Pavel Tatashin <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
|
|
Remove unused addr in __page_table_check_pte_set and
page_table_check_pte_set.
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Kemeng Shi <[email protected]>
Cc: Pavel Tatashin <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
|
|
Remove unused addr in page_table_check_pmd_clear and
__page_table_check_pmd_clear.
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Kemeng Shi <[email protected]>
Cc: Pavel Tatashin <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
|
|
Remove unused addr in page_table_check_pte_clear and
__page_table_check_pte_clear.
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Kemeng Shi <[email protected]>
Cc: Pavel Tatashin <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
|
|
As a result of the patches "mm: Call arch_swap_restore() from
do_swap_page()" and "mm: Call arch_swap_restore() from unuse_pte()", there
are no circumstances in which a swapped-in page is installed in a page
table without first having arch_swap_restore() called on it. Therefore,
we no longer need the logic in set_pte_at() that restores the tags, so
remove it.
Link: https://lkml.kernel.org/r/[email protected]
Link: https://linux-review.googlesource.com/id/I8ad54476f3b2d0144ccd8ce0c1d7a2963e5ff6f3
Signed-off-by: Peter Collingbourne <[email protected]>
Reviewed-by: Steven Price <[email protected]>
Reviewed-by: Catalin Marinas <[email protected]>
Cc: Alexandru Elisei <[email protected]>
Cc: Chinwen Chang <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: Evgenii Stepanov <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: [email protected]
Cc: kasan-dev <[email protected]>
Cc: "Kuan-Ying Lee (李冠穎)" <[email protected]>
Cc: Qun-Wei Lin <[email protected]>
Cc: Suren Baghdasaryan <[email protected]>
Cc: Vincenzo Frascino <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: "Huang, Ying" <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
|
|
ACPI TRBE does not have a HID for identification which could create and add
a platform device into the platform bus. Also without a platform device, it
cannot be probed and bound to a platform driver.
This creates a dummy platform device for TRBE after ascertaining that ACPI
provides required interrupts uniformly across all cpus on the system. This
device gets created inside drivers/perf/arm_pmu_acpi.c to accommodate TRBE
being built as a module.
Cc: Catalin Marinas <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: [email protected]
Cc: [email protected]
Signed-off-by: Anshuman Khandual <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Will Deacon <[email protected]>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/arnd/asm-generic
Pull asm-generic regression fix from Arnd Bergmann:
"Just one partial revert for a commit from the merge window that caused
annoying behavior when building old kernels on arm64 hosts"
* tag 'asm-generic-fix-6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/arnd/asm-generic:
asm-generic: partially revert "Unify uapi bitsperlong.h for arm64, riscv and loongarch"
|
|
To support BPF sign-extend load instructions, add encoders for
LDRSB/LDRSH/LDRSW.
LDRSB/LDRSH/LDRSW (immediate) is encoded as follows:
3 2 2 2 2 1 0 0
0 7 6 4 2 0 5 0
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| sz|1 1 1|0|0 1|opc| imm12 | Rn | Rt |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
LDRSB/LDRSH/LDRSW (register) is encoded as follows:
3 2 2 2 2 2 1 1 1 1 0 0
0 7 6 4 2 1 6 3 2 0 5 0
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| sz|1 1 1|0|0 0|opc|1| Rm | opt |S|1 0| Rn | Rt |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
where:
- sz
indicates whether 8-bit, 16-bit or 32-bit data is to be loaded
- opc
opc[1] (bit 23) is always 1 and opc[0] == 1 indicates regsize
is 32-bit. Since BPF signed load instructions always exend the
sign bit to bit 63 regardless of whether it loads an 8-bit,
16-bit or 32-bit data. So only 64-bit register size is required.
That is, it's sufficient to set field opc fixed to 0x2.
- opt
Indicates whether to sign extend the offset register Rm and the
effective bits of Rm. We set opt to 0x7 (SXTX) since we'll use
Rm as a sgined 64-bit value in BPF.
- S
Optional only when opt field is 0x3 (LSL)
In short, the above fields are encoded to the values listed below.
sz opc opt S
LDRSB (immediate) 0x0 0x2 na na
LDRSH (immediate) 0x1 0x2 na na
LDRSW (immediate) 0x2 0x2 na na
LDRSB (register) 0x0 0x2 0x7 0
LDRSH (register) 0x1 0x2 0x7 0
LDRSW (register) 0x2 0x2 0x7 0
Signed-off-by: Xu Kuohai <[email protected]>
Signed-off-by: Daniel Borkmann <[email protected]>
Tested-by: Florent Revest <[email protected]>
Acked-by: Florent Revest <[email protected]>
Link: https://lore.kernel.org/bpf/[email protected]
|
|
When we use NT_ARM_SSVE to either enable streaming mode or change the
vector length for a process we do not currently do anything to ensure that
there is storage allocated for the SME specific register state. If the
task had not previously used SME or we changed the vector length then
the task will not have had TIF_SME set or backing storage for ZA/ZT
allocated, resulting in inconsistent register sizes when saving state
and spurious traps which flush the newly set register state.
We should set TIF_SME to disable traps and ensure that storage is
allocated for ZA and ZT if it is not already allocated. This requires
modifying sme_alloc() to make the flush of any existing register state
optional so we don't disturb existing state for ZA and ZT.
Fixes: e12310a0d30f ("arm64/sme: Implement ptrace support for streaming mode SVE registers")
Reported-by: David Spickett <[email protected]>
Signed-off-by: Mark Brown <[email protected]>
Cc: <[email protected]> # 5.19.x
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Catalin Marinas <[email protected]>
|
|
loongarch"
Unifying the asm-generic headers across 32-bit and 64-bit architectures
based on the compiler provided macros was a good idea and appears to work
with all user space, but it caused a regression when building old kernels
on systems that have the new headers installed in /usr/include, as this
combination trips an inconsistency in the kernel's own tools/include
headers that are a mix of userspace and kernel-internal headers.
This affects kernel builds on arm64, riscv64 and loongarch64 systems that
might end up using the "#define __BITS_PER_LONG 32" default from the old
tools headers. Backporting the commit into stable kernels would address
this, but it would still break building kernels without that backport,
and waste time for developers trying to understand the problem.
arm64 build machines are rather common, and on riscv64 this can also
happen in practice, but loongarch64 is probably new enough to not
be used much for building old kernels, so only revert the bits
for arm64 and riscv.
Link: https://lore.kernel.org/all/[email protected]/
Reported-by: Nathan Chancellor <[email protected]>
Fixes: 8386f58f8deda ("asm-generic: Unify uapi bitsperlong.h for arm64, riscv and loongarch")
Acked-by: Catalin Marinas <[email protected]>
Acked-by: Palmer Dabbelt <[email protected]>
Tested-by: Nathan Chancellor <[email protected]>
Signed-off-by: Arnd Bergmann <[email protected]>
|
|
HCRX_EL2 has an interesting effect on HFGITR_EL2, as it conditions
the traps of TLBI*nXS.
Expand the FGT support to add a new Fine Grained Filter that will
get checked when the instruction gets trapped, allowing the shadow
register to override the trap as needed.
Reviewed-by: Eric Auger <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Reviewed-by: Jing Zhang <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
... and finally, the Debug version of FGT, with its *enormous*
list of trapped registers.
Reviewed-by: Suzuki K Poulose <[email protected]>
Reviewed-by: Eric Auger <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Reviewed-by: Jing Zhang <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
Similarly, implement the trap forwarding for instructions affected
by HFGITR_EL2.
Note that the TLBI*nXS instructions should be affected by HCRX_EL2,
which will be dealt with down the line. Also, ERET* and SVC traps
are handled separately.
Reviewed-by: Eric Auger <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Reviewed-by: Jing Zhang <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
A significant part of what a NV hypervisor needs to do is to decide
whether a trap from a L2+ guest has to be forwarded to a L1 guest
or handled locally. This is done by checking for the trap bits that
the guest hypervisor has set and acting accordingly, as described by
the architecture.
A previous approach was to sprinkle a bunch of checks in all the
system register accessors, but this is pretty error prone and doesn't
help getting an overview of what is happening.
Instead, implement a set of global tables that describe a trap bit,
combinations of trap bits, behaviours on trap, and what bits must
be evaluated on a system register trap.
Although this is painful to describe, this allows to specify each
and every control bit in a static manner. To make it efficient,
the table is inserted in an xarray that is global to the system,
and checked each time we trap a system register while running
a L2 guest.
Add the basic infrastructure for now, while additional patches will
implement configuration registers.
Signed-off-by: Marc Zyngier <[email protected]>
Reviewed-by: Jing Zhang <[email protected]>
Reviewed-by: Miguel Luis <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
As we're about to majorly extend the handling of FGT registers,
restructure the code to actually save/restore the registers
as required. This is made easy thanks to the previous addition
of the EL2 registers, allowing us to use the host context for
this purpose.
Reviewed-by: Eric Auger <[email protected]>
Reviewed-by: Oliver Upton <[email protected]>
Reviewed-by: Miguel Luis <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
Add the 5 registers covering FEAT_FGT. The AMU-related registers
are currently left out as we don't have a plan for them. Yet.
Reviewed-by: Eric Auger <[email protected]>
Reviewed-by: Miguel Luis <[email protected]>
Reviewed-by: Jing Zhang <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
We're still missing a handfull of HCR_EL2 trap bits. Add them.
Reviewed-by: Eric Auger <[email protected]>
Reviewed-by: Miguel Luis <[email protected]>
Reviewed-by: Jing Zhang <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
As we blindly reset some HFGxTR_EL2 bits to 0, we also randomly trap
unsuspecting sysregs that have their trap bits with a negative
polarity.
ACCDATA_EL1 is one such register that can be accessed by the guest,
causing a splat on the host as we don't have a proper handler for
it.
Adding such handler addresses the issue, though there are a number
of other registers missing as the current architecture documentation
doesn't describe them yet.
Reviewed-by: Eric Auger <[email protected]>
Reviewed-by: Miguel Luis <[email protected]>
Reviewed-by: Jing Zhang <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
As we're about to implement full support for FEAT_FGT, add the
full HDFGRTR_EL2 and HDFGWTR_EL2 layouts.
Reviewed-by: Mark Brown <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Reviewed-by: Miguel Luis <[email protected]>
Reviewed-by: Jing Zhang <[email protected]>
Reviewed-by: Eric Auger <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
HFGITR_EL2 traps a bunch of instructions for which we don't have
encodings yet. Add them.
Reviewed-by: Miguel Luis <[email protected]>
Reviewed-by: Eric Auger <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Reviewed-by: Jing Zhang <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
The HDFGxTR_EL2 registers trap a (huge) set of debug and trace
related registers. Add their encodings (and only that, because
we really don't care about what these registers actually do at
this stage).
Acked-by: Catalin Marinas <[email protected]>
Reviewed-by: Eric Auger <[email protected]>
Reviewed-by: Jing Zhang <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
Add the encodings for the AT operation that are usable from NS.
Reviewed-by: Eric Auger <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Reviewed-by: Miguel Luis <[email protected]>
Reviewed-by: Zenghui Yu <[email protected]>
Reviewed-by: Jing Zhang <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
Add all the TLBI encodings that are usable from Non-Secure.
Reviewed-by: Eric Auger <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Reviewed-by: Miguel Luis <[email protected]>
Reviewed-by: Zenghui Yu <[email protected]>
Reviewed-by: Jing Zhang <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
Add the missing DC *VA encodings.
Reviewed-by: Eric Auger <[email protected]>
Reviewed-by: Miguel Luis <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Reviewed-by: Zenghui Yu <[email protected]>
Reviewed-by: Jing Zhang <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
We only describe a few of the ERX*_EL1 registers. Add the missing
ones (ERXPFGF_EL1, ERXPFGCTL_EL1, ERXPFGCDN_EL1, ERXMISC2_EL1 and
ERXMISC3_EL1).
Reviewed-by: Eric Auger <[email protected]>
Reviewed-by: Miguel Luis <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Reviewed-by: Zenghui Yu <[email protected]>
Reviewed-by: Jing Zhang <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
Add the missing VA-based CMOs encodings.
Reviewed-by: Eric Auger <[email protected]>
Reviewed-by: Miguel Luis <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Reviewed-by: Zenghui Yu <[email protected]>
Reviewed-by: Jing Zhang <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
Implement kvm_arch_flush_remote_tlbs_range() for arm64
to invalidate the given range in the TLB.
Signed-off-by: Raghavendra Rao Ananta <[email protected]>
Reviewed-by: Gavin Shan <[email protected]>
Reviewed-by: Shaoqin Huang <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
Implement the helper kvm_tlb_flush_vmid_range() that acts
as a wrapper for range-based TLB invalidations. For the
given VMID, use the range-based TLBI instructions to do
the job or fallback to invalidating all the TLB entries.
Signed-off-by: Raghavendra Rao Ananta <[email protected]>
Reviewed-by: Gavin Shan <[email protected]>
Reviewed-by: Shaoqin Huang <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|
|
Define __kvm_tlb_flush_vmid_range() (for VHE and nVHE)
to flush a range of stage-2 page-tables using IPA in one go.
If the system supports FEAT_TLBIRANGE, the following patches
would conveniently replace global TLBI such as vmalls12e1is
in the map, unmap, and dirty-logging paths with ripas2e1is
instead.
Signed-off-by: Raghavendra Rao Ananta <[email protected]>
Reviewed-by: Gavin Shan <[email protected]>
Reviewed-by: Shaoqin Huang <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
|