diff options
| author | Steve Capper <[email protected]> | 2014-10-09 15:29:25 -0700 |
|---|---|---|
| committer | Linus Torvalds <[email protected]> | 2014-10-09 22:26:01 -0400 |
| commit | 29e5694054149acd25b0d5538c95fb6d64478315 (patch) | |
| tree | 1f496525d51e1989b3415bf275034213a639bbc8 /arch/arm64/mm/flush.c | |
| parent | 5e5f6dc10546f5c03bc572e3ba3089af30c66e2d (diff) | |
arm64: mm: enable RCU fast_gup
Activate the RCU fast_gup for ARM64. We also need to force THP splits to
broadcast an IPI s.t. we block in the fast_gup page walker. As THP
splits are comparatively rare, this should not lead to a noticeable
performance degradation.
Some pre-requisite functions pud_write and pud_page are also added.
[[email protected]: coding-style fixes]
Signed-off-by: Steve Capper <[email protected]>
Tested-by: Dann Frazier <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Cc: Hugh Dickins <[email protected]>
Cc: Russell King <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Mel Gorman <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Christoffer Dall <[email protected]>
Cc: Andrea Arcangeli <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
Diffstat (limited to 'arch/arm64/mm/flush.c')
| -rw-r--r-- | arch/arm64/mm/flush.c | 16 |
1 files changed, 16 insertions, 0 deletions
diff --git a/arch/arm64/mm/flush.c b/arch/arm64/mm/flush.c index 0d64089d28b5..b6f14e8d2121 100644 --- a/arch/arm64/mm/flush.c +++ b/arch/arm64/mm/flush.c @@ -104,3 +104,19 @@ EXPORT_SYMBOL(flush_dcache_page); */ EXPORT_SYMBOL(flush_cache_all); EXPORT_SYMBOL(flush_icache_range); + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_HAVE_RCU_TABLE_FREE +void pmdp_splitting_flush(struct vm_area_struct *vma, unsigned long address, + pmd_t *pmdp) +{ + pmd_t pmd = pmd_mksplitting(*pmdp); + + VM_BUG_ON(address & ~PMD_MASK); + set_pmd_at(vma->vm_mm, address, pmdp, pmd); + + /* dummy IPI to serialise against fast_gup */ + kick_all_cpus_sync(); +} +#endif /* CONFIG_HAVE_RCU_TABLE_FREE */ +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ |