diff options
| author | Jesper Dangaard Brouer <[email protected]> | 2021-04-29 23:01:51 -0700 |
|---|---|---|
| committer | Linus Torvalds <[email protected]> | 2021-04-30 11:20:43 -0700 |
| commit | ce76f9a1d9a21c2633dcd2a5605f923286e16e1d (patch) | |
| tree | ac88cdbe2dcb1b12d204bbe2fb0401d98b61d81b | |
| parent | 0f87d9d30f21390dd71114f30e63038980e6eb3f (diff) | |
mm/page_alloc: optimize code layout for __alloc_pages_bulk
Looking at perf-report and ASM-code for __alloc_pages_bulk() it is clear
that the code activated is suboptimal. The compiler guesses wrong and
places unlikely code at the beginning. Due to the use of WARN_ON_ONCE()
macro the UD2 asm instruction is added to the code, which confuse the
I-cache prefetcher in the CPU.
[[email protected]: minor changes and rebasing]
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Jesper Dangaard Brouer <[email protected]>
Signed-off-by: Mel Gorman <[email protected]>
Reviewed-by: Alexander Lobakin <[email protected]>
Acked-By: Vlastimil Babka <[email protected]>
Cc: Alexander Duyck <[email protected]>
Cc: Christoph Hellwig <[email protected]>
Cc: Chuck Lever <[email protected]>
Cc: David Miller <[email protected]>
Cc: Ilias Apalodimas <[email protected]>
Cc: Matthew Wilcox (Oracle) <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
| -rw-r--r-- | mm/page_alloc.c | 6 |
1 files changed, 3 insertions, 3 deletions
diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e240704b5d39..3f2291695b19 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5042,7 +5042,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, unsigned int alloc_flags = ALLOC_WMARK_LOW; int nr_populated = 0; - if (WARN_ON_ONCE(nr_pages <= 0)) + if (unlikely(nr_pages <= 0)) return 0; /* @@ -5089,7 +5089,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, * If there are no allowed local zones that meets the watermarks then * try to allocate a single page and reclaim if necessary. */ - if (!zone) + if (unlikely(!zone)) goto failed; /* Attempt the batch allocation */ @@ -5107,7 +5107,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, page = __rmqueue_pcplist(zone, ac.migratetype, alloc_flags, pcp, pcp_list); - if (!page) { + if (unlikely(!page)) { /* Try and get at least one page */ if (!nr_populated) goto failed_irq; |