rwsem: Implement writer lock-stealing for better scalability

Commit 5a505085f043 ("mm/rmap: Convert the struct anon_vma::mutex
to an rwsem") changed struct anon_vma::mutex to an rwsem, which
caused aim7 fork_test performance to drop by 50%.

Yuanhan Liu did the following excellent analysis:

    https://lkml.org/lkml/2013/1/29/84

and found that the regression is caused by strict, serialized,
FIFO sequential write-ownership of rwsems. Ingo suggested
implementing opportunistic lock-stealing for the front writer
task in the waitqueue.

Yuanhan Liu implemented lock-stealing for spinlock-rwsems,
which indeed recovered much of the regression - confirming
the analysis that the main factor in the regression was the
FIFO writer-fairness of rwsems.

In this patch we allow lock-stealing to happen when the first
waiter is also writer. With that change in place the
aim7 fork_test performance is fully recovered on my
Intel NHM EP, NHM EX, SNB EP 2S and 4S test-machines.

Reported-by: [email protected]
Reported-by: Yuanhan Liu <[email protected]>
Signed-off-by: Alex Shi <[email protected]>
Cc: David Howells <[email protected]>
Cc: Michel Lespinasse <[email protected]>
Cc: Linus Torvalds <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Anton Blanchard <[email protected]>
Cc: Arjan van de Ven <[email protected]>
Cc: [email protected]
Link: https://lkml.org/lkml/2013/1/29/84
Link: http://lkml.kernel.org/r/[email protected]
[ Small stylistic fixes, updated changelog. ]
Signed-off-by: Ingo Molnar <[email protected]>

0 files changed, 0 insertions, 0 deletions