| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
The aio code tries really hard to avoid having to deal with the
completion ringbuffer overflowing. To do that, it has to keep track of
the number of outstanding kiocbs, and the number of completions
currently in the ringbuffer - and it's got to check that every time we
allocate a kiocb. Ouch.
But - we can improve this quite a bit if we just change reqs_active to
mean "number of outstanding requests and unreaped completions" - that
means kiocb allocation doesn't have to look at the ringbuffer, which is
a fairly significant win.
Signed-off-by: Kent Overstreet <[email protected]>
Cc: Zach Brown <[email protected]>
Cc: Felipe Balbi <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Mark Fasheh <[email protected]>
Cc: Joel Becker <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Jens Axboe <[email protected]>
Cc: Asai Thambi S P <[email protected]>
Cc: Selvan Mani <[email protected]>
Cc: Sam Bradshaw <[email protected]>
Cc: Jeff Moyer <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Benjamin LaHaise <[email protected]>
Signed-off-by: "Theodore Ts'o" <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
Cancelling kiocbs requires adding them to a per kioctx linked list,
which is one of the few things we need to take the kioctx lock for in
the fast path. But most kiocbs can't be cancelled - so if we just do
this lazily, we can avoid quite a bit of locking overhead.
While we're at it, instead of using a flag bit switch to using ki_cancel
itself to indicate that a kiocb has been cancelled/completed. This lets
us get rid of ki_flags entirely.
[[email protected]: remove buggy BUG()]
Signed-off-by: Kent Overstreet <[email protected]>
Cc: Zach Brown <[email protected]>
Cc: Felipe Balbi <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Mark Fasheh <[email protected]>
Cc: Joel Becker <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Jens Axboe <[email protected]>
Cc: Asai Thambi S P <[email protected]>
Cc: Selvan Mani <[email protected]>
Cc: Sam Bradshaw <[email protected]>
Cc: Jeff Moyer <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Benjamin LaHaise <[email protected]>
Reviewed-by: "Theodore Ts'o" <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
This wasn't causing problems before because it's not needed on x86, but
it is needed on other architectures.
Signed-off-by: Kent Overstreet <[email protected]>
Cc: Zach Brown <[email protected]>
Cc: Felipe Balbi <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Mark Fasheh <[email protected]>
Cc: Joel Becker <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Jens Axboe <[email protected]>
Cc: Asai Thambi S P <[email protected]>
Cc: Selvan Mani <[email protected]>
Cc: Sam Bradshaw <[email protected]>
Cc: Jeff Moyer <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Benjamin LaHaise <[email protected]>
Cc: Theodore Ts'o <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
Previously, aio_read_event() pulled a single completion off the
ringbuffer at a time, locking and unlocking each time. Change it to
pull off as many events as it can at a time, and copy them directly to
userspace.
This also fixes a bug where if copying the event to userspace failed,
we'd lose the event.
Also convert it to wait_event_interruptible_hrtimeout(), which
simplifies it quite a bit.
[[email protected]: coding-style fixes]
Signed-off-by: Kent Overstreet <[email protected]>
Cc: Zach Brown <[email protected]>
Cc: Felipe Balbi <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Mark Fasheh <[email protected]>
Cc: Joel Becker <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Jens Axboe <[email protected]>
Cc: Asai Thambi S P <[email protected]>
Cc: Selvan Mani <[email protected]>
Cc: Sam Bradshaw <[email protected]>
Cc: Jeff Moyer <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Benjamin LaHaise <[email protected]>
Reviewed-by: "Theodore Ts'o" <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
Analagous to wait_event_timeout() and friends, this adds
wait_event_hrtimeout() and wait_event_interruptible_hrtimeout().
Note that unlike the versions that use regular timers, these don't
return the amount of time remaining when they return - instead, they
return 0 or -ETIME if they timed out. because I was uncomfortable with
the semantics of doing it the other way (that I could get it right,
anyways).
If the timer expires, there's no real guarantee that expire_time -
current_time would be <= 0 - due to timer slack certainly, and I'm not
sure I want to know the implications of the different clock bases in
hrtimers.
If the timer does expire and the code calculates that the time remaining
is nonnegative, that could be even worse if the calling code then reuses
that timeout. Probably safer to just return 0 then, but I could imagine
weird bugs or at least unintended behaviour arising from that too.
I came to the conclusion that if other users end up actually needing the
amount of time remaining, the sanest thing to do would be to create a
version that uses absolute timeouts instead of relative.
[[email protected]: fix description of `timeout' arg]
Signed-off-by: Kent Overstreet <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Zach Brown <[email protected]>
Cc: Felipe Balbi <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Mark Fasheh <[email protected]>
Cc: Joel Becker <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Jens Axboe <[email protected]>
Cc: Asai Thambi S P <[email protected]>
Cc: Selvan Mani <[email protected]>
Cc: Sam Bradshaw <[email protected]>
Cc: Jeff Moyer <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Benjamin LaHaise <[email protected]>
Reviewed-by: "Theodore Ts'o" <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
The usage of ctx->dead was fubar - it makes no sense to explicitly check
it all over the place, especially when we're already using RCU.
Now, ctx->dead only indicates whether we've dropped the initial
refcount. The new teardown sequence is:
set ctx->dead
hlist_del_rcu();
synchronize_rcu();
Now we know no system calls can take a new ref, and it's safe to drop
the initial ref:
put_ioctx();
We also need to ensure there are no more outstanding kiocbs. This was
done incorrectly - it was being done in kill_ctx(), and before dropping
the initial refcount. At this point, other syscalls may still be
submitting kiocbs!
Now, we cancel and wait for outstanding kiocbs in free_ioctx(), after
kioctx->users has dropped to 0 and we know no more iocbs could be
submitted.
[[email protected]: coding-style fixes]
Signed-off-by: Kent Overstreet <[email protected]>
Cc: Zach Brown <[email protected]>
Cc: Felipe Balbi <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Mark Fasheh <[email protected]>
Cc: Joel Becker <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Jens Axboe <[email protected]>
Cc: Asai Thambi S P <[email protected]>
Cc: Selvan Mani <[email protected]>
Cc: Sam Bradshaw <[email protected]>
Cc: Jeff Moyer <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Benjamin LaHaise <[email protected]>
Reviewed-by: "Theodore Ts'o" <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
Freeing a kiocb needed to touch the kioctx for three things:
* Pull it off the reqs_active list
* Decrementing reqs_active
* Issuing a wakeup, if the kioctx was in the process of being freed.
This patch moves these to aio_complete(), for a couple reasons:
* aio_complete() already has to issue the wakeup, so if we drop the
kioctx refcount before aio_complete does its wakeup we don't have to
do it twice.
* aio_complete currently has to take the kioctx lock, so it makes sense
for it to pull the kiocb off the reqs_active list too.
* A later patch is going to change reqs_active to include unreaped
completions - this will mean allocating a kiocb doesn't have to look
at the ringbuffer. So taking the decrement of reqs_active out of
kiocb_free() is useful prep work for that patch.
This doesn't really affect cancellation, since existing (usb) code that
implements a cancel function still calls aio_complete() - we just have
to make sure that aio_complete does the necessary teardown for cancelled
kiocbs.
It does affect code paths where we free kiocbs that were never
submitted; they need to decrement reqs_active and pull the kiocb off the
reqs_active list. This occurs in two places: kiocb_batch_free(), which
is going away in a later patch, and the error path in io_submit_one.
[[email protected]: coding-style fixes]
Signed-off-by: Kent Overstreet <[email protected]>
Cc: Zach Brown <[email protected]>
Cc: Felipe Balbi <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Mark Fasheh <[email protected]>
Cc: Joel Becker <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Jens Axboe <[email protected]>
Cc: Asai Thambi S P <[email protected]>
Cc: Selvan Mani <[email protected]>
Cc: Sam Bradshaw <[email protected]>
Acked-by: Jeff Moyer <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Benjamin LaHaise <[email protected]>
Reviewed-by: "Theodore Ts'o" <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
aio_get_req() will fail if we have the maximum number of requests
outstanding, which depending on the application may not be uncommon. So
avoid doing an unnecessary fget().
Signed-off-by: Kent Overstreet <[email protected]>
Cc: Zach Brown <[email protected]>
Cc: Felipe Balbi <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Mark Fasheh <[email protected]>
Cc: Joel Becker <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Jens Axboe <[email protected]>
Cc: Asai Thambi S P <[email protected]>
Cc: Selvan Mani <[email protected]>
Cc: Sam Bradshaw <[email protected]>
Acked-by: Jeff Moyer <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Benjamin LaHaise <[email protected]>
Reviewed-by: "Theodore Ts'o" <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
Signed-off-by: Kent Overstreet <[email protected]>
Cc: Zach Brown <[email protected]>
Cc: Felipe Balbi <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Mark Fasheh <[email protected]>
Cc: Joel Becker <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Jens Axboe <[email protected]>
Cc: Asai Thambi S P <[email protected]>
Cc: Selvan Mani <[email protected]>
Cc: Sam Bradshaw <[email protected]>
Acked-by: Jeff Moyer <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Benjamin LaHaise <[email protected]>
Cc: Theodore Ts'o <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
Signed-off-by: Kent Overstreet <[email protected]>
Cc: Zach Brown <[email protected]>
Cc: Felipe Balbi <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Mark Fasheh <[email protected]>
Cc: Joel Becker <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Jens Axboe <[email protected]>
Cc: Asai Thambi S P <[email protected]>
Cc: Selvan Mani <[email protected]>
Cc: Sam Bradshaw <[email protected]>
Acked-by: Jeff Moyer <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Benjamin LaHaise <[email protected]>
Cc: Theodore Ts'o <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
Minor refactoring, to get rid of some duplicated code
[[email protected]: fix warning]
Signed-off-by: Kent Overstreet <[email protected]>
Cc: Zach Brown <[email protected]>
Cc: Felipe Balbi <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Mark Fasheh <[email protected]>
Cc: Joel Becker <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Jens Axboe <[email protected]>
Cc: Asai Thambi S P <[email protected]>
Cc: Selvan Mani <[email protected]>
Cc: Sam Bradshaw <[email protected]>
Acked-by: Jeff Moyer <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Benjamin LaHaise <[email protected]>
Reviewed-by: "Theodore Ts'o" <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
Nothing used the return value, and it probably wasn't possible to use it
safely for the locked versions (aio_complete(), aio_put_req()). Just
kill it.
Signed-off-by: Kent Overstreet <[email protected]>
Acked-by: Zach Brown <[email protected]>
Cc: Felipe Balbi <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Mark Fasheh <[email protected]>
Cc: Joel Becker <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Jens Axboe <[email protected]>
Cc: Asai Thambi S P <[email protected]>
Cc: Selvan Mani <[email protected]>
Cc: Sam Bradshaw <[email protected]>
Acked-by: Jeff Moyer <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Benjamin LaHaise <[email protected]>
Reviewed-by: "Theodore Ts'o" <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
These are handy for measuring the cost of the aio infrastructure with
operations that do very little and complete immediately.
Signed-off-by: Zach Brown <[email protected]>
Signed-off-by: Kent Overstreet <[email protected]>
Cc: Felipe Balbi <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Mark Fasheh <[email protected]>
Cc: Joel Becker <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Jens Axboe <[email protected]>
Cc: Asai Thambi S P <[email protected]>
Cc: Selvan Mani <[email protected]>
Cc: Sam Bradshaw <[email protected]>
Acked-by: Jeff Moyer <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Benjamin LaHaise <[email protected]>
Reviewed-by: "Theodore Ts'o" <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
This removes the retry-based AIO infrastructure now that nothing in tree
is using it.
We want to remove retry-based AIO because it is fundemantally unsafe.
It retries IO submission from a kernel thread that has only assumed the
mm of the submitting task. All other task_struct references in the IO
submission path will see the kernel thread, not the submitting task.
This design flaw means that nothing of any meaningful complexity can use
retry-based AIO.
This removes all the code and data associated with the retry machinery.
The most significant benefit of this is the removal of the locking
around the unused run list in the submission path.
[[email protected]: coding-style fixes]
Signed-off-by: Kent Overstreet <[email protected]>
Signed-off-by: Zach Brown <[email protected]>
Cc: Zach Brown <[email protected]>
Cc: Felipe Balbi <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Mark Fasheh <[email protected]>
Cc: Joel Becker <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Jens Axboe <[email protected]>
Cc: Asai Thambi S P <[email protected]>
Cc: Selvan Mani <[email protected]>
Cc: Sam Bradshaw <[email protected]>
Acked-by: Jeff Moyer <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Benjamin LaHaise <[email protected]>
Reviewed-by: "Theodore Ts'o" <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
This removes the only in-tree user of aio retry. This will let us
remove the retry code from the aio core.
Removing retry is relatively easy as the USB gadget wasn't using it to
retry IOs at all. It always fully submitted the IO in the context of
the initial io_submit() call. It only used the AIO retry facility to
get the submitter's mm context for copying the result of a read back to
user space. This is easy to implement with use_mm() and a work struct,
much like kvm does with async_pf_execute() for get_user_pages().
[[email protected]: coding-style fixes]
Signed-off-by: Zach Brown <[email protected]>
Signed-off-by: Kent Overstreet <[email protected]>
Cc: Felipe Balbi <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Mark Fasheh <[email protected]>
Cc: Joel Becker <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Jens Axboe <[email protected]>
Cc: Asai Thambi S P <[email protected]>
Cc: Selvan Mani <[email protected]>
Cc: Sam Bradshaw <[email protected]>
Acked-by: Jeff Moyer <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Benjamin LaHaise <[email protected]>
Cc: Theodore Ts'o <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
Signed-off-by: Zach Brown <[email protected]>
Signed-off-by: Kent Overstreet <[email protected]>
Cc: Felipe Balbi <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Mark Fasheh <[email protected]>
Cc: Joel Becker <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Jens Axboe <[email protected]>
Cc: Asai Thambi S P <[email protected]>
Cc: Selvan Mani <[email protected]>
Cc: Sam Bradshaw <[email protected]>
Acked-by: Jeff Moyer <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Benjamin LaHaise <[email protected]>
Reviewed-by: "Theodore Ts'o" <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
Bunch of performance improvements and cleanups Zach Brown and I have
been working on. The code should be pretty solid at this point, though
it could of course use more review and testing.
The results in my testing are pretty impressive, particularly when an
ioctx is being shared between multiple threads. In my crappy synthetic
benchmark, with 4 threads submitting and one thread reaping completions,
I saw overhead in the aio code go from ~50% (mostly ioctx lock
contention) to low single digits. Performance with ioctx per thread
improved too, but I'd have to rerun those benchmarks.
The reason I've been focused on performance when the ioctx is shared is
that for a fair number of real world completions, userspace needs the
completions aggregated somehow - in practice people just end up
implementing this aggregation in userspace today, but if it's done right
we can do it much more efficiently in the kernel.
Performance wise, the end result of this patch series is that submitting
a kiocb writes to _no_ shared cachelines - the penalty for sharing an
ioctx is gone there. There's still going to be some cacheline
contention when we deliver the completions to the aio ringbuffer (at
least if you have interrupts being delivered on multiple cores, which
for high end stuff you do) but I have a couple more patches not in this
series that implement coalescing for that (by taking advantage of
interrupt coalescing). With that, there's basically no bottlenecks or
performance issues to speak of in the aio code.
This patch:
use_mm() is used in more places than just aio. There's no need to mention
callers when describing the function.
Signed-off-by: Zach Brown <[email protected]>
Signed-off-by: Kent Overstreet <[email protected]>
Cc: Felipe Balbi <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Mark Fasheh <[email protected]>
Cc: Joel Becker <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Jens Axboe <[email protected]>
Cc: Asai Thambi S P <[email protected]>
Cc: Selvan Mani <[email protected]>
Cc: Sam Bradshaw <[email protected]>
Acked-by: Jeff Moyer <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Benjamin LaHaise <[email protected]>
Reviewed-by: "Theodore Ts'o" <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
Cc: Al Viro <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
After finishing a naming transition, remove unused backward
compatibility wrapper macros
Signed-off-by: Akinobu Mita <[email protected]>
Cc: "Theodore Ts'o" <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
Use preferable function name which implies using a pseudo-random number
generator.
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
Use preferable function name which implies using a pseudo-random number
generator.
[[email protected]: convert team_mode_random.c]
Signed-off-by: Akinobu Mita <[email protected]>
Acked-by: Thomas Sailer <[email protected]>
Acked-by: Bing Zhao <[email protected]> [mwifiex]
Cc: "David S. Miller" <[email protected]>
Cc: Michael Chan <[email protected]>
Cc: Thomas Sailer <[email protected]>
Cc: Jean-Paul Roubelat <[email protected]>
Cc: Bing Zhao <[email protected]>
Cc: Brett Rudley <[email protected]>
Cc: Arend van Spriel <[email protected]>
Cc: "Franky (Zhenhui) Lin" <[email protected]>
Cc: Hante Meuleman <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
The current kernel returns -EINVAL unless a given mmap length is
"almost" hugepage aligned. This is because in sys_mmap_pgoff() the
given length is passed to vm_mmap_pgoff() as it is without being aligned
with hugepage boundary.
This is a regression introduced in commit 40716e29243d ("hugetlbfs: fix
alignment of huge page requests"), where alignment code is pushed into
hugetlb_file_setup() and the variable len in caller side is not changed.
To fix this, this patch partially reverts that commit, and adds
alignment code in caller side. And it also introduces hstate_sizelog()
in order to get proper hstate to specified hugepage size.
Addresses https://bugzilla.kernel.org/show_bug.cgi?id=56881
[[email protected]: fix warning when CONFIG_HUGETLB_PAGE=n]
Signed-off-by: Naoya Horiguchi <[email protected]>
Signed-off-by: Johannes Weiner <[email protected]>
Reported-by: <[email protected]>
Cc: Steven Truelove <[email protected]>
Cc: Jianguo Wu <[email protected]>
Cc: Hugh Dickins <[email protected]>
Cc: <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
kmap_atomic() requires only one argument now.
Signed-off-by: Zhao Hongjiang <[email protected]>
Cc: "James E.J. Bottomley" <[email protected]>
Cc: Helge Deller <[email protected]>
Cc: Rolf Eike Beer <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
Register layout is the same, so just add the variant to the appropriate
places.
Signed-off-by: Lucas Stach <[email protected]>
Signed-off-by: Jan Luebbe <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
That nameless-function-arguments thing drives me batty. Fix.
Cc: Dave Hansen <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
This exports the amount of anonymous transparent hugepages for each
memcg via the new "rss_huge" stat in memory.stat. The units are in
bytes.
This is helpful to determine the hugepage utilization for individual
jobs on the system in comparison to rss and opportunities where
MADV_HUGEPAGE may be helpful.
The amount of anonymous transparent hugepages is also included in "rss"
for backwards compatibility.
Signed-off-by: David Rientjes <[email protected]>
Acked-by: Michal Hocko <[email protected]>
Acked-by: Johannes Weiner <[email protected]>
Cc: KAMEZAWA Hiroyuki <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
Use common help functions to free reserved pages.
Signed-off-by: Jiang Liu <[email protected]>
Acked-by: David S. Miller <[email protected]>
Acked-by: Sam Ravnborg <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
I badly screwed up the merge in commit 6fa52ed33bea ("Merge tag
'drivers-for-linus' of git://git.kernel.org/pub/.../arm-soc") by
incorrectly taking the arch/arm/mach-omap2/* data fully from the merge
target because the 'drivers-for-linus' branch seemed to be a proper
superset of the duplicate ARM commits.
That was bogus: commit ff931c821bab ("ARM: OMAP: clocks: Delay clk inits
atleast until slab is initialized") only existed in head, and the
changes to arch/arm/mach-omap2/timer.c from that commit got list.
Re-doing the merge more carefully, I do think this part was the only
thing I screwed up. Knock wood.
Reported-by: Tony Lindgren <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Olof Johansson <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
Shamelessly copied from dchinner's:
ad650f5b xfs: fallback to vmalloc for large buffers in xfs_attrmulti_attr_get
xfsdump uses a large buffer for extended attributes, which has a
kmalloc'd shadow buffer in the kernel. This can fail after the
system has been running for some time as it is a high order
allocation. Add a fallback to vmalloc so that it doesn't require
contiguous memory and so won't randomly fail while xfsdump is
running.
This was done for xfs_attrlist_by_handle but
xfs_compat_attrlist_by_handle (the 32-bit version) needs the same
attention.
Signed-off-by: Eric Sandeen <[email protected]>
Reviewed-by: Mark Tinguely <[email protected]>
Signed-off-by: Ben Myers <[email protected]>
|
|
Shamelessly copied from dchinner's:
ad650f5b xfs: fallback to vmalloc for large buffers in xfs_attrmulti_attr_get
xfsdump uses for a large buffer for extended attributes, which has a
kmalloc'd shadow buffer in the kernel. This can fail after the
system has been running for some time as it is a high order
allocation. Add a fallback to vmalloc so that it doesn't require
contiguous memory and so won't randomly fail while xfsdump is
running.
Signed-off-by: Eric Sandeen <[email protected]>
Reviewed-by: Mark Tinguely <[email protected]>
Signed-off-by: Ben Myers <[email protected]>
|
|
Running a CONFIG_XFS_DEBUG kernel in production environments is not
the best idea as it introduces significant overhead, can change
the behaviour of algorithms (such as allocation) to improve test
coverage, and (most importantly) panic the machine on non-fatal
errors.
There are many cases where all we want to do is run a
kernel with more bounds checking enabled, such as is provided by the
ASSERT() statements throughout the code, but without all the
potential overhead and drawbacks.
This patch converts all the ASSERT statements to evaluate as
WARN_ON(1) statements and hence if they fail dump a warning and a
stack trace to the log. This has minimal overhead and does not
change any algorithms, and will allow us to find strange "out of
bounds" problems more easily on production machines.
There are a few places where assert statements contain debug only
code. These are converted to be debug-or-warn only code so that we
still get all the assert checks in the code.
Signed-off-by: Dave Chinner <[email protected]>
Reviewed-by: Brian Foster <[email protected]>
Signed-off-by: Ben Myers <[email protected]>
|
|
This patch tries to reduce the amount of cmpxchg calls in the writer
failed path by checking the counter value first before issuing the
instruction. If ->count is not set to RWSEM_WAITING_BIAS then there is
no point wasting a cmpxchg call.
Furthermore, Michel states "I suppose it helps due to the case where
someone else steals the lock while we're trying to acquire
sem->wait_lock."
Two very different workloads and machines were used to see how this
patch improves throughput: pgbench on a quad-core laptop and aim7 on a
large 8 socket box with 80 cores.
Some results comparing Michel's fast-path write lock stealing
(tps-rwsem) on a quad-core laptop running pgbench:
| db_size | clients | tps-rwsem | tps-patch |
+---------+----------+----------------+--------------+
| 160 MB | 1 | 6906 | 9153 | + 32.5
| 160 MB | 2 | 15931 | 22487 | + 41.1%
| 160 MB | 4 | 33021 | 32503 |
| 160 MB | 8 | 34626 | 34695 |
| 160 MB | 16 | 33098 | 34003 |
| 160 MB | 20 | 31343 | 31440 |
| 160 MB | 30 | 28961 | 28987 |
| 160 MB | 40 | 26902 | 26970 |
| 160 MB | 50 | 25760 | 25810 |
------------------------------------------------------
| 1.6 GB | 1 | 7729 | 7537 |
| 1.6 GB | 2 | 19009 | 23508 | + 23.7%
| 1.6 GB | 4 | 33185 | 32666 |
| 1.6 GB | 8 | 34550 | 34318 |
| 1.6 GB | 16 | 33079 | 32689 |
| 1.6 GB | 20 | 31494 | 31702 |
| 1.6 GB | 30 | 28535 | 28755 |
| 1.6 GB | 40 | 27054 | 27017 |
| 1.6 GB | 50 | 25591 | 25560 |
------------------------------------------------------
| 7.6 GB | 1 | 6224 | 7469 | + 20.0%
| 7.6 GB | 2 | 13611 | 12778 |
| 7.6 GB | 4 | 33108 | 32927 |
| 7.6 GB | 8 | 34712 | 34878 |
| 7.6 GB | 16 | 32895 | 33003 |
| 7.6 GB | 20 | 31689 | 31974 |
| 7.6 GB | 30 | 29003 | 28806 |
| 7.6 GB | 40 | 26683 | 26976 |
| 7.6 GB | 50 | 25925 | 25652 |
------------------------------------------------------
For the aim7 worloads, they overall improved on top of Michel's
patchset. For full graphs on how the rwsem series plus this patch
behaves on a large 8 socket machine against a vanilla kernel:
http://stgolabs.net/rwsem-aim7-results.tar.gz
Signed-off-by: Davidlohr Bueso <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
- make warning smp-safe
- result of atomic _unless_zero functions should be checked by caller
to avoid use-after-free error
- trivial whitespace fix.
Link: https://lkml.org/lkml/2013/4/12/391
Tested: compile x86, boot machine and run xfstests
Signed-off-by: Anatol Pomozov <[email protected]>
[ Removed line-break, changed to use WARN_ON_ONCE() - Linus ]
Signed-off-by: Linus Torvalds <[email protected]>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
Pull more vfs updates from Al Viro:
"A couple of fixes + getting rid of __blkdev_put() return value"
* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs:
proc: Use PDE attribute setting accessor functions
make blkdev_put() return void
block_device_operations->release() should return void
mtd_blktrans_ops->release() should return void
hfs: SMP race on directory close()
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/deller/parisc-linux
Pull parisc updates from Helge Deller:
"Main fixes and updates in this patch series are:
- we faced kernel stack corruptions because of multiple delivery of
interrupts
- added kernel stack overflow checks
- added possibility to use dedicated stacks for irq processing
- initial support for page sizes > 4k
- more information in /proc/interrupts (e.g. TLB flushes and number
of IPI calls)
- documented how the parisc gateway page works
- and of course quite some other smaller cleanups and fixes."
* 'parisc-for-3.10' of git://git.kernel.org/pub/scm/linux/kernel/git/deller/parisc-linux:
parisc: tlb flush counting fix for SMP and UP
parisc: more irq statistics in /proc/interrupts
parisc: implement irq stacks
parisc: add kernel stack overflow check
parisc: only re-enable interrupts if we need to schedule or deliver signals when returning to userspace
parisc: implement atomic64_dec_if_positive()
parisc: use long branch in fork_like macro
parisc: fix NATIVE set up in build
parisc: document the parisc gateway page
parisc: fix partly 16/64k PAGE_SIZE boot
parisc: Provide default implementation for dma_{alloc, free}_attrs
parisc: fix whitespace errors in arch/parisc/kernel/traps.c
parisc: remove the second argument of kmap_atomic
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/sstabellini/xen
Pull ARM Xen SMP updates from Stefano Stabellini:
"This contains a bunch of Xen/ARM specific changes, including some
fixes, SMP support for Xen on ARM, and moving the xenvm machine from
mach-vexpress to mach-virt.
The non-Xen files that are touched are arch/arm/Kconfig, to select
ARM_PSCI on XEN, and arch/arm/boot/dts/Makefile, to build the xenvm
DTB if CONFIG_ARCH_VIRT.
Highlights:
- Move xenvm to mach-virt.
- Implement SMP support in Xen on ARM.
- Add support for machine reboot and power off via Xen hypercalls"
* tag '3.9-rc3-smp-6-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/sstabellini/xen:
xen/arm: remove duplicated include from enlighten.c
xen/arm: use sched_op hypercalls for machine reboot and power off
xenvm: add a simple PSCI node and a second cpu
xen/arm: XEN selects ARM_PSCI
xen: move the xenvm machine to mach-virt
xen/arm: SMP support
xen/arm: implement HYPERVISOR_vcpu_op
xen/arm: actually pass a non-NULL percpu pointer to request_percpu_irq
|
|
Fix up build error on UP and show correctly number of function call
(ipi) irqs.
Signed-off-by: Helge Deller <[email protected]>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/ohad/remoteproc
Pull remoteproc update from Ohad Ben-Cohen:
- Some refactoring, cleanups and small improvements from Sjur
Brændeland. The improvements are mainly about better supporting
varios virtio properties (such as virtio's config space, status and
features). I now see that I messed up while commiting one of Sjur's
patches and erroneously put myself as the author, as well as letting
a nasty typo sneak in. I will not fix this in order to avoid
rebasing the patches. Sjur - sorry!
- A new remoteproc driver for OMAP-L13x (technically a DaVinci
platform) from Robert Tivy.
- Extend OMAP support to OMAP5 as well, from Vincent Stehlé.
- Fix Kconfig VIRTUALIZATION dependency, from Suman Anna (a
non-critical fix which arrived late during the rc cycle).
* tag 'remoteproc-3.10' of git://git.kernel.org/pub/scm/linux/kernel/git/ohad/remoteproc:
remoteproc: fix kconfig dependencies for VIRTIO
remoteproc/davinci: add a remoteproc driver for OMAP-L13x DSP
remoteproc: support default firmware name in rproc_alloc()
remoteproc/omap: support OMAP5 too
remoteproc: set vring addresses in resource table
remoteproc: support virtio config space.
remoteproc: perserve resource table data
remoteproc: calculate max_notifyid by counting vrings
remoteproc: code cleanup of resource parsing
remoteproc: parse STE-firmware and find resource table address
remoteproc: add find_loaded_rsc_table firmware ops
remoteproc: refactor rproc_elf_find_rsc_table()
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/ohad/rpmsg
Pull rpmsg changes from Ohad Ben-Cohen:
"A small pull request consisting of:
- Make rpmsg process all pending messages instead of just one, from
Robert Tivy
- Fix Kconfig dependency on VIRTUALIZATION, from Suman.
Note: this was submitted late during the 3.9 rc cycle and it seemed
appropriate to wait with it for the merge window.
- Belated addition of an rpmsg entry to the MAINTAINERS file. People
seem to look for this"
* tag 'rpmsg-3.10' of git://git.kernel.org/pub/scm/linux/kernel/git/ohad/rpmsg:
rpmsg: fix kconfig dependencies for VIRTIO
MAINTAINERS: add rpmsg entry
rpmsg: process _all_ pending messages in rpmsg_recv_done
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/ohad/hwspinlock
Pullhwspinlock update from Ohad Ben-Cohen:
"A single patch from Vincent extending OMAP's hwspinlock support to
OMAP5"
* tag 'hwspinlock-3.10' of git://git.kernel.org/pub/scm/linux/kernel/git/ohad/hwspinlock:
hwspinlock/omap: support OMAP5 as well
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/tj/libata
Pull libata maintainership change from Tejun Heo.
Tejun is taking over from Jeff, after many many years. Thanks Jeff.
* 'for-3.10' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/libata:
libata: change maintainer
|
|
Add framework and initial values for more fine grained statistics in
/proc/interrupts.
Signed-off-by: Helge Deller <[email protected]>
|
|
Commit 4f535093cf "PCI: Put pci_dev in device tree as early as possible"
moved final fixups from pci_bus_add_device() to pci_device_add(). But
pci_device_add() happens before resource assignment, so BARs may not be
valid yet.
Typical flow for hot-add:
pciehp_configure_device
pci_scan_slot
pci_scan_single_device
pci_device_add
pci_fixup_device(pci_fixup_final, dev) # previous location
# resource assignment happens here
pci_bus_add_devices
pci_bus_add_device
pci_fixup_device(pci_fixup_final, dev) # new location
[bhelgaas: changelog, move fixups to pci_bus_add_device()]
Reference: https://lkml.kernel.org/r/20130415182614.GB9224@xanatos
Reported-by: David Bulkow <[email protected]>
Tested-by: David Bulkow <[email protected]>
Signed-off-by: Yinghai Lu <[email protected]>
Signed-off-by: Bjorn Helgaas <[email protected]>
CC: [email protected] # v3.9+
|
|
Default kernel stack size on parisc is 16k. During tests we found that the
kernel stack can easily grow beyond 13k, which leaves 3k left for irq
processing.
This patch adds the possibility to activate an additional stack of 16k per CPU
which is being used during irq processing. This implementation does not yet
uses this irq stack for the irq bh handler.
The assembler code for call_on_stack was heavily cleaned up by John
David Anglin.
CC: John David Anglin <[email protected]>
Signed-off-by: Helge Deller <[email protected]>
|
|
Add the CONFIG_DEBUG_STACKOVERFLOW config option to enable checks to
detect kernel stack overflows.
Stack overflows can not be detected reliable since we do not want to
introduce too much overhead.
Instead, during irq processing in do_cpu_irq_mask() we check kernel
stack usage of the interrupted kernel process. Kernel threads can be
easily detected by checking the value of space register 7 (sr7) which
is zero when running inside the kernel.
Since THREAD_SIZE is 16k and PAGE_SIZE is 4k, reduce the alignment of
the init thread to the lower value (PAGE_SIZE) in the kernel
vmlinux.ld.S linker script.
Signed-off-by: Helge Deller <[email protected]>
|
|
arch/arm/mach-msm/last_radio_log.c: In function 'msm_init_last_radio_log':
arch/arm/mach-msm/last_radio_log.c:69:7: error: dereferencing pointer to incomplete type
arch/cris/kernel/profile.c: In function 'init_cris_profile':
arch/cris/kernel/profile.c:79:8: error: dereferencing pointer to incomplete type
Use proc_set_size(), cfr. commit 271a15eabe094538d958dc68ccfc9c36b699247a
("proc: Supply PDE attribute setting accessor functions")
Signed-off-by: Geert Uytterhoeven <[email protected]>
Signed-off-by: Al Viro <[email protected]>
|
|
when returning to userspace
Helge and I have found that we have a kernel stack overflow problem
which causes a variety of random failures.
Currently, we re-enable interrupts when returning from an external
interrupt incase we need to schedule or delivery
signals. As a result, a potentially unlimited number of interrupts
can occur while we are running on the kernel
stack. It is very limited in space (currently, 16k). This change
defers enabling interrupts until we have
actually decided to schedule or delivery signals. This only occurs
when we about to return to userspace. This
limits the number of interrupts on the kernel stack to one. In other
cases, interrupts remain disabled until the
final return from interrupt (rfi).
Signed-off-by: John David Anglin <[email protected]>
Signed-off-by: Helge Deller <[email protected]>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
Pull late ARM Exynos multiplatform changes from Arnd Bergmann:
"These continue the multiplatform support for exynos, adding support
for building most of the essential drivers (clocksource, clk, irqchip)
when combined with other platforms. As a result, it should become
really easy to add full multiplatform exynos support in 3.11, although
we don't yet enable it for 3.10.
The changes were not included in the earlier multiplatform series in
order to avoid clashes with the other Exynos updates.
This also includes work from Tomasz Figa to fix the pwm clocksource
code on Exynos, which is not strictly required for multiplatform, but
related to the other patches in this set and needed as a bug fix for
at least one board."
* tag 'multiplatform-for-linus-2' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc: (22 commits)
ARM: dts: exynops4210: really add universal_c210 dts
ARM: dts: exynos4210: Add basic dts file for universal_c210 board
ARM: dts: exynos4: Add node for PWM device
ARM: SAMSUNG: Do not register legacy timer interrupts on Exynos
clocksource: samsung_pwm_timer: Work around rounding errors in clockevents core
clocksource: samsung_pwm_timer: Correct programming of clock events
clocksource: samsung_pwm_timer: Use proper clockevents max_delta
clocksource: samsung_pwm_timer: Add support for non-DT platforms
clocksource: samsung_pwm_timer: Drop unused samsung_pwm struct
clocksource: samsung_pwm_timer: Keep all driver data in a structure
clocksource: samsung_pwm_timer: Make PWM spinlock global
clocksource: samsung_pwm_timer: Let platforms select the driver
Documentation: Add device tree bindings for Samsung PWM timers
clocksource: add samsung pwm timer driver
irqchip: exynos: look up irq using irq_find_mapping
irqchip: exynos: pass irq_base from platform
irqchip: exynos: localize irq lookup for ATAGS
irqchip: exynos: allocate combiner_data dynamically
irqchip: exynos: pass max combiner number to combiner_init
ARM: exynos: add missing properties for combiner IRQs
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
Pull ARM SoC late cleanups from Arnd Bergmann:
"These are cleanups and smaller changes that either depend on earlier
feature branches or came in late during the development cycle. We
normally try to get all cleanups early, so these are the exceptions:
- A follow-up on the clocksource reworks, hopefully the last time we
need to merge clocksource subsystem changes through arm-soc.
A first set of patches was part of the original 3.10 arm-soc
cleanup series because of interdependencies with timer drivers now
moved out of arch/arm.
- Migrating the SPEAr13xx platform away from using auxdata for DMA
channel descriptions towards using information in device tree,
based on the earlier SPEAr multiplatform series
- A few follow-ups on the Atmel SAMA5 support and other changes for
Atmel at91 based on the larger at91 reworks.
- Moving the armada irqchip implementation to drivers/irqchip
- Several OMAP cleanups following up on the larger series already
merged in 3.10."
* tag 'cleanup-for-linus-2' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc: (50 commits)
ARM: OMAP4: change the device names in usb_bind_phy
ARM: OMAP2+: Fix mismerge for timer.c between ff931c82 and da4a686a
ARM: SPEAr: conditionalize SMP code
ARM: arch_timer: Silence debug preempt warnings
ARM: OMAP: remove unused variable
serial: amba-pl011: fix !CONFIG_DMA_ENGINE case
ata: arasan: remove the need for platform_data
ARM: at91/sama5d34ek.dts: remove not needed compatibility string
ARM: at91: dts: add MCI DMA support
ARM: at91: dts: add i2c dma support
ARM: at91: dts: set #dma-cells to the correct value
ARM: at91: suspend both memory controllers on at91sam9263
irqchip: armada-370-xp: slightly cleanup irq controller driver
irqchip: armada-370-xp: move IRQ handler to avoid forward declaration
irqchip: move IRQ driver for Armada 370/XP
ARM: mvebu: move L2 cache initialization in init_early()
devtree: add binding documentation for sp804
ARM: integrator-cp: convert use CLKSRC_OF for timer init
ARM: versatile: use OF init for sp804 timer
ARM: versatile: add versatile dtbs to dtbs target
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
Pull ARM SoC device tree updates (part 2) from Arnd Bergmann:
"These are mostly new device tree bindings for existing drivers, as
well as changes to the device tree source files to add support for
those devices, and a couple of new boards, most notably Samsung's
Exynos5 based Chromebook.
The changes depend on earlier platform specific updates and touch the
usual platforms: omap, exynos, tegra, mxs, mvebu and davinci."
* tag 'dt-for-linus-2' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc: (169 commits)
ARM: exynos: dts: cros5250: add EC device
ARM: dts: Add sbs-battery for exynos5250-snow
ARM: dts: Add i2c-arbitrator bus for exynos5250-snow
ARM: dts: add mshc controller node for Exynos4x12 SoCs
ARM: dts: Add chip-id controller node on Exynos4/5 SoC
ARM: EXYNOS: Create virtual I/O mapping for Chip-ID controller using device tree
ARM: davinci: da850-evm: add SPI flash support
ARM: davinci: da850: override SPI DT node device name
ARM: davinci: da850: add SPI1 DT node
spi/davinci: add DT binding documentation
spi/davinci: no wildcards in DT compatible property
ARM: dts: mvebu: Convert mvebu device tree files to 64 bits
ARM: dts: mvebu: introduce internal-regs node
ARM: dts: mvebu: Convert all the mvebu files to use the range property
ARM: dts: mvebu: move all peripherals inside soc
ARM: dts: mvebu: fix cpus section indentation
ARM: davinci: da850: add EHRPWM & ECAP DT node
ARM/dts: OMAP3: fix pinctrl-single configuration
ARM: dts: Add OMAP3430 SDP NOR flash memory binding
ARM: dts: Add NOR flash bindings for OMAP2420 H4
...
|