aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorFilesLines
2021-03-05io-wq: kill hashed waitqueue before manager exitsJens Axboe1-4/+5
If we race with shutting down the io-wq context and someone queueing a hashed entry, then we can exit the manager with it armed. If it then triggers after the manager has exited, we can have a use-after-free where io_wqe_hash_wake() attempts to wake a now gone manager process. Move the killing of the hashed write queue into the manager itself, so that we know we've killed it before the task exits. Fixes: e941894eae31 ("io-wq: make buffered file write hashed work map per-ctx") Signed-off-by: Jens Axboe <[email protected]>
2021-03-05io_uring: clear IOCB_WAITQ for non -EIOCBQUEUED returnJens Axboe1-0/+1
The callback can only be armed, if we get -EIOCBQUEUED returned. It's important that we clear the WAITQ bit for other cases, otherwise we can queue for async retry and filemap will assume that we're armed and return -EAGAIN instead of just blocking for the IO. Cc: [email protected] # 5.9+ Signed-off-by: Jens Axboe <[email protected]>
2021-03-05io_uring: don't keep looping for more events if we can't flush overflowJens Axboe1-3/+12
It doesn't make sense to wait for more events to come in, if we can't even flush the overflow we already have to the ring. Return -EBUSY for that condition, just like we do for attempts to submit with overflow pending. Cc: [email protected] # 5.11 Signed-off-by: Jens Axboe <[email protected]>
2021-03-05io_uring: move to using create_io_thread()Jens Axboe3-109/+54
This allows us to do task creation and setup without needing to use completions to try and synchronize with the starting thread. Get rid of the old io_wq_fork_thread() wrapper, and the 'wq' and 'worker' startup completion events - we can now do setup before the task is running. Signed-off-by: Jens Axboe <[email protected]>
2021-03-04kernel: provide create_io_thread() helperJens Axboe2-0/+32
Provide a generic helper for setting up an io_uring worker. Returns a task_struct so that the caller can do whatever setup is needed, then call wake_up_new_task() to kick it into gear. Add a kernel_clone_args member, io_thread, which tells copy_process() to mark the task with PF_IO_WORKER. Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: reliably cancel linked timeoutsPavel Begunkov1-0/+1
Linked timeouts are fired asynchronously (i.e. soft-irq), and use generic cancellation paths to do its stuff, including poking into io-wq. The problem is that it's racy to access tctx->io_wq, as io_uring_task_cancel() and others may be happening at this exact moment. Mark linked timeouts with REQ_F_INLIFGHT for now, making sure there are no timeouts before io-wq destraction. Signed-off-by: Pavel Begunkov <[email protected]> Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: cancel-match based on flagsPavel Begunkov1-2/+2
Instead of going into request internals, like checking req->file->f_op, do match them based on REQ_F_INFLIGHT, it's set only when we want it to be reliably cancelled. Signed-off-by: Pavel Begunkov <[email protected]> Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io-wq: ensure all pending work is canceled on exitJens Axboe1-9/+33
If we race on shutting down the io-wq, then we should ensure that any work that was queued after workers shutdown is canceled. Harden the add work check a bit too, checking for IO_WQ_BIT_EXIT and cancel if it's set. Add a WARN_ON() for having any work before we kill the io-wq context. Reported-by: [email protected] Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: ensure that threads freeze on suspendJens Axboe2-3/+5
Alex reports that his system fails to suspend using 5.12-rc1, with the following dump: [ 240.650300] PM: suspend entry (deep) [ 240.650748] Filesystems sync: 0.000 seconds [ 240.725605] Freezing user space processes ... [ 260.739483] Freezing of tasks failed after 20.013 seconds (3 tasks refusing to freeze, wq_busy=0): [ 260.739497] task:iou-mgr-446 state:S stack: 0 pid: 516 ppid: 439 flags:0x00004224 [ 260.739504] Call Trace: [ 260.739507] ? sysvec_apic_timer_interrupt+0xb/0x81 [ 260.739515] ? pick_next_task_fair+0x197/0x1cde [ 260.739519] ? sysvec_reschedule_ipi+0x2f/0x6a [ 260.739522] ? asm_sysvec_reschedule_ipi+0x12/0x20 [ 260.739525] ? __schedule+0x57/0x6d6 [ 260.739529] ? del_timer_sync+0xb9/0x115 [ 260.739533] ? schedule+0x63/0xd5 [ 260.739536] ? schedule_timeout+0x219/0x356 [ 260.739540] ? __next_timer_interrupt+0xf1/0xf1 [ 260.739544] ? io_wq_manager+0x73/0xb1 [ 260.739549] ? io_wq_create+0x262/0x262 [ 260.739553] ? ret_from_fork+0x22/0x30 [ 260.739557] task:iou-mgr-517 state:S stack: 0 pid: 522 ppid: 439 flags:0x00004224 [ 260.739561] Call Trace: [ 260.739563] ? sysvec_apic_timer_interrupt+0xb/0x81 [ 260.739566] ? pick_next_task_fair+0x16f/0x1cde [ 260.739569] ? sysvec_apic_timer_interrupt+0xb/0x81 [ 260.739571] ? asm_sysvec_apic_timer_interrupt+0x12/0x20 [ 260.739574] ? __schedule+0x5b7/0x6d6 [ 260.739578] ? del_timer_sync+0x70/0x115 [ 260.739581] ? schedule_timeout+0x211/0x356 [ 260.739585] ? __next_timer_interrupt+0xf1/0xf1 [ 260.739588] ? io_wq_check_workers+0x15/0x11f [ 260.739592] ? io_wq_manager+0x69/0xb1 [ 260.739596] ? io_wq_create+0x262/0x262 [ 260.739600] ? ret_from_fork+0x22/0x30 [ 260.739603] task:iou-wrk-517 state:S stack: 0 pid: 523 ppid: 439 flags:0x00004224 [ 260.739607] Call Trace: [ 260.739609] ? __schedule+0x5b7/0x6d6 [ 260.739614] ? schedule+0x63/0xd5 [ 260.739617] ? schedule_timeout+0x219/0x356 [ 260.739621] ? __next_timer_interrupt+0xf1/0xf1 [ 260.739624] ? task_thread.isra.0+0x148/0x3af [ 260.739628] ? task_thread_unbound+0xa/0xa [ 260.739632] ? task_thread_bound+0x7/0x7 [ 260.739636] ? ret_from_fork+0x22/0x30 [ 260.739647] OOM killer enabled. [ 260.739648] Restarting tasks ... done. [ 260.740077] PM: suspend exit Play nice and ensure that any thread we create will call try_to_freeze() at an opportune time so that memory suspend can proceed. For the io-wq worker threads, mark them as PF_NOFREEZE. They could potentially be blocked for a long time. Reported-by: Alex Xu (Hello71) <[email protected]> Tested-by: Alex Xu (Hello71) <[email protected]> Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: remove extra in_idle wake upPavel Begunkov1-3/+1
io_dismantle_req() is always followed by io_put_task(), which already do proper in_idle wake ups, so we can skip waking the owner task in io_dismantle_req(). The rules are simpler now, do io_put_task() shortly after ending a request, and it will be fine. Signed-off-by: Pavel Begunkov <[email protected]> Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: inline __io_queue_async_work()Pavel Begunkov1-11/+2
__io_queue_async_work() is only called from io_queue_async_work(), inline it. Signed-off-by: Pavel Begunkov <[email protected]> Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: inline io_req_clean_work()Pavel Begunkov1-17/+13
Inline io_req_clean_work(), less code and easier to analyse tctx dependencies and refs usage. Signed-off-by: Pavel Begunkov <[email protected]> Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: choose right tctx->io_wq for try cancelPavel Begunkov1-1/+2
When we cancel SQPOLL, @task in io_uring_try_cancel_requests() will differ from current. Use the right tctx from passed in @task, and don't forget that it can be NULL when the io_uring ctx exits. Signed-off-by: Pavel Begunkov <[email protected]> Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: fix -EAGAIN retry with IOPOLLJens Axboe1-5/+31
We no longer revert the iovec on -EIOCBQUEUED, see commit ab2125df921d, and this started causing issues for IOPOLL on devies that run out of request slots. Turns out what outside of needing a revert for those, we also had a bug where we didn't properly setup retry inside the submission path. That could cause re-import of the iovec, if any, and that could lead to spurious results if the application had those allocated on the stack. Catch -EAGAIN retry and make the iovec stable for IOPOLL, just like we do for !IOPOLL retries. Cc: <[email protected]> # 5.9+ Reported-by: Abaci Robot <[email protected]> Reported-by: Xiaoguang Wang <[email protected]> Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io-wq: fix error path leak of buffered write hash mapJens Axboe1-1/+1
The 'err' path should include the hash put, we already grabbed a reference once we get that far. Fixes: e941894eae31 ("io-wq: make buffered file write hashed work map per-ctx") Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: remove sqo_taskPavel Begunkov1-10/+0
Now, sqo_task is used only for a warning that is not interesting anymore since sqo_dead is gone, remove all of that including ctx->sqo_task. Signed-off-by: Pavel Begunkov <[email protected]> Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: kill sqo_dead and sqo submission haltingPavel Begunkov1-37/+8
As SQPOLL task doesn't poke into ->sqo_task anymore, there is no need to kill the sqo when the master task exits. Before it was necessary to avoid races accessing sqo_task->files with removing them. Signed-off-by: Pavel Begunkov <[email protected]> [axboe: don't forget to enable SQPOLL before exit, if started disabled] Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: ignore double poll add on the same waitqueue headJens Axboe1-0/+3
syzbot reports a deadlock, attempting to lock the same spinlock twice: ============================================ WARNING: possible recursive locking detected 5.11.0-syzkaller #0 Not tainted -------------------------------------------- swapper/1/0 is trying to acquire lock: ffff88801b2b1130 (&runtime->sleep){..-.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline] ffff88801b2b1130 (&runtime->sleep){..-.}-{2:2}, at: io_poll_double_wake+0x25f/0x6a0 fs/io_uring.c:4960 but task is already holding lock: ffff88801b2b3130 (&runtime->sleep){..-.}-{2:2}, at: __wake_up_common_lock+0xb4/0x130 kernel/sched/wait.c:137 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(&runtime->sleep); lock(&runtime->sleep); *** DEADLOCK *** May be due to missing lock nesting notation 2 locks held by swapper/1/0: #0: ffff888147474908 (&group->lock){..-.}-{2:2}, at: _snd_pcm_stream_lock_irqsave+0x9f/0xd0 sound/core/pcm_native.c:170 #1: ffff88801b2b3130 (&runtime->sleep){..-.}-{2:2}, at: __wake_up_common_lock+0xb4/0x130 kernel/sched/wait.c:137 stack backtrace: CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.11.0-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: <IRQ> __dump_stack lib/dump_stack.c:79 [inline] dump_stack+0xfa/0x151 lib/dump_stack.c:120 print_deadlock_bug kernel/locking/lockdep.c:2829 [inline] check_deadlock kernel/locking/lockdep.c:2872 [inline] validate_chain kernel/locking/lockdep.c:3661 [inline] __lock_acquire.cold+0x14c/0x3b4 kernel/locking/lockdep.c:4900 lock_acquire kernel/locking/lockdep.c:5510 [inline] lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151 spin_lock include/linux/spinlock.h:354 [inline] io_poll_double_wake+0x25f/0x6a0 fs/io_uring.c:4960 __wake_up_common+0x147/0x650 kernel/sched/wait.c:108 __wake_up_common_lock+0xd0/0x130 kernel/sched/wait.c:138 snd_pcm_update_state+0x46a/0x540 sound/core/pcm_lib.c:203 snd_pcm_update_hw_ptr0+0xa75/0x1a50 sound/core/pcm_lib.c:464 snd_pcm_period_elapsed+0x160/0x250 sound/core/pcm_lib.c:1805 dummy_hrtimer_callback+0x94/0x1b0 sound/drivers/dummy.c:378 __run_hrtimer kernel/time/hrtimer.c:1519 [inline] __hrtimer_run_queues+0x609/0xe40 kernel/time/hrtimer.c:1583 hrtimer_run_softirq+0x17b/0x360 kernel/time/hrtimer.c:1600 __do_softirq+0x29b/0x9f6 kernel/softirq.c:345 invoke_softirq kernel/softirq.c:221 [inline] __irq_exit_rcu kernel/softirq.c:422 [inline] irq_exit_rcu+0x134/0x200 kernel/softirq.c:434 sysvec_apic_timer_interrupt+0x93/0xc0 arch/x86/kernel/apic/apic.c:1100 </IRQ> asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:632 RIP: 0010:native_save_fl arch/x86/include/asm/irqflags.h:29 [inline] RIP: 0010:arch_local_save_flags arch/x86/include/asm/irqflags.h:70 [inline] RIP: 0010:arch_irqs_disabled arch/x86/include/asm/irqflags.h:137 [inline] RIP: 0010:acpi_safe_halt drivers/acpi/processor_idle.c:111 [inline] RIP: 0010:acpi_idle_do_entry+0x1c9/0x250 drivers/acpi/processor_idle.c:516 Code: dd 38 6e f8 84 db 75 ac e8 54 32 6e f8 e8 0f 1c 74 f8 e9 0c 00 00 00 e8 45 32 6e f8 0f 00 2d 4e 4a c5 00 e8 39 32 6e f8 fb f4 <9c> 5b 81 e3 00 02 00 00 fa 31 ff 48 89 de e8 14 3a 6e f8 48 85 db RSP: 0018:ffffc90000d47d18 EFLAGS: 00000293 RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 RDX: ffff8880115c3780 RSI: ffffffff89052537 RDI: 0000000000000000 RBP: ffff888141127064 R08: 0000000000000001 R09: 0000000000000001 R10: ffffffff81794168 R11: 0000000000000000 R12: 0000000000000001 R13: ffff888141127000 R14: ffff888141127064 R15: ffff888143331804 acpi_idle_enter+0x361/0x500 drivers/acpi/processor_idle.c:647 cpuidle_enter_state+0x1b1/0xc80 drivers/cpuidle/cpuidle.c:237 cpuidle_enter+0x4a/0xa0 drivers/cpuidle/cpuidle.c:351 call_cpuidle kernel/sched/idle.c:158 [inline] cpuidle_idle_call kernel/sched/idle.c:239 [inline] do_idle+0x3e1/0x590 kernel/sched/idle.c:300 cpu_startup_entry+0x14/0x20 kernel/sched/idle.c:397 start_secondary+0x274/0x350 arch/x86/kernel/smpboot.c:272 secondary_startup_64_no_verify+0xb0/0xbb which is due to the driver doing poll_wait() twice on the same wait_queue_head. That is perfectly valid, but from checking the rest of the kernel tree, it's the only driver that does this. We can handle this just fine, we just need to ignore the second addition as we'll get woken just fine on the first one. Cc: [email protected] # 5.8+ Fixes: 18bceab101ad ("io_uring: allow POLL_ADD with double poll_wait() users") Reported-by: [email protected] Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: ensure that SQPOLL thread is started for exitJens Axboe1-3/+3
If we create it in a disabled state because IORING_SETUP_R_DISABLED is set on ring creation, we need to ensure that we've kicked the thread if we're exiting before it's been explicitly disabled. Otherwise we can run into a deadlock where exit is waiting go park the SQPOLL thread, but the SQPOLL thread itself is waiting to get a signal to start. That results in the below trace of both tasks hung, waiting on each other: INFO: task syz-executor458:8401 blocked for more than 143 seconds. Not tainted 5.11.0-next-20210226-syzkaller #0 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:syz-executor458 state:D stack:27536 pid: 8401 ppid: 8400 flags:0x00004004 Call Trace: context_switch kernel/sched/core.c:4324 [inline] __schedule+0x90c/0x21a0 kernel/sched/core.c:5075 schedule+0xcf/0x270 kernel/sched/core.c:5154 schedule_timeout+0x1db/0x250 kernel/time/timer.c:1868 do_wait_for_common kernel/sched/completion.c:85 [inline] __wait_for_common kernel/sched/completion.c:106 [inline] wait_for_common kernel/sched/completion.c:117 [inline] wait_for_completion+0x168/0x270 kernel/sched/completion.c:138 io_sq_thread_park fs/io_uring.c:7115 [inline] io_sq_thread_park+0xd5/0x130 fs/io_uring.c:7103 io_uring_cancel_task_requests+0x24c/0xd90 fs/io_uring.c:8745 __io_uring_files_cancel+0x110/0x230 fs/io_uring.c:8840 io_uring_files_cancel include/linux/io_uring.h:47 [inline] do_exit+0x299/0x2a60 kernel/exit.c:780 do_group_exit+0x125/0x310 kernel/exit.c:922 __do_sys_exit_group kernel/exit.c:933 [inline] __se_sys_exit_group kernel/exit.c:931 [inline] __x64_sys_exit_group+0x3a/0x50 kernel/exit.c:931 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x43e899 RSP: 002b:00007ffe89376d48 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 RAX: ffffffffffffffda RBX: 00000000004af2f0 RCX: 000000000043e899 RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000010000000 R10: 0000000000008011 R11: 0000000000000246 R12: 00000000004af2f0 R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001 INFO: task iou-sqp-8401:8402 can't die for more than 143 seconds. task:iou-sqp-8401 state:D stack:30272 pid: 8402 ppid: 8400 flags:0x00004004 Call Trace: context_switch kernel/sched/core.c:4324 [inline] __schedule+0x90c/0x21a0 kernel/sched/core.c:5075 schedule+0xcf/0x270 kernel/sched/core.c:5154 schedule_timeout+0x1db/0x250 kernel/time/timer.c:1868 do_wait_for_common kernel/sched/completion.c:85 [inline] __wait_for_common kernel/sched/completion.c:106 [inline] wait_for_common kernel/sched/completion.c:117 [inline] wait_for_completion+0x168/0x270 kernel/sched/completion.c:138 io_sq_thread+0x27d/0x1ae0 fs/io_uring.c:6717 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294 INFO: task iou-sqp-8401:8402 blocked for more than 143 seconds. Reported-by: [email protected] Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: replace cmpxchg in fallback with xchgPavel Begunkov1-6/+2
io_run_ctx_fallback() can use xchg() instead of cmpxchg(). It's simpler and faster. Signed-off-by: Pavel Begunkov <[email protected]> Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: fix __tctx_task_work() ctx racePavel Begunkov1-17/+19
There is an unlikely but possible race using a freed context. That's because req->task_work.func() can free a request, but we won't necessarily find a completion in submit_state.comp and so all ctx refs may be put by the time we do mutex_lock(&ctx->uring_ctx); There are several reasons why it can miss going through submit_state.comp: 1) req->task_work.func() didn't complete it itself, but punted to iowq (e.g. reissue) and it got freed later, or a similar situation with it overflowing and getting flushed by someone else, or being submitted to IRQ completion, 2) As we don't hold the uring_lock, someone else can do io_submit_flush_completions() and put our ref. 3) Bugs and code obscurities, e.g. failing to propagate issue_flags properly. One example is as follows CPU1 | CPU2 ======================================================================= @req->task_work.func() | -> @req overflwed, | so submit_state.comp,nr==0 | | flush overflows, and free @req | ctx refs == 0, free it ctx is dead, but we do | lock + flush + unlock | So take a ctx reference for each new ctx we see in __tctx_task_work(), and do release it until we do all our flushing. Fixes: 65453d1efbd2 ("io_uring: enable req cache for task_work items") Reported-by: [email protected] Signed-off-by: Pavel Begunkov <[email protected]> [axboe: fold in my one-liner and fix ref mismatch] Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: kill io_uring_flush()Jens Axboe1-47/+0
This was always a weird work-around or file referencing, and we don't need it anymore. Get rid of it. Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: kill unnecessary io_run_ctx_fallback() in io_ring_exit_work()Jens Axboe1-1/+0
We already run the fallback task_work in io_uring_try_cancel_requests(), no need to duplicate at ring exit explicitly. Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: move cred assignment into io_issue_sqe()Jens Axboe3-81/+22
If we move it in there, then we no longer have to care about it in io-wq. This means we can drop the cred handling in io-wq, and we can drop the REQ_F_WORK_INITIALIZED flag and async init functions as that was the last user of it since we moved to the new workers. Then we can also drop io_wq_work->creds, and just hold the personality u16 in there instead. Suggested-by: Pavel Begunkov <[email protected]> Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: kill unnecessary REQ_F_WORK_INITIALIZED checksJens Axboe1-11/+1
We're no longer checking anything that requires the work item to be initialized, as we're not carrying any file related state there. Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: remove unused argument 'tsk' from io_req_caches_free()Jens Axboe1-3/+3
We prune the full cache regardless, get rid of the dead argument. Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: destroy io-wq on execPavel Begunkov2-10/+11
Destroy current's io-wq backend and tctx on __io_uring_task_cancel(), aka exec(). Looks it's not strictly necessary, because it will be done at some point when the task dies and changes of creds/files/etc. are handled, but better to do that earlier to free io-wq and not potentially lock previous mm and other resources for the time being. It's safe to do because we wait for all requests of the current task to complete, so no request will use tctx afterwards. Note, that io_uring_files_cancel() may leave some requests for later reaping, so it leaves tctx intact, that's ok as the task is dying anyway. Signed-off-by: Pavel Begunkov <[email protected]> Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: warn on not destroyed io-wqPavel Begunkov1-0/+2
Make sure that we killed an io-wq by the time a task is dead. Signed-off-by: Pavel Begunkov <[email protected]> Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: fix race condition in task_work add and clearJens Axboe1-2/+2
We clear the bit marking the ctx task_work as active after having run the queued work, but we really should be clearing it before. Otherwise we can hit a tiny race ala: CPU0 CPU1 io_task_work_add() tctx_task_work() run_work add_to_list test_and_set_bit clear_bit already set and CPU0 will return thinking the task_work is queued, while in reality it's already being run. If we hit the condition after __tctx_task_work() found no more work, but before we've cleared the bit, then we'll end up thinking it's queued and will be run. In reality it is queued, but we didn't queue the ctx task_work to ensure that it gets run. Fixes: 7cbf1722d5fc ("io_uring: provide FIFO ordering for task_work") Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io-wq: provide an io_wq_put_and_exit() helperJens Axboe3-11/+21
If we put the io-wq from io_uring, we really want it to exit. Provide a helper that does that for us. Couple that with not having the manager hold a reference to the 'wq' and the normal SQPOLL exit will tear down the io-wq context appropriate. On the io-wq side, our wq context is per task, so only the task itself is manipulating ->manager and hence it's safe to check and clear without any extra locking. We just need to ensure that the manager task stays around, in case it exits. Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: don't use complete_all() on SQPOLL thread exitJens Axboe1-1/+4
We want to reuse this completion, and a single complete should do just fine. Ensure that we park ourselves first if requested, as that is what lead to the initial deadlock in this area. If we've got someone attempting to park us, then we can't proceed without having them finish first. Fixes: 37d1e2e3642e ("io_uring: move SQPOLL thread io-wq forked worker") Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: run fallback on cancellationPavel Begunkov1-1/+6
io_uring_try_cancel_requests() matches not only current's requests, but also of other exiting tasks, so we need to actively cancel them and not just wait, especially since the function can be called on flush during do_exit() -> exit_files(). Even if it's not a problem for now, it's much nicer to know that the function tries to cancel everything it can. Signed-off-by: Pavel Begunkov <[email protected]> Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io_uring: SQPOLL stop error handling fixesJens Axboe1-7/+18
If we fail to fork an SQPOLL worker, we can hit cancel, and hence attempted thread stop, with the thread already being stopped. Ensure we check for that. Also guard thread stop fully by the sqd mutex, just like we do for park. Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io-wq: fix double put of 'wq' in error pathJens Axboe1-2/+0
We are already freeing the wq struct in both spots, so don't put it and get it freed twice. Reported-by: [email protected] Fixes: 4fb6ac326204 ("io-wq: improve manager/worker handling over exec") Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io-wq: wait for manager exit on wq destroyJens Axboe1-1/+6
The manager waits for the workers, hence the manager is always valid if workers are running. Now also have wq destroy wait for the manager on exit, so we now everything is gone. Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io-wq: rename wq->done completion to wq->startedJens Axboe1-4/+4
This is a leftover from a different use cases, it's used to wait for the manager to startup. Rename it as such. Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io-wq: don't ask for a new worker if we're exitingJens Axboe1-0/+2
If we're in the process of shutting down the async context, then don't create new workers if we already have at least the fixed one. Signed-off-by: Jens Axboe <[email protected]>
2021-03-04io-wq: have manager wait for all workers to exitJens Axboe1-8/+22
Instead of having to wait separately on workers and manager, just have the manager wait on the workers. We use an atomic_t for the reference here, as we need to start at 0 and allow increment from that. Since the number of workers is naturally capped by the allowed nr of processes, and that uses an int, there is no risk of overflow. Signed-off-by: Jens Axboe <[email protected]>
2021-03-01io-wq: wait for worker startup when forking a new oneJens Axboe1-0/+4
We need to have our worker count updated before continuing, to avoid cases where we repeatedly think we need a new worker, but a fork is already in progress. Signed-off-by: Jens Axboe <[email protected]>
2021-02-28Linux 5.12-rc1Linus Torvalds1-3/+3
2021-02-28Merge tag 'ide-5.11-2021-02-28' of git://git.kernel.dk/linux-blockLinus Torvalds1-1/+2
Pull ide fix from Jens Axboe: "This is a leftover fix from 5.11, where I forgot to ship it your way" * tag 'ide-5.11-2021-02-28' of git://git.kernel.dk/linux-block: ide/falconide: Fix module unload
2021-02-28Merge tag 'kbuild-fixes-v5.12' of ↵Linus Torvalds6-21/+31
git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild Pull Kbuild fixes from Masahiro Yamada: - Fix UNUSED_KSYMS_WHITELIST for Clang LTO - Make -s builds really silent irrespective of V= option - Fix build error when SUBLEVEL or PATCHLEVEL is empty * tag 'kbuild-fixes-v5.12' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild: kbuild: Fix <linux/version.h> for empty SUBLEVEL or PATCHLEVEL again kbuild: make -s option take precedence over V=1 ia64: remove redundant READELF from arch/ia64/Makefile kbuild: do not include include/config/auto.conf from adjust_autoksyms.sh kbuild: fix UNUSED_KSYMS_WHITELIST for Clang LTO kbuild: lto: add _mcount to list of used symbols
2021-02-28Merge tag 'csky-for-linus-5.12-rc1' of git://github.com/c-sky/csky-linuxLinus Torvalds94-992/+1347
Pull arch/csky updates from Guo Ren: "Features: - add new memory layout 2.5G(user):1.5G(kernel) - add kmemleak support - reconstruct VDSO framework: add VDSO with GENERIC_GETTIMEOFDAY, GENERIC_TIME_VSYSCALL, HAVE_GENERIC_VDSO - add faulthandler_disabled() check - support (fix) swapon - add (fix) _PAGE_ACCESSED for default pgprot - abort uaccess retries upon fatal signal (from arm) Fixes and optimizations: - fix perf probe failure - fix show_regs doesn't contain regs->usp - remove custom asm/atomic.h implementation - fix barrier design - fix futex SMP implementation - fix asm/cmpxchg.h with correct ordering barrier - cleanup asm/spinlock.h - fix PTE global for 2.5:1.5 virtual memory - remove prologue of page fault handler in entry.S - fix TLB maintenance synchronization problem - add show_tlb for CPU_CK860 debug - fix FAULT_FLAG_XXX param for handle_mm_fault - fix update_mmu_cache called with user io mapping - fix do_page_fault parent irq status - fix a size determination in gpr_get() - pgtable.h: Coding convention - kprobe: Fix code in simulate without 'long' - fix pfn_valid error with wrong max_mapnr - use free_initmem_default() in free_initmem() - fix compile error" * tag 'csky-for-linus-5.12-rc1' of git://github.com/c-sky/csky-linux: (30 commits) csky: Fixup compile error csky: use free_initmem_default() in free_initmem() csky: Fixup pfn_valid error with wrong max_mapnr csky: Add VDSO with GENERIC_GETTIMEOFDAY, GENERIC_TIME_VSYSCALL, HAVE_GENERIC_VDSO csky: kprobe: Fixup code in simulate without 'long' csky: Fixup swapon csky: pgtable.h: Coding convention csky: Fixup _PAGE_ACCESSED for default pgprot csky: remove unused including <linux/version.h> csky: Fix a size determination in gpr_get() csky: Reconstruct VDSO framework csky: mm: abort uaccess retries upon fatal signal csky: Sync riscv mm/fault.c for easy maintenance csky: Fixup do_page_fault parent irq status csky: Add faulthandler_disabled() check csky: Fixup update_mmu_cache called with user io mapping csky: Fixup FAULT_FLAG_XXX param for handle_mm_fault csky: Add show_tlb for CPU_CK860 debug csky: Fix TLB maintenance synchronization problem csky: Add kmemleak support ...
2021-02-28Merge tag 'riscv-for-linus-5.12-mw1' of ↵Linus Torvalds4-19/+5
git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux Pull more RISC-V updates from Palmer Dabbelt: "A pair of patches that slipped through the cracks: - enable CPU hotplug in the defconfigs - some cleanups to setup_bootmem There's also a single fix for some randconfig build failures: - make NUMA depend on SMP" * tag 'riscv-for-linus-5.12-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux: riscv: Cleanup setup_bootmem() RISC-V: Enable CPU Hotplug in defconfigs RISC-V: Make NUMA depend on SMP
2021-02-28Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsiLinus Torvalds30-316/+746
Pull more SCSI updates from James Bottomley: "This is a few driver updates (iscsi, mpt3sas) that were still in the staging queue when the merge window opened (all committed on or before 8 Feb) and some small bug fixes which came in during the merge window (all committed on 22 Feb)" * tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi: (30 commits) scsi: hpsa: Correct dev cmds outstanding for retried cmds scsi: sd: Fix Opal support scsi: target: tcmu: Fix memory leak caused by wrong uio usage scsi: target: tcmu: Move some functions without code change scsi: sd: sd_zbc: Don't pass GFP_NOIO to kvcalloc scsi: aic7xxx: Remove unused function pointer typedef ahc_bus_suspend/resume_t scsi: bnx2fc: Fix Kconfig warning & CNIC build errors scsi: ufs: Fix a duplicate dev quirk number scsi: aic79xx: Fix spelling of version scsi: target: core: Prevent underflow for service actions scsi: target: core: Add cmd length set before cmd complete scsi: iscsi: Drop session lock in iscsi_session_chkready() scsi: qla4xxx: Use iscsi_is_session_online() scsi: libiscsi: Reset max/exp cmdsn during recovery scsi: iscsi_tcp: Fix shost can_queue initialization scsi: libiscsi: Add helper to calculate max SCSI cmds per session scsi: libiscsi: Fix iSCSI host workq destruction scsi: libiscsi: Fix iscsi_task use after free() scsi: libiscsi: Drop taskqueuelock scsi: libiscsi: Fix iscsi_prep_scsi_cmd_pdu() error handling ...
2021-02-28Merge tag 'xfs-5.12-merge-6' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linuxLinus Torvalds8-70/+94
Pull more xfs updates from Darrick Wong: "The most notable fix here prevents premature reuse of freed metadata blocks, and adding the ability to detect accidental nested transactions, which are not allowed here. - Restore a disused sysctl control knob that was inadvertently dropped during the merge window to avoid fstests regressions. - Don't speculatively release freed blocks from the busy list until we're actually allocating them, which fixes a rare log recovery regression. - Don't nest transactions when scanning for free space. - Add an idiot^Wmaintainer light to detect nested transactions. ;)" * tag 'xfs-5.12-merge-6' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux: xfs: use current->journal_info for detecting transaction recursion xfs: don't nest transactions when scanning for eofblocks xfs: don't reuse busy extents on extent trim xfs: restore speculative_cow_prealloc_lifetime sysctl
2021-02-28Merge tag 'block-5.12-2021-02-27' of git://git.kernel.dk/linux-blockLinus Torvalds40-180/+173
Pull more block updates from Jens Axboe: "A few stragglers (and one due to me missing it originally), and fixes for changes in this merge window mostly. In particular: - blktrace cleanups (Chaitanya, Greg) - Kill dead blk_pm_* functions (Bart) - Fixes for the bio alloc changes (Christoph) - Fix for the partition changes (Christoph, Ming) - Fix for turning off iopoll with polled IO inflight (Jeffle) - nbd disconnect fix (Josef) - loop fsync error fix (Mauricio) - kyber update depth fix (Yang) - max_sectors alignment fix (Mikulas) - Add bio_max_segs helper (Matthew)" * tag 'block-5.12-2021-02-27' of git://git.kernel.dk/linux-block: (21 commits) block: Add bio_max_segs blktrace: fix documentation for blk_fill_rw() block: memory allocations in bounce_clone_bio must not fail block: remove the gfp_mask argument to bounce_clone_bio block: fix bounce_clone_bio for passthrough bios block-crypto-fallback: use a bio_set for splitting bios block: fix logging on capacity change blk-settings: align max_sectors on "logical_block_size" boundary block: reopen the device in blkdev_reread_part block: don't skip empty device in in disk_uevent blktrace: remove debugfs file dentries from struct blk_trace nbd: handle device refs for DESTROY_ON_DISCONNECT properly kyber: introduce kyber_depth_updated() loop: fix I/O error on fsync() in detached loop devices block: fix potential IO hang when turning off io_poll block: get rid of the trace rq insert wrapper blktrace: fix blk_rq_merge documentation blktrace: fix blk_rq_issue documentation blktrace: add blk_fill_rwbs documentation comment block: remove superfluous param in blk_fill_rwbs() ...
2021-02-28kbuild: Fix <linux/version.h> for empty SUBLEVEL or PATCHLEVEL againMasahiro Yamada1-2/+4
Commit 78d3bb4483ba ("kbuild: Fix <linux/version.h> for empty SUBLEVEL or PATCHLEVEL") fixed the build error for empty SUBLEVEL or PATCHLEVEL by prepending a zero. Commit 9b82f13e7ef3 ("kbuild: clamp SUBLEVEL to 255") re-introduced this issue. This time, we cannot take the same approach because we have C code: #define LINUX_VERSION_PATCHLEVEL $(PATCHLEVEL) #define LINUX_VERSION_SUBLEVEL $(SUBLEVEL) Replace empty SUBLEVEL/PATCHLEVEL with a zero. Fixes: 9b82f13e7ef3 ("kbuild: clamp SUBLEVEL to 255") Reported-by: Christian Zigotzky <[email protected]> Signed-off-by: Masahiro Yamada <[email protected]> Reviewed-and-tested-by: Sasha Levin <[email protected]>
2021-02-28kbuild: make -s option take precedence over V=1Masahiro Yamada1-0/+1
'make -s' should be really silent. However, 'make -s V=1' prints noisy log messages from some shell scripts. Of course, such a combination is odd, but the build system needs to do the right thing even if a user gives strange input. If -s is given, KBUILD_VERBOSE should be forced to 0. Signed-off-by: Masahiro Yamada <[email protected]>
2021-02-28ia64: remove redundant READELF from arch/ia64/MakefileMasahiro Yamada1-1/+0
READELF is defined by the top Makefile. Signed-off-by: Masahiro Yamada <[email protected]>
Powered by cgit, Gruvboxed by Blaster4385.