| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
The ppc_md and pci_controller_ops methods are unused now and can be
removed. The dma_nommu implementation is generic to the generic one
except for using max_pfn instead of calling into the memblock API,
and all other dma_map_ops instances implement a method of their own.
Signed-off-by: Christoph Hellwig <[email protected]>
Tested-by: Christian Zigotzky <[email protected]>
Signed-off-by: Michael Ellerman <[email protected]>
|
|
If there is no ZONE_DMA32 we might need GFP_DMA to be able to
allocate memory that satisfies a 32-bit DMA mask.
Reported-by: Christian Zigotzky <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
Tested-by: Christian Zigotzky <[email protected]>
Signed-off-by: Michael Ellerman <[email protected]>
|
|
There is no need to have the 'struct dentry *d_swiotlb_usage' variable
static since new value always be assigned before use it.
Signed-off-by: YueHaibing <[email protected]>
Signed-off-by: Konrad Rzeszutek Wilk <[email protected]>
|
|
memmap return a regular void pointer, not and __iomem one.
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Catalin Marinas <[email protected]> # arm64
|
|
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Paul Burton <[email protected]> # MIPS
Acked-by: Catalin Marinas <[email protected]> # arm64
|
|
This is a follow up to the commit cf65a0f6f6ff
("dma-mapping: move all DMA mapping code to kernel/dma")
which moved source code of DMA API to kernel/dma folder. Since there is
no file left in the lib that require DMA API debugging options move the
latter to kernel/dma as well.
Cc: Christoph Hellwig <[email protected]>
Signed-off-by: Andy Shevchenko <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
This patch uses io_tlb_used to help check whether swiotlb buffer is full.
io_tlb_used is no longer used for only debugfs. It is also used to help
optimize swiotlb_tbl_map_single().
Suggested-by: Joe Jin <[email protected]>
Signed-off-by: Dongli Zhang <[email protected]>
Signed-off-by: Konrad Rzeszutek Wilk <[email protected]>
|
|
The device driver will not be able to do dma operations once swiotlb buffer
is full, either because the driver is using so many IO TLB blocks inflight,
or because there is memory leak issue in device driver. To export the
swiotlb buffer usage via debugfs would help the user estimate the size of
swiotlb buffer to pre-allocate or analyze device driver memory leak issue.
Signed-off-by: Dongli Zhang <[email protected]>
Signed-off-by: Konrad Rzeszutek Wilk <[email protected]>
|
|
Fix the comment as swiotlb_bounce() is used to copy from original dma
location to swiotlb buffer during swiotlb_tbl_map_single(), while to
copy from swiotlb buffer to original dma location during
swiotlb_tbl_unmap_single().
Signed-off-by: Dongli Zhang <[email protected]>
Signed-off-by: Konrad Rzeszutek Wilk <[email protected]>
|
|
While debugging a DMA mapping leak, I needed to access
debug_dma_dump_mappings() but easily from user space.
This patch adds a /sys/kernel/debug/dma-api/dump file which contain all
current DMA mapping.
Signed-off-by: Corentin Labbe <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
When calling debugfs functions, there is no need to ever check the
return value. The function can work or not, but the code logic should
never do something different based on this.
Also delete the variables for the file dentries for the debugfs entries
as they are never used at all once they are created.
Signed-off-by: Greg Kroah-Hartman <[email protected]>
Reviewed-by: Robin Murphy <[email protected]>
[hch: moved dma_debug_dent to function scope and renamed it]
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
Instead provide a proper implementation in the direct mapping code, and
also wire it up for arm and powerpc, leaving an error return for all the
IOMMU or virtual mapping instances for which we'd have to wire up an
actual implementation
Signed-off-by: Christoph Hellwig <[email protected]>
Tested-by: Marek Szyprowski <[email protected]>
|
|
Otherwise is_swiotlb_buffer will return false positives when
we first initialize a swiotlb buffer, but then free it because
we have an IOMMU available.
Fixes: 55897af63091 ("dma-direct: merge swiotlb_dma_ops into the dma_direct code")
Reported-by: Sibren Vasse <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
Tested-by: Sibren Vasse <[email protected]>
Signed-off-by: Konrad Rzeszutek Wilk <[email protected]>
|
|
We need to return a dma_addr_t even if we don't have a kernel mapping.
Do so by consolidating the phys_to_dma call in a single place and jump
to it from all the branches that return successfully.
Fixes: bfd56cd60521 ("dma-mapping: support highmem in the generic remap allocator")
Reported-by: Liviu Dudau <[email protected]
Signed-off-by: Christoph Hellwig <[email protected]>
Tested-by: Liviu Dudau <[email protected]>
|
|
Now that the slow path DMA API calls are implemented out of line a few
helpers only used by them don't need to be exported anymore.
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
These functions have never been used.
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
dmam_alloc_coherent is just the default no-flags case of
dmam_alloc_attrs, so take advantage of this similar to the non-managed
version.
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
And also switch the way we implement the unmap side around to stay
consistent. This ensures dma-debug works again because it records which
function we used for mapping to ensure it is also used for unmapping,
and also reduces further code duplication. Last but not least this
also officially allows calling dma_sync_single_* for mappings created
using dma_map_page, which is perfectly fine given that the sync calls
only take a dma_addr_t, but not a virtual address or struct page.
Fixes: 7f0fee242e ("dma-mapping: merge dma_unmap_page_attrs and dma_unmap_single_attrs")
Signed-off-by: Christoph Hellwig <[email protected]>
Tested-by: LABBE Corentin <[email protected]>
|
|
Pull DMA mapping updates from Christoph Hellwig:
"A huge update this time, but a lot of that is just consolidating or
removing code:
- provide a common DMA_MAPPING_ERROR definition and avoid indirect
calls for dma_map_* error checking
- use direct calls for the DMA direct mapping case, avoiding huge
retpoline overhead for high performance workloads
- merge the swiotlb dma_map_ops into dma-direct
- provide a generic remapping DMA consistent allocator for
architectures that have devices that perform DMA that is not cache
coherent. Based on the existing arm64 implementation and also used
for csky now.
- improve the dma-debug infrastructure, including dynamic allocation
of entries (Robin Murphy)
- default to providing chaining scatterlist everywhere, with opt-outs
for the few architectures (alpha, parisc, most arm32 variants) that
can't cope with it
- misc sparc32 dma-related cleanups
- remove the dma_mark_clean arch hook used by swiotlb on ia64 and
replace it with the generic noncoherent infrastructure
- fix the return type of dma_set_max_seg_size (Niklas Söderlund)
- move the dummy dma ops for not DMA capable devices from arm64 to
common code (Robin Murphy)
- ensure dma_alloc_coherent returns zeroed memory to avoid kernel
data leaks through userspace. We already did this for most common
architectures, but this ensures we do it everywhere.
dma_zalloc_coherent has been deprecated and can hopefully be
removed after -rc1 with a coccinelle script"
* tag 'dma-mapping-4.21' of git://git.infradead.org/users/hch/dma-mapping: (73 commits)
dma-mapping: fix inverted logic in dma_supported
dma-mapping: deprecate dma_zalloc_coherent
dma-mapping: zero memory returned from dma_alloc_*
sparc/iommu: fix ->map_sg return value
sparc/io-unit: fix ->map_sg return value
arm64: default to the direct mapping in get_arch_dma_ops
PCI: Remove unused attr variable in pci_dma_configure
ia64: only select ARCH_HAS_DMA_COHERENT_TO_PFN if swiotlb is enabled
dma-mapping: bypass indirect calls for dma-direct
vmd: use the proper dma_* APIs instead of direct methods calls
dma-direct: merge swiotlb_dma_ops into the dma_direct code
dma-direct: use dma_direct_map_page to implement dma_direct_map_sg
dma-direct: improve addressability error reporting
swiotlb: remove dma_mark_clean
swiotlb: remove SWIOTLB_MAP_ERROR
ACPI / scan: Refactor _CCA enforcement
dma-mapping: factor out dummy DMA ops
dma-mapping: always build the direct mapping code
dma-mapping: move dma_cache_sync out of line
dma-mapping: move various slow path functions out of line
...
|
|
The cleanup in commit 356da6d0cde3 ("dma-mapping: bypass indirect calls
for dma-direct") accidentally inverted the logic in the check for the
presence of a ->dma_supported() callback. Switch this back to the way it
was to prevent a crash on boot.
Fixes: 356da6d0cde3 ("dma-mapping: bypass indirect calls for dma-direct")
Signed-off-by: Thierry Reding <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
If we want to map memory from the DMA allocator to userspace it must be
zeroed at allocation time to prevent stale data leaks. We already do
this on most common architectures, but some architectures don't do this
yet, fix them up, either by passing GFP_ZERO when we use the normal page
allocator or doing a manual memset otherwise.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Geert Uytterhoeven <[email protected]> [m68k]
Acked-by: Sam Ravnborg <[email protected]> [sparc]
|
|
The dma_direct_supported() function intends to check the DMA mask against
specific values. However, the phys_to_dma() function includes the SME
encryption mask, which defeats the intended purpose of the check. This
results in drivers that support less than 48-bit DMA (SME encryption mask
is bit 47) from being able to set the DMA mask successfully when SME is
active, which results in the driver failing to initialize.
Change the function used to check the mask from phys_to_dma() to
__phys_to_dma() so that the SME encryption mask is not part of the check.
Fixes: c1d0af1a1d5d ("kernel/dma/direct: take DMA offset into account in dma_direct_supported")
Signed-off-by: Tom Lendacky <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
Avoid expensive indirect calls in the fast path DMA mapping
operations by directly calling the dma_direct_* ops if we are using
the directly mapped DMA operations.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
|
|
While the dma-direct code is (relatively) clean and simple we actually
have to use the swiotlb ops for the mapping on many architectures due
to devices with addressing limits. Instead of keeping two
implementations around this commit allows the dma-direct
implementation to call the swiotlb bounce buffering functions and
thus share the guts of the mapping implementation. This also
simplified the dma-mapping setup on a few architectures where we
don't have to differenciate which implementation to use.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
|
|
No need to duplicate the mapping logic.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
|
|
Only report report a DMA addressability report once to avoid spewing the
kernel log with repeated message. Also provide a stack trace to make it
easy to find the actual caller that caused the problem.
Last but not least move the actual check into the fast path and only
leave the error reporting in a helper.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
|
|
Instead of providing a special dma_mark_clean hook just for ia64, switch
ia64 to use the normal arch_sync_dma_for_cpu hooks instead.
This means that we now also set the PG_arch_1 bit for pages in the
swiotlb buffer, which isn't stricly needed as we will never execute code
out of the swiotlb buffer, but otherwise harmless.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
|
|
We can use DMA_MAPPING_ERROR instead, which already maps to the same
value.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
|
|
The dummy DMA ops are currently used by arm64 for any device which has
an invalid ACPI description and is thus barred from using DMA due to not
knowing whether is is cache-coherent or not. Factor these out into
general dma-mapping code so that they can be referenced from other
common code paths. In the process, we can prune all the optional
callbacks which just do the same thing as the default behaviour, and
fill in .map_resource for completeness.
Signed-off-by: Robin Murphy <[email protected]>
[hch: moved to a separate source file]
Reviewed-by: Rafael J. Wysocki <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
All architectures except for sparc64 use the dma-direct code in some
form, and even for sparc64 we had the discussion of a direct mapping
mode a while ago. In preparation for directly calling the direct
mapping code don't bother having it optionally but always build the
code in. This is a minor hardship for some powerpc and arm configs
that don't pull it in yet (although they should in a relase ot two),
and sparc64 which currently doesn't need it at all, but it will
reduce the ifdef mess we'd otherwise need significantly.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
|
|
This isn't exactly a slow path routine, but it is not super critical
either, and moving it out of line will help to keep the include chain
clean for the following DMA indirection bypass work.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
|
|
There is no need to have all setup and coherent allocation / freeing
routines inline. Move them out of line to keep the implemeation
nicely encapsulated and save some kernel text size.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
|
|
dma_get_required_mask should really be with the rest of the DMA mapping
implementation instead of in drivers/base as a lone outlier.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
|
|
We can just call the regular calls after adding offset the the address instead
of reimplementing them.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
|
|
We already zero the memory after allocating it from the pool that
this function fills, and having the memset here in this form means
we can't support CMA highmem allocations.
Signed-off-by: Christoph Hellwig <[email protected]>
Reported-by: Russell King - ARM Linux <[email protected]>
|
|
DMA debug entries are one of those things which aren't that useful
individually - we will always want some larger quantity of them - and
which we don't really need to manage the exact number of - we only care
about having 'enough'. In that regard, the current behaviour of creating
them one-by-one leads to a lot of unwarranted function call overhead and
memory wasted on alignment padding.
Now that we don't have to worry about freeing anything via
dma_debug_resize_entries(), we can optimise the allocation behaviour by
grabbing whole pages at once, which will save considerably on the
aforementioned overheads, and probably offer a little more cache/TLB
locality benefit for traversing the lists under normal operation. This
should also give even less reason for an architecture-level override of
the preallocation size, so make the definition unconditional - if there
is still any desire to change the compile-time value for some platforms
it would be better off as a Kconfig option anyway.
Since freeing a whole page of entries at once becomes enough of a
challenge that it's not really worth complicating dma_debug_init(), we
may as well tweak the preallocation behaviour such that as long as we
manage to allocate *some* pages, we can leave debugging enabled on a
best-effort basis rather than otherwise wasting them.
Signed-off-by: Robin Murphy <[email protected]>
Tested-by: Qian Cai <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
With the only caller now gone, we can clean up this part of dma-debug's
exposed internals and make way to tweak the allocation behaviour.
Signed-off-by: Robin Murphy <[email protected]>
Tested-by: Qian Cai <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
Now that we can dynamically allocate DMA debug entries to cope with
drivers maintaining excessively large numbers of live mappings, a driver
which *does* actually have a bug leaking mappings (and is not unloaded)
will no longer trigger the "DMA-API: debugging out of memory - disabling"
message until it gets to actual kernel OOM conditions, which means it
could go unnoticed for a while. To that end, let's inform the user each
time the pool has grown to a multiple of its initial size, which should
make it apparent that they either have a leak or might want to increase
the preallocation size.
Signed-off-by: Robin Murphy <[email protected]>
Tested-by: Qian Cai <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
Certain drivers such as large multi-queue network adapters can use pools
of mapped DMA buffers larger than the default dma_debug_entry pool of
65536 entries, with the result that merely probing such a device can
cause DMA debug to disable itself during boot unless explicitly given an
appropriate "dma_debug_entries=..." option.
Developers trying to debug some other driver on such a system may not be
immediately aware of this, and at worst it can hide bugs if they fail to
realise that dma-debug has already disabled itself unexpectedly by the
time their code of interest gets to run. Even once they do realise, it
can be a bit of a pain to emprirically determine a suitable number of
preallocated entries to configure, short of massively over-allocating.
There's really no need for such a static limit, though, since we can
quite easily expand the pool at runtime in those rare cases that the
preallocated entries are insufficient, which is arguably the least
surprising and most useful behaviour. To that end, refactor the
prealloc_memory() logic a little bit to generalise it for runtime
reallocations as well.
Signed-off-by: Robin Murphy <[email protected]>
Tested-by: Qian Cai <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
Use pr_fmt() to generate the "DMA-API: " prefix consistently. This
results in it being added to a couple of pr_*() messages which were
missing it before, and for the err_printk() calls moves it to the actual
start of the message instead of somewhere in the middle.
Signed-off-by: Robin Murphy <[email protected]>
Tested-by: Qian Cai <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
Expose nr_total_entries in debugfs, so that {num,min}_free_entries
become even more meaningful to users interested in current/maximum
utilisation. This becomes even more relevant once nr_total_entries
may change at runtime beyond just the existing AMD GART debug code.
Suggested-by: John Garry <[email protected]>
Signed-off-by: Robin Murphy <[email protected]>
Tested-by: Qian Cai <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
The dma-direct code already returns (~(dma_addr_t)0x0) on mapping
failures, so we can switch over to returning DMA_MAPPING_ERROR and let
the core dma-mapping code handle the rest.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Linus Torvalds <[email protected]>
|
|
Commit bfd56cd60521 ("dma-mapping: support highmem in the generic remap
allocator") replaced dma_direct_alloc_pages() with __dma_direct_alloc_pages(),
which doesn't set dma_handle and zero allocated memory. Fix it by doing this
directly in the caller function.
Fixes: bfd56cd60521 ("dma-mapping: support highmem in the generic remap allocator")
Signed-off-by: Marek Szyprowski <[email protected]>
Tested-by: Thierry Reding <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
Do not waste vmalloc space on allocations that do not require a mapping
into the kernel address space.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Robin Murphy <[email protected]>
|
|
By using __dma_direct_alloc_pages we can deal entirely with struct page
instead of having to derive a kernel virtual address.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Robin Murphy <[email protected]>
|
|
The arm64 codebase to implement coherent dma allocation for architectures
with non-coherent DMA is a good start for a generic implementation, given
that is uses the generic remap helpers, provides the atomic pool for
allocations that can't sleep and still is realtively simple and well
tested. Move it to kernel/dma and allow architectures to opt into it
using a config symbol. Architectures just need to provide a new
arch_dma_prep_coherent helper to writeback an invalidate the caches
for any memory that gets remapped for uncached access.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Will Deacon <[email protected]>
Reviewed-by: Robin Murphy <[email protected]>
|
|
The dma remap code only makes sense for not cache coherent architectures
(or possibly the corner case of highmem CMA allocations) and currently
is only used by arm, arm64, csky and xtensa. Split it out into a
separate file with a separate Kconfig symbol, which gets the right
copyright notice given that this code was written by Laura Abbott
working for Code Aurora at that point.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Laura Abbott <[email protected]>
Reviewed-by: Robin Murphy <[email protected]>
|
|
dma_alloc_from_contiguous can return highmem pages depending on the
setup, which a plain non-remapping DMA allocator can't handle. Detect
this case and fail the allocation.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Robin Murphy <[email protected]>
|
|
Some architectures support remapping highmem into DMA coherent
allocations. To use the common code for them we need variants of
dma_direct_{alloc,free}_pages that do not use kernel virtual addresses.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Robin Murphy <[email protected]>
|