| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Use WARN_ON_ONCE to print a stack trace and return a proper error
code instead.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Robin Murphy <[email protected]>
Tested-by: Marek Szyprowski <[email protected]>
|
|
Instead provide a proper implementation in the direct mapping code, and
also wire it up for arm and powerpc, leaving an error return for all the
IOMMU or virtual mapping instances for which we'd have to wire up an
actual implementation
Signed-off-by: Christoph Hellwig <[email protected]>
Tested-by: Marek Szyprowski <[email protected]>
|
|
dma_zalloc_coherent() is no longer needed as it has no users because
dma_alloc_coherent() already zeroes out memory for us.
The Coccinelle grammar rule that used to check for dma_alloc_coherent()
+ memset() is modified so that it just tells the user that the memset is
not needed anymore.
Suggested-by: Christoph Hellwig <[email protected]>
Signed-off-by: Luis Chamberlain <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
This avoids link failures in drivers using the DMA API, when they
are compiled for user mode Linux with CONFIG_COMPILE_TEST=y.
Fixes: 356da6d0cd ("dma-mapping: bypass indirect calls for dma-direct")
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
These functions have never been used.
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
dmam_alloc_coherent is just the default no-flags case of
dmam_alloc_attrs, so take advantage of this similar to the non-managed
version.
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
And also switch the way we implement the unmap side around to stay
consistent. This ensures dma-debug works again because it records which
function we used for mapping to ensure it is also used for unmapping,
and also reduces further code duplication. Last but not least this
also officially allows calling dma_sync_single_* for mappings created
using dma_map_page, which is perfectly fine given that the sync calls
only take a dma_addr_t, but not a virtual address or struct page.
Fixes: 7f0fee242e ("dma-mapping: merge dma_unmap_page_attrs and dma_unmap_single_attrs")
Signed-off-by: Christoph Hellwig <[email protected]>
Tested-by: LABBE Corentin <[email protected]>
|
|
Pull DMA mapping updates from Christoph Hellwig:
"A huge update this time, but a lot of that is just consolidating or
removing code:
- provide a common DMA_MAPPING_ERROR definition and avoid indirect
calls for dma_map_* error checking
- use direct calls for the DMA direct mapping case, avoiding huge
retpoline overhead for high performance workloads
- merge the swiotlb dma_map_ops into dma-direct
- provide a generic remapping DMA consistent allocator for
architectures that have devices that perform DMA that is not cache
coherent. Based on the existing arm64 implementation and also used
for csky now.
- improve the dma-debug infrastructure, including dynamic allocation
of entries (Robin Murphy)
- default to providing chaining scatterlist everywhere, with opt-outs
for the few architectures (alpha, parisc, most arm32 variants) that
can't cope with it
- misc sparc32 dma-related cleanups
- remove the dma_mark_clean arch hook used by swiotlb on ia64 and
replace it with the generic noncoherent infrastructure
- fix the return type of dma_set_max_seg_size (Niklas Söderlund)
- move the dummy dma ops for not DMA capable devices from arm64 to
common code (Robin Murphy)
- ensure dma_alloc_coherent returns zeroed memory to avoid kernel
data leaks through userspace. We already did this for most common
architectures, but this ensures we do it everywhere.
dma_zalloc_coherent has been deprecated and can hopefully be
removed after -rc1 with a coccinelle script"
* tag 'dma-mapping-4.21' of git://git.infradead.org/users/hch/dma-mapping: (73 commits)
dma-mapping: fix inverted logic in dma_supported
dma-mapping: deprecate dma_zalloc_coherent
dma-mapping: zero memory returned from dma_alloc_*
sparc/iommu: fix ->map_sg return value
sparc/io-unit: fix ->map_sg return value
arm64: default to the direct mapping in get_arch_dma_ops
PCI: Remove unused attr variable in pci_dma_configure
ia64: only select ARCH_HAS_DMA_COHERENT_TO_PFN if swiotlb is enabled
dma-mapping: bypass indirect calls for dma-direct
vmd: use the proper dma_* APIs instead of direct methods calls
dma-direct: merge swiotlb_dma_ops into the dma_direct code
dma-direct: use dma_direct_map_page to implement dma_direct_map_sg
dma-direct: improve addressability error reporting
swiotlb: remove dma_mark_clean
swiotlb: remove SWIOTLB_MAP_ERROR
ACPI / scan: Refactor _CCA enforcement
dma-mapping: factor out dummy DMA ops
dma-mapping: always build the direct mapping code
dma-mapping: move dma_cache_sync out of line
dma-mapping: move various slow path functions out of line
...
|
|
We really need the writecombine flag in dma_alloc_wc, fix a stupid
oversight.
Fixes: 7ed1d91a9e ("dma-mapping: translate __GFP_NOFAIL to DMA_ATTR_NO_WARN")
Signed-off-by: Christoph Hellwig <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|
|
We now always return zeroed memory from dma_alloc_coherent. Note that
simply passing GFP_ZERO to dma_alloc_coherent wasn't always doing the
right thing to start with given that various allocators are not backed
by the page allocator and thus would ignore GFP_ZERO.
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
Avoid expensive indirect calls in the fast path DMA mapping
operations by directly calling the dma_direct_* ops if we are using
the directly mapped DMA operations.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
|
|
The dummy DMA ops are currently used by arm64 for any device which has
an invalid ACPI description and is thus barred from using DMA due to not
knowing whether is is cache-coherent or not. Factor these out into
general dma-mapping code so that they can be referenced from other
common code paths. In the process, we can prune all the optional
callbacks which just do the same thing as the default behaviour, and
fill in .map_resource for completeness.
Signed-off-by: Robin Murphy <[email protected]>
[hch: moved to a separate source file]
Reviewed-by: Rafael J. Wysocki <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
This isn't exactly a slow path routine, but it is not super critical
either, and moving it out of line will help to keep the include chain
clean for the following DMA indirection bypass work.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
|
|
There is no need to have all setup and coherent allocation / freeing
routines inline. Move them out of line to keep the implemeation
nicely encapsulated and save some kernel text size.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
|
|
The two functions are exactly the same, so don't bother implementing
them twice.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
|
|
We can just call the regular calls after adding offset the the address instead
of reimplementing them.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
|
|
Currently dma_mapping_error returns a boolean as int, with 1 meaning
error. This is rather unusual and many callers have to convert it to
errno value. The callers are highly inconsistent with error codes
ranging from -ENOMEM over -EIO, -EINVAL and -EFAULT ranging to -EAGAIN.
Return -ENOMEM which seems to be what the largest number of callers
convert it to, and which also matches the typical error case where
we are out of resources.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Russell King <[email protected]>
Acked-by: Linus Torvalds <[email protected]>
|
|
No users left except for vmd which just forwards it.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Linus Torvalds <[email protected]>
|
|
Error handling of the dma_map_single and dma_map_page APIs is a little
problematic at the moment, in that we use different encodings in the
returned dma_addr_t to indicate an error. That means we require an
additional indirect call to figure out if a dma mapping call returned
an error, and a lot of boilerplate code to implement these semantics.
Instead return the maximum addressable value as the error. As long
as we don't allow mapping single-byte ranges with single-byte alignment
this value can never be a valid return. Additionaly if drivers do
not check the return value from the dma_map* routines this values means
they will generally not be pointed to actual memory.
Once the default value is added here we can start removing the
various mapping_error methods and just rely on this generic check.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Robin Murphy <[email protected]>
Acked-by: Russell King <[email protected]>
Acked-by: Linus Torvalds <[email protected]>
|
|
The arm64 codebase to implement coherent dma allocation for architectures
with non-coherent DMA is a good start for a generic implementation, given
that is uses the generic remap helpers, provides the atomic pool for
allocations that can't sleep and still is realtively simple and well
tested. Move it to kernel/dma and allow architectures to opt into it
using a config symbol. Architectures just need to provide a new
arch_dma_prep_coherent helper to writeback an invalidate the caches
for any memory that gets remapped for uncached access.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Will Deacon <[email protected]>
Reviewed-by: Robin Murphy <[email protected]>
|
|
The function dma_set_max_seg_size() can return either 0 on success or
-EIO on error. Change its return type from unsigned int to int to
capture this.
Signed-off-by: Niklas Söderlund <[email protected]>
Reviewed-by: Geert Uytterhoeven <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
This allows all dma_map_ops instances to entirely rely on
DMA_ATTR_NO_WARN going forward.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Robin Murphy <[email protected]>
|
|
I recently debugged a DMA mapping oops where a driver was trying to map
a buffer returned from request_firmware() with dma_map_single(). Memory
returned from request_firmware() is mapped into the vmalloc region and
this isn't a valid region to map with dma_map_single() per the DMA
documentation's "What memory is DMA'able?" section.
Unfortunately, we don't really check that in the DMA debugging code, so
enabling DMA debugging doesn't help catch this problem. Let's add a new
DMA debug function to check for a vmalloc address or an invalid virtual
address and print a warning if this happens. This makes it a little
easier to debug these sorts of problems, instead of seeing odd behavior
or crashes when drivers attempt to map the vmalloc space for DMA.
Cc: Marek Szyprowski <[email protected]>
Reviewed-by: Robin Murphy <[email protected]>
Signed-off-by: Stephen Boyd <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
This save some duplication for ia64, and makes the interface more
general. In the long run we want each dma_map_ops instance to fill this
out, but this will take a little more prep work.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Benjamin Herrenschmidt <[email protected]>
|
|
This reverts commit 46053c73685411915d3de50c5a0045beef32806b.
This change breaks architectures setting up dma_ops in their own magic
way and not using arch_setup_dma_ops, so revert it.
Reported-by: Guenter Roeck <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
We can use the arch_dma_coherent_to_pfn hook to provide a ->get_sgtable
implementation. Note that this isn't an endorsement of this interface
(which is a horrible bad idea), but it is required to move arm64 over
to the generic code without a loss of functionality.
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
The only functional differences (modulo a few missing fixes in the arch
code) is that architectures without coherent caches need a hook to
convert a virtual or dma address into a pfn, given that we don't have
the kernel linear mapping available for the otherwise easy virt_to_page
call. As a side effect we can support mmap of the per-device coherent
area even on architectures not providing the callback, and we make
previous dangerous default methods dma_common_mmap actually save for
non-coherent architectures by rejecting it without the right helper.
In addition to that we need a hook so that some architectures can
override the protection bits when mmaping a dma coherent allocations.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Paul Burton <[email protected]> # MIPS parts
|
|
All the cache maintainance is already stubbed out when not enabled,
but merging the two allows us to nicely handle the case where
cache maintainance is required for some devices, but not others.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Paul Burton <[email protected]> # MIPS parts
|
|
There is no reason to leave the per-device dma_ops around when
deconfiguring a device, so move this code from arm64 into the
common code.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Robin Murphy <[email protected]>
|
|
This goes through a lot of hooks just to call arch_teardown_dma_ops.
Replace it with a direct call instead.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Robin Murphy <[email protected]>
|
|
There is no good reason for this indirection given that the method
always exists.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Robin Murphy <[email protected]>
|
|
The reasons why dma_free_attrs() should not be called from IRQ context
are not necessarily obvious and somewhat buried in the development
history, so let's start by documenting the warning itself to help anyone
who does happen to hit it and wonder what the deal is.
However, this check turns out to be slightly over-restrictive for the
way that per-device memory has been spliced into the general API, since
for that case we know that dma_declare_coherent_memory() has created an
appropriate CPU mapping for the entire area and nothing dynamic should
be happening. Given that the usage model for per-device memory is often
more akin to streaming DMA than 'real' coherent DMA (e.g. allocating and
freeing space to copy short-lived packets in and out), it is also
somewhat more reasonable for those operations to happen in IRQ handlers
for such devices.
Therefore, let's move the irqs_disabled() check down past the per-device
area hook, so that that gets a chance to resolve the request before we
reach definite "you're doing it wrong" territory.
Reported-by: Fredrik Noring <[email protected]>
Tested-by: Fredrik Noring <[email protected]>
Signed-off-by: Robin Murphy <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
Instead of globally disabling > 32bit DMA using the arch_dma_supported
hook walk the PCI bus under the actually affected bridge and mark every
device with the dma_32bit_limit flag. This also gets rid of the
arch_dma_supported hook entirely.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Thomas Gleixner <[email protected]>
|
|
Signed-off-by: Huaisheng Ye <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
Add a new dma_map_ops implementation that uses dma-direct for the
address mapping of streaming mappings, and which requires arch-specific
implemenations of coherent allocate/free.
Architectures have to provide flushing helpers to ownership trasnfers
to the device and/or CPU, and can provide optional implementations of
the coherent mmap functionality, and the cache_flush routines for
non-coherent long term allocations.
Signed-off-by: Christoph Hellwig <[email protected]>
Tested-by: Alexey Brodkin <[email protected]>
Acked-by: Vineet Gupta <[email protected]>
|
|
This way we have one central definition of it, and user can select it as
needed. Note that we now also always select it when CONFIG_DMA_API_DEBUG
is select, which fixes some incorrect checks in a few network drivers.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Anshuman Khandual <[email protected]>
|
|
This was used by the ide, scsi and networking code in the past to
determine if they should bounce payloads. Now that the dma mapping
always have to support dma to all physical memory (thanks to swiotlb
for non-iommu systems) there is no need to this crude hack any more.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Palmer Dabbelt <[email protected]> (for riscv)
Reviewed-by: Jens Axboe <[email protected]>
|
|
Pull dma-mapping updates from Christoph Hellwig:
"Very light this round as the interesting dma mapping changes went
through the x86 tree.
This just provides proper stubs for architectures not supporting dma
(Geert Uytterhoeven)"
* tag 'dma-mapping-4.17' of git://git.infradead.org/users/hch/dma-mapping:
usb: gadget: Add NO_DMA dummies for DMA mapping API
scsi: Add NO_DMA dummies for SCSI DMA mapping API
mm: Add NO_DMA dummies for DMA pool API
dma-coherent: Add NO_DMA dummies for managed DMA API
dma-mapping: Convert NO_DMA get_dma_ops() into a real dummy
|
|
Revert the clearing of __GFP_ZERO in dma_alloc_attrs and move it to
dma_direct_alloc for now. While most common architectures always zero dma
cohereny allocations (and x86 did so since day one) this is not documented
and at least arc and s390 do not zero without the explicit __GFP_ZERO
argument.
Fixes: 57bf5a8963f8 ("dma-mapping: clear harmful GFP_* flags in common code")
Reported-by: Evgeniy Didin <[email protected]>
Reported-by: Sebastian Ott <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
Signed-off-by: Thomas Gleixner <[email protected]>
Tested-by: Evgeniy Didin <[email protected]>
Cc: [email protected]
Link: https://lkml.kernel.org/r/[email protected]
|
|
Add dummies for dmam_{alloc,free}_coherent(), to allow compile-testing
if NO_DMA=y.
This prevents the following from showing up later:
ERROR: "dmam_alloc_coherent" [drivers/net/ethernet/arc/arc_emac.ko] undefined!
ERROR: "dmam_free_coherent" [drivers/net/ethernet/apm/xgene/xgene-enet.ko] undefined!
ERROR: "dmam_alloc_coherent" [drivers/net/ethernet/apm/xgene/xgene-enet.ko] undefined!
ERROR: "dmam_alloc_coherent" [drivers/mtd/nand/hisi504_nand.ko] undefined!
ERROR: "dmam_alloc_coherent" [drivers/mmc/host/dw_mmc.ko] undefined!
Signed-off-by: Geert Uytterhoeven <[email protected]>
Reviewed-by: Mark Brown <[email protected]>
Acked-by: Robin Murphy <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
If NO_DMA=y, get_dma_ops() returns a reference to the
non-existing symbol bad_dma_ops, thus causing a link failure if it is
ever used.
Make get_dma_ops() return NULL instead, to avoid the link failure.
This allows to improve compile-testing, and limits the need to keep on
sprinkling dependencies on HAS_DMA all over the place.
Signed-off-by: Geert Uytterhoeven <[email protected]>
Reviewed-by: Mark Brown <[email protected]>
Acked-by: Robin Murphy <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
Reported-by: Randy Dunlap <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
|
|
The trivial direct mapping implementation already does a virtual to
physical translation which isn't strictly a noop, and will soon learn
to do non-direct but linear physical to dma translations through the
device offset and a few small tricks. Rename it to a better fitting
name.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Vladimir Murzin <[email protected]>
|
|
To implement the x86 forbid_dac and iommu_sac_force we want an arch hook
so that it can apply the global options across all dma_map_ops
implementations.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Konrad Rzeszutek Wilk <[email protected]>
|
|
Lift the code from x86 so that we behave consistently. In the future we
should probably warn if any of these is set.
Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Jesper Nilsson <[email protected]>
Acked-by: Geert Uytterhoeven <[email protected]> [m68k]
|
|
These days all devices should have a DMA coherent mask, and most dma_ops
implementations rely on that fact. But just to be sure add an assert to
ring the warning bell if that is not the case.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Vladimir Murzin <[email protected]>
Reviewed-by: Konrad Rzeszutek Wilk <[email protected]>
|
|
This makes sure the generic version can be used with architectures /
devices that have a DMA offset in the direct mapping.
Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Robin Murphy <[email protected]>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
Pull SCSI fixes from James Bottomley:
"A bunch of fixes for aacraid, a set of coherency fixes that only
affect non-coherent platforms and one coccinelle detected null check
after use"
* tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi:
scsi: libsas: align sata_device's rps_resp on a cacheline
scsi: use dma_get_cache_alignment() as minimum DMA alignment
scsi: dma-mapping: always provide dma_get_cache_alignment
scsi: ufs: ufshcd: fix potential NULL pointer dereference in ufshcd_config_vreg
scsi: aacraid: Prevent crash in case of free interrupt during scsi EH path
scsi: aacraid: Perform initialization reset only once
scsi: aacraid: Check for PCI state of device in a generic way
|
|
Provide the dummy version of dma_get_cache_alignment that always returns
1 even if CONFIG_HAS_DMA is not set, so that drivers and subsystems can
use it without ifdefs.
Cc: [email protected]
Signed-off-by: Christoph Hellwig <[email protected]>
Signed-off-by: Martin K. Petersen <[email protected]>
|
|
Patch series "kmemcheck: kill kmemcheck", v2.
As discussed at LSF/MM, kill kmemcheck.
KASan is a replacement that is able to work without the limitation of
kmemcheck (single CPU, slow). KASan is already upstream.
We are also not aware of any users of kmemcheck (or users who don't
consider KASan as a suitable replacement).
The only objection was that since KASAN wasn't supported by all GCC
versions provided by distros at that time we should hold off for 2
years, and try again.
Now that 2 years have passed, and all distros provide gcc that supports
KASAN, kill kmemcheck again for the very same reasons.
This patch (of 4):
Remove kmemcheck annotations, and calls to kmemcheck from the kernel.
[[email protected]: correctly remove kmemcheck call from dma_map_sg_attrs]
Link: http://lkml.kernel.org/r/[email protected]
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Sasha Levin <[email protected]>
Cc: Alexander Potapenko <[email protected]>
Cc: Eric W. Biederman <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: Pekka Enberg <[email protected]>
Cc: Steven Rostedt <[email protected]>
Cc: Tim Hansen <[email protected]>
Cc: Vegard Nossum <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
|