diff options
668 files changed, 46020 insertions, 13641 deletions
@@ -232,6 +232,8 @@ Johan Hovold <[email protected]> <[email protected]> John Crispin <[email protected]> <[email protected]> John Paul Adrian Glaubitz <[email protected]> John Stultz <[email protected]> Jordan Crouse <[email protected]> <[email protected]> @@ -297,6 +299,8 @@ Martin Kepplinger <[email protected]> <[email protected]> Martin Kepplinger <[email protected]> <[email protected]> Martyna Szapar-Mudlaw <[email protected]> <[email protected]> Mathieu Othacehe <[email protected]> +Mat Martineau <[email protected]> <[email protected]> +Mat Martineau <[email protected]> <[email protected]> Matthew Wilcox <[email protected]> <[email protected]> Matthew Wilcox <[email protected]> <[email protected]> Matthew Wilcox <[email protected]> <[email protected]> diff --git a/Documentation/admin-guide/kernel-parameters.rst b/Documentation/admin-guide/kernel-parameters.rst index 19600c50277b..6ae5f129fbca 100644 --- a/Documentation/admin-guide/kernel-parameters.rst +++ b/Documentation/admin-guide/kernel-parameters.rst @@ -128,6 +128,7 @@ parameter is applicable:: KVM Kernel Virtual Machine support is enabled. LIBATA Libata driver is enabled LP Printer support is enabled. + LOONGARCH LoongArch architecture is enabled. LOOP Loopback device support is enabled. M68k M68k architecture is enabled. These options have more detailed description inside of diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 6221a1d057dd..7016cb12dc4e 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -6933,6 +6933,12 @@ When enabled, memory and cache locality will be impacted. + writecombine= [LOONGARCH] Control the MAT (Memory Access Type) of + ioremap_wc(). + + on - Enable writecombine, use WUC for ioremap_wc() + off - Disable writecombine, use SUC for ioremap_wc() + x2apic_phys [X86-64,APIC] Use x2apic physical mode instead of default x2apic cluster mode on platforms supporting x2apic. diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst index 3b42cfe12437..ea2516374d92 100644 --- a/Documentation/bpf/kfuncs.rst +++ b/Documentation/bpf/kfuncs.rst @@ -184,16 +184,7 @@ in. All copies of the pointer being released are invalidated as a result of invoking kfunc with this flag. KF_RELEASE kfuncs automatically receive the protection afforded by the KF_TRUSTED_ARGS flag described below. -2.4.4 KF_KPTR_GET flag ----------------------- - -The KF_KPTR_GET flag is used to indicate that the kfunc takes the first argument -as a pointer to kptr, safely increments the refcount of the object it points to, -and returns a reference to the user. The rest of the arguments may be normal -arguments of a kfunc. The KF_KPTR_GET flag should be used in conjunction with -KF_ACQUIRE and KF_RET_NULL flags. - -2.4.5 KF_TRUSTED_ARGS flag +2.4.4 KF_TRUSTED_ARGS flag -------------------------- The KF_TRUSTED_ARGS flag is used for kfuncs taking pointer arguments. It @@ -205,7 +196,7 @@ exception described below). There are two types of pointers to kernel objects which are considered "valid": 1. Pointers which are passed as tracepoint or struct_ops callback arguments. -2. Pointers which were returned from a KF_ACQUIRE or KF_KPTR_GET kfunc. +2. Pointers which were returned from a KF_ACQUIRE kfunc. Pointers to non-BTF objects (e.g. scalar pointers) may also be passed to KF_TRUSTED_ARGS kfuncs, and may have a non-zero offset. @@ -232,13 +223,13 @@ In other words, you must: 2. Specify the type and name of the trusted nested field. This field must match the field in the original type definition exactly. -2.4.6 KF_SLEEPABLE flag +2.4.5 KF_SLEEPABLE flag ----------------------- The KF_SLEEPABLE flag is used for kfuncs that may sleep. Such kfuncs can only be called by sleepable BPF programs (BPF_F_SLEEPABLE). -2.4.7 KF_DESTRUCTIVE flag +2.4.6 KF_DESTRUCTIVE flag -------------------------- The KF_DESTRUCTIVE flag is used to indicate functions calling which is @@ -247,7 +238,7 @@ rebooting or panicking. Due to this additional restrictions apply to these calls. At the moment they only require CAP_SYS_BOOT capability, but more can be added later. -2.4.8 KF_RCU flag +2.4.7 KF_RCU flag ----------------- The KF_RCU flag is a weaker version of KF_TRUSTED_ARGS. The kfuncs marked with @@ -260,7 +251,7 @@ also be KF_RET_NULL. .. _KF_deprecated_flag: -2.4.9 KF_DEPRECATED flag +2.4.8 KF_DEPRECATED flag ------------------------ The KF_DEPRECATED flag is used for kfuncs which are scheduled to be diff --git a/Documentation/devicetree/bindings/net/wireless/mediatek,mt76.yaml b/Documentation/devicetree/bindings/net/wireless/mediatek,mt76.yaml index 7d526ff53fb7..67b63f119f64 100644 --- a/Documentation/devicetree/bindings/net/wireless/mediatek,mt76.yaml +++ b/Documentation/devicetree/bindings/net/wireless/mediatek,mt76.yaml @@ -111,6 +111,11 @@ properties: $ref: /schemas/leds/common.yaml# additionalProperties: false properties: + led-active-low: + description: + LED is enabled with ground signal. + type: boolean + led-sources: maxItems: 1 diff --git a/Documentation/devicetree/bindings/net/wireless/qcom,ath10k.txt b/Documentation/devicetree/bindings/net/wireless/qcom,ath10k.txt deleted file mode 100644 index b61c2d5a0ff7..000000000000 --- a/Documentation/devicetree/bindings/net/wireless/qcom,ath10k.txt +++ /dev/null @@ -1,215 +0,0 @@ -* Qualcomm Atheros ath10k wireless devices - -Required properties: -- compatible: Should be one of the following: - * "qcom,ath10k" - * "qcom,ipq4019-wifi" - * "qcom,wcn3990-wifi" - -PCI based devices uses compatible string "qcom,ath10k" and takes calibration -data along with board specific data via "qcom,ath10k-calibration-data". -Rest of the properties are not applicable for PCI based devices. - -AHB based devices (i.e. ipq4019) uses compatible string "qcom,ipq4019-wifi" -and also uses most of the properties defined in this doc (except -"qcom,ath10k-calibration-data"). It uses "qcom,ath10k-pre-calibration-data" -to carry pre calibration data. - -In general, entry "qcom,ath10k-pre-calibration-data" and -"qcom,ath10k-calibration-data" conflict with each other and only one -can be provided per device. - -SNOC based devices (i.e. wcn3990) uses compatible string "qcom,wcn3990-wifi". - -- reg: Address and length of the register set for the device. -- reg-names: Must include the list of following reg names, - "membase" -- interrupts: reference to the list of 17 interrupt numbers for "qcom,ipq4019-wifi" - compatible target. - reference to the list of 12 interrupt numbers for "qcom,wcn3990-wifi" - compatible target. - Must contain interrupt-names property per entry for - "qcom,ath10k", "qcom,ipq4019-wifi" compatible targets. - -- interrupt-names: Must include the entries for MSI interrupt - names ("msi0" to "msi15") and legacy interrupt - name ("legacy") for "qcom,ath10k", "qcom,ipq4019-wifi" - compatible targets. - -Optional properties: -- resets: Must contain an entry for each entry in reset-names. - See ../reset/reseti.txt for details. -- reset-names: Must include the list of following reset names, - "wifi_cpu_init" - "wifi_radio_srif" - "wifi_radio_warm" - "wifi_radio_cold" - "wifi_core_warm" - "wifi_core_cold" -- clocks: List of clock specifiers, must contain an entry for each required - entry in clock-names. -- clock-names: Should contain the clock names "wifi_wcss_cmd", "wifi_wcss_ref", - "wifi_wcss_rtc" for "qcom,ipq4019-wifi" compatible target and - "cxo_ref_clk_pin" and optionally "qdss" for "qcom,wcn3990-wifi" - compatible target. -- qcom,msi_addr: MSI interrupt address. -- qcom,msi_base: Base value to add before writing MSI data into - MSI address register. -- qcom,ath10k-calibration-variant: string to search for in the board-2.bin - variant list with the same bus and device - specific ids -- qcom,ath10k-calibration-data : calibration data + board specific data - as an array, the length can vary between - hw versions. -- qcom,ath10k-pre-calibration-data : pre calibration data as an array, - the length can vary between hw versions. -- <supply-name>-supply: handle to the regulator device tree node - optional "supply-name" are "vdd-0.8-cx-mx", - "vdd-1.8-xo", "vdd-1.3-rfa", "vdd-3.3-ch0", - and "vdd-3.3-ch1". -- memory-region: - Usage: optional - Value type: <phandle> - Definition: reference to the reserved-memory for the msa region - used by the wifi firmware running in Q6. -- iommus: - Usage: optional - Value type: <prop-encoded-array> - Definition: A list of phandle and IOMMU specifier pairs. -- ext-fem-name: - Usage: Optional - Value type: string - Definition: Name of external front end module used. Some valid FEM names - for example: "microsemi-lx5586", "sky85703-11" - and "sky85803" etc. -- qcom,snoc-host-cap-8bit-quirk: - Usage: Optional - Value type: <empty> - Definition: Quirk specifying that the firmware expects the 8bit version - of the host capability QMI request -- qcom,xo-cal-data: xo cal offset to be configured in xo trim register. - -- qcom,msa-fixed-perm: Boolean context flag to disable SCM call for statically - mapped msa region. - -- qcom,coexist-support : should contain eithr "0" or "1" to indicate coex - support by the hardware. -- qcom,coexist-gpio-pin : gpio pin number information to support coex - which will be used by wifi firmware. - -* Subnodes -The ath10k wifi node can contain one optional firmware subnode. -Firmware subnode is needed when the platform does not have TustZone. -The firmware subnode must have: - -- iommus: - Usage: required - Value type: <prop-encoded-array> - Definition: A list of phandle and IOMMU specifier pairs. - - -Example (to supply PCI based wifi block details): - -In this example, the node is defined as child node of the PCI controller. - -pci { - pcie@0 { - reg = <0 0 0 0 0>; - #interrupt-cells = <1>; - #size-cells = <2>; - #address-cells = <3>; - device_type = "pci"; - - wifi@0,0 { - reg = <0 0 0 0 0>; - qcom,ath10k-calibration-data = [ 01 02 03 ... ]; - ext-fem-name = "microsemi-lx5586"; - }; - }; -}; - -Example (to supply ipq4019 SoC wifi block details): - -wifi0: wifi@a000000 { - compatible = "qcom,ipq4019-wifi"; - reg = <0xa000000 0x200000>; - resets = <&gcc WIFI0_CPU_INIT_RESET>, - <&gcc WIFI0_RADIO_SRIF_RESET>, - <&gcc WIFI0_RADIO_WARM_RESET>, - <&gcc WIFI0_RADIO_COLD_RESET>, - <&gcc WIFI0_CORE_WARM_RESET>, - <&gcc WIFI0_CORE_COLD_RESET>; - reset-names = "wifi_cpu_init", - "wifi_radio_srif", - "wifi_radio_warm", - "wifi_radio_cold", - "wifi_core_warm", - "wifi_core_cold"; - clocks = <&gcc GCC_WCSS2G_CLK>, - <&gcc GCC_WCSS2G_REF_CLK>, - <&gcc GCC_WCSS2G_RTC_CLK>; - clock-names = "wifi_wcss_cmd", - "wifi_wcss_ref", - "wifi_wcss_rtc"; - interrupts = <0 0x20 0x1>, - <0 0x21 0x1>, - <0 0x22 0x1>, - <0 0x23 0x1>, - <0 0x24 0x1>, - <0 0x25 0x1>, - <0 0x26 0x1>, - <0 0x27 0x1>, - <0 0x28 0x1>, - <0 0x29 0x1>, - <0 0x2a 0x1>, - <0 0x2b 0x1>, - <0 0x2c 0x1>, - <0 0x2d 0x1>, - <0 0x2e 0x1>, - <0 0x2f 0x1>, - <0 0xa8 0x0>; - interrupt-names = "msi0", "msi1", "msi2", "msi3", - "msi4", "msi5", "msi6", "msi7", - "msi8", "msi9", "msi10", "msi11", - "msi12", "msi13", "msi14", "msi15", - "legacy"; - qcom,msi_addr = <0x0b006040>; - qcom,msi_base = <0x40>; - qcom,ath10k-pre-calibration-data = [ 01 02 03 ... ]; - qcom,coexist-support = <1>; - qcom,coexist-gpio-pin = <0x33>; -}; - -Example (to supply wcn3990 SoC wifi block details): - -wifi@18000000 { - compatible = "qcom,wcn3990-wifi"; - reg = <0x18800000 0x800000>; - reg-names = "membase"; - clocks = <&clock_gcc clk_rf_clk2_pin>; - clock-names = "cxo_ref_clk_pin"; - interrupts = - <GIC_SPI 414 IRQ_TYPE_LEVEL_HIGH>, - <GIC_SPI 415 IRQ_TYPE_LEVEL_HIGH>, - <GIC_SPI 416 IRQ_TYPE_LEVEL_HIGH>, - <GIC_SPI 417 IRQ_TYPE_LEVEL_HIGH>, - <GIC_SPI 418 IRQ_TYPE_LEVEL_HIGH>, - <GIC_SPI 419 IRQ_TYPE_LEVEL_HIGH>, - <GIC_SPI 420 IRQ_TYPE_LEVEL_HIGH>, - <GIC_SPI 421 IRQ_TYPE_LEVEL_HIGH>, - <GIC_SPI 422 IRQ_TYPE_LEVEL_HIGH>, - <GIC_SPI 423 IRQ_TYPE_LEVEL_HIGH>, - <GIC_SPI 424 IRQ_TYPE_LEVEL_HIGH>, - <GIC_SPI 425 IRQ_TYPE_LEVEL_HIGH>; - vdd-0.8-cx-mx-supply = <&pm8998_l5>; - vdd-1.8-xo-supply = <&vreg_l7a_1p8>; - vdd-1.3-rfa-supply = <&vreg_l17a_1p3>; - vdd-3.3-ch0-supply = <&vreg_l25a_3p3>; - vdd-3.3-ch1-supply = <&vreg_l26a_3p3>; - memory-region = <&wifi_msa_mem>; - iommus = <&apps_smmu 0x0040 0x1>; - qcom,msa-fixed-perm; - wifi-firmware { - iommus = <&apps_iommu 0xc22 0x1>; - }; -}; diff --git a/Documentation/devicetree/bindings/net/wireless/qcom,ath10k.yaml b/Documentation/devicetree/bindings/net/wireless/qcom,ath10k.yaml new file mode 100644 index 000000000000..c85ed330426d --- /dev/null +++ b/Documentation/devicetree/bindings/net/wireless/qcom,ath10k.yaml @@ -0,0 +1,358 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/net/wireless/qcom,ath10k.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Qualcomm Technologies ath10k wireless devices + +maintainers: + - Kalle Valo <[email protected]> + +description: + Qualcomm Technologies, Inc. IEEE 802.11ac devices. + +properties: + compatible: + enum: + - qcom,ath10k # SDIO-based devices + - qcom,ipq4019-wifi + - qcom,wcn3990-wifi # SNoC-based devices + + reg: + maxItems: 1 + + reg-names: + items: + - const: membase + + interrupts: + minItems: 12 + maxItems: 17 + + interrupt-names: + minItems: 12 + maxItems: 17 + + memory-region: + maxItems: 1 + description: + Reference to the MSA memory region used by the Wi-Fi firmware + running on the Q6 core. + + iommus: + minItems: 1 + maxItems: 2 + + clocks: + minItems: 1 + maxItems: 3 + + clock-names: + minItems: 1 + maxItems: 3 + + resets: + maxItems: 6 + + reset-names: + items: + - const: wifi_cpu_init + - const: wifi_radio_srif + - const: wifi_radio_warm + - const: wifi_radio_cold + - const: wifi_core_warm + - const: wifi_core_cold + + ext-fem-name: + $ref: /schemas/types.yaml#/definitions/string + description: Name of external front end module used. + enum: + - microsemi-lx5586 + - sky85703-11 + - sky85803 + + wifi-firmware: + type: object + additionalProperties: false + description: | + The ath10k Wi-Fi node can contain one optional firmware subnode. + Firmware subnode is needed when the platform does not have Trustzone. + properties: + iommus: + maxItems: 1 + required: + - iommus + + qcom,ath10k-calibration-data: + $ref: /schemas/types.yaml#/definitions/uint8-array + description: + Calibration data + board-specific data as a byte array. The length + can vary between hardware versions. + + qcom,ath10k-calibration-variant: + $ref: /schemas/types.yaml#/definitions/string + description: + Unique variant identifier of the calibration data in board-2.bin + for designs with colliding bus and device specific ids + + qcom,ath10k-pre-calibration-data: + $ref: /schemas/types.yaml#/definitions/uint8-array + description: + Pre-calibration data as a byte array. The length can vary between + hardware versions. + + qcom,coexist-support: + $ref: /schemas/types.yaml#/definitions/uint8 + enum: [0, 1] + description: + Indicate coex support by the hardware. + + qcom,coexist-gpio-pin: + $ref: /schemas/types.yaml#/definitions/uint32 + description: + COEX GPIO number provided to the Wi-Fi firmware. + + qcom,msa-fixed-perm: + type: boolean + description: + Whether to skip executing an SCM call that reassigns the memory + region ownership. + + qcom,smem-states: + $ref: /schemas/types.yaml#/definitions/phandle-array + description: State bits used by the AP to signal the WLAN Q6. + items: + - description: Signal bits used to enable/disable low power mode + on WCN in the case of WoW (Wake on Wireless). + + qcom,smem-state-names: + description: The names of the state bits used for SMP2P output. + items: + - const: wlan-smp2p-out + + qcom,snoc-host-cap-8bit-quirk: + type: boolean + description: + Quirk specifying that the firmware expects the 8bit version + of the host capability QMI request + + qcom,xo-cal-data: + $ref: /schemas/types.yaml#/definitions/uint32 + description: + XO cal offset to be configured in XO trim register. + + vdd-0.8-cx-mx-supply: + description: Main logic power rail + + vdd-1.8-xo-supply: + description: Crystal oscillator supply + + vdd-1.3-rfa-supply: + description: RFA supply + + vdd-3.3-ch0-supply: + description: Primary Wi-Fi antenna supply + + vdd-3.3-ch1-supply: + description: Secondary Wi-Fi antenna supply + +required: + - compatible + - reg + +additionalProperties: false + +allOf: + - if: + properties: + compatible: + contains: + enum: + - qcom,ipq4019-wifi + then: + properties: + interrupts: + minItems: 17 + maxItems: 17 + + interrupt-names: + items: + - const: msi0 + - const: msi1 + - const: msi2 + - const: msi3 + - const: msi4 + - const: msi5 + - const: msi6 + - const: msi7 + - const: msi8 + - const: msi9 + - const: msi10 + - const: msi11 + - const: msi12 + - const: msi13 + - const: msi14 + - const: msi15 + - const: legacy + + clocks: + items: + - description: Wi-Fi command clock + - description: Wi-Fi reference clock + - description: Wi-Fi RTC clock + + clock-names: + items: + - const: wifi_wcss_cmd + - const: wifi_wcss_ref + - const: wifi_wcss_rtc + + required: + - clocks + - clock-names + - interrupts + - interrupt-names + - resets + - reset-names + + - if: + properties: + compatible: + contains: + enum: + - qcom,wcn3990-wifi + + then: + properties: + clocks: + minItems: 1 + items: + - description: XO reference clock + - description: Qualcomm Debug Subsystem clock + + clock-names: + minItems: 1 + items: + - const: cxo_ref_clk_pin + - const: qdss + + interrupts: + items: + - description: CE0 + - description: CE1 + - description: CE2 + - description: CE3 + - description: CE4 + - description: CE5 + - description: CE6 + - description: CE7 + - description: CE8 + - description: CE9 + - description: CE10 + - description: CE11 + + interrupt-names: false + + required: + - interrupts + +examples: + # SNoC + - | + #include <dt-bindings/clock/qcom,rpmcc.h> + #include <dt-bindings/interrupt-controller/arm-gic.h> + + wifi@18800000 { + compatible = "qcom,wcn3990-wifi"; + reg = <0x18800000 0x800000>; + reg-names = "membase"; + memory-region = <&wlan_msa_mem>; + clocks = <&rpmcc RPM_SMD_RF_CLK2_PIN>; + clock-names = "cxo_ref_clk_pin"; + interrupts = <GIC_SPI 413 IRQ_TYPE_LEVEL_HIGH>, + <GIC_SPI 414 IRQ_TYPE_LEVEL_HIGH>, + <GIC_SPI 415 IRQ_TYPE_LEVEL_HIGH>, + <GIC_SPI 416 IRQ_TYPE_LEVEL_HIGH>, + <GIC_SPI 417 IRQ_TYPE_LEVEL_HIGH>, + <GIC_SPI 418 IRQ_TYPE_LEVEL_HIGH>, + <GIC_SPI 420 IRQ_TYPE_LEVEL_HIGH>, + <GIC_SPI 421 IRQ_TYPE_LEVEL_HIGH>, + <GIC_SPI 422 IRQ_TYPE_LEVEL_HIGH>, + <GIC_SPI 423 IRQ_TYPE_LEVEL_HIGH>, + <GIC_SPI 424 IRQ_TYPE_LEVEL_HIGH>, + <GIC_SPI 425 IRQ_TYPE_LEVEL_HIGH>; + iommus = <&anoc2_smmu 0x1900>, + <&anoc2_smmu 0x1901>; + qcom,snoc-host-cap-8bit-quirk; + vdd-0.8-cx-mx-supply = <&vreg_l5a_0p8>; + vdd-1.8-xo-supply = <&vreg_l7a_1p8>; + vdd-1.3-rfa-supply = <&vreg_l17a_1p3>; + vdd-3.3-ch0-supply = <&vreg_l25a_3p3>; + vdd-3.3-ch1-supply = <&vreg_l23a_3p3>; + + wifi-firmware { + iommus = <&apps_smmu 0x1c02 0x1>; + }; + }; + + # AHB + - | + #include <dt-bindings/clock/qcom,gcc-ipq4019.h> + + wifi@a000000 { + compatible = "qcom,ipq4019-wifi"; + reg = <0xa000000 0x200000>; + resets = <&gcc WIFI0_CPU_INIT_RESET>, + <&gcc WIFI0_RADIO_SRIF_RESET>, + <&gcc WIFI0_RADIO_WARM_RESET>, + <&gcc WIFI0_RADIO_COLD_RESET>, + <&gcc WIFI0_CORE_WARM_RESET>, + <&gcc WIFI0_CORE_COLD_RESET>; + reset-names = "wifi_cpu_init", + "wifi_radio_srif", + "wifi_radio_warm", + "wifi_radio_cold", + "wifi_core_warm", + "wifi_core_cold"; + clocks = <&gcc GCC_WCSS2G_CLK>, + <&gcc GCC_WCSS2G_REF_CLK>, + <&gcc GCC_WCSS2G_RTC_CLK>; + clock-names = "wifi_wcss_cmd", + "wifi_wcss_ref", + "wifi_wcss_rtc"; + interrupts = <GIC_SPI 32 IRQ_TYPE_EDGE_RISING>, + <GIC_SPI 33 IRQ_TYPE_EDGE_RISING>, + <GIC_SPI 34 IRQ_TYPE_EDGE_RISING>, + <GIC_SPI 35 IRQ_TYPE_EDGE_RISING>, + <GIC_SPI 36 IRQ_TYPE_EDGE_RISING>, + <GIC_SPI 37 IRQ_TYPE_EDGE_RISING>, + <GIC_SPI 38 IRQ_TYPE_EDGE_RISING>, + <GIC_SPI 39 IRQ_TYPE_EDGE_RISING>, + <GIC_SPI 40 IRQ_TYPE_EDGE_RISING>, + <GIC_SPI 41 IRQ_TYPE_EDGE_RISING>, + <GIC_SPI 42 IRQ_TYPE_EDGE_RISING>, + <GIC_SPI 43 IRQ_TYPE_EDGE_RISING>, + <GIC_SPI 44 IRQ_TYPE_EDGE_RISING>, + <GIC_SPI 45 IRQ_TYPE_EDGE_RISING>, + <GIC_SPI 46 IRQ_TYPE_EDGE_RISING>, + <GIC_SPI 47 IRQ_TYPE_EDGE_RISING>, + <GIC_SPI 168 IRQ_TYPE_LEVEL_HIGH>; + interrupt-names = "msi0", + "msi1", + "msi2", + "msi3", + "msi4", + "msi5", + "msi6", + "msi7", + "msi8", + "msi9", + "msi10", + "msi11", + "msi12", + "msi13", + "msi14", + "msi15", + "legacy"; + }; diff --git a/Documentation/devicetree/bindings/net/wireless/qcom,ath11k-pci.yaml b/Documentation/devicetree/bindings/net/wireless/qcom,ath11k-pci.yaml new file mode 100644 index 000000000000..817f02a8b481 --- /dev/null +++ b/Documentation/devicetree/bindings/net/wireless/qcom,ath11k-pci.yaml @@ -0,0 +1,58 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +# Copyright (c) 2023 Linaro Limited +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/net/wireless/qcom,ath11k-pci.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Qualcomm Technologies ath11k wireless devices (PCIe) + +maintainers: + - Kalle Valo <[email protected]> + +description: | + Qualcomm Technologies IEEE 802.11ax PCIe devices + +properties: + compatible: + enum: + - pci17cb,1103 # WCN6855 + + reg: + maxItems: 1 + + qcom,ath11k-calibration-variant: + $ref: /schemas/types.yaml#/definitions/string + description: | + string to uniquely identify variant of the calibration data for designs + with colliding bus and device ids + +required: + - compatible + - reg + +additionalProperties: false + +examples: + - | + pcie { + #address-cells = <3>; + #size-cells = <2>; + + pcie@0 { + device_type = "pci"; + reg = <0x0 0x0 0x0 0x0 0x0>; + #address-cells = <3>; + #size-cells = <2>; + ranges; + + bus-range = <0x01 0xff>; + + wifi@0 { + compatible = "pci17cb,1103"; + reg = <0x10000 0x0 0x0 0x0 0x0>; + + qcom,ath11k-calibration-variant = "LE_X13S"; + }; + }; + }; diff --git a/Documentation/kbuild/llvm.rst b/Documentation/kbuild/llvm.rst index bfb51685073c..c3851fe1900d 100644 --- a/Documentation/kbuild/llvm.rst +++ b/Documentation/kbuild/llvm.rst @@ -171,6 +171,10 @@ Getting Help Getting LLVM ------------- +We provide prebuilt stable versions of LLVM on `kernel.org <https://kernel.org/pub/tools/llvm/>`_. +Below are links that may be useful for building LLVM from source or procuring +it through a distribution's package manager. + - https://releases.llvm.org/download.html - https://github.com/llvm/llvm-project - https://llvm.org/docs/GettingStarted.html diff --git a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst new file mode 100644 index 000000000000..9e8a16c44102 --- /dev/null +++ b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst @@ -0,0 +1,139 @@ +.. SPDX-License-Identifier: GPL-2.0+ + +======================================================== +Linux Driver for the AMD/Pensando(R) DSC adapter family +======================================================== + +Copyright(c) 2023 Advanced Micro Devices, Inc + +Identifying the Adapter +======================= + +To find if one or more AMD/Pensando PCI Core devices are installed on the +host, check for the PCI devices:: + + # lspci -d 1dd8:100c + b5:00.0 Processing accelerators: Pensando Systems Device 100c + b6:00.0 Processing accelerators: Pensando Systems Device 100c + +If such devices are listed as above, then the pds_core.ko driver should find +and configure them for use. There should be log entries in the kernel +messages such as these:: + + $ dmesg | grep pds_core + pds_core 0000:b5:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link) + pds_core 0000:b5:00.0: FW: 1.60.0-73 + pds_core 0000:b6:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link) + pds_core 0000:b6:00.0: FW: 1.60.0-73 + +Driver and firmware version information can be gathered with devlink:: + + $ devlink dev info pci/0000:b5:00.0 + pci/0000:b5:00.0: + driver pds_core + serial_number FLM18420073 + versions: + fixed: + asic.id 0x0 + asic.rev 0x0 + running: + fw 1.51.0-73 + stored: + fw.goldfw 1.15.9-C-22 + fw.mainfwa 1.60.0-73 + fw.mainfwb 1.60.0-57 + +Info versions +============= + +The ``pds_core`` driver reports the following versions + +.. list-table:: devlink info versions implemented + :widths: 5 5 90 + + * - Name + - Type + - Description + * - ``fw`` + - running + - Version of firmware running on the device + * - ``fw.goldfw`` + - stored + - Version of firmware stored in the goldfw slot + * - ``fw.mainfwa`` + - stored + - Version of firmware stored in the mainfwa slot + * - ``fw.mainfwb`` + - stored + - Version of firmware stored in the mainfwb slot + * - ``asic.id`` + - fixed + - The ASIC type for this device + * - ``asic.rev`` + - fixed + - The revision of the ASIC for this device + +Parameters +========== + +The ``pds_core`` driver implements the following generic +parameters for controlling the functionality to be made available +as auxiliary_bus devices. + +.. list-table:: Generic parameters implemented + :widths: 5 5 8 82 + + * - Name + - Mode + - Type + - Description + * - ``enable_vnet`` + - runtime + - Boolean + - Enables vDPA functionality through an auxiliary_bus device + +Firmware Management +=================== + +The ``flash`` command can update a the DSC firmware. The downloaded firmware +will be saved into either of firmware bank 1 or bank 2, whichever is not +currently in use, and that bank will used for the next boot:: + + # devlink dev flash pci/0000:b5:00.0 \ + file pensando/dsc_fw_1.63.0-22.tar + +Health Reporters +================ + +The driver supports a devlink health reporter for FW status:: + + # devlink health show pci/0000:2b:00.0 reporter fw + pci/0000:2b:00.0: + reporter fw + state healthy error 0 recover 0 + # devlink health diagnose pci/0000:2b:00.0 reporter fw + Status: healthy State: 1 Generation: 0 Recoveries: 0 + +Enabling the driver +=================== + +The driver is enabled via the standard kernel configuration system, +using the make command:: + + make oldconfig/menuconfig/etc. + +The driver is located in the menu structure at: + + -> Device Drivers + -> Network device support (NETDEVICES [=y]) + -> Ethernet driver support + -> AMD devices + -> AMD/Pensando Ethernet PDS_CORE Support + +Support +======= + +For general Linux networking support, please use the netdev mailing +list, which is monitored by AMD/Pensando personnel:: + diff --git a/Documentation/networking/device_drivers/ethernet/index.rst b/Documentation/networking/device_drivers/ethernet/index.rst index 6e9e7012d000..417ca514a4d0 100644 --- a/Documentation/networking/device_drivers/ethernet/index.rst +++ b/Documentation/networking/device_drivers/ethernet/index.rst @@ -14,6 +14,7 @@ Contents: 3com/vortex amazon/ena altera/altera_tse + amd/pds_core aquantia/atlantic chelsio/cxgb cirrus/cs89x0 diff --git a/Documentation/networking/device_drivers/ethernet/mellanox/mlx5/devlink.rst b/Documentation/networking/device_drivers/ethernet/mellanox/mlx5/devlink.rst index 0995e4e5acd7..3a7a714cc08f 100644 --- a/Documentation/networking/device_drivers/ethernet/mellanox/mlx5/devlink.rst +++ b/Documentation/networking/device_drivers/ethernet/mellanox/mlx5/devlink.rst @@ -257,3 +257,36 @@ User commands examples: $ devlink health dump show pci/0000:82:00.1 reporter fw_fatal NOTE: This command can run only on PF. + +vnic reporter +------------- +The vnic reporter implements only the `diagnose` callback. +It is responsible for querying the vnic diagnostic counters from fw and displaying +them in realtime. + +Description of the vnic counters: +total_q_under_processor_handle: number of queues in an error state due to +an async error or errored command. +send_queue_priority_update_flow: number of QP/SQ priority/SL update +events. +cq_overrun: number of times CQ entered an error state due to an +overflow. +async_eq_overrun: number of times an EQ mapped to async events was +overrun. +comp_eq_overrun: number of times an EQ mapped to completion events was +overrun. +quota_exceeded_command: number of commands issued and failed due to quota +exceeded. +invalid_command: number of commands issued and failed dues to any reason +other than quota exceeded. +nic_receive_steering_discard: number of packets that completed RX flow +steering but were discarded due to a mismatch in flow table. + +User commands examples: +- Diagnose PF/VF vnic counters + $ devlink health diagnose pci/0000:82:00.1 reporter vnic +- Diagnose representor vnic counters (performed by supplying devlink port of the + representor, which can be obtained via devlink port command) + $ devlink health diagnose pci/0000:82:00.1/65537 reporter vnic + +NOTE: This command can run over all interfaces such as PF/VF and representor ports. diff --git a/Documentation/networking/devlink/ice.rst b/Documentation/networking/devlink/ice.rst index 10f282c2117c..2f60e34ab926 100644 --- a/Documentation/networking/devlink/ice.rst +++ b/Documentation/networking/devlink/ice.rst @@ -7,6 +7,21 @@ ice devlink support This document describes the devlink features implemented by the ``ice`` device driver. +Parameters +========== + +.. list-table:: Generic parameters implemented + + * - Name + - Mode + - Notes + * - ``enable_roce`` + - runtime + - mutually exclusive with ``enable_iwarp`` + * - ``enable_iwarp`` + - runtime + - mutually exclusive with ``enable_roce`` + Info versions ============= diff --git a/Documentation/networking/ip-sysctl.rst b/Documentation/networking/ip-sysctl.rst index 58a78a316697..6ec06a33688a 100644 --- a/Documentation/networking/ip-sysctl.rst +++ b/Documentation/networking/ip-sysctl.rst @@ -2721,6 +2721,13 @@ echo_ignore_anycast - BOOLEAN Default: 0 +error_anycast_as_unicast - BOOLEAN + If set to 1, then the kernel will respond with ICMP Errors + resulting from requests sent to it over the IPv6 protocol destined + to anycast address essentially treating anycast as unicast. + + Default: 0 + xfrm6_gc_thresh - INTEGER (Obsolete since linux-4.14) The threshold at which we will start garbage collecting for IPv6 diff --git a/Documentation/riscv/vm-layout.rst b/Documentation/riscv/vm-layout.rst index 3be44e74ec5d..5462c84f4723 100644 --- a/Documentation/riscv/vm-layout.rst +++ b/Documentation/riscv/vm-layout.rst @@ -47,7 +47,7 @@ RISC-V Linux Kernel SV39 | Kernel-space virtual memory, shared between all processes: ____________________________________________________________|___________________________________________________________ | | | | - ffffffc6fee00000 | -228 GB | ffffffc6feffffff | 2 MB | fixmap + ffffffc6fea00000 | -228 GB | ffffffc6feffffff | 6 MB | fixmap ffffffc6ff000000 | -228 GB | ffffffc6ffffffff | 16 MB | PCI io ffffffc700000000 | -228 GB | ffffffc7ffffffff | 4 GB | vmemmap ffffffc800000000 | -224 GB | ffffffd7ffffffff | 64 GB | vmalloc/ioremap space @@ -83,7 +83,7 @@ RISC-V Linux Kernel SV48 | Kernel-space virtual memory, shared between all processes: ____________________________________________________________|___________________________________________________________ | | | | - ffff8d7ffee00000 | -114.5 TB | ffff8d7ffeffffff | 2 MB | fixmap + ffff8d7ffea00000 | -114.5 TB | ffff8d7ffeffffff | 6 MB | fixmap ffff8d7fff000000 | -114.5 TB | ffff8d7fffffffff | 16 MB | PCI io ffff8d8000000000 | -114.5 TB | ffff8f7fffffffff | 2 TB | vmemmap ffff8f8000000000 | -112.5 TB | ffffaf7fffffffff | 32 TB | vmalloc/ioremap space @@ -119,7 +119,7 @@ RISC-V Linux Kernel SV57 | Kernel-space virtual memory, shared between all processes: ____________________________________________________________|___________________________________________________________ | | | | - ff1bfffffee00000 | -57 PB | ff1bfffffeffffff | 2 MB | fixmap + ff1bfffffea00000 | -57 PB | ff1bfffffeffffff | 6 MB | fixmap ff1bffffff000000 | -57 PB | ff1bffffffffffff | 16 MB | PCI io ff1c000000000000 | -57 PB | ff1fffffffffffff | 1 PB | vmemmap ff20000000000000 | -56 PB | ff5fffffffffffff | 16 PB | vmalloc/ioremap space diff --git a/Documentation/rust/arch-support.rst b/Documentation/rust/arch-support.rst index ed7f4f5b3cf1..b91e9ef4d0c2 100644 --- a/Documentation/rust/arch-support.rst +++ b/Documentation/rust/arch-support.rst @@ -15,7 +15,7 @@ support corresponds to ``S`` values in the ``MAINTAINERS`` file. ============ ================ ============================================== Architecture Level of support Constraints ============ ================ ============================================== -``x86`` Maintained ``x86_64`` only. ``um`` Maintained ``x86_64`` only. +``x86`` Maintained ``x86_64`` only. ============ ================ ============================================== diff --git a/Documentation/sound/hd-audio/models.rst b/Documentation/sound/hd-audio/models.rst index 9b52f50a6854..120430450014 100644 --- a/Documentation/sound/hd-audio/models.rst +++ b/Documentation/sound/hd-audio/models.rst @@ -704,7 +704,7 @@ ref no-jd BIOS setup but without jack-detection intel - Intel DG45* mobos + Intel D*45* mobos dell-m6-amic Dell desktops/laptops with analog mics dell-m6-dmic diff --git a/MAINTAINERS b/MAINTAINERS index 04ebde8ccb75..6ac562e0381e 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1041,6 +1041,15 @@ F: drivers/gpu/drm/amd/include/vi_structs.h F: include/uapi/linux/kfd_ioctl.h F: include/uapi/linux/kfd_sysfs.h +AMD PDS CORE DRIVER +M: Shannon Nelson <[email protected]> +M: Brett Creeley <[email protected]> +S: Supported +F: Documentation/networking/device_drivers/ethernet/amd/pds_core.rst +F: drivers/net/ethernet/amd/pds_core/ +F: include/linux/pds/ + AMD SPI DRIVER M: Sanjay R Mehta <[email protected]> S: Maintained @@ -14623,6 +14632,7 @@ F: net/netlabel/ NETWORKING [MPTCP] M: Matthieu Baerts <[email protected]> +M: Mat Martineau <[email protected]> S: Maintained @@ -17231,7 +17241,7 @@ S: Supported W: https://wireless.wiki.kernel.org/en/users/Drivers/ath10k T: git git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git F: drivers/net/wireless/ath/ath10k/ -F: Documentation/devicetree/bindings/net/wireless/qcom,ath10k.txt +F: Documentation/devicetree/bindings/net/wireless/qcom,ath10k.yaml QUALCOMM ATHEROS ATH11K WIRELESS DRIVER M: Kalle Valo <[email protected]> @@ -2,7 +2,7 @@ VERSION = 6 PATCHLEVEL = 3 SUBLEVEL = 0 -EXTRAVERSION = -rc6 +EXTRAVERSION = -rc7 NAME = Hurr durr I'ma ninja sloth # *DOCUMENTATION* diff --git a/arch/arm/boot/dts/imx6ull-colibri.dtsi b/arch/arm/boot/dts/imx6ull-colibri.dtsi index bf64ba84b358..fde8a19aac0f 100644 --- a/arch/arm/boot/dts/imx6ull-colibri.dtsi +++ b/arch/arm/boot/dts/imx6ull-colibri.dtsi @@ -33,15 +33,9 @@ self-powered; type = "micro"; - ports { - #address-cells = <1>; - #size-cells = <0>; - - port@0 { - reg = <0>; - usb_dr_connector: endpoint { - remote-endpoint = <&usb1_drd_sw>; - }; + port { + usb_dr_connector: endpoint { + remote-endpoint = <&usb1_drd_sw>; }; }; }; diff --git a/arch/arm/boot/dts/imx7d-remarkable2.dts b/arch/arm/boot/dts/imx7d-remarkable2.dts index 8b2f11e85e05..427f8d04ec89 100644 --- a/arch/arm/boot/dts/imx7d-remarkable2.dts +++ b/arch/arm/boot/dts/imx7d-remarkable2.dts @@ -118,8 +118,6 @@ reg = <0x62>; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_epdpmic>; - #address-cells = <1>; - #size-cells = <0>; #thermal-sensor-cells = <0>; epd-pwr-good-gpios = <&gpio6 21 GPIO_ACTIVE_HIGH>; diff --git a/arch/arm/boot/dts/rk3288.dtsi b/arch/arm/boot/dts/rk3288.dtsi index 2ca76b69add7..511ca864c1b2 100644 --- a/arch/arm/boot/dts/rk3288.dtsi +++ b/arch/arm/boot/dts/rk3288.dtsi @@ -942,7 +942,7 @@ status = "disabled"; }; - spdif: sound@ff88b0000 { + spdif: sound@ff8b0000 { compatible = "rockchip,rk3288-spdif", "rockchip,rk3066-spdif"; reg = <0x0 0xff8b0000 0x0 0x10000>; #sound-dai-cells = <0>; diff --git a/arch/arm/configs/imx_v6_v7_defconfig b/arch/arm/configs/imx_v6_v7_defconfig index 6dc6fed12af8..8d002c6e6cb3 100644 --- a/arch/arm/configs/imx_v6_v7_defconfig +++ b/arch/arm/configs/imx_v6_v7_defconfig @@ -76,7 +76,7 @@ CONFIG_RFKILL=y CONFIG_RFKILL_INPUT=y CONFIG_PCI=y CONFIG_PCI_MSI=y -CONFIG_PCI_IMX6=y +CONFIG_PCI_IMX6_HOST=y CONFIG_DEVTMPFS=y CONFIG_DEVTMPFS_MOUNT=y # CONFIG_STANDALONE is not set diff --git a/arch/arm64/boot/dts/amlogic/meson-g12-common.dtsi b/arch/arm64/boot/dts/amlogic/meson-g12-common.dtsi index 123a56f7f818..feb27a0ccfb4 100644 --- a/arch/arm64/boot/dts/amlogic/meson-g12-common.dtsi +++ b/arch/arm64/boot/dts/amlogic/meson-g12-common.dtsi @@ -1571,15 +1571,20 @@ dmc: bus@38000 { compatible = "simple-bus"; - reg = <0x0 0x38000 0x0 0x400>; #address-cells = <2>; #size-cells = <2>; - ranges = <0x0 0x0 0x0 0x38000 0x0 0x400>; + ranges = <0x0 0x0 0x0 0x38000 0x0 0x2000>; canvas: video-lut@48 { compatible = "amlogic,canvas"; reg = <0x0 0x48 0x0 0x14>; }; + + pmu: pmu@80 { + reg = <0x0 0x80 0x0 0x40>, + <0x0 0xc00 0x0 0x40>; + interrupts = <GIC_SPI 52 IRQ_TYPE_EDGE_RISING>; + }; }; usb2_phy1: phy@3a000 { @@ -1705,12 +1710,6 @@ }; }; - pmu: pmu@ff638000 { - reg = <0x0 0xff638000 0x0 0x100>, - <0x0 0xff638c00 0x0 0x100>; - interrupts = <GIC_SPI 52 IRQ_TYPE_EDGE_RISING>; - }; - aobus: bus@ff800000 { compatible = "simple-bus"; reg = <0x0 0xff800000 0x0 0x100000>; diff --git a/arch/arm64/boot/dts/freescale/imx8mm-evk.dtsi b/arch/arm64/boot/dts/freescale/imx8mm-evk.dtsi index d1a6390976a9..3f9dfd4d3884 100644 --- a/arch/arm64/boot/dts/freescale/imx8mm-evk.dtsi +++ b/arch/arm64/boot/dts/freescale/imx8mm-evk.dtsi @@ -194,7 +194,7 @@ rohm,reset-snvs-powered; #clock-cells = <0>; - clocks = <&osc_32k 0>; + clocks = <&osc_32k>; clock-output-names = "clk-32k-out"; regulators { diff --git a/arch/arm64/boot/dts/freescale/imx8mm-verdin.dtsi b/arch/arm64/boot/dts/freescale/imx8mm-verdin.dtsi index 88321b5b0693..6f0811587142 100644 --- a/arch/arm64/boot/dts/freescale/imx8mm-verdin.dtsi +++ b/arch/arm64/boot/dts/freescale/imx8mm-verdin.dtsi @@ -99,7 +99,7 @@ compatible = "regulator-fixed"; enable-active-high; gpio = <&gpio2 20 GPIO_ACTIVE_HIGH>; /* PMIC_EN_ETH */ - off-on-delay = <500000>; + off-on-delay-us = <500000>; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_reg_eth>; regulator-always-on; @@ -139,7 +139,7 @@ enable-active-high; /* Verdin SD_1_PWR_EN (SODIMM 76) */ gpio = <&gpio3 5 GPIO_ACTIVE_HIGH>; - off-on-delay = <100000>; + off-on-delay-us = <100000>; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_usdhc2_pwr_en>; regulator-max-microvolt = <3300000>; diff --git a/arch/arm64/boot/dts/freescale/imx8mp-verdin-dev.dtsi b/arch/arm64/boot/dts/freescale/imx8mp-verdin-dev.dtsi index 361426c0da0a..c29622529200 100644 --- a/arch/arm64/boot/dts/freescale/imx8mp-verdin-dev.dtsi +++ b/arch/arm64/boot/dts/freescale/imx8mp-verdin-dev.dtsi @@ -10,7 +10,7 @@ compatible = "regulator-fixed"; enable-active-high; gpio = <&gpio_expander_21 4 GPIO_ACTIVE_HIGH>; /* ETH_PWR_EN */ - off-on-delay = <500000>; + off-on-delay-us = <500000>; regulator-max-microvolt = <3300000>; regulator-min-microvolt = <3300000>; regulator-name = "+V3.3_ETH"; diff --git a/arch/arm64/boot/dts/freescale/imx8mp-verdin.dtsi b/arch/arm64/boot/dts/freescale/imx8mp-verdin.dtsi index 0dd6180a8e39..1608775da0ad 100644 --- a/arch/arm64/boot/dts/freescale/imx8mp-verdin.dtsi +++ b/arch/arm64/boot/dts/freescale/imx8mp-verdin.dtsi @@ -87,7 +87,7 @@ compatible = "regulator-fixed"; enable-active-high; gpio = <&gpio2 20 GPIO_ACTIVE_HIGH>; /* PMIC_EN_ETH */ - off-on-delay = <500000>; + off-on-delay-us = <500000>; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_reg_eth>; regulator-always-on; @@ -128,7 +128,7 @@ enable-active-high; /* Verdin SD_1_PWR_EN (SODIMM 76) */ gpio = <&gpio4 22 GPIO_ACTIVE_HIGH>; - off-on-delay = <100000>; + off-on-delay-us = <100000>; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_usdhc2_pwr_en>; regulator-max-microvolt = <3300000>; diff --git a/arch/arm64/boot/dts/freescale/imx8mp.dtsi b/arch/arm64/boot/dts/freescale/imx8mp.dtsi index 2dd60e3252f3..a237275ee017 100644 --- a/arch/arm64/boot/dts/freescale/imx8mp.dtsi +++ b/arch/arm64/boot/dts/freescale/imx8mp.dtsi @@ -1128,7 +1128,7 @@ lcdif2: display-controller@32e90000 { compatible = "fsl,imx8mp-lcdif"; - reg = <0x32e90000 0x238>; + reg = <0x32e90000 0x10000>; interrupts = <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>; clocks = <&clk IMX8MP_CLK_MEDIA_DISP2_PIX_ROOT>, <&clk IMX8MP_CLK_MEDIA_APB_ROOT>, diff --git a/arch/arm64/boot/dts/qcom/ipq8074-hk01.dts b/arch/arm64/boot/dts/qcom/ipq8074-hk01.dts index ca3f96646b90..5cf07caf4103 100644 --- a/arch/arm64/boot/dts/qcom/ipq8074-hk01.dts +++ b/arch/arm64/boot/dts/qcom/ipq8074-hk01.dts @@ -62,11 +62,11 @@ perst-gpios = <&tlmm 58 GPIO_ACTIVE_LOW>; }; -&pcie_phy0 { +&pcie_qmp0 { status = "okay"; }; -&pcie_phy1 { +&pcie_qmp1 { status = "okay"; }; diff --git a/arch/arm64/boot/dts/qcom/ipq8074-hk10.dtsi b/arch/arm64/boot/dts/qcom/ipq8074-hk10.dtsi index 651a231554e0..1b8379ba87f9 100644 --- a/arch/arm64/boot/dts/qcom/ipq8074-hk10.dtsi +++ b/arch/arm64/boot/dts/qcom/ipq8074-hk10.dtsi @@ -48,11 +48,11 @@ perst-gpios = <&tlmm 61 GPIO_ACTIVE_LOW>; }; -&pcie_phy0 { +&pcie_qmp0 { status = "okay"; }; -&pcie_phy1 { +&pcie_qmp1 { status = "okay"; }; diff --git a/arch/arm64/boot/dts/qcom/qrb5165-rb5.dts b/arch/arm64/boot/dts/qcom/qrb5165-rb5.dts index aa0a7bd7307c..dd924331b0ee 100644 --- a/arch/arm64/boot/dts/qcom/qrb5165-rb5.dts +++ b/arch/arm64/boot/dts/qcom/qrb5165-rb5.dts @@ -1012,7 +1012,7 @@ left_spkr: speaker@0,3 { compatible = "sdw10217211000"; reg = <0 3>; - powerdown-gpios = <&tlmm 130 GPIO_ACTIVE_HIGH>; + powerdown-gpios = <&tlmm 130 GPIO_ACTIVE_LOW>; #thermal-sensor-cells = <0>; sound-name-prefix = "SpkrLeft"; #sound-dai-cells = <0>; @@ -1021,7 +1021,7 @@ right_spkr: speaker@0,4 { compatible = "sdw10217211000"; reg = <0 4>; - powerdown-gpios = <&tlmm 130 GPIO_ACTIVE_HIGH>; + powerdown-gpios = <&tlmm 130 GPIO_ACTIVE_LOW>; #thermal-sensor-cells = <0>; sound-name-prefix = "SpkrRight"; #sound-dai-cells = <0>; diff --git a/arch/arm64/boot/dts/qcom/sc7280-herobrine.dtsi b/arch/arm64/boot/dts/qcom/sc7280-herobrine.dtsi index b6137816f2f3..313083ec1f39 100644 --- a/arch/arm64/boot/dts/qcom/sc7280-herobrine.dtsi +++ b/arch/arm64/boot/dts/qcom/sc7280-herobrine.dtsi @@ -464,7 +464,7 @@ ap_i2c_tpm: &i2c14 { &mdss_dp_out { data-lanes = <0 1>; - link-frequencies = /bits/ 64 <1620000000 2700000000 5400000000 8100000000>; + link-frequencies = /bits/ 64 <1620000000 2700000000 5400000000>; }; &mdss_mdp { diff --git a/arch/arm64/boot/dts/qcom/sc8280xp-pmics.dtsi b/arch/arm64/boot/dts/qcom/sc8280xp-pmics.dtsi index df7d28f7ae60..be446eba4fa7 100644 --- a/arch/arm64/boot/dts/qcom/sc8280xp-pmics.dtsi +++ b/arch/arm64/boot/dts/qcom/sc8280xp-pmics.dtsi @@ -59,8 +59,9 @@ #size-cells = <0>; pmk8280_pon: pon@1300 { - compatible = "qcom,pm8998-pon"; - reg = <0x1300>; + compatible = "qcom,pmk8350-pon"; + reg = <0x1300>, <0x800>; + reg-names = "hlos", "pbs"; pmk8280_pon_pwrkey: pwrkey { compatible = "qcom,pmk8350-pwrkey"; diff --git a/arch/arm64/boot/dts/qcom/sdm850-lenovo-yoga-c630.dts b/arch/arm64/boot/dts/qcom/sdm850-lenovo-yoga-c630.dts index 67d2a663ce75..5c688cb6a7ce 100644 --- a/arch/arm64/boot/dts/qcom/sdm850-lenovo-yoga-c630.dts +++ b/arch/arm64/boot/dts/qcom/sdm850-lenovo-yoga-c630.dts @@ -753,7 +753,7 @@ left_spkr: speaker@0,3 { compatible = "sdw10217211000"; reg = <0 3>; - powerdown-gpios = <&wcdgpio 1 GPIO_ACTIVE_HIGH>; + powerdown-gpios = <&wcdgpio 1 GPIO_ACTIVE_LOW>; #thermal-sensor-cells = <0>; sound-name-prefix = "SpkrLeft"; #sound-dai-cells = <0>; @@ -761,7 +761,7 @@ right_spkr: speaker@0,4 { compatible = "sdw10217211000"; - powerdown-gpios = <&wcdgpio 2 GPIO_ACTIVE_HIGH>; + powerdown-gpios = <&wcdgpio 2 GPIO_ACTIVE_LOW>; reg = <0 4>; #thermal-sensor-cells = <0>; sound-name-prefix = "SpkrRight"; diff --git a/arch/arm64/boot/dts/qcom/sdm850-samsung-w737.dts b/arch/arm64/boot/dts/qcom/sdm850-samsung-w737.dts index 9850140514ba..41f59e32af64 100644 --- a/arch/arm64/boot/dts/qcom/sdm850-samsung-w737.dts +++ b/arch/arm64/boot/dts/qcom/sdm850-samsung-w737.dts @@ -662,7 +662,7 @@ left_spkr: speaker@0,3 { compatible = "sdw10217211000"; reg = <0 3>; - powerdown-gpios = <&wcdgpio 1 GPIO_ACTIVE_HIGH>; + powerdown-gpios = <&wcdgpio 1 GPIO_ACTIVE_LOW>; #thermal-sensor-cells = <0>; sound-name-prefix = "SpkrLeft"; #sound-dai-cells = <0>; @@ -670,7 +670,7 @@ right_spkr: speaker@0,4 { compatible = "sdw10217211000"; - powerdown-gpios = <&wcdgpio 2 GPIO_ACTIVE_HIGH>; + powerdown-gpios = <&wcdgpio 2 GPIO_ACTIVE_LOW>; reg = <0 4>; #thermal-sensor-cells = <0>; sound-name-prefix = "SpkrRight"; diff --git a/arch/arm64/boot/dts/qcom/sm8250-mtp.dts b/arch/arm64/boot/dts/qcom/sm8250-mtp.dts index e54cdc8bc31f..4c9de236676d 100644 --- a/arch/arm64/boot/dts/qcom/sm8250-mtp.dts +++ b/arch/arm64/boot/dts/qcom/sm8250-mtp.dts @@ -764,7 +764,7 @@ left_spkr: speaker@0,3 { compatible = "sdw10217211000"; reg = <0 3>; - powerdown-gpios = <&tlmm 26 GPIO_ACTIVE_HIGH>; + powerdown-gpios = <&tlmm 26 GPIO_ACTIVE_LOW>; #thermal-sensor-cells = <0>; sound-name-prefix = "SpkrLeft"; #sound-dai-cells = <0>; @@ -773,7 +773,7 @@ right_spkr: speaker@0,4 { compatible = "sdw10217211000"; reg = <0 4>; - powerdown-gpios = <&tlmm 127 GPIO_ACTIVE_HIGH>; + powerdown-gpios = <&tlmm 127 GPIO_ACTIVE_LOW>; #thermal-sensor-cells = <0>; sound-name-prefix = "SpkrRight"; #sound-dai-cells = <0>; diff --git a/arch/arm64/boot/dts/rockchip/rk3326-anbernic-rg351m.dts b/arch/arm64/boot/dts/rockchip/rk3326-anbernic-rg351m.dts index 61b31688b469..ce318e05f0a6 100644 --- a/arch/arm64/boot/dts/rockchip/rk3326-anbernic-rg351m.dts +++ b/arch/arm64/boot/dts/rockchip/rk3326-anbernic-rg351m.dts @@ -24,6 +24,8 @@ &internal_display { compatible = "elida,kd35t133"; + iovcc-supply = <&vcc_lcd>; + vdd-supply = <&vcc_lcd>; }; &pwm0 { diff --git a/arch/arm64/boot/dts/rockchip/rk3326-odroid-go.dtsi b/arch/arm64/boot/dts/rockchip/rk3326-odroid-go.dtsi index 04eba432fb0e..80fc53c807a4 100644 --- a/arch/arm64/boot/dts/rockchip/rk3326-odroid-go.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3326-odroid-go.dtsi @@ -235,10 +235,8 @@ internal_display: panel@0 { reg = <0>; backlight = <&backlight>; - iovcc-supply = <&vcc_lcd>; reset-gpios = <&gpio3 RK_PC0 GPIO_ACTIVE_LOW>; rotation = <270>; - vdd-supply = <&vcc_lcd>; port { mipi_in_panel: endpoint { diff --git a/arch/arm64/boot/dts/rockchip/rk3326-odroid-go2-v11.dts b/arch/arm64/boot/dts/rockchip/rk3326-odroid-go2-v11.dts index 139c898e590e..d94ac81eb4e6 100644 --- a/arch/arm64/boot/dts/rockchip/rk3326-odroid-go2-v11.dts +++ b/arch/arm64/boot/dts/rockchip/rk3326-odroid-go2-v11.dts @@ -83,6 +83,8 @@ &internal_display { compatible = "elida,kd35t133"; + iovcc-supply = <&vcc_lcd>; + vdd-supply = <&vcc_lcd>; }; &rk817 { diff --git a/arch/arm64/boot/dts/rockchip/rk3326-odroid-go2.dts b/arch/arm64/boot/dts/rockchip/rk3326-odroid-go2.dts index 4702183b673c..aa6f5b12206b 100644 --- a/arch/arm64/boot/dts/rockchip/rk3326-odroid-go2.dts +++ b/arch/arm64/boot/dts/rockchip/rk3326-odroid-go2.dts @@ -59,6 +59,8 @@ &internal_display { compatible = "elida,kd35t133"; + iovcc-supply = <&vcc_lcd>; + vdd-supply = <&vcc_lcd>; }; &rk817_charger { diff --git a/arch/arm64/boot/dts/rockchip/rk3368-evb.dtsi b/arch/arm64/boot/dts/rockchip/rk3368-evb.dtsi index 083452c67711..e47d1398aeca 100644 --- a/arch/arm64/boot/dts/rockchip/rk3368-evb.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3368-evb.dtsi @@ -61,7 +61,6 @@ pinctrl-names = "default"; pinctrl-0 = <&bl_en>; pwms = <&pwm0 0 1000000 PWM_POLARITY_INVERTED>; - pwm-delay-us = <10000>; }; emmc_pwrseq: emmc-pwrseq { diff --git a/arch/arm64/boot/dts/rockchip/rk3399-gru-chromebook.dtsi b/arch/arm64/boot/dts/rockchip/rk3399-gru-chromebook.dtsi index ee6095baba4d..5c1929d41cc0 100644 --- a/arch/arm64/boot/dts/rockchip/rk3399-gru-chromebook.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3399-gru-chromebook.dtsi @@ -198,7 +198,6 @@ power-supply = <&pp3300_disp>; pinctrl-names = "default"; pinctrl-0 = <&bl_en>; - pwm-delay-us = <10000>; }; gpio_keys: gpio-keys { diff --git a/arch/arm64/boot/dts/rockchip/rk3399-gru-scarlet.dtsi b/arch/arm64/boot/dts/rockchip/rk3399-gru-scarlet.dtsi index a47d9f758611..c5e7de60c121 100644 --- a/arch/arm64/boot/dts/rockchip/rk3399-gru-scarlet.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3399-gru-scarlet.dtsi @@ -167,7 +167,6 @@ pinctrl-names = "default"; pinctrl-0 = <&bl_en>; pwms = <&pwm1 0 1000000 0>; - pwm-delay-us = <10000>; }; dmic: dmic { diff --git a/arch/arm64/boot/dts/rockchip/rk3399-pinebook-pro.dts b/arch/arm64/boot/dts/rockchip/rk3399-pinebook-pro.dts index 194e48c755f6..ddd45de97950 100644 --- a/arch/arm64/boot/dts/rockchip/rk3399-pinebook-pro.dts +++ b/arch/arm64/boot/dts/rockchip/rk3399-pinebook-pro.dts @@ -50,19 +50,9 @@ pinctrl-0 = <&panel_en_pin>; power-supply = <&vcc3v3_panel>; - ports { - #address-cells = <1>; - #size-cells = <0>; - - port@0 { - reg = <0>; - #address-cells = <1>; - #size-cells = <0>; - - panel_in_edp: endpoint@0 { - reg = <0>; - remote-endpoint = <&edp_out_panel>; - }; + port { + panel_in_edp: endpoint { + remote-endpoint = <&edp_out_panel>; }; }; }; @@ -943,7 +933,7 @@ disable-wp; pinctrl-names = "default"; pinctrl-0 = <&sdmmc_clk &sdmmc_cmd &sdmmc_bus4>; - sd-uhs-sdr104; + sd-uhs-sdr50; vmmc-supply = <&vcc3v0_sd>; vqmmc-supply = <&vcc_sdio>; status = "okay"; diff --git a/arch/arm64/boot/dts/rockchip/rk3399-rockpro64.dtsi b/arch/arm64/boot/dts/rockchip/rk3399-rockpro64.dtsi index 78157521e944..bca2b50e0a93 100644 --- a/arch/arm64/boot/dts/rockchip/rk3399-rockpro64.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3399-rockpro64.dtsi @@ -647,16 +647,10 @@ avdd-supply = <&avdd>; backlight = <&backlight>; dvdd-supply = <&vcc3v3_s0>; - ports { - #address-cells = <1>; - #size-cells = <0>; - port@0 { - reg = <0>; - - mipi_in_panel: endpoint { - remote-endpoint = <&mipi_out_panel>; - }; + port { + mipi_in_panel: endpoint { + remote-endpoint = <&mipi_out_panel>; }; }; }; diff --git a/arch/arm64/boot/dts/rockchip/rk3399.dtsi b/arch/arm64/boot/dts/rockchip/rk3399.dtsi index 1881b4b71f91..40e7c4a70055 100644 --- a/arch/arm64/boot/dts/rockchip/rk3399.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3399.dtsi @@ -552,7 +552,7 @@ <0x0 0xfff10000 0 0x10000>, /* GICH */ <0x0 0xfff20000 0 0x10000>; /* GICV */ interrupts = <GIC_PPI 9 IRQ_TYPE_LEVEL_HIGH 0>; - its: interrupt-controller@fee20000 { + its: msi-controller@fee20000 { compatible = "arm,gic-v3-its"; msi-controller; #msi-cells = <1>; diff --git a/arch/arm64/boot/dts/rockchip/rk3566-anbernic-rg353x.dtsi b/arch/arm64/boot/dts/rockchip/rk3566-anbernic-rg353x.dtsi index 65a80d1f6d91..9a0e217f069f 100644 --- a/arch/arm64/boot/dts/rockchip/rk3566-anbernic-rg353x.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3566-anbernic-rg353x.dtsi @@ -16,8 +16,10 @@ }; &cru { - assigned-clocks = <&cru PLL_GPLL>, <&pmucru PLL_PPLL>, <&cru PLL_VPLL>; - assigned-clock-rates = <1200000000>, <200000000>, <241500000>; + assigned-clocks = <&pmucru CLK_RTC_32K>, <&cru PLL_GPLL>, + <&pmucru PLL_PPLL>, <&cru PLL_VPLL>; + assigned-clock-rates = <32768>, <1200000000>, + <200000000>, <241500000>; }; &gpio_keys_control { diff --git a/arch/arm64/boot/dts/rockchip/rk3566-anbernic-rg503.dts b/arch/arm64/boot/dts/rockchip/rk3566-anbernic-rg503.dts index b4b2df821cba..c763c7f3b1b3 100644 --- a/arch/arm64/boot/dts/rockchip/rk3566-anbernic-rg503.dts +++ b/arch/arm64/boot/dts/rockchip/rk3566-anbernic-rg503.dts @@ -105,8 +105,10 @@ }; &cru { - assigned-clocks = <&cru PLL_GPLL>, <&pmucru PLL_PPLL>, <&cru PLL_VPLL>; - assigned-clock-rates = <1200000000>, <200000000>, <500000000>; + assigned-clocks = <&pmucru CLK_RTC_32K>, <&cru PLL_GPLL>, + <&pmucru PLL_PPLL>, <&cru PLL_VPLL>; + assigned-clock-rates = <32768>, <1200000000>, + <200000000>, <500000000>; }; &dsi_dphy0 { diff --git a/arch/arm64/boot/dts/rockchip/rk3566-soquartz.dtsi b/arch/arm64/boot/dts/rockchip/rk3566-soquartz.dtsi index ce7165d7f1a1..102e448bc026 100644 --- a/arch/arm64/boot/dts/rockchip/rk3566-soquartz.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3566-soquartz.dtsi @@ -598,7 +598,7 @@ non-removable; pinctrl-names = "default"; pinctrl-0 = <&sdmmc1_bus4 &sdmmc1_cmd &sdmmc1_clk>; - sd-uhs-sdr104; + sd-uhs-sdr50; vmmc-supply = <&vcc3v3_sys>; vqmmc-supply = <&vcc_1v8>; status = "okay"; diff --git a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi index 005cde61b4b2..a506948b5572 100644 --- a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi @@ -222,6 +222,7 @@ cache-size = <131072>; cache-line-size = <64>; cache-sets = <512>; + cache-level = <2>; next-level-cache = <&l3_cache>; }; @@ -230,6 +231,7 @@ cache-size = <131072>; cache-line-size = <64>; cache-sets = <512>; + cache-level = <2>; next-level-cache = <&l3_cache>; }; @@ -238,6 +240,7 @@ cache-size = <131072>; cache-line-size = <64>; cache-sets = <512>; + cache-level = <2>; next-level-cache = <&l3_cache>; }; @@ -246,6 +249,7 @@ cache-size = <131072>; cache-line-size = <64>; cache-sets = <512>; + cache-level = <2>; next-level-cache = <&l3_cache>; }; @@ -254,6 +258,7 @@ cache-size = <524288>; cache-line-size = <64>; cache-sets = <1024>; + cache-level = <2>; next-level-cache = <&l3_cache>; }; @@ -262,6 +267,7 @@ cache-size = <524288>; cache-line-size = <64>; cache-sets = <1024>; + cache-level = <2>; next-level-cache = <&l3_cache>; }; @@ -270,6 +276,7 @@ cache-size = <524288>; cache-line-size = <64>; cache-sets = <1024>; + cache-level = <2>; next-level-cache = <&l3_cache>; }; @@ -278,6 +285,7 @@ cache-size = <524288>; cache-line-size = <64>; cache-sets = <1024>; + cache-level = <2>; next-level-cache = <&l3_cache>; }; @@ -286,6 +294,7 @@ cache-size = <3145728>; cache-line-size = <64>; cache-sets = <4096>; + cache-level = <3>; }; }; diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index 7fd51257e0ed..3ddde336e6a5 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -447,6 +447,22 @@ config ARCH_IOREMAP protection support. However, you can enable LoongArch DMW-based ioremap() for better performance. +config ARCH_WRITECOMBINE + bool "Enable WriteCombine (WUC) for ioremap()" + help + LoongArch maintains cache coherency in hardware, but when paired + with LS7A chipsets the WUC attribute (Weak-ordered UnCached, which + is similar to WriteCombine) is out of the scope of cache coherency + machanism for PCIe devices (this is a PCIe protocol violation, which + may be fixed in newer chipsets). + + This means WUC can only used for write-only memory regions now, so + this option is disabled by default, making WUC silently fallback to + SUC for ioremap(). You can enable this option if the kernel is ensured + to run on hardware without this bug. + + You can override this setting via writecombine=on/off boot parameter. + config ARCH_STRICT_ALIGN bool "Enable -mstrict-align to prevent unaligned accesses" if EXPERT default y diff --git a/arch/loongarch/include/asm/acpi.h b/arch/loongarch/include/asm/acpi.h index 4198753aa1d0..976a810352c6 100644 --- a/arch/loongarch/include/asm/acpi.h +++ b/arch/loongarch/include/asm/acpi.h @@ -41,8 +41,11 @@ extern void loongarch_suspend_enter(void); static inline unsigned long acpi_get_wakeup_address(void) { +#ifdef CONFIG_SUSPEND extern void loongarch_wakeup_start(void); return (unsigned long)loongarch_wakeup_start; +#endif + return 0UL; } #endif /* _ASM_LOONGARCH_ACPI_H */ diff --git a/arch/loongarch/include/asm/addrspace.h b/arch/loongarch/include/asm/addrspace.h index 8fb699b4d40a..5c9c03bdf915 100644 --- a/arch/loongarch/include/asm/addrspace.h +++ b/arch/loongarch/include/asm/addrspace.h @@ -71,9 +71,9 @@ extern unsigned long vm_map_base; #define _ATYPE32_ int #define _ATYPE64_ __s64 #ifdef CONFIG_64BIT -#define _CONST64_(x) x ## L +#define _CONST64_(x) x ## UL #else -#define _CONST64_(x) x ## LL +#define _CONST64_(x) x ## ULL #endif #endif diff --git a/arch/loongarch/include/asm/bootinfo.h b/arch/loongarch/include/asm/bootinfo.h index 0051b526ac6d..c60796869b2b 100644 --- a/arch/loongarch/include/asm/bootinfo.h +++ b/arch/loongarch/include/asm/bootinfo.h @@ -13,7 +13,6 @@ const char *get_system_type(void); extern void init_environ(void); extern void memblock_init(void); extern void platform_init(void); -extern void plat_swiotlb_setup(void); extern int __init init_numa_memory(void); struct loongson_board_info { diff --git a/arch/loongarch/include/asm/cpu-features.h b/arch/loongarch/include/asm/cpu-features.h index b07974218393..f6177f133477 100644 --- a/arch/loongarch/include/asm/cpu-features.h +++ b/arch/loongarch/include/asm/cpu-features.h @@ -42,6 +42,7 @@ #define cpu_has_fpu cpu_opt(LOONGARCH_CPU_FPU) #define cpu_has_lsx cpu_opt(LOONGARCH_CPU_LSX) #define cpu_has_lasx cpu_opt(LOONGARCH_CPU_LASX) +#define cpu_has_crc32 cpu_opt(LOONGARCH_CPU_CRC32) #define cpu_has_complex cpu_opt(LOONGARCH_CPU_COMPLEX) #define cpu_has_crypto cpu_opt(LOONGARCH_CPU_CRYPTO) #define cpu_has_lvz cpu_opt(LOONGARCH_CPU_LVZ) diff --git a/arch/loongarch/include/asm/cpu.h b/arch/loongarch/include/asm/cpu.h index c3da91759472..88773d849e33 100644 --- a/arch/loongarch/include/asm/cpu.h +++ b/arch/loongarch/include/asm/cpu.h @@ -78,25 +78,26 @@ enum cpu_type_enum { #define CPU_FEATURE_FPU 3 /* CPU has FPU */ #define CPU_FEATURE_LSX 4 /* CPU has LSX (128-bit SIMD) */ #define CPU_FEATURE_LASX 5 /* CPU has LASX (256-bit SIMD) */ -#define CPU_FEATURE_COMPLEX 6 /* CPU has Complex instructions */ -#define CPU_FEATURE_CRYPTO 7 /* CPU has Crypto instructions */ -#define CPU_FEATURE_LVZ 8 /* CPU has Virtualization extension */ -#define CPU_FEATURE_LBT_X86 9 /* CPU has X86 Binary Translation */ -#define CPU_FEATURE_LBT_ARM 10 /* CPU has ARM Binary Translation */ -#define CPU_FEATURE_LBT_MIPS 11 /* CPU has MIPS Binary Translation */ -#define CPU_FEATURE_TLB 12 /* CPU has TLB */ -#define CPU_FEATURE_CSR 13 /* CPU has CSR */ -#define CPU_FEATURE_WATCH 14 /* CPU has watchpoint registers */ -#define CPU_FEATURE_VINT 15 /* CPU has vectored interrupts */ -#define CPU_FEATURE_CSRIPI 16 /* CPU has CSR-IPI */ -#define CPU_FEATURE_EXTIOI 17 /* CPU has EXT-IOI */ -#define CPU_FEATURE_PREFETCH 18 /* CPU has prefetch instructions */ -#define CPU_FEATURE_PMP 19 /* CPU has perfermance counter */ -#define CPU_FEATURE_SCALEFREQ 20 /* CPU supports cpufreq scaling */ -#define CPU_FEATURE_FLATMODE 21 /* CPU has flat mode */ -#define CPU_FEATURE_EIODECODE 22 /* CPU has EXTIOI interrupt pin decode mode */ -#define CPU_FEATURE_GUESTID 23 /* CPU has GuestID feature */ -#define CPU_FEATURE_HYPERVISOR 24 /* CPU has hypervisor (running in VM) */ +#define CPU_FEATURE_CRC32 6 /* CPU has CRC32 instructions */ +#define CPU_FEATURE_COMPLEX 7 /* CPU has Complex instructions */ +#define CPU_FEATURE_CRYPTO 8 /* CPU has Crypto instructions */ +#define CPU_FEATURE_LVZ 9 /* CPU has Virtualization extension */ +#define CPU_FEATURE_LBT_X86 10 /* CPU has X86 Binary Translation */ +#define CPU_FEATURE_LBT_ARM 11 /* CPU has ARM Binary Translation */ +#define CPU_FEATURE_LBT_MIPS 12 /* CPU has MIPS Binary Translation */ +#define CPU_FEATURE_TLB 13 /* CPU has TLB */ +#define CPU_FEATURE_CSR 14 /* CPU has CSR */ +#define CPU_FEATURE_WATCH 15 /* CPU has watchpoint registers */ +#define CPU_FEATURE_VINT 16 /* CPU has vectored interrupts */ +#define CPU_FEATURE_CSRIPI 17 /* CPU has CSR-IPI */ +#define CPU_FEATURE_EXTIOI 18 /* CPU has EXT-IOI */ +#define CPU_FEATURE_PREFETCH 19 /* CPU has prefetch instructions */ +#define CPU_FEATURE_PMP 20 /* CPU has perfermance counter */ +#define CPU_FEATURE_SCALEFREQ 21 /* CPU supports cpufreq scaling */ +#define CPU_FEATURE_FLATMODE 22 /* CPU has flat mode */ +#define CPU_FEATURE_EIODECODE 23 /* CPU has EXTIOI interrupt pin decode mode */ +#define CPU_FEATURE_GUESTID 24 /* CPU has GuestID feature */ +#define CPU_FEATURE_HYPERVISOR 25 /* CPU has hypervisor (running in VM) */ #define LOONGARCH_CPU_CPUCFG BIT_ULL(CPU_FEATURE_CPUCFG) #define LOONGARCH_CPU_LAM BIT_ULL(CPU_FEATURE_LAM) @@ -104,6 +105,7 @@ enum cpu_type_enum { #define LOONGARCH_CPU_FPU BIT_ULL(CPU_FEATURE_FPU) #define LOONGARCH_CPU_LSX BIT_ULL(CPU_FEATURE_LSX) #define LOONGARCH_CPU_LASX BIT_ULL(CPU_FEATURE_LASX) +#define LOONGARCH_CPU_CRC32 BIT_ULL(CPU_FEATURE_CRC32) #define LOONGARCH_CPU_COMPLEX BIT_ULL(CPU_FEATURE_COMPLEX) #define LOONGARCH_CPU_CRYPTO BIT_ULL(CPU_FEATURE_CRYPTO) #define LOONGARCH_CPU_LVZ BIT_ULL(CPU_FEATURE_LVZ) diff --git a/arch/loongarch/include/asm/io.h b/arch/loongarch/include/asm/io.h index 402a7d9e3a53..545e2708fbf7 100644 --- a/arch/loongarch/include/asm/io.h +++ b/arch/loongarch/include/asm/io.h @@ -54,8 +54,10 @@ static inline void __iomem *ioremap_prot(phys_addr_t offset, unsigned long size, * @offset: bus address of the memory * @size: size of the resource to map */ +extern pgprot_t pgprot_wc; + #define ioremap_wc(offset, size) \ - ioremap_prot((offset), (size), pgprot_val(PAGE_KERNEL_WUC)) + ioremap_prot((offset), (size), pgprot_val(pgprot_wc)) #define ioremap_cache(offset, size) \ ioremap_prot((offset), (size), pgprot_val(PAGE_KERNEL)) diff --git a/arch/loongarch/include/asm/loongarch.h b/arch/loongarch/include/asm/loongarch.h index 65b7dcdea16d..83da5d29e2d1 100644 --- a/arch/loongarch/include/asm/loongarch.h +++ b/arch/loongarch/include/asm/loongarch.h @@ -117,7 +117,7 @@ static inline u32 read_cpucfg(u32 reg) #define CPUCFG1_EP BIT(22) #define CPUCFG1_RPLV BIT(23) #define CPUCFG1_HUGEPG BIT(24) -#define CPUCFG1_IOCSRBRD BIT(25) +#define CPUCFG1_CRC32 BIT(25) #define CPUCFG1_MSGINT BIT(26) #define LOONGARCH_CPUCFG2 0x2 @@ -423,9 +423,9 @@ static __always_inline void iocsr_write64(u64 val, u32 reg) #define CSR_ASID_ASID_WIDTH 10 #define CSR_ASID_ASID (_ULCAST_(0x3ff) << CSR_ASID_ASID_SHIFT) -#define LOONGARCH_CSR_PGDL 0x19 /* Page table base address when VA[47] = 0 */ +#define LOONGARCH_CSR_PGDL 0x19 /* Page table base address when VA[VALEN-1] = 0 */ -#define LOONGARCH_CSR_PGDH 0x1a /* Page table base address when VA[47] = 1 */ +#define LOONGARCH_CSR_PGDH 0x1a /* Page table base address when VA[VALEN-1] = 1 */ #define LOONGARCH_CSR_PGD 0x1b /* Page table base */ diff --git a/arch/loongarch/include/asm/module.lds.h b/arch/loongarch/include/asm/module.lds.h index 438f09d4ccf4..88554f92e010 100644 --- a/arch/loongarch/include/asm/module.lds.h +++ b/arch/loongarch/include/asm/module.lds.h @@ -2,8 +2,8 @@ /* Copyright (C) 2020-2022 Loongson Technology Corporation Limited */ SECTIONS { . = ALIGN(4); - .got : { BYTE(0) } - .plt : { BYTE(0) } - .plt.idx : { BYTE(0) } - .ftrace_trampoline : { BYTE(0) } + .got 0 : { BYTE(0) } + .plt 0 : { BYTE(0) } + .plt.idx 0 : { BYTE(0) } + .ftrace_trampoline 0 : { BYTE(0) } } diff --git a/arch/loongarch/include/uapi/asm/ptrace.h b/arch/loongarch/include/uapi/asm/ptrace.h index cc48ed262021..82d811b5c6e9 100644 --- a/arch/loongarch/include/uapi/asm/ptrace.h +++ b/arch/loongarch/include/uapi/asm/ptrace.h @@ -47,11 +47,12 @@ struct user_fp_state { }; struct user_watch_state { - uint16_t dbg_info; + uint64_t dbg_info; struct { uint64_t addr; uint64_t mask; uint32_t ctrl; + uint32_t pad; } dbg_regs[8]; }; diff --git a/arch/loongarch/kernel/cpu-probe.c b/arch/loongarch/kernel/cpu-probe.c index 3a3fce2d7846..5adf0f736c6d 100644 --- a/arch/loongarch/kernel/cpu-probe.c +++ b/arch/loongarch/kernel/cpu-probe.c @@ -60,7 +60,7 @@ static inline void set_elf_platform(int cpu, const char *plat) /* MAP BASE */ unsigned long vm_map_base; -EXPORT_SYMBOL_GPL(vm_map_base); +EXPORT_SYMBOL(vm_map_base); static void cpu_probe_addrbits(struct cpuinfo_loongarch *c) { @@ -94,13 +94,18 @@ static void cpu_probe_common(struct cpuinfo_loongarch *c) c->options = LOONGARCH_CPU_CPUCFG | LOONGARCH_CPU_CSR | LOONGARCH_CPU_TLB | LOONGARCH_CPU_VINT | LOONGARCH_CPU_WATCH; - elf_hwcap = HWCAP_LOONGARCH_CPUCFG | HWCAP_LOONGARCH_CRC32; + elf_hwcap = HWCAP_LOONGARCH_CPUCFG; config = read_cpucfg(LOONGARCH_CPUCFG1); if (config & CPUCFG1_UAL) { c->options |= LOONGARCH_CPU_UAL; elf_hwcap |= HWCAP_LOONGARCH_UAL; } + if (config & CPUCFG1_CRC32) { + c->options |= LOONGARCH_CPU_CRC32; + elf_hwcap |= HWCAP_LOONGARCH_CRC32; + } + config = read_cpucfg(LOONGARCH_CPUCFG2); if (config & CPUCFG2_LAM) { diff --git a/arch/loongarch/kernel/proc.c b/arch/loongarch/kernel/proc.c index 5c67cc4fd56d..0d82907b5404 100644 --- a/arch/loongarch/kernel/proc.c +++ b/arch/loongarch/kernel/proc.c @@ -76,6 +76,7 @@ static int show_cpuinfo(struct seq_file *m, void *v) if (cpu_has_fpu) seq_printf(m, " fpu"); if (cpu_has_lsx) seq_printf(m, " lsx"); if (cpu_has_lasx) seq_printf(m, " lasx"); + if (cpu_has_crc32) seq_printf(m, " crc32"); if (cpu_has_complex) seq_printf(m, " complex"); if (cpu_has_crypto) seq_printf(m, " crypto"); if (cpu_has_lvz) seq_printf(m, " lvz"); diff --git a/arch/loongarch/kernel/ptrace.c b/arch/loongarch/kernel/ptrace.c index 06bceae7d104..5fcffb452367 100644 --- a/arch/loongarch/kernel/ptrace.c +++ b/arch/loongarch/kernel/ptrace.c @@ -391,10 +391,10 @@ static int ptrace_hbp_fill_attr_ctrl(unsigned int note_type, return 0; } -static int ptrace_hbp_get_resource_info(unsigned int note_type, u16 *info) +static int ptrace_hbp_get_resource_info(unsigned int note_type, u64 *info) { u8 num; - u16 reg = 0; + u64 reg = 0; switch (note_type) { case NT_LOONGARCH_HW_BREAK: @@ -524,15 +524,16 @@ static int ptrace_hbp_set_addr(unsigned int note_type, return modify_user_hw_breakpoint(bp, &attr); } -#define PTRACE_HBP_CTRL_SZ sizeof(u32) #define PTRACE_HBP_ADDR_SZ sizeof(u64) #define PTRACE_HBP_MASK_SZ sizeof(u64) +#define PTRACE_HBP_CTRL_SZ sizeof(u32) +#define PTRACE_HBP_PAD_SZ sizeof(u32) static int hw_break_get(struct task_struct *target, const struct user_regset *regset, struct membuf to) { - u16 info; + u64 info; u32 ctrl; u64 addr, mask; int ret, idx = 0; @@ -545,7 +546,7 @@ static int hw_break_get(struct task_struct *target, membuf_write(&to, &info, sizeof(info)); - /* (address, ctrl) registers */ + /* (address, mask, ctrl) registers */ while (to.left) { ret = ptrace_hbp_get_addr(note_type, target, idx, &addr); if (ret) @@ -562,6 +563,7 @@ static int hw_break_get(struct task_struct *target, membuf_store(&to, addr); membuf_store(&to, mask); membuf_store(&to, ctrl); + membuf_zero(&to, sizeof(u32)); idx++; } @@ -582,7 +584,7 @@ static int hw_break_set(struct task_struct *target, offset = offsetof(struct user_watch_state, dbg_regs); user_regset_copyin_ignore(&pos, &count, &kbuf, &ubuf, 0, offset); - /* (address, ctrl) registers */ + /* (address, mask, ctrl) registers */ limit = regset->n * regset->size; while (count && offset < limit) { if (count < PTRACE_HBP_ADDR_SZ) @@ -602,7 +604,7 @@ static int hw_break_set(struct task_struct *target, break; ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, &mask, - offset, offset + PTRACE_HBP_ADDR_SZ); + offset, offset + PTRACE_HBP_MASK_SZ); if (ret) return ret; @@ -611,8 +613,8 @@ static int hw_break_set(struct task_struct *target, return ret; offset += PTRACE_HBP_MASK_SZ; - ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, &mask, - offset, offset + PTRACE_HBP_MASK_SZ); + ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, &ctrl, + offset, offset + PTRACE_HBP_CTRL_SZ); if (ret) return ret; @@ -620,6 +622,11 @@ static int hw_break_set(struct task_struct *target, if (ret) return ret; offset += PTRACE_HBP_CTRL_SZ; + + user_regset_copyin_ignore(&pos, &count, &kbuf, &ubuf, + offset, offset + PTRACE_HBP_PAD_SZ); + offset += PTRACE_HBP_PAD_SZ; + idx++; } diff --git a/arch/loongarch/kernel/setup.c b/arch/loongarch/kernel/setup.c index bae84ccf6d36..4444b13418f0 100644 --- a/arch/loongarch/kernel/setup.c +++ b/arch/loongarch/kernel/setup.c @@ -160,6 +160,27 @@ static void __init smbios_parse(void) dmi_walk(find_tokens, NULL); } +#ifdef CONFIG_ARCH_WRITECOMBINE +pgprot_t pgprot_wc = PAGE_KERNEL_WUC; +#else +pgprot_t pgprot_wc = PAGE_KERNEL_SUC; +#endif + +EXPORT_SYMBOL(pgprot_wc); + +static int __init setup_writecombine(char *p) +{ + if (!strcmp(p, "on")) + pgprot_wc = PAGE_KERNEL_WUC; + else if (!strcmp(p, "off")) + pgprot_wc = PAGE_KERNEL_SUC; + else + pr_warn("Unknown writecombine setting \"%s\".\n", p); + + return 0; +} +early_param("writecombine", setup_writecombine); + static int usermem __initdata; static int __init early_parse_mem(char *p) @@ -368,8 +389,8 @@ static void __init arch_mem_init(char **cmdline_p) /* * In order to reduce the possibility of kernel panic when failed to * get IO TLB memory under CONFIG_SWIOTLB, it is better to allocate - * low memory as small as possible before plat_swiotlb_setup(), so - * make sparse_init() using top-down allocation. + * low memory as small as possible before swiotlb_init(), so make + * sparse_init() using top-down allocation. */ memblock_set_bottom_up(false); sparse_init(); diff --git a/arch/loongarch/kernel/stacktrace.c b/arch/loongarch/kernel/stacktrace.c index 3a690f96f00c..2463d2fea21f 100644 --- a/arch/loongarch/kernel/stacktrace.c +++ b/arch/loongarch/kernel/stacktrace.c @@ -30,7 +30,7 @@ void arch_stack_walk(stack_trace_consume_fn consume_entry, void *cookie, regs->regs[1] = 0; for (unwind_start(&state, task, regs); - !unwind_done(&state); unwind_next_frame(&state)) { + !unwind_done(&state) && !unwind_error(&state); unwind_next_frame(&state)) { addr = unwind_get_return_address(&state); if (!addr || !consume_entry(cookie, addr)) break; diff --git a/arch/loongarch/kernel/unwind.c b/arch/loongarch/kernel/unwind.c index a463d6961344..ba324ba76fa1 100644 --- a/arch/loongarch/kernel/unwind.c +++ b/arch/loongarch/kernel/unwind.c @@ -28,5 +28,6 @@ bool default_next_frame(struct unwind_state *state) } while (!get_stack_info(state->sp, state->task, info)); + state->error = true; return false; } diff --git a/arch/loongarch/kernel/unwind_prologue.c b/arch/loongarch/kernel/unwind_prologue.c index 9095fde8e55d..55afc27320e1 100644 --- a/arch/loongarch/kernel/unwind_prologue.c +++ b/arch/loongarch/kernel/unwind_prologue.c @@ -211,7 +211,7 @@ static bool next_frame(struct unwind_state *state) pc = regs->csr_era; if (user_mode(regs) || !__kernel_text_address(pc)) - return false; + goto out; state->first = true; state->pc = pc; @@ -226,6 +226,8 @@ static bool next_frame(struct unwind_state *state) } while (!get_stack_info(state->sp, state->task, info)); +out: + state->error = true; return false; } diff --git a/arch/loongarch/mm/init.c b/arch/loongarch/mm/init.c index e018aed34586..3b7d8129570b 100644 --- a/arch/loongarch/mm/init.c +++ b/arch/loongarch/mm/init.c @@ -41,7 +41,7 @@ * don't have to care about aliases on other CPUs. */ unsigned long empty_zero_page, zero_page_mask; -EXPORT_SYMBOL_GPL(empty_zero_page); +EXPORT_SYMBOL(empty_zero_page); EXPORT_SYMBOL(zero_page_mask); void setup_zero_pages(void) @@ -270,7 +270,7 @@ pud_t invalid_pud_table[PTRS_PER_PUD] __page_aligned_bss; #endif #ifndef __PAGETABLE_PMD_FOLDED pmd_t invalid_pmd_table[PTRS_PER_PMD] __page_aligned_bss; -EXPORT_SYMBOL_GPL(invalid_pmd_table); +EXPORT_SYMBOL(invalid_pmd_table); #endif pte_t invalid_pte_table[PTRS_PER_PTE] __page_aligned_bss; EXPORT_SYMBOL(invalid_pte_table); diff --git a/arch/loongarch/power/suspend_asm.S b/arch/loongarch/power/suspend_asm.S index 90da899c06a1..e2fc3b4e31f0 100644 --- a/arch/loongarch/power/suspend_asm.S +++ b/arch/loongarch/power/suspend_asm.S @@ -80,6 +80,10 @@ SYM_INNER_LABEL(loongarch_wakeup_start, SYM_L_GLOBAL) JUMP_VIRT_ADDR t0, t1 + /* Enable PG */ + li.w t0, 0xb0 # PLV=0, IE=0, PG=1 + csrwr t0, LOONGARCH_CSR_CRMD + la.pcrel t0, acpi_saved_sp ld.d sp, t0, 0 SETUP_WAKEUP diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c index b44ce71917d7..16cfe56be05b 100644 --- a/arch/powerpc/mm/numa.c +++ b/arch/powerpc/mm/numa.c @@ -366,6 +366,7 @@ void update_numa_distance(struct device_node *node) WARN(numa_distance_table[nid][nid] == -1, "NUMA distance details for node %d not provided\n", nid); } +EXPORT_SYMBOL_GPL(update_numa_distance); /* * ibm,numa-lookup-index-table= {N, domainid1, domainid2, ..... domainidN} diff --git a/arch/powerpc/platforms/pseries/papr_scm.c b/arch/powerpc/platforms/pseries/papr_scm.c index 2f8385523a13..1a53e048ceb7 100644 --- a/arch/powerpc/platforms/pseries/papr_scm.c +++ b/arch/powerpc/platforms/pseries/papr_scm.c @@ -1428,6 +1428,13 @@ static int papr_scm_probe(struct platform_device *pdev) return -ENODEV; } + /* + * open firmware platform device create won't update the NUMA + * distance table. For PAPR SCM devices we use numa_map_to_online_node() + * to find the nearest online NUMA node and that requires correct + * distance table information. + */ + update_numa_distance(dn); p = kzalloc(sizeof(*p), GFP_KERNEL); if (!p) diff --git a/arch/riscv/boot/dts/canaan/k210.dtsi b/arch/riscv/boot/dts/canaan/k210.dtsi index 07e2e2649604..f87c5164d9cf 100644 --- a/arch/riscv/boot/dts/canaan/k210.dtsi +++ b/arch/riscv/boot/dts/canaan/k210.dtsi @@ -259,7 +259,6 @@ <&sysclk K210_CLK_APB0>; clock-names = "ssi_clk", "pclk"; resets = <&sysrst K210_RST_SPI2>; - spi-max-frequency = <25000000>; }; i2s0: i2s@50250000 { diff --git a/arch/riscv/include/asm/fixmap.h b/arch/riscv/include/asm/fixmap.h index 5c3e7b97fcc6..0a55099bb734 100644 --- a/arch/riscv/include/asm/fixmap.h +++ b/arch/riscv/include/asm/fixmap.h @@ -22,6 +22,14 @@ */ enum fixed_addresses { FIX_HOLE, + /* + * The fdt fixmap mapping must be PMD aligned and will be mapped + * using PMD entries in fixmap_pmd in 64-bit and a PGD entry in 32-bit. + */ + FIX_FDT_END, + FIX_FDT = FIX_FDT_END + FIX_FDT_SIZE / PAGE_SIZE - 1, + + /* Below fixmaps will be mapped using fixmap_pte */ FIX_PTE, FIX_PMD, FIX_PUD, diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index ab05f892d317..f641837ccf31 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -87,9 +87,13 @@ #define FIXADDR_TOP PCI_IO_START #ifdef CONFIG_64BIT -#define FIXADDR_SIZE PMD_SIZE +#define MAX_FDT_SIZE PMD_SIZE +#define FIX_FDT_SIZE (MAX_FDT_SIZE + SZ_2M) +#define FIXADDR_SIZE (PMD_SIZE + FIX_FDT_SIZE) #else -#define FIXADDR_SIZE PGDIR_SIZE +#define MAX_FDT_SIZE PGDIR_SIZE +#define FIX_FDT_SIZE MAX_FDT_SIZE +#define FIXADDR_SIZE (PGDIR_SIZE + FIX_FDT_SIZE) #endif #define FIXADDR_START (FIXADDR_TOP - FIXADDR_SIZE) diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index 376d2827e736..a059b73f4ddb 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -278,12 +278,8 @@ void __init setup_arch(char **cmdline_p) #if IS_ENABLED(CONFIG_BUILTIN_DTB) unflatten_and_copy_device_tree(); #else - if (early_init_dt_verify(__va(XIP_FIXUP(dtb_early_pa)))) - unflatten_device_tree(); - else - pr_err("No DTB found in kernel mappings\n"); + unflatten_device_tree(); #endif - early_init_fdt_scan_reserved_mem(); misc_mem_init(); init_resources(); diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index bfb2afa4135f..dee66c9290cc 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -19,6 +19,7 @@ #include <asm/signal32.h> #include <asm/switch_to.h> #include <asm/csr.h> +#include <asm/cacheflush.h> extern u32 __user_rt_sigreturn[2]; @@ -181,6 +182,7 @@ static int setup_rt_frame(struct ksignal *ksig, sigset_t *set, { struct rt_sigframe __user *frame; long err = 0; + unsigned long __maybe_unused addr; frame = get_sigframe(ksig, regs, sizeof(*frame)); if (!access_ok(frame, sizeof(*frame))) @@ -209,7 +211,12 @@ static int setup_rt_frame(struct ksignal *ksig, sigset_t *set, if (copy_to_user(&frame->sigreturn_code, __user_rt_sigreturn, sizeof(frame->sigreturn_code))) return -EFAULT; - regs->ra = (unsigned long)&frame->sigreturn_code; + + addr = (unsigned long)&frame->sigreturn_code; + /* Make sure the two instructions are pushed to icache. */ + flush_icache_range(addr, addr + sizeof(frame->sigreturn_code)); + + regs->ra = addr; #endif /* CONFIG_MMU */ /* diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 478d6763a01a..0f14f4a8d179 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -57,7 +57,6 @@ unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)] EXPORT_SYMBOL(empty_zero_page); extern char _start[]; -#define DTB_EARLY_BASE_VA PGDIR_SIZE void *_dtb_early_va __initdata; uintptr_t _dtb_early_pa __initdata; @@ -236,31 +235,22 @@ static void __init setup_bootmem(void) set_max_mapnr(max_low_pfn - ARCH_PFN_OFFSET); reserve_initrd_mem(); + + /* + * No allocation should be done before reserving the memory as defined + * in the device tree, otherwise the allocation could end up in a + * reserved region. + */ + early_init_fdt_scan_reserved_mem(); + /* * If DTB is built in, no need to reserve its memblock. * Otherwise, do reserve it but avoid using * early_init_fdt_reserve_self() since __pa() does * not work for DTB pointers that are fixmap addresses */ - if (!IS_ENABLED(CONFIG_BUILTIN_DTB)) { - /* - * In case the DTB is not located in a memory region we won't - * be able to locate it later on via the linear mapping and - * get a segfault when accessing it via __va(dtb_early_pa). - * To avoid this situation copy DTB to a memory region. - * Note that memblock_phys_alloc will also reserve DTB region. - */ - if (!memblock_is_memory(dtb_early_pa)) { - size_t fdt_size = fdt_totalsize(dtb_early_va); - phys_addr_t new_dtb_early_pa = memblock_phys_alloc(fdt_size, PAGE_SIZE); - void *new_dtb_early_va = early_memremap(new_dtb_early_pa, fdt_size); - - memcpy(new_dtb_early_va, dtb_early_va, fdt_size); - early_memunmap(new_dtb_early_va, fdt_size); - _dtb_early_pa = new_dtb_early_pa; - } else - memblock_reserve(dtb_early_pa, fdt_totalsize(dtb_early_va)); - } + if (!IS_ENABLED(CONFIG_BUILTIN_DTB)) + memblock_reserve(dtb_early_pa, fdt_totalsize(dtb_early_va)); dma_contiguous_reserve(dma32_phys_limit); if (IS_ENABLED(CONFIG_64BIT)) @@ -279,9 +269,6 @@ pgd_t trampoline_pg_dir[PTRS_PER_PGD] __page_aligned_bss; static pte_t fixmap_pte[PTRS_PER_PTE] __page_aligned_bss; pgd_t early_pg_dir[PTRS_PER_PGD] __initdata __aligned(PAGE_SIZE); -static p4d_t __maybe_unused early_dtb_p4d[PTRS_PER_P4D] __initdata __aligned(PAGE_SIZE); -static pud_t __maybe_unused early_dtb_pud[PTRS_PER_PUD] __initdata __aligned(PAGE_SIZE); -static pmd_t __maybe_unused early_dtb_pmd[PTRS_PER_PMD] __initdata __aligned(PAGE_SIZE); #ifdef CONFIG_XIP_KERNEL #define pt_ops (*(struct pt_alloc_ops *)XIP_FIXUP(&pt_ops)) @@ -626,9 +613,6 @@ static void __init create_p4d_mapping(p4d_t *p4dp, #define trampoline_pgd_next (pgtable_l5_enabled ? \ (uintptr_t)trampoline_p4d : (pgtable_l4_enabled ? \ (uintptr_t)trampoline_pud : (uintptr_t)trampoline_pmd)) -#define early_dtb_pgd_next (pgtable_l5_enabled ? \ - (uintptr_t)early_dtb_p4d : (pgtable_l4_enabled ? \ - (uintptr_t)early_dtb_pud : (uintptr_t)early_dtb_pmd)) #else #define pgd_next_t pte_t #define alloc_pgd_next(__va) pt_ops.alloc_pte(__va) @@ -636,7 +620,6 @@ static void __init create_p4d_mapping(p4d_t *p4dp, #define create_pgd_next_mapping(__nextp, __va, __pa, __sz, __prot) \ create_pte_mapping(__nextp, __va, __pa, __sz, __prot) #define fixmap_pgd_next ((uintptr_t)fixmap_pte) -#define early_dtb_pgd_next ((uintptr_t)early_dtb_pmd) #define create_p4d_mapping(__pmdp, __va, __pa, __sz, __prot) do {} while(0) #define create_pud_mapping(__pmdp, __va, __pa, __sz, __prot) do {} while(0) #define create_pmd_mapping(__pmdp, __va, __pa, __sz, __prot) do {} while(0) @@ -860,32 +843,28 @@ static void __init create_kernel_page_table(pgd_t *pgdir, bool early) * this means 2 PMD entries whereas for 32-bit kernel, this is only 1 PGDIR * entry. */ -static void __init create_fdt_early_page_table(pgd_t *pgdir, uintptr_t dtb_pa) +static void __init create_fdt_early_page_table(pgd_t *pgdir, + uintptr_t fix_fdt_va, + uintptr_t dtb_pa) { -#ifndef CONFIG_BUILTIN_DTB uintptr_t pa = dtb_pa & ~(PMD_SIZE - 1); - create_pgd_mapping(early_pg_dir, DTB_EARLY_BASE_VA, - IS_ENABLED(CONFIG_64BIT) ? early_dtb_pgd_next : pa, - PGDIR_SIZE, - IS_ENABLED(CONFIG_64BIT) ? PAGE_TABLE : PAGE_KERNEL); - - if (pgtable_l5_enabled) - create_p4d_mapping(early_dtb_p4d, DTB_EARLY_BASE_VA, - (uintptr_t)early_dtb_pud, P4D_SIZE, PAGE_TABLE); - - if (pgtable_l4_enabled) - create_pud_mapping(early_dtb_pud, DTB_EARLY_BASE_VA, - (uintptr_t)early_dtb_pmd, PUD_SIZE, PAGE_TABLE); +#ifndef CONFIG_BUILTIN_DTB + /* Make sure the fdt fixmap address is always aligned on PMD size */ + BUILD_BUG_ON(FIX_FDT % (PMD_SIZE / PAGE_SIZE)); - if (IS_ENABLED(CONFIG_64BIT)) { - create_pmd_mapping(early_dtb_pmd, DTB_EARLY_BASE_VA, + /* In 32-bit only, the fdt lies in its own PGD */ + if (!IS_ENABLED(CONFIG_64BIT)) { + create_pgd_mapping(early_pg_dir, fix_fdt_va, + pa, MAX_FDT_SIZE, PAGE_KERNEL); + } else { + create_pmd_mapping(fixmap_pmd, fix_fdt_va, pa, PMD_SIZE, PAGE_KERNEL); - create_pmd_mapping(early_dtb_pmd, DTB_EARLY_BASE_VA + PMD_SIZE, + create_pmd_mapping(fixmap_pmd, fix_fdt_va + PMD_SIZE, pa + PMD_SIZE, PMD_SIZE, PAGE_KERNEL); } - dtb_early_va = (void *)DTB_EARLY_BASE_VA + (dtb_pa & (PMD_SIZE - 1)); + dtb_early_va = (void *)fix_fdt_va + (dtb_pa & (PMD_SIZE - 1)); #else /* * For 64-bit kernel, __va can't be used since it would return a linear @@ -1055,7 +1034,8 @@ asmlinkage void __init setup_vm(uintptr_t dtb_pa) create_kernel_page_table(early_pg_dir, true); /* Setup early mapping for FDT early scan */ - create_fdt_early_page_table(early_pg_dir, dtb_pa); + create_fdt_early_page_table(early_pg_dir, + __fix_to_virt(FIX_FDT), dtb_pa); /* * Bootime fixmap only can handle PMD_SIZE mapping. Thus, boot-ioremap @@ -1097,6 +1077,16 @@ static void __init setup_vm_final(void) u64 i; /* Setup swapper PGD for fixmap */ +#if !defined(CONFIG_64BIT) + /* + * In 32-bit, the device tree lies in a pgd entry, so it must be copied + * directly in swapper_pg_dir in addition to the pgd entry that points + * to fixmap_pte. + */ + unsigned long idx = pgd_index(__fix_to_virt(FIX_FDT)); + + set_pgd(&swapper_pg_dir[idx], early_pg_dir[idx]); +#endif create_pgd_mapping(swapper_pg_dir, FIXADDR_START, __pa_symbol(fixmap_pgd_next), PGDIR_SIZE, PAGE_TABLE); diff --git a/arch/riscv/purgatory/Makefile b/arch/riscv/purgatory/Makefile index d16bf715a586..5730797a6b40 100644 --- a/arch/riscv/purgatory/Makefile +++ b/arch/riscv/purgatory/Makefile @@ -84,12 +84,7 @@ CFLAGS_string.o += $(PURGATORY_CFLAGS) CFLAGS_REMOVE_ctype.o += $(PURGATORY_CFLAGS_REMOVE) CFLAGS_ctype.o += $(PURGATORY_CFLAGS) -AFLAGS_REMOVE_entry.o += -Wa,-gdwarf-2 -AFLAGS_REMOVE_memcpy.o += -Wa,-gdwarf-2 -AFLAGS_REMOVE_memset.o += -Wa,-gdwarf-2 -AFLAGS_REMOVE_strcmp.o += -Wa,-gdwarf-2 -AFLAGS_REMOVE_strlen.o += -Wa,-gdwarf-2 -AFLAGS_REMOVE_strncmp.o += -Wa,-gdwarf-2 +asflags-remove-y += $(foreach x, -g -gdwarf-4 -gdwarf-5, $(x) -Wa,$(x)) $(obj)/purgatory.ro: $(PURGATORY_OBJS) FORCE $(call if_changed,ld) diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c index d0846ba818ee..f95d7e401b96 100644 --- a/arch/s390/net/bpf_jit_comp.c +++ b/arch/s390/net/bpf_jit_comp.c @@ -539,7 +539,7 @@ static void bpf_jit_plt(void *plt, void *ret, void *target) { memcpy(plt, bpf_plt, BPF_PLT_SIZE); *(void **)((char *)plt + (bpf_plt_ret - bpf_plt)) = ret; - *(void **)((char *)plt + (bpf_plt_target - bpf_plt)) = target; + *(void **)((char *)plt + (bpf_plt_target - bpf_plt)) = target ?: ret; } /* @@ -2001,6 +2001,11 @@ bool bpf_jit_supports_kfunc_call(void) return true; } +bool bpf_jit_supports_far_kfunc_call(void) +{ + return true; +} + int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, void *old_addr, void *new_addr) { @@ -2010,7 +2015,9 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, } __packed insn; char expected_plt[BPF_PLT_SIZE]; char current_plt[BPF_PLT_SIZE]; + char new_plt[BPF_PLT_SIZE]; char *plt; + char *ret; int err; /* Verify the branch to be patched. */ @@ -2032,12 +2039,15 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, err = copy_from_kernel_nofault(current_plt, plt, BPF_PLT_SIZE); if (err < 0) return err; - bpf_jit_plt(expected_plt, (char *)ip + 6, old_addr); + ret = (char *)ip + 6; + bpf_jit_plt(expected_plt, ret, old_addr); if (memcmp(current_plt, expected_plt, BPF_PLT_SIZE)) return -EINVAL; /* Adjust the call address. */ + bpf_jit_plt(new_plt, ret, new_addr); s390_kernel_write(plt + (bpf_plt_target - bpf_plt), - &new_addr, sizeof(void *)); + new_plt + (bpf_plt_target - bpf_plt), + sizeof(void *)); } /* Adjust the mask of the branch. */ diff --git a/arch/x86/kernel/x86_init.c b/arch/x86/kernel/x86_init.c index ef80d361b463..10622cf2b30f 100644 --- a/arch/x86/kernel/x86_init.c +++ b/arch/x86/kernel/x86_init.c @@ -33,8 +33,8 @@ static int __init iommu_init_noop(void) { return 0; } static void iommu_shutdown_noop(void) { } bool __init bool_x86_init_noop(void) { return false; } void x86_op_int_noop(int cpu) { } -static __init int set_rtc_noop(const struct timespec64 *now) { return -EINVAL; } -static __init void get_rtc_noop(struct timespec64 *now) { } +static int set_rtc_noop(const struct timespec64 *now) { return -EINVAL; } +static void get_rtc_noop(struct timespec64 *now) { } static __initconst const struct of_device_id of_cmos_match[] = { { .compatible = "motorola,mc146818" }, diff --git a/arch/x86/purgatory/Makefile b/arch/x86/purgatory/Makefile index 17f09dc26381..82fec66d46d2 100644 --- a/arch/x86/purgatory/Makefile +++ b/arch/x86/purgatory/Makefile @@ -69,8 +69,7 @@ CFLAGS_sha256.o += $(PURGATORY_CFLAGS) CFLAGS_REMOVE_string.o += $(PURGATORY_CFLAGS_REMOVE) CFLAGS_string.o += $(PURGATORY_CFLAGS) -AFLAGS_REMOVE_setup-x86_$(BITS).o += -Wa,-gdwarf-2 -AFLAGS_REMOVE_entry64.o += -Wa,-gdwarf-2 +asflags-remove-y += $(foreach x, -g -gdwarf-4 -gdwarf-5, $(x) -Wa,$(x)) $(obj)/purgatory.ro: $(PURGATORY_OBJS) FORCE $(call if_changed,ld) diff --git a/drivers/acpi/resource.c b/drivers/acpi/resource.c index 7b4801ce62d6..e8492b3a393a 100644 --- a/drivers/acpi/resource.c +++ b/drivers/acpi/resource.c @@ -440,6 +440,13 @@ static const struct dmi_system_id asus_laptop[] = { }, }, { + .ident = "Asus ExpertBook B1502CBA", + .matches = { + DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."), + DMI_MATCH(DMI_BOARD_NAME, "B1502CBA"), + }, + }, + { .ident = "Asus ExpertBook B2402CBA", .matches = { DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."), diff --git a/drivers/acpi/x86/utils.c b/drivers/acpi/x86/utils.c index da5727069d85..ba420a28a4aa 100644 --- a/drivers/acpi/x86/utils.c +++ b/drivers/acpi/x86/utils.c @@ -213,6 +213,7 @@ bool acpi_device_override_status(struct acpi_device *adev, unsigned long long *s disk in the system. */ static const struct x86_cpu_id storage_d3_cpu_ids[] = { + X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 24, NULL), /* Picasso */ X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 96, NULL), /* Renoir */ X86_MATCH_VENDOR_FAM_MODEL(AMD, 23, 104, NULL), /* Lucienne */ X86_MATCH_VENDOR_FAM_MODEL(AMD, 25, 80, NULL), /* Cezanne */ diff --git a/drivers/bcma/main.c b/drivers/bcma/main.c index 5e438f74ee4c..7061d3ee836a 100644 --- a/drivers/bcma/main.c +++ b/drivers/bcma/main.c @@ -14,6 +14,7 @@ #include <linux/slab.h> #include <linux/of_address.h> #include <linux/of_irq.h> +#include <linux/of_device.h> #include <linux/of_platform.h> MODULE_DESCRIPTION("Broadcom's specific AMBA driver"); diff --git a/drivers/clk/clk-renesas-pcie.c b/drivers/clk/clk-renesas-pcie.c index f91f30560820..ff3a52d48479 100644 --- a/drivers/clk/clk-renesas-pcie.c +++ b/drivers/clk/clk-renesas-pcie.c @@ -143,8 +143,9 @@ static int rs9_regmap_i2c_read(void *context, static const struct regmap_config rs9_regmap_config = { .reg_bits = 8, .val_bits = 8, - .cache_type = REGCACHE_NONE, + .cache_type = REGCACHE_FLAT, .max_register = RS9_REG_BCP, + .num_reg_defaults_raw = 0x8, .rd_table = &rs9_readable_table, .wr_table = &rs9_writeable_table, .reg_write = rs9_regmap_i2c_write, diff --git a/drivers/clk/imx/clk-imx6ul.c b/drivers/clk/imx/clk-imx6ul.c index 2836adb817b7..e3696a88b5a3 100644 --- a/drivers/clk/imx/clk-imx6ul.c +++ b/drivers/clk/imx/clk-imx6ul.c @@ -95,14 +95,16 @@ static const struct clk_div_table video_div_table[] = { { } }; -static const char * enet1_ref_sels[] = { "enet1_ref_125m", "enet1_ref_pad", }; +static const char * enet1_ref_sels[] = { "enet1_ref_125m", "enet1_ref_pad", "dummy", "dummy"}; static const u32 enet1_ref_sels_table[] = { IMX6UL_GPR1_ENET1_TX_CLK_DIR, - IMX6UL_GPR1_ENET1_CLK_SEL }; + IMX6UL_GPR1_ENET1_CLK_SEL, 0, + IMX6UL_GPR1_ENET1_TX_CLK_DIR | IMX6UL_GPR1_ENET1_CLK_SEL }; static const u32 enet1_ref_sels_table_mask = IMX6UL_GPR1_ENET1_TX_CLK_DIR | IMX6UL_GPR1_ENET1_CLK_SEL; -static const char * enet2_ref_sels[] = { "enet2_ref_125m", "enet2_ref_pad", }; +static const char * enet2_ref_sels[] = { "enet2_ref_125m", "enet2_ref_pad", "dummy", "dummy"}; static const u32 enet2_ref_sels_table[] = { IMX6UL_GPR1_ENET2_TX_CLK_DIR, - IMX6UL_GPR1_ENET2_CLK_SEL }; + IMX6UL_GPR1_ENET2_CLK_SEL, 0, + IMX6UL_GPR1_ENET2_TX_CLK_DIR | IMX6UL_GPR1_ENET2_CLK_SEL }; static const u32 enet2_ref_sels_table_mask = IMX6UL_GPR1_ENET2_TX_CLK_DIR | IMX6UL_GPR1_ENET2_CLK_SEL; diff --git a/drivers/clk/sprd/common.c b/drivers/clk/sprd/common.c index ce81e4087a8f..2bfbab8db94b 100644 --- a/drivers/clk/sprd/common.c +++ b/drivers/clk/sprd/common.c @@ -17,7 +17,6 @@ static const struct regmap_config sprdclk_regmap_config = { .reg_bits = 32, .reg_stride = 4, .val_bits = 32, - .max_register = 0xffff, .fast_io = true, }; @@ -43,6 +42,8 @@ int sprd_clk_regmap_init(struct platform_device *pdev, struct device *dev = &pdev->dev; struct device_node *node = dev->of_node, *np; struct regmap *regmap; + struct resource *res; + struct regmap_config reg_config = sprdclk_regmap_config; if (of_find_property(node, "sprd,syscon", NULL)) { regmap = syscon_regmap_lookup_by_phandle(node, "sprd,syscon"); @@ -59,12 +60,14 @@ int sprd_clk_regmap_init(struct platform_device *pdev, return PTR_ERR(regmap); } } else { - base = devm_platform_ioremap_resource(pdev, 0); + base = devm_platform_get_and_ioremap_resource(pdev, 0, &res); if (IS_ERR(base)) return PTR_ERR(base); + reg_config.max_register = resource_size(res) - reg_config.reg_stride; + regmap = devm_regmap_init_mmio(&pdev->dev, base, - &sprdclk_regmap_config); + ®_config); if (IS_ERR(regmap)) { pr_err("failed to init regmap\n"); return PTR_ERR(regmap); diff --git a/drivers/cpufreq/amd-pstate.c b/drivers/cpufreq/amd-pstate.c index 73c7643b2697..8dd46fad151e 100644 --- a/drivers/cpufreq/amd-pstate.c +++ b/drivers/cpufreq/amd-pstate.c @@ -840,22 +840,20 @@ static int amd_pstate_update_status(const char *buf, size_t size) switch(mode_idx) { case AMD_PSTATE_DISABLE: - if (!current_pstate_driver) - return -EINVAL; - if (cppc_state == AMD_PSTATE_ACTIVE) - return -EBUSY; - cpufreq_unregister_driver(current_pstate_driver); - amd_pstate_driver_cleanup(); + if (current_pstate_driver) { + cpufreq_unregister_driver(current_pstate_driver); + amd_pstate_driver_cleanup(); + } break; case AMD_PSTATE_PASSIVE: if (current_pstate_driver) { if (current_pstate_driver == &amd_pstate_driver) return 0; cpufreq_unregister_driver(current_pstate_driver); - cppc_state = AMD_PSTATE_PASSIVE; - current_pstate_driver = &amd_pstate_driver; } + current_pstate_driver = &amd_pstate_driver; + cppc_state = AMD_PSTATE_PASSIVE; ret = cpufreq_register_driver(current_pstate_driver); break; case AMD_PSTATE_ACTIVE: @@ -863,10 +861,10 @@ static int amd_pstate_update_status(const char *buf, size_t size) if (current_pstate_driver == &amd_pstate_epp_driver) return 0; cpufreq_unregister_driver(current_pstate_driver); - current_pstate_driver = &amd_pstate_epp_driver; - cppc_state = AMD_PSTATE_ACTIVE; } + current_pstate_driver = &amd_pstate_epp_driver; + cppc_state = AMD_PSTATE_ACTIVE; ret = cpufreq_register_driver(current_pstate_driver); break; default: diff --git a/drivers/firmware/psci/psci.c b/drivers/firmware/psci/psci.c index 29619f49873a..d9629ff87861 100644 --- a/drivers/firmware/psci/psci.c +++ b/drivers/firmware/psci/psci.c @@ -167,7 +167,8 @@ int psci_set_osi_mode(bool enable) err = invoke_psci_fn(PSCI_1_0_FN_SET_SUSPEND_MODE, suspend_mode, 0, 0); if (err < 0) - pr_warn("failed to set %s mode: %d\n", enable ? "OSI" : "PC", err); + pr_info(FW_BUG "failed to set %s mode: %d\n", + enable ? "OSI" : "PC", err); return psci_to_linux_errno(err); } diff --git a/drivers/i2c/busses/i2c-mchp-pci1xxxx.c b/drivers/i2c/busses/i2c-mchp-pci1xxxx.c index 09af75921147..b21ffd6df927 100644 --- a/drivers/i2c/busses/i2c-mchp-pci1xxxx.c +++ b/drivers/i2c/busses/i2c-mchp-pci1xxxx.c @@ -48,9 +48,9 @@ * SR_HOLD_TIME_XK_TICKS field will indicate the number of ticks of the * baud clock required to program 'Hold Time' at X KHz. */ -#define SR_HOLD_TIME_100K_TICKS 133 -#define SR_HOLD_TIME_400K_TICKS 20 -#define SR_HOLD_TIME_1000K_TICKS 11 +#define SR_HOLD_TIME_100K_TICKS 150 +#define SR_HOLD_TIME_400K_TICKS 20 +#define SR_HOLD_TIME_1000K_TICKS 12 #define SMB_CORE_COMPLETION_REG_OFF3 (SMBUS_MAST_CORE_ADDR_BASE + 0x23) @@ -65,17 +65,17 @@ * the baud clock required to program 'fair idle delay' at X KHz. Fair idle * delay establishes the MCTP T(IDLE_DELAY) period. */ -#define FAIR_BUS_IDLE_MIN_100K_TICKS 969 -#define FAIR_BUS_IDLE_MIN_400K_TICKS 157 -#define FAIR_BUS_IDLE_MIN_1000K_TICKS 157 +#define FAIR_BUS_IDLE_MIN_100K_TICKS 992 +#define FAIR_BUS_IDLE_MIN_400K_TICKS 500 +#define FAIR_BUS_IDLE_MIN_1000K_TICKS 500 /* * FAIR_IDLE_DELAY_XK_TICKS field will indicate the number of ticks of the * baud clock required to satisfy the fairness protocol at X KHz. */ -#define FAIR_IDLE_DELAY_100K_TICKS 1000 -#define FAIR_IDLE_DELAY_400K_TICKS 500 -#define FAIR_IDLE_DELAY_1000K_TICKS 500 +#define FAIR_IDLE_DELAY_100K_TICKS 963 +#define FAIR_IDLE_DELAY_400K_TICKS 156 +#define FAIR_IDLE_DELAY_1000K_TICKS 156 #define SMB_IDLE_SCALING_100K \ ((FAIR_IDLE_DELAY_100K_TICKS << 16) | FAIR_BUS_IDLE_MIN_100K_TICKS) @@ -105,7 +105,7 @@ */ #define BUS_CLK_100K_LOW_PERIOD_TICKS 156 #define BUS_CLK_400K_LOW_PERIOD_TICKS 41 -#define BUS_CLK_1000K_LOW_PERIOD_TICKS 15 +#define BUS_CLK_1000K_LOW_PERIOD_TICKS 15 /* * BUS_CLK_XK_HIGH_PERIOD_TICKS field defines the number of I2C Baud Clock @@ -131,7 +131,7 @@ */ #define CLK_SYNC_100K 4 #define CLK_SYNC_400K 4 -#define CLK_SYNC_1000K 4 +#define CLK_SYNC_1000K 4 #define SMB_CORE_DATA_TIMING_REG_OFF (SMBUS_MAST_CORE_ADDR_BASE + 0x40) @@ -142,25 +142,25 @@ * determines the SCLK hold time following SDAT driven low during the first * START bit in a transfer. */ -#define FIRST_START_HOLD_100K_TICKS 22 -#define FIRST_START_HOLD_400K_TICKS 16 -#define FIRST_START_HOLD_1000K_TICKS 6 +#define FIRST_START_HOLD_100K_TICKS 23 +#define FIRST_START_HOLD_400K_TICKS 8 +#define FIRST_START_HOLD_1000K_TICKS 12 /* * STOP_SETUP_XK_TICKS will indicate the number of ticks of the baud clock * required to program 'STOP_SETUP' timer at X KHz. This timer determines the * SDAT setup time from the rising edge of SCLK for a STOP condition. */ -#define STOP_SETUP_100K_TICKS 157 +#define STOP_SETUP_100K_TICKS 150 #define STOP_SETUP_400K_TICKS 20 -#define STOP_SETUP_1000K_TICKS 12 +#define STOP_SETUP_1000K_TICKS 12 /* * RESTART_SETUP_XK_TICKS will indicate the number of ticks of the baud clock * required to program 'RESTART_SETUP' timer at X KHz. This timer determines the * SDAT setup time from the rising edge of SCLK for a repeated START condition. */ -#define RESTART_SETUP_100K_TICKS 157 +#define RESTART_SETUP_100K_TICKS 156 #define RESTART_SETUP_400K_TICKS 20 #define RESTART_SETUP_1000K_TICKS 12 @@ -169,7 +169,7 @@ * required to program 'DATA_HOLD' timer at X KHz. This timer determines the * SDAT hold time following SCLK driven low. */ -#define DATA_HOLD_100K_TICKS 2 +#define DATA_HOLD_100K_TICKS 12 #define DATA_HOLD_400K_TICKS 2 #define DATA_HOLD_1000K_TICKS 2 @@ -190,35 +190,35 @@ * Bus Idle Minimum time = BUS_IDLE_MIN[7:0] x Baud_Clock_Period x * (BUS_IDLE_MIN_XK_TICKS[7] ? 4,1) */ -#define BUS_IDLE_MIN_100K_TICKS 167UL -#define BUS_IDLE_MIN_400K_TICKS 139UL -#define BUS_IDLE_MIN_1000K_TICKS 133UL +#define BUS_IDLE_MIN_100K_TICKS 36UL +#define BUS_IDLE_MIN_400K_TICKS 10UL +#define BUS_IDLE_MIN_1000K_TICKS 4UL /* * CTRL_CUM_TIME_OUT_XK_TICKS defines SMBus Controller Cumulative Time-Out. * SMBus Controller Cumulative Time-Out duration = * CTRL_CUM_TIME_OUT_XK_TICKS[7:0] x Baud_Clock_Period x 2048 */ -#define CTRL_CUM_TIME_OUT_100K_TICKS 159 -#define CTRL_CUM_TIME_OUT_400K_TICKS 159 -#define CTRL_CUM_TIME_OUT_1000K_TICKS 159 +#define CTRL_CUM_TIME_OUT_100K_TICKS 76 +#define CTRL_CUM_TIME_OUT_400K_TICKS 76 +#define CTRL_CUM_TIME_OUT_1000K_TICKS 76 /* * TARGET_CUM_TIME_OUT_XK_TICKS defines SMBus Target Cumulative Time-Out duration. * SMBus Target Cumulative Time-Out duration = TARGET_CUM_TIME_OUT_XK_TICKS[7:0] x * Baud_Clock_Period x 4096 */ -#define TARGET_CUM_TIME_OUT_100K_TICKS 199 -#define TARGET_CUM_TIME_OUT_400K_TICKS 199 -#define TARGET_CUM_TIME_OUT_1000K_TICKS 199 +#define TARGET_CUM_TIME_OUT_100K_TICKS 95 +#define TARGET_CUM_TIME_OUT_400K_TICKS 95 +#define TARGET_CUM_TIME_OUT_1000K_TICKS 95 /* * CLOCK_HIGH_TIME_OUT_XK defines Clock High time out period. * Clock High time out period = CLOCK_HIGH_TIME_OUT_XK[7:0] x Baud_Clock_Period x 8 */ -#define CLOCK_HIGH_TIME_OUT_100K_TICKS 204 -#define CLOCK_HIGH_TIME_OUT_400K_TICKS 204 -#define CLOCK_HIGH_TIME_OUT_1000K_TICKS 204 +#define CLOCK_HIGH_TIME_OUT_100K_TICKS 97 +#define CLOCK_HIGH_TIME_OUT_400K_TICKS 97 +#define CLOCK_HIGH_TIME_OUT_1000K_TICKS 97 #define TO_SCALING_100K \ ((BUS_IDLE_MIN_100K_TICKS << 24) | (CTRL_CUM_TIME_OUT_100K_TICKS << 16) | \ diff --git a/drivers/i2c/busses/i2c-ocores.c b/drivers/i2c/busses/i2c-ocores.c index a0af027db04c..2e575856c5cd 100644 --- a/drivers/i2c/busses/i2c-ocores.c +++ b/drivers/i2c/busses/i2c-ocores.c @@ -342,18 +342,18 @@ static int ocores_poll_wait(struct ocores_i2c *i2c) * ocores_isr(), we just add our polling code around it. * * It can run in atomic context + * + * Return: 0 on success, -ETIMEDOUT on timeout */ -static void ocores_process_polling(struct ocores_i2c *i2c) +static int ocores_process_polling(struct ocores_i2c *i2c) { - while (1) { - irqreturn_t ret; - int err; + irqreturn_t ret; + int err = 0; + while (1) { err = ocores_poll_wait(i2c); - if (err) { - i2c->state = STATE_ERROR; + if (err) break; /* timeout */ - } ret = ocores_isr(-1, i2c); if (ret == IRQ_NONE) @@ -364,13 +364,15 @@ static void ocores_process_polling(struct ocores_i2c *i2c) break; } } + + return err; } static int ocores_xfer_core(struct ocores_i2c *i2c, struct i2c_msg *msgs, int num, bool polling) { - int ret; + int ret = 0; u8 ctrl; ctrl = oc_getreg(i2c, OCI2C_CONTROL); @@ -388,15 +390,16 @@ static int ocores_xfer_core(struct ocores_i2c *i2c, oc_setreg(i2c, OCI2C_CMD, OCI2C_CMD_START); if (polling) { - ocores_process_polling(i2c); + ret = ocores_process_polling(i2c); } else { - ret = wait_event_timeout(i2c->wait, - (i2c->state == STATE_ERROR) || - (i2c->state == STATE_DONE), HZ); - if (ret == 0) { - ocores_process_timeout(i2c); - return -ETIMEDOUT; - } + if (wait_event_timeout(i2c->wait, + (i2c->state == STATE_ERROR) || + (i2c->state == STATE_DONE), HZ) == 0) + ret = -ETIMEDOUT; + } + if (ret) { + ocores_process_timeout(i2c); + return ret; } return (i2c->state == STATE_DONE) ? num : -EIO; diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c index 308155937713..6b9563d4f23c 100644 --- a/drivers/infiniband/core/cma.c +++ b/drivers/infiniband/core/cma.c @@ -624,22 +624,11 @@ static inline unsigned short cma_family(struct rdma_id_private *id_priv) return id_priv->id.route.addr.src_addr.ss_family; } -static int cma_set_qkey(struct rdma_id_private *id_priv, u32 qkey) +static int cma_set_default_qkey(struct rdma_id_private *id_priv) { struct ib_sa_mcmember_rec rec; int ret = 0; - if (id_priv->qkey) { - if (qkey && id_priv->qkey != qkey) - return -EINVAL; - return 0; - } - - if (qkey) { - id_priv->qkey = qkey; - return 0; - } - switch (id_priv->id.ps) { case RDMA_PS_UDP: case RDMA_PS_IB: @@ -659,6 +648,16 @@ static int cma_set_qkey(struct rdma_id_private *id_priv, u32 qkey) return ret; } +static int cma_set_qkey(struct rdma_id_private *id_priv, u32 qkey) +{ + if (!qkey || + (id_priv->qkey && (id_priv->qkey != qkey))) + return -EINVAL; + + id_priv->qkey = qkey; + return 0; +} + static void cma_translate_ib(struct sockaddr_ib *sib, struct rdma_dev_addr *dev_addr) { dev_addr->dev_type = ARPHRD_INFINIBAND; @@ -1229,7 +1228,7 @@ static int cma_ib_init_qp_attr(struct rdma_id_private *id_priv, *qp_attr_mask = IB_QP_STATE | IB_QP_PKEY_INDEX | IB_QP_PORT; if (id_priv->id.qp_type == IB_QPT_UD) { - ret = cma_set_qkey(id_priv, 0); + ret = cma_set_default_qkey(id_priv); if (ret) return ret; @@ -4569,7 +4568,10 @@ static int cma_send_sidr_rep(struct rdma_id_private *id_priv, memset(&rep, 0, sizeof rep); rep.status = status; if (status == IB_SIDR_SUCCESS) { - ret = cma_set_qkey(id_priv, qkey); + if (qkey) + ret = cma_set_qkey(id_priv, qkey); + else + ret = cma_set_default_qkey(id_priv); if (ret) return ret; rep.qp_num = id_priv->qp_num; @@ -4774,9 +4776,7 @@ static void cma_make_mc_event(int status, struct rdma_id_private *id_priv, enum ib_gid_type gid_type; struct net_device *ndev; - if (!status) - status = cma_set_qkey(id_priv, be32_to_cpu(multicast->rec.qkey)); - else + if (status) pr_debug_ratelimited("RDMA CM: MULTICAST_ERROR: failed to join multicast. status %d\n", status); @@ -4804,7 +4804,7 @@ static void cma_make_mc_event(int status, struct rdma_id_private *id_priv, } event->param.ud.qp_num = 0xFFFFFF; - event->param.ud.qkey = be32_to_cpu(multicast->rec.qkey); + event->param.ud.qkey = id_priv->qkey; out: if (ndev) @@ -4823,8 +4823,11 @@ static int cma_ib_mc_handler(int status, struct ib_sa_multicast *multicast) READ_ONCE(id_priv->state) == RDMA_CM_DESTROYING) goto out; - cma_make_mc_event(status, id_priv, multicast, &event, mc); - ret = cma_cm_event_handler(id_priv, &event); + ret = cma_set_qkey(id_priv, be32_to_cpu(multicast->rec.qkey)); + if (!ret) { + cma_make_mc_event(status, id_priv, multicast, &event, mc); + ret = cma_cm_event_handler(id_priv, &event); + } rdma_destroy_ah_attr(&event.param.ud.ah_attr); WARN_ON(ret); @@ -4877,9 +4880,11 @@ static int cma_join_ib_multicast(struct rdma_id_private *id_priv, if (ret) return ret; - ret = cma_set_qkey(id_priv, 0); - if (ret) - return ret; + if (!id_priv->qkey) { + ret = cma_set_default_qkey(id_priv); + if (ret) + return ret; + } cma_set_mgid(id_priv, (struct sockaddr *) &mc->addr, &rec.mgid); rec.qkey = cpu_to_be32(id_priv->qkey); @@ -4956,9 +4961,6 @@ static int cma_iboe_join_multicast(struct rdma_id_private *id_priv, cma_iboe_set_mgid(addr, &ib.rec.mgid, gid_type); ib.rec.pkey = cpu_to_be16(0xffff); - if (id_priv->id.ps == RDMA_PS_UDP) - ib.rec.qkey = cpu_to_be32(RDMA_UDP_QKEY); - if (dev_addr->bound_dev_if) ndev = dev_get_by_index(dev_addr->net, dev_addr->bound_dev_if); if (!ndev) @@ -4984,6 +4986,9 @@ static int cma_iboe_join_multicast(struct rdma_id_private *id_priv, if (err || !ib.rec.mtu) return err ?: -EINVAL; + if (!id_priv->qkey) + cma_set_default_qkey(id_priv); + rdma_ip2gid((struct sockaddr *)&id_priv->id.route.addr.src_addr, &ib.rec.port_gid); INIT_WORK(&mc->iboe_join.work, cma_iboe_join_work_handler); @@ -5009,6 +5014,9 @@ int rdma_join_multicast(struct rdma_cm_id *id, struct sockaddr *addr, READ_ONCE(id_priv->state) != RDMA_CM_ADDR_RESOLVED)) return -EINVAL; + if (id_priv->id.qp_type != IB_QPT_UD) + return -EINVAL; + mc = kzalloc(sizeof(*mc), GFP_KERNEL); if (!mc) return -ENOMEM; diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c index 11b1c1603aeb..b99b3cc283b6 100644 --- a/drivers/infiniband/core/verbs.c +++ b/drivers/infiniband/core/verbs.c @@ -532,6 +532,8 @@ static struct ib_ah *_rdma_create_ah(struct ib_pd *pd, else ret = device->ops.create_ah(ah, &init_attr, NULL); if (ret) { + if (ah->sgid_attr) + rdma_put_gid_attr(ah->sgid_attr); kfree(ah); return ERR_PTR(ret); } diff --git a/drivers/infiniband/hw/erdma/erdma_cq.c b/drivers/infiniband/hw/erdma/erdma_cq.c index cabd8678b355..7bc354273d4e 100644 --- a/drivers/infiniband/hw/erdma/erdma_cq.c +++ b/drivers/infiniband/hw/erdma/erdma_cq.c @@ -65,7 +65,7 @@ static const enum ib_wc_opcode wc_mapping_table[ERDMA_NUM_OPCODES] = { [ERDMA_OP_LOCAL_INV] = IB_WC_LOCAL_INV, [ERDMA_OP_READ_WITH_INV] = IB_WC_RDMA_READ, [ERDMA_OP_ATOMIC_CAS] = IB_WC_COMP_SWAP, - [ERDMA_OP_ATOMIC_FAD] = IB_WC_FETCH_ADD, + [ERDMA_OP_ATOMIC_FAA] = IB_WC_FETCH_ADD, }; static const struct { diff --git a/drivers/infiniband/hw/erdma/erdma_hw.h b/drivers/infiniband/hw/erdma/erdma_hw.h index 4c38d99c73f1..37ad1bb1917c 100644 --- a/drivers/infiniband/hw/erdma/erdma_hw.h +++ b/drivers/infiniband/hw/erdma/erdma_hw.h @@ -441,7 +441,7 @@ struct erdma_reg_mr_sqe { }; /* EQ related. */ -#define ERDMA_DEFAULT_EQ_DEPTH 256 +#define ERDMA_DEFAULT_EQ_DEPTH 4096 /* ceqe */ #define ERDMA_CEQE_HDR_DB_MASK BIT_ULL(63) @@ -491,7 +491,7 @@ enum erdma_opcode { ERDMA_OP_LOCAL_INV = 15, ERDMA_OP_READ_WITH_INV = 16, ERDMA_OP_ATOMIC_CAS = 17, - ERDMA_OP_ATOMIC_FAD = 18, + ERDMA_OP_ATOMIC_FAA = 18, ERDMA_NUM_OPCODES = 19, ERDMA_OP_INVALID = ERDMA_NUM_OPCODES + 1 }; diff --git a/drivers/infiniband/hw/erdma/erdma_main.c b/drivers/infiniband/hw/erdma/erdma_main.c index 5dc31e5df5cb..4a29a53a6652 100644 --- a/drivers/infiniband/hw/erdma/erdma_main.c +++ b/drivers/infiniband/hw/erdma/erdma_main.c @@ -56,7 +56,7 @@ done: static int erdma_enum_and_get_netdev(struct erdma_dev *dev) { struct net_device *netdev; - int ret = -ENODEV; + int ret = -EPROBE_DEFER; /* Already binded to a net_device, so we skip. */ if (dev->netdev) diff --git a/drivers/infiniband/hw/erdma/erdma_qp.c b/drivers/infiniband/hw/erdma/erdma_qp.c index d088d6bef431..44923c51a01b 100644 --- a/drivers/infiniband/hw/erdma/erdma_qp.c +++ b/drivers/infiniband/hw/erdma/erdma_qp.c @@ -405,7 +405,7 @@ static int erdma_push_one_sqe(struct erdma_qp *qp, u16 *pi, FIELD_PREP(ERDMA_SQE_MR_MTT_CNT_MASK, mr->mem.mtt_nents); - if (mr->mem.mtt_nents < ERDMA_MAX_INLINE_MTT_ENTRIES) { + if (mr->mem.mtt_nents <= ERDMA_MAX_INLINE_MTT_ENTRIES) { attrs |= FIELD_PREP(ERDMA_SQE_MR_MTT_TYPE_MASK, 0); /* Copy SGLs to SQE content to accelerate */ memcpy(get_queue_entry(qp->kern_qp.sq_buf, idx + 1, @@ -439,7 +439,7 @@ static int erdma_push_one_sqe(struct erdma_qp *qp, u16 *pi, cpu_to_le64(atomic_wr(send_wr)->compare_add); } else { wqe_hdr |= FIELD_PREP(ERDMA_SQE_HDR_OPCODE_MASK, - ERDMA_OP_ATOMIC_FAD); + ERDMA_OP_ATOMIC_FAA); atomic_sqe->fetchadd_swap_data = cpu_to_le64(atomic_wr(send_wr)->compare_add); } diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.h b/drivers/infiniband/hw/erdma/erdma_verbs.h index e0a993bc032a..131cf5f40982 100644 --- a/drivers/infiniband/hw/erdma/erdma_verbs.h +++ b/drivers/infiniband/hw/erdma/erdma_verbs.h @@ -11,7 +11,7 @@ /* RDMA Capability. */ #define ERDMA_MAX_PD (128 * 1024) -#define ERDMA_MAX_SEND_WR 4096 +#define ERDMA_MAX_SEND_WR 8192 #define ERDMA_MAX_ORD 128 #define ERDMA_MAX_IRD 128 #define ERDMA_MAX_SGE_RD 1 diff --git a/drivers/infiniband/hw/irdma/cm.c b/drivers/infiniband/hw/irdma/cm.c index 195aa9ea18b6..8817864154af 100644 --- a/drivers/infiniband/hw/irdma/cm.c +++ b/drivers/infiniband/hw/irdma/cm.c @@ -1458,13 +1458,15 @@ static int irdma_send_fin(struct irdma_cm_node *cm_node) * irdma_find_listener - find a cm node listening on this addr-port pair * @cm_core: cm's core * @dst_addr: listener ip addr + * @ipv4: flag indicating IPv4 when true * @dst_port: listener tcp port num * @vlan_id: virtual LAN ID * @listener_state: state to match with listen node's */ static struct irdma_cm_listener * -irdma_find_listener(struct irdma_cm_core *cm_core, u32 *dst_addr, u16 dst_port, - u16 vlan_id, enum irdma_cm_listener_state listener_state) +irdma_find_listener(struct irdma_cm_core *cm_core, u32 *dst_addr, bool ipv4, + u16 dst_port, u16 vlan_id, + enum irdma_cm_listener_state listener_state) { struct irdma_cm_listener *listen_node; static const u32 ip_zero[4] = { 0, 0, 0, 0 }; @@ -1477,7 +1479,7 @@ irdma_find_listener(struct irdma_cm_core *cm_core, u32 *dst_addr, u16 dst_port, list_for_each_entry (listen_node, &cm_core->listen_list, list) { memcpy(listen_addr, listen_node->loc_addr, sizeof(listen_addr)); listen_port = listen_node->loc_port; - if (listen_port != dst_port || + if (listen_node->ipv4 != ipv4 || listen_port != dst_port || !(listener_state & listen_node->listener_state)) continue; /* compare node pair, return node handle if a match */ @@ -2902,9 +2904,10 @@ irdma_make_listen_node(struct irdma_cm_core *cm_core, unsigned long flags; /* cannot have multiple matching listeners */ - listener = irdma_find_listener(cm_core, cm_info->loc_addr, - cm_info->loc_port, cm_info->vlan_id, - IRDMA_CM_LISTENER_EITHER_STATE); + listener = + irdma_find_listener(cm_core, cm_info->loc_addr, cm_info->ipv4, + cm_info->loc_port, cm_info->vlan_id, + IRDMA_CM_LISTENER_EITHER_STATE); if (listener && listener->listener_state == IRDMA_CM_LISTENER_ACTIVE_STATE) { refcount_dec(&listener->refcnt); @@ -3153,6 +3156,7 @@ void irdma_receive_ilq(struct irdma_sc_vsi *vsi, struct irdma_puda_buf *rbuf) listener = irdma_find_listener(cm_core, cm_info.loc_addr, + cm_info.ipv4, cm_info.loc_port, cm_info.vlan_id, IRDMA_CM_LISTENER_ACTIVE_STATE); diff --git a/drivers/infiniband/hw/irdma/cm.h b/drivers/infiniband/hw/irdma/cm.h index 19c284975fc7..7feadb3e1eda 100644 --- a/drivers/infiniband/hw/irdma/cm.h +++ b/drivers/infiniband/hw/irdma/cm.h @@ -41,7 +41,7 @@ #define TCP_OPTIONS_PADDING 3 #define IRDMA_DEFAULT_RETRYS 64 -#define IRDMA_DEFAULT_RETRANS 8 +#define IRDMA_DEFAULT_RETRANS 32 #define IRDMA_DEFAULT_TTL 0x40 #define IRDMA_DEFAULT_RTT_VAR 6 #define IRDMA_DEFAULT_SS_THRESH 0x3fffffff diff --git a/drivers/infiniband/hw/irdma/hw.c b/drivers/infiniband/hw/irdma/hw.c index 2e1e2bad0401..43dfa4761f06 100644 --- a/drivers/infiniband/hw/irdma/hw.c +++ b/drivers/infiniband/hw/irdma/hw.c @@ -41,6 +41,7 @@ static enum irdma_hmc_rsrc_type iw_hmc_obj_types[] = { IRDMA_HMC_IW_XFFL, IRDMA_HMC_IW_Q1, IRDMA_HMC_IW_Q1FL, + IRDMA_HMC_IW_PBLE, IRDMA_HMC_IW_TIMER, IRDMA_HMC_IW_FSIMC, IRDMA_HMC_IW_FSIAV, @@ -827,6 +828,8 @@ static int irdma_create_hmc_objs(struct irdma_pci_f *rf, bool privileged, info.entry_type = rf->sd_type; for (i = 0; i < IW_HMC_OBJ_TYPE_NUM; i++) { + if (iw_hmc_obj_types[i] == IRDMA_HMC_IW_PBLE) + continue; if (dev->hmc_info->hmc_obj[iw_hmc_obj_types[i]].cnt) { info.rsrc_type = iw_hmc_obj_types[i]; info.count = dev->hmc_info->hmc_obj[info.rsrc_type].cnt; diff --git a/drivers/infiniband/hw/irdma/utils.c b/drivers/infiniband/hw/irdma/utils.c index 445e69e86409..7887230c867b 100644 --- a/drivers/infiniband/hw/irdma/utils.c +++ b/drivers/infiniband/hw/irdma/utils.c @@ -2595,7 +2595,10 @@ void irdma_generate_flush_completions(struct irdma_qp *iwqp) /* remove the SQ WR by moving SQ tail*/ IRDMA_RING_SET_TAIL(*sq_ring, sq_ring->tail + qp->sq_wrtrk_array[sq_ring->tail].quanta); - + if (cmpl->cpi.op_type == IRDMAQP_OP_NOP) { + kfree(cmpl); + continue; + } ibdev_dbg(iwqp->iwscq->ibcq.device, "DEV: %s: adding wr_id = 0x%llx SQ Completion to list qp_id=%d\n", __func__, cmpl->cpi.wr_id, qp->qp_id); diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c index 5b988db66b8f..5d45de223c43 100644 --- a/drivers/infiniband/hw/mlx5/main.c +++ b/drivers/infiniband/hw/mlx5/main.c @@ -442,6 +442,10 @@ static int translate_eth_ext_proto_oper(u32 eth_proto_oper, u16 *active_speed, *active_width = IB_WIDTH_2X; *active_speed = IB_SPEED_NDR; break; + case MLX5E_PROT_MASK(MLX5E_400GAUI_8): + *active_width = IB_WIDTH_8X; + *active_speed = IB_SPEED_HDR; + break; case MLX5E_PROT_MASK(MLX5E_400GAUI_4_400GBASE_CR4_KR4): *active_width = IB_WIDTH_4X; *active_speed = IB_SPEED_NDR; diff --git a/drivers/memstick/core/memstick.c b/drivers/memstick/core/memstick.c index bf7667845459..bbfaf6536903 100644 --- a/drivers/memstick/core/memstick.c +++ b/drivers/memstick/core/memstick.c @@ -410,6 +410,7 @@ static struct memstick_dev *memstick_alloc_card(struct memstick_host *host) return card; err_out: host->card = old_card; + kfree_const(card->dev.kobj.name); kfree(card); return NULL; } @@ -468,8 +469,10 @@ static void memstick_check(struct work_struct *work) put_device(&card->dev); host->card = NULL; } - } else + } else { + kfree_const(card->dev.kobj.name); kfree(card); + } } out_power_off: diff --git a/drivers/mmc/host/sdhci_am654.c b/drivers/mmc/host/sdhci_am654.c index 89953093e20c..672d37ea98d0 100644 --- a/drivers/mmc/host/sdhci_am654.c +++ b/drivers/mmc/host/sdhci_am654.c @@ -351,8 +351,6 @@ static void sdhci_am654_write_b(struct sdhci_host *host, u8 val, int reg) */ case MMC_TIMING_SD_HS: case MMC_TIMING_MMC_HS: - case MMC_TIMING_UHS_SDR12: - case MMC_TIMING_UHS_SDR25: val &= ~SDHCI_CTRL_HISPD; } } diff --git a/drivers/mtd/ubi/build.c b/drivers/mtd/ubi/build.c index 0904eb40c95f..ad025b2ee417 100644 --- a/drivers/mtd/ubi/build.c +++ b/drivers/mtd/ubi/build.c @@ -666,12 +666,6 @@ static int io_init(struct ubi_device *ubi, int max_beb_per1024) ubi->ec_hdr_alsize = ALIGN(UBI_EC_HDR_SIZE, ubi->hdrs_min_io_size); ubi->vid_hdr_alsize = ALIGN(UBI_VID_HDR_SIZE, ubi->hdrs_min_io_size); - if (ubi->vid_hdr_offset && ((ubi->vid_hdr_offset + UBI_VID_HDR_SIZE) > - ubi->vid_hdr_alsize)) { - ubi_err(ubi, "VID header offset %d too large.", ubi->vid_hdr_offset); - return -EINVAL; - } - dbg_gen("min_io_size %d", ubi->min_io_size); dbg_gen("max_write_size %d", ubi->max_write_size); dbg_gen("hdrs_min_io_size %d", ubi->hdrs_min_io_size); @@ -689,6 +683,21 @@ static int io_init(struct ubi_device *ubi, int max_beb_per1024) ubi->vid_hdr_aloffset; } + /* + * Memory allocation for VID header is ubi->vid_hdr_alsize + * which is described in comments in io.c. + * Make sure VID header shift + UBI_VID_HDR_SIZE not exceeds + * ubi->vid_hdr_alsize, so that all vid header operations + * won't access memory out of bounds. + */ + if ((ubi->vid_hdr_shift + UBI_VID_HDR_SIZE) > ubi->vid_hdr_alsize) { + ubi_err(ubi, "Invalid VID header offset %d, VID header shift(%d)" + " + VID header size(%zu) > VID header aligned size(%d).", + ubi->vid_hdr_offset, ubi->vid_hdr_shift, + UBI_VID_HDR_SIZE, ubi->vid_hdr_alsize); + return -EINVAL; + } + /* Similar for the data offset */ ubi->leb_start = ubi->vid_hdr_offset + UBI_VID_HDR_SIZE; ubi->leb_start = ALIGN(ubi->leb_start, ubi->min_io_size); diff --git a/drivers/mtd/ubi/wl.c b/drivers/mtd/ubi/wl.c index 40f39e5d6dfc..26a214f016c1 100644 --- a/drivers/mtd/ubi/wl.c +++ b/drivers/mtd/ubi/wl.c @@ -575,7 +575,7 @@ static int erase_worker(struct ubi_device *ubi, struct ubi_work *wl_wrk, * @vol_id: the volume ID that last used this PEB * @lnum: the last used logical eraseblock number for the PEB * @torture: if the physical eraseblock has to be tortured - * @nested: denotes whether the work_sem is already held in read mode + * @nested: denotes whether the work_sem is already held * * This function returns zero in case of success and a %-ENOMEM in case of * failure. @@ -1131,7 +1131,7 @@ static int __erase_worker(struct ubi_device *ubi, struct ubi_work *wl_wrk) int err1; /* Re-schedule the LEB for erasure */ - err1 = schedule_erase(ubi, e, vol_id, lnum, 0, false); + err1 = schedule_erase(ubi, e, vol_id, lnum, 0, true); if (err1) { spin_lock(&ubi->wl_lock); wl_entry_destroy(ubi, e); diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c index db7e650d9ebb..710548dbd0c1 100644 --- a/drivers/net/bonding/bond_main.c +++ b/drivers/net/bonding/bond_main.c @@ -1777,14 +1777,15 @@ void bond_lower_state_changed(struct slave *slave) /* The bonding driver uses ether_setup() to convert a master bond device * to ARPHRD_ETHER, that resets the target netdevice's flags so we always - * have to restore the IFF_MASTER flag, and only restore IFF_SLAVE if it was set + * have to restore the IFF_MASTER flag, and only restore IFF_SLAVE and IFF_UP + * if they were set */ static void bond_ether_setup(struct net_device *bond_dev) { - unsigned int slave_flag = bond_dev->flags & IFF_SLAVE; + unsigned int flags = bond_dev->flags & (IFF_SLAVE | IFF_UP); ether_setup(bond_dev); - bond_dev->flags |= IFF_MASTER | slave_flag; + bond_dev->flags |= IFF_MASTER | flags; bond_dev->priv_flags &= ~IFF_TX_SKB_SHARING; } diff --git a/drivers/net/dsa/microchip/ksz8795.c b/drivers/net/dsa/microchip/ksz8795.c index 23614a937cc3..f56fca1b1a22 100644 --- a/drivers/net/dsa/microchip/ksz8795.c +++ b/drivers/net/dsa/microchip/ksz8795.c @@ -96,7 +96,7 @@ static int ksz8795_change_mtu(struct ksz_device *dev, int frame_size) if (frame_size > KSZ8_LEGAL_PACKET_SIZE) ctrl2 |= SW_LEGAL_PACKET_DISABLE; - else if (frame_size > KSZ8863_NORMAL_PACKET_SIZE) + if (frame_size > KSZ8863_NORMAL_PACKET_SIZE) ctrl1 |= SW_HUGE_PACKET; ret = ksz_rmw8(dev, REG_SW_CTRL_1, SW_HUGE_PACKET, ctrl1); diff --git a/drivers/net/dsa/qca/Kconfig b/drivers/net/dsa/qca/Kconfig index 7a86d6d6a246..4347b42c50fd 100644 --- a/drivers/net/dsa/qca/Kconfig +++ b/drivers/net/dsa/qca/Kconfig @@ -19,7 +19,7 @@ config NET_DSA_QCA8K config NET_DSA_QCA8K_LEDS_SUPPORT bool "Qualcomm Atheros QCA8K Ethernet switch family LEDs support" depends on NET_DSA_QCA8K - depends on LEDS_CLASS + depends on LEDS_CLASS=y || LEDS_CLASS=NET_DSA_QCA8K help This enabled support for LEDs present on the Qualcomm Atheros QCA8K Ethernet switch chips. diff --git a/drivers/net/ethernet/amd/Kconfig b/drivers/net/ethernet/amd/Kconfig index ab42f75b9413..235fcacef5c5 100644 --- a/drivers/net/ethernet/amd/Kconfig +++ b/drivers/net/ethernet/amd/Kconfig @@ -186,4 +186,16 @@ config AMD_XGBE_HAVE_ECC bool default n +config PDS_CORE + tristate "AMD/Pensando Data Systems Core Device Support" + depends on 64BIT && PCI + help + This enables the support for the AMD/Pensando Core device family of + adapters. More specific information on this driver can be + found in + <file:Documentation/networking/device_drivers/ethernet/amd/pds_core.rst>. + + To compile this driver as a module, choose M here. The module + will be called pds_core. + endif # NET_VENDOR_AMD diff --git a/drivers/net/ethernet/amd/Makefile b/drivers/net/ethernet/amd/Makefile index 42742afe9115..2dcfb84731e1 100644 --- a/drivers/net/ethernet/amd/Makefile +++ b/drivers/net/ethernet/amd/Makefile @@ -17,3 +17,4 @@ obj-$(CONFIG_PCNET32) += pcnet32.o obj-$(CONFIG_SUN3LANCE) += sun3lance.o obj-$(CONFIG_SUNLANCE) += sunlance.o obj-$(CONFIG_AMD_XGBE) += xgbe/ +obj-$(CONFIG_PDS_CORE) += pds_core/ diff --git a/drivers/net/ethernet/amd/pds_core/Makefile b/drivers/net/ethernet/amd/pds_core/Makefile new file mode 100644 index 000000000000..0abc33ce826c --- /dev/null +++ b/drivers/net/ethernet/amd/pds_core/Makefile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Advanced Micro Devices, Inc. + +obj-$(CONFIG_PDS_CORE) := pds_core.o + +pds_core-y := main.o \ + devlink.o \ + auxbus.o \ + dev.o \ + adminq.o \ + core.o \ + fw.o + +pds_core-$(CONFIG_DEBUG_FS) += debugfs.o diff --git a/drivers/net/ethernet/amd/pds_core/adminq.c b/drivers/net/ethernet/amd/pds_core/adminq.c new file mode 100644 index 000000000000..045fe133f6ee --- /dev/null +++ b/drivers/net/ethernet/amd/pds_core/adminq.c @@ -0,0 +1,290 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include <linux/dynamic_debug.h> + +#include "core.h" + +struct pdsc_wait_context { + struct pdsc_qcq *qcq; + struct completion wait_completion; +}; + +static int pdsc_process_notifyq(struct pdsc_qcq *qcq) +{ + union pds_core_notifyq_comp *comp; + struct pdsc *pdsc = qcq->pdsc; + struct pdsc_cq *cq = &qcq->cq; + struct pdsc_cq_info *cq_info; + int nq_work = 0; + u64 eid; + + cq_info = &cq->info[cq->tail_idx]; + comp = cq_info->comp; + eid = le64_to_cpu(comp->event.eid); + while (eid > pdsc->last_eid) { + u16 ecode = le16_to_cpu(comp->event.ecode); + + switch (ecode) { + case PDS_EVENT_LINK_CHANGE: + dev_info(pdsc->dev, "NotifyQ LINK_CHANGE ecode %d eid %lld\n", + ecode, eid); + pdsc_notify(PDS_EVENT_LINK_CHANGE, comp); + break; + + case PDS_EVENT_RESET: + dev_info(pdsc->dev, "NotifyQ RESET ecode %d eid %lld\n", + ecode, eid); + pdsc_notify(PDS_EVENT_RESET, comp); + break; + + case PDS_EVENT_XCVR: + dev_info(pdsc->dev, "NotifyQ XCVR ecode %d eid %lld\n", + ecode, eid); + break; + + default: + dev_info(pdsc->dev, "NotifyQ ecode %d eid %lld\n", + ecode, eid); + break; + } + + pdsc->last_eid = eid; + cq->tail_idx = (cq->tail_idx + 1) & (cq->num_descs - 1); + cq_info = &cq->info[cq->tail_idx]; + comp = cq_info->comp; + eid = le64_to_cpu(comp->event.eid); + + nq_work++; + } + + qcq->accum_work += nq_work; + + return nq_work; +} + +void pdsc_process_adminq(struct pdsc_qcq *qcq) +{ + union pds_core_adminq_comp *comp; + struct pdsc_queue *q = &qcq->q; + struct pdsc *pdsc = qcq->pdsc; + struct pdsc_cq *cq = &qcq->cq; + struct pdsc_q_info *q_info; + unsigned long irqflags; + int nq_work = 0; + int aq_work = 0; + int credits; + + /* Don't process AdminQ when shutting down */ + if (pdsc->state & BIT_ULL(PDSC_S_STOPPING_DRIVER)) { + dev_err(pdsc->dev, "%s: called while PDSC_S_STOPPING_DRIVER\n", + __func__); + return; + } + + /* Check for NotifyQ event */ + nq_work = pdsc_process_notifyq(&pdsc->notifyqcq); + + /* Check for empty queue, which can happen if the interrupt was + * for a NotifyQ event and there are no new AdminQ completions. + */ + if (q->tail_idx == q->head_idx) + goto credits; + + /* Find the first completion to clean, + * run the callback in the related q_info, + * and continue while we still match done color + */ + spin_lock_irqsave(&pdsc->adminq_lock, irqflags); + comp = cq->info[cq->tail_idx].comp; + while (pdsc_color_match(comp->color, cq->done_color)) { + q_info = &q->info[q->tail_idx]; + q->tail_idx = (q->tail_idx + 1) & (q->num_descs - 1); + + /* Copy out the completion data */ + memcpy(q_info->dest, comp, sizeof(*comp)); + + complete_all(&q_info->wc->wait_completion); + + if (cq->tail_idx == cq->num_descs - 1) + cq->done_color = !cq->done_color; + cq->tail_idx = (cq->tail_idx + 1) & (cq->num_descs - 1); + comp = cq->info[cq->tail_idx].comp; + + aq_work++; + } + spin_unlock_irqrestore(&pdsc->adminq_lock, irqflags); + + qcq->accum_work += aq_work; + +credits: + /* Return the interrupt credits, one for each completion */ + credits = nq_work + aq_work; + if (credits) + pds_core_intr_credits(&pdsc->intr_ctrl[qcq->intx], + credits, + PDS_CORE_INTR_CRED_REARM); +} + +void pdsc_work_thread(struct work_struct *work) +{ + struct pdsc_qcq *qcq = container_of(work, struct pdsc_qcq, work); + + pdsc_process_adminq(qcq); +} + +irqreturn_t pdsc_adminq_isr(int irq, void *data) +{ + struct pdsc_qcq *qcq = data; + struct pdsc *pdsc = qcq->pdsc; + + /* Don't process AdminQ when shutting down */ + if (pdsc->state & BIT_ULL(PDSC_S_STOPPING_DRIVER)) { + dev_err(pdsc->dev, "%s: called while PDSC_S_STOPPING_DRIVER\n", + __func__); + return IRQ_HANDLED; + } + + queue_work(pdsc->wq, &qcq->work); + pds_core_intr_mask(&pdsc->intr_ctrl[irq], PDS_CORE_INTR_MASK_CLEAR); + + return IRQ_HANDLED; +} + +static int __pdsc_adminq_post(struct pdsc *pdsc, + struct pdsc_qcq *qcq, + union pds_core_adminq_cmd *cmd, + union pds_core_adminq_comp *comp, + struct pdsc_wait_context *wc) +{ + struct pdsc_queue *q = &qcq->q; + struct pdsc_q_info *q_info; + unsigned long irqflags; + unsigned int avail; + int index; + int ret; + + spin_lock_irqsave(&pdsc->adminq_lock, irqflags); + + /* Check for space in the queue */ + avail = q->tail_idx; + if (q->head_idx >= avail) + avail += q->num_descs - q->head_idx - 1; + else + avail -= q->head_idx + 1; + if (!avail) { + ret = -ENOSPC; + goto err_out_unlock; + } + + /* Check that the FW is running */ + if (!pdsc_is_fw_running(pdsc)) { + u8 fw_status = ioread8(&pdsc->info_regs->fw_status); + + dev_info(pdsc->dev, "%s: post failed - fw not running %#02x:\n", + __func__, fw_status); + ret = -ENXIO; + + goto err_out_unlock; + } + + /* Post the request */ + index = q->head_idx; + q_info = &q->info[index]; + q_info->wc = wc; + q_info->dest = comp; + memcpy(q_info->desc, cmd, sizeof(*cmd)); + + dev_dbg(pdsc->dev, "head_idx %d tail_idx %d\n", + q->head_idx, q->tail_idx); + dev_dbg(pdsc->dev, "post admin queue command:\n"); + dynamic_hex_dump("cmd ", DUMP_PREFIX_OFFSET, 16, 1, + cmd, sizeof(*cmd), true); + + q->head_idx = (q->head_idx + 1) & (q->num_descs - 1); + + pds_core_dbell_ring(pdsc->kern_dbpage, + q->hw_type, q->dbval | q->head_idx); + ret = index; + +err_out_unlock: + spin_unlock_irqrestore(&pdsc->adminq_lock, irqflags); + return ret; +} + +int pdsc_adminq_post(struct pdsc *pdsc, + union pds_core_adminq_cmd *cmd, + union pds_core_adminq_comp *comp, + bool fast_poll) +{ + struct pdsc_wait_context wc = { + .wait_completion = + COMPLETION_INITIALIZER_ONSTACK(wc.wait_completion), + }; + unsigned long poll_interval = 1; + unsigned long poll_jiffies; + unsigned long time_limit; + unsigned long time_start; + unsigned long time_done; + unsigned long remaining; + int err = 0; + int index; + + wc.qcq = &pdsc->adminqcq; + index = __pdsc_adminq_post(pdsc, &pdsc->adminqcq, cmd, comp, &wc); + if (index < 0) { + err = index; + goto err_out; + } + + time_start = jiffies; + time_limit = time_start + HZ * pdsc->devcmd_timeout; + do { + /* Timeslice the actual wait to catch IO errors etc early */ + poll_jiffies = msecs_to_jiffies(poll_interval); + remaining = wait_for_completion_timeout(&wc.wait_completion, + poll_jiffies); + if (remaining) + break; + + if (!pdsc_is_fw_running(pdsc)) { + u8 fw_status = ioread8(&pdsc->info_regs->fw_status); + + dev_dbg(pdsc->dev, "%s: post wait failed - fw not running %#02x:\n", + __func__, fw_status); + err = -ENXIO; + break; + } + + /* When fast_poll is not requested, prevent aggressive polling + * on failures due to timeouts by doing exponential back off. + */ + if (!fast_poll && poll_interval < PDSC_ADMINQ_MAX_POLL_INTERVAL) + poll_interval <<= 1; + } while (time_before(jiffies, time_limit)); + time_done = jiffies; + dev_dbg(pdsc->dev, "%s: elapsed %d msecs\n", + __func__, jiffies_to_msecs(time_done - time_start)); + + /* Check the results */ + if (time_after_eq(time_done, time_limit)) + err = -ETIMEDOUT; + + dev_dbg(pdsc->dev, "read admin queue completion idx %d:\n", index); + dynamic_hex_dump("comp ", DUMP_PREFIX_OFFSET, 16, 1, + comp, sizeof(*comp), true); + + if (remaining && comp->status) + err = pdsc_err_to_errno(comp->status); + +err_out: + if (err) { + dev_dbg(pdsc->dev, "%s: opcode %d status %d err %pe\n", + __func__, cmd->opcode, comp->status, ERR_PTR(err)); + if (err == -ENXIO || err == -ETIMEDOUT) + queue_work(pdsc->wq, &pdsc->health_work); + } + + return err; +} +EXPORT_SYMBOL_GPL(pdsc_adminq_post); diff --git a/drivers/net/ethernet/amd/pds_core/auxbus.c b/drivers/net/ethernet/amd/pds_core/auxbus.c new file mode 100644 index 000000000000..561af8e5b3ea --- /dev/null +++ b/drivers/net/ethernet/amd/pds_core/auxbus.c @@ -0,0 +1,264 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include <linux/pci.h> + +#include "core.h" +#include <linux/pds/pds_auxbus.h> + +/** + * pds_client_register - Link the client to the firmware + * @pf_pdev: ptr to the PF driver struct + * @devname: name that includes service into, e.g. pds_core.vDPA + * + * Return: 0 on success, or + * negative for error + */ +int pds_client_register(struct pci_dev *pf_pdev, char *devname) +{ + union pds_core_adminq_comp comp = {}; + union pds_core_adminq_cmd cmd = {}; + struct pdsc *pf; + int err; + u16 ci; + + pf = pci_get_drvdata(pf_pdev); + if (pf->state) + return -ENXIO; + + cmd.client_reg.opcode = PDS_AQ_CMD_CLIENT_REG; + strscpy(cmd.client_reg.devname, devname, + sizeof(cmd.client_reg.devname)); + + err = pdsc_adminq_post(pf, &cmd, &comp, false); + if (err) { + dev_info(pf->dev, "register dev_name %s with DSC failed, status %d: %pe\n", + devname, comp.status, ERR_PTR(err)); + return err; + } + + ci = le16_to_cpu(comp.client_reg.client_id); + if (!ci) { + dev_err(pf->dev, "%s: device returned null client_id\n", + __func__); + return -EIO; + } + + dev_dbg(pf->dev, "%s: device returned client_id %d for %s\n", + __func__, ci, devname); + + return ci; +} +EXPORT_SYMBOL_GPL(pds_client_register); + +/** + * pds_client_unregister - Unlink the client from the firmware + * @pf_pdev: ptr to the PF driver struct + * @client_id: id returned from pds_client_register() + * + * Return: 0 on success, or + * negative for error + */ +int pds_client_unregister(struct pci_dev *pf_pdev, u16 client_id) +{ + union pds_core_adminq_comp comp = {}; + union pds_core_adminq_cmd cmd = {}; + struct pdsc *pf; + int err; + + pf = pci_get_drvdata(pf_pdev); + if (pf->state) + return -ENXIO; + + cmd.client_unreg.opcode = PDS_AQ_CMD_CLIENT_UNREG; + cmd.client_unreg.client_id = cpu_to_le16(client_id); + + err = pdsc_adminq_post(pf, &cmd, &comp, false); + if (err) + dev_info(pf->dev, "unregister client_id %d failed, status %d: %pe\n", + client_id, comp.status, ERR_PTR(err)); + + return err; +} +EXPORT_SYMBOL_GPL(pds_client_unregister); + +/** + * pds_client_adminq_cmd - Process an adminq request for the client + * @padev: ptr to the client device + * @req: ptr to buffer with request + * @req_len: length of actual struct used for request + * @resp: ptr to buffer where answer is to be copied + * @flags: optional flags from pds_core_adminq_flags + * + * Return: 0 on success, or + * negative for error + * + * Client sends pointers to request and response buffers + * Core copies request data into pds_core_client_request_cmd + * Core sets other fields as needed + * Core posts to AdminQ + * Core copies completion data into response buffer + */ +int pds_client_adminq_cmd(struct pds_auxiliary_dev *padev, + union pds_core_adminq_cmd *req, + size_t req_len, + union pds_core_adminq_comp *resp, + u64 flags) +{ + union pds_core_adminq_cmd cmd = {}; + struct pci_dev *pf_pdev; + struct pdsc *pf; + size_t cp_len; + int err; + + pf_pdev = pci_physfn(padev->vf_pdev); + pf = pci_get_drvdata(pf_pdev); + + dev_dbg(pf->dev, "%s: %s opcode %d\n", + __func__, dev_name(&padev->aux_dev.dev), req->opcode); + + if (pf->state) + return -ENXIO; + + /* Wrap the client's request */ + cmd.client_request.opcode = PDS_AQ_CMD_CLIENT_CMD; + cmd.client_request.client_id = cpu_to_le16(padev->client_id); + cp_len = min_t(size_t, req_len, sizeof(cmd.client_request.client_cmd)); + memcpy(cmd.client_request.client_cmd, req, cp_len); + + err = pdsc_adminq_post(pf, &cmd, resp, + !!(flags & PDS_AQ_FLAG_FASTPOLL)); + if (err && err != -EAGAIN) + dev_info(pf->dev, "client admin cmd failed: %pe\n", + ERR_PTR(err)); + + return err; +} +EXPORT_SYMBOL_GPL(pds_client_adminq_cmd); + +static void pdsc_auxbus_dev_release(struct device *dev) +{ + struct pds_auxiliary_dev *padev = + container_of(dev, struct pds_auxiliary_dev, aux_dev.dev); + + kfree(padev); +} + +static struct pds_auxiliary_dev *pdsc_auxbus_dev_register(struct pdsc *cf, + struct pdsc *pf, + u16 client_id, + char *name) +{ + struct auxiliary_device *aux_dev; + struct pds_auxiliary_dev *padev; + int err; + + padev = kzalloc(sizeof(*padev), GFP_KERNEL); + if (!padev) + return ERR_PTR(-ENOMEM); + + padev->vf_pdev = cf->pdev; + padev->client_id = client_id; + + aux_dev = &padev->aux_dev; + aux_dev->name = name; + aux_dev->id = cf->uid; + aux_dev->dev.parent = cf->dev; + aux_dev->dev.release = pdsc_auxbus_dev_release; + + err = auxiliary_device_init(aux_dev); + if (err < 0) { + dev_warn(cf->dev, "auxiliary_device_init of %s failed: %pe\n", + name, ERR_PTR(err)); + goto err_out; + } + + err = auxiliary_device_add(aux_dev); + if (err) { + dev_warn(cf->dev, "auxiliary_device_add of %s failed: %pe\n", + name, ERR_PTR(err)); + goto err_out_uninit; + } + + return padev; + +err_out_uninit: + auxiliary_device_uninit(aux_dev); +err_out: + kfree(padev); + return ERR_PTR(err); +} + +int pdsc_auxbus_dev_del(struct pdsc *cf, struct pdsc *pf) +{ + struct pds_auxiliary_dev *padev; + int err = 0; + + mutex_lock(&pf->config_lock); + + padev = pf->vfs[cf->vf_id].padev; + if (padev) { + pds_client_unregister(pf->pdev, padev->client_id); + auxiliary_device_delete(&padev->aux_dev); + auxiliary_device_uninit(&padev->aux_dev); + padev->client_id = 0; + } + pf->vfs[cf->vf_id].padev = NULL; + + mutex_unlock(&pf->config_lock); + return err; +} + +int pdsc_auxbus_dev_add(struct pdsc *cf, struct pdsc *pf) +{ + struct pds_auxiliary_dev *padev; + enum pds_core_vif_types vt; + char devname[PDS_DEVNAME_LEN]; + u16 vt_support; + int client_id; + int err = 0; + + mutex_lock(&pf->config_lock); + + /* We only support vDPA so far, so it is the only one to + * be verified that it is available in the Core device and + * enabled in the devlink param. In the future this might + * become a loop for several VIF types. + */ + + /* Verify that the type is supported and enabled. It is not + * an error if there is no auxbus device support for this + * VF, it just means something else needs to happen with it. + */ + vt = PDS_DEV_TYPE_VDPA; + vt_support = !!le16_to_cpu(pf->dev_ident.vif_types[vt]); + if (!(vt_support && + pf->viftype_status[vt].supported && + pf->viftype_status[vt].enabled)) + goto out_unlock; + + /* Need to register with FW and get the client_id before + * creating the aux device so that the aux client can run + * adminq commands as part its probe + */ + snprintf(devname, sizeof(devname), "%s.%s.%d", + PDS_CORE_DRV_NAME, pf->viftype_status[vt].name, cf->uid); + client_id = pds_client_register(pf->pdev, devname); + if (client_id < 0) { + err = client_id; + goto out_unlock; + } + + padev = pdsc_auxbus_dev_register(cf, pf, client_id, + pf->viftype_status[vt].name); + if (IS_ERR(padev)) { + pds_client_unregister(pf->pdev, client_id); + err = PTR_ERR(padev); + goto out_unlock; + } + pf->vfs[cf->vf_id].padev = padev; + +out_unlock: + mutex_unlock(&pf->config_lock); + return err; +} diff --git a/drivers/net/ethernet/amd/pds_core/core.c b/drivers/net/ethernet/amd/pds_core/core.c new file mode 100644 index 000000000000..483a070d96fa --- /dev/null +++ b/drivers/net/ethernet/amd/pds_core/core.c @@ -0,0 +1,597 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include <linux/pci.h> +#include <linux/vmalloc.h> + +#include "core.h" + +static BLOCKING_NOTIFIER_HEAD(pds_notify_chain); + +int pdsc_register_notify(struct notifier_block *nb) +{ + return blocking_notifier_chain_register(&pds_notify_chain, nb); +} +EXPORT_SYMBOL_GPL(pdsc_register_notify); + +void pdsc_unregister_notify(struct notifier_block *nb) +{ + blocking_notifier_chain_unregister(&pds_notify_chain, nb); +} +EXPORT_SYMBOL_GPL(pdsc_unregister_notify); + +void pdsc_notify(unsigned long event, void *data) +{ + blocking_notifier_call_chain(&pds_notify_chain, event, data); +} + +void pdsc_intr_free(struct pdsc *pdsc, int index) +{ + struct pdsc_intr_info *intr_info; + + if (index >= pdsc->nintrs || index < 0) { + WARN(true, "bad intr index %d\n", index); + return; + } + + intr_info = &pdsc->intr_info[index]; + if (!intr_info->vector) + return; + dev_dbg(pdsc->dev, "%s: idx %d vec %d name %s\n", + __func__, index, intr_info->vector, intr_info->name); + + pds_core_intr_mask(&pdsc->intr_ctrl[index], PDS_CORE_INTR_MASK_SET); + pds_core_intr_clean(&pdsc->intr_ctrl[index]); + + free_irq(intr_info->vector, intr_info->data); + + memset(intr_info, 0, sizeof(*intr_info)); +} + +int pdsc_intr_alloc(struct pdsc *pdsc, char *name, + irq_handler_t handler, void *data) +{ + struct pdsc_intr_info *intr_info; + unsigned int index; + int err; + + /* Find the first available interrupt */ + for (index = 0; index < pdsc->nintrs; index++) + if (!pdsc->intr_info[index].vector) + break; + if (index >= pdsc->nintrs) { + dev_warn(pdsc->dev, "%s: no intr, index=%d nintrs=%d\n", + __func__, index, pdsc->nintrs); + return -ENOSPC; + } + + pds_core_intr_clean_flags(&pdsc->intr_ctrl[index], + PDS_CORE_INTR_CRED_RESET_COALESCE); + + intr_info = &pdsc->intr_info[index]; + + intr_info->index = index; + intr_info->data = data; + strscpy(intr_info->name, name, sizeof(intr_info->name)); + + /* Get the OS vector number for the interrupt */ + err = pci_irq_vector(pdsc->pdev, index); + if (err < 0) { + dev_err(pdsc->dev, "failed to get intr vector index %d: %pe\n", + index, ERR_PTR(err)); + goto err_out_free_intr; + } + intr_info->vector = err; + + /* Init the device's intr mask */ + pds_core_intr_clean(&pdsc->intr_ctrl[index]); + pds_core_intr_mask_assert(&pdsc->intr_ctrl[index], 1); + pds_core_intr_mask(&pdsc->intr_ctrl[index], PDS_CORE_INTR_MASK_SET); + + /* Register the isr with a name */ + err = request_irq(intr_info->vector, handler, 0, intr_info->name, data); + if (err) { + dev_err(pdsc->dev, "failed to get intr irq vector %d: %pe\n", + intr_info->vector, ERR_PTR(err)); + goto err_out_free_intr; + } + + return index; + +err_out_free_intr: + pdsc_intr_free(pdsc, index); + return err; +} + +static void pdsc_qcq_intr_free(struct pdsc *pdsc, struct pdsc_qcq *qcq) +{ + if (!(qcq->flags & PDS_CORE_QCQ_F_INTR) || + qcq->intx == PDS_CORE_INTR_INDEX_NOT_ASSIGNED) + return; + + pdsc_intr_free(pdsc, qcq->intx); + qcq->intx = PDS_CORE_INTR_INDEX_NOT_ASSIGNED; +} + +static int pdsc_qcq_intr_alloc(struct pdsc *pdsc, struct pdsc_qcq *qcq) +{ + char name[PDSC_INTR_NAME_MAX_SZ]; + int index; + + if (!(qcq->flags & PDS_CORE_QCQ_F_INTR)) { + qcq->intx = PDS_CORE_INTR_INDEX_NOT_ASSIGNED; + return 0; + } + + snprintf(name, sizeof(name), "%s-%d-%s", + PDS_CORE_DRV_NAME, pdsc->pdev->bus->number, qcq->q.name); + index = pdsc_intr_alloc(pdsc, name, pdsc_adminq_isr, qcq); + if (index < 0) + return index; + qcq->intx = index; + + return 0; +} + +void pdsc_qcq_free(struct pdsc *pdsc, struct pdsc_qcq *qcq) +{ + struct device *dev = pdsc->dev; + + if (!(qcq && qcq->pdsc)) + return; + + pdsc_debugfs_del_qcq(qcq); + + pdsc_qcq_intr_free(pdsc, qcq); + + if (qcq->q_base) + dma_free_coherent(dev, qcq->q_size, + qcq->q_base, qcq->q_base_pa); + + if (qcq->cq_base) + dma_free_coherent(dev, qcq->cq_size, + qcq->cq_base, qcq->cq_base_pa); + + if (qcq->cq.info) + vfree(qcq->cq.info); + + if (qcq->q.info) + vfree(qcq->q.info); + + memset(qcq, 0, sizeof(*qcq)); +} + +static void pdsc_q_map(struct pdsc_queue *q, void *base, dma_addr_t base_pa) +{ + struct pdsc_q_info *cur; + unsigned int i; + + q->base = base; + q->base_pa = base_pa; + + for (i = 0, cur = q->info; i < q->num_descs; i++, cur++) + cur->desc = base + (i * q->desc_size); +} + +static void pdsc_cq_map(struct pdsc_cq *cq, void *base, dma_addr_t base_pa) +{ + struct pdsc_cq_info *cur; + unsigned int i; + + cq->base = base; + cq->base_pa = base_pa; + + for (i = 0, cur = cq->info; i < cq->num_descs; i++, cur++) + cur->comp = base + (i * cq->desc_size); +} + +int pdsc_qcq_alloc(struct pdsc *pdsc, unsigned int type, unsigned int index, + const char *name, unsigned int flags, unsigned int num_descs, + unsigned int desc_size, unsigned int cq_desc_size, + unsigned int pid, struct pdsc_qcq *qcq) +{ + struct device *dev = pdsc->dev; + void *q_base, *cq_base; + dma_addr_t cq_base_pa; + dma_addr_t q_base_pa; + int err; + + qcq->q.info = vzalloc(num_descs * sizeof(*qcq->q.info)); + if (!qcq->q.info) { + err = -ENOMEM; + goto err_out; + } + + qcq->pdsc = pdsc; + qcq->flags = flags; + INIT_WORK(&qcq->work, pdsc_work_thread); + + qcq->q.type = type; + qcq->q.index = index; + qcq->q.num_descs = num_descs; + qcq->q.desc_size = desc_size; + qcq->q.tail_idx = 0; + qcq->q.head_idx = 0; + qcq->q.pid = pid; + snprintf(qcq->q.name, sizeof(qcq->q.name), "%s%u", name, index); + + err = pdsc_qcq_intr_alloc(pdsc, qcq); + if (err) + goto err_out_free_q_info; + + qcq->cq.info = vzalloc(num_descs * sizeof(*qcq->cq.info)); + if (!qcq->cq.info) { + err = -ENOMEM; + goto err_out_free_irq; + } + + qcq->cq.bound_intr = &pdsc->intr_info[qcq->intx]; + qcq->cq.num_descs = num_descs; + qcq->cq.desc_size = cq_desc_size; + qcq->cq.tail_idx = 0; + qcq->cq.done_color = 1; + + if (flags & PDS_CORE_QCQ_F_NOTIFYQ) { + /* q & cq need to be contiguous in case of notifyq */ + qcq->q_size = PDS_PAGE_SIZE + + ALIGN(num_descs * desc_size, PDS_PAGE_SIZE) + + ALIGN(num_descs * cq_desc_size, PDS_PAGE_SIZE); + qcq->q_base = dma_alloc_coherent(dev, + qcq->q_size + qcq->cq_size, + &qcq->q_base_pa, + GFP_KERNEL); + if (!qcq->q_base) { + err = -ENOMEM; + goto err_out_free_cq_info; + } + q_base = PTR_ALIGN(qcq->q_base, PDS_PAGE_SIZE); + q_base_pa = ALIGN(qcq->q_base_pa, PDS_PAGE_SIZE); + pdsc_q_map(&qcq->q, q_base, q_base_pa); + + cq_base = PTR_ALIGN(q_base + + ALIGN(num_descs * desc_size, PDS_PAGE_SIZE), + PDS_PAGE_SIZE); + cq_base_pa = ALIGN(qcq->q_base_pa + + ALIGN(num_descs * desc_size, PDS_PAGE_SIZE), + PDS_PAGE_SIZE); + + } else { + /* q DMA descriptors */ + qcq->q_size = PDS_PAGE_SIZE + (num_descs * desc_size); + qcq->q_base = dma_alloc_coherent(dev, qcq->q_size, + &qcq->q_base_pa, + GFP_KERNEL); + if (!qcq->q_base) { + err = -ENOMEM; + goto err_out_free_cq_info; + } + q_base = PTR_ALIGN(qcq->q_base, PDS_PAGE_SIZE); + q_base_pa = ALIGN(qcq->q_base_pa, PDS_PAGE_SIZE); + pdsc_q_map(&qcq->q, q_base, q_base_pa); + + /* cq DMA descriptors */ + qcq->cq_size = PDS_PAGE_SIZE + (num_descs * cq_desc_size); + qcq->cq_base = dma_alloc_coherent(dev, qcq->cq_size, + &qcq->cq_base_pa, + GFP_KERNEL); + if (!qcq->cq_base) { + err = -ENOMEM; + goto err_out_free_q; + } + cq_base = PTR_ALIGN(qcq->cq_base, PDS_PAGE_SIZE); + cq_base_pa = ALIGN(qcq->cq_base_pa, PDS_PAGE_SIZE); + } + + pdsc_cq_map(&qcq->cq, cq_base, cq_base_pa); + qcq->cq.bound_q = &qcq->q; + + pdsc_debugfs_add_qcq(pdsc, qcq); + + return 0; + +err_out_free_q: + dma_free_coherent(dev, qcq->q_size, qcq->q_base, qcq->q_base_pa); +err_out_free_cq_info: + vfree(qcq->cq.info); +err_out_free_irq: + pdsc_qcq_intr_free(pdsc, qcq); +err_out_free_q_info: + vfree(qcq->q.info); + memset(qcq, 0, sizeof(*qcq)); +err_out: + dev_err(dev, "qcq alloc of %s%d failed %d\n", name, index, err); + return err; +} + +static int pdsc_core_init(struct pdsc *pdsc) +{ + union pds_core_dev_comp comp = {}; + union pds_core_dev_cmd cmd = { + .init.opcode = PDS_CORE_CMD_INIT, + }; + struct pds_core_dev_init_data_out cido; + struct pds_core_dev_init_data_in cidi; + u32 dbid_count; + u32 dbpage_num; + size_t sz; + int err; + + cidi.adminq_q_base = cpu_to_le64(pdsc->adminqcq.q_base_pa); + cidi.adminq_cq_base = cpu_to_le64(pdsc->adminqcq.cq_base_pa); + cidi.notifyq_cq_base = cpu_to_le64(pdsc->notifyqcq.cq.base_pa); + cidi.flags = cpu_to_le32(PDS_CORE_QINIT_F_IRQ | PDS_CORE_QINIT_F_ENA); + cidi.intr_index = cpu_to_le16(pdsc->adminqcq.intx); + cidi.adminq_ring_size = ilog2(pdsc->adminqcq.q.num_descs); + cidi.notifyq_ring_size = ilog2(pdsc->notifyqcq.q.num_descs); + + mutex_lock(&pdsc->devcmd_lock); + + sz = min_t(size_t, sizeof(cidi), sizeof(pdsc->cmd_regs->data)); + memcpy_toio(&pdsc->cmd_regs->data, &cidi, sz); + + err = pdsc_devcmd_locked(pdsc, &cmd, &comp, pdsc->devcmd_timeout); + if (!err) { + sz = min_t(size_t, sizeof(cido), sizeof(pdsc->cmd_regs->data)); + memcpy_fromio(&cido, &pdsc->cmd_regs->data, sz); + } + + mutex_unlock(&pdsc->devcmd_lock); + if (err) { + dev_err(pdsc->dev, "Device init command failed: %pe\n", + ERR_PTR(err)); + return err; + } + + pdsc->hw_index = le32_to_cpu(cido.core_hw_index); + + dbid_count = le32_to_cpu(pdsc->dev_ident.ndbpgs_per_lif); + dbpage_num = pdsc->hw_index * dbid_count; + pdsc->kern_dbpage = pdsc_map_dbpage(pdsc, dbpage_num); + if (!pdsc->kern_dbpage) { + dev_err(pdsc->dev, "Cannot map dbpage, aborting\n"); + return -ENOMEM; + } + + pdsc->adminqcq.q.hw_type = cido.adminq_hw_type; + pdsc->adminqcq.q.hw_index = le32_to_cpu(cido.adminq_hw_index); + pdsc->adminqcq.q.dbval = PDS_CORE_DBELL_QID(pdsc->adminqcq.q.hw_index); + + pdsc->notifyqcq.q.hw_type = cido.notifyq_hw_type; + pdsc->notifyqcq.q.hw_index = le32_to_cpu(cido.notifyq_hw_index); + pdsc->notifyqcq.q.dbval = PDS_CORE_DBELL_QID(pdsc->notifyqcq.q.hw_index); + + pdsc->last_eid = 0; + + return err; +} + +static struct pdsc_viftype pdsc_viftype_defaults[] = { + [PDS_DEV_TYPE_VDPA] = { .name = PDS_DEV_TYPE_VDPA_STR, + .vif_id = PDS_DEV_TYPE_VDPA, + .dl_id = DEVLINK_PARAM_GENERIC_ID_ENABLE_VNET }, + [PDS_DEV_TYPE_MAX] = {} +}; + +static int pdsc_viftypes_init(struct pdsc *pdsc) +{ + enum pds_core_vif_types vt; + + pdsc->viftype_status = kzalloc(sizeof(pdsc_viftype_defaults), + GFP_KERNEL); + if (!pdsc->viftype_status) + return -ENOMEM; + + for (vt = 0; vt < PDS_DEV_TYPE_MAX; vt++) { + bool vt_support; + + if (!pdsc_viftype_defaults[vt].name) + continue; + + /* Grab the defaults */ + pdsc->viftype_status[vt] = pdsc_viftype_defaults[vt]; + + /* See what the Core device has for support */ + vt_support = !!le16_to_cpu(pdsc->dev_ident.vif_types[vt]); + dev_dbg(pdsc->dev, "VIF %s is %ssupported\n", + pdsc->viftype_status[vt].name, + vt_support ? "" : "not "); + + pdsc->viftype_status[vt].supported = vt_support; + } + + return 0; +} + +int pdsc_setup(struct pdsc *pdsc, bool init) +{ + int numdescs; + int err; + + if (init) + err = pdsc_dev_init(pdsc); + else + err = pdsc_dev_reinit(pdsc); + if (err) + return err; + + /* Scale the descriptor ring length based on number of CPUs and VFs */ + numdescs = max_t(int, PDSC_ADMINQ_MIN_LENGTH, num_online_cpus()); + numdescs += 2 * pci_sriov_get_totalvfs(pdsc->pdev); + numdescs = roundup_pow_of_two(numdescs); + err = pdsc_qcq_alloc(pdsc, PDS_CORE_QTYPE_ADMINQ, 0, "adminq", + PDS_CORE_QCQ_F_CORE | PDS_CORE_QCQ_F_INTR, + numdescs, + sizeof(union pds_core_adminq_cmd), + sizeof(union pds_core_adminq_comp), + 0, &pdsc->adminqcq); + if (err) + goto err_out_teardown; + + err = pdsc_qcq_alloc(pdsc, PDS_CORE_QTYPE_NOTIFYQ, 0, "notifyq", + PDS_CORE_QCQ_F_NOTIFYQ, + PDSC_NOTIFYQ_LENGTH, + sizeof(struct pds_core_notifyq_cmd), + sizeof(union pds_core_notifyq_comp), + 0, &pdsc->notifyqcq); + if (err) + goto err_out_teardown; + + /* NotifyQ rides on the AdminQ interrupt */ + pdsc->notifyqcq.intx = pdsc->adminqcq.intx; + + /* Set up the Core with the AdminQ and NotifyQ info */ + err = pdsc_core_init(pdsc); + if (err) + goto err_out_teardown; + + /* Set up the VIFs */ + err = pdsc_viftypes_init(pdsc); + if (err) + goto err_out_teardown; + + if (init) + pdsc_debugfs_add_viftype(pdsc); + + clear_bit(PDSC_S_FW_DEAD, &pdsc->state); + return 0; + +err_out_teardown: + pdsc_teardown(pdsc, init); + return err; +} + +void pdsc_teardown(struct pdsc *pdsc, bool removing) +{ + int i; + + pdsc_devcmd_reset(pdsc); + pdsc_qcq_free(pdsc, &pdsc->notifyqcq); + pdsc_qcq_free(pdsc, &pdsc->adminqcq); + + kfree(pdsc->viftype_status); + pdsc->viftype_status = NULL; + + if (pdsc->intr_info) { + for (i = 0; i < pdsc->nintrs; i++) + pdsc_intr_free(pdsc, i); + + if (removing) { + kfree(pdsc->intr_info); + pdsc->intr_info = NULL; + } + } + + if (pdsc->kern_dbpage) { + iounmap(pdsc->kern_dbpage); + pdsc->kern_dbpage = NULL; + } + + set_bit(PDSC_S_FW_DEAD, &pdsc->state); +} + +int pdsc_start(struct pdsc *pdsc) +{ + pds_core_intr_mask(&pdsc->intr_ctrl[pdsc->adminqcq.intx], + PDS_CORE_INTR_MASK_CLEAR); + + return 0; +} + +void pdsc_stop(struct pdsc *pdsc) +{ + int i; + + if (!pdsc->intr_info) + return; + + /* Mask interrupts that are in use */ + for (i = 0; i < pdsc->nintrs; i++) + if (pdsc->intr_info[i].vector) + pds_core_intr_mask(&pdsc->intr_ctrl[i], + PDS_CORE_INTR_MASK_SET); +} + +static void pdsc_fw_down(struct pdsc *pdsc) +{ + union pds_core_notifyq_comp reset_event = { + .reset.ecode = cpu_to_le16(PDS_EVENT_RESET), + .reset.state = 0, + }; + + if (test_and_set_bit(PDSC_S_FW_DEAD, &pdsc->state)) { + dev_err(pdsc->dev, "%s: already happening\n", __func__); + return; + } + + /* Notify clients of fw_down */ + devlink_health_report(pdsc->fw_reporter, "FW down reported", pdsc); + pdsc_notify(PDS_EVENT_RESET, &reset_event); + + pdsc_stop(pdsc); + pdsc_teardown(pdsc, PDSC_TEARDOWN_RECOVERY); +} + +static void pdsc_fw_up(struct pdsc *pdsc) +{ + union pds_core_notifyq_comp reset_event = { + .reset.ecode = cpu_to_le16(PDS_EVENT_RESET), + .reset.state = 1, + }; + int err; + + if (!test_bit(PDSC_S_FW_DEAD, &pdsc->state)) { + dev_err(pdsc->dev, "%s: fw not dead\n", __func__); + return; + } + + err = pdsc_setup(pdsc, PDSC_SETUP_RECOVERY); + if (err) + goto err_out; + + err = pdsc_start(pdsc); + if (err) + goto err_out; + + /* Notify clients of fw_up */ + pdsc->fw_recoveries++; + devlink_health_reporter_state_update(pdsc->fw_reporter, + DEVLINK_HEALTH_REPORTER_STATE_HEALTHY); + pdsc_notify(PDS_EVENT_RESET, &reset_event); + + return; + +err_out: + pdsc_teardown(pdsc, PDSC_TEARDOWN_RECOVERY); +} + +void pdsc_health_thread(struct work_struct *work) +{ + struct pdsc *pdsc = container_of(work, struct pdsc, health_work); + unsigned long mask; + bool healthy; + + mutex_lock(&pdsc->config_lock); + + /* Don't do a check when in a transition state */ + mask = BIT_ULL(PDSC_S_INITING_DRIVER) | + BIT_ULL(PDSC_S_STOPPING_DRIVER); + if (pdsc->state & mask) + goto out_unlock; + + healthy = pdsc_is_fw_good(pdsc); + dev_dbg(pdsc->dev, "%s: health %d fw_status %#02x fw_heartbeat %d\n", + __func__, healthy, pdsc->fw_status, pdsc->last_hb); + + if (test_bit(PDSC_S_FW_DEAD, &pdsc->state)) { + if (healthy) + pdsc_fw_up(pdsc); + } else { + if (!healthy) + pdsc_fw_down(pdsc); + } + + pdsc->fw_generation = pdsc->fw_status & PDS_CORE_FW_STS_F_GENERATION; + +out_unlock: + mutex_unlock(&pdsc->config_lock); +} diff --git a/drivers/net/ethernet/amd/pds_core/core.h b/drivers/net/ethernet/amd/pds_core/core.h new file mode 100644 index 000000000000..e545fafc4819 --- /dev/null +++ b/drivers/net/ethernet/amd/pds_core/core.h @@ -0,0 +1,312 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#ifndef _PDSC_H_ +#define _PDSC_H_ + +#include <linux/debugfs.h> +#include <net/devlink.h> + +#include <linux/pds/pds_common.h> +#include <linux/pds/pds_core_if.h> +#include <linux/pds/pds_adminq.h> +#include <linux/pds/pds_intr.h> + +#define PDSC_DRV_DESCRIPTION "AMD/Pensando Core Driver" + +#define PDSC_WATCHDOG_SECS 5 +#define PDSC_QUEUE_NAME_MAX_SZ 32 +#define PDSC_ADMINQ_MIN_LENGTH 16 /* must be a power of two */ +#define PDSC_NOTIFYQ_LENGTH 64 /* must be a power of two */ +#define PDSC_TEARDOWN_RECOVERY false +#define PDSC_TEARDOWN_REMOVING true +#define PDSC_SETUP_RECOVERY false +#define PDSC_SETUP_INIT true + +struct pdsc_dev_bar { + void __iomem *vaddr; + phys_addr_t bus_addr; + unsigned long len; + int res_index; +}; + +struct pdsc; + +struct pdsc_vf { + struct pds_auxiliary_dev *padev; + struct pdsc *vf; + u16 index; + __le16 vif_types[PDS_DEV_TYPE_MAX]; +}; + +struct pdsc_devinfo { + u8 asic_type; + u8 asic_rev; + char fw_version[PDS_CORE_DEVINFO_FWVERS_BUFLEN + 1]; + char serial_num[PDS_CORE_DEVINFO_SERIAL_BUFLEN + 1]; +}; + +struct pdsc_queue { + struct pdsc_q_info *info; + u64 dbval; + u16 head_idx; + u16 tail_idx; + u8 hw_type; + unsigned int index; + unsigned int num_descs; + u64 dbell_count; + u64 features; + unsigned int type; + unsigned int hw_index; + union { + void *base; + struct pds_core_admin_cmd *adminq; + }; + dma_addr_t base_pa; /* must be page aligned */ + unsigned int desc_size; + unsigned int pid; + char name[PDSC_QUEUE_NAME_MAX_SZ]; +}; + +#define PDSC_INTR_NAME_MAX_SZ 32 + +struct pdsc_intr_info { + char name[PDSC_INTR_NAME_MAX_SZ]; + unsigned int index; + unsigned int vector; + void *data; +}; + +struct pdsc_cq_info { + void *comp; +}; + +struct pdsc_buf_info { + struct page *page; + dma_addr_t dma_addr; + u32 page_offset; + u32 len; +}; + +struct pdsc_q_info { + union { + void *desc; + struct pdsc_admin_cmd *adminq_desc; + }; + unsigned int bytes; + unsigned int nbufs; + struct pdsc_buf_info bufs[PDS_CORE_MAX_FRAGS]; + struct pdsc_wait_context *wc; + void *dest; +}; + +struct pdsc_cq { + struct pdsc_cq_info *info; + struct pdsc_queue *bound_q; + struct pdsc_intr_info *bound_intr; + u16 tail_idx; + bool done_color; + unsigned int num_descs; + unsigned int desc_size; + void *base; + dma_addr_t base_pa; /* must be page aligned */ +} ____cacheline_aligned_in_smp; + +struct pdsc_qcq { + struct pdsc *pdsc; + void *q_base; + dma_addr_t q_base_pa; /* might not be page aligned */ + void *cq_base; + dma_addr_t cq_base_pa; /* might not be page aligned */ + u32 q_size; + u32 cq_size; + bool armed; + unsigned int flags; + + struct work_struct work; + struct pdsc_queue q; + struct pdsc_cq cq; + int intx; + + u32 accum_work; + struct dentry *dentry; +}; + +struct pdsc_viftype { + char *name; + bool supported; + bool enabled; + int dl_id; + int vif_id; + struct pds_auxiliary_dev *padev; +}; + +/* No state flags set means we are in a steady running state */ +enum pdsc_state_flags { + PDSC_S_FW_DEAD, /* stopped, wait on startup or recovery */ + PDSC_S_INITING_DRIVER, /* initial startup from probe */ + PDSC_S_STOPPING_DRIVER, /* driver remove */ + + /* leave this as last */ + PDSC_S_STATE_SIZE +}; + +struct pdsc { + struct pci_dev *pdev; + struct dentry *dentry; + struct device *dev; + struct pdsc_dev_bar bars[PDS_CORE_BARS_MAX]; + struct pdsc_vf *vfs; + int num_vfs; + int vf_id; + int hw_index; + int uid; + + unsigned long state; + u8 fw_status; + u8 fw_generation; + unsigned long last_fw_time; + u32 last_hb; + struct timer_list wdtimer; + unsigned int wdtimer_period; + struct work_struct health_work; + struct devlink_health_reporter *fw_reporter; + u32 fw_recoveries; + + struct pdsc_devinfo dev_info; + struct pds_core_dev_identity dev_ident; + unsigned int nintrs; + struct pdsc_intr_info *intr_info; /* array of nintrs elements */ + + struct workqueue_struct *wq; + + unsigned int devcmd_timeout; + struct mutex devcmd_lock; /* lock for dev_cmd operations */ + struct mutex config_lock; /* lock for configuration operations */ + spinlock_t adminq_lock; /* lock for adminq operations */ + struct pds_core_dev_info_regs __iomem *info_regs; + struct pds_core_dev_cmd_regs __iomem *cmd_regs; + struct pds_core_intr __iomem *intr_ctrl; + u64 __iomem *intr_status; + u64 __iomem *db_pages; + dma_addr_t phy_db_pages; + u64 __iomem *kern_dbpage; + + struct pdsc_qcq adminqcq; + struct pdsc_qcq notifyqcq; + u64 last_eid; + struct pdsc_viftype *viftype_status; +}; + +/** enum pds_core_dbell_bits - bitwise composition of dbell values. + * + * @PDS_CORE_DBELL_QID_MASK: unshifted mask of valid queue id bits. + * @PDS_CORE_DBELL_QID_SHIFT: queue id shift amount in dbell value. + * @PDS_CORE_DBELL_QID: macro to build QID component of dbell value. + * + * @PDS_CORE_DBELL_RING_MASK: unshifted mask of valid ring bits. + * @PDS_CORE_DBELL_RING_SHIFT: ring shift amount in dbell value. + * @PDS_CORE_DBELL_RING: macro to build ring component of dbell value. + * + * @PDS_CORE_DBELL_RING_0: ring zero dbell component value. + * @PDS_CORE_DBELL_RING_1: ring one dbell component value. + * @PDS_CORE_DBELL_RING_2: ring two dbell component value. + * @PDS_CORE_DBELL_RING_3: ring three dbell component value. + * + * @PDS_CORE_DBELL_INDEX_MASK: bit mask of valid index bits, no shift needed. + */ +enum pds_core_dbell_bits { + PDS_CORE_DBELL_QID_MASK = 0xffffff, + PDS_CORE_DBELL_QID_SHIFT = 24, + +#define PDS_CORE_DBELL_QID(n) \ + (((u64)(n) & PDS_CORE_DBELL_QID_MASK) << PDS_CORE_DBELL_QID_SHIFT) + + PDS_CORE_DBELL_RING_MASK = 0x7, + PDS_CORE_DBELL_RING_SHIFT = 16, + +#define PDS_CORE_DBELL_RING(n) \ + (((u64)(n) & PDS_CORE_DBELL_RING_MASK) << PDS_CORE_DBELL_RING_SHIFT) + + PDS_CORE_DBELL_RING_0 = 0, + PDS_CORE_DBELL_RING_1 = PDS_CORE_DBELL_RING(1), + PDS_CORE_DBELL_RING_2 = PDS_CORE_DBELL_RING(2), + PDS_CORE_DBELL_RING_3 = PDS_CORE_DBELL_RING(3), + + PDS_CORE_DBELL_INDEX_MASK = 0xffff, +}; + +static inline void pds_core_dbell_ring(u64 __iomem *db_page, + enum pds_core_logical_qtype qtype, + u64 val) +{ + writeq(val, &db_page[qtype]); +} + +int pdsc_fw_reporter_diagnose(struct devlink_health_reporter *reporter, + struct devlink_fmsg *fmsg, + struct netlink_ext_ack *extack); +int pdsc_dl_info_get(struct devlink *dl, struct devlink_info_req *req, + struct netlink_ext_ack *extack); +int pdsc_dl_flash_update(struct devlink *dl, + struct devlink_flash_update_params *params, + struct netlink_ext_ack *extack); +int pdsc_dl_enable_get(struct devlink *dl, u32 id, + struct devlink_param_gset_ctx *ctx); +int pdsc_dl_enable_set(struct devlink *dl, u32 id, + struct devlink_param_gset_ctx *ctx); +int pdsc_dl_enable_validate(struct devlink *dl, u32 id, + union devlink_param_value val, + struct netlink_ext_ack *extack); + +void __iomem *pdsc_map_dbpage(struct pdsc *pdsc, int page_num); + +void pdsc_debugfs_create(void); +void pdsc_debugfs_destroy(void); +void pdsc_debugfs_add_dev(struct pdsc *pdsc); +void pdsc_debugfs_del_dev(struct pdsc *pdsc); +void pdsc_debugfs_add_ident(struct pdsc *pdsc); +void pdsc_debugfs_add_viftype(struct pdsc *pdsc); +void pdsc_debugfs_add_irqs(struct pdsc *pdsc); +void pdsc_debugfs_add_qcq(struct pdsc *pdsc, struct pdsc_qcq *qcq); +void pdsc_debugfs_del_qcq(struct pdsc_qcq *qcq); + +int pdsc_err_to_errno(enum pds_core_status_code code); +bool pdsc_is_fw_running(struct pdsc *pdsc); +bool pdsc_is_fw_good(struct pdsc *pdsc); +int pdsc_devcmd(struct pdsc *pdsc, union pds_core_dev_cmd *cmd, + union pds_core_dev_comp *comp, int max_seconds); +int pdsc_devcmd_locked(struct pdsc *pdsc, union pds_core_dev_cmd *cmd, + union pds_core_dev_comp *comp, int max_seconds); +int pdsc_devcmd_init(struct pdsc *pdsc); +int pdsc_devcmd_reset(struct pdsc *pdsc); +int pdsc_dev_reinit(struct pdsc *pdsc); +int pdsc_dev_init(struct pdsc *pdsc); + +int pdsc_intr_alloc(struct pdsc *pdsc, char *name, + irq_handler_t handler, void *data); +void pdsc_intr_free(struct pdsc *pdsc, int index); +void pdsc_qcq_free(struct pdsc *pdsc, struct pdsc_qcq *qcq); +int pdsc_qcq_alloc(struct pdsc *pdsc, unsigned int type, unsigned int index, + const char *name, unsigned int flags, unsigned int num_descs, + unsigned int desc_size, unsigned int cq_desc_size, + unsigned int pid, struct pdsc_qcq *qcq); +int pdsc_setup(struct pdsc *pdsc, bool init); +void pdsc_teardown(struct pdsc *pdsc, bool removing); +int pdsc_start(struct pdsc *pdsc); +void pdsc_stop(struct pdsc *pdsc); +void pdsc_health_thread(struct work_struct *work); + +int pdsc_register_notify(struct notifier_block *nb); +void pdsc_unregister_notify(struct notifier_block *nb); +void pdsc_notify(unsigned long event, void *data); +int pdsc_auxbus_dev_add(struct pdsc *cf, struct pdsc *pf); +int pdsc_auxbus_dev_del(struct pdsc *cf, struct pdsc *pf); + +void pdsc_process_adminq(struct pdsc_qcq *qcq); +void pdsc_work_thread(struct work_struct *work); +irqreturn_t pdsc_adminq_isr(int irq, void *data); + +int pdsc_firmware_update(struct pdsc *pdsc, const struct firmware *fw, + struct netlink_ext_ack *extack); +#endif /* _PDSC_H_ */ diff --git a/drivers/net/ethernet/amd/pds_core/debugfs.c b/drivers/net/ethernet/amd/pds_core/debugfs.c new file mode 100644 index 000000000000..8ec392299b7d --- /dev/null +++ b/drivers/net/ethernet/amd/pds_core/debugfs.c @@ -0,0 +1,170 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include <linux/pci.h> + +#include "core.h" + +static struct dentry *pdsc_dir; + +void pdsc_debugfs_create(void) +{ + pdsc_dir = debugfs_create_dir(PDS_CORE_DRV_NAME, NULL); +} + +void pdsc_debugfs_destroy(void) +{ + debugfs_remove_recursive(pdsc_dir); +} + +void pdsc_debugfs_add_dev(struct pdsc *pdsc) +{ + pdsc->dentry = debugfs_create_dir(pci_name(pdsc->pdev), pdsc_dir); + + debugfs_create_ulong("state", 0400, pdsc->dentry, &pdsc->state); +} + +void pdsc_debugfs_del_dev(struct pdsc *pdsc) +{ + debugfs_remove_recursive(pdsc->dentry); + pdsc->dentry = NULL; +} + +static int identity_show(struct seq_file *seq, void *v) +{ + struct pdsc *pdsc = seq->private; + struct pds_core_dev_identity *ident; + int vt; + + ident = &pdsc->dev_ident; + + seq_printf(seq, "fw_heartbeat: 0x%x\n", + ioread32(&pdsc->info_regs->fw_heartbeat)); + + seq_printf(seq, "nlifs: %d\n", + le32_to_cpu(ident->nlifs)); + seq_printf(seq, "nintrs: %d\n", + le32_to_cpu(ident->nintrs)); + seq_printf(seq, "ndbpgs_per_lif: %d\n", + le32_to_cpu(ident->ndbpgs_per_lif)); + seq_printf(seq, "intr_coal_mult: %d\n", + le32_to_cpu(ident->intr_coal_mult)); + seq_printf(seq, "intr_coal_div: %d\n", + le32_to_cpu(ident->intr_coal_div)); + + seq_puts(seq, "vif_types: "); + for (vt = 0; vt < PDS_DEV_TYPE_MAX; vt++) + seq_printf(seq, "%d ", + le16_to_cpu(pdsc->dev_ident.vif_types[vt])); + seq_puts(seq, "\n"); + + return 0; +} +DEFINE_SHOW_ATTRIBUTE(identity); + +void pdsc_debugfs_add_ident(struct pdsc *pdsc) +{ + debugfs_create_file("identity", 0400, pdsc->dentry, + pdsc, &identity_fops); +} + +static int viftype_show(struct seq_file *seq, void *v) +{ + struct pdsc *pdsc = seq->private; + int vt; + + for (vt = 0; vt < PDS_DEV_TYPE_MAX; vt++) { + if (!pdsc->viftype_status[vt].name) + continue; + + seq_printf(seq, "%s\t%d supported %d enabled\n", + pdsc->viftype_status[vt].name, + pdsc->viftype_status[vt].supported, + pdsc->viftype_status[vt].enabled); + } + return 0; +} +DEFINE_SHOW_ATTRIBUTE(viftype); + +void pdsc_debugfs_add_viftype(struct pdsc *pdsc) +{ + debugfs_create_file("viftypes", 0400, pdsc->dentry, + pdsc, &viftype_fops); +} + +static const struct debugfs_reg32 intr_ctrl_regs[] = { + { .name = "coal_init", .offset = 0, }, + { .name = "mask", .offset = 4, }, + { .name = "credits", .offset = 8, }, + { .name = "mask_on_assert", .offset = 12, }, + { .name = "coal_timer", .offset = 16, }, +}; + +void pdsc_debugfs_add_qcq(struct pdsc *pdsc, struct pdsc_qcq *qcq) +{ + struct dentry *qcq_dentry, *q_dentry, *cq_dentry; + struct dentry *intr_dentry; + struct debugfs_regset32 *intr_ctrl_regset; + struct pdsc_intr_info *intr = &pdsc->intr_info[qcq->intx]; + struct pdsc_queue *q = &qcq->q; + struct pdsc_cq *cq = &qcq->cq; + + qcq_dentry = debugfs_create_dir(q->name, pdsc->dentry); + if (IS_ERR_OR_NULL(qcq_dentry)) + return; + qcq->dentry = qcq_dentry; + + debugfs_create_x64("q_base_pa", 0400, qcq_dentry, &qcq->q_base_pa); + debugfs_create_x32("q_size", 0400, qcq_dentry, &qcq->q_size); + debugfs_create_x64("cq_base_pa", 0400, qcq_dentry, &qcq->cq_base_pa); + debugfs_create_x32("cq_size", 0400, qcq_dentry, &qcq->cq_size); + debugfs_create_x32("accum_work", 0400, qcq_dentry, &qcq->accum_work); + + q_dentry = debugfs_create_dir("q", qcq->dentry); + if (IS_ERR_OR_NULL(q_dentry)) + return; + + debugfs_create_u32("index", 0400, q_dentry, &q->index); + debugfs_create_u32("num_descs", 0400, q_dentry, &q->num_descs); + debugfs_create_u32("desc_size", 0400, q_dentry, &q->desc_size); + debugfs_create_u32("pid", 0400, q_dentry, &q->pid); + + debugfs_create_u16("tail", 0400, q_dentry, &q->tail_idx); + debugfs_create_u16("head", 0400, q_dentry, &q->head_idx); + + cq_dentry = debugfs_create_dir("cq", qcq->dentry); + if (IS_ERR_OR_NULL(cq_dentry)) + return; + + debugfs_create_x64("base_pa", 0400, cq_dentry, &cq->base_pa); + debugfs_create_u32("num_descs", 0400, cq_dentry, &cq->num_descs); + debugfs_create_u32("desc_size", 0400, cq_dentry, &cq->desc_size); + debugfs_create_bool("done_color", 0400, cq_dentry, &cq->done_color); + debugfs_create_u16("tail", 0400, cq_dentry, &cq->tail_idx); + + if (qcq->flags & PDS_CORE_QCQ_F_INTR) { + intr_dentry = debugfs_create_dir("intr", qcq->dentry); + if (IS_ERR_OR_NULL(intr_dentry)) + return; + + debugfs_create_u32("index", 0400, intr_dentry, &intr->index); + debugfs_create_u32("vector", 0400, intr_dentry, &intr->vector); + + intr_ctrl_regset = kzalloc(sizeof(*intr_ctrl_regset), + GFP_KERNEL); + if (!intr_ctrl_regset) + return; + intr_ctrl_regset->regs = intr_ctrl_regs; + intr_ctrl_regset->nregs = ARRAY_SIZE(intr_ctrl_regs); + intr_ctrl_regset->base = &pdsc->intr_ctrl[intr->index]; + + debugfs_create_regset32("intr_ctrl", 0400, intr_dentry, + intr_ctrl_regset); + } +}; + +void pdsc_debugfs_del_qcq(struct pdsc_qcq *qcq) +{ + debugfs_remove_recursive(qcq->dentry); + qcq->dentry = NULL; +} diff --git a/drivers/net/ethernet/amd/pds_core/dev.c b/drivers/net/ethernet/amd/pds_core/dev.c new file mode 100644 index 000000000000..f7c597ea5daf --- /dev/null +++ b/drivers/net/ethernet/amd/pds_core/dev.c @@ -0,0 +1,351 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include <linux/errno.h> +#include <linux/pci.h> +#include <linux/utsname.h> + +#include "core.h" + +int pdsc_err_to_errno(enum pds_core_status_code code) +{ + switch (code) { + case PDS_RC_SUCCESS: + return 0; + case PDS_RC_EVERSION: + case PDS_RC_EQTYPE: + case PDS_RC_EQID: + case PDS_RC_EINVAL: + case PDS_RC_ENOSUPP: + return -EINVAL; + case PDS_RC_EPERM: + return -EPERM; + case PDS_RC_ENOENT: + return -ENOENT; + case PDS_RC_EAGAIN: + return -EAGAIN; + case PDS_RC_ENOMEM: + return -ENOMEM; + case PDS_RC_EFAULT: + return -EFAULT; + case PDS_RC_EBUSY: + return -EBUSY; + case PDS_RC_EEXIST: + return -EEXIST; + case PDS_RC_EVFID: + return -ENODEV; + case PDS_RC_ECLIENT: + return -ECHILD; + case PDS_RC_ENOSPC: + return -ENOSPC; + case PDS_RC_ERANGE: + return -ERANGE; + case PDS_RC_BAD_ADDR: + return -EFAULT; + case PDS_RC_EOPCODE: + case PDS_RC_EINTR: + case PDS_RC_DEV_CMD: + case PDS_RC_ERROR: + case PDS_RC_ERDMA: + case PDS_RC_EIO: + default: + return -EIO; + } +} + +bool pdsc_is_fw_running(struct pdsc *pdsc) +{ + pdsc->fw_status = ioread8(&pdsc->info_regs->fw_status); + pdsc->last_fw_time = jiffies; + pdsc->last_hb = ioread32(&pdsc->info_regs->fw_heartbeat); + + /* Firmware is useful only if the running bit is set and + * fw_status != 0xff (bad PCI read) + */ + return (pdsc->fw_status != 0xff) && + (pdsc->fw_status & PDS_CORE_FW_STS_F_RUNNING); +} + +bool pdsc_is_fw_good(struct pdsc *pdsc) +{ + u8 gen = pdsc->fw_status & PDS_CORE_FW_STS_F_GENERATION; + + return pdsc_is_fw_running(pdsc) && gen == pdsc->fw_generation; +} + +static u8 pdsc_devcmd_status(struct pdsc *pdsc) +{ + return ioread8(&pdsc->cmd_regs->comp.status); +} + +static bool pdsc_devcmd_done(struct pdsc *pdsc) +{ + return ioread32(&pdsc->cmd_regs->done) & PDS_CORE_DEV_CMD_DONE; +} + +static void pdsc_devcmd_dbell(struct pdsc *pdsc) +{ + iowrite32(0, &pdsc->cmd_regs->done); + iowrite32(1, &pdsc->cmd_regs->doorbell); +} + +static void pdsc_devcmd_clean(struct pdsc *pdsc) +{ + iowrite32(0, &pdsc->cmd_regs->doorbell); + memset_io(&pdsc->cmd_regs->cmd, 0, sizeof(pdsc->cmd_regs->cmd)); +} + +static const char *pdsc_devcmd_str(int opcode) +{ + switch (opcode) { + case PDS_CORE_CMD_NOP: + return "PDS_CORE_CMD_NOP"; + case PDS_CORE_CMD_IDENTIFY: + return "PDS_CORE_CMD_IDENTIFY"; + case PDS_CORE_CMD_RESET: + return "PDS_CORE_CMD_RESET"; + case PDS_CORE_CMD_INIT: + return "PDS_CORE_CMD_INIT"; + case PDS_CORE_CMD_FW_DOWNLOAD: + return "PDS_CORE_CMD_FW_DOWNLOAD"; + case PDS_CORE_CMD_FW_CONTROL: + return "PDS_CORE_CMD_FW_CONTROL"; + default: + return "PDS_CORE_CMD_UNKNOWN"; + } +} + +static int pdsc_devcmd_wait(struct pdsc *pdsc, int max_seconds) +{ + struct device *dev = pdsc->dev; + unsigned long start_time; + unsigned long max_wait; + unsigned long duration; + int timeout = 0; + int done = 0; + int err = 0; + int status; + int opcode; + + opcode = ioread8(&pdsc->cmd_regs->cmd.opcode); + + start_time = jiffies; + max_wait = start_time + (max_seconds * HZ); + + while (!done && !timeout) { + done = pdsc_devcmd_done(pdsc); + if (done) + break; + + timeout = time_after(jiffies, max_wait); + if (timeout) + break; + + usleep_range(100, 200); + } + duration = jiffies - start_time; + + if (done && duration > HZ) + dev_dbg(dev, "DEVCMD %d %s after %ld secs\n", + opcode, pdsc_devcmd_str(opcode), duration / HZ); + + if (!done || timeout) { + dev_err(dev, "DEVCMD %d %s timeout, done %d timeout %d max_seconds=%d\n", + opcode, pdsc_devcmd_str(opcode), done, timeout, + max_seconds); + err = -ETIMEDOUT; + pdsc_devcmd_clean(pdsc); + } + + status = pdsc_devcmd_status(pdsc); + err = pdsc_err_to_errno(status); + if (err && err != -EAGAIN) + dev_err(dev, "DEVCMD %d %s failed, status=%d err %d %pe\n", + opcode, pdsc_devcmd_str(opcode), status, err, + ERR_PTR(err)); + + return err; +} + +int pdsc_devcmd_locked(struct pdsc *pdsc, union pds_core_dev_cmd *cmd, + union pds_core_dev_comp *comp, int max_seconds) +{ + int err; + + memcpy_toio(&pdsc->cmd_regs->cmd, cmd, sizeof(*cmd)); + pdsc_devcmd_dbell(pdsc); + err = pdsc_devcmd_wait(pdsc, max_seconds); + memcpy_fromio(comp, &pdsc->cmd_regs->comp, sizeof(*comp)); + + if (err == -ENXIO || err == -ETIMEDOUT) + queue_work(pdsc->wq, &pdsc->health_work); + + return err; +} + +int pdsc_devcmd(struct pdsc *pdsc, union pds_core_dev_cmd *cmd, + union pds_core_dev_comp *comp, int max_seconds) +{ + int err; + + mutex_lock(&pdsc->devcmd_lock); + err = pdsc_devcmd_locked(pdsc, cmd, comp, max_seconds); + mutex_unlock(&pdsc->devcmd_lock); + + return err; +} + +int pdsc_devcmd_init(struct pdsc *pdsc) +{ + union pds_core_dev_comp comp = {}; + union pds_core_dev_cmd cmd = { + .opcode = PDS_CORE_CMD_INIT, + }; + + return pdsc_devcmd(pdsc, &cmd, &comp, pdsc->devcmd_timeout); +} + +int pdsc_devcmd_reset(struct pdsc *pdsc) +{ + union pds_core_dev_comp comp = {}; + union pds_core_dev_cmd cmd = { + .reset.opcode = PDS_CORE_CMD_RESET, + }; + + return pdsc_devcmd(pdsc, &cmd, &comp, pdsc->devcmd_timeout); +} + +static int pdsc_devcmd_identify_locked(struct pdsc *pdsc) +{ + union pds_core_dev_comp comp = {}; + union pds_core_dev_cmd cmd = { + .identify.opcode = PDS_CORE_CMD_IDENTIFY, + .identify.ver = PDS_CORE_IDENTITY_VERSION_1, + }; + + return pdsc_devcmd_locked(pdsc, &cmd, &comp, pdsc->devcmd_timeout); +} + +static void pdsc_init_devinfo(struct pdsc *pdsc) +{ + pdsc->dev_info.asic_type = ioread8(&pdsc->info_regs->asic_type); + pdsc->dev_info.asic_rev = ioread8(&pdsc->info_regs->asic_rev); + pdsc->fw_generation = PDS_CORE_FW_STS_F_GENERATION & + ioread8(&pdsc->info_regs->fw_status); + + memcpy_fromio(pdsc->dev_info.fw_version, + pdsc->info_regs->fw_version, + PDS_CORE_DEVINFO_FWVERS_BUFLEN); + pdsc->dev_info.fw_version[PDS_CORE_DEVINFO_FWVERS_BUFLEN] = 0; + + memcpy_fromio(pdsc->dev_info.serial_num, + pdsc->info_regs->serial_num, + PDS_CORE_DEVINFO_SERIAL_BUFLEN); + pdsc->dev_info.serial_num[PDS_CORE_DEVINFO_SERIAL_BUFLEN] = 0; + + dev_dbg(pdsc->dev, "fw_version %s\n", pdsc->dev_info.fw_version); +} + +static int pdsc_identify(struct pdsc *pdsc) +{ + struct pds_core_drv_identity drv = {}; + size_t sz; + int err; + + drv.drv_type = cpu_to_le32(PDS_DRIVER_LINUX); + snprintf(drv.driver_ver_str, sizeof(drv.driver_ver_str), + "%s %s", PDS_CORE_DRV_NAME, utsname()->release); + + /* Next let's get some info about the device + * We use the devcmd_lock at this level in order to + * get safe access to the cmd_regs->data before anyone + * else can mess it up + */ + mutex_lock(&pdsc->devcmd_lock); + + sz = min_t(size_t, sizeof(drv), sizeof(pdsc->cmd_regs->data)); + memcpy_toio(&pdsc->cmd_regs->data, &drv, sz); + + err = pdsc_devcmd_identify_locked(pdsc); + if (!err) { + sz = min_t(size_t, sizeof(pdsc->dev_ident), + sizeof(pdsc->cmd_regs->data)); + memcpy_fromio(&pdsc->dev_ident, &pdsc->cmd_regs->data, sz); + } + mutex_unlock(&pdsc->devcmd_lock); + + if (err) { + dev_err(pdsc->dev, "Cannot identify device: %pe\n", + ERR_PTR(err)); + return err; + } + + if (isprint(pdsc->dev_info.fw_version[0]) && + isascii(pdsc->dev_info.fw_version[0])) + dev_info(pdsc->dev, "FW: %.*s\n", + (int)(sizeof(pdsc->dev_info.fw_version) - 1), + pdsc->dev_info.fw_version); + else + dev_info(pdsc->dev, "FW: (invalid string) 0x%02x 0x%02x 0x%02x 0x%02x ...\n", + (u8)pdsc->dev_info.fw_version[0], + (u8)pdsc->dev_info.fw_version[1], + (u8)pdsc->dev_info.fw_version[2], + (u8)pdsc->dev_info.fw_version[3]); + + return 0; +} + +int pdsc_dev_reinit(struct pdsc *pdsc) +{ + pdsc_init_devinfo(pdsc); + + return pdsc_identify(pdsc); +} + +int pdsc_dev_init(struct pdsc *pdsc) +{ + unsigned int nintrs; + int err; + + /* Initial init and reset of device */ + pdsc_init_devinfo(pdsc); + pdsc->devcmd_timeout = PDS_CORE_DEVCMD_TIMEOUT; + + err = pdsc_devcmd_reset(pdsc); + if (err) + return err; + + err = pdsc_identify(pdsc); + if (err) + return err; + + pdsc_debugfs_add_ident(pdsc); + + /* Now we can reserve interrupts */ + nintrs = le32_to_cpu(pdsc->dev_ident.nintrs); + nintrs = min_t(unsigned int, num_online_cpus(), nintrs); + + /* Get intr_info struct array for tracking */ + pdsc->intr_info = kcalloc(nintrs, sizeof(*pdsc->intr_info), GFP_KERNEL); + if (!pdsc->intr_info) { + err = -ENOMEM; + goto err_out; + } + + err = pci_alloc_irq_vectors(pdsc->pdev, nintrs, nintrs, PCI_IRQ_MSIX); + if (err != nintrs) { + dev_err(pdsc->dev, "Can't get %d intrs from OS: %pe\n", + nintrs, ERR_PTR(err)); + err = -ENOSPC; + goto err_out; + } + pdsc->nintrs = nintrs; + + return 0; + +err_out: + kfree(pdsc->intr_info); + pdsc->intr_info = NULL; + + return err; +} diff --git a/drivers/net/ethernet/amd/pds_core/devlink.c b/drivers/net/ethernet/amd/pds_core/devlink.c new file mode 100644 index 000000000000..9c6b3653c1c7 --- /dev/null +++ b/drivers/net/ethernet/amd/pds_core/devlink.c @@ -0,0 +1,183 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include "core.h" +#include <linux/pds/pds_auxbus.h> + +static struct +pdsc_viftype *pdsc_dl_find_viftype_by_id(struct pdsc *pdsc, + enum devlink_param_type dl_id) +{ + int vt; + + for (vt = 0; vt < PDS_DEV_TYPE_MAX; vt++) { + if (pdsc->viftype_status[vt].dl_id == dl_id) + return &pdsc->viftype_status[vt]; + } + + return NULL; +} + +int pdsc_dl_enable_get(struct devlink *dl, u32 id, + struct devlink_param_gset_ctx *ctx) +{ + struct pdsc *pdsc = devlink_priv(dl); + struct pdsc_viftype *vt_entry; + + vt_entry = pdsc_dl_find_viftype_by_id(pdsc, id); + if (!vt_entry) + return -ENOENT; + + ctx->val.vbool = vt_entry->enabled; + + return 0; +} + +int pdsc_dl_enable_set(struct devlink *dl, u32 id, + struct devlink_param_gset_ctx *ctx) +{ + struct pdsc *pdsc = devlink_priv(dl); + struct pdsc_viftype *vt_entry; + int err = 0; + int vf_id; + + vt_entry = pdsc_dl_find_viftype_by_id(pdsc, id); + if (!vt_entry || !vt_entry->supported) + return -EOPNOTSUPP; + + if (vt_entry->enabled == ctx->val.vbool) + return 0; + + vt_entry->enabled = ctx->val.vbool; + for (vf_id = 0; vf_id < pdsc->num_vfs; vf_id++) { + struct pdsc *vf = pdsc->vfs[vf_id].vf; + + err = ctx->val.vbool ? pdsc_auxbus_dev_add(vf, pdsc) : + pdsc_auxbus_dev_del(vf, pdsc); + } + + return err; +} + +int pdsc_dl_enable_validate(struct devlink *dl, u32 id, + union devlink_param_value val, + struct netlink_ext_ack *extack) +{ + struct pdsc *pdsc = devlink_priv(dl); + struct pdsc_viftype *vt_entry; + + vt_entry = pdsc_dl_find_viftype_by_id(pdsc, id); + if (!vt_entry || !vt_entry->supported) + return -EOPNOTSUPP; + + if (!pdsc->viftype_status[vt_entry->vif_id].supported) + return -ENODEV; + + return 0; +} + +int pdsc_dl_flash_update(struct devlink *dl, + struct devlink_flash_update_params *params, + struct netlink_ext_ack *extack) +{ + struct pdsc *pdsc = devlink_priv(dl); + + return pdsc_firmware_update(pdsc, params->fw, extack); +} + +static char *fw_slotnames[] = { + "fw.goldfw", + "fw.mainfwa", + "fw.mainfwb", +}; + +int pdsc_dl_info_get(struct devlink *dl, struct devlink_info_req *req, + struct netlink_ext_ack *extack) +{ + union pds_core_dev_cmd cmd = { + .fw_control.opcode = PDS_CORE_CMD_FW_CONTROL, + .fw_control.oper = PDS_CORE_FW_GET_LIST, + }; + struct pds_core_fw_list_info fw_list; + struct pdsc *pdsc = devlink_priv(dl); + union pds_core_dev_comp comp; + char buf[16]; + int listlen; + int err; + int i; + + mutex_lock(&pdsc->devcmd_lock); + err = pdsc_devcmd_locked(pdsc, &cmd, &comp, pdsc->devcmd_timeout * 2); + memcpy_fromio(&fw_list, pdsc->cmd_regs->data, sizeof(fw_list)); + mutex_unlock(&pdsc->devcmd_lock); + if (err && err != -EIO) + return err; + + listlen = fw_list.num_fw_slots; + for (i = 0; i < listlen; i++) { + if (i < ARRAY_SIZE(fw_slotnames)) + strscpy(buf, fw_slotnames[i], sizeof(buf)); + else + snprintf(buf, sizeof(buf), "fw.slot_%d", i); + err = devlink_info_version_stored_put(req, buf, + fw_list.fw_names[i].fw_version); + } + + err = devlink_info_version_running_put(req, + DEVLINK_INFO_VERSION_GENERIC_FW, + pdsc->dev_info.fw_version); + if (err) + return err; + + snprintf(buf, sizeof(buf), "0x%x", pdsc->dev_info.asic_type); + err = devlink_info_version_fixed_put(req, + DEVLINK_INFO_VERSION_GENERIC_ASIC_ID, + buf); + if (err) + return err; + + snprintf(buf, sizeof(buf), "0x%x", pdsc->dev_info.asic_rev); + err = devlink_info_version_fixed_put(req, + DEVLINK_INFO_VERSION_GENERIC_ASIC_REV, + buf); + if (err) + return err; + + return devlink_info_serial_number_put(req, pdsc->dev_info.serial_num); +} + +int pdsc_fw_reporter_diagnose(struct devlink_health_reporter *reporter, + struct devlink_fmsg *fmsg, + struct netlink_ext_ack *extack) +{ + struct pdsc *pdsc = devlink_health_reporter_priv(reporter); + int err; + + mutex_lock(&pdsc->config_lock); + + if (test_bit(PDSC_S_FW_DEAD, &pdsc->state)) + err = devlink_fmsg_string_pair_put(fmsg, "Status", "dead"); + else if (!pdsc_is_fw_good(pdsc)) + err = devlink_fmsg_string_pair_put(fmsg, "Status", "unhealthy"); + else + err = devlink_fmsg_string_pair_put(fmsg, "Status", "healthy"); + + mutex_unlock(&pdsc->config_lock); + + if (err) + return err; + + err = devlink_fmsg_u32_pair_put(fmsg, "State", + pdsc->fw_status & + ~PDS_CORE_FW_STS_F_GENERATION); + if (err) + return err; + + err = devlink_fmsg_u32_pair_put(fmsg, "Generation", + pdsc->fw_generation >> 4); + if (err) + return err; + + return devlink_fmsg_u32_pair_put(fmsg, "Recoveries", + pdsc->fw_recoveries); +} diff --git a/drivers/net/ethernet/amd/pds_core/fw.c b/drivers/net/ethernet/amd/pds_core/fw.c new file mode 100644 index 000000000000..90a811f3878a --- /dev/null +++ b/drivers/net/ethernet/amd/pds_core/fw.c @@ -0,0 +1,194 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#include "core.h" + +/* The worst case wait for the install activity is about 25 minutes when + * installing a new CPLD, which is very seldom. Normal is about 30-35 + * seconds. Since the driver can't tell if a CPLD update will happen we + * set the timeout for the ugly case. + */ +#define PDSC_FW_INSTALL_TIMEOUT (25 * 60) +#define PDSC_FW_SELECT_TIMEOUT 30 + +/* Number of periodic log updates during fw file download */ +#define PDSC_FW_INTERVAL_FRACTION 32 + +static int pdsc_devcmd_fw_download_locked(struct pdsc *pdsc, u64 addr, + u32 offset, u32 length) +{ + union pds_core_dev_cmd cmd = { + .fw_download.opcode = PDS_CORE_CMD_FW_DOWNLOAD, + .fw_download.offset = cpu_to_le32(offset), + .fw_download.addr = cpu_to_le64(addr), + .fw_download.length = cpu_to_le32(length), + }; + union pds_core_dev_comp comp; + + return pdsc_devcmd_locked(pdsc, &cmd, &comp, pdsc->devcmd_timeout); +} + +static int pdsc_devcmd_fw_install(struct pdsc *pdsc) +{ + union pds_core_dev_cmd cmd = { + .fw_control.opcode = PDS_CORE_CMD_FW_CONTROL, + .fw_control.oper = PDS_CORE_FW_INSTALL_ASYNC + }; + union pds_core_dev_comp comp; + int err; + + err = pdsc_devcmd(pdsc, &cmd, &comp, pdsc->devcmd_timeout); + if (err < 0) + return err; + + return comp.fw_control.slot; +} + +static int pdsc_devcmd_fw_activate(struct pdsc *pdsc, + enum pds_core_fw_slot slot) +{ + union pds_core_dev_cmd cmd = { + .fw_control.opcode = PDS_CORE_CMD_FW_CONTROL, + .fw_control.oper = PDS_CORE_FW_ACTIVATE_ASYNC, + .fw_control.slot = slot + }; + union pds_core_dev_comp comp; + + return pdsc_devcmd(pdsc, &cmd, &comp, pdsc->devcmd_timeout); +} + +static int pdsc_fw_status_long_wait(struct pdsc *pdsc, + const char *label, + unsigned long timeout, + u8 fw_cmd, + struct netlink_ext_ack *extack) +{ + union pds_core_dev_cmd cmd = { + .fw_control.opcode = PDS_CORE_CMD_FW_CONTROL, + .fw_control.oper = fw_cmd, + }; + union pds_core_dev_comp comp; + unsigned long start_time; + unsigned long end_time; + int err; + + /* Ping on the status of the long running async install + * command. We get EAGAIN while the command is still + * running, else we get the final command status. + */ + start_time = jiffies; + end_time = start_time + (timeout * HZ); + do { + err = pdsc_devcmd(pdsc, &cmd, &comp, pdsc->devcmd_timeout); + msleep(20); + } while (time_before(jiffies, end_time) && + (err == -EAGAIN || err == -ETIMEDOUT)); + + if (err == -EAGAIN || err == -ETIMEDOUT) { + NL_SET_ERR_MSG_MOD(extack, "Firmware wait timed out"); + dev_err(pdsc->dev, "DEV_CMD firmware wait %s timed out\n", + label); + } else if (err) { + NL_SET_ERR_MSG_MOD(extack, "Firmware wait failed"); + } + + return err; +} + +int pdsc_firmware_update(struct pdsc *pdsc, const struct firmware *fw, + struct netlink_ext_ack *extack) +{ + u32 buf_sz, copy_sz, offset; + struct devlink *dl; + int next_interval; + u64 data_addr; + int err = 0; + int fw_slot; + + dev_info(pdsc->dev, "Installing firmware\n"); + + dl = priv_to_devlink(pdsc); + devlink_flash_update_status_notify(dl, "Preparing to flash", + NULL, 0, 0); + + buf_sz = sizeof(pdsc->cmd_regs->data); + + dev_dbg(pdsc->dev, + "downloading firmware - size %d part_sz %d nparts %lu\n", + (int)fw->size, buf_sz, DIV_ROUND_UP(fw->size, buf_sz)); + + offset = 0; + next_interval = 0; + data_addr = offsetof(struct pds_core_dev_cmd_regs, data); + while (offset < fw->size) { + if (offset >= next_interval) { + devlink_flash_update_status_notify(dl, "Downloading", + NULL, offset, + fw->size); + next_interval = offset + + (fw->size / PDSC_FW_INTERVAL_FRACTION); + } + + copy_sz = min_t(unsigned int, buf_sz, fw->size - offset); + mutex_lock(&pdsc->devcmd_lock); + memcpy_toio(&pdsc->cmd_regs->data, fw->data + offset, copy_sz); + err = pdsc_devcmd_fw_download_locked(pdsc, data_addr, + offset, copy_sz); + mutex_unlock(&pdsc->devcmd_lock); + if (err) { + dev_err(pdsc->dev, + "download failed offset 0x%x addr 0x%llx len 0x%x: %pe\n", + offset, data_addr, copy_sz, ERR_PTR(err)); + NL_SET_ERR_MSG_MOD(extack, "Segment download failed"); + goto err_out; + } + offset += copy_sz; + } + devlink_flash_update_status_notify(dl, "Downloading", NULL, + fw->size, fw->size); + + devlink_flash_update_timeout_notify(dl, "Installing", NULL, + PDSC_FW_INSTALL_TIMEOUT); + + fw_slot = pdsc_devcmd_fw_install(pdsc); + if (fw_slot < 0) { + err = fw_slot; + dev_err(pdsc->dev, "install failed: %pe\n", ERR_PTR(err)); + NL_SET_ERR_MSG_MOD(extack, "Failed to start firmware install"); + goto err_out; + } + + err = pdsc_fw_status_long_wait(pdsc, "Installing", + PDSC_FW_INSTALL_TIMEOUT, + PDS_CORE_FW_INSTALL_STATUS, + extack); + if (err) + goto err_out; + + devlink_flash_update_timeout_notify(dl, "Selecting", NULL, + PDSC_FW_SELECT_TIMEOUT); + + err = pdsc_devcmd_fw_activate(pdsc, fw_slot); + if (err) { + NL_SET_ERR_MSG_MOD(extack, "Failed to start firmware select"); + goto err_out; + } + + err = pdsc_fw_status_long_wait(pdsc, "Selecting", + PDSC_FW_SELECT_TIMEOUT, + PDS_CORE_FW_ACTIVATE_STATUS, + extack); + if (err) + goto err_out; + + dev_info(pdsc->dev, "Firmware update completed, slot %d\n", fw_slot); + +err_out: + if (err) + devlink_flash_update_status_notify(dl, "Flash failed", + NULL, 0, 0); + else + devlink_flash_update_status_notify(dl, "Flash done", + NULL, 0, 0); + return err; +} diff --git a/drivers/net/ethernet/amd/pds_core/main.c b/drivers/net/ethernet/amd/pds_core/main.c new file mode 100644 index 000000000000..e2d14b1ca471 --- /dev/null +++ b/drivers/net/ethernet/amd/pds_core/main.c @@ -0,0 +1,475 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include <linux/pci.h> + +#include <linux/pds/pds_common.h> + +#include "core.h" + +MODULE_DESCRIPTION(PDSC_DRV_DESCRIPTION); +MODULE_AUTHOR("Advanced Micro Devices, Inc"); +MODULE_LICENSE("GPL"); + +/* Supported devices */ +static const struct pci_device_id pdsc_id_table[] = { + { PCI_VDEVICE(PENSANDO, PCI_DEVICE_ID_PENSANDO_CORE_PF) }, + { PCI_VDEVICE(PENSANDO, PCI_DEVICE_ID_PENSANDO_VDPA_VF) }, + { 0, } /* end of table */ +}; +MODULE_DEVICE_TABLE(pci, pdsc_id_table); + +static void pdsc_wdtimer_cb(struct timer_list *t) +{ + struct pdsc *pdsc = from_timer(pdsc, t, wdtimer); + + dev_dbg(pdsc->dev, "%s: jiffies %ld\n", __func__, jiffies); + mod_timer(&pdsc->wdtimer, + round_jiffies(jiffies + pdsc->wdtimer_period)); + + queue_work(pdsc->wq, &pdsc->health_work); +} + +static void pdsc_unmap_bars(struct pdsc *pdsc) +{ + struct pdsc_dev_bar *bars = pdsc->bars; + unsigned int i; + + for (i = 0; i < PDS_CORE_BARS_MAX; i++) { + if (bars[i].vaddr) + pci_iounmap(pdsc->pdev, bars[i].vaddr); + } +} + +static int pdsc_map_bars(struct pdsc *pdsc) +{ + struct pdsc_dev_bar *bar = pdsc->bars; + struct pci_dev *pdev = pdsc->pdev; + struct device *dev = pdsc->dev; + struct pdsc_dev_bar *bars; + unsigned int i, j; + int num_bars = 0; + int err; + u32 sig; + + bars = pdsc->bars; + + /* Since the PCI interface in the hardware is configurable, + * we need to poke into all the bars to find the set we're + * expecting. + */ + for (i = 0, j = 0; i < PDS_CORE_BARS_MAX; i++) { + if (!(pci_resource_flags(pdev, i) & IORESOURCE_MEM)) + continue; + + bars[j].len = pci_resource_len(pdev, i); + bars[j].bus_addr = pci_resource_start(pdev, i); + bars[j].res_index = i; + + /* only map the whole bar 0 */ + if (j > 0) { + bars[j].vaddr = NULL; + } else { + bars[j].vaddr = pci_iomap(pdev, i, bars[j].len); + if (!bars[j].vaddr) { + dev_err(dev, "Cannot map BAR %d, aborting\n", i); + return -ENODEV; + } + } + + j++; + } + num_bars = j; + + /* BAR0: dev_cmd and interrupts */ + if (num_bars < 1) { + dev_err(dev, "No bars found\n"); + err = -EFAULT; + goto err_out; + } + + if (bar->len < PDS_CORE_BAR0_SIZE) { + dev_err(dev, "Resource bar size %lu too small\n", bar->len); + err = -EFAULT; + goto err_out; + } + + pdsc->info_regs = bar->vaddr + PDS_CORE_BAR0_DEV_INFO_REGS_OFFSET; + pdsc->cmd_regs = bar->vaddr + PDS_CORE_BAR0_DEV_CMD_REGS_OFFSET; + pdsc->intr_status = bar->vaddr + PDS_CORE_BAR0_INTR_STATUS_OFFSET; + pdsc->intr_ctrl = bar->vaddr + PDS_CORE_BAR0_INTR_CTRL_OFFSET; + + sig = ioread32(&pdsc->info_regs->signature); + if (sig != PDS_CORE_DEV_INFO_SIGNATURE) { + dev_err(dev, "Incompatible firmware signature %x", sig); + err = -EFAULT; + goto err_out; + } + + /* BAR1: doorbells */ + bar++; + if (num_bars < 2) { + dev_err(dev, "Doorbell bar missing\n"); + err = -EFAULT; + goto err_out; + } + + pdsc->db_pages = bar->vaddr; + pdsc->phy_db_pages = bar->bus_addr; + + return 0; + +err_out: + pdsc_unmap_bars(pdsc); + return err; +} + +void __iomem *pdsc_map_dbpage(struct pdsc *pdsc, int page_num) +{ + return pci_iomap_range(pdsc->pdev, + pdsc->bars[PDS_CORE_PCI_BAR_DBELL].res_index, + (u64)page_num << PAGE_SHIFT, PAGE_SIZE); +} + +static int pdsc_sriov_configure(struct pci_dev *pdev, int num_vfs) +{ + struct pdsc *pdsc = pci_get_drvdata(pdev); + struct device *dev = pdsc->dev; + int ret = 0; + + if (num_vfs > 0) { + pdsc->vfs = kcalloc(num_vfs, sizeof(struct pdsc_vf), + GFP_KERNEL); + if (!pdsc->vfs) + return -ENOMEM; + pdsc->num_vfs = num_vfs; + + ret = pci_enable_sriov(pdev, num_vfs); + if (ret) { + dev_err(dev, "Cannot enable SRIOV: %pe\n", + ERR_PTR(ret)); + goto no_vfs; + } + + return num_vfs; + } + +no_vfs: + pci_disable_sriov(pdev); + + kfree(pdsc->vfs); + pdsc->vfs = NULL; + pdsc->num_vfs = 0; + + return ret; +} + +static int pdsc_init_vf(struct pdsc *vf) +{ + struct devlink *dl; + struct pdsc *pf; + int err; + + pf = pdsc_get_pf_struct(vf->pdev); + if (IS_ERR_OR_NULL(pf)) + return PTR_ERR(pf) ?: -1; + + vf->vf_id = pci_iov_vf_id(vf->pdev); + + dl = priv_to_devlink(vf); + devl_lock(dl); + devl_register(dl); + devl_unlock(dl); + + pf->vfs[vf->vf_id].vf = vf; + err = pdsc_auxbus_dev_add(vf, pf); + if (err) { + devl_lock(dl); + devl_unregister(dl); + devl_unlock(dl); + } + + return err; +} + +static const struct devlink_health_reporter_ops pdsc_fw_reporter_ops = { + .name = "fw", + .diagnose = pdsc_fw_reporter_diagnose, +}; + +static const struct devlink_param pdsc_dl_params[] = { + DEVLINK_PARAM_GENERIC(ENABLE_VNET, + BIT(DEVLINK_PARAM_CMODE_RUNTIME), + pdsc_dl_enable_get, + pdsc_dl_enable_set, + pdsc_dl_enable_validate), +}; + +#define PDSC_WQ_NAME_LEN 24 + +static int pdsc_init_pf(struct pdsc *pdsc) +{ + struct devlink_health_reporter *hr; + char wq_name[PDSC_WQ_NAME_LEN]; + struct devlink *dl; + int err; + + pcie_print_link_status(pdsc->pdev); + + err = pci_request_regions(pdsc->pdev, PDS_CORE_DRV_NAME); + if (err) { + dev_err(pdsc->dev, "Cannot request PCI regions: %pe\n", + ERR_PTR(err)); + return err; + } + + err = pdsc_map_bars(pdsc); + if (err) + goto err_out_release_regions; + + /* General workqueue and timer, but don't start timer yet */ + snprintf(wq_name, sizeof(wq_name), "%s.%d", PDS_CORE_DRV_NAME, pdsc->uid); + pdsc->wq = create_singlethread_workqueue(wq_name); + INIT_WORK(&pdsc->health_work, pdsc_health_thread); + timer_setup(&pdsc->wdtimer, pdsc_wdtimer_cb, 0); + pdsc->wdtimer_period = PDSC_WATCHDOG_SECS * HZ; + + mutex_init(&pdsc->devcmd_lock); + mutex_init(&pdsc->config_lock); + spin_lock_init(&pdsc->adminq_lock); + + mutex_lock(&pdsc->config_lock); + set_bit(PDSC_S_FW_DEAD, &pdsc->state); + + err = pdsc_setup(pdsc, PDSC_SETUP_INIT); + if (err) + goto err_out_unmap_bars; + err = pdsc_start(pdsc); + if (err) + goto err_out_teardown; + + mutex_unlock(&pdsc->config_lock); + + dl = priv_to_devlink(pdsc); + devl_lock(dl); + err = devl_params_register(dl, pdsc_dl_params, + ARRAY_SIZE(pdsc_dl_params)); + if (err) { + dev_warn(pdsc->dev, "Failed to register devlink params: %pe\n", + ERR_PTR(err)); + goto err_out_unlock_dl; + } + + hr = devl_health_reporter_create(dl, &pdsc_fw_reporter_ops, 0, pdsc); + if (IS_ERR(hr)) { + dev_warn(pdsc->dev, "Failed to create fw reporter: %pe\n", hr); + err = PTR_ERR(hr); + goto err_out_unreg_params; + } + pdsc->fw_reporter = hr; + + devl_register(dl); + devl_unlock(dl); + + /* Lastly, start the health check timer */ + mod_timer(&pdsc->wdtimer, round_jiffies(jiffies + pdsc->wdtimer_period)); + + return 0; + +err_out_unreg_params: + devl_params_unregister(dl, pdsc_dl_params, + ARRAY_SIZE(pdsc_dl_params)); +err_out_unlock_dl: + devl_unlock(dl); + pdsc_stop(pdsc); +err_out_teardown: + pdsc_teardown(pdsc, PDSC_TEARDOWN_REMOVING); +err_out_unmap_bars: + mutex_unlock(&pdsc->config_lock); + del_timer_sync(&pdsc->wdtimer); + if (pdsc->wq) + destroy_workqueue(pdsc->wq); + mutex_destroy(&pdsc->config_lock); + mutex_destroy(&pdsc->devcmd_lock); + pci_free_irq_vectors(pdsc->pdev); + pdsc_unmap_bars(pdsc); +err_out_release_regions: + pci_release_regions(pdsc->pdev); + + return err; +} + +static const struct devlink_ops pdsc_dl_ops = { + .info_get = pdsc_dl_info_get, + .flash_update = pdsc_dl_flash_update, +}; + +static const struct devlink_ops pdsc_dl_vf_ops = { +}; + +static DEFINE_IDA(pdsc_ida); + +static int pdsc_probe(struct pci_dev *pdev, const struct pci_device_id *ent) +{ + struct device *dev = &pdev->dev; + const struct devlink_ops *ops; + struct devlink *dl; + struct pdsc *pdsc; + bool is_pf; + int err; + + is_pf = !pdev->is_virtfn; + ops = is_pf ? &pdsc_dl_ops : &pdsc_dl_vf_ops; + dl = devlink_alloc(ops, sizeof(struct pdsc), dev); + if (!dl) + return -ENOMEM; + pdsc = devlink_priv(dl); + + pdsc->pdev = pdev; + pdsc->dev = &pdev->dev; + set_bit(PDSC_S_INITING_DRIVER, &pdsc->state); + pci_set_drvdata(pdev, pdsc); + pdsc_debugfs_add_dev(pdsc); + + err = ida_alloc(&pdsc_ida, GFP_KERNEL); + if (err < 0) { + dev_err(pdsc->dev, "%s: id alloc failed: %pe\n", + __func__, ERR_PTR(err)); + goto err_out_free_devlink; + } + pdsc->uid = err; + + /* Query system for DMA addressing limitation for the device. */ + err = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(PDS_CORE_ADDR_LEN)); + if (err) { + dev_err(dev, "Unable to obtain 64-bit DMA for consistent allocations, aborting: %pe\n", + ERR_PTR(err)); + goto err_out_free_ida; + } + + err = pci_enable_device(pdev); + if (err) { + dev_err(dev, "Cannot enable PCI device: %pe\n", ERR_PTR(err)); + goto err_out_free_ida; + } + pci_set_master(pdev); + + if (is_pf) + err = pdsc_init_pf(pdsc); + else + err = pdsc_init_vf(pdsc); + if (err) { + dev_err(dev, "Cannot init device: %pe\n", ERR_PTR(err)); + goto err_out_clear_master; + } + + clear_bit(PDSC_S_INITING_DRIVER, &pdsc->state); + return 0; + +err_out_clear_master: + pci_clear_master(pdev); + pci_disable_device(pdev); +err_out_free_ida: + ida_free(&pdsc_ida, pdsc->uid); +err_out_free_devlink: + pdsc_debugfs_del_dev(pdsc); + devlink_free(dl); + + return err; +} + +static void pdsc_remove(struct pci_dev *pdev) +{ + struct pdsc *pdsc = pci_get_drvdata(pdev); + struct devlink *dl; + + /* Unhook the registrations first to be sure there + * are no requests while we're stopping. + */ + dl = priv_to_devlink(pdsc); + devl_lock(dl); + devl_unregister(dl); + if (!pdev->is_virtfn) { + if (pdsc->fw_reporter) { + devl_health_reporter_destroy(pdsc->fw_reporter); + pdsc->fw_reporter = NULL; + } + devl_params_unregister(dl, pdsc_dl_params, + ARRAY_SIZE(pdsc_dl_params)); + } + devl_unlock(dl); + + if (pdev->is_virtfn) { + struct pdsc *pf; + + pf = pdsc_get_pf_struct(pdsc->pdev); + if (!IS_ERR(pf)) { + pdsc_auxbus_dev_del(pdsc, pf); + pf->vfs[pdsc->vf_id].vf = NULL; + } + } else { + /* Remove the VFs and their aux_bus connections before other + * cleanup so that the clients can use the AdminQ to cleanly + * shut themselves down. + */ + pdsc_sriov_configure(pdev, 0); + + del_timer_sync(&pdsc->wdtimer); + if (pdsc->wq) + destroy_workqueue(pdsc->wq); + + mutex_lock(&pdsc->config_lock); + set_bit(PDSC_S_STOPPING_DRIVER, &pdsc->state); + + pdsc_stop(pdsc); + pdsc_teardown(pdsc, PDSC_TEARDOWN_REMOVING); + mutex_unlock(&pdsc->config_lock); + mutex_destroy(&pdsc->config_lock); + mutex_destroy(&pdsc->devcmd_lock); + + pci_free_irq_vectors(pdev); + pdsc_unmap_bars(pdsc); + pci_release_regions(pdev); + } + + pci_clear_master(pdev); + pci_disable_device(pdev); + + ida_free(&pdsc_ida, pdsc->uid); + pdsc_debugfs_del_dev(pdsc); + devlink_free(dl); +} + +static struct pci_driver pdsc_driver = { + .name = PDS_CORE_DRV_NAME, + .id_table = pdsc_id_table, + .probe = pdsc_probe, + .remove = pdsc_remove, + .sriov_configure = pdsc_sriov_configure, +}; + +void *pdsc_get_pf_struct(struct pci_dev *vf_pdev) +{ + return pci_iov_get_pf_drvdata(vf_pdev, &pdsc_driver); +} +EXPORT_SYMBOL_GPL(pdsc_get_pf_struct); + +static int __init pdsc_init_module(void) +{ + if (strcmp(KBUILD_MODNAME, PDS_CORE_DRV_NAME)) + return -EINVAL; + + pdsc_debugfs_create(); + return pci_register_driver(&pdsc_driver); +} + +static void __exit pdsc_cleanup_module(void) +{ + pci_unregister_driver(&pdsc_driver); + pdsc_debugfs_destroy(); +} + +module_init(pdsc_init_module); +module_exit(pdsc_cleanup_module); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index 8fb9d1bbe56f..dcd9367f05af 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -2361,7 +2361,7 @@ static int bnxt_async_event_process(struct bnxt *bp, case ASYNC_EVENT_CMPL_EVENT_ID_PHC_UPDATE: { switch (BNXT_EVENT_PHC_EVENT_TYPE(data1)) { case ASYNC_EVENT_CMPL_PHC_UPDATE_EVENT_DATA1_FLAGS_PHC_RTC_UPDATE: - if (bp->fw_cap & BNXT_FW_CAP_PTP_RTC) { + if (BNXT_PTP_USE_RTC(bp)) { struct bnxt_ptp_cfg *ptp = bp->ptp_cfg; u64 ns; @@ -7601,7 +7601,7 @@ static int __bnxt_hwrm_ptp_qcfg(struct bnxt *bp) u8 flags; int rc; - if (bp->hwrm_spec_code < 0x10801) { + if (bp->hwrm_spec_code < 0x10801 || !BNXT_CHIP_P5_THOR(bp)) { rc = -ENODEV; goto no_ptp; } diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index e7b5e28ee29f..852eb449ccae 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -304,7 +304,7 @@ void bnxt_rdma_aux_device_uninit(struct bnxt *bp) struct auxiliary_device *adev; /* Skip if no auxiliary device init was done. */ - if (!(bp->flags & BNXT_FLAG_ROCE_CAP)) + if (!bp->aux_priv) return; aux_priv = bp->aux_priv; @@ -324,6 +324,7 @@ static void bnxt_aux_dev_release(struct device *dev) bp->edev = NULL; kfree(aux_priv->edev); kfree(aux_priv); + bp->aux_priv = NULL; } static void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp) @@ -359,19 +360,18 @@ void bnxt_rdma_aux_device_init(struct bnxt *bp) if (!(bp->flags & BNXT_FLAG_ROCE_CAP)) return; - bp->aux_priv = kzalloc(sizeof(*bp->aux_priv), GFP_KERNEL); - if (!bp->aux_priv) + aux_priv = kzalloc(sizeof(*bp->aux_priv), GFP_KERNEL); + if (!aux_priv) goto exit; - bp->aux_priv->id = ida_alloc(&bnxt_aux_dev_ids, GFP_KERNEL); - if (bp->aux_priv->id < 0) { + aux_priv->id = ida_alloc(&bnxt_aux_dev_ids, GFP_KERNEL); + if (aux_priv->id < 0) { netdev_warn(bp->dev, "ida alloc failed for ROCE auxiliary device\n"); - kfree(bp->aux_priv); + kfree(aux_priv); goto exit; } - aux_priv = bp->aux_priv; aux_dev = &aux_priv->aux_dev; aux_dev->id = aux_priv->id; aux_dev->name = "rdma"; @@ -380,10 +380,11 @@ void bnxt_rdma_aux_device_init(struct bnxt *bp) rc = auxiliary_device_init(aux_dev); if (rc) { - ida_free(&bnxt_aux_dev_ids, bp->aux_priv->id); - kfree(bp->aux_priv); + ida_free(&bnxt_aux_dev_ids, aux_priv->id); + kfree(aux_priv); goto exit; } + bp->aux_priv = aux_priv; /* From this point, all cleanup will happen via the .release callback & * any error unwinding will need to include a call to diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_tc_flower.c b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_tc_flower.c index dd9be229819a..d3541159487d 100644 --- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_tc_flower.c +++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_tc_flower.c @@ -1135,7 +1135,7 @@ void cxgb4_cleanup_tc_flower(struct adapter *adap) return; if (adap->flower_stats_timer.function) - del_timer_sync(&adap->flower_stats_timer); + timer_shutdown_sync(&adap->flower_stats_timer); cancel_work_sync(&adap->flower_stats_work); rhashtable_destroy(&adap->flower_tbl); adap->tc_flower_initialized = false; diff --git a/drivers/net/ethernet/freescale/enetc/enetc.c b/drivers/net/ethernet/freescale/enetc/enetc.c index 41c194c1672d..3c4fa26f0f9b 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc.c +++ b/drivers/net/ethernet/freescale/enetc/enetc.c @@ -25,23 +25,12 @@ void enetc_port_mac_wr(struct enetc_si *si, u32 reg, u32 val) } EXPORT_SYMBOL_GPL(enetc_port_mac_wr); -void enetc_set_ptcfpr(struct enetc_hw *hw, unsigned long preemptible_tcs) +static void enetc_change_preemptible_tcs(struct enetc_ndev_priv *priv, + u8 preemptible_tcs) { - u32 val; - int tc; - - for (tc = 0; tc < 8; tc++) { - val = enetc_port_rd(hw, ENETC_PTCFPR(tc)); - - if (preemptible_tcs & BIT(tc)) - val |= ENETC_PTCFPR_FPE; - else - val &= ~ENETC_PTCFPR_FPE; - - enetc_port_wr(hw, ENETC_PTCFPR(tc), val); - } + priv->preemptible_tcs = preemptible_tcs; + enetc_mm_commit_preemptible_tcs(priv); } -EXPORT_SYMBOL_GPL(enetc_set_ptcfpr); static int enetc_num_stack_tx_queues(struct enetc_ndev_priv *priv) { @@ -2659,7 +2648,7 @@ static void enetc_reset_tc_mqprio(struct net_device *ndev) enetc_debug_tx_ring_prios(priv); - enetc_set_ptcfpr(hw, 0); + enetc_change_preemptible_tcs(priv, 0); } int enetc_setup_tc_mqprio(struct net_device *ndev, void *type_data) @@ -2714,7 +2703,7 @@ int enetc_setup_tc_mqprio(struct net_device *ndev, void *type_data) enetc_debug_tx_ring_prios(priv); - enetc_set_ptcfpr(hw, mqprio->preemptible_tcs); + enetc_change_preemptible_tcs(priv, mqprio->preemptible_tcs); return 0; diff --git a/drivers/net/ethernet/freescale/enetc/enetc.h b/drivers/net/ethernet/freescale/enetc/enetc.h index 143078a9ef16..c97a8e3d7a7f 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc.h +++ b/drivers/net/ethernet/freescale/enetc/enetc.h @@ -355,6 +355,9 @@ struct enetc_ndev_priv { u16 rx_bd_count, tx_bd_count; u16 msg_enable; + + u8 preemptible_tcs; + enum enetc_active_offloads active_offloads; u32 speed; /* store speed for compare update pspeed */ @@ -433,6 +436,7 @@ int enetc_xdp_xmit(struct net_device *ndev, int num_frames, /* ethtool */ void enetc_set_ethtool_ops(struct net_device *ndev); void enetc_mm_link_state_update(struct enetc_ndev_priv *priv, bool link); +void enetc_mm_commit_preemptible_tcs(struct enetc_ndev_priv *priv); /* control buffer descriptor ring (CBDR) */ int enetc_setup_cbdr(struct device *dev, struct enetc_hw *hw, int bd_count, @@ -486,7 +490,6 @@ static inline void enetc_cbd_free_data_mem(struct enetc_si *si, int size, void enetc_reset_ptcmsdur(struct enetc_hw *hw); void enetc_set_ptcmsdur(struct enetc_hw *hw, u32 *queue_max_sdu); -void enetc_set_ptcfpr(struct enetc_hw *hw, unsigned long preemptible_tcs); #ifdef CONFIG_FSL_ENETC_QOS int enetc_qos_query_caps(struct net_device *ndev, void *type_data); diff --git a/drivers/net/ethernet/freescale/enetc/enetc_ethtool.c b/drivers/net/ethernet/freescale/enetc/enetc_ethtool.c index 838750a03cf6..e993ed04ab57 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc_ethtool.c +++ b/drivers/net/ethernet/freescale/enetc/enetc_ethtool.c @@ -32,6 +32,12 @@ static const u32 enetc_port_regs[] = { ENETC_PM0_CMD_CFG, ENETC_PM0_MAXFRM, ENETC_PM0_IF_MODE }; +static const u32 enetc_port_mm_regs[] = { + ENETC_MMCSR, ENETC_PFPMR, ENETC_PTCFPR(0), ENETC_PTCFPR(1), + ENETC_PTCFPR(2), ENETC_PTCFPR(3), ENETC_PTCFPR(4), ENETC_PTCFPR(5), + ENETC_PTCFPR(6), ENETC_PTCFPR(7), +}; + static int enetc_get_reglen(struct net_device *ndev) { struct enetc_ndev_priv *priv = netdev_priv(ndev); @@ -45,6 +51,9 @@ static int enetc_get_reglen(struct net_device *ndev) if (hw->port) len += ARRAY_SIZE(enetc_port_regs); + if (hw->port && !!(priv->si->hw_features & ENETC_SI_F_QBU)) + len += ARRAY_SIZE(enetc_port_mm_regs); + len *= sizeof(u32) * 2; /* store 2 entries per reg: addr and value */ return len; @@ -90,6 +99,14 @@ static void enetc_get_regs(struct net_device *ndev, struct ethtool_regs *regs, *buf++ = addr; *buf++ = enetc_rd(hw, addr); } + + if (priv->si->hw_features & ENETC_SI_F_QBU) { + for (i = 0; i < ARRAY_SIZE(enetc_port_mm_regs); i++) { + addr = ENETC_PORT_BASE + enetc_port_mm_regs[i]; + *buf++ = addr; + *buf++ = enetc_rd(hw, addr); + } + } } static const struct { @@ -976,7 +993,9 @@ static int enetc_get_mm(struct net_device *ndev, struct ethtool_mm_state *state) lafs = ENETC_MMCSR_GET_LAFS(val); state->rx_min_frag_size = ethtool_mm_frag_size_add_to_min(lafs); state->tx_enabled = !!(val & ENETC_MMCSR_LPE); /* mirror of MMCSR_ME */ - state->tx_active = !!(val & ENETC_MMCSR_LPA); + state->tx_active = state->tx_enabled && + (state->verify_status == ETHTOOL_MM_VERIFY_STATUS_SUCCEEDED || + state->verify_status == ETHTOOL_MM_VERIFY_STATUS_DISABLED); state->verify_enabled = !(val & ENETC_MMCSR_VDIS); state->verify_time = ENETC_MMCSR_GET_VT(val); /* A verifyTime of 128 ms would exceed the 7 bit width @@ -989,6 +1008,64 @@ static int enetc_get_mm(struct net_device *ndev, struct ethtool_mm_state *state) return 0; } +static int enetc_mm_wait_tx_active(struct enetc_hw *hw, int verify_time) +{ + int timeout = verify_time * USEC_PER_MSEC * ENETC_MM_VERIFY_RETRIES; + u32 val; + + /* This will time out after the standard value of 3 verification + * attempts. To not sleep forever, it relies on a non-zero verify_time, + * guarantee which is provided by the ethtool nlattr policy. + */ + return read_poll_timeout(enetc_port_rd, val, + ENETC_MMCSR_GET_VSTS(val) == 3, + ENETC_MM_VERIFY_SLEEP_US, timeout, + true, hw, ENETC_MMCSR); +} + +static void enetc_set_ptcfpr(struct enetc_hw *hw, u8 preemptible_tcs) +{ + u32 val; + int tc; + + for (tc = 0; tc < 8; tc++) { + val = enetc_port_rd(hw, ENETC_PTCFPR(tc)); + + if (preemptible_tcs & BIT(tc)) + val |= ENETC_PTCFPR_FPE; + else + val &= ~ENETC_PTCFPR_FPE; + + enetc_port_wr(hw, ENETC_PTCFPR(tc), val); + } +} + +/* ENETC does not have an IRQ to notify changes to the MAC Merge TX status + * (active/inactive), but the preemptible traffic classes should only be + * committed to hardware once TX is active. Resort to polling. + */ +void enetc_mm_commit_preemptible_tcs(struct enetc_ndev_priv *priv) +{ + struct enetc_hw *hw = &priv->si->hw; + u8 preemptible_tcs = 0; + u32 val; + int err; + + val = enetc_port_rd(hw, ENETC_MMCSR); + if (!(val & ENETC_MMCSR_ME)) + goto out; + + if (!(val & ENETC_MMCSR_VDIS)) { + err = enetc_mm_wait_tx_active(hw, ENETC_MMCSR_GET_VT(val)); + if (err) + goto out; + } + + preemptible_tcs = priv->preemptible_tcs; +out: + enetc_set_ptcfpr(hw, preemptible_tcs); +} + /* FIXME: Workaround for the link partner's verification failing if ENETC * priorly received too much express traffic. The documentation doesn't * suggest this is needed. @@ -1041,10 +1118,13 @@ static int enetc_set_mm(struct net_device *ndev, struct ethtool_mm_cfg *cfg, else priv->active_offloads &= ~ENETC_F_QBU; - /* If link is up, enable MAC Merge right away */ - if (!!(priv->active_offloads & ENETC_F_QBU) && - !(val & ENETC_MMCSR_LINK_FAIL)) - val |= ENETC_MMCSR_ME; + /* If link is up, enable/disable MAC Merge right away */ + if (!(val & ENETC_MMCSR_LINK_FAIL)) { + if (!!(priv->active_offloads & ENETC_F_QBU)) + val |= ENETC_MMCSR_ME; + else + val &= ~ENETC_MMCSR_ME; + } val &= ~ENETC_MMCSR_VT_MASK; val |= ENETC_MMCSR_VT(cfg->verify_time); @@ -1056,6 +1136,8 @@ static int enetc_set_mm(struct net_device *ndev, struct ethtool_mm_cfg *cfg, enetc_restart_emac_rx(priv->si); + enetc_mm_commit_preemptible_tcs(priv); + mutex_unlock(&priv->mm_lock); return 0; @@ -1089,6 +1171,8 @@ void enetc_mm_link_state_update(struct enetc_ndev_priv *priv, bool link) enetc_port_wr(hw, ENETC_MMCSR, val); + enetc_mm_commit_preemptible_tcs(priv); + mutex_unlock(&priv->mm_lock); } EXPORT_SYMBOL_GPL(enetc_mm_link_state_update); diff --git a/drivers/net/ethernet/freescale/enetc/enetc_hw.h b/drivers/net/ethernet/freescale/enetc/enetc_hw.h index 36bb2d6d5658..1619943fb263 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc_hw.h +++ b/drivers/net/ethernet/freescale/enetc/enetc_hw.h @@ -3,6 +3,9 @@ #include <linux/bitops.h> +#define ENETC_MM_VERIFY_SLEEP_US USEC_PER_MSEC +#define ENETC_MM_VERIFY_RETRIES 3 + /* ENETC device IDs */ #define ENETC_DEV_ID_PF 0xe100 #define ENETC_DEV_ID_VF 0xef00 diff --git a/drivers/net/ethernet/intel/e1000e/netdev.c b/drivers/net/ethernet/intel/e1000e/netdev.c index 6f5c16aebcbf..bd7ef59b1f2e 100644 --- a/drivers/net/ethernet/intel/e1000e/netdev.c +++ b/drivers/net/ethernet/intel/e1000e/netdev.c @@ -5287,31 +5287,6 @@ static void e1000_watchdog_task(struct work_struct *work) ew32(TARC(0), tarc0); } - /* disable TSO for pcie and 10/100 speeds, to avoid - * some hardware issues - */ - if (!(adapter->flags & FLAG_TSO_FORCE)) { - switch (adapter->link_speed) { - case SPEED_10: - case SPEED_100: - e_info("10/100 speed: disabling TSO\n"); - netdev->features &= ~NETIF_F_TSO; - netdev->features &= ~NETIF_F_TSO6; - break; - case SPEED_1000: - netdev->features |= NETIF_F_TSO; - netdev->features |= NETIF_F_TSO6; - break; - default: - /* oops */ - break; - } - if (hw->mac.type == e1000_pch_spt) { - netdev->features &= ~NETIF_F_TSO; - netdev->features &= ~NETIF_F_TSO6; - } - } - /* enable transmits in the hardware, need to do this * after setting TARC(0) */ @@ -7525,6 +7500,32 @@ static int e1000_probe(struct pci_dev *pdev, const struct pci_device_id *ent) NETIF_F_RXCSUM | NETIF_F_HW_CSUM); + /* disable TSO for pcie and 10/100 speeds to avoid + * some hardware issues and for i219 to fix transfer + * speed being capped at 60% + */ + if (!(adapter->flags & FLAG_TSO_FORCE)) { + switch (adapter->link_speed) { + case SPEED_10: + case SPEED_100: + e_info("10/100 speed: disabling TSO\n"); + netdev->features &= ~NETIF_F_TSO; + netdev->features &= ~NETIF_F_TSO6; + break; + case SPEED_1000: + netdev->features |= NETIF_F_TSO; + netdev->features |= NETIF_F_TSO6; + break; + default: + /* oops */ + break; + } + if (hw->mac.type == e1000_pch_spt) { + netdev->features &= ~NETIF_F_TSO; + netdev->features &= ~NETIF_F_TSO6; + } + } + /* Set user-changeable features (subset of all device features) */ netdev->hw_features = netdev->features; netdev->hw_features |= NETIF_F_RXFCS; diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c index c8ff5675b29d..b847bd105b16 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_main.c +++ b/drivers/net/ethernet/intel/i40e/i40e_main.c @@ -11072,8 +11072,11 @@ static void i40e_rebuild(struct i40e_pf *pf, bool reinit, bool lock_acquired) pf->hw.aq.asq_last_status)); } /* reinit the misc interrupt */ - if (pf->flags & I40E_FLAG_MSIX_ENABLED) + if (pf->flags & I40E_FLAG_MSIX_ENABLED) { ret = i40e_setup_misc_vector(pf); + if (ret) + goto end_unlock; + } /* Add a filter to drop all Flow control frames from any VSI from being * transmitted. By doing so we stop a malicious VF from sending out @@ -14147,15 +14150,15 @@ static int i40e_add_vsi(struct i40e_vsi *vsi) vsi->id = ctxt.vsi_number; } - vsi->active_filters = 0; - clear_bit(__I40E_VSI_OVERFLOW_PROMISC, vsi->state); spin_lock_bh(&vsi->mac_filter_hash_lock); + vsi->active_filters = 0; /* If macvlan filters already exist, force them to get loaded */ hash_for_each_safe(vsi->mac_filter_hash, bkt, h, f, hlist) { f->state = I40E_FILTER_NEW; f_count++; } spin_unlock_bh(&vsi->mac_filter_hash_lock); + clear_bit(__I40E_VSI_OVERFLOW_PROMISC, vsi->state); if (f_count) { vsi->flags |= I40E_VSI_FLAG_FILTER_CHANGED; diff --git a/drivers/net/ethernet/mediatek/mtk_ppe.c b/drivers/net/ethernet/mediatek/mtk_ppe.c index dd9581334b05..9129821f3ab8 100644 --- a/drivers/net/ethernet/mediatek/mtk_ppe.c +++ b/drivers/net/ethernet/mediatek/mtk_ppe.c @@ -601,6 +601,7 @@ __mtk_foe_entry_commit(struct mtk_ppe *ppe, struct mtk_foe_entry *entry, struct mtk_eth *eth = ppe->eth; u16 timestamp = mtk_eth_timestamp(eth); struct mtk_foe_entry *hwe; + u32 val; if (MTK_HAS_CAPS(eth->soc->caps, MTK_NETSYS_V2)) { entry->ib1 &= ~MTK_FOE_IB1_BIND_TIMESTAMP_V2; @@ -617,8 +618,13 @@ __mtk_foe_entry_commit(struct mtk_ppe *ppe, struct mtk_foe_entry *entry, wmb(); hwe->ib1 = entry->ib1; - if (ppe->accounting) - *mtk_foe_entry_ib2(eth, hwe) |= MTK_FOE_IB2_MIB_CNT; + if (ppe->accounting) { + if (MTK_HAS_CAPS(eth->soc->caps, MTK_NETSYS_V2)) + val = MTK_FOE_IB2_MIB_CNT_V2; + else + val = MTK_FOE_IB2_MIB_CNT; + *mtk_foe_entry_ib2(eth, hwe) |= val; + } dma_wmb(); diff --git a/drivers/net/ethernet/mediatek/mtk_ppe.h b/drivers/net/ethernet/mediatek/mtk_ppe.h index e1aab2e8e262..e51de31a52ec 100644 --- a/drivers/net/ethernet/mediatek/mtk_ppe.h +++ b/drivers/net/ethernet/mediatek/mtk_ppe.h @@ -55,9 +55,10 @@ enum { #define MTK_FOE_IB2_PSE_QOS BIT(4) #define MTK_FOE_IB2_DEST_PORT GENMASK(7, 5) #define MTK_FOE_IB2_MULTICAST BIT(8) +#define MTK_FOE_IB2_MIB_CNT BIT(10) #define MTK_FOE_IB2_WDMA_QID2 GENMASK(13, 12) -#define MTK_FOE_IB2_MIB_CNT BIT(15) +#define MTK_FOE_IB2_MIB_CNT_V2 BIT(15) #define MTK_FOE_IB2_WDMA_DEVIDX BIT(16) #define MTK_FOE_IB2_WDMA_WINFO BIT(17) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile index ca3c66cd47ec..ddf1e352f51d 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile +++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile @@ -16,7 +16,7 @@ mlx5_core-y := main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \ transobj.o vport.o sriov.o fs_cmd.o fs_core.o pci_irq.o \ fs_counters.o fs_ft_pool.o rl.o lag/debugfs.o lag/lag.o dev.o events.o wq.o lib/gid.o \ lib/devcom.o lib/pci_vsc.o lib/dm.o lib/fs_ttc.o diag/fs_tracepoint.o \ - diag/fw_tracer.o diag/crdump.o devlink.o diag/rsc_dump.o \ + diag/fw_tracer.o diag/crdump.o devlink.o diag/rsc_dump.o diag/reporter_vnic.o \ fw_reset.o qos.o lib/tout.o lib/aso.o # @@ -69,7 +69,7 @@ mlx5_core-$(CONFIG_MLX5_TC_SAMPLE) += en/tc/sample.o # mlx5_core-$(CONFIG_MLX5_ESWITCH) += eswitch.o eswitch_offloads.o eswitch_offloads_termtbl.o \ ecpf.o rdma.o esw/legacy.o \ - esw/debugfs.o esw/devlink_port.o esw/vporttbl.o esw/qos.o + esw/devlink_port.o esw/vporttbl.o esw/qos.o mlx5_core-$(CONFIG_MLX5_ESWITCH) += esw/acl/helper.o \ esw/acl/egress_lgcy.o esw/acl/egress_ofld.o \ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/dev.c b/drivers/net/ethernet/mellanox/mlx5/core/dev.c index e7739acc926e..1b33533b15de 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/dev.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/dev.c @@ -60,9 +60,6 @@ bool mlx5_eth_supported(struct mlx5_core_dev *dev) if (!IS_ENABLED(CONFIG_MLX5_CORE_EN)) return false; - if (mlx5_core_is_management_pf(dev)) - return false; - if (MLX5_CAP_GEN(dev, port_type) != MLX5_CAP_PORT_TYPE_ETH) return false; @@ -191,9 +188,6 @@ bool mlx5_rdma_supported(struct mlx5_core_dev *dev) if (!IS_ENABLED(CONFIG_MLX5_INFINIBAND)) return false; - if (mlx5_core_is_management_pf(dev)) - return false; - if (dev->priv.flags & MLX5_PRIV_FLAGS_DISABLE_IB_ADEV) return false; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/diag/reporter_vnic.c b/drivers/net/ethernet/mellanox/mlx5/core/diag/reporter_vnic.c new file mode 100644 index 000000000000..9114661cd967 --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/reporter_vnic.c @@ -0,0 +1,125 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. */ + +#include "reporter_vnic.h" +#include "devlink.h" + +#define VNIC_ENV_GET64(vnic_env_stats, c) \ + MLX5_GET64(query_vnic_env_out, (vnic_env_stats)->query_vnic_env_out, \ + vport_env.c) + +struct mlx5_vnic_diag_stats { + __be64 query_vnic_env_out[MLX5_ST_SZ_QW(query_vnic_env_out)]; +}; + +int mlx5_reporter_vnic_diagnose_counters(struct mlx5_core_dev *dev, + struct devlink_fmsg *fmsg, + u16 vport_num, bool other_vport) +{ + u32 in[MLX5_ST_SZ_DW(query_vnic_env_in)] = {}; + struct mlx5_vnic_diag_stats vnic; + int err; + + MLX5_SET(query_vnic_env_in, in, opcode, MLX5_CMD_OP_QUERY_VNIC_ENV); + MLX5_SET(query_vnic_env_in, in, vport_number, vport_num); + MLX5_SET(query_vnic_env_in, in, other_vport, !!other_vport); + + err = mlx5_cmd_exec_inout(dev, query_vnic_env, in, &vnic.query_vnic_env_out); + if (err) + return err; + + err = devlink_fmsg_pair_nest_start(fmsg, "vNIC env counters"); + if (err) + return err; + + err = devlink_fmsg_obj_nest_start(fmsg); + if (err) + return err; + + err = devlink_fmsg_u64_pair_put(fmsg, "total_error_queues", + VNIC_ENV_GET64(&vnic, total_error_queues)); + if (err) + return err; + + err = devlink_fmsg_u64_pair_put(fmsg, "send_queue_priority_update_flow", + VNIC_ENV_GET64(&vnic, send_queue_priority_update_flow)); + if (err) + return err; + + err = devlink_fmsg_u64_pair_put(fmsg, "comp_eq_overrun", + VNIC_ENV_GET64(&vnic, comp_eq_overrun)); + if (err) + return err; + + err = devlink_fmsg_u64_pair_put(fmsg, "async_eq_overrun", + VNIC_ENV_GET64(&vnic, async_eq_overrun)); + if (err) + return err; + + err = devlink_fmsg_u64_pair_put(fmsg, "cq_overrun", + VNIC_ENV_GET64(&vnic, cq_overrun)); + if (err) + return err; + + err = devlink_fmsg_u64_pair_put(fmsg, "invalid_command", + VNIC_ENV_GET64(&vnic, invalid_command)); + if (err) + return err; + + err = devlink_fmsg_u64_pair_put(fmsg, "quota_exceeded_command", + VNIC_ENV_GET64(&vnic, quota_exceeded_command)); + if (err) + return err; + + err = devlink_fmsg_u64_pair_put(fmsg, "nic_receive_steering_discard", + VNIC_ENV_GET64(&vnic, nic_receive_steering_discard)); + if (err) + return err; + + err = devlink_fmsg_obj_nest_end(fmsg); + if (err) + return err; + + err = devlink_fmsg_pair_nest_end(fmsg); + if (err) + return err; + + return 0; +} + +static int mlx5_reporter_vnic_diagnose(struct devlink_health_reporter *reporter, + struct devlink_fmsg *fmsg, + struct netlink_ext_ack *extack) +{ + struct mlx5_core_dev *dev = devlink_health_reporter_priv(reporter); + + return mlx5_reporter_vnic_diagnose_counters(dev, fmsg, 0, false); +} + +static const struct devlink_health_reporter_ops mlx5_reporter_vnic_ops = { + .name = "vnic", + .diagnose = mlx5_reporter_vnic_diagnose, +}; + +void mlx5_reporter_vnic_create(struct mlx5_core_dev *dev) +{ + struct mlx5_core_health *health = &dev->priv.health; + struct devlink *devlink = priv_to_devlink(dev); + + health->vnic_reporter = + devlink_health_reporter_create(devlink, + &mlx5_reporter_vnic_ops, + 0, dev); + if (IS_ERR(health->vnic_reporter)) + mlx5_core_warn(dev, + "Failed to create vnic reporter, err = %ld\n", + PTR_ERR(health->vnic_reporter)); +} + +void mlx5_reporter_vnic_destroy(struct mlx5_core_dev *dev) +{ + struct mlx5_core_health *health = &dev->priv.health; + + if (!IS_ERR_OR_NULL(health->vnic_reporter)) + devlink_health_reporter_destroy(health->vnic_reporter); +} diff --git a/drivers/net/ethernet/mellanox/mlx5/core/diag/reporter_vnic.h b/drivers/net/ethernet/mellanox/mlx5/core/diag/reporter_vnic.h new file mode 100644 index 000000000000..eba87a39e9b1 --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/reporter_vnic.h @@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB + * Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. + */ +#ifndef __MLX5_REPORTER_VNIC_H +#define __MLX5_REPORTER_VNIC_H + +#include "mlx5_core.h" + +void mlx5_reporter_vnic_create(struct mlx5_core_dev *dev); +void mlx5_reporter_vnic_destroy(struct mlx5_core_dev *dev); + +int mlx5_reporter_vnic_diagnose_counters(struct mlx5_core_dev *dev, + struct devlink_fmsg *fmsg, + u16 vport_num, bool other_vport); + +#endif /* __MLX5_REPORTER_VNIC_H */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/ecpf.c b/drivers/net/ethernet/mellanox/mlx5/core/ecpf.c index 7c9c4e40c019..d000236ddbac 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/ecpf.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/ecpf.c @@ -75,10 +75,6 @@ int mlx5_ec_init(struct mlx5_core_dev *dev) if (!mlx5_core_is_ecpf(dev)) return 0; - /* Management PF don't have a peer PF */ - if (mlx5_core_is_management_pf(dev)) - return 0; - return mlx5_host_pf_init(dev); } @@ -89,10 +85,6 @@ void mlx5_ec_cleanup(struct mlx5_core_dev *dev) if (!mlx5_core_is_ecpf(dev)) return; - /* Management PF don't have a peer PF */ - if (mlx5_core_is_management_pf(dev)) - return; - mlx5_host_pf_cleanup(dev); err = mlx5_wait_for_pages(dev, &dev->priv.page_counters[MLX5_HOST_PF]); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c index 5fd609d1120e..55b38544422f 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c @@ -252,6 +252,8 @@ static void mlx5e_ipsec_init_macs(struct mlx5e_ipsec_sa_entry *sa_entry, struct net_device *netdev; struct neighbour *n; u8 addr[ETH_ALEN]; + const void *pkey; + u8 *dst, *src; if (attrs->mode != XFRM_MODE_TUNNEL || attrs->type != XFRM_DEV_OFFLOAD_PACKET) @@ -262,36 +264,31 @@ static void mlx5e_ipsec_init_macs(struct mlx5e_ipsec_sa_entry *sa_entry, mlx5_query_mac_address(mdev, addr); switch (attrs->dir) { case XFRM_DEV_OFFLOAD_IN: - ether_addr_copy(attrs->dmac, addr); - n = neigh_lookup(&arp_tbl, &attrs->saddr.a4, netdev); - if (!n) { - n = neigh_create(&arp_tbl, &attrs->saddr.a4, netdev); - if (IS_ERR(n)) - return; - neigh_event_send(n, NULL); - attrs->drop = true; - break; - } - neigh_ha_snapshot(addr, n, netdev); - ether_addr_copy(attrs->smac, addr); + src = attrs->dmac; + dst = attrs->smac; + pkey = &attrs->saddr.a4; break; case XFRM_DEV_OFFLOAD_OUT: - ether_addr_copy(attrs->smac, addr); - n = neigh_lookup(&arp_tbl, &attrs->daddr.a4, netdev); - if (!n) { - n = neigh_create(&arp_tbl, &attrs->daddr.a4, netdev); - if (IS_ERR(n)) - return; - neigh_event_send(n, NULL); - attrs->drop = true; - break; - } - neigh_ha_snapshot(addr, n, netdev); - ether_addr_copy(attrs->dmac, addr); + src = attrs->smac; + dst = attrs->dmac; + pkey = &attrs->daddr.a4; break; default: return; } + + ether_addr_copy(src, addr); + n = neigh_lookup(&arp_tbl, pkey, netdev); + if (!n) { + n = neigh_create(&arp_tbl, pkey, netdev); + if (IS_ERR(n)) + return; + neigh_event_send(n, NULL); + attrs->drop = true; + } else { + neigh_ha_snapshot(addr, n, netdev); + ether_addr_copy(dst, addr); + } neigh_release(n); } @@ -708,11 +705,12 @@ err_hw_ctx: release_dwork: kfree(sa_entry->dwork); release_work: - kfree(sa_entry->work->data); + if (sa_entry->work) + kfree(sa_entry->work->data); kfree(sa_entry->work); err_xfrm: kfree(sa_entry); - NL_SET_ERR_MSG_MOD(extack, "Device failed to offload this policy"); + NL_SET_ERR_MSG_WEAK_MOD(extack, "Device failed to offload this state"); return err; } @@ -752,7 +750,8 @@ static void mlx5e_xfrm_free_state(struct xfrm_state *x) mlx5e_accel_ipsec_fs_del_rule(sa_entry); mlx5_ipsec_free_sa_ctx(sa_entry); kfree(sa_entry->dwork); - kfree(sa_entry->work->data); + if (sa_entry->work) + kfree(sa_entry->work->data); kfree(sa_entry->work); sa_entry_free: kfree(sa_entry); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.h index f7f7c09d2b32..4e9887171508 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.h @@ -287,7 +287,7 @@ static inline bool addr6_all_zero(__be32 *addr6) { static const __be32 zaddr6[4] = {}; - return !memcmp(addr6, zaddr6, sizeof(*zaddr6)); + return !memcmp(addr6, zaddr6, sizeof(zaddr6)); } #else static inline void mlx5e_ipsec_init(struct mlx5e_priv *priv) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_fs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_fs.c index 5a8fcd30fcb1..dbe87bf89c0d 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_fs.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_fs.c @@ -1252,16 +1252,16 @@ static int tx_add_policy(struct mlx5e_ipsec_pol_entry *pol_entry) setup_fte_no_frags(spec); setup_fte_upper_proto_match(spec, &attrs->upspec); - if (attrs->reqid) { + switch (attrs->action) { + case XFRM_POLICY_ALLOW: + flow_act.action |= MLX5_FLOW_CONTEXT_ACTION_FWD_DEST; + if (!attrs->reqid) + break; + err = setup_modify_header(mdev, attrs->reqid, XFRM_DEV_OFFLOAD_OUT, &flow_act); if (err) goto err_mod_header; - } - - switch (attrs->action) { - case XFRM_POLICY_ALLOW: - flow_act.action |= MLX5_FLOW_CONTEXT_ACTION_FWD_DEST; break; case XFRM_POLICY_BLOCK: flow_act.action |= MLX5_FLOW_CONTEXT_ACTION_DROP | @@ -1273,7 +1273,7 @@ static int tx_add_policy(struct mlx5e_ipsec_pol_entry *pol_entry) default: WARN_ON(true); err = -EINVAL; - goto err_action; + goto err_mod_header; } flow_act.flags |= FLOW_ACT_NO_APPEND; @@ -1293,7 +1293,7 @@ static int tx_add_policy(struct mlx5e_ipsec_pol_entry *pol_entry) return 0; err_action: - if (attrs->reqid) + if (flow_act.modify_hdr) mlx5_modify_header_dealloc(mdev, flow_act.modify_hdr); err_mod_header: kvfree(spec); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/macsec.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/macsec.c index 33b3620ea45c..f3428dbeb298 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/macsec.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/macsec.c @@ -4,6 +4,7 @@ #include <linux/mlx5/device.h> #include <linux/mlx5/mlx5_ifc.h> #include <linux/xarray.h> +#include <linux/if_vlan.h> #include "en.h" #include "lib/aso.h" @@ -348,12 +349,21 @@ static void mlx5e_macsec_cleanup_sa(struct mlx5e_macsec *macsec, sa->macsec_rule = NULL; } +static struct mlx5e_priv *macsec_netdev_priv(const struct net_device *dev) +{ +#if IS_ENABLED(CONFIG_VLAN_8021Q) + if (is_vlan_dev(dev)) + return netdev_priv(vlan_dev_priv(dev)->real_dev); +#endif + return netdev_priv(dev); +} + static int mlx5e_macsec_init_sa(struct macsec_context *ctx, struct mlx5e_macsec_sa *sa, bool encrypt, bool is_tx) { - struct mlx5e_priv *priv = netdev_priv(ctx->netdev); + struct mlx5e_priv *priv = macsec_netdev_priv(ctx->netdev); struct mlx5e_macsec *macsec = priv->macsec; struct mlx5_macsec_rule_attrs rule_attrs; struct mlx5_core_dev *mdev = priv->mdev; @@ -427,7 +437,7 @@ static int macsec_rx_sa_active_update(struct macsec_context *ctx, struct mlx5e_macsec_sa *rx_sa, bool active) { - struct mlx5e_priv *priv = netdev_priv(ctx->netdev); + struct mlx5e_priv *priv = macsec_netdev_priv(ctx->netdev); struct mlx5e_macsec *macsec = priv->macsec; int err = 0; @@ -508,9 +518,9 @@ static void update_macsec_epn(struct mlx5e_macsec_sa *sa, const struct macsec_ke static int mlx5e_macsec_add_txsa(struct macsec_context *ctx) { + struct mlx5e_priv *priv = macsec_netdev_priv(ctx->netdev); const struct macsec_tx_sc *tx_sc = &ctx->secy->tx_sc; const struct macsec_tx_sa *ctx_tx_sa = ctx->sa.tx_sa; - struct mlx5e_priv *priv = netdev_priv(ctx->netdev); const struct macsec_secy *secy = ctx->secy; struct mlx5e_macsec_device *macsec_device; struct mlx5_core_dev *mdev = priv->mdev; @@ -583,9 +593,9 @@ out: static int mlx5e_macsec_upd_txsa(struct macsec_context *ctx) { + struct mlx5e_priv *priv = macsec_netdev_priv(ctx->netdev); const struct macsec_tx_sc *tx_sc = &ctx->secy->tx_sc; const struct macsec_tx_sa *ctx_tx_sa = ctx->sa.tx_sa; - struct mlx5e_priv *priv = netdev_priv(ctx->netdev); struct mlx5e_macsec_device *macsec_device; u8 assoc_num = ctx->sa.assoc_num; struct mlx5e_macsec_sa *tx_sa; @@ -645,7 +655,7 @@ out: static int mlx5e_macsec_del_txsa(struct macsec_context *ctx) { - struct mlx5e_priv *priv = netdev_priv(ctx->netdev); + struct mlx5e_priv *priv = macsec_netdev_priv(ctx->netdev); struct mlx5e_macsec_device *macsec_device; u8 assoc_num = ctx->sa.assoc_num; struct mlx5e_macsec_sa *tx_sa; @@ -696,7 +706,7 @@ static u32 mlx5e_macsec_get_sa_from_hashtable(struct rhashtable *sci_hash, sci_t static int mlx5e_macsec_add_rxsc(struct macsec_context *ctx) { struct mlx5e_macsec_rx_sc_xarray_element *sc_xarray_element; - struct mlx5e_priv *priv = netdev_priv(ctx->netdev); + struct mlx5e_priv *priv = macsec_netdev_priv(ctx->netdev); const struct macsec_rx_sc *ctx_rx_sc = ctx->rx_sc; struct mlx5e_macsec_device *macsec_device; struct mlx5e_macsec_rx_sc *rx_sc; @@ -776,7 +786,7 @@ out: static int mlx5e_macsec_upd_rxsc(struct macsec_context *ctx) { - struct mlx5e_priv *priv = netdev_priv(ctx->netdev); + struct mlx5e_priv *priv = macsec_netdev_priv(ctx->netdev); const struct macsec_rx_sc *ctx_rx_sc = ctx->rx_sc; struct mlx5e_macsec_device *macsec_device; struct mlx5e_macsec_rx_sc *rx_sc; @@ -854,7 +864,7 @@ static void macsec_del_rxsc_ctx(struct mlx5e_macsec *macsec, struct mlx5e_macsec static int mlx5e_macsec_del_rxsc(struct macsec_context *ctx) { - struct mlx5e_priv *priv = netdev_priv(ctx->netdev); + struct mlx5e_priv *priv = macsec_netdev_priv(ctx->netdev); struct mlx5e_macsec_device *macsec_device; struct mlx5e_macsec_rx_sc *rx_sc; struct mlx5e_macsec *macsec; @@ -890,8 +900,8 @@ out: static int mlx5e_macsec_add_rxsa(struct macsec_context *ctx) { + struct mlx5e_priv *priv = macsec_netdev_priv(ctx->netdev); const struct macsec_rx_sa *ctx_rx_sa = ctx->sa.rx_sa; - struct mlx5e_priv *priv = netdev_priv(ctx->netdev); struct mlx5e_macsec_device *macsec_device; struct mlx5_core_dev *mdev = priv->mdev; u8 assoc_num = ctx->sa.assoc_num; @@ -976,8 +986,8 @@ out: static int mlx5e_macsec_upd_rxsa(struct macsec_context *ctx) { + struct mlx5e_priv *priv = macsec_netdev_priv(ctx->netdev); const struct macsec_rx_sa *ctx_rx_sa = ctx->sa.rx_sa; - struct mlx5e_priv *priv = netdev_priv(ctx->netdev); struct mlx5e_macsec_device *macsec_device; u8 assoc_num = ctx->sa.assoc_num; struct mlx5e_macsec_rx_sc *rx_sc; @@ -1033,7 +1043,7 @@ out: static int mlx5e_macsec_del_rxsa(struct macsec_context *ctx) { - struct mlx5e_priv *priv = netdev_priv(ctx->netdev); + struct mlx5e_priv *priv = macsec_netdev_priv(ctx->netdev); struct mlx5e_macsec_device *macsec_device; sci_t sci = ctx->sa.rx_sa->sc->sci; struct mlx5e_macsec_rx_sc *rx_sc; @@ -1085,7 +1095,7 @@ out: static int mlx5e_macsec_add_secy(struct macsec_context *ctx) { - struct mlx5e_priv *priv = netdev_priv(ctx->netdev); + struct mlx5e_priv *priv = macsec_netdev_priv(ctx->netdev); const struct net_device *dev = ctx->secy->netdev; const struct net_device *netdev = ctx->netdev; struct mlx5e_macsec_device *macsec_device; @@ -1137,7 +1147,7 @@ out: static int macsec_upd_secy_hw_address(struct macsec_context *ctx, struct mlx5e_macsec_device *macsec_device) { - struct mlx5e_priv *priv = netdev_priv(ctx->netdev); + struct mlx5e_priv *priv = macsec_netdev_priv(ctx->netdev); const struct net_device *dev = ctx->secy->netdev; struct mlx5e_macsec *macsec = priv->macsec; struct mlx5e_macsec_rx_sc *rx_sc, *tmp; @@ -1184,8 +1194,8 @@ out: */ static int mlx5e_macsec_upd_secy(struct macsec_context *ctx) { + struct mlx5e_priv *priv = macsec_netdev_priv(ctx->netdev); const struct macsec_tx_sc *tx_sc = &ctx->secy->tx_sc; - struct mlx5e_priv *priv = netdev_priv(ctx->netdev); const struct net_device *dev = ctx->secy->netdev; struct mlx5e_macsec_device *macsec_device; struct mlx5e_macsec_sa *tx_sa; @@ -1240,7 +1250,7 @@ out: static int mlx5e_macsec_del_secy(struct macsec_context *ctx) { - struct mlx5e_priv *priv = netdev_priv(ctx->netdev); + struct mlx5e_priv *priv = macsec_netdev_priv(ctx->netdev); struct mlx5e_macsec_device *macsec_device; struct mlx5e_macsec_rx_sc *rx_sc, *tmp; struct mlx5e_macsec_sa *tx_sa; @@ -1741,7 +1751,7 @@ void mlx5e_macsec_offload_handle_rx_skb(struct net_device *netdev, { struct mlx5e_macsec_rx_sc_xarray_element *sc_xarray_element; u32 macsec_meta_data = be32_to_cpu(cqe->ft_metadata); - struct mlx5e_priv *priv = netdev_priv(netdev); + struct mlx5e_priv *priv = macsec_netdev_priv(netdev); struct mlx5e_macsec_rx_sc *rx_sc; struct mlx5e_macsec *macsec; u32 fs_id; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/macsec_fs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/macsec_fs.c index 9173b67becef..7fc901a6ec5f 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/macsec_fs.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/macsec_fs.c @@ -4,6 +4,7 @@ #include <net/macsec.h> #include <linux/netdevice.h> #include <linux/mlx5/qp.h> +#include <linux/if_vlan.h> #include "fs_core.h" #include "en/fs.h" #include "en_accel/macsec_fs.h" @@ -508,6 +509,8 @@ static void macsec_fs_tx_del_rule(struct mlx5e_macsec_fs *macsec_fs, macsec_fs_tx_ft_put(macsec_fs); } +#define MLX5_REFORMAT_PARAM_ADD_MACSEC_OFFSET_4_BYTES 1 + static union mlx5e_macsec_rule * macsec_fs_tx_add_rule(struct mlx5e_macsec_fs *macsec_fs, const struct macsec_context *macsec_ctx, @@ -553,6 +556,10 @@ macsec_fs_tx_add_rule(struct mlx5e_macsec_fs *macsec_fs, reformat_params.type = MLX5_REFORMAT_TYPE_ADD_MACSEC; reformat_params.size = reformat_size; reformat_params.data = reformatbf; + + if (is_vlan_dev(macsec_ctx->netdev)) + reformat_params.param_0 = MLX5_REFORMAT_PARAM_ADD_MACSEC_OFFSET_4_BYTES; + flow_act.pkt_reformat = mlx5_packet_reformat_alloc(macsec_fs->mdev, &reformat_params, MLX5_FLOW_NAMESPACE_EGRESS_MACSEC); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c index 7eb1eeb115ca..1ff09ef38a28 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c @@ -857,6 +857,7 @@ static int mlx5e_alloc_rq(struct mlx5e_params *params, pp_params.pool_size = pool_size; pp_params.nid = node; pp_params.dev = rq->pdev; + pp_params.napi = rq->cq.napi; pp_params.dma_dir = rq->buff.map_dir; pp_params.max_len = PAGE_SIZE; @@ -5109,6 +5110,7 @@ static void mlx5e_build_nic_netdev(struct net_device *netdev) netdev->vlan_features |= NETIF_F_SG; netdev->vlan_features |= NETIF_F_HW_CSUM; + netdev->vlan_features |= NETIF_F_HW_MACSEC; netdev->vlan_features |= NETIF_F_GRO; netdev->vlan_features |= NETIF_F_TSO; netdev->vlan_features |= NETIF_F_TSO6; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c index 8ff654b4e9e1..2d87068f63fd 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c @@ -53,6 +53,7 @@ #include "lib/vxlan.h" #define CREATE_TRACE_POINTS #include "diag/en_rep_tracepoint.h" +#include "diag/reporter_vnic.h" #include "en_accel/ipsec.h" #include "en/tc/int_port.h" #include "en/ptp.h" @@ -1294,6 +1295,50 @@ static unsigned int mlx5e_ul_rep_stats_grps_num(struct mlx5e_priv *priv) return ARRAY_SIZE(mlx5e_ul_rep_stats_grps); } +static int +mlx5e_rep_vnic_reporter_diagnose(struct devlink_health_reporter *reporter, + struct devlink_fmsg *fmsg, + struct netlink_ext_ack *extack) +{ + struct mlx5e_rep_priv *rpriv = devlink_health_reporter_priv(reporter); + struct mlx5_eswitch_rep *rep = rpriv->rep; + + return mlx5_reporter_vnic_diagnose_counters(rep->esw->dev, fmsg, + rep->vport, true); +} + +static const struct devlink_health_reporter_ops mlx5_rep_vnic_reporter_ops = { + .name = "vnic", + .diagnose = mlx5e_rep_vnic_reporter_diagnose, +}; + +static void mlx5e_rep_vnic_reporter_create(struct mlx5e_priv *priv, + struct devlink_port *dl_port) +{ + struct mlx5e_rep_priv *rpriv = priv->ppriv; + struct devlink_health_reporter *reporter; + + reporter = devl_port_health_reporter_create(dl_port, + &mlx5_rep_vnic_reporter_ops, + 0, rpriv); + if (IS_ERR(reporter)) { + mlx5_core_err(priv->mdev, + "Failed to create representor vnic reporter, err = %ld\n", + PTR_ERR(reporter)); + return; + } + + rpriv->rep_vnic_reporter = reporter; +} + +static void mlx5e_rep_vnic_reporter_destroy(struct mlx5e_priv *priv) +{ + struct mlx5e_rep_priv *rpriv = priv->ppriv; + + if (!IS_ERR_OR_NULL(rpriv->rep_vnic_reporter)) + devl_health_reporter_destroy(rpriv->rep_vnic_reporter); +} + static const struct mlx5e_profile mlx5e_rep_profile = { .init = mlx5e_init_rep, .cleanup = mlx5e_cleanup_rep, @@ -1394,8 +1439,10 @@ mlx5e_vport_vf_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep) dl_port = mlx5_esw_offloads_devlink_port(dev->priv.eswitch, rpriv->rep->vport); - if (dl_port) + if (dl_port) { SET_NETDEV_DEVLINK_PORT(netdev, dl_port); + mlx5e_rep_vnic_reporter_create(priv, dl_port); + } err = register_netdev(netdev); if (err) { @@ -1408,8 +1455,8 @@ mlx5e_vport_vf_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep) return 0; err_detach_netdev: + mlx5e_rep_vnic_reporter_destroy(priv); mlx5e_detach_netdev(netdev_priv(netdev)); - err_cleanup_profile: priv->profile->cleanup(priv); @@ -1458,6 +1505,7 @@ mlx5e_vport_rep_unload(struct mlx5_eswitch_rep *rep) } unregister_netdev(netdev); + mlx5e_rep_vnic_reporter_destroy(priv); mlx5e_detach_netdev(priv); priv->profile->cleanup(priv); mlx5e_destroy_netdev(priv); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.h b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.h index dcfad0bf0f45..80b7f5079a5a 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.h @@ -118,6 +118,7 @@ struct mlx5e_rep_priv { struct rtnl_link_stats64 prev_vf_vport_stats; struct mlx5_flow_handle *send_to_vport_meta_rule; struct rhashtable tc_ht; + struct devlink_health_reporter *rep_vnic_reporter; }; static inline diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c index a8c2ae389d6c..69634829558e 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -861,6 +861,11 @@ static void mlx5e_dealloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix) struct mlx5e_mpw_info *wi = mlx5e_get_mpw_info(rq, ix); /* This function is called on rq/netdev close. */ mlx5e_free_rx_mpwqe(rq, wi); + + /* Avoid a second release of the wqe pages: dealloc is called also + * for missing wqes on an already flushed RQ. + */ + bitmap_fill(wi->skip_release_bitmap, rq->mpwqe.pages_per_wqe); } INDIRECT_CALLABLE_SCOPE bool mlx5e_post_rx_wqes(struct mlx5e_rq *rq) @@ -1741,10 +1746,10 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5e_wqe_frag_info *wi prog = rcu_dereference(rq->xdp_prog); if (prog && mlx5e_xdp_handle(rq, prog, &mxbuf)) { if (test_bit(MLX5E_RQ_FLAG_XDP_XMIT, rq->flags)) { - int i; + struct mlx5e_wqe_frag_info *pwi; - for (i = wi - head_wi; i < rq->wqe.info.num_frags; i++) - mlx5e_put_rx_frag(rq, &head_wi[i]); + for (pwi = head_wi; pwi < wi; pwi++) + pwi->flags |= BIT(MLX5E_WQE_FRAG_SKIP_RELEASE); } return NULL; /* page/packet was consumed by XDP */ } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eq.c b/drivers/net/ethernet/mellanox/mlx5/core/eq.c index eb41f0abf798..1c35d721a31d 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eq.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/eq.c @@ -1070,10 +1070,11 @@ mlx5_comp_irq_get_affinity_mask(struct mlx5_core_dev *dev, int vector) list_for_each_entry(eq, &table->comp_eqs_list, list) { if (i++ == vector) - break; + return mlx5_irq_get_affinity_mask(eq->core.irq); } - return mlx5_irq_get_affinity_mask(eq->core.irq); + WARN_ON_ONCE(1); + return NULL; } EXPORT_SYMBOL(mlx5_comp_irq_get_affinity_mask); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/esw/debugfs.c b/drivers/net/ethernet/mellanox/mlx5/core/esw/debugfs.c deleted file mode 100644 index 3d0bbcca1cb9..000000000000 --- a/drivers/net/ethernet/mellanox/mlx5/core/esw/debugfs.c +++ /dev/null @@ -1,198 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB -/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */ - -#include <linux/debugfs.h> -#include "eswitch.h" - -enum vnic_diag_counter { - MLX5_VNIC_DIAG_TOTAL_Q_UNDER_PROCESSOR_HANDLE, - MLX5_VNIC_DIAG_SEND_QUEUE_PRIORITY_UPDATE_FLOW, - MLX5_VNIC_DIAG_COMP_EQ_OVERRUN, - MLX5_VNIC_DIAG_ASYNC_EQ_OVERRUN, - MLX5_VNIC_DIAG_CQ_OVERRUN, - MLX5_VNIC_DIAG_INVALID_COMMAND, - MLX5_VNIC_DIAG_QOUTA_EXCEEDED_COMMAND, - MLX5_VNIC_DIAG_RX_STEERING_DISCARD, -}; - -static int mlx5_esw_query_vnic_diag(struct mlx5_vport *vport, enum vnic_diag_counter counter, - u64 *val) -{ - u32 out[MLX5_ST_SZ_DW(query_vnic_env_out)] = {}; - u32 in[MLX5_ST_SZ_DW(query_vnic_env_in)] = {}; - struct mlx5_core_dev *dev = vport->dev; - u16 vport_num = vport->vport; - void *vnic_diag_out; - int err; - - MLX5_SET(query_vnic_env_in, in, opcode, MLX5_CMD_OP_QUERY_VNIC_ENV); - MLX5_SET(query_vnic_env_in, in, vport_number, vport_num); - if (!mlx5_esw_is_manager_vport(dev->priv.eswitch, vport_num)) - MLX5_SET(query_vnic_env_in, in, other_vport, 1); - - err = mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); - if (err) - return err; - - vnic_diag_out = MLX5_ADDR_OF(query_vnic_env_out, out, vport_env); - switch (counter) { - case MLX5_VNIC_DIAG_TOTAL_Q_UNDER_PROCESSOR_HANDLE: - *val = MLX5_GET(vnic_diagnostic_statistics, vnic_diag_out, total_error_queues); - break; - case MLX5_VNIC_DIAG_SEND_QUEUE_PRIORITY_UPDATE_FLOW: - *val = MLX5_GET(vnic_diagnostic_statistics, vnic_diag_out, - send_queue_priority_update_flow); - break; - case MLX5_VNIC_DIAG_COMP_EQ_OVERRUN: - *val = MLX5_GET(vnic_diagnostic_statistics, vnic_diag_out, comp_eq_overrun); - break; - case MLX5_VNIC_DIAG_ASYNC_EQ_OVERRUN: - *val = MLX5_GET(vnic_diagnostic_statistics, vnic_diag_out, async_eq_overrun); - break; - case MLX5_VNIC_DIAG_CQ_OVERRUN: - *val = MLX5_GET(vnic_diagnostic_statistics, vnic_diag_out, cq_overrun); - break; - case MLX5_VNIC_DIAG_INVALID_COMMAND: - *val = MLX5_GET(vnic_diagnostic_statistics, vnic_diag_out, invalid_command); - break; - case MLX5_VNIC_DIAG_QOUTA_EXCEEDED_COMMAND: - *val = MLX5_GET(vnic_diagnostic_statistics, vnic_diag_out, quota_exceeded_command); - break; - case MLX5_VNIC_DIAG_RX_STEERING_DISCARD: - *val = MLX5_GET64(vnic_diagnostic_statistics, vnic_diag_out, - nic_receive_steering_discard); - break; - } - - return 0; -} - -static int __show_vnic_diag(struct seq_file *file, struct mlx5_vport *vport, - enum vnic_diag_counter type) -{ - u64 val = 0; - int ret; - - ret = mlx5_esw_query_vnic_diag(vport, type, &val); - if (ret) - return ret; - - seq_printf(file, "%llu\n", val); - return 0; -} - -static int total_q_under_processor_handle_show(struct seq_file *file, void *priv) -{ - return __show_vnic_diag(file, file->private, MLX5_VNIC_DIAG_TOTAL_Q_UNDER_PROCESSOR_HANDLE); -} - -static int send_queue_priority_update_flow_show(struct seq_file *file, void *priv) -{ - return __show_vnic_diag(file, file->private, - MLX5_VNIC_DIAG_SEND_QUEUE_PRIORITY_UPDATE_FLOW); -} - -static int comp_eq_overrun_show(struct seq_file *file, void *priv) -{ - return __show_vnic_diag(file, file->private, MLX5_VNIC_DIAG_COMP_EQ_OVERRUN); -} - -static int async_eq_overrun_show(struct seq_file *file, void *priv) -{ - return __show_vnic_diag(file, file->private, MLX5_VNIC_DIAG_ASYNC_EQ_OVERRUN); -} - -static int cq_overrun_show(struct seq_file *file, void *priv) -{ - return __show_vnic_diag(file, file->private, MLX5_VNIC_DIAG_CQ_OVERRUN); -} - -static int invalid_command_show(struct seq_file *file, void *priv) -{ - return __show_vnic_diag(file, file->private, MLX5_VNIC_DIAG_INVALID_COMMAND); -} - -static int quota_exceeded_command_show(struct seq_file *file, void *priv) -{ - return __show_vnic_diag(file, file->private, MLX5_VNIC_DIAG_QOUTA_EXCEEDED_COMMAND); -} - -static int rx_steering_discard_show(struct seq_file *file, void *priv) -{ - return __show_vnic_diag(file, file->private, MLX5_VNIC_DIAG_RX_STEERING_DISCARD); -} - -DEFINE_SHOW_ATTRIBUTE(total_q_under_processor_handle); -DEFINE_SHOW_ATTRIBUTE(send_queue_priority_update_flow); -DEFINE_SHOW_ATTRIBUTE(comp_eq_overrun); -DEFINE_SHOW_ATTRIBUTE(async_eq_overrun); -DEFINE_SHOW_ATTRIBUTE(cq_overrun); -DEFINE_SHOW_ATTRIBUTE(invalid_command); -DEFINE_SHOW_ATTRIBUTE(quota_exceeded_command); -DEFINE_SHOW_ATTRIBUTE(rx_steering_discard); - -void mlx5_esw_vport_debugfs_destroy(struct mlx5_eswitch *esw, u16 vport_num) -{ - struct mlx5_vport *vport = mlx5_eswitch_get_vport(esw, vport_num); - - debugfs_remove_recursive(vport->dbgfs); - vport->dbgfs = NULL; -} - -/* vnic diag dir name is "pf", "ecpf" or "{vf/sf}_xxxx" */ -#define VNIC_DIAG_DIR_NAME_MAX_LEN 8 - -void mlx5_esw_vport_debugfs_create(struct mlx5_eswitch *esw, u16 vport_num, bool is_sf, u16 sf_num) -{ - struct mlx5_vport *vport = mlx5_eswitch_get_vport(esw, vport_num); - struct dentry *vnic_diag; - char dir_name[VNIC_DIAG_DIR_NAME_MAX_LEN]; - int err; - - if (!MLX5_CAP_GEN(esw->dev, vport_group_manager)) - return; - - if (vport_num == MLX5_VPORT_PF) { - strcpy(dir_name, "pf"); - } else if (vport_num == MLX5_VPORT_ECPF) { - strcpy(dir_name, "ecpf"); - } else { - err = snprintf(dir_name, VNIC_DIAG_DIR_NAME_MAX_LEN, "%s_%d", is_sf ? "sf" : "vf", - is_sf ? sf_num : vport_num - MLX5_VPORT_FIRST_VF); - if (WARN_ON(err < 0)) - return; - } - - vport->dbgfs = debugfs_create_dir(dir_name, esw->dbgfs); - vnic_diag = debugfs_create_dir("vnic_diag", vport->dbgfs); - - if (MLX5_CAP_GEN(esw->dev, vnic_env_queue_counters)) { - debugfs_create_file("total_q_under_processor_handle", 0444, vnic_diag, vport, - &total_q_under_processor_handle_fops); - debugfs_create_file("send_queue_priority_update_flow", 0444, vnic_diag, vport, - &send_queue_priority_update_flow_fops); - } - - if (MLX5_CAP_GEN(esw->dev, eq_overrun_count)) { - debugfs_create_file("comp_eq_overrun", 0444, vnic_diag, vport, - &comp_eq_overrun_fops); - debugfs_create_file("async_eq_overrun", 0444, vnic_diag, vport, - &async_eq_overrun_fops); - } - - if (MLX5_CAP_GEN(esw->dev, vnic_env_cq_overrun)) - debugfs_create_file("cq_overrun", 0444, vnic_diag, vport, &cq_overrun_fops); - - if (MLX5_CAP_GEN(esw->dev, invalid_command_count)) - debugfs_create_file("invalid_command", 0444, vnic_diag, vport, - &invalid_command_fops); - - if (MLX5_CAP_GEN(esw->dev, quota_exceeded_count)) - debugfs_create_file("quota_exceeded_command", 0444, vnic_diag, vport, - "a_exceeded_command_fops); - - if (MLX5_CAP_GEN(esw->dev, nic_receive_steering_discard)) - debugfs_create_file("rx_steering_discard", 0444, vnic_diag, vport, - &rx_steering_discard_fops); - -} diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c index 8bdf28762f41..901c53751b0a 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c @@ -36,7 +36,6 @@ #include <linux/mlx5/vport.h> #include <linux/mlx5/fs.h> #include <linux/mlx5/mpfs.h> -#include <linux/debugfs.h> #include "esw/acl/lgcy.h" #include "esw/legacy.h" #include "esw/qos.h" @@ -1056,7 +1055,6 @@ int mlx5_eswitch_load_vport(struct mlx5_eswitch *esw, u16 vport_num, if (err) return err; - mlx5_esw_vport_debugfs_create(esw, vport_num, false, 0); err = esw_offloads_load_rep(esw, vport_num); if (err) goto err_rep; @@ -1064,7 +1062,6 @@ int mlx5_eswitch_load_vport(struct mlx5_eswitch *esw, u16 vport_num, return err; err_rep: - mlx5_esw_vport_debugfs_destroy(esw, vport_num); mlx5_esw_vport_disable(esw, vport_num); return err; } @@ -1072,7 +1069,6 @@ err_rep: void mlx5_eswitch_unload_vport(struct mlx5_eswitch *esw, u16 vport_num) { esw_offloads_unload_rep(esw, vport_num); - mlx5_esw_vport_debugfs_destroy(esw, vport_num); mlx5_esw_vport_disable(esw, vport_num); } @@ -1488,7 +1484,7 @@ int mlx5_esw_sf_max_hpf_functions(struct mlx5_core_dev *dev, u16 *max_sfs, u16 * void *hca_caps; int err; - if (!mlx5_core_is_ecpf(dev) || mlx5_core_is_management_pf(dev)) { + if (!mlx5_core_is_ecpf(dev)) { *max_sfs = 0; return 0; } @@ -1510,7 +1506,7 @@ out_free: return err; } -static int mlx5_esw_vport_alloc(struct mlx5_eswitch *esw, struct mlx5_core_dev *dev, +static int mlx5_esw_vport_alloc(struct mlx5_eswitch *esw, int index, u16 vport_num) { struct mlx5_vport *vport; @@ -1564,7 +1560,7 @@ static int mlx5_esw_vports_init(struct mlx5_eswitch *esw) xa_init(&esw->vports); - err = mlx5_esw_vport_alloc(esw, dev, idx, MLX5_VPORT_PF); + err = mlx5_esw_vport_alloc(esw, idx, MLX5_VPORT_PF); if (err) goto err; if (esw->first_host_vport == MLX5_VPORT_PF) @@ -1572,7 +1568,7 @@ static int mlx5_esw_vports_init(struct mlx5_eswitch *esw) idx++; for (i = 0; i < mlx5_core_max_vfs(dev); i++) { - err = mlx5_esw_vport_alloc(esw, dev, idx, idx); + err = mlx5_esw_vport_alloc(esw, idx, idx); if (err) goto err; xa_set_mark(&esw->vports, idx, MLX5_ESW_VPT_VF); @@ -1581,7 +1577,7 @@ static int mlx5_esw_vports_init(struct mlx5_eswitch *esw) } base_sf_num = mlx5_sf_start_function_id(dev); for (i = 0; i < mlx5_sf_max_functions(dev); i++) { - err = mlx5_esw_vport_alloc(esw, dev, idx, base_sf_num + i); + err = mlx5_esw_vport_alloc(esw, idx, base_sf_num + i); if (err) goto err; xa_set_mark(&esw->vports, base_sf_num + i, MLX5_ESW_VPT_SF); @@ -1592,7 +1588,7 @@ static int mlx5_esw_vports_init(struct mlx5_eswitch *esw) if (err) goto err; for (i = 0; i < max_host_pf_sfs; i++) { - err = mlx5_esw_vport_alloc(esw, dev, idx, base_sf_num + i); + err = mlx5_esw_vport_alloc(esw, idx, base_sf_num + i); if (err) goto err; xa_set_mark(&esw->vports, base_sf_num + i, MLX5_ESW_VPT_SF); @@ -1600,12 +1596,12 @@ static int mlx5_esw_vports_init(struct mlx5_eswitch *esw) } if (mlx5_ecpf_vport_exists(dev)) { - err = mlx5_esw_vport_alloc(esw, dev, idx, MLX5_VPORT_ECPF); + err = mlx5_esw_vport_alloc(esw, idx, MLX5_VPORT_ECPF); if (err) goto err; idx++; } - err = mlx5_esw_vport_alloc(esw, dev, idx, MLX5_VPORT_UPLINK); + err = mlx5_esw_vport_alloc(esw, idx, MLX5_VPORT_UPLINK); if (err) goto err; return 0; @@ -1672,7 +1668,6 @@ int mlx5_eswitch_init(struct mlx5_core_dev *dev) dev->priv.eswitch = esw; BLOCKING_INIT_NOTIFIER_HEAD(&esw->n_head); - esw->dbgfs = debugfs_create_dir("esw", mlx5_debugfs_get_dev_root(esw->dev)); esw_info(dev, "Total vports %d, per vport: max uc(%d) max mc(%d)\n", esw->total_vports, @@ -1696,7 +1691,6 @@ void mlx5_eswitch_cleanup(struct mlx5_eswitch *esw) esw_info(esw->dev, "cleanup\n"); - debugfs_remove_recursive(esw->dbgfs); esw->dev->priv.eswitch = NULL; destroy_workqueue(esw->work_queue); WARN_ON(refcount_read(&esw->qos.refcnt)); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h index e9d68fdf68f5..62f01d4600fe 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h @@ -195,7 +195,6 @@ struct mlx5_vport { enum mlx5_eswitch_vport_event enabled_events; int index; struct devlink_port *dl_port; - struct dentry *dbgfs; }; struct mlx5_esw_indir_table; @@ -343,7 +342,6 @@ struct mlx5_eswitch { u32 large_group_num; } params; struct blocking_notifier_head n_head; - struct dentry *dbgfs; }; void esw_offloads_disable(struct mlx5_eswitch *esw); @@ -356,7 +354,6 @@ mlx5_eswitch_add_send_to_vport_meta_rule(struct mlx5_eswitch *esw, u16 vport_num void mlx5_eswitch_del_send_to_vport_meta_rule(struct mlx5_flow_handle *rule); bool mlx5_esw_vport_match_metadata_supported(const struct mlx5_eswitch *esw); -int mlx5_esw_offloads_vport_metadata_set(struct mlx5_eswitch *esw, bool enable); u32 mlx5_esw_match_metadata_alloc(struct mlx5_eswitch *esw); void mlx5_esw_match_metadata_free(struct mlx5_eswitch *esw, u32 metadata); @@ -704,9 +701,6 @@ int mlx5_esw_offloads_devlink_port_register(struct mlx5_eswitch *esw, u16 vport_ void mlx5_esw_offloads_devlink_port_unregister(struct mlx5_eswitch *esw, u16 vport_num); struct devlink_port *mlx5_esw_offloads_devlink_port(struct mlx5_eswitch *esw, u16 vport_num); -void mlx5_esw_vport_debugfs_create(struct mlx5_eswitch *esw, u16 vport_num, bool is_sf, u16 sf_num); -void mlx5_esw_vport_debugfs_destroy(struct mlx5_eswitch *esw, u16 vport_num); - int mlx5_esw_devlink_sf_port_register(struct mlx5_eswitch *esw, struct devlink_port *dl_port, u16 vport_num, u32 controller, u32 sfnum); void mlx5_esw_devlink_sf_port_unregister(struct mlx5_eswitch *esw, u16 vport_num); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c index b6e2709c1371..12c07a44aa4f 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c @@ -2939,28 +2939,6 @@ metadata_err: return err; } -int mlx5_esw_offloads_vport_metadata_set(struct mlx5_eswitch *esw, bool enable) -{ - int err = 0; - - down_write(&esw->mode_lock); - if (mlx5_esw_is_fdb_created(esw)) { - err = -EBUSY; - goto done; - } - if (!mlx5_esw_vport_match_metadata_supported(esw)) { - err = -EOPNOTSUPP; - goto done; - } - if (enable) - esw->flags |= MLX5_ESWITCH_VPORT_MATCH_METADATA; - else - esw->flags &= ~MLX5_ESWITCH_VPORT_MATCH_METADATA; -done: - up_write(&esw->mode_lock); - return err; -} - int esw_vport_create_offloads_acl_tables(struct mlx5_eswitch *esw, struct mlx5_vport *vport) @@ -3828,14 +3806,12 @@ int mlx5_esw_offloads_sf_vport_enable(struct mlx5_eswitch *esw, struct devlink_p if (err) goto devlink_err; - mlx5_esw_vport_debugfs_create(esw, vport_num, true, sfnum); err = mlx5_esw_offloads_rep_load(esw, vport_num); if (err) goto rep_err; return 0; rep_err: - mlx5_esw_vport_debugfs_destroy(esw, vport_num); mlx5_esw_devlink_sf_port_unregister(esw, vport_num); devlink_err: mlx5_esw_vport_disable(esw, vport_num); @@ -3845,7 +3821,6 @@ devlink_err: void mlx5_esw_offloads_sf_vport_disable(struct mlx5_eswitch *esw, u16 vport_num) { mlx5_esw_offloads_rep_unload(esw, vport_num); - mlx5_esw_vport_debugfs_destroy(esw, vport_num); mlx5_esw_devlink_sf_port_unregister(esw, vport_num); mlx5_esw_vport_disable(esw, vport_num); } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c index 016c5f99c470..871c32dda66e 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/health.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c @@ -42,6 +42,7 @@ #include "lib/pci_vsc.h" #include "lib/tout.h" #include "diag/fw_tracer.h" +#include "diag/reporter_vnic.h" enum { MAX_MISSES = 3, @@ -898,6 +899,7 @@ void mlx5_health_cleanup(struct mlx5_core_dev *dev) cancel_delayed_work_sync(&health->update_fw_log_ts_work); destroy_workqueue(health->wq); + mlx5_reporter_vnic_destroy(dev); mlx5_fw_reporters_destroy(dev); } @@ -907,6 +909,7 @@ int mlx5_health_init(struct mlx5_core_dev *dev) char *name; mlx5_fw_reporters_create(dev); + mlx5_reporter_vnic_create(dev); health = &dev->priv.health; name = kmalloc(64, GFP_KERNEL); @@ -926,6 +929,7 @@ int mlx5_health_init(struct mlx5_core_dev *dev) return 0; out_err: + mlx5_reporter_vnic_destroy(dev); mlx5_fw_reporters_destroy(dev); return -ENOMEM; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c index a95d1218def9..89a65779494e 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -717,7 +717,7 @@ static int handle_hca_cap_port_selection(struct mlx5_core_dev *dev, MLX5_ST_SZ_BYTES(port_selection_cap)); MLX5_SET(port_selection_cap, set_hca_cap, port_select_flow_table_bypass, 1); - err = set_caps(dev, set_ctx, MLX5_SET_HCA_CAP_OP_MODE_PORT_SELECTION); + err = set_caps(dev, set_ctx, MLX5_SET_HCA_CAP_OP_MOD_PORT_SELECTION); return err; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c b/drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c index e12e528c09f5..2245d3b2f393 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB /* Copyright (c) 2019 Mellanox Technologies. */ +#include <linux/pci.h> #include <linux/interrupt.h> #include <linux/notifier.h> #include <linux/mlx5/driver.h> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_dbg.c b/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_dbg.c index 1ff8bde90e1e..7e36e1062139 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_dbg.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_dbg.c @@ -4,6 +4,7 @@ #include <linux/debugfs.h> #include <linux/kernel.h> #include <linux/seq_file.h> +#include <linux/version.h> #include "dr_types.h" #define DR_DBG_PTR_TO_ID(p) ((u64)(uintptr_t)(p) & 0xFFFFFFFFULL) @@ -153,13 +154,15 @@ dr_dump_rule_action_mem(struct seq_file *file, const u64 rule_id, DR_DUMP_REC_TYPE_ACTION_MODIFY_HDR, action_id, rule_id, action->rewrite->index, action->rewrite->single_action_opt, - action->rewrite->num_of_actions, + ptrn_arg ? action->rewrite->num_of_actions : 0, ptrn_arg ? ptrn->index : 0, ptrn_arg ? mlx5dr_arg_get_obj_id(arg) : 0); - for (i = 0; i < action->rewrite->num_of_actions; i++) { - seq_printf(file, ",0x%016llx", - be64_to_cpu(((__be64 *)rewrite_data)[i])); + if (ptrn_arg) { + for (i = 0; i < action->rewrite->num_of_actions; i++) { + seq_printf(file, ",0x%016llx", + be64_to_cpu(((__be64 *)rewrite_data)[i])); + } } seq_puts(file, "\n"); @@ -630,9 +633,18 @@ dr_dump_domain(struct seq_file *file, struct mlx5dr_domain *dmn) u64 domain_id = DR_DBG_PTR_TO_ID(dmn); int ret; - seq_printf(file, "%d,0x%llx,%d,0%x,%d,%s\n", DR_DUMP_REC_TYPE_DOMAIN, + seq_printf(file, "%d,0x%llx,%d,0%x,%d,%u.%u.%u,%s,%d,%u,%u,%u\n", + DR_DUMP_REC_TYPE_DOMAIN, domain_id, dmn->type, dmn->info.caps.gvmi, - dmn->info.supp_sw_steering, pci_name(dmn->mdev->pdev)); + dmn->info.supp_sw_steering, + /* package version */ + LINUX_VERSION_MAJOR, LINUX_VERSION_PATCHLEVEL, + LINUX_VERSION_SUBLEVEL, + pci_name(dmn->mdev->pdev), + 0, /* domain flags */ + dmn->num_buddies[DR_ICM_TYPE_STE], + dmn->num_buddies[DR_ICM_TYPE_MODIFY_ACTION], + dmn->num_buddies[DR_ICM_TYPE_MODIFY_HDR_PTRN]); ret = dr_dump_domain_info(file, &dmn->info, domain_id); if (ret < 0) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_icm_pool.c b/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_icm_pool.c index 04fc170a6c16..0b5af9f3f605 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_icm_pool.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_icm_pool.c @@ -4,7 +4,9 @@ #include "dr_types.h" #define DR_ICM_MODIFY_HDR_ALIGN_BASE 64 -#define DR_ICM_POOL_HOT_MEMORY_FRACTION 4 +#define DR_ICM_POOL_STE_HOT_MEM_PERCENT 25 +#define DR_ICM_POOL_MODIFY_HDR_PTRN_HOT_MEM_PERCENT 50 +#define DR_ICM_POOL_MODIFY_ACTION_HOT_MEM_PERCENT 90 struct mlx5dr_icm_hot_chunk { struct mlx5dr_icm_buddy_mem *buddy_mem; @@ -29,6 +31,8 @@ struct mlx5dr_icm_pool { struct mlx5dr_icm_hot_chunk *hot_chunks_arr; u32 hot_chunks_num; u64 hot_memory_size; + /* hot memory size threshold for triggering sync */ + u64 th; }; struct mlx5dr_icm_dm { @@ -284,6 +288,8 @@ static int dr_icm_buddy_create(struct mlx5dr_icm_pool *pool) /* add it to the -start- of the list in order to search in it first */ list_add(&buddy->list_node, &pool->buddy_mem_list); + pool->dmn->num_buddies[pool->icm_type]++; + return 0; err_cleanup_buddy: @@ -297,13 +303,17 @@ free_mr: static void dr_icm_buddy_destroy(struct mlx5dr_icm_buddy_mem *buddy) { + enum mlx5dr_icm_type icm_type = buddy->pool->icm_type; + dr_icm_pool_mr_destroy(buddy->icm_mr); mlx5dr_buddy_cleanup(buddy); - if (buddy->pool->icm_type == DR_ICM_TYPE_STE) + if (icm_type == DR_ICM_TYPE_STE) dr_icm_buddy_cleanup_ste_cache(buddy); + buddy->pool->dmn->num_buddies[icm_type]--; + kvfree(buddy); } @@ -330,15 +340,7 @@ dr_icm_chunk_init(struct mlx5dr_icm_chunk *chunk, static bool dr_icm_pool_is_sync_required(struct mlx5dr_icm_pool *pool) { - int allow_hot_size; - - /* sync when hot memory reaches a certain fraction of the pool size */ - allow_hot_size = - mlx5dr_icm_pool_chunk_size_to_byte(pool->max_log_chunk_sz, - pool->icm_type) / - DR_ICM_POOL_HOT_MEMORY_FRACTION; - - return pool->hot_memory_size > allow_hot_size; + return pool->hot_memory_size > pool->th; } static void dr_icm_pool_clear_hot_chunks_arr(struct mlx5dr_icm_pool *pool) @@ -503,8 +505,9 @@ void mlx5dr_icm_pool_free_htbl(struct mlx5dr_icm_pool *pool, struct mlx5dr_ste_h struct mlx5dr_icm_pool *mlx5dr_icm_pool_create(struct mlx5dr_domain *dmn, enum mlx5dr_icm_type icm_type) { - u32 num_of_chunks, entry_size, max_hot_size; + u32 num_of_chunks, entry_size; struct mlx5dr_icm_pool *pool; + u32 max_hot_size = 0; pool = kvzalloc(sizeof(*pool), GFP_KERNEL); if (!pool) @@ -520,12 +523,21 @@ struct mlx5dr_icm_pool *mlx5dr_icm_pool_create(struct mlx5dr_domain *dmn, switch (icm_type) { case DR_ICM_TYPE_STE: pool->max_log_chunk_sz = dmn->info.max_log_sw_icm_sz; + max_hot_size = mlx5dr_icm_pool_chunk_size_to_byte(pool->max_log_chunk_sz, + pool->icm_type) * + DR_ICM_POOL_STE_HOT_MEM_PERCENT / 100; break; case DR_ICM_TYPE_MODIFY_ACTION: pool->max_log_chunk_sz = dmn->info.max_log_action_icm_sz; + max_hot_size = mlx5dr_icm_pool_chunk_size_to_byte(pool->max_log_chunk_sz, + pool->icm_type) * + DR_ICM_POOL_MODIFY_ACTION_HOT_MEM_PERCENT / 100; break; case DR_ICM_TYPE_MODIFY_HDR_PTRN: pool->max_log_chunk_sz = dmn->info.max_log_modify_hdr_pattern_icm_sz; + max_hot_size = mlx5dr_icm_pool_chunk_size_to_byte(pool->max_log_chunk_sz, + pool->icm_type) * + DR_ICM_POOL_MODIFY_HDR_PTRN_HOT_MEM_PERCENT / 100; break; default: WARN_ON(icm_type); @@ -533,11 +545,8 @@ struct mlx5dr_icm_pool *mlx5dr_icm_pool_create(struct mlx5dr_domain *dmn, entry_size = mlx5dr_icm_pool_dm_type_to_entry_size(pool->icm_type); - max_hot_size = mlx5dr_icm_pool_chunk_size_to_byte(pool->max_log_chunk_sz, - pool->icm_type) / - DR_ICM_POOL_HOT_MEMORY_FRACTION; - num_of_chunks = DIV_ROUND_UP(max_hot_size, entry_size) + 1; + pool->th = max_hot_size; pool->hot_chunks_arr = kvcalloc(num_of_chunks, sizeof(struct mlx5dr_icm_hot_chunk), diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_types.h b/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_types.h index 37b7b1a79f93..678a993ab053 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_types.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_types.h @@ -72,6 +72,7 @@ enum mlx5dr_icm_type { DR_ICM_TYPE_STE, DR_ICM_TYPE_MODIFY_ACTION, DR_ICM_TYPE_MODIFY_HDR_PTRN, + DR_ICM_TYPE_MAX, }; static inline enum mlx5dr_icm_chunk_size @@ -955,6 +956,8 @@ struct mlx5dr_domain { struct list_head dbg_tbl_list; struct mlx5dr_dbg_dump_info dump_info; struct xarray definers_xa; + /* memory management statistics */ + u32 num_buddies[DR_ICM_TYPE_MAX]; }; struct mlx5dr_table_rx_tx { diff --git a/drivers/net/ethernet/mellanox/mlxfw/mlxfw_mfa2_tlv_multi.c b/drivers/net/ethernet/mellanox/mlxfw/mlxfw_mfa2_tlv_multi.c index 017d68f1e123..972c571b4158 100644 --- a/drivers/net/ethernet/mellanox/mlxfw/mlxfw_mfa2_tlv_multi.c +++ b/drivers/net/ethernet/mellanox/mlxfw/mlxfw_mfa2_tlv_multi.c @@ -31,6 +31,8 @@ mlxfw_mfa2_tlv_next(const struct mlxfw_mfa2_file *mfa2_file, if (tlv->type == MLXFW_MFA2_TLV_MULTI_PART) { multi = mlxfw_mfa2_tlv_multi_get(mfa2_file, tlv); + if (!multi) + return NULL; tlv_len = NLA_ALIGN(tlv_len + be16_to_cpu(multi->total_len)); } diff --git a/drivers/net/ethernet/mellanox/mlxsw/pci_hw.h b/drivers/net/ethernet/mellanox/mlxsw/pci_hw.h index 48dbfea0a2a1..7cdf0ce24f28 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/pci_hw.h +++ b/drivers/net/ethernet/mellanox/mlxsw/pci_hw.h @@ -26,7 +26,7 @@ #define MLXSW_PCI_CIR_TIMEOUT_MSECS 1000 #define MLXSW_PCI_SW_RESET_TIMEOUT_MSECS 900000 -#define MLXSW_PCI_SW_RESET_WAIT_MSECS 200 +#define MLXSW_PCI_SW_RESET_WAIT_MSECS 400 #define MLXSW_PCI_FW_READY 0xA1844 #define MLXSW_PCI_FW_READY_MASK 0xFFFF #define MLXSW_PCI_FW_READY_MAGIC 0x5E diff --git a/drivers/net/ethernet/sfc/efx.c b/drivers/net/ethernet/sfc/efx.c index 746fd9164e30..a4f22d8e6ac7 100644 --- a/drivers/net/ethernet/sfc/efx.c +++ b/drivers/net/ethernet/sfc/efx.c @@ -540,7 +540,6 @@ int efx_net_open(struct net_device *net_dev) else efx->state = STATE_NET_UP; - efx_selftest_async_start(efx); return 0; } diff --git a/drivers/net/ethernet/sfc/efx_common.c b/drivers/net/ethernet/sfc/efx_common.c index cc30524c2fe4..361687de308d 100644 --- a/drivers/net/ethernet/sfc/efx_common.c +++ b/drivers/net/ethernet/sfc/efx_common.c @@ -544,6 +544,8 @@ void efx_start_all(struct efx_nic *efx) /* Start the hardware monitor if there is one */ efx_start_monitor(efx); + efx_selftest_async_start(efx); + /* Link state detection is normally event-driven; we have * to poll now because we could have missed a change */ diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c index 47534310365a..f116e4ae293b 100644 --- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c +++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c @@ -6377,6 +6377,10 @@ static int stmmac_vlan_rx_add_vid(struct net_device *ndev, __be16 proto, u16 vid bool is_double = false; int ret; + ret = pm_runtime_resume_and_get(priv->device); + if (ret < 0) + return ret; + if (be16_to_cpu(proto) == ETH_P_8021AD) is_double = true; @@ -6384,16 +6388,18 @@ static int stmmac_vlan_rx_add_vid(struct net_device *ndev, __be16 proto, u16 vid ret = stmmac_vlan_update(priv, is_double); if (ret) { clear_bit(vid, priv->active_vlans); - return ret; + goto err_pm_put; } if (priv->hw->num_vlan) { ret = stmmac_add_hw_vlan_rx_fltr(priv, ndev, priv->hw, proto, vid); if (ret) - return ret; + goto err_pm_put; } +err_pm_put: + pm_runtime_put(priv->device); - return 0; + return ret; } static int stmmac_vlan_rx_kill_vid(struct net_device *ndev, __be16 proto, u16 vid) diff --git a/drivers/net/hamradio/Kconfig b/drivers/net/hamradio/Kconfig index a9c44f08199d..a94c7bd5db2e 100644 --- a/drivers/net/hamradio/Kconfig +++ b/drivers/net/hamradio/Kconfig @@ -47,7 +47,7 @@ config BPQETHER config SCC tristate "Z8530 SCC driver" - depends on ISA && AX25 && ISA_DMA_API + depends on ISA && AX25 help These cards are used to connect your Linux box to an amateur radio in order to communicate with other computers. If you want to use diff --git a/drivers/net/macsec.c b/drivers/net/macsec.c index 25616247d7a5..3427993f94f7 100644 --- a/drivers/net/macsec.c +++ b/drivers/net/macsec.c @@ -1021,8 +1021,12 @@ static enum rx_handler_result handle_not_macsec(struct sk_buff *skb) * the SecTAG, so we have to deduce which port to deliver to. */ if (macsec_is_offloaded(macsec) && netif_running(ndev)) { - if (md_dst && md_dst->type == METADATA_MACSEC && - (!find_rx_sc(&macsec->secy, md_dst->u.macsec_info.sci))) + struct macsec_rx_sc *rx_sc = NULL; + + if (md_dst && md_dst->type == METADATA_MACSEC) + rx_sc = find_rx_sc(&macsec->secy, md_dst->u.macsec_info.sci); + + if (md_dst && md_dst->type == METADATA_MACSEC && !rx_sc) continue; if (ether_addr_equal_64bits(hdr->h_dest, @@ -1047,7 +1051,13 @@ static enum rx_handler_result handle_not_macsec(struct sk_buff *skb) nskb->pkt_type = PACKET_MULTICAST; __netif_rx(nskb); + } else if (rx_sc || ndev->flags & IFF_PROMISC) { + skb->dev = ndev; + skb->pkt_type = PACKET_HOST; + ret = RX_HANDLER_ANOTHER; + goto out; } + continue; } diff --git a/drivers/net/phy/Kconfig b/drivers/net/phy/Kconfig index b8cc49820ced..bcfba077b80e 100644 --- a/drivers/net/phy/Kconfig +++ b/drivers/net/phy/Kconfig @@ -18,7 +18,6 @@ menuconfig PHYLIB depends on NETDEVICES select MDIO_DEVICE select MDIO_DEVRES - depends on LEDS_CLASS || LEDS_CLASS=n help Ethernet controllers are usually attached to PHY devices. This option provides infrastructure for @@ -45,6 +44,14 @@ config LED_TRIGGER_PHY <Speed in megabits>Mbps OR <Speed in gigabits>Gbps OR link for any speed known to the PHY. +config PHYLIB_LEDS + bool "Support probing LEDs from device tree" + depends on LEDS_CLASS=y || LEDS_CLASS=PHYLIB + depends on OF + default y + help + When LED class support is enabled, phylib can automatically + probe LED setting from device tree. config FIXED_PHY tristate "MDIO Bus/PHY emulation with fixed speed/link PHYs" @@ -266,6 +273,12 @@ config NATIONAL_PHY help Currently supports the DP83865 PHY. +config NXP_CBTX_PHY + tristate "NXP 100BASE-TX PHYs" + help + Support the 100BASE-TX PHY integrated on the SJA1110 automotive + switch family. + config NXP_C45_TJA11XX_PHY tristate "NXP C45 TJA11XX PHYs" depends on PTP_1588_CLOCK_OPTIONAL diff --git a/drivers/net/phy/Makefile b/drivers/net/phy/Makefile index b5138066ba04..ae11bf20b46e 100644 --- a/drivers/net/phy/Makefile +++ b/drivers/net/phy/Makefile @@ -79,6 +79,7 @@ obj-$(CONFIG_MOTORCOMM_PHY) += motorcomm.o obj-$(CONFIG_NATIONAL_PHY) += national.o obj-$(CONFIG_NCN26000_PHY) += ncn26000.o obj-$(CONFIG_NXP_C45_TJA11XX_PHY) += nxp-c45-tja11xx.o +obj-$(CONFIG_NXP_CBTX_PHY) += nxp-cbtx.o obj-$(CONFIG_NXP_TJA11XX_PHY) += nxp-tja11xx.o obj-$(CONFIG_QSEMI_PHY) += qsemi.o obj-$(CONFIG_REALTEK_PHY) += realtek.o diff --git a/drivers/net/phy/nxp-cbtx.c b/drivers/net/phy/nxp-cbtx.c new file mode 100644 index 000000000000..145703f0a406 --- /dev/null +++ b/drivers/net/phy/nxp-cbtx.c @@ -0,0 +1,227 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Driver for 100BASE-TX PHY embedded into NXP SJA1110 switch + * + * Copyright 2022-2023 NXP + */ + +#include <linux/kernel.h> +#include <linux/mii.h> +#include <linux/module.h> +#include <linux/phy.h> + +#define PHY_ID_CBTX_SJA1110 0x001bb020 + +/* Registers */ +#define CBTX_MODE_CTRL_STAT 0x11 +#define CBTX_PDOWN_CTRL 0x18 +#define CBTX_RX_ERR_COUNTER 0x1a +#define CBTX_IRQ_STAT 0x1d +#define CBTX_IRQ_ENABLE 0x1e + +/* Fields */ +#define CBTX_MODE_CTRL_STAT_AUTO_MDIX_EN BIT(7) +#define CBTX_MODE_CTRL_STAT_MDIX_MODE BIT(6) + +#define CBTX_PDOWN_CTL_TRUE_PDOWN BIT(0) + +#define CBTX_IRQ_ENERGYON BIT(7) +#define CBTX_IRQ_AN_COMPLETE BIT(6) +#define CBTX_IRQ_REM_FAULT BIT(5) +#define CBTX_IRQ_LINK_DOWN BIT(4) +#define CBTX_IRQ_AN_LP_ACK BIT(3) +#define CBTX_IRQ_PARALLEL_DETECT_FAULT BIT(2) +#define CBTX_IRQ_AN_PAGE_RECV BIT(1) + +static int cbtx_soft_reset(struct phy_device *phydev) +{ + int ret; + + /* Can't soft reset unless we remove PHY from true power down mode */ + ret = phy_clear_bits(phydev, CBTX_PDOWN_CTRL, + CBTX_PDOWN_CTL_TRUE_PDOWN); + if (ret) + return ret; + + return genphy_soft_reset(phydev); +} + +static int cbtx_config_init(struct phy_device *phydev) +{ + /* Wait for cbtx_config_aneg() to kick in and apply this */ + phydev->mdix_ctrl = ETH_TP_MDI_AUTO; + + return 0; +} + +static int cbtx_mdix_status(struct phy_device *phydev) +{ + int ret; + + ret = phy_read(phydev, CBTX_MODE_CTRL_STAT); + if (ret < 0) + return ret; + + if (ret & CBTX_MODE_CTRL_STAT_MDIX_MODE) + phydev->mdix = ETH_TP_MDI_X; + else + phydev->mdix = ETH_TP_MDI; + + return 0; +} + +static int cbtx_read_status(struct phy_device *phydev) +{ + int ret; + + ret = cbtx_mdix_status(phydev); + if (ret) + return ret; + + return genphy_read_status(phydev); +} + +static int cbtx_mdix_config(struct phy_device *phydev) +{ + int ret; + + switch (phydev->mdix_ctrl) { + case ETH_TP_MDI_AUTO: + return phy_set_bits(phydev, CBTX_MODE_CTRL_STAT, + CBTX_MODE_CTRL_STAT_AUTO_MDIX_EN); + case ETH_TP_MDI: + ret = phy_clear_bits(phydev, CBTX_MODE_CTRL_STAT, + CBTX_MODE_CTRL_STAT_AUTO_MDIX_EN); + if (ret) + return ret; + + return phy_clear_bits(phydev, CBTX_MODE_CTRL_STAT, + CBTX_MODE_CTRL_STAT_MDIX_MODE); + case ETH_TP_MDI_X: + ret = phy_clear_bits(phydev, CBTX_MODE_CTRL_STAT, + CBTX_MODE_CTRL_STAT_AUTO_MDIX_EN); + if (ret) + return ret; + + return phy_set_bits(phydev, CBTX_MODE_CTRL_STAT, + CBTX_MODE_CTRL_STAT_MDIX_MODE); + } + + return 0; +} + +static int cbtx_config_aneg(struct phy_device *phydev) +{ + int ret; + + ret = cbtx_mdix_config(phydev); + if (ret) + return ret; + + return genphy_config_aneg(phydev); +} + +static int cbtx_ack_interrupts(struct phy_device *phydev) +{ + return phy_read(phydev, CBTX_IRQ_STAT); +} + +static int cbtx_config_intr(struct phy_device *phydev) +{ + int ret; + + if (phydev->interrupts == PHY_INTERRUPT_ENABLED) { + ret = cbtx_ack_interrupts(phydev); + if (ret < 0) + return ret; + + ret = phy_write(phydev, CBTX_IRQ_ENABLE, CBTX_IRQ_LINK_DOWN | + CBTX_IRQ_AN_COMPLETE | CBTX_IRQ_ENERGYON); + if (ret) + return ret; + } else { + ret = phy_write(phydev, CBTX_IRQ_ENABLE, 0); + if (ret) + return ret; + + ret = cbtx_ack_interrupts(phydev); + if (ret < 0) + return ret; + } + + return 0; +} + +static irqreturn_t cbtx_handle_interrupt(struct phy_device *phydev) +{ + int irq_stat, irq_enabled; + + irq_stat = cbtx_ack_interrupts(phydev); + if (irq_stat < 0) { + phy_error(phydev); + return IRQ_NONE; + } + + irq_enabled = phy_read(phydev, CBTX_IRQ_ENABLE); + if (irq_enabled < 0) { + phy_error(phydev); + return IRQ_NONE; + } + + if (!(irq_enabled & irq_stat)) + return IRQ_NONE; + + phy_trigger_machine(phydev); + + return IRQ_HANDLED; +} + +static int cbtx_get_sset_count(struct phy_device *phydev) +{ + return 1; +} + +static void cbtx_get_strings(struct phy_device *phydev, u8 *data) +{ + strncpy(data, "100btx_rx_err", ETH_GSTRING_LEN); +} + +static void cbtx_get_stats(struct phy_device *phydev, + struct ethtool_stats *stats, u64 *data) +{ + int ret; + + ret = phy_read(phydev, CBTX_RX_ERR_COUNTER); + data[0] = (ret < 0) ? U64_MAX : ret; +} + +static struct phy_driver cbtx_driver[] = { + { + PHY_ID_MATCH_MODEL(PHY_ID_CBTX_SJA1110), + .name = "NXP CBTX (SJA1110)", + /* PHY_BASIC_FEATURES */ + .soft_reset = cbtx_soft_reset, + .config_init = cbtx_config_init, + .suspend = genphy_suspend, + .resume = genphy_resume, + .config_intr = cbtx_config_intr, + .handle_interrupt = cbtx_handle_interrupt, + .read_status = cbtx_read_status, + .config_aneg = cbtx_config_aneg, + .get_sset_count = cbtx_get_sset_count, + .get_strings = cbtx_get_strings, + .get_stats = cbtx_get_stats, + }, +}; + +module_phy_driver(cbtx_driver); + +static struct mdio_device_id __maybe_unused cbtx_tbl[] = { + { PHY_ID_MATCH_MODEL(PHY_ID_CBTX_SJA1110) }, + { }, +}; + +MODULE_DEVICE_TABLE(mdio, cbtx_tbl); + +MODULE_AUTHOR("Vladimir Oltean <[email protected]>"); +MODULE_DESCRIPTION("NXP CBTX PHY driver"); +MODULE_LICENSE("GPL"); diff --git a/drivers/net/phy/phy_device.c b/drivers/net/phy/phy_device.c index 538523a7cd51..d373446ab5ac 100644 --- a/drivers/net/phy/phy_device.c +++ b/drivers/net/phy/phy_device.c @@ -3284,7 +3284,8 @@ static int phy_probe(struct device *dev) /* Get the LEDs from the device tree, and instantiate standard * LEDs for them. */ - err = of_phy_leds(phydev); + if (IS_ENABLED(CONFIG_PHYLIB_LEDS)) + err = of_phy_leds(phydev); out: /* Re-assert the reset signal on error */ diff --git a/drivers/net/veth.c b/drivers/net/veth.c index e1b38fbf1dd9..4b3c6647edc6 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -1262,11 +1262,12 @@ static void veth_set_xdp_features(struct net_device *dev) peer = rtnl_dereference(priv->peer); if (peer && peer->real_num_tx_queues <= dev->real_num_rx_queues) { + struct veth_priv *priv_peer = netdev_priv(peer); xdp_features_t val = NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_REDIRECT | NETDEV_XDP_ACT_RX_SG; - if (priv->_xdp_prog || veth_gro_requested(dev)) + if (priv_peer->_xdp_prog || veth_gro_requested(peer)) val |= NETDEV_XDP_ACT_NDO_XMIT | NETDEV_XDP_ACT_NDO_XMIT_SG; xdp_set_features_flag(dev, val); @@ -1504,19 +1505,23 @@ static int veth_set_features(struct net_device *dev, { netdev_features_t changed = features ^ dev->features; struct veth_priv *priv = netdev_priv(dev); + struct net_device *peer; int err; if (!(changed & NETIF_F_GRO) || !(dev->flags & IFF_UP) || priv->_xdp_prog) return 0; + peer = rtnl_dereference(priv->peer); if (features & NETIF_F_GRO) { err = veth_napi_enable(dev); if (err) return err; - xdp_features_set_redirect_target(dev, true); + if (peer) + xdp_features_set_redirect_target(peer, true); } else { - xdp_features_clear_redirect_target(dev); + if (peer) + xdp_features_clear_redirect_target(peer); veth_napi_del(dev); } return 0; @@ -1598,13 +1603,13 @@ static int veth_xdp_set(struct net_device *dev, struct bpf_prog *prog, peer->max_mtu = max_mtu; } - xdp_features_set_redirect_target(dev, true); + xdp_features_set_redirect_target(peer, true); } if (old_prog) { if (!prog) { - if (!veth_gro_requested(dev)) - xdp_features_clear_redirect_target(dev); + if (peer && !veth_gro_requested(dev)) + xdp_features_clear_redirect_target(peer); if (dev->flags & IFF_UP) veth_disable_xdp(dev); diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index e2560b6f7980..8d8038538fc4 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -815,8 +815,13 @@ static struct page *xdp_linearize_page(struct receive_queue *rq, int page_off, unsigned int *len) { - struct page *page = alloc_page(GFP_ATOMIC); + int tailroom = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); + struct page *page; + + if (page_off + *len + tailroom > PAGE_SIZE) + return NULL; + page = alloc_page(GFP_ATOMIC); if (!page) return NULL; @@ -824,7 +829,6 @@ static struct page *xdp_linearize_page(struct receive_queue *rq, page_off += *len; while (--*num_buf) { - int tailroom = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); unsigned int buflen; void *buf; int off; diff --git a/drivers/net/vmxnet3/vmxnet3_drv.c b/drivers/net/vmxnet3/vmxnet3_drv.c index da488cbb0542..f2b76ee866a4 100644 --- a/drivers/net/vmxnet3/vmxnet3_drv.c +++ b/drivers/net/vmxnet3/vmxnet3_drv.c @@ -1504,7 +1504,7 @@ vmxnet3_rq_rx_complete(struct vmxnet3_rx_queue *rq, goto rcd_done; } - if (rxDataRingUsed) { + if (rxDataRingUsed && adapter->rxdataring_enabled) { size_t sz; BUG_ON(rcd->len > rq->data_ring.desc_size); diff --git a/drivers/net/wireless/ath/ath10k/ce.c b/drivers/net/wireless/ath/ath10k/ce.c index b656cfc03648..c27b8204718a 100644 --- a/drivers/net/wireless/ath/ath10k/ce.c +++ b/drivers/net/wireless/ath/ath10k/ce.c @@ -84,13 +84,6 @@ ath10k_set_ring_byte(unsigned int offset, return ((offset << addr_map->lsb) & addr_map->mask); } -static inline unsigned int -ath10k_get_ring_byte(unsigned int offset, - struct ath10k_hw_ce_regs_addr_map *addr_map) -{ - return ((offset & addr_map->mask) >> (addr_map->lsb)); -} - static inline u32 ath10k_ce_read32(struct ath10k *ar, u32 offset) { struct ath10k_ce *ce = ath10k_ce_priv(ar); diff --git a/drivers/net/wireless/ath/ath10k/pci.c b/drivers/net/wireless/ath/ath10k/pci.c index 728d607289c3..a7f44f6335fb 100644 --- a/drivers/net/wireless/ath/ath10k/pci.c +++ b/drivers/net/wireless/ath/ath10k/pci.c @@ -3406,15 +3406,12 @@ static int ath10k_pci_claim(struct ath10k *ar) if (!ar_pci->mem) { ath10k_err(ar, "failed to iomap BAR%d\n", BAR_NUM); ret = -EIO; - goto err_master; + goto err_region; } ath10k_dbg(ar, ATH10K_DBG_BOOT, "boot pci_mem 0x%pK\n", ar_pci->mem); return 0; -err_master: - pci_clear_master(pdev); - err_region: pci_release_region(pdev, BAR_NUM); @@ -3431,7 +3428,6 @@ static void ath10k_pci_release(struct ath10k *ar) pci_iounmap(pdev, ar_pci->mem); pci_release_region(pdev, BAR_NUM); - pci_clear_master(pdev); pci_disable_device(pdev); } diff --git a/drivers/net/wireless/ath/ath11k/ahb.c b/drivers/net/wireless/ath/ath11k/ahb.c index b549576d0b51..5cbba9a8b6ba 100644 --- a/drivers/net/wireless/ath/ath11k/ahb.c +++ b/drivers/net/wireless/ath/ath11k/ahb.c @@ -1078,6 +1078,12 @@ static int ath11k_ahb_fw_resource_deinit(struct ath11k_base *ab) struct iommu_domain *iommu; size_t unmapped_size; + /* Chipsets not requiring MSA would have not initialized + * MSA resources, return success in such cases. + */ + if (!ab->hw_params.fixed_fw_mem) + return 0; + if (ab_ahb->fw.use_tz) return 0; diff --git a/drivers/net/wireless/ath/ath11k/core.c b/drivers/net/wireless/ath/ath11k/core.c index 75fdbe4ef83a..b1b90bd34d67 100644 --- a/drivers/net/wireless/ath/ath11k/core.c +++ b/drivers/net/wireless/ath/ath11k/core.c @@ -116,7 +116,6 @@ static const struct ath11k_hw_params ath11k_hw_params[] = { .tcl_ring_retry = true, .tx_ring_size = DP_TCL_DATA_RING_SIZE, .smp2p_wow_exit = false, - .ftm_responder = true, }, { .hw_rev = ATH11K_HW_IPQ6018_HW10, @@ -199,7 +198,6 @@ static const struct ath11k_hw_params ath11k_hw_params[] = { .tx_ring_size = DP_TCL_DATA_RING_SIZE, .smp2p_wow_exit = false, .support_fw_mac_sequence = false, - .ftm_responder = true, }, { .name = "qca6390 hw2.0", @@ -284,7 +282,6 @@ static const struct ath11k_hw_params ath11k_hw_params[] = { .tx_ring_size = DP_TCL_DATA_RING_SIZE, .smp2p_wow_exit = false, .support_fw_mac_sequence = true, - .ftm_responder = false, }, { .name = "qcn9074 hw1.0", @@ -366,7 +363,6 @@ static const struct ath11k_hw_params ath11k_hw_params[] = { .tx_ring_size = DP_TCL_DATA_RING_SIZE, .smp2p_wow_exit = false, .support_fw_mac_sequence = false, - .ftm_responder = true, }, { .name = "wcn6855 hw2.0", @@ -451,7 +447,6 @@ static const struct ath11k_hw_params ath11k_hw_params[] = { .tx_ring_size = DP_TCL_DATA_RING_SIZE, .smp2p_wow_exit = false, .support_fw_mac_sequence = true, - .ftm_responder = false, }, { .name = "wcn6855 hw2.1", @@ -534,7 +529,6 @@ static const struct ath11k_hw_params ath11k_hw_params[] = { .tx_ring_size = DP_TCL_DATA_RING_SIZE, .smp2p_wow_exit = false, .support_fw_mac_sequence = true, - .ftm_responder = false, }, { .name = "wcn6750 hw1.0", @@ -599,7 +593,7 @@ static const struct ath11k_hw_params ath11k_hw_params[] = { .current_cc_support = true, .dbr_debug_support = false, .global_reset = false, - .bios_sar_capa = NULL, + .bios_sar_capa = &ath11k_hw_sar_capa_wcn6855, .m3_fw_support = false, .fixed_bdf_addr = false, .fixed_mem_region = false, @@ -615,7 +609,6 @@ static const struct ath11k_hw_params ath11k_hw_params[] = { .tx_ring_size = DP_TCL_DATA_RING_SIZE_WCN6750, .smp2p_wow_exit = true, .support_fw_mac_sequence = true, - .ftm_responder = false, }, { .hw_rev = ATH11K_HW_IPQ5018_HW10, @@ -695,7 +688,6 @@ static const struct ath11k_hw_params ath11k_hw_params[] = { .tx_ring_size = DP_TCL_DATA_RING_SIZE, .smp2p_wow_exit = false, .support_fw_mac_sequence = false, - .ftm_responder = true, }, }; diff --git a/drivers/net/wireless/ath/ath11k/dbring.c b/drivers/net/wireless/ath/ath11k/dbring.c index 2107ec05d14f..5536e8642331 100644 --- a/drivers/net/wireless/ath/ath11k/dbring.c +++ b/drivers/net/wireless/ath/ath11k/dbring.c @@ -26,13 +26,13 @@ int ath11k_dbring_validate_buffer(struct ath11k *ar, void *buffer, u32 size) static void ath11k_dbring_fill_magic_value(struct ath11k *ar, void *buffer, u32 size) { - u32 *temp; - int idx; - - size = size >> 2; + /* memset32 function fills buffer payload with the ATH11K_DB_MAGIC_VALUE + * and the variable size is expected to be the number of u32 values + * to be stored, not the number of bytes. + */ + size = size / sizeof(u32); - for (idx = 0, temp = buffer; idx < size; idx++, temp++) - *temp++ = ATH11K_DB_MAGIC_VALUE; + memset32(buffer, ATH11K_DB_MAGIC_VALUE, size); } static int ath11k_dbring_bufs_replenish(struct ath11k *ar, diff --git a/drivers/net/wireless/ath/ath11k/debugfs_htt_stats.h b/drivers/net/wireless/ath/ath11k/debugfs_htt_stats.h index 2b97cbbd28cb..0bbd58a380de 100644 --- a/drivers/net/wireless/ath/ath11k/debugfs_htt_stats.h +++ b/drivers/net/wireless/ath/ath11k/debugfs_htt_stats.h @@ -143,7 +143,8 @@ enum htt_tx_pdev_underrun_enum { /* Bytes stored in little endian order */ /* Length should be multiple of DWORD */ struct htt_stats_string_tlv { - u32 data[0]; /* Can be variable length */ + /* Can be variable length */ + DECLARE_FLEX_ARRAY(u32, data); } __packed; #define HTT_STATS_MAC_ID GENMASK(7, 0) @@ -205,27 +206,32 @@ struct htt_tx_pdev_stats_cmn_tlv { /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_tx_pdev_stats_urrn_tlv_v { - u32 urrn_stats[0]; /* HTT_TX_PDEV_MAX_URRN_STATS */ + /* HTT_TX_PDEV_MAX_URRN_STATS */ + DECLARE_FLEX_ARRAY(u32, urrn_stats); }; /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_tx_pdev_stats_flush_tlv_v { - u32 flush_errs[0]; /* HTT_TX_PDEV_MAX_FLUSH_REASON_STATS */ + /* HTT_TX_PDEV_MAX_FLUSH_REASON_STATS */ + DECLARE_FLEX_ARRAY(u32, flush_errs); }; /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_tx_pdev_stats_sifs_tlv_v { - u32 sifs_status[0]; /* HTT_TX_PDEV_MAX_SIFS_BURST_STATS */ + /* HTT_TX_PDEV_MAX_SIFS_BURST_STATS */ + DECLARE_FLEX_ARRAY(u32, sifs_status); }; /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_tx_pdev_stats_phy_err_tlv_v { - u32 phy_errs[0]; /* HTT_TX_PDEV_MAX_PHY_ERR_STATS */ + /* HTT_TX_PDEV_MAX_PHY_ERR_STATS */ + DECLARE_FLEX_ARRAY(u32, phy_errs); }; /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_tx_pdev_stats_sifs_hist_tlv_v { - u32 sifs_hist_status[0]; /* HTT_TX_PDEV_SIFS_BURST_HIST_STATS */ + /* HTT_TX_PDEV_SIFS_BURST_HIST_STATS */ + DECLARE_FLEX_ARRAY(u32, sifs_hist_status); }; struct htt_tx_pdev_stats_tx_ppdu_stats_tlv_v { @@ -590,20 +596,20 @@ struct htt_tx_hwq_difs_latency_stats_tlv_v { /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_tx_hwq_cmd_result_stats_tlv_v { - /* Histogram of sched cmd result */ - u32 cmd_result[0]; /* HTT_TX_HWQ_MAX_CMD_RESULT_STATS */ + /* Histogram of sched cmd result, HTT_TX_HWQ_MAX_CMD_RESULT_STATS */ + DECLARE_FLEX_ARRAY(u32, cmd_result); }; /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_tx_hwq_cmd_stall_stats_tlv_v { - /* Histogram of various pause conitions */ - u32 cmd_stall_status[0]; /* HTT_TX_HWQ_MAX_CMD_STALL_STATS */ + /* Histogram of various pause conitions, HTT_TX_HWQ_MAX_CMD_STALL_STATS */ + DECLARE_FLEX_ARRAY(u32, cmd_stall_status); }; /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_tx_hwq_fes_result_stats_tlv_v { - /* Histogram of number of user fes result */ - u32 fes_result[0]; /* HTT_TX_HWQ_MAX_FES_RESULT_STATS */ + /* Histogram of number of user fes result, HTT_TX_HWQ_MAX_FES_RESULT_STATS */ + DECLARE_FLEX_ARRAY(u32, fes_result); }; /* NOTE: Variable length TLV, use length spec to infer array size @@ -635,8 +641,8 @@ struct htt_tx_hwq_tried_mpdu_cnt_hist_tlv_v { * #define WAL_TXOP_USED_HISTOGRAM_INTERVAL 1000 ( 1 ms ) */ struct htt_tx_hwq_txop_used_cnt_hist_tlv_v { - /* Histogram of txop used cnt */ - u32 txop_used_cnt_hist[0]; /* HTT_TX_HWQ_TXOP_USED_CNT_HIST */ + /* Histogram of txop used cnt, HTT_TX_HWQ_TXOP_USED_CNT_HIST */ + DECLARE_FLEX_ARRAY(u32, txop_used_cnt_hist); }; /* == TX SELFGEN STATS == */ @@ -804,17 +810,20 @@ struct htt_tx_pdev_mpdu_stats_tlv { /* == TX SCHED STATS == */ /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_sched_txq_cmd_posted_tlv_v { - u32 sched_cmd_posted[0]; /* HTT_TX_PDEV_SCHED_TX_MODE_MAX */ + /* HTT_TX_PDEV_SCHED_TX_MODE_MAX */ + DECLARE_FLEX_ARRAY(u32, sched_cmd_posted); }; /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_sched_txq_cmd_reaped_tlv_v { - u32 sched_cmd_reaped[0]; /* HTT_TX_PDEV_SCHED_TX_MODE_MAX */ + /* HTT_TX_PDEV_SCHED_TX_MODE_MAX */ + DECLARE_FLEX_ARRAY(u32, sched_cmd_reaped); }; /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_sched_txq_sched_order_su_tlv_v { - u32 sched_order_su[0]; /* HTT_TX_PDEV_NUM_SCHED_ORDER_LOG */ + /* HTT_TX_PDEV_NUM_SCHED_ORDER_LOG */ + DECLARE_FLEX_ARRAY(u32, sched_order_su); }; enum htt_sched_txq_sched_ineligibility_tlv_enum { @@ -842,7 +851,7 @@ enum htt_sched_txq_sched_ineligibility_tlv_enum { /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_sched_txq_sched_ineligibility_tlv_v { /* indexed by htt_sched_txq_sched_ineligibility_tlv_enum */ - u32 sched_ineligibility[0]; + DECLARE_FLEX_ARRAY(u32, sched_ineligibility); }; #define HTT_TX_PDEV_STATS_SCHED_PER_TXQ_MAC_ID GENMASK(7, 0) @@ -888,18 +897,20 @@ struct htt_stats_tx_sched_cmn_tlv { /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_tx_tqm_gen_mpdu_stats_tlv_v { - u32 gen_mpdu_end_reason[0]; /* HTT_TX_TQM_MAX_GEN_MPDU_END_REASON */ + /* HTT_TX_TQM_MAX_GEN_MPDU_END_REASON */ + DECLARE_FLEX_ARRAY(u32, gen_mpdu_end_reason); }; /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_tx_tqm_list_mpdu_stats_tlv_v { - u32 list_mpdu_end_reason[0]; /* HTT_TX_TQM_MAX_LIST_MPDU_END_REASON */ + /* HTT_TX_TQM_MAX_LIST_MPDU_END_REASON */ + DECLARE_FLEX_ARRAY(u32, list_mpdu_end_reason); }; /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_tx_tqm_list_mpdu_cnt_tlv_v { - u32 list_mpdu_cnt_hist[0]; - /* HTT_TX_TQM_MAX_LIST_MPDU_CNT_HISTOGRAM_BINS */ + /* HTT_TX_TQM_MAX_LIST_MPDU_CNT_HISTOGRAM_BINS */ + DECLARE_FLEX_ARRAY(u32, list_mpdu_cnt_hist); }; struct htt_tx_tqm_pdev_stats_tlv_v { @@ -1098,7 +1109,7 @@ struct htt_tx_de_compl_stats_tlv { * ENTRIES_PER_BIN_COUNT) */ struct htt_tx_de_fw2wbm_ring_full_hist_tlv { - u32 fw2wbm_ring_full_hist[0]; + DECLARE_FLEX_ARRAY(u32, fw2wbm_ring_full_hist); }; struct htt_tx_de_cmn_stats_tlv { @@ -1151,7 +1162,7 @@ struct htt_ring_if_cmn_tlv { /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_sfm_client_user_tlv_v { /* Number of DWORDS used per user and per client */ - u32 dwords_used_by_user_n[0]; + DECLARE_FLEX_ARRAY(u32, dwords_used_by_user_n); }; struct htt_sfm_client_tlv { @@ -1436,12 +1447,14 @@ struct htt_rx_soc_fw_stats_tlv { /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_rx_soc_fw_refill_ring_empty_tlv_v { - u32 refill_ring_empty_cnt[0]; /* HTT_RX_STATS_REFILL_MAX_RING */ + /* HTT_RX_STATS_REFILL_MAX_RING */ + DECLARE_FLEX_ARRAY(u32, refill_ring_empty_cnt); }; /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_rx_soc_fw_refill_ring_num_refill_tlv_v { - u32 refill_ring_num_refill[0]; /* HTT_RX_STATS_REFILL_MAX_RING */ + /* HTT_RX_STATS_REFILL_MAX_RING */ + DECLARE_FLEX_ARRAY(u32, refill_ring_num_refill); }; /* RXDMA error code from WBM released packets */ @@ -1473,7 +1486,7 @@ enum htt_rx_rxdma_error_code_enum { /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_rx_soc_fw_refill_ring_num_rxdma_err_tlv_v { - u32 rxdma_err[0]; /* HTT_RX_RXDMA_MAX_ERR_CODE */ + DECLARE_FLEX_ARRAY(u32, rxdma_err); /* HTT_RX_RXDMA_MAX_ERR_CODE */ }; /* REO error code from WBM released packets */ @@ -1505,7 +1518,7 @@ enum htt_rx_reo_error_code_enum { /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_rx_soc_fw_refill_ring_num_reo_err_tlv_v { - u32 reo_err[0]; /* HTT_RX_REO_MAX_ERR_CODE */ + DECLARE_FLEX_ARRAY(u32, reo_err); /* HTT_RX_REO_MAX_ERR_CODE */ }; /* == RX PDEV STATS == */ @@ -1622,13 +1635,13 @@ struct htt_rx_pdev_fw_stats_phy_err_tlv { /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_rx_pdev_fw_ring_mpdu_err_tlv_v { /* Num error MPDU for each RxDMA error type */ - u32 fw_ring_mpdu_err[0]; /* HTT_RX_STATS_RXDMA_MAX_ERR */ + DECLARE_FLEX_ARRAY(u32, fw_ring_mpdu_err); /* HTT_RX_STATS_RXDMA_MAX_ERR */ }; /* NOTE: Variable length TLV, use length spec to infer array size */ struct htt_rx_pdev_fw_mpdu_drop_tlv_v { /* Num MPDU dropped */ - u32 fw_mpdu_drop[0]; /* HTT_RX_STATS_FW_DROP_REASON_MAX */ + DECLARE_FLEX_ARRAY(u32, fw_mpdu_drop); /* HTT_RX_STATS_FW_DROP_REASON_MAX */ }; #define HTT_PDEV_CCA_STATS_TX_FRAME_INFO_PRESENT (0x1) diff --git a/drivers/net/wireless/ath/ath11k/dp.c b/drivers/net/wireless/ath/ath11k/dp.c index f5156a7fbdd7..d070bcb3fe24 100644 --- a/drivers/net/wireless/ath/ath11k/dp.c +++ b/drivers/net/wireless/ath/ath11k/dp.c @@ -36,6 +36,7 @@ void ath11k_dp_peer_cleanup(struct ath11k *ar, int vdev_id, const u8 *addr) } ath11k_peer_rx_tid_cleanup(ar, peer); + peer->dp_setup_done = false; crypto_free_shash(peer->tfm_mmic); spin_unlock_bh(&ab->base_lock); } @@ -72,7 +73,8 @@ int ath11k_dp_peer_setup(struct ath11k *ar, int vdev_id, const u8 *addr) ret = ath11k_peer_rx_frag_setup(ar, addr, vdev_id); if (ret) { ath11k_warn(ab, "failed to setup rx defrag context\n"); - return ret; + tid--; + goto peer_clean; } /* TODO: Setup other peer specific resource used in data path */ diff --git a/drivers/net/wireless/ath/ath11k/dp.h b/drivers/net/wireless/ath/ath11k/dp.h index be9eafc872b3..d04f78ab6b37 100644 --- a/drivers/net/wireless/ath/ath11k/dp.h +++ b/drivers/net/wireless/ath/ath11k/dp.h @@ -214,7 +214,7 @@ struct ath11k_pdev_dp { #define DP_REO_REINJECT_RING_SIZE 32 #define DP_RX_RELEASE_RING_SIZE 1024 #define DP_REO_EXCEPTION_RING_SIZE 128 -#define DP_REO_CMD_RING_SIZE 128 +#define DP_REO_CMD_RING_SIZE 256 #define DP_REO_STATUS_RING_SIZE 2048 #define DP_RXDMA_BUF_RING_SIZE 4096 #define DP_RXDMA_REFILL_RING_SIZE 2048 @@ -303,12 +303,16 @@ struct ath11k_dp { #define HTT_TX_WBM_COMP_STATUS_OFFSET 8 +#define HTT_INVALID_PEER_ID 0xffff + /* HTT tx completion is overlaid in wbm_release_ring */ #define HTT_TX_WBM_COMP_INFO0_STATUS GENMASK(12, 9) #define HTT_TX_WBM_COMP_INFO0_REINJECT_REASON GENMASK(16, 13) #define HTT_TX_WBM_COMP_INFO0_REINJECT_REASON GENMASK(16, 13) #define HTT_TX_WBM_COMP_INFO1_ACK_RSSI GENMASK(31, 24) +#define HTT_TX_WBM_COMP_INFO2_SW_PEER_ID GENMASK(15, 0) +#define HTT_TX_WBM_COMP_INFO2_VALID BIT(21) struct htt_tx_wbm_completion { u32 info0; diff --git a/drivers/net/wireless/ath/ath11k/dp_rx.c b/drivers/net/wireless/ath/ath11k/dp_rx.c index b65a84a88264..f67ce62b2b48 100644 --- a/drivers/net/wireless/ath/ath11k/dp_rx.c +++ b/drivers/net/wireless/ath/ath11k/dp_rx.c @@ -389,10 +389,10 @@ int ath11k_dp_rxbufs_replenish(struct ath11k_base *ab, int mac_id, goto fail_free_skb; spin_lock_bh(&rx_ring->idr_lock); - buf_id = idr_alloc(&rx_ring->bufs_idr, skb, 0, - rx_ring->bufs_max * 3, GFP_ATOMIC); + buf_id = idr_alloc(&rx_ring->bufs_idr, skb, 1, + (rx_ring->bufs_max * 3) + 1, GFP_ATOMIC); spin_unlock_bh(&rx_ring->idr_lock); - if (buf_id < 0) + if (buf_id <= 0) goto fail_dma_unmap; desc = ath11k_hal_srng_src_get_next_entry(ab, srng); @@ -435,7 +435,6 @@ fail_free_skb: static int ath11k_dp_rxdma_buf_ring_free(struct ath11k *ar, struct dp_rxdma_ring *rx_ring) { - struct ath11k_pdev_dp *dp = &ar->dp; struct sk_buff *skb; int buf_id; @@ -453,28 +452,6 @@ static int ath11k_dp_rxdma_buf_ring_free(struct ath11k *ar, idr_destroy(&rx_ring->bufs_idr); spin_unlock_bh(&rx_ring->idr_lock); - /* if rxdma1_enable is false, mon_status_refill_ring - * isn't setup, so don't clean. - */ - if (!ar->ab->hw_params.rxdma1_enable) - return 0; - - rx_ring = &dp->rx_mon_status_refill_ring[0]; - - spin_lock_bh(&rx_ring->idr_lock); - idr_for_each_entry(&rx_ring->bufs_idr, skb, buf_id) { - idr_remove(&rx_ring->bufs_idr, buf_id); - /* XXX: Understand where internal driver does this dma_unmap - * of rxdma_buffer. - */ - dma_unmap_single(ar->ab->dev, ATH11K_SKB_RXCB(skb)->paddr, - skb->len + skb_tailroom(skb), DMA_BIDIRECTIONAL); - dev_kfree_skb_any(skb); - } - - idr_destroy(&rx_ring->bufs_idr); - spin_unlock_bh(&rx_ring->idr_lock); - return 0; } @@ -691,13 +668,18 @@ void ath11k_dp_reo_cmd_list_cleanup(struct ath11k_base *ab) struct ath11k_dp *dp = &ab->dp; struct dp_reo_cmd *cmd, *tmp; struct dp_reo_cache_flush_elem *cmd_cache, *tmp_cache; + struct dp_rx_tid *rx_tid; spin_lock_bh(&dp->reo_cmd_lock); list_for_each_entry_safe(cmd, tmp, &dp->reo_cmd_list, list) { list_del(&cmd->list); - dma_unmap_single(ab->dev, cmd->data.paddr, - cmd->data.size, DMA_BIDIRECTIONAL); - kfree(cmd->data.vaddr); + rx_tid = &cmd->data; + if (rx_tid->vaddr) { + dma_unmap_single(ab->dev, rx_tid->paddr, + rx_tid->size, DMA_BIDIRECTIONAL); + kfree(rx_tid->vaddr); + rx_tid->vaddr = NULL; + } kfree(cmd); } @@ -705,9 +687,13 @@ void ath11k_dp_reo_cmd_list_cleanup(struct ath11k_base *ab) &dp->reo_cmd_cache_flush_list, list) { list_del(&cmd_cache->list); dp->reo_cmd_cache_flush_count--; - dma_unmap_single(ab->dev, cmd_cache->data.paddr, - cmd_cache->data.size, DMA_BIDIRECTIONAL); - kfree(cmd_cache->data.vaddr); + rx_tid = &cmd_cache->data; + if (rx_tid->vaddr) { + dma_unmap_single(ab->dev, rx_tid->paddr, + rx_tid->size, DMA_BIDIRECTIONAL); + kfree(rx_tid->vaddr); + rx_tid->vaddr = NULL; + } kfree(cmd_cache); } spin_unlock_bh(&dp->reo_cmd_lock); @@ -721,10 +707,12 @@ static void ath11k_dp_reo_cmd_free(struct ath11k_dp *dp, void *ctx, if (status != HAL_REO_CMD_SUCCESS) ath11k_warn(dp->ab, "failed to flush rx tid hw desc, tid %d status %d\n", rx_tid->tid, status); - - dma_unmap_single(dp->ab->dev, rx_tid->paddr, rx_tid->size, - DMA_BIDIRECTIONAL); - kfree(rx_tid->vaddr); + if (rx_tid->vaddr) { + dma_unmap_single(dp->ab->dev, rx_tid->paddr, rx_tid->size, + DMA_BIDIRECTIONAL); + kfree(rx_tid->vaddr); + rx_tid->vaddr = NULL; + } } static void ath11k_dp_reo_cache_flush(struct ath11k_base *ab, @@ -763,6 +751,7 @@ static void ath11k_dp_reo_cache_flush(struct ath11k_base *ab, dma_unmap_single(ab->dev, rx_tid->paddr, rx_tid->size, DMA_BIDIRECTIONAL); kfree(rx_tid->vaddr); + rx_tid->vaddr = NULL; } } @@ -815,6 +804,7 @@ free_desc: dma_unmap_single(ab->dev, rx_tid->paddr, rx_tid->size, DMA_BIDIRECTIONAL); kfree(rx_tid->vaddr); + rx_tid->vaddr = NULL; } void ath11k_peer_rx_tid_delete(struct ath11k *ar, @@ -827,6 +817,8 @@ void ath11k_peer_rx_tid_delete(struct ath11k *ar, if (!rx_tid->active) return; + rx_tid->active = false; + cmd.flag = HAL_REO_CMD_FLG_NEED_STATUS; cmd.addr_lo = lower_32_bits(rx_tid->paddr); cmd.addr_hi = upper_32_bits(rx_tid->paddr); @@ -841,9 +833,11 @@ void ath11k_peer_rx_tid_delete(struct ath11k *ar, dma_unmap_single(ar->ab->dev, rx_tid->paddr, rx_tid->size, DMA_BIDIRECTIONAL); kfree(rx_tid->vaddr); + rx_tid->vaddr = NULL; } - rx_tid->active = false; + rx_tid->paddr = 0; + rx_tid->size = 0; } static int ath11k_dp_rx_link_desc_return(struct ath11k_base *ab, @@ -990,6 +984,7 @@ static void ath11k_dp_rx_tid_mem_free(struct ath11k_base *ab, dma_unmap_single(ab->dev, rx_tid->paddr, rx_tid->size, DMA_BIDIRECTIONAL); kfree(rx_tid->vaddr); + rx_tid->vaddr = NULL; rx_tid->active = false; @@ -1014,7 +1009,8 @@ int ath11k_peer_rx_tid_setup(struct ath11k *ar, const u8 *peer_mac, int vdev_id, peer = ath11k_peer_find(ab, vdev_id, peer_mac); if (!peer) { - ath11k_warn(ab, "failed to find the peer to set up rx tid\n"); + ath11k_warn(ab, "failed to find the peer %pM to set up rx tid\n", + peer_mac); spin_unlock_bh(&ab->base_lock); return -ENOENT; } @@ -1027,7 +1023,8 @@ int ath11k_peer_rx_tid_setup(struct ath11k *ar, const u8 *peer_mac, int vdev_id, ba_win_sz, ssn, true); spin_unlock_bh(&ab->base_lock); if (ret) { - ath11k_warn(ab, "failed to update reo for rx tid %d\n", tid); + ath11k_warn(ab, "failed to update reo for peer %pM rx tid %d\n: %d", + peer_mac, tid, ret); return ret; } @@ -1035,8 +1032,8 @@ int ath11k_peer_rx_tid_setup(struct ath11k *ar, const u8 *peer_mac, int vdev_id, peer_mac, paddr, tid, 1, ba_win_sz); if (ret) - ath11k_warn(ab, "failed to send wmi command to update rx reorder queue, tid :%d (%d)\n", - tid, ret); + ath11k_warn(ab, "failed to send wmi rx reorder queue for peer %pM tid %d: %d\n", + peer_mac, tid, ret); return ret; } @@ -1069,6 +1066,8 @@ int ath11k_peer_rx_tid_setup(struct ath11k *ar, const u8 *peer_mac, int vdev_id, ret = dma_mapping_error(ab->dev, paddr); if (ret) { spin_unlock_bh(&ab->base_lock); + ath11k_warn(ab, "failed to setup dma map for peer %pM rx tid %d: %d\n", + peer_mac, tid, ret); goto err_mem_free; } @@ -1082,15 +1081,16 @@ int ath11k_peer_rx_tid_setup(struct ath11k *ar, const u8 *peer_mac, int vdev_id, ret = ath11k_wmi_peer_rx_reorder_queue_setup(ar, vdev_id, peer_mac, paddr, tid, 1, ba_win_sz); if (ret) { - ath11k_warn(ar->ab, "failed to setup rx reorder queue, tid :%d (%d)\n", - tid, ret); + ath11k_warn(ar->ab, "failed to setup rx reorder queue for peer %pM tid %d: %d\n", + peer_mac, tid, ret); ath11k_dp_rx_tid_mem_free(ab, peer_mac, vdev_id, tid); } return ret; err_mem_free: - kfree(vaddr); + kfree(rx_tid->vaddr); + rx_tid->vaddr = NULL; return ret; } @@ -2665,6 +2665,9 @@ try_again: cookie); mac_id = FIELD_GET(DP_RXDMA_BUF_COOKIE_PDEV_ID, cookie); + if (unlikely(buf_id == 0)) + continue; + ar = ab->pdevs[mac_id].ar; rx_ring = &ar->dp.rx_refill_buf_ring; spin_lock_bh(&rx_ring->idr_lock); @@ -3029,39 +3032,51 @@ static int ath11k_dp_rx_reap_mon_status_ring(struct ath11k_base *ab, int mac_id, spin_lock_bh(&rx_ring->idr_lock); skb = idr_find(&rx_ring->bufs_idr, buf_id); + spin_unlock_bh(&rx_ring->idr_lock); + if (!skb) { ath11k_warn(ab, "rx monitor status with invalid buf_id %d\n", buf_id); - spin_unlock_bh(&rx_ring->idr_lock); pmon->buf_state = DP_MON_STATUS_REPLINISH; goto move_next; } - idr_remove(&rx_ring->bufs_idr, buf_id); - spin_unlock_bh(&rx_ring->idr_lock); - rxcb = ATH11K_SKB_RXCB(skb); - dma_unmap_single(ab->dev, rxcb->paddr, - skb->len + skb_tailroom(skb), - DMA_FROM_DEVICE); + dma_sync_single_for_cpu(ab->dev, rxcb->paddr, + skb->len + skb_tailroom(skb), + DMA_FROM_DEVICE); tlv = (struct hal_tlv_hdr *)skb->data; if (FIELD_GET(HAL_TLV_HDR_TAG, tlv->tl) != HAL_RX_STATUS_BUFFER_DONE) { - ath11k_warn(ab, "mon status DONE not set %lx\n", + ath11k_warn(ab, "mon status DONE not set %lx, buf_id %d\n", FIELD_GET(HAL_TLV_HDR_TAG, - tlv->tl)); - dev_kfree_skb_any(skb); + tlv->tl), buf_id); + /* If done status is missing, hold onto status + * ring until status is done for this status + * ring buffer. + * Keep HP in mon_status_ring unchanged, + * and break from here. + * Check status for same buffer for next time + */ pmon->buf_state = DP_MON_STATUS_NO_DMA; - goto move_next; + break; } + spin_lock_bh(&rx_ring->idr_lock); + idr_remove(&rx_ring->bufs_idr, buf_id); + spin_unlock_bh(&rx_ring->idr_lock); if (ab->hw_params.full_monitor_mode) { ath11k_dp_rx_mon_update_status_buf_state(pmon, tlv); if (paddr == pmon->mon_status_paddr) pmon->buf_state = DP_MON_STATUS_MATCH; } + + dma_unmap_single(ab->dev, rxcb->paddr, + skb->len + skb_tailroom(skb), + DMA_FROM_DEVICE); + __skb_queue_tail(skb_list, skb); } else { pmon->buf_state = DP_MON_STATUS_REPLINISH; @@ -3117,8 +3132,11 @@ int ath11k_peer_rx_frag_setup(struct ath11k *ar, const u8 *peer_mac, int vdev_id int i; tfm = crypto_alloc_shash("michael_mic", 0, 0); - if (IS_ERR(tfm)) + if (IS_ERR(tfm)) { + ath11k_warn(ab, "failed to allocate michael_mic shash: %ld\n", + PTR_ERR(tfm)); return PTR_ERR(tfm); + } spin_lock_bh(&ab->base_lock); @@ -3138,6 +3156,7 @@ int ath11k_peer_rx_frag_setup(struct ath11k *ar, const u8 *peer_mac, int vdev_id } peer->tfm_mmic = tfm; + peer->dp_setup_done = true; spin_unlock_bh(&ab->base_lock); return 0; @@ -3583,6 +3602,13 @@ static int ath11k_dp_rx_frag_h_mpdu(struct ath11k *ar, ret = -ENOENT; goto out_unlock; } + if (!peer->dp_setup_done) { + ath11k_warn(ab, "The peer %pM [%d] has uninitialized datapath\n", + peer->addr, peer_id); + ret = -ENOENT; + goto out_unlock; + } + rx_tid = &peer->rx_tid[tid]; if ((!skb_queue_empty(&rx_tid->rx_frags) && seqno != rx_tid->cur_sn) || @@ -3598,7 +3624,7 @@ static int ath11k_dp_rx_frag_h_mpdu(struct ath11k *ar, goto out_unlock; } - if (frag_no > __fls(rx_tid->rx_frag_bitmap)) + if (!rx_tid->rx_frag_bitmap || (frag_no > __fls(rx_tid->rx_frag_bitmap))) __skb_queue_tail(&rx_tid->rx_frags, msdu); else ath11k_dp_rx_h_sort_frags(ar, &rx_tid->rx_frags, msdu); diff --git a/drivers/net/wireless/ath/ath11k/dp_tx.c b/drivers/net/wireless/ath/ath11k/dp_tx.c index 8afbba236935..08a28464eb7a 100644 --- a/drivers/net/wireless/ath/ath11k/dp_tx.c +++ b/drivers/net/wireless/ath/ath11k/dp_tx.c @@ -316,10 +316,12 @@ ath11k_dp_tx_htt_tx_complete_buf(struct ath11k_base *ab, struct dp_tx_ring *tx_ring, struct ath11k_dp_htt_wbm_tx_status *ts) { + struct ieee80211_tx_status status = { 0 }; struct sk_buff *msdu; struct ieee80211_tx_info *info; struct ath11k_skb_cb *skb_cb; struct ath11k *ar; + struct ath11k_peer *peer; spin_lock(&tx_ring->tx_idr_lock); msdu = idr_remove(&tx_ring->txbuf_idr, ts->msdu_id); @@ -341,6 +343,11 @@ ath11k_dp_tx_htt_tx_complete_buf(struct ath11k_base *ab, dma_unmap_single(ab->dev, skb_cb->paddr, msdu->len, DMA_TO_DEVICE); + if (!skb_cb->vif) { + dev_kfree_skb_any(msdu); + return; + } + memset(&info->status, 0, sizeof(info->status)); if (ts->acked) { @@ -355,7 +362,23 @@ ath11k_dp_tx_htt_tx_complete_buf(struct ath11k_base *ab, } } - ieee80211_tx_status(ar->hw, msdu); + spin_lock_bh(&ab->base_lock); + peer = ath11k_peer_find_by_id(ab, ts->peer_id); + if (!peer || !peer->sta) { + ath11k_dbg(ab, ATH11K_DBG_DATA, + "dp_tx: failed to find the peer with peer_id %d\n", + ts->peer_id); + spin_unlock_bh(&ab->base_lock); + dev_kfree_skb_any(msdu); + return; + } + spin_unlock_bh(&ab->base_lock); + + status.sta = peer->sta; + status.info = info; + status.skb = msdu; + + ieee80211_tx_status_ext(ar->hw, &status); } static void @@ -379,7 +402,15 @@ ath11k_dp_tx_process_htt_tx_complete(struct ath11k_base *ab, ts.msdu_id = msdu_id; ts.ack_rssi = FIELD_GET(HTT_TX_WBM_COMP_INFO1_ACK_RSSI, status_desc->info1); + + if (FIELD_GET(HTT_TX_WBM_COMP_INFO2_VALID, status_desc->info2)) + ts.peer_id = FIELD_GET(HTT_TX_WBM_COMP_INFO2_SW_PEER_ID, + status_desc->info2); + else + ts.peer_id = HTT_INVALID_PEER_ID; + ath11k_dp_tx_htt_tx_complete_buf(ab, tx_ring, &ts); + break; case HAL_WBM_REL_HTT_TX_COMP_STATUS_REINJ: case HAL_WBM_REL_HTT_TX_COMP_STATUS_INSPECT: diff --git a/drivers/net/wireless/ath/ath11k/dp_tx.h b/drivers/net/wireless/ath/ath11k/dp_tx.h index e87d65bfbf06..68a21ea9b934 100644 --- a/drivers/net/wireless/ath/ath11k/dp_tx.h +++ b/drivers/net/wireless/ath/ath11k/dp_tx.h @@ -13,6 +13,7 @@ struct ath11k_dp_htt_wbm_tx_status { u32 msdu_id; bool acked; int ack_rssi; + u16 peer_id; }; void ath11k_dp_tx_update_txcompl(struct ath11k *ar, struct hal_tx_status *ts); diff --git a/drivers/net/wireless/ath/ath11k/hal_rx.c b/drivers/net/wireless/ath/ath11k/hal_rx.c index 7f39c6fb7408..bb1d40034aa8 100644 --- a/drivers/net/wireless/ath/ath11k/hal_rx.c +++ b/drivers/net/wireless/ath/ath11k/hal_rx.c @@ -865,6 +865,12 @@ ath11k_hal_rx_populate_mu_user_info(void *rx_tlv, struct hal_rx_mon_ppdu_info *p ath11k_hal_rx_populate_byte_count(rx_tlv, ppdu_info, rx_user_status); } +static u16 ath11k_hal_rx_mpduinfo_get_peerid(struct ath11k_base *ab, + struct hal_rx_mpdu_info *mpdu_info) +{ + return ab->hw_params.hw_ops->mpdu_info_get_peerid(mpdu_info); +} + static enum hal_rx_mon_status ath11k_hal_rx_parse_mon_status_tlv(struct ath11k_base *ab, struct hal_rx_mon_ppdu_info *ppdu_info, @@ -1023,7 +1029,7 @@ ath11k_hal_rx_parse_mon_status_tlv(struct ath11k_base *ab, info1 = __le32_to_cpu(vht_sig->info1); ppdu_info->ldpc = FIELD_GET(HAL_RX_VHT_SIG_A_INFO_INFO1_SU_MU_CODING, - info0); + info1); ppdu_info->mcs = FIELD_GET(HAL_RX_VHT_SIG_A_INFO_INFO1_MCS, info1); gi_setting = FIELD_GET(HAL_RX_VHT_SIG_A_INFO_INFO1_GI_SETTING, @@ -1446,7 +1452,7 @@ ath11k_hal_rx_parse_mon_status_tlv(struct ath11k_base *ab, * PHYRX_OTHER_RECEIVE_INFO TLV. */ ppdu_info->rssi_comb = - FIELD_GET(HAL_RX_PHYRX_RSSI_LEGACY_INFO_INFO1_RSSI_COMB, + FIELD_GET(HAL_RX_PHYRX_RSSI_LEGACY_INFO_INFO0_RSSI_COMB, __le32_to_cpu(rssi->info0)); if (db2dbm) { @@ -1459,9 +1465,11 @@ ath11k_hal_rx_parse_mon_status_tlv(struct ath11k_base *ab, break; } case HAL_RX_MPDU_START: { + struct hal_rx_mpdu_info *mpdu_info = + (struct hal_rx_mpdu_info *)tlv_data; u16 peer_id; - peer_id = ab->hw_params.hw_ops->mpdu_info_get_peerid(tlv_data); + peer_id = ath11k_hal_rx_mpduinfo_get_peerid(ab, mpdu_info); if (peer_id) ppdu_info->peer_id = peer_id; break; diff --git a/drivers/net/wireless/ath/ath11k/hal_rx.h b/drivers/net/wireless/ath/ath11k/hal_rx.h index f6bae07abfd3..61bd8416c4fd 100644 --- a/drivers/net/wireless/ath/ath11k/hal_rx.h +++ b/drivers/net/wireless/ath/ath11k/hal_rx.h @@ -385,7 +385,7 @@ struct hal_rx_he_sig_b2_ofdma_info { __le32 info0; } __packed; -#define HAL_RX_PHYRX_RSSI_LEGACY_INFO_INFO1_RSSI_COMB GENMASK(15, 8) +#define HAL_RX_PHYRX_RSSI_LEGACY_INFO_INFO0_RSSI_COMB GENMASK(15, 8) #define HAL_RX_PHYRX_RSSI_PREAMBLE_PRI20 GENMASK(7, 0) @@ -405,7 +405,7 @@ struct hal_rx_phyrx_rssi_legacy_info { #define HAL_RX_MPDU_INFO_INFO0_PEERID_WCN6855 GENMASK(15, 0) #define HAL_RX_MPDU_INFO_INFO1_MPDU_LEN GENMASK(13, 0) -struct hal_rx_mpdu_info { +struct hal_rx_mpdu_info_ipq8074 { __le32 rsvd0; __le32 info0; __le32 rsvd1[11]; @@ -413,12 +413,28 @@ struct hal_rx_mpdu_info { __le32 rsvd2[9]; } __packed; +struct hal_rx_mpdu_info_qcn9074 { + __le32 rsvd0[10]; + __le32 info0; + __le32 rsvd1[2]; + __le32 info1; + __le32 rsvd2[9]; +} __packed; + struct hal_rx_mpdu_info_wcn6855 { __le32 rsvd0[8]; __le32 info0; __le32 rsvd1[14]; } __packed; +struct hal_rx_mpdu_info { + union { + struct hal_rx_mpdu_info_ipq8074 ipq8074; + struct hal_rx_mpdu_info_qcn9074 qcn9074; + struct hal_rx_mpdu_info_wcn6855 wcn6855; + } u; +} __packed; + #define HAL_RX_PPDU_END_DURATION GENMASK(23, 0) struct hal_rx_ppdu_end_duration { __le32 rsvd0[9]; diff --git a/drivers/net/wireless/ath/ath11k/hw.c b/drivers/net/wireless/ath/ath11k/hw.c index 60ac215e0678..eb995f9cf0fa 100644 --- a/drivers/net/wireless/ath/ath11k/hw.c +++ b/drivers/net/wireless/ath/ath11k/hw.c @@ -835,26 +835,35 @@ static void ath11k_hw_ipq5018_reo_setup(struct ath11k_base *ab) ring_hash_map); } -static u16 ath11k_hw_ipq8074_mpdu_info_get_peerid(u8 *tlv_data) +static u16 +ath11k_hw_ipq8074_mpdu_info_get_peerid(struct hal_rx_mpdu_info *mpdu_info) { u16 peer_id = 0; - struct hal_rx_mpdu_info *mpdu_info = - (struct hal_rx_mpdu_info *)tlv_data; peer_id = FIELD_GET(HAL_RX_MPDU_INFO_INFO0_PEERID, - __le32_to_cpu(mpdu_info->info0)); + __le32_to_cpu(mpdu_info->u.ipq8074.info0)); return peer_id; } -static u16 ath11k_hw_wcn6855_mpdu_info_get_peerid(u8 *tlv_data) +static u16 +ath11k_hw_qcn9074_mpdu_info_get_peerid(struct hal_rx_mpdu_info *mpdu_info) +{ + u16 peer_id = 0; + + peer_id = FIELD_GET(HAL_RX_MPDU_INFO_INFO0_PEERID, + __le32_to_cpu(mpdu_info->u.qcn9074.info0)); + + return peer_id; +} + +static u16 +ath11k_hw_wcn6855_mpdu_info_get_peerid(struct hal_rx_mpdu_info *mpdu_info) { u16 peer_id = 0; - struct hal_rx_mpdu_info_wcn6855 *mpdu_info = - (struct hal_rx_mpdu_info_wcn6855 *)tlv_data; peer_id = FIELD_GET(HAL_RX_MPDU_INFO_INFO0_PEERID_WCN6855, - __le32_to_cpu(mpdu_info->info0)); + __le32_to_cpu(mpdu_info->u.wcn6855.info0)); return peer_id; } @@ -1042,7 +1051,7 @@ const struct ath11k_hw_ops qcn9074_ops = { .rx_desc_get_attention = ath11k_hw_qcn9074_rx_desc_get_attention, .rx_desc_get_msdu_payload = ath11k_hw_qcn9074_rx_desc_get_msdu_payload, .reo_setup = ath11k_hw_ipq8074_reo_setup, - .mpdu_info_get_peerid = ath11k_hw_ipq8074_mpdu_info_get_peerid, + .mpdu_info_get_peerid = ath11k_hw_qcn9074_mpdu_info_get_peerid, .rx_desc_mac_addr2_valid = ath11k_hw_ipq9074_rx_desc_mac_addr2_valid, .rx_desc_mpdu_start_addr2 = ath11k_hw_ipq9074_rx_desc_mpdu_start_addr2, .get_ring_selector = ath11k_hw_ipq8074_get_tcl_ring_selector, @@ -1224,6 +1233,7 @@ const struct ath11k_hw_ring_mask ath11k_hw_ring_mask_ipq8074 = { ATH11K_RX_WBM_REL_RING_MASK_0, }, .reo_status = { + 0, 0, 0, ATH11K_REO_STATUS_RING_MASK_0, }, .rxdma2host = { diff --git a/drivers/net/wireless/ath/ath11k/hw.h b/drivers/net/wireless/ath/ath11k/hw.h index 0be4e1232384..6a5dd2dbdb3a 100644 --- a/drivers/net/wireless/ath/ath11k/hw.h +++ b/drivers/net/wireless/ath/ath11k/hw.h @@ -224,7 +224,6 @@ struct ath11k_hw_params { u32 tx_ring_size; bool smp2p_wow_exit; bool support_fw_mac_sequence; - bool ftm_responder; }; struct ath11k_hw_ops { @@ -264,7 +263,7 @@ struct ath11k_hw_ops { struct rx_attention *(*rx_desc_get_attention)(struct hal_rx_desc *desc); u8 *(*rx_desc_get_msdu_payload)(struct hal_rx_desc *desc); void (*reo_setup)(struct ath11k_base *ab); - u16 (*mpdu_info_get_peerid)(u8 *tlv_data); + u16 (*mpdu_info_get_peerid)(struct hal_rx_mpdu_info *mpdu_info); bool (*rx_desc_mac_addr2_valid)(struct hal_rx_desc *desc); u8* (*rx_desc_mpdu_start_addr2)(struct hal_rx_desc *desc); u32 (*get_ring_selector)(struct sk_buff *skb); diff --git a/drivers/net/wireless/ath/ath11k/mac.c b/drivers/net/wireless/ath/ath11k/mac.c index cad832e0e6b8..1c93f1afccc5 100644 --- a/drivers/net/wireless/ath/ath11k/mac.c +++ b/drivers/net/wireless/ath/ath11k/mac.c @@ -3538,7 +3538,7 @@ static void ath11k_mac_op_bss_info_changed(struct ieee80211_hw *hw, if (changed & BSS_CHANGED_FTM_RESPONDER && arvif->ftm_responder != info->ftm_responder && - ar->ab->hw_params.ftm_responder && + test_bit(WMI_TLV_SERVICE_RTT, ar->ab->wmi_ab.svc_map) && (vif->type == NL80211_IFTYPE_AP || vif->type == NL80211_IFTYPE_MESH_POINT)) { arvif->ftm_responder = info->ftm_responder; @@ -3755,6 +3755,18 @@ static int ath11k_mac_op_hw_scan(struct ieee80211_hw *hw, int i; u32 scan_timeout; + /* Firmwares advertising the support of triggering 11D algorithm + * on the scan results of a regular scan expects driver to send + * WMI_11D_SCAN_START_CMDID before sending WMI_START_SCAN_CMDID. + * With this feature, separate 11D scan can be avoided since + * regdomain can be determined with the scan results of the + * regular scan. + */ + if (ar->state_11d == ATH11K_11D_PREPARING && + test_bit(WMI_TLV_SERVICE_SUPPORT_11D_FOR_HOST_SCAN, + ar->ab->wmi_ab.svc_map)) + ath11k_mac_11d_scan_start(ar, arvif->vdev_id); + mutex_lock(&ar->conf_mutex); spin_lock_bh(&ar->data_lock); @@ -3819,8 +3831,29 @@ static int ath11k_mac_op_hw_scan(struct ieee80211_hw *hw, goto exit; } - for (i = 0; i < arg->num_chan; i++) - arg->chan_list[i] = req->channels[i]->center_freq; + for (i = 0; i < arg->num_chan; i++) { + if (test_bit(WMI_TLV_SERVICE_SCAN_CONFIG_PER_CHANNEL, + ar->ab->wmi_ab.svc_map)) { + arg->chan_list[i] = + u32_encode_bits(req->channels[i]->center_freq, + WMI_SCAN_CONFIG_PER_CHANNEL_MASK); + + /* If NL80211_SCAN_FLAG_COLOCATED_6GHZ is set in scan + * flags, then scan all PSC channels in 6 GHz band and + * those non-PSC channels where RNR IE is found during + * the legacy 2.4/5 GHz scan. + * If NL80211_SCAN_FLAG_COLOCATED_6GHZ is not set, + * then all channels in 6 GHz will be scanned. + */ + if (req->channels[i]->band == NL80211_BAND_6GHZ && + req->flags & NL80211_SCAN_FLAG_COLOCATED_6GHZ && + !cfg80211_channel_is_psc(req->channels[i])) + arg->chan_list[i] |= + WMI_SCAN_CH_FLAG_SCAN_ONLY_IF_RNR_FOUND; + } else { + arg->chan_list[i] = req->channels[i]->center_freq; + } + } } if (req->flags & NL80211_SCAN_FLAG_RANDOM_ADDR) { @@ -5552,10 +5585,6 @@ static int ath11k_mac_copy_he_cap(struct ath11k *ar, he_cap_elem->mac_cap_info[1] &= IEEE80211_HE_MAC_CAP1_TF_MAC_PAD_DUR_MASK; - he_cap_elem->phy_cap_info[0] &= - ~IEEE80211_HE_PHY_CAP0_CHANNEL_WIDTH_SET_160MHZ_IN_5G; - he_cap_elem->phy_cap_info[0] &= - ~IEEE80211_HE_PHY_CAP0_CHANNEL_WIDTH_SET_80PLUS80_MHZ_IN_5G; he_cap_elem->phy_cap_info[5] &= ~IEEE80211_HE_PHY_CAP5_BEAMFORMEE_NUM_SND_DIM_UNDER_80MHZ_MASK; @@ -6652,6 +6681,11 @@ static void ath11k_mac_op_remove_interface(struct ieee80211_hw *hw, ath11k_dbg(ab, ATH11K_DBG_MAC, "mac remove interface (vdev %d)\n", arvif->vdev_id); + ret = ath11k_spectral_vif_stop(arvif); + if (ret) + ath11k_warn(ab, "failed to stop spectral for vdev %i: %d\n", + arvif->vdev_id, ret); + if (arvif->vdev_type == WMI_VDEV_TYPE_STA) ath11k_mac_11d_scan_stop(ar); @@ -9213,7 +9247,7 @@ static int __ath11k_mac_register(struct ath11k *ar) wiphy_ext_feature_set(ar->hw->wiphy, NL80211_EXT_FEATURE_SET_SCAN_DWELL); - if (ab->hw_params.ftm_responder) + if (test_bit(WMI_TLV_SERVICE_RTT, ar->ab->wmi_ab.svc_map)) wiphy_ext_feature_set(ar->hw->wiphy, NL80211_EXT_FEATURE_ENABLE_FTM_RESPONDER); diff --git a/drivers/net/wireless/ath/ath11k/pci.c b/drivers/net/wireless/ath/ath11k/pci.c index 0aeef2948ff5..7b33731a50ee 100644 --- a/drivers/net/wireless/ath/ath11k/pci.c +++ b/drivers/net/wireless/ath/ath11k/pci.c @@ -540,7 +540,7 @@ static int ath11k_pci_claim(struct ath11k_pci *ab_pci, struct pci_dev *pdev) if (!ab->mem) { ath11k_err(ab, "failed to map pci bar %d\n", ATH11K_PCI_BAR_NUM); ret = -EIO; - goto clear_master; + goto release_region; } ab->mem_ce = ab->mem; @@ -548,8 +548,6 @@ static int ath11k_pci_claim(struct ath11k_pci *ab_pci, struct pci_dev *pdev) ath11k_dbg(ab, ATH11K_DBG_BOOT, "boot pci_mem 0x%pK\n", ab->mem); return 0; -clear_master: - pci_clear_master(pdev); release_region: pci_release_region(pdev, ATH11K_PCI_BAR_NUM); disable_device: @@ -565,7 +563,6 @@ static void ath11k_pci_free_region(struct ath11k_pci *ab_pci) pci_iounmap(pci_dev, ab->mem); ab->mem = NULL; - pci_clear_master(pci_dev); pci_release_region(pci_dev, ATH11K_PCI_BAR_NUM); if (pci_is_enabled(pci_dev)) pci_disable_device(pci_dev); @@ -1039,7 +1036,8 @@ module_exit(ath11k_pci_exit); MODULE_DESCRIPTION("Driver support for Qualcomm Technologies 802.11ax WLAN PCIe devices"); MODULE_LICENSE("Dual BSD/GPL"); -/* QCA639x 2.0 firmware files */ -MODULE_FIRMWARE(ATH11K_FW_DIR "/QCA6390/hw2.0/" ATH11K_BOARD_API2_FILE); -MODULE_FIRMWARE(ATH11K_FW_DIR "/QCA6390/hw2.0/" ATH11K_AMSS_FILE); -MODULE_FIRMWARE(ATH11K_FW_DIR "/QCA6390/hw2.0/" ATH11K_M3_FILE); +/* firmware files */ +MODULE_FIRMWARE(ATH11K_FW_DIR "/QCA6390/hw2.0/*"); +MODULE_FIRMWARE(ATH11K_FW_DIR "/QCN9074/hw1.0/*"); +MODULE_FIRMWARE(ATH11K_FW_DIR "/WCN6855/hw2.0/*"); +MODULE_FIRMWARE(ATH11K_FW_DIR "/WCN6855/hw2.1/*"); diff --git a/drivers/net/wireless/ath/ath11k/peer.h b/drivers/net/wireless/ath/ath11k/peer.h index 6dd17bafe3a0..9bd385d0a38c 100644 --- a/drivers/net/wireless/ath/ath11k/peer.h +++ b/drivers/net/wireless/ath/ath11k/peer.h @@ -35,6 +35,7 @@ struct ath11k_peer { u16 sec_type; u16 sec_type_grp; bool is_authorized; + bool dp_setup_done; }; void ath11k_peer_unmap_event(struct ath11k_base *ab, u16 peer_id); diff --git a/drivers/net/wireless/ath/ath11k/wmi.c b/drivers/net/wireless/ath/ath11k/wmi.c index 27f3fceb33c5..d0b59bc2905a 100644 --- a/drivers/net/wireless/ath/ath11k/wmi.c +++ b/drivers/net/wireless/ath/ath11k/wmi.c @@ -82,6 +82,12 @@ struct wmi_tlv_fw_stats_parse { bool chain_rssi_done; }; +struct wmi_tlv_mgmt_rx_parse { + const struct wmi_mgmt_rx_hdr *fixed; + const u8 *frame_buf; + bool frame_buf_done; +}; + static const struct wmi_tlv_policy wmi_tlv_policies[] = { [WMI_TAG_ARRAY_BYTE] = { .min_len = 0 }, @@ -865,7 +871,8 @@ static void ath11k_wmi_put_wmi_channel(struct wmi_channel *chan, chan->band_center_freq2 = arg->channel.band_center_freq1; - } else if (arg->channel.mode == MODE_11AC_VHT80_80) { + } else if ((arg->channel.mode == MODE_11AC_VHT80_80) || + (arg->channel.mode == MODE_11AX_HE80_80)) { chan->band_center_freq2 = arg->channel.band_center_freq2; } else { chan->band_center_freq2 = 0; @@ -5633,28 +5640,49 @@ static int ath11k_pull_vdev_stopped_param_tlv(struct ath11k_base *ab, struct sk_ return 0; } +static int ath11k_wmi_tlv_mgmt_rx_parse(struct ath11k_base *ab, + u16 tag, u16 len, + const void *ptr, void *data) +{ + struct wmi_tlv_mgmt_rx_parse *parse = data; + + switch (tag) { + case WMI_TAG_MGMT_RX_HDR: + parse->fixed = ptr; + break; + case WMI_TAG_ARRAY_BYTE: + if (!parse->frame_buf_done) { + parse->frame_buf = ptr; + parse->frame_buf_done = true; + } + break; + } + return 0; +} + static int ath11k_pull_mgmt_rx_params_tlv(struct ath11k_base *ab, struct sk_buff *skb, struct mgmt_rx_event_params *hdr) { - const void **tb; + struct wmi_tlv_mgmt_rx_parse parse = { }; const struct wmi_mgmt_rx_hdr *ev; const u8 *frame; int ret; - tb = ath11k_wmi_tlv_parse_alloc(ab, skb->data, skb->len, GFP_ATOMIC); - if (IS_ERR(tb)) { - ret = PTR_ERR(tb); - ath11k_warn(ab, "failed to parse tlv: %d\n", ret); + ret = ath11k_wmi_tlv_iter(ab, skb->data, skb->len, + ath11k_wmi_tlv_mgmt_rx_parse, + &parse); + if (ret) { + ath11k_warn(ab, "failed to parse mgmt rx tlv %d\n", + ret); return ret; } - ev = tb[WMI_TAG_MGMT_RX_HDR]; - frame = tb[WMI_TAG_ARRAY_BYTE]; + ev = parse.fixed; + frame = parse.frame_buf; if (!ev || !frame) { ath11k_warn(ab, "failed to fetch mgmt rx hdr"); - kfree(tb); return -EPROTO; } @@ -5673,7 +5701,6 @@ static int ath11k_pull_mgmt_rx_params_tlv(struct ath11k_base *ab, if (skb->len < (frame - skb->data) + hdr->buf_len) { ath11k_warn(ab, "invalid length in mgmt rx hdr ev"); - kfree(tb); return -EPROTO; } @@ -5685,7 +5712,6 @@ static int ath11k_pull_mgmt_rx_params_tlv(struct ath11k_base *ab, ath11k_ce_byte_swap(skb->data, hdr->buf_len); - kfree(tb); return 0; } diff --git a/drivers/net/wireless/ath/ath11k/wmi.h b/drivers/net/wireless/ath/ath11k/wmi.h index b23b7a22bc9a..92fddb77669c 100644 --- a/drivers/net/wireless/ath/ath11k/wmi.h +++ b/drivers/net/wireless/ath/ath11k/wmi.h @@ -2100,8 +2100,10 @@ enum wmi_tlv_service { /* The second 128 bits */ WMI_MAX_EXT_SERVICE = 256, + WMI_TLV_SERVICE_SCAN_CONFIG_PER_CHANNEL = 265, WMI_TLV_SERVICE_REG_CC_EXT_EVENT_SUPPORT = 281, WMI_TLV_SERVICE_BIOS_SAR_SUPPORT = 326, + WMI_TLV_SERVICE_SUPPORT_11D_FOR_HOST_SCAN = 357, /* The third 128 bits */ WMI_MAX_EXT2_SERVICE = 384 @@ -3249,6 +3251,9 @@ struct wmi_start_scan_cmd { #define WMI_SCAN_DWELL_MODE_SHIFT 21 #define WMI_SCAN_FLAG_EXT_PASSIVE_SCAN_START_TIME_ENHANCE 0x00000800 +#define WMI_SCAN_CONFIG_PER_CHANNEL_MASK GENMASK(19, 0) +#define WMI_SCAN_CH_FLAG_SCAN_ONLY_IF_RNR_FOUND BIT(20) + enum { WMI_SCAN_DWELL_MODE_DEFAULT = 0, WMI_SCAN_DWELL_MODE_CONSERVATIVE = 1, diff --git a/drivers/net/wireless/ath/ath12k/core.h b/drivers/net/wireless/ath/ath12k/core.h index dffa687ee40e..9439052a652e 100644 --- a/drivers/net/wireless/ath/ath12k/core.h +++ b/drivers/net/wireless/ath/ath12k/core.h @@ -395,6 +395,7 @@ struct ath12k_sta { u8 rssi_comb; struct ath12k_rx_peer_stats *rx_stats; struct ath12k_wbm_tx_stats *wbm_tx_stats; + u32 bw_prev; }; #define ATH12K_MIN_5G_FREQ 4150 diff --git a/drivers/net/wireless/ath/ath12k/dp_rx.c b/drivers/net/wireless/ath/ath12k/dp_rx.c index 0adcbcfa0db5..e78478a5b978 100644 --- a/drivers/net/wireless/ath/ath12k/dp_rx.c +++ b/drivers/net/wireless/ath/ath12k/dp_rx.c @@ -196,7 +196,8 @@ static void ath12k_dp_rxdesc_set_msdu_len(struct ath12k_base *ab, static bool ath12k_dp_rx_h_is_mcbc(struct ath12k_base *ab, struct hal_rx_desc *desc) { - return ab->hw_params->hal_ops->rx_desc_is_mcbc(desc); + return (ath12k_dp_rx_h_first_msdu(ab, desc) && + ab->hw_params->hal_ops->rx_desc_is_mcbc(desc)); } static bool ath12k_dp_rxdesc_mac_addr2_valid(struct ath12k_base *ab, @@ -3047,10 +3048,14 @@ static int ath12k_dp_rx_h_defrag_reo_reinject(struct ath12k *ar, reo_ent_ring->rx_mpdu_info.peer_meta_data = reo_dest_ring->rx_mpdu_info.peer_meta_data; - reo_ent_ring->queue_addr_lo = cpu_to_le32(lower_32_bits(rx_tid->paddr)); - reo_ent_ring->info0 = le32_encode_bits(upper_32_bits(rx_tid->paddr), - HAL_REO_ENTR_RING_INFO0_QUEUE_ADDR_HI) | - le32_encode_bits(dst_ind, HAL_REO_ENTR_RING_INFO0_DEST_IND); + /* Firmware expects physical address to be filled in queue_addr_lo in + * the MLO scenario and in case of non MLO peer meta data needs to be + * filled. + * TODO: Need to handle for MLO scenario. + */ + reo_ent_ring->queue_addr_lo = reo_dest_ring->rx_mpdu_info.peer_meta_data; + reo_ent_ring->info0 = le32_encode_bits(dst_ind, + HAL_REO_ENTR_RING_INFO0_DEST_IND); reo_ent_ring->info1 = le32_encode_bits(rx_tid->cur_sn, HAL_REO_ENTR_RING_INFO1_MPDU_SEQ_NUM); diff --git a/drivers/net/wireless/ath/ath12k/dp_tx.c b/drivers/net/wireless/ath/ath12k/dp_tx.c index fd8d850f9818..d3c7c76d6b75 100644 --- a/drivers/net/wireless/ath/ath12k/dp_tx.c +++ b/drivers/net/wireless/ath/ath12k/dp_tx.c @@ -13,6 +13,10 @@ static enum hal_tcl_encap_type ath12k_dp_tx_get_encap_type(struct ath12k_vif *arvif, struct sk_buff *skb) { struct ieee80211_tx_info *tx_info = IEEE80211_SKB_CB(skb); + struct ath12k_base *ab = arvif->ar->ab; + + if (test_bit(ATH12K_FLAG_RAW_MODE, &ab->dev_flags)) + return HAL_TCL_ENCAP_TYPE_RAW; if (tx_info->flags & IEEE80211_TX_CTL_HW_80211_ENCAP) return HAL_TCL_ENCAP_TYPE_ETHERNET; diff --git a/drivers/net/wireless/ath/ath12k/hw.c b/drivers/net/wireless/ath/ath12k/hw.c index 91d576fd4b0f..1ffac7e3deaa 100644 --- a/drivers/net/wireless/ath/ath12k/hw.c +++ b/drivers/net/wireless/ath/ath12k/hw.c @@ -944,7 +944,7 @@ static const struct ath12k_hw_params ath12k_hw_params[] = { .interface_modes = BIT(NL80211_IFTYPE_STATION), .supports_monitor = false, - .idle_ps = false, + .idle_ps = true, .download_calib = false, .supports_suspend = false, .tcl_ring_retry = false, diff --git a/drivers/net/wireless/ath/ath12k/mac.c b/drivers/net/wireless/ath/ath12k/mac.c index bf7e5b6977b2..ee792822b411 100644 --- a/drivers/net/wireless/ath/ath12k/mac.c +++ b/drivers/net/wireless/ath/ath12k/mac.c @@ -3220,10 +3220,11 @@ static void ath12k_sta_rc_update_wk(struct work_struct *wk) enum nl80211_band band; const u8 *ht_mcs_mask; const u16 *vht_mcs_mask; - u32 changed, bw, nss, smps; + u32 changed, bw, nss, smps, bw_prev; int err, num_vht_rates; const struct cfg80211_bitrate_mask *mask; struct ath12k_wmi_peer_assoc_arg peer_arg; + enum wmi_phy_mode peer_phymode; arsta = container_of(wk, struct ath12k_sta, update_wk); sta = container_of((void *)arsta, struct ieee80211_sta, drv_priv); @@ -3243,6 +3244,7 @@ static void ath12k_sta_rc_update_wk(struct work_struct *wk) arsta->changed = 0; bw = arsta->bw; + bw_prev = arsta->bw_prev; nss = arsta->nss; smps = arsta->smps; @@ -3255,11 +3257,53 @@ static void ath12k_sta_rc_update_wk(struct work_struct *wk) ath12k_mac_max_vht_nss(vht_mcs_mask))); if (changed & IEEE80211_RC_BW_CHANGED) { - err = ath12k_wmi_set_peer_param(ar, sta->addr, arvif->vdev_id, - WMI_PEER_CHWIDTH, bw); - if (err) - ath12k_warn(ar->ab, "failed to update STA %pM peer bw %d: %d\n", - sta->addr, bw, err); + ath12k_peer_assoc_h_phymode(ar, arvif->vif, sta, &peer_arg); + peer_phymode = peer_arg.peer_phymode; + + if (bw > bw_prev) { + /* Phymode shows maximum supported channel width, if we + * upgrade bandwidth then due to sanity check of firmware, + * we have to send WMI_PEER_PHYMODE followed by + * WMI_PEER_CHWIDTH + */ + ath12k_dbg(ar->ab, ATH12K_DBG_MAC, "mac bandwidth upgrade for sta %pM new %d old %d\n", + sta->addr, bw, bw_prev); + err = ath12k_wmi_set_peer_param(ar, sta->addr, + arvif->vdev_id, WMI_PEER_PHYMODE, + peer_phymode); + if (err) { + ath12k_warn(ar->ab, "failed to update STA %pM to peer phymode %d: %d\n", + sta->addr, peer_phymode, err); + goto err_rc_bw_changed; + } + err = ath12k_wmi_set_peer_param(ar, sta->addr, + arvif->vdev_id, WMI_PEER_CHWIDTH, + bw); + if (err) + ath12k_warn(ar->ab, "failed to update STA %pM to peer bandwidth %d: %d\n", + sta->addr, bw, err); + } else { + /* When we downgrade bandwidth this will conflict with phymode + * and cause to trigger firmware crash. In this case we send + * WMI_PEER_CHWIDTH followed by WMI_PEER_PHYMODE + */ + ath12k_dbg(ar->ab, ATH12K_DBG_MAC, "mac bandwidth downgrade for sta %pM new %d old %d\n", + sta->addr, bw, bw_prev); + err = ath12k_wmi_set_peer_param(ar, sta->addr, + arvif->vdev_id, WMI_PEER_CHWIDTH, + bw); + if (err) { + ath12k_warn(ar->ab, "failed to update STA %pM peer to bandwidth %d: %d\n", + sta->addr, bw, err); + goto err_rc_bw_changed; + } + err = ath12k_wmi_set_peer_param(ar, sta->addr, + arvif->vdev_id, WMI_PEER_PHYMODE, + peer_phymode); + if (err) + ath12k_warn(ar->ab, "failed to update STA %pM to peer phymode %d: %d\n", + sta->addr, peer_phymode, err); + } } if (changed & IEEE80211_RC_NSS_CHANGED) { @@ -3321,7 +3365,7 @@ static void ath12k_sta_rc_update_wk(struct work_struct *wk) sta->addr, arvif->vdev_id); } } - +err_rc_bw_changed: mutex_unlock(&ar->conf_mutex); } @@ -3433,6 +3477,34 @@ exit: return ret; } +static u32 ath12k_mac_ieee80211_sta_bw_to_wmi(struct ath12k *ar, + struct ieee80211_sta *sta) +{ + u32 bw = WMI_PEER_CHWIDTH_20MHZ; + + switch (sta->deflink.bandwidth) { + case IEEE80211_STA_RX_BW_20: + bw = WMI_PEER_CHWIDTH_20MHZ; + break; + case IEEE80211_STA_RX_BW_40: + bw = WMI_PEER_CHWIDTH_40MHZ; + break; + case IEEE80211_STA_RX_BW_80: + bw = WMI_PEER_CHWIDTH_80MHZ; + break; + case IEEE80211_STA_RX_BW_160: + bw = WMI_PEER_CHWIDTH_160MHZ; + break; + default: + ath12k_warn(ar->ab, "Invalid bandwidth %d in rc update for %pM\n", + sta->deflink.bandwidth, sta->addr); + bw = WMI_PEER_CHWIDTH_20MHZ; + break; + } + + return bw; +} + static int ath12k_mac_op_sta_state(struct ieee80211_hw *hw, struct ieee80211_vif *vif, struct ieee80211_sta *sta, @@ -3498,6 +3570,13 @@ static int ath12k_mac_op_sta_state(struct ieee80211_hw *hw, if (ret) ath12k_warn(ar->ab, "Failed to associate station: %pM\n", sta->addr); + + spin_lock_bh(&ar->data_lock); + + arsta->bw = ath12k_mac_ieee80211_sta_bw_to_wmi(ar, sta); + arsta->bw_prev = sta->deflink.bandwidth; + + spin_unlock_bh(&ar->data_lock); } else if (old_state == IEEE80211_STA_ASSOC && new_state == IEEE80211_STA_AUTHORIZED) { spin_lock_bh(&ar->ab->base_lock); @@ -3607,28 +3686,8 @@ static void ath12k_mac_op_sta_rc_update(struct ieee80211_hw *hw, spin_lock_bh(&ar->data_lock); if (changed & IEEE80211_RC_BW_CHANGED) { - bw = WMI_PEER_CHWIDTH_20MHZ; - - switch (sta->deflink.bandwidth) { - case IEEE80211_STA_RX_BW_20: - bw = WMI_PEER_CHWIDTH_20MHZ; - break; - case IEEE80211_STA_RX_BW_40: - bw = WMI_PEER_CHWIDTH_40MHZ; - break; - case IEEE80211_STA_RX_BW_80: - bw = WMI_PEER_CHWIDTH_80MHZ; - break; - case IEEE80211_STA_RX_BW_160: - bw = WMI_PEER_CHWIDTH_160MHZ; - break; - default: - ath12k_warn(ar->ab, "Invalid bandwidth %d in rc update for %pM\n", - sta->deflink.bandwidth, sta->addr); - bw = WMI_PEER_CHWIDTH_20MHZ; - break; - } - + bw = ath12k_mac_ieee80211_sta_bw_to_wmi(ar, sta); + arsta->bw_prev = arsta->bw; arsta->bw = bw; } diff --git a/drivers/net/wireless/ath/ath12k/pci.c b/drivers/net/wireless/ath/ath12k/pci.c index d32637b0113d..9f174daf324c 100644 --- a/drivers/net/wireless/ath/ath12k/pci.c +++ b/drivers/net/wireless/ath/ath12k/pci.c @@ -755,14 +755,12 @@ static int ath12k_pci_claim(struct ath12k_pci *ab_pci, struct pci_dev *pdev) if (!ab->mem) { ath12k_err(ab, "failed to map pci bar %d\n", ATH12K_PCI_BAR_NUM); ret = -EIO; - goto clear_master; + goto release_region; } ath12k_dbg(ab, ATH12K_DBG_BOOT, "boot pci_mem 0x%pK\n", ab->mem); return 0; -clear_master: - pci_clear_master(pdev); release_region: pci_release_region(pdev, ATH12K_PCI_BAR_NUM); disable_device: @@ -778,7 +776,6 @@ static void ath12k_pci_free_region(struct ath12k_pci *ab_pci) pci_iounmap(pci_dev, ab->mem); ab->mem = NULL; - pci_clear_master(pci_dev); pci_release_region(pci_dev, ATH12K_PCI_BAR_NUM); if (pci_is_enabled(pci_dev)) pci_disable_device(pci_dev); @@ -1223,7 +1220,8 @@ static int ath12k_pci_probe(struct pci_dev *pdev, dev_err(&pdev->dev, "Unknown hardware version found for QCN9274: 0x%x\n", soc_hw_version_major); - return -EOPNOTSUPP; + ret = -EOPNOTSUPP; + goto err_pci_free_region; } break; case WCN7850_DEVICE_ID: diff --git a/drivers/net/wireless/ath/ath12k/qmi.c b/drivers/net/wireless/ath/ath12k/qmi.c index 979a63f2e2ab..03ba245fbee9 100644 --- a/drivers/net/wireless/ath/ath12k/qmi.c +++ b/drivers/net/wireless/ath/ath12k/qmi.c @@ -2991,7 +2991,7 @@ static void ath12k_qmi_driver_event_work(struct work_struct *work) spin_unlock(&qmi->event_lock); if (test_bit(ATH12K_FLAG_UNREGISTERING, &ab->dev_flags)) - return; + goto skip; switch (event->type) { case ATH12K_QMI_EVENT_SERVER_ARRIVE: @@ -3032,6 +3032,8 @@ static void ath12k_qmi_driver_event_work(struct work_struct *work) ath12k_warn(ab, "invalid event type: %d", event->type); break; } + +skip: kfree(event); spin_lock(&qmi->event_lock); } diff --git a/drivers/net/wireless/ath/ath12k/wmi.c b/drivers/net/wireless/ath/ath12k/wmi.c index 3e6991120e53..7ae0bb78b2b5 100644 --- a/drivers/net/wireless/ath/ath12k/wmi.c +++ b/drivers/net/wireless/ath/ath12k/wmi.c @@ -2438,6 +2438,9 @@ int ath12k_wmi_send_scan_chan_list_cmd(struct ath12k *ar, if (channel_arg->psc_channel) chan_info->info |= cpu_to_le32(WMI_CHAN_INFO_PSC); + if (channel_arg->dfs_set) + chan_info->info |= cpu_to_le32(WMI_CHAN_INFO_DFS); + chan_info->info |= le32_encode_bits(channel_arg->phy_mode, WMI_CHAN_INFO_MODE); *reg1 |= le32_encode_bits(channel_arg->minpower, @@ -4934,6 +4937,9 @@ static int freq_to_idx(struct ath12k *ar, int freq) int band, ch, idx = 0; for (band = NL80211_BAND_2GHZ; band < NUM_NL80211_BANDS; band++) { + if (!ar->mac.sbands[band].channels) + continue; + sband = ar->hw->wiphy->bands[band]; if (!sband) continue; diff --git a/drivers/net/wireless/ath/ath9k/hif_usb.c b/drivers/net/wireless/ath/ath9k/hif_usb.c index e0130beb304d..27ff1ca2631f 100644 --- a/drivers/net/wireless/ath/ath9k/hif_usb.c +++ b/drivers/net/wireless/ath/ath9k/hif_usb.c @@ -42,8 +42,6 @@ static const struct usb_device_id ath9k_hif_usb_ids[] = { { USB_DEVICE(0x0cf3, 0x7015), .driver_info = AR9287_USB }, /* Atheros */ - { USB_DEVICE(0x1668, 0x1200), - .driver_info = AR9287_USB }, /* Verizon */ { USB_DEVICE(0x0cf3, 0x7010), .driver_info = AR9280_USB }, /* Atheros */ diff --git a/drivers/net/wireless/ath/ath9k/mci.c b/drivers/net/wireless/ath/ath9k/mci.c index 3363fc4e8966..a0845002d6fe 100644 --- a/drivers/net/wireless/ath/ath9k/mci.c +++ b/drivers/net/wireless/ath/ath9k/mci.c @@ -646,9 +646,7 @@ void ath9k_mci_update_wlan_channels(struct ath_softc *sc, bool allow_all) struct ath_hw *ah = sc->sc_ah; struct ath9k_hw_mci *mci = &ah->btcoex_hw.mci; struct ath9k_channel *chan = ah->curchan; - static const u32 channelmap[] = { - 0x00000000, 0xffff0000, 0xffffffff, 0x7fffffff - }; + u32 channelmap[] = {0x00000000, 0xffff0000, 0xffffffff, 0x7fffffff}; int i; s16 chan_start, chan_end; u16 wlan_chan; diff --git a/drivers/net/wireless/ath/ath9k/xmit.c b/drivers/net/wireless/ath/ath9k/xmit.c index ef9a8e0b75e6..f6f2ab7a63ff 100644 --- a/drivers/net/wireless/ath/ath9k/xmit.c +++ b/drivers/net/wireless/ath/ath9k/xmit.c @@ -34,6 +34,12 @@ #define NUM_SYMBOLS_PER_USEC(_usec) (_usec >> 2) #define NUM_SYMBOLS_PER_USEC_HALFGI(_usec) (((_usec*5)-4)/18) +/* Shifts in ar5008_phy.c and ar9003_phy.c are equal for all revisions */ +#define ATH9K_PWRTBL_11NA_OFDM_SHIFT 0 +#define ATH9K_PWRTBL_11NG_OFDM_SHIFT 4 +#define ATH9K_PWRTBL_11NA_HT_SHIFT 8 +#define ATH9K_PWRTBL_11NG_HT_SHIFT 12 + static u16 bits_per_symbol[][2] = { /* 20MHz 40MHz */ @@ -1169,13 +1175,14 @@ void ath_update_max_aggr_framelen(struct ath_softc *sc, int queue, int txop) } static u8 ath_get_rate_txpower(struct ath_softc *sc, struct ath_buf *bf, - u8 rateidx, bool is_40, bool is_cck) + u8 rateidx, bool is_40, bool is_cck, bool is_mcs) { u8 max_power; struct sk_buff *skb; struct ath_frame_info *fi; struct ieee80211_tx_info *info; struct ath_hw *ah = sc->sc_ah; + bool is_2ghz, is_5ghz, use_stbc; if (sc->tx99_state || !ah->tpc_enabled) return MAX_RATE_POWER; @@ -1184,6 +1191,19 @@ static u8 ath_get_rate_txpower(struct ath_softc *sc, struct ath_buf *bf, fi = get_frame_info(skb); info = IEEE80211_SKB_CB(skb); + is_2ghz = info->band == NL80211_BAND_2GHZ; + is_5ghz = info->band == NL80211_BAND_5GHZ; + use_stbc = is_mcs && rateidx < 8 && (info->flags & + IEEE80211_TX_CTL_STBC); + + if (is_mcs) + rateidx += is_5ghz ? ATH9K_PWRTBL_11NA_HT_SHIFT + : ATH9K_PWRTBL_11NG_HT_SHIFT; + else if (is_2ghz && !is_cck) + rateidx += ATH9K_PWRTBL_11NG_OFDM_SHIFT; + else + rateidx += ATH9K_PWRTBL_11NA_OFDM_SHIFT; + if (!AR_SREV_9300_20_OR_LATER(ah)) { int txpower = fi->tx_power; @@ -1193,10 +1213,8 @@ static u8 ath_get_rate_txpower(struct ath_softc *sc, struct ath_buf *bf, u16 eeprom_rev = ah->eep_ops->get_eeprom_rev(ah); if (eeprom_rev >= AR5416_EEP_MINOR_VER_2) { - bool is_2ghz; struct modal_eep_header *pmodal; - is_2ghz = info->band == NL80211_BAND_2GHZ; pmodal = &eep->modalHeader[is_2ghz]; power_ht40delta = pmodal->ht40PowerIncForPdadc; } else { @@ -1229,7 +1247,7 @@ static u8 ath_get_rate_txpower(struct ath_softc *sc, struct ath_buf *bf, if (!max_power && !AR_SREV_9280_20_OR_LATER(ah)) max_power = 1; } else if (!bf->bf_state.bfs_paprd) { - if (rateidx < 8 && (info->flags & IEEE80211_TX_CTL_STBC)) + if (use_stbc) max_power = min_t(u8, ah->tx_power_stbc[rateidx], fi->tx_power); else @@ -1319,7 +1337,7 @@ static void ath_buf_set_rate(struct ath_softc *sc, struct ath_buf *bf, } info->txpower[i] = ath_get_rate_txpower(sc, bf, rix, - is_40, false); + is_40, false, true); continue; } @@ -1350,7 +1368,7 @@ static void ath_buf_set_rate(struct ath_softc *sc, struct ath_buf *bf, is_cck = IS_CCK_RATE(info->rates[i].Rate); info->txpower[i] = ath_get_rate_txpower(sc, bf, rix, false, - is_cck); + is_cck, false); } /* For AR5416 - RTS cannot be followed by a frame larger than 8K */ diff --git a/drivers/net/wireless/ath/carl9170/cmd.c b/drivers/net/wireless/ath/carl9170/cmd.c index f2b4f537e4c1..b8ed193c0195 100644 --- a/drivers/net/wireless/ath/carl9170/cmd.c +++ b/drivers/net/wireless/ath/carl9170/cmd.c @@ -120,7 +120,7 @@ struct carl9170_cmd *carl9170_cmd_buf(struct ar9170 *ar, { struct carl9170_cmd *tmp; - tmp = kzalloc(sizeof(struct carl9170_cmd_head) + len, GFP_ATOMIC); + tmp = kzalloc(sizeof(*tmp), GFP_ATOMIC); if (tmp) { tmp->hdr.cmd = cmd; tmp->hdr.len = len; diff --git a/drivers/net/wireless/ath/carl9170/fwcmd.h b/drivers/net/wireless/ath/carl9170/fwcmd.h index ff4b3b50250c..e5bcc364f088 100644 --- a/drivers/net/wireless/ath/carl9170/fwcmd.h +++ b/drivers/net/wireless/ath/carl9170/fwcmd.h @@ -320,9 +320,9 @@ struct carl9170_rsp { struct carl9170_u32_list rreg_res; struct carl9170_u32_list echo; #ifdef __CARL9170FW__ - struct carl9170_tx_status tx_status[0]; + DECLARE_FLEX_ARRAY(struct carl9170_tx_status, tx_status); #endif /* __CARL9170FW__ */ - struct _carl9170_tx_status _tx_status[0]; + DECLARE_FLEX_ARRAY(struct _carl9170_tx_status, _tx_status); struct carl9170_gpio gpio; struct carl9170_tsf_rsp tsf; struct carl9170_psm psm; diff --git a/drivers/net/wireless/ath/wcn36xx/dxe.c b/drivers/net/wireless/ath/wcn36xx/dxe.c index 4e9e13941c8f..9013f056eecb 100644 --- a/drivers/net/wireless/ath/wcn36xx/dxe.c +++ b/drivers/net/wireless/ath/wcn36xx/dxe.c @@ -112,8 +112,8 @@ int wcn36xx_dxe_alloc_ctl_blks(struct wcn36xx *wcn) wcn->dxe_rx_l_ch.desc_num = WCN36XX_DXE_CH_DESC_NUMB_RX_L; wcn->dxe_rx_h_ch.desc_num = WCN36XX_DXE_CH_DESC_NUMB_RX_H; - wcn->dxe_tx_l_ch.dxe_wq = WCN36XX_DXE_WQ_TX_L; - wcn->dxe_tx_h_ch.dxe_wq = WCN36XX_DXE_WQ_TX_H; + wcn->dxe_tx_l_ch.dxe_wq = WCN36XX_DXE_WQ_TX_L(wcn); + wcn->dxe_tx_h_ch.dxe_wq = WCN36XX_DXE_WQ_TX_H(wcn); wcn->dxe_tx_l_ch.ctrl_bd = WCN36XX_DXE_CTRL_TX_L_BD; wcn->dxe_tx_h_ch.ctrl_bd = WCN36XX_DXE_CTRL_TX_H_BD; @@ -165,8 +165,9 @@ void wcn36xx_dxe_free_ctl_blks(struct wcn36xx *wcn) wcn36xx_dxe_free_ctl_block(&wcn->dxe_rx_h_ch); } -static int wcn36xx_dxe_init_descs(struct device *dev, struct wcn36xx_dxe_ch *wcn_ch) +static int wcn36xx_dxe_init_descs(struct wcn36xx *wcn, struct wcn36xx_dxe_ch *wcn_ch) { + struct device *dev = wcn->dev; struct wcn36xx_dxe_desc *cur_dxe = NULL; struct wcn36xx_dxe_desc *prev_dxe = NULL; struct wcn36xx_dxe_ctl *cur_ctl = NULL; @@ -190,11 +191,11 @@ static int wcn36xx_dxe_init_descs(struct device *dev, struct wcn36xx_dxe_ch *wcn switch (wcn_ch->ch_type) { case WCN36XX_DXE_CH_TX_L: cur_dxe->ctrl = WCN36XX_DXE_CTRL_TX_L; - cur_dxe->dst_addr_l = WCN36XX_DXE_WQ_TX_L; + cur_dxe->dst_addr_l = WCN36XX_DXE_WQ_TX_L(wcn); break; case WCN36XX_DXE_CH_TX_H: cur_dxe->ctrl = WCN36XX_DXE_CTRL_TX_H; - cur_dxe->dst_addr_l = WCN36XX_DXE_WQ_TX_H; + cur_dxe->dst_addr_l = WCN36XX_DXE_WQ_TX_H(wcn); break; case WCN36XX_DXE_CH_RX_L: cur_dxe->ctrl = WCN36XX_DXE_CTRL_RX_L; @@ -914,7 +915,7 @@ int wcn36xx_dxe_init(struct wcn36xx *wcn) /***************************************/ /* Init descriptors for TX LOW channel */ /***************************************/ - ret = wcn36xx_dxe_init_descs(wcn->dev, &wcn->dxe_tx_l_ch); + ret = wcn36xx_dxe_init_descs(wcn, &wcn->dxe_tx_l_ch); if (ret) { dev_err(wcn->dev, "Error allocating descriptor\n"); return ret; @@ -928,14 +929,14 @@ int wcn36xx_dxe_init(struct wcn36xx *wcn) /* Program DMA destination addr for TX LOW */ wcn36xx_dxe_write_register(wcn, WCN36XX_DXE_CH_DEST_ADDR_TX_L, - WCN36XX_DXE_WQ_TX_L); + WCN36XX_DXE_WQ_TX_L(wcn)); wcn36xx_dxe_read_register(wcn, WCN36XX_DXE_REG_CH_EN, ®_data); /***************************************/ /* Init descriptors for TX HIGH channel */ /***************************************/ - ret = wcn36xx_dxe_init_descs(wcn->dev, &wcn->dxe_tx_h_ch); + ret = wcn36xx_dxe_init_descs(wcn, &wcn->dxe_tx_h_ch); if (ret) { dev_err(wcn->dev, "Error allocating descriptor\n"); goto out_err_txh_ch; @@ -950,14 +951,14 @@ int wcn36xx_dxe_init(struct wcn36xx *wcn) /* Program DMA destination addr for TX HIGH */ wcn36xx_dxe_write_register(wcn, WCN36XX_DXE_CH_DEST_ADDR_TX_H, - WCN36XX_DXE_WQ_TX_H); + WCN36XX_DXE_WQ_TX_H(wcn)); wcn36xx_dxe_read_register(wcn, WCN36XX_DXE_REG_CH_EN, ®_data); /***************************************/ /* Init descriptors for RX LOW channel */ /***************************************/ - ret = wcn36xx_dxe_init_descs(wcn->dev, &wcn->dxe_rx_l_ch); + ret = wcn36xx_dxe_init_descs(wcn, &wcn->dxe_rx_l_ch); if (ret) { dev_err(wcn->dev, "Error allocating descriptor\n"); goto out_err_rxl_ch; @@ -988,7 +989,7 @@ int wcn36xx_dxe_init(struct wcn36xx *wcn) /***************************************/ /* Init descriptors for RX HIGH channel */ /***************************************/ - ret = wcn36xx_dxe_init_descs(wcn->dev, &wcn->dxe_rx_h_ch); + ret = wcn36xx_dxe_init_descs(wcn, &wcn->dxe_rx_h_ch); if (ret) { dev_err(wcn->dev, "Error allocating descriptor\n"); goto out_err_rxh_ch; diff --git a/drivers/net/wireless/ath/wcn36xx/dxe.h b/drivers/net/wireless/ath/wcn36xx/dxe.h index 26a31edf52e9..dd8c684a3ba7 100644 --- a/drivers/net/wireless/ath/wcn36xx/dxe.h +++ b/drivers/net/wireless/ath/wcn36xx/dxe.h @@ -135,8 +135,8 @@ H2H_TEST_RX_TX = DMA2 WCN36xx_DXE_CTRL_ENDIANNESS) /* TODO This must calculated properly but not hardcoded */ -#define WCN36XX_DXE_WQ_TX_L 0x17 -#define WCN36XX_DXE_WQ_TX_H 0x17 +#define WCN36XX_DXE_WQ_TX_L(wcn) ((wcn)->is_pronto_v3 ? 0x6 : 0x17) +#define WCN36XX_DXE_WQ_TX_H(wcn) ((wcn)->is_pronto_v3 ? 0x6 : 0x17) #define WCN36XX_DXE_WQ_RX_L 0xB #define WCN36XX_DXE_WQ_RX_H 0x4 diff --git a/drivers/net/wireless/ath/wcn36xx/main.c b/drivers/net/wireless/ath/wcn36xx/main.c index 3b79cc1c7c5b..8dbd115a393c 100644 --- a/drivers/net/wireless/ath/wcn36xx/main.c +++ b/drivers/net/wireless/ath/wcn36xx/main.c @@ -1508,6 +1508,7 @@ static int wcn36xx_platform_get_resources(struct wcn36xx *wcn, } wcn->is_pronto = !!of_device_is_compatible(mmio_node, "qcom,pronto"); + wcn->is_pronto_v3 = !!of_device_is_compatible(mmio_node, "qcom,pronto-v3-pil"); /* Map the CCU memory */ index = of_property_match_string(mmio_node, "reg-names", "ccu"); diff --git a/drivers/net/wireless/ath/wcn36xx/wcn36xx.h b/drivers/net/wireless/ath/wcn36xx/wcn36xx.h index 9aa08b636d08..ff4a8e5d7209 100644 --- a/drivers/net/wireless/ath/wcn36xx/wcn36xx.h +++ b/drivers/net/wireless/ath/wcn36xx/wcn36xx.h @@ -217,6 +217,7 @@ struct wcn36xx { u8 fw_major; u32 fw_feat_caps[WCN36XX_HAL_CAPS_SIZE]; bool is_pronto; + bool is_pronto_v3; /* extra byte for the NULL termination */ u8 crm_version[WCN36XX_HAL_VERSION_LENGTH + 1]; diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcmsdh.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcmsdh.c index 65d4799a5658..ff710b0b5071 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcmsdh.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcmsdh.c @@ -965,6 +965,12 @@ out: .driver_data = BRCMF_FWVENDOR_ ## fw_vend \ } +#define CYW_SDIO_DEVICE(dev_id, fw_vend) \ + { \ + SDIO_DEVICE(SDIO_VENDOR_ID_CYPRESS, dev_id), \ + .driver_data = BRCMF_FWVENDOR_ ## fw_vend \ + } + /* devices we support, null terminated */ static const struct sdio_device_id brcmf_sdmmc_ids[] = { BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_43143, WCC), @@ -979,6 +985,7 @@ static const struct sdio_device_id brcmf_sdmmc_ids[] = { BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_4335_4339, WCC), BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_4339, WCC), BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_43430, WCC), + BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_43439, WCC), BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_4345, WCC), BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_43455, WCC), BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_4354, WCC), @@ -986,9 +993,9 @@ static const struct sdio_device_id brcmf_sdmmc_ids[] = { BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_4359, WCC), BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_CYPRESS_4373, CYW), BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_CYPRESS_43012, CYW), - BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_CYPRESS_43439, CYW), BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_CYPRESS_43752, CYW), BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_CYPRESS_89359, CYW), + CYW_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_CYPRESS_43439, CYW), { /* end: all zeroes */ } }; MODULE_DEVICE_TABLE(sdio, brcmf_sdmmc_ids); diff --git a/drivers/net/wireless/cisco/Kconfig b/drivers/net/wireless/cisco/Kconfig index 681bfc2d740a..b40ee25aca99 100644 --- a/drivers/net/wireless/cisco/Kconfig +++ b/drivers/net/wireless/cisco/Kconfig @@ -14,7 +14,7 @@ if WLAN_VENDOR_CISCO config AIRO tristate "Cisco/Aironet 34X/35X/4500/4800 ISA and PCI cards" - depends on CFG80211 && ISA_DMA_API && (PCI || BROKEN) + depends on CFG80211 && (PCI || BROKEN) select WIRELESS_EXT select CRYPTO select CRYPTO_SKCIPHER diff --git a/drivers/net/wireless/intel/iwlwifi/cfg/22000.c b/drivers/net/wireless/intel/iwlwifi/cfg/22000.c index 0b10b34a77a8..b6f82510e980 100644 --- a/drivers/net/wireless/intel/iwlwifi/cfg/22000.c +++ b/drivers/net/wireless/intel/iwlwifi/cfg/22000.c @@ -10,7 +10,7 @@ #include "fw/api/txq.h" /* Highest firmware API version supported */ -#define IWL_22000_UCODE_API_MAX 75 +#define IWL_22000_UCODE_API_MAX 78 /* Lowest firmware API version supported */ #define IWL_22000_UCODE_API_MIN 39 @@ -50,7 +50,13 @@ #define IWL_MA_A_GF4_A_FW_PRE "iwlwifi-ma-a0-gf4-a0-" #define IWL_MA_A_MR_A_FW_PRE "iwlwifi-ma-a0-mr-a0-" #define IWL_MA_A_FM_A_FW_PRE "iwlwifi-ma-a0-fm-a0-" +#define IWL_MA_B_HR_B_FW_PRE "iwlwifi-ma-b0-hr-b0-" +#define IWL_MA_B_GF_A_FW_PRE "iwlwifi-ma-b0-gf-a0-" +#define IWL_MA_B_GF4_A_FW_PRE "iwlwifi-ma-b0-gf4-a0-" +#define IWL_MA_B_MR_A_FW_PRE "iwlwifi-ma-b0-mr-a0-" +#define IWL_MA_B_FM_A_FW_PRE "iwlwifi-ma-b0-fm-a0-" #define IWL_SNJ_A_MR_A_FW_PRE "iwlwifi-SoSnj-a0-mr-a0-" +#define IWL_BZ_A_HR_A_FW_PRE "iwlwifi-bz-a0-hr-b0-" #define IWL_BZ_A_HR_B_FW_PRE "iwlwifi-bz-a0-hr-b0-" #define IWL_BZ_A_GF_A_FW_PRE "iwlwifi-bz-a0-gf-a0-" #define IWL_BZ_A_GF4_A_FW_PRE "iwlwifi-bz-a0-gf4-a0-" @@ -69,7 +75,9 @@ #define IWL_BNJ_B_GF_A_FW_PRE "iwlwifi-BzBnj-b0-gf-a0-" #define IWL_BNJ_A_GF4_A_FW_PRE "iwlwifi-BzBnj-a0-gf4-a0-" #define IWL_BNJ_B_GF4_A_FW_PRE "iwlwifi-BzBnj-b0-gf4-a0-" +#define IWL_BNJ_A_HR_A_FW_PRE "iwlwifi-BzBnj-a0-hr-b0-" #define IWL_BNJ_A_HR_B_FW_PRE "iwlwifi-BzBnj-a0-hr-b0-" +#define IWL_BNJ_B_HR_A_FW_PRE "iwlwifi-BzBnj-b0-hr-b0-" #define IWL_BNJ_B_HR_B_FW_PRE "iwlwifi-BzBnj-b0-hr-b0-" #define IWL_BNJ_B_FM_B_FW_PRE "iwlwifi-BzBnj-b0-fm-b0-" @@ -116,8 +124,20 @@ IWL_MA_A_MR_A_FW_PRE __stringify(api) ".ucode" #define IWL_MA_A_FM_A_FW_MODULE_FIRMWARE(api) \ IWL_MA_A_FM_A_FW_PRE __stringify(api) ".ucode" +#define IWL_MA_B_HR_B_FW_MODULE_FIRMWARE(api) \ + IWL_MA_B_HR_B_FW_PRE __stringify(api) ".ucode" +#define IWL_MA_B_GF_A_FW_MODULE_FIRMWARE(api) \ + IWL_MA_B_GF_A_FW_PRE __stringify(api) ".ucode" +#define IWL_MA_B_GF4_A_FW_MODULE_FIRMWARE(api) \ + IWL_MA_B_GF4_A_FW_PRE __stringify(api) ".ucode" +#define IWL_MA_B_MR_A_FW_MODULE_FIRMWARE(api) \ + IWL_MA_B_MR_A_FW_PRE __stringify(api) ".ucode" +#define IWL_MA_B_FM_A_FW_MODULE_FIRMWARE(api) \ + IWL_MA_B_FM_A_FW_PRE __stringify(api) ".ucode" #define IWL_SNJ_A_MR_A_MODULE_FIRMWARE(api) \ IWL_SNJ_A_MR_A_FW_PRE __stringify(api) ".ucode" +#define IWL_BZ_A_HR_A_MODULE_FIRMWARE(api) \ + IWL_BZ_A_HR_A_FW_PRE __stringify(api) ".ucode" #define IWL_BZ_A_HR_B_MODULE_FIRMWARE(api) \ IWL_BZ_A_HR_B_FW_PRE __stringify(api) ".ucode" #define IWL_BZ_A_GF_A_MODULE_FIRMWARE(api) \ @@ -127,17 +147,17 @@ #define IWL_BZ_A_MR_A_MODULE_FIRMWARE(api) \ IWL_BZ_A_MR_A_FW_PRE __stringify(api) ".ucode" #define IWL_BZ_A_FM_A_MODULE_FIRMWARE(api) \ - IWL_BZ_A_FM_A_FW_PRE __stringify(api) ".ucode" + IWL_BZ_A_FM_A_FW_PRE __stringify(api) ".ucode" #define IWL_BZ_A_FM4_A_MODULE_FIRMWARE(api) \ - IWL_BZ_A_FM4_A_FW_PRE __stringify(api) ".ucode" + IWL_BZ_A_FM4_A_FW_PRE __stringify(api) ".ucode" #define IWL_BZ_A_FM_B_MODULE_FIRMWARE(api) \ - IWL_BZ_A_FM_B_FW_PRE __stringify(api) ".ucode" + IWL_BZ_A_FM_B_FW_PRE __stringify(api) ".ucode" #define IWL_BZ_A_FM4_B_MODULE_FIRMWARE(api) \ - IWL_BZ_A_FM4_B_FW_PRE __stringify(api) ".ucode" + IWL_BZ_A_FM4_B_FW_PRE __stringify(api) ".ucode" #define IWL_GL_A_FM_A_MODULE_FIRMWARE(api) \ - IWL_GL_A_FM_A_FW_PRE __stringify(api) ".ucode" + IWL_GL_A_FM_A_FW_PRE __stringify(api) ".ucode" #define IWL_GL_B_FM_B_MODULE_FIRMWARE(api) \ - IWL_GL_B_FM_B_FW_PRE __stringify(api) ".ucode" + IWL_GL_B_FM_B_FW_PRE __stringify(api) ".ucode" #define IWL_BNJ_A_FM_A_MODULE_FIRMWARE(api) \ IWL_BNJ_A_FM_A_FW_PRE __stringify(api) ".ucode" #define IWL_BNJ_A_FM4_A_MODULE_FIRMWARE(api) \ @@ -152,8 +172,12 @@ IWL_BNJ_A_GF4_A_FW_PRE __stringify(api) ".ucode" #define IWL_BNJ_B_GF4_A_MODULE_FIRMWARE(api) \ IWL_BNJ_B_GF4_A_FW_PRE __stringify(api) ".ucode" +#define IWL_BNJ_A_HR_A_MODULE_FIRMWARE(api) \ + IWL_BNJ_A_HR_A_FW_PRE __stringify(api) ".ucode" #define IWL_BNJ_A_HR_B_MODULE_FIRMWARE(api) \ IWL_BNJ_A_HR_B_FW_PRE __stringify(api) ".ucode" +#define IWL_BNJ_B_HR_A_MODULE_FIRMWARE(api) \ + IWL_BNJ_B_HR_A_FW_PRE __stringify(api) ".ucode" #define IWL_BNJ_B_HR_B_MODULE_FIRMWARE(api) \ IWL_BNJ_B_HR_B_FW_PRE __stringify(api) ".ucode" #define IWL_BNJ_B_FM_B_MODULE_FIRMWARE(api) \ @@ -882,6 +906,41 @@ const struct iwl_cfg iwl_cfg_ma_a0_ms_a0 = { .num_rbds = IWL_NUM_RBDS_AX210_HE, }; +const struct iwl_cfg iwl_cfg_ma_b0_fm_a0 = { + .fw_name_pre = IWL_MA_B_FM_A_FW_PRE, + .uhb_supported = true, + IWL_DEVICE_AX210, + .num_rbds = IWL_NUM_RBDS_AX210_HE, +}; + +const struct iwl_cfg iwl_cfg_ma_b0_hr_b0 = { + .fw_name_pre = IWL_MA_B_HR_B_FW_PRE, + .uhb_supported = true, + IWL_DEVICE_AX210, + .num_rbds = IWL_NUM_RBDS_AX210_HE, +}; + +const struct iwl_cfg iwl_cfg_ma_b0_gf_a0 = { + .fw_name_pre = IWL_MA_B_GF_A_FW_PRE, + .uhb_supported = true, + IWL_DEVICE_AX210, + .num_rbds = IWL_NUM_RBDS_AX210_HE, +}; + +const struct iwl_cfg iwl_cfg_ma_b0_gf4_a0 = { + .fw_name_pre = IWL_MA_B_GF4_A_FW_PRE, + .uhb_supported = true, + IWL_DEVICE_AX210, + .num_rbds = IWL_NUM_RBDS_AX210_HE, +}; + +const struct iwl_cfg iwl_cfg_ma_b0_mr_a0 = { + .fw_name_pre = IWL_MA_B_MR_A_FW_PRE, + .uhb_supported = true, + IWL_DEVICE_AX210, + .num_rbds = IWL_NUM_RBDS_AX210_HE, +}; + const struct iwl_cfg iwl_cfg_so_a0_ms_a0 = { .fw_name_pre = IWL_SO_A_MR_A_FW_PRE, .uhb_supported = false, @@ -928,6 +987,14 @@ const struct iwl_cfg iwl_cfg_quz_a0_hr_b0 = { .num_rbds = IWL_NUM_RBDS_22000_HE, }; +const struct iwl_cfg iwl_cfg_bz_a0_hr_a0 = { + .fw_name_pre = IWL_BZ_A_HR_A_FW_PRE, + .uhb_supported = true, + IWL_DEVICE_BZ, + .features = IWL_TX_CSUM_NETIF_FLAGS_BZ | NETIF_F_RXCSUM, + .num_rbds = IWL_NUM_RBDS_AX210_HE, +}; + const struct iwl_cfg iwl_cfg_bz_a0_hr_b0 = { .fw_name_pre = IWL_BZ_A_HR_B_FW_PRE, .uhb_supported = true, @@ -1072,6 +1139,14 @@ const struct iwl_cfg iwl_cfg_bnj_b0_gf4_a0 = { .num_rbds = IWL_NUM_RBDS_AX210_HE, }; +const struct iwl_cfg iwl_cfg_bnj_a0_hr_a0 = { + .fw_name_pre = IWL_BNJ_A_HR_A_FW_PRE, + .uhb_supported = true, + IWL_DEVICE_BZ, + .features = IWL_TX_CSUM_NETIF_FLAGS | NETIF_F_RXCSUM, + .num_rbds = IWL_NUM_RBDS_AX210_HE, +}; + const struct iwl_cfg iwl_cfg_bnj_a0_hr_b0 = { .fw_name_pre = IWL_BNJ_A_HR_B_FW_PRE, .uhb_supported = true, @@ -1080,6 +1155,14 @@ const struct iwl_cfg iwl_cfg_bnj_a0_hr_b0 = { .num_rbds = IWL_NUM_RBDS_AX210_HE, }; +const struct iwl_cfg iwl_cfg_bnj_b0_hr_a0 = { + .fw_name_pre = IWL_BNJ_B_HR_A_FW_PRE, + .uhb_supported = true, + IWL_DEVICE_BZ, + .features = IWL_TX_CSUM_NETIF_FLAGS | NETIF_F_RXCSUM, + .num_rbds = IWL_NUM_RBDS_AX210_HE, +}; + const struct iwl_cfg iwl_cfg_bnj_b0_hr_b0 = { .fw_name_pre = IWL_BNJ_B_HR_B_FW_PRE, .uhb_supported = true, @@ -1116,7 +1199,13 @@ MODULE_FIRMWARE(IWL_MA_A_GF_A_FW_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_MA_A_GF4_A_FW_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_MA_A_MR_A_FW_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_MA_A_FM_A_FW_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); +MODULE_FIRMWARE(IWL_MA_B_HR_B_FW_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); +MODULE_FIRMWARE(IWL_MA_B_GF_A_FW_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); +MODULE_FIRMWARE(IWL_MA_B_GF4_A_FW_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); +MODULE_FIRMWARE(IWL_MA_B_MR_A_FW_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); +MODULE_FIRMWARE(IWL_MA_B_FM_A_FW_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_SNJ_A_MR_A_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); +MODULE_FIRMWARE(IWL_BZ_A_HR_A_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_BZ_A_HR_B_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_BZ_A_GF_A_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_BZ_A_GF4_A_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); @@ -1132,6 +1221,7 @@ MODULE_FIRMWARE(IWL_BNJ_B_GF_A_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_BNJ_A_GF4_A_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_BNJ_B_GF4_A_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_BNJ_A_HR_B_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); +MODULE_FIRMWARE(IWL_BNJ_B_HR_A_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_BNJ_B_HR_B_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_BZ_A_FM4_A_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_BZ_A_FM4_B_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); diff --git a/drivers/net/wireless/intel/iwlwifi/dvm/sta.c b/drivers/net/wireless/intel/iwlwifi/dvm/sta.c index cef43cf80620..8b01ab986cb1 100644 --- a/drivers/net/wireless/intel/iwlwifi/dvm/sta.c +++ b/drivers/net/wireless/intel/iwlwifi/dvm/sta.c @@ -1081,6 +1081,7 @@ static int iwlagn_send_sta_key(struct iwl_priv *priv, { __le16 key_flags; struct iwl_addsta_cmd sta_cmd; + size_t to_copy; int i; spin_lock_bh(&priv->sta_lock); @@ -1100,7 +1101,9 @@ static int iwlagn_send_sta_key(struct iwl_priv *priv, sta_cmd.key.tkip_rx_tsc_byte2 = tkip_iv32; for (i = 0; i < 5; i++) sta_cmd.key.tkip_rx_ttak[i] = cpu_to_le16(tkip_p1k[i]); - memcpy(sta_cmd.key.key, keyconf->key, keyconf->keylen); + /* keyconf may contain MIC rx/tx keys which iwl does not use */ + to_copy = min_t(size_t, sizeof(sta_cmd.key.key), keyconf->keylen); + memcpy(sta_cmd.key.key, keyconf->key, to_copy); break; case WLAN_CIPHER_SUITE_WEP104: key_flags |= STA_KEY_FLG_KEY_SIZE_MSK; diff --git a/drivers/net/wireless/intel/iwlwifi/fw/acpi.c b/drivers/net/wireless/intel/iwlwifi/fw/acpi.c index a02e5a67b706..5f4a51310add 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/acpi.c +++ b/drivers/net/wireless/intel/iwlwifi/fw/acpi.c @@ -1006,8 +1006,10 @@ int iwl_acpi_get_ppag_table(struct iwl_fw_runtime *fwrt) union acpi_object *wifi_pkg, *data, *flags; int i, j, ret, tbl_rev, num_sub_bands = 0; int idx = 2; + u8 cmd_ver; fwrt->ppag_flags = 0; + fwrt->ppag_table_valid = false; data = iwl_acpi_get_object(fwrt->dev, ACPI_PPAG_METHOD); if (IS_ERR(data)) @@ -1054,8 +1056,15 @@ read_table: } fwrt->ppag_flags = flags->integer.value & ACPI_PPAG_MASK; - - if (!fwrt->ppag_flags) { + cmd_ver = iwl_fw_lookup_cmd_ver(fwrt->fw, + WIDE_ID(PHY_OPS_GROUP, + PER_PLATFORM_ANT_GAIN_CMD), + IWL_FW_CMD_VER_UNKNOWN); + if (cmd_ver == IWL_FW_CMD_VER_UNKNOWN) { + ret = -EINVAL; + goto out_free; + } + if (!fwrt->ppag_flags && cmd_ver <= 3) { ret = 0; goto out_free; } @@ -1076,21 +1085,22 @@ read_table: } fwrt->ppag_chains[i].subbands[j] = ent->integer.value; - + /* from ver 4 the fw deals with out of range values */ + if (cmd_ver >= 4) + continue; if ((j == 0 && (fwrt->ppag_chains[i].subbands[j] > ACPI_PPAG_MAX_LB || fwrt->ppag_chains[i].subbands[j] < ACPI_PPAG_MIN_LB)) || (j != 0 && (fwrt->ppag_chains[i].subbands[j] > ACPI_PPAG_MAX_HB || fwrt->ppag_chains[i].subbands[j] < ACPI_PPAG_MIN_HB))) { - fwrt->ppag_flags = 0; ret = -EINVAL; goto out_free; } } } - + fwrt->ppag_table_valid = true; ret = 0; out_free: @@ -1115,19 +1125,22 @@ int iwl_read_ppag_table(struct iwl_fw_runtime *fwrt, union iwl_ppag_table_cmd *c IWL_DEBUG_RADIO(fwrt, "PPAG capability not supported by FW, command not sent.\n"); return -EINVAL; - } - if (!fwrt->ppag_flags) { - IWL_DEBUG_RADIO(fwrt, "PPAG not enabled, command not sent.\n"); - return -EINVAL; - } + } + + cmd_ver = iwl_fw_lookup_cmd_ver(fwrt->fw, + WIDE_ID(PHY_OPS_GROUP, + PER_PLATFORM_ANT_GAIN_CMD), + IWL_FW_CMD_VER_UNKNOWN); + if (!fwrt->ppag_table_valid || (cmd_ver <= 3 && !fwrt->ppag_flags)) { + IWL_DEBUG_RADIO(fwrt, "PPAG not enabled, command not sent.\n"); + return -EINVAL; + } /* The 'flags' field is the same in v1 and in v2 so we can just * use v1 to access it. */ cmd->v1.flags = cpu_to_le32(fwrt->ppag_flags); - cmd_ver = iwl_fw_lookup_cmd_ver(fwrt->fw, - WIDE_ID(PHY_OPS_GROUP, PER_PLATFORM_ANT_GAIN_CMD), - IWL_FW_CMD_VER_UNKNOWN); + if (cmd_ver == 1) { num_sub_bands = IWL_NUM_SUB_BANDS_V1; gain = cmd->v1.gain[0]; @@ -1138,7 +1151,7 @@ int iwl_read_ppag_table(struct iwl_fw_runtime *fwrt, union iwl_ppag_table_cmd *c fwrt->ppag_ver); cmd->v1.flags &= cpu_to_le32(IWL_PPAG_ETSI_MASK); } - } else if (cmd_ver == 2 || cmd_ver == 3) { + } else if (cmd_ver >= 2 && cmd_ver <= 4) { num_sub_bands = IWL_NUM_SUB_BANDS_V2; gain = cmd->v2.gain[0]; *cmd_size = sizeof(cmd->v2); diff --git a/drivers/net/wireless/intel/iwlwifi/fw/api/d3.h b/drivers/net/wireless/intel/iwlwifi/fw/api/d3.h index df0833890e55..8a613e150a02 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/api/d3.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/api/d3.h @@ -767,7 +767,7 @@ struct iwl_wowlan_status_v12 { } __packed; /* WOWLAN_STATUSES_RSP_API_S_VER_12 */ /** - * struct iwl_wowlan_info_notif - WoWLAN information notification + * struct iwl_wowlan_info_notif_v1 - WoWLAN information notification * @gtk: GTK data * @igtk: IGTK data * @replay_ctr: GTK rekey replay counter @@ -785,7 +785,7 @@ struct iwl_wowlan_status_v12 { * @station_id: station id * @reserved2: reserved */ -struct iwl_wowlan_info_notif { +struct iwl_wowlan_info_notif_v1 { struct iwl_wowlan_gtk_status_v3 gtk[WOWLAN_GTK_KEYS_NUM]; struct iwl_wowlan_igtk_status igtk[WOWLAN_IGTK_KEYS_NUM]; __le64 replay_ctr; @@ -804,6 +804,39 @@ struct iwl_wowlan_info_notif { } __packed; /* WOWLAN_INFO_NTFY_API_S_VER_1 */ /** + * struct iwl_wowlan_info_notif - WoWLAN information notification + * @gtk: GTK data + * @igtk: IGTK data + * @replay_ctr: GTK rekey replay counter + * @pattern_number: number of the matched patterns + * @reserved1: reserved + * @qos_seq_ctr: QoS sequence counters to use next + * @wakeup_reasons: wakeup reasons, see &enum iwl_wowlan_wakeup_reason + * @num_of_gtk_rekeys: number of GTK rekeys + * @transmitted_ndps: number of transmitted neighbor discovery packets + * @received_beacons: number of received beacons + * @tid_tear_down: bit mask of tids whose BA sessions were closed + * in suspend state + * @station_id: station id + * @reserved2: reserved + */ +struct iwl_wowlan_info_notif { + struct iwl_wowlan_gtk_status_v3 gtk[WOWLAN_GTK_KEYS_NUM]; + struct iwl_wowlan_igtk_status igtk[WOWLAN_IGTK_KEYS_NUM]; + __le64 replay_ctr; + __le16 pattern_number; + __le16 reserved1; + __le16 qos_seq_ctr[8]; + __le32 wakeup_reasons; + __le32 num_of_gtk_rekeys; + __le32 transmitted_ndps; + __le32 received_beacons; + u8 tid_tear_down; + u8 station_id; + u8 reserved2[2]; +} __packed; /* WOWLAN_INFO_NTFY_API_S_VER_2 */ + +/** * struct iwl_wowlan_wake_pkt_notif - WoWLAN wake packet notification * @wake_packet_length: wakeup packet length * @station_id: station id diff --git a/drivers/net/wireless/intel/iwlwifi/fw/api/mac-cfg.h b/drivers/net/wireless/intel/iwlwifi/fw/api/mac-cfg.h index e236d1b0aae2..74f2efbad34e 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/api/mac-cfg.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/api/mac-cfg.h @@ -224,15 +224,6 @@ struct iwl_mac_client_data { } __packed; /* MAC_CONTEXT_CONFIG_CLIENT_DATA_API_S_VER_1 */ /** - * struct iwl_mac_go_ibss_data - configuration data for GO and IBSS MAC context - * - * @beacon_template: beacon template ID - */ -struct iwl_mac_go_ibss_data { - __le32 beacon_template; -} __packed; /* MAC_CONTEXT_CONFIG_GO_IBSS_DATA_API_S_VER_1 */ - -/** * struct iwl_mac_p2p_dev_data - configuration data for P2P device MAC context * * @is_disc_extended: if set to true, P2P Device discoverability is enabled on @@ -278,6 +269,7 @@ enum iwl_mac_config_filter_flags { * @reserved_for_local_mld_addr: reserved * @filter_flags: combination of &enum iwl_mac_config_filter_flags * @he_support: does this MAC support HE + * @he_ap_support: HE AP enabled, "pseudo HE", no trigger frame handling * @eht_support: does this MAC support EHT. Requires he_support * @nic_not_ack_enabled: mark that the NIC doesn't support receiving * ACK-enabled AGG, (i.e. both BACK and non-BACK frames in single AGG). @@ -296,13 +288,13 @@ struct iwl_mac_config_cmd { u8 local_mld_addr[6]; __le16 reserved_for_local_mld_addr; __le32 filter_flags; - __le32 he_support; + __le16 he_support; + __le16 he_ap_support; __le32 eht_support; __le32 nic_not_ack_enabled; /* MAC_CONTEXT_CONFIG_SPECIFIC_DATA_API_U_VER_1 */ union { struct iwl_mac_client_data client; - struct iwl_mac_go_ibss_data go_ibss; struct iwl_mac_p2p_dev_data p2p_dev; }; } __packed; /* MAC_CONTEXT_CONFIG_CMD_API_S_VER_1 */ diff --git a/drivers/net/wireless/intel/iwlwifi/fw/api/scan.h b/drivers/net/wireless/intel/iwlwifi/fw/api/scan.h index 7ba0e3409199..ec96ba053a5c 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/api/scan.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/api/scan.h @@ -709,10 +709,13 @@ enum iwl_umac_scan_general_flags_v2 { * should be aware of a P2P GO operation on the 2GHz band. * @IWL_UMAC_SCAN_GEN_PARAMS_FLAGS2_RESPECT_P2P_GO_HB: scan event scheduling * should be aware of a P2P GO operation on the 5GHz or 6GHz band. + * @IWL_UMAC_SCAN_GEN_PARAMS_FLAGS2_DONT_TOGGLE_ANT: don't toggle between + * valid antennas, and use the same antenna as in previous scan */ enum iwl_umac_scan_general_params_flags2 { IWL_UMAC_SCAN_GEN_PARAMS_FLAGS2_RESPECT_P2P_GO_LB = BIT(0), IWL_UMAC_SCAN_GEN_PARAMS_FLAGS2_RESPECT_P2P_GO_HB = BIT(1), + IWL_UMAC_SCAN_GEN_PARAMS_FLAGS2_DONT_TOGGLE_ANT = BIT(2), }; /** diff --git a/drivers/net/wireless/intel/iwlwifi/fw/dbg.c b/drivers/net/wireless/intel/iwlwifi/fw/dbg.c index ca97f2fcb693..d9faaae01abd 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/dbg.c +++ b/drivers/net/wireless/intel/iwlwifi/fw/dbg.c @@ -1038,7 +1038,7 @@ iwl_dump_ini_prph_mac_iter(struct iwl_fw_runtime *fwrt, range->range_data_size = reg->dev_addr.size; for (i = 0; i < le32_to_cpu(reg->dev_addr.size); i += 4) { prph_val = iwl_read_prph(fwrt->trans, addr + i); - if (prph_val == 0x5a5a5a5a) + if ((prph_val & ~0xf) == 0xa5a5a5a0) return -EBUSY; *val++ = cpu_to_le32(prph_val); } @@ -1388,13 +1388,13 @@ static void iwl_ini_get_rxf_data(struct iwl_fw_runtime *fwrt, if (!data) return; + memset(data, 0, sizeof(*data)); + /* make sure only one bit is set in only one fid */ if (WARN_ONCE(hweight_long(fid1) + hweight_long(fid2) != 1, "fid1=%x, fid2=%x\n", fid1, fid2)) return; - memset(data, 0, sizeof(*data)); - if (fid1) { fifo_idx = ffs(fid1) - 1; if (WARN_ONCE(fifo_idx >= MAX_NUM_LMAC, "fifo_idx=%d\n", @@ -1562,7 +1562,7 @@ iwl_dump_ini_dbgi_sram_iter(struct iwl_fw_runtime *fwrt, prph_data = iwl_read_prph_no_grab(fwrt->trans, (i % 2) ? DBGI_SRAM_TARGET_ACCESS_RDATA_MSB : DBGI_SRAM_TARGET_ACCESS_RDATA_LSB); - if (prph_data == 0x5a5a5a5a) { + if ((prph_data & ~0xf) == 0xa5a5a5a0) { iwl_trans_release_nic_access(fwrt->trans); return -EBUSY; } @@ -2345,6 +2345,8 @@ static u32 iwl_dump_ini_file_name_info(struct iwl_fw_runtime *fwrt, /* add the dump file name extension tlv to the list */ list_add_tail(&entry->list, list); + fwrt->trans->dbg.dump_file_name_ext_valid = false; + return entry->size; } diff --git a/drivers/net/wireless/intel/iwlwifi/fw/debugfs.c b/drivers/net/wireless/intel/iwlwifi/fw/debugfs.c index 43e997283db0..607e07ed2477 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/debugfs.c +++ b/drivers/net/wireless/intel/iwlwifi/fw/debugfs.c @@ -317,8 +317,10 @@ static void *iwl_dbgfs_fw_info_seq_next(struct seq_file *seq, const struct iwl_fw *fw = priv->fwrt->fw; *pos = ++state->pos; - if (*pos >= fw->ucode_capa.n_cmd_versions) + if (*pos >= fw->ucode_capa.n_cmd_versions) { + kfree(state); return NULL; + } return state; } diff --git a/drivers/net/wireless/intel/iwlwifi/fw/dump.c b/drivers/net/wireless/intel/iwlwifi/fw/dump.c index 59ed321bcc27..f86f7b4baa18 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/dump.c +++ b/drivers/net/wireless/intel/iwlwifi/fw/dump.c @@ -484,6 +484,9 @@ static void iwl_fwrt_dump_fseq_regs(struct iwl_fw_runtime *fwrt) void iwl_fwrt_dump_error_logs(struct iwl_fw_runtime *fwrt) { + struct iwl_pc_data *pc_data; + u8 count; + if (!test_bit(STATUS_DEVICE_ENABLED, &fwrt->trans->status)) { IWL_ERR(fwrt, "DEVICE_ENABLED bit is not set. Aborting dump.\n"); @@ -502,6 +505,14 @@ void iwl_fwrt_dump_error_logs(struct iwl_fw_runtime *fwrt) iwl_fwrt_dump_rcm_error_log(fwrt, 1); iwl_fwrt_dump_iml_error_log(fwrt); iwl_fwrt_dump_fseq_regs(fwrt); + if (fwrt->trans->trans_cfg->device_family >= IWL_DEVICE_FAMILY_22000) { + pc_data = fwrt->trans->dbg.pc_data; + for (count = 0; count < fwrt->trans->dbg.num_pc; + count++, pc_data++) + IWL_ERR(fwrt, "%s: 0x%x\n", + pc_data->pc_name, + pc_data->pc_address); + } if (fwrt->trans->trans_cfg->device_family >= IWL_DEVICE_FAMILY_BZ) { u32 scratch = iwl_read32(fwrt->trans, CSR_FUNC_SCRATCH); diff --git a/drivers/net/wireless/intel/iwlwifi/fw/file.h b/drivers/net/wireless/intel/iwlwifi/fw/file.h index f2eefca6e260..cddf09d6be1c 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/file.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/file.h @@ -101,8 +101,10 @@ enum iwl_ucode_tlv_type { IWL_UCODE_TLV_SEC_TABLE_ADDR = 66, IWL_UCODE_TLV_D3_KEK_KCK_ADDR = 67, + IWL_UCODE_TLV_CURRENT_PC = 68, IWL_UCODE_TLV_FW_NUM_STATIONS = IWL_UCODE_TLV_CONST_BASE + 0, + IWL_UCODE_TLV_FW_NUM_BEACONS = IWL_UCODE_TLV_CONST_BASE + 2, IWL_UCODE_TLV_TYPE_DEBUG_INFO = IWL_UCODE_TLV_DEBUG_BASE + 0, IWL_UCODE_TLV_TYPE_BUFFER_ALLOCATION = IWL_UCODE_TLV_DEBUG_BASE + 1, @@ -458,6 +460,8 @@ enum iwl_ucode_tlv_capa { IWL_UCODE_TLV_CAPA_SYNCED_TIME = (__force iwl_ucode_tlv_capa_t)106, IWL_UCODE_TLV_CAPA_TIME_SYNC_BOTH_FTM_TM = (__force iwl_ucode_tlv_capa_t)108, IWL_UCODE_TLV_CAPA_BIGTK_TX_SUPPORT = (__force iwl_ucode_tlv_capa_t)109, + IWL_UCODE_TLV_CAPA_MLD_API_SUPPORT = (__force iwl_ucode_tlv_capa_t)110, + IWL_UCODE_TLV_CAPA_SCAN_DONT_TOGGLE_ANT = (__force iwl_ucode_tlv_capa_t)111, #ifdef __CHECKER__ /* sparse says it cannot increment the previous enum member */ diff --git a/drivers/net/wireless/intel/iwlwifi/fw/img.h b/drivers/net/wireless/intel/iwlwifi/fw/img.h index f5c4d93d1033..8d0d58d61892 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/img.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/img.h @@ -51,6 +51,7 @@ struct iwl_ucode_capabilities { u32 error_log_addr; u32 error_log_size; u32 num_stations; + u32 num_beacons; unsigned long _api[BITS_TO_LONGS(NUM_IWL_UCODE_TLV_API)]; unsigned long _capa[BITS_TO_LONGS(NUM_IWL_UCODE_TLV_CAPA)]; diff --git a/drivers/net/wireless/intel/iwlwifi/fw/runtime.h b/drivers/net/wireless/intel/iwlwifi/fw/runtime.h index a59cf4d9567c..df689a9b7e2c 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/runtime.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/runtime.h @@ -165,6 +165,7 @@ struct iwl_fw_runtime { struct iwl_ppag_chain ppag_chains[IWL_NUM_CHAIN_LIMITS]; u32 ppag_flags; u32 ppag_ver; + bool ppag_table_valid; struct iwl_sar_offset_mapping_cmd sgom_table; bool sgom_enabled; u8 reduced_power_flags; diff --git a/drivers/net/wireless/intel/iwlwifi/fw/uefi.c b/drivers/net/wireless/intel/iwlwifi/fw/uefi.c index 0b6f694cf30d..01afea33c38c 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/uefi.c +++ b/drivers/net/wireless/intel/iwlwifi/fw/uefi.c @@ -222,7 +222,7 @@ void *iwl_uefi_get_reduced_power(struct iwl_trans *trans, size_t *len) return ERR_PTR(-ENOMEM); status = efi.get_variable(IWL_UEFI_REDUCED_POWER_NAME, &IWL_EFI_VAR_GUID, - NULL, &package_size, data); + NULL, &package_size, package); if (status != EFI_SUCCESS) { IWL_DEBUG_FW(trans, "Reduced Power UEFI variable not found 0x%lx (len %lu)\n", diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-config.h b/drivers/net/wireless/intel/iwlwifi/iwl-config.h index 9b7b6fca2b1b..411b7d4fcc9a 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-config.h +++ b/drivers/net/wireless/intel/iwlwifi/iwl-config.h @@ -469,6 +469,7 @@ struct iwl_dev_info { u16 mac_type; u16 rf_type; u8 mac_step; + u8 rf_step; u8 rf_id; u8 no_160; u8 cores; @@ -639,11 +640,17 @@ extern const struct iwl_cfg iwl_cfg_ma_a0_gf4_a0; extern const struct iwl_cfg iwl_cfg_ma_a0_mr_a0; extern const struct iwl_cfg iwl_cfg_ma_a0_ms_a0; extern const struct iwl_cfg iwl_cfg_ma_a0_fm_a0; +extern const struct iwl_cfg iwl_cfg_ma_b0_hr_b0; +extern const struct iwl_cfg iwl_cfg_ma_b0_gf_a0; +extern const struct iwl_cfg iwl_cfg_ma_b0_gf4_a0; +extern const struct iwl_cfg iwl_cfg_ma_b0_mr_a0; +extern const struct iwl_cfg iwl_cfg_ma_b0_fm_a0; extern const struct iwl_cfg iwl_cfg_snj_a0_mr_a0; extern const struct iwl_cfg iwl_cfg_snj_a0_ms_a0; extern const struct iwl_cfg iwl_cfg_so_a0_hr_a0; extern const struct iwl_cfg iwl_cfg_so_a0_ms_a0; extern const struct iwl_cfg iwl_cfg_quz_a0_hr_b0; +extern const struct iwl_cfg iwl_cfg_bz_a0_hr_a0; extern const struct iwl_cfg iwl_cfg_bz_a0_hr_b0; extern const struct iwl_cfg iwl_cfg_bz_a0_gf_a0; extern const struct iwl_cfg iwl_cfg_bz_a0_gf4_a0; @@ -661,7 +668,9 @@ extern const struct iwl_cfg iwl_cfg_bnj_a0_gf_a0; extern const struct iwl_cfg iwl_cfg_bnj_b0_gf_a0; extern const struct iwl_cfg iwl_cfg_bnj_a0_gf4_a0; extern const struct iwl_cfg iwl_cfg_bnj_b0_gf4_a0; +extern const struct iwl_cfg iwl_cfg_bnj_a0_hr_a0; extern const struct iwl_cfg iwl_cfg_bnj_a0_hr_b0; +extern const struct iwl_cfg iwl_cfg_bnj_b0_hr_a0; extern const struct iwl_cfg iwl_cfg_bnj_b0_hr_b0; extern const struct iwl_cfg iwl_cfg_bnj_b0_fm_b0; extern const struct iwl_cfg iwl_cfg_bnj_b0_fm4_b0; diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-csr.h b/drivers/net/wireless/intel/iwlwifi/iwl-csr.h index bece76b1a514..587368a0ad4a 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-csr.h +++ b/drivers/net/wireless/intel/iwlwifi/iwl-csr.h @@ -102,6 +102,8 @@ #define CSR_LTR_LONG_VAL_AD_SNOOP_VAL 0x000003ff #define CSR_LTR_LONG_VAL_AD_SCALE_USEC 2 +#define CSR_LTR_LAST_MSG (CSR_BASE + 0x0DC) + /* GIO Chicken Bits (PCI Express bus link power management) */ #define CSR_GIO_CHICKEN_BITS (CSR_BASE+0x100) @@ -309,6 +311,8 @@ enum { SILICON_A_STEP = 0, SILICON_B_STEP, SILICON_C_STEP, + SILICON_D_STEP, + SILICON_E_STEP, SILICON_Z_STEP = 0xf, }; diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-dbg-tlv.c b/drivers/net/wireless/intel/iwlwifi/iwl-dbg-tlv.c index 87366b70b17f..898d5dcf1012 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-dbg-tlv.c +++ b/drivers/net/wireless/intel/iwlwifi/iwl-dbg-tlv.c @@ -138,6 +138,12 @@ static int iwl_dbg_tlv_alloc_buf_alloc(struct iwl_trans *trans, alloc_id != IWL_FW_INI_ALLOCATION_ID_DBGC1) goto err; + if (buf_location == IWL_FW_INI_LOCATION_DRAM_PATH && + alloc->req_size == 0) { + IWL_ERR(trans, "WRT: Invalid DRAM buffer allocation requested size (0)\n"); + return -EINVAL; + } + trans->dbg.fw_mon_cfg[alloc_id] = *alloc; return 0; @@ -797,7 +803,7 @@ static void iwl_dbg_tlv_update_drams(struct iwl_fw_runtime *fwrt) if (!ret) dram_alloc = true; else - IWL_WARN(fwrt, + IWL_INFO(fwrt, "WRT: Failed to set DRAM buffer for alloc id %d, ret=%d\n", i, ret); } diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-debug.c b/drivers/net/wireless/intel/iwlwifi/iwl-debug.c index ae4c2a3d63d5..3a3c13a41fc6 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-debug.c +++ b/drivers/net/wireless/intel/iwlwifi/iwl-debug.c @@ -1,6 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause /* - * Copyright (C) 2005-2011, 2021 Intel Corporation + * Copyright (C) 2005-2011, 2021-2022 Intel Corporation */ #include <linux/device.h> #include <linux/interrupt.h> @@ -57,6 +57,7 @@ void __iwl_err(struct device *dev, enum iwl_err_mode mode, const char *fmt, ...) default: break; } + vaf.va = &args; trace_iwlwifi_err(&vaf); va_end(args); } diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-drv.c b/drivers/net/wireless/intel/iwlwifi/iwl-drv.c index 4c977ba9cd85..34feb4d29adc 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-drv.c +++ b/drivers/net/wireless/intel/iwlwifi/iwl-drv.c @@ -127,6 +127,7 @@ static void iwl_dealloc_ucode(struct iwl_drv *drv) kfree(drv->fw.iml); kfree(drv->fw.ucode_capa.cmd_versions); kfree(drv->fw.phy_integration_ver); + kfree(drv->trans->dbg.pc_data); for (i = 0; i < IWL_UCODE_TYPE_MAX; i++) iwl_free_fw_img(drv, drv->fw.img + i); @@ -894,7 +895,7 @@ static int iwl_parse_tlv_firmware(struct iwl_drv *drv, drv->fw.img[IWL_UCODE_WOWLAN].is_dual_cpus = true; } else if ((num_of_cpus > 2) || (num_of_cpus < 1)) { - IWL_ERR(drv, "Driver support upto 2 CPUs\n"); + IWL_ERR(drv, "Driver support up to 2 CPUs\n"); return -EINVAL; } break; @@ -1154,6 +1155,12 @@ static int iwl_parse_tlv_firmware(struct iwl_drv *drv, capa->num_stations = le32_to_cpup((const __le32 *)tlv_data); break; + case IWL_UCODE_TLV_FW_NUM_BEACONS: + if (tlv_len != sizeof(u32)) + goto invalid_tlv_len; + capa->num_beacons = + le32_to_cpup((const __le32 *)tlv_data); + break; case IWL_UCODE_TLV_UMAC_DEBUG_ADDRS: { const struct iwl_umac_debug_addrs *dbg_ptrs = (const void *)tlv_data; @@ -1232,6 +1239,14 @@ static int iwl_parse_tlv_firmware(struct iwl_drv *drv, iwl_drv_set_dump_exclude(drv, tlv_type, tlv_data, tlv_len); break; + case IWL_UCODE_TLV_CURRENT_PC: + if (tlv_len < sizeof(struct iwl_pc_data)) + goto invalid_tlv_len; + drv->trans->dbg.num_pc = + tlv_len / sizeof(struct iwl_pc_data); + drv->trans->dbg.pc_data = + kmemdup(tlv_data, tlv_len, GFP_KERNEL); + break; default: IWL_DEBUG_INFO(drv, "unknown TLV: %d\n", tlv_type); break; @@ -1406,6 +1421,7 @@ static void iwl_req_fw_callback(const struct firmware *ucode_raw, void *context) IWL_DEFAULT_STANDARD_PHY_CALIBRATE_TBL_SIZE; fw->ucode_capa.n_scan_channels = IWL_DEFAULT_SCAN_CHANNELS; fw->ucode_capa.num_stations = IWL_MVM_STATION_COUNT_MAX; + fw->ucode_capa.num_beacons = 1; /* dump all fw memory areas by default */ fw->dbg.dump_mask = 0xffffffff; diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-eeprom-parse.h b/drivers/net/wireless/intel/iwlwifi/iwl-eeprom-parse.h index baa643386018..0e8ca761d24b 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-eeprom-parse.h +++ b/drivers/net/wireless/intel/iwlwifi/iwl-eeprom-parse.h @@ -47,13 +47,12 @@ struct iwl_nvm_data { struct ieee80211_supported_band bands[NUM_NL80211_BANDS]; /* - * iftype data for low (2.4 GHz) and high (5 and 6 GHz) bands, - * we can use the same for 5 and 6 GHz bands because they have - * the same data + * iftype data for low (2.4 GHz) high (5 GHz) and uhb (6 GHz) bands */ struct { struct ieee80211_sband_iftype_data low[2]; struct ieee80211_sband_iftype_data high[2]; + struct ieee80211_sband_iftype_data uhb[2]; } iftd; struct ieee80211_channel channels[]; diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-nvm-parse.c b/drivers/net/wireless/intel/iwlwifi/iwl-nvm-parse.c index 923bbfc151dd..7dcb1c3ab728 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-nvm-parse.c +++ b/drivers/net/wireless/intel/iwlwifi/iwl-nvm-parse.c @@ -860,7 +860,10 @@ iwl_nvm_fixup_sband_iftd(struct iwl_trans *trans, /* Advertise an A-MPDU exponent extension based on * operating band */ - if (sband->band != NL80211_BAND_2GHZ) + if (sband->band == NL80211_BAND_6GHZ && iftype_data->eht_cap.has_eht) + iftype_data->he_cap.he_cap_elem.mac_cap_info[3] |= + IEEE80211_HE_MAC_CAP3_MAX_AMPDU_LEN_EXP_EXT_2; + else if (sband->band != NL80211_BAND_2GHZ) iftype_data->he_cap.he_cap_elem.mac_cap_info[3] |= IEEE80211_HE_MAC_CAP3_MAX_AMPDU_LEN_EXP_EXT_1; else @@ -876,16 +879,13 @@ iwl_nvm_fixup_sband_iftd(struct iwl_trans *trans, IEEE80211_EHT_MAC_CAP0_MAX_MPDU_LEN_MASK); break; case NL80211_BAND_6GHZ: - if (!is_ap || iwlwifi_mod_params.nvm_file) - iftype_data->eht_cap.eht_cap_elem.phy_cap_info[0] |= - IEEE80211_EHT_PHY_CAP0_320MHZ_IN_6GHZ; + iftype_data->eht_cap.eht_cap_elem.phy_cap_info[0] |= + IEEE80211_EHT_PHY_CAP0_320MHZ_IN_6GHZ; fallthrough; case NL80211_BAND_5GHZ: iftype_data->he_cap.he_cap_elem.phy_cap_info[0] |= - IEEE80211_HE_PHY_CAP0_CHANNEL_WIDTH_SET_40MHZ_80MHZ_IN_5G; - if (!is_ap || iwlwifi_mod_params.nvm_file) - iftype_data->he_cap.he_cap_elem.phy_cap_info[0] |= - IEEE80211_HE_PHY_CAP0_CHANNEL_WIDTH_SET_160MHZ_IN_5G; + IEEE80211_HE_PHY_CAP0_CHANNEL_WIDTH_SET_40MHZ_80MHZ_IN_5G | + IEEE80211_HE_PHY_CAP0_CHANNEL_WIDTH_SET_160MHZ_IN_5G; break; default: WARN_ON(1); @@ -938,6 +938,10 @@ iwl_nvm_fixup_sband_iftd(struct iwl_trans *trans, } } + if (trans->trans_cfg->device_family >= IWL_DEVICE_FAMILY_AX210 && !is_ap) + iftype_data->he_cap.he_cap_elem.phy_cap_info[2] |= + IEEE80211_HE_PHY_CAP2_UL_MU_FULL_MU_MIMO; + switch (CSR_HW_RFID_TYPE(trans->hw_rf_id)) { case IWL_CFG_RF_TYPE_GF: case IWL_CFG_RF_TYPE_MR: @@ -999,15 +1003,18 @@ static void iwl_init_he_hw_capab(struct iwl_trans *trans, BUILD_BUG_ON(sizeof(data->iftd.low) != sizeof(iwl_he_eht_capa)); BUILD_BUG_ON(sizeof(data->iftd.high) != sizeof(iwl_he_eht_capa)); + BUILD_BUG_ON(sizeof(data->iftd.uhb) != sizeof(iwl_he_eht_capa)); switch (sband->band) { case NL80211_BAND_2GHZ: iftype_data = data->iftd.low; break; case NL80211_BAND_5GHZ: - case NL80211_BAND_6GHZ: iftype_data = data->iftd.high; break; + case NL80211_BAND_6GHZ: + iftype_data = data->iftd.uhb; + break; default: WARN_ON(1); return; diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-prph.h b/drivers/net/wireless/intel/iwlwifi/iwl-prph.h index 62ce116d3783..0dfe00eae05d 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-prph.h +++ b/drivers/net/wireless/intel/iwlwifi/iwl-prph.h @@ -350,6 +350,11 @@ #define WFPM_OTP_CFG1_ADDR 0x00a03098 #define WFPM_OTP_CFG1_IS_JACKET_BIT BIT(4) #define WFPM_OTP_CFG1_IS_CDB_BIT BIT(5) +#define WFPM_OTP_BZ_BNJ_JACKET_BIT 5 +#define WFPM_OTP_BZ_BNJ_CDB_BIT 4 +#define WFPM_OTP_CFG1_IS_JACKET(_val) (((_val) & 0x00000020) >> WFPM_OTP_BZ_BNJ_JACKET_BIT) +#define WFPM_OTP_CFG1_IS_CDB(_val) (((_val) & 0x00000010) >> WFPM_OTP_BZ_BNJ_CDB_BIT) + #define WFPM_GP2 0xA030B4 @@ -445,6 +450,8 @@ enum { #define REG_CRF_ID_TYPE_GF_TC 0xF08 #define REG_CRF_ID_TYPE_MR 0x810 #define REG_CRF_ID_TYPE_FM 0x910 +#define REG_CRF_ID_TYPE_FMI 0x930 +#define REG_CRF_ID_TYPE_FMR 0x900 #define HPM_DEBUG 0xA03440 #define PERSISTENCE_BIT BIT(12) diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-trans.h b/drivers/net/wireless/intel/iwlwifi/iwl-trans.h index dd277a4fa8dd..9f1228b5a384 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-trans.h +++ b/drivers/net/wireless/intel/iwlwifi/iwl-trans.h @@ -748,6 +748,18 @@ struct iwl_imr_data { __le64 imr_base_addr; }; +#define IWL_TRANS_CURRENT_PC_NAME_MAX_BYTES 32 + +/** + * struct iwl_pc_data - program counter details + * @pc_name: cpu name + * @pc_address: cpu program counter + */ +struct iwl_pc_data { + u8 pc_name[IWL_TRANS_CURRENT_PC_NAME_MAX_BYTES]; + u32 pc_address; +}; + /** * struct iwl_trans_debug - transport debug related data * @@ -777,6 +789,8 @@ struct iwl_imr_data { * @ucode_preset: preset based on ucode * @dump_file_name_ext: dump file name extension * @dump_file_name_ext_valid: dump file name extension if valid or not + * @num_pc: number of program counter for cpu + * @pc_data: details of the program counter */ struct iwl_trans_debug { u8 n_dest_reg; @@ -817,6 +831,8 @@ struct iwl_trans_debug { struct iwl_imr_data imr_data; u8 dump_file_name_ext[IWL_FW_INI_MAX_NAME]; bool dump_file_name_ext_valid; + u32 num_pc; + struct iwl_pc_data *pc_data; }; struct iwl_dma_ptr { @@ -981,7 +997,7 @@ struct iwl_trans_txqs { * 0 indicates that frag SKBs (NETIF_F_SG) aren't supported. * @hw_rf_id a u32 with the device RF ID * @hw_crf_id a u32 with the device CRF ID - * @hw_cdb_id a u32 with the device CDB ID + * @hw_wfpm_id a u32 with the device wfpm ID * @hw_id: a u32 with the ID of the device / sub-device. * Set during transport allocation. * @hw_id_str: a string with info about HW ID. Set during transport allocation. @@ -1024,7 +1040,8 @@ struct iwl_trans { u32 hw_rev_step; u32 hw_rf_id; u32 hw_crf_id; - u32 hw_cdb_id; + u32 hw_cnv_id; + u32 hw_wfpm_id; u32 hw_id; char hw_id_str[52]; u32 sku_id[3]; diff --git a/drivers/net/wireless/intel/iwlwifi/mei/iwl-mei.h b/drivers/net/wireless/intel/iwlwifi/mei/iwl-mei.h index ae66192feefe..655d95d3a068 100644 --- a/drivers/net/wireless/intel/iwlwifi/mei/iwl-mei.h +++ b/drivers/net/wireless/intel/iwlwifi/mei/iwl-mei.h @@ -1,6 +1,6 @@ /* SPDX-License-Identifier: GPL-2.0-only */ /* - * Copyright (C) 2021 Intel Corporation + * Copyright (C) 2021 - 2022 Intel Corporation */ #ifndef __iwl_mei_h__ @@ -301,7 +301,7 @@ struct iwl_mei_colloc_info { struct iwl_mei_ops { void (*me_conn_status)(void *priv, const struct iwl_mei_conn_info *conn_info); - void (*rfkill)(void *priv, bool blocked); + void (*rfkill)(void *priv, bool blocked, bool csme_taking_ownership); void (*roaming_forbidden)(void *priv, bool forbidden); void (*sap_connected)(void *priv); void (*nic_stolen)(void *priv); diff --git a/drivers/net/wireless/intel/iwlwifi/mei/main.c b/drivers/net/wireless/intel/iwlwifi/mei/main.c index 67dfb77fedf7..0a29fb013005 100644 --- a/drivers/net/wireless/intel/iwlwifi/mei/main.c +++ b/drivers/net/wireless/intel/iwlwifi/mei/main.c @@ -31,6 +31,11 @@ MODULE_LICENSE("GPL"); #define MEI_WLAN_UUID UUID_LE(0x13280904, 0x7792, 0x4fcb, \ 0xa1, 0xaa, 0x5e, 0x70, 0xcb, 0xb1, 0xe8, 0x65) +/* After CSME takes ownership, it won't release it for 60 seconds to avoid + * frequent ownership transitions. + */ +#define MEI_OWNERSHIP_RETAKE_TIMEOUT_MS msecs_to_jiffies(60000) + /* * Since iwlwifi calls iwlmei without any context, hold a pointer to the * mei_cl_device structure here. @@ -156,6 +161,8 @@ struct iwl_mei_filters { * accessed without the mutex. * @netdev_work: used to defer registering and unregistering of the netdev to * avoid taking the rtnl lock in the SAP messages handlers. + * @ownership_dwork: used to re-ask for NIC ownership after ownership was taken + * by CSME or when a previous ownership request failed. * @sap_seq_no: the sequence number for the SAP messages * @seq_no: the sequence number for the SAP messages * @dbgfs_dir: the debugfs dir entry @@ -179,6 +186,7 @@ struct iwl_mei { bool pldr_active; spinlock_t data_q_lock; struct work_struct netdev_work; + struct delayed_work ownership_dwork; atomic_t sap_seq_no; atomic_t seq_no; @@ -716,7 +724,7 @@ iwl_mei_handle_conn_status(struct mei_cl_device *cldev, status->link_prot_state); else iwl_mei_cache.ops->rfkill(iwl_mei_cache.priv, - status->link_prot_state); + status->link_prot_state, false); } static void iwl_mei_set_init_conf(struct iwl_mei *mei) @@ -788,7 +796,7 @@ static void iwl_mei_handle_amt_state(struct mei_cl_device *cldev, if (mei->amt_enabled) iwl_mei_set_init_conf(mei); else if (iwl_mei_cache.ops) - iwl_mei_cache.ops->rfkill(iwl_mei_cache.priv, false); + iwl_mei_cache.ops->rfkill(iwl_mei_cache.priv, false, false); schedule_work(&mei->netdev_work); @@ -829,10 +837,12 @@ static void iwl_mei_handle_csme_taking_ownership(struct mei_cl_device *cldev, */ mei->csme_taking_ownership = true; - iwl_mei_cache.ops->rfkill(iwl_mei_cache.priv, true); + iwl_mei_cache.ops->rfkill(iwl_mei_cache.priv, true, true); } else { iwl_mei_send_sap_msg(cldev, SAP_MSG_NOTIF_CSME_OWNERSHIP_CONFIRMED); + schedule_delayed_work(&mei->ownership_dwork, + MEI_OWNERSHIP_RETAKE_TIMEOUT_MS); } } @@ -882,7 +892,7 @@ static void iwl_mei_handle_rx_host_own_req(struct mei_cl_device *cldev, /* We can now start the connection, unblock rfkill */ if (iwl_mei_cache.ops) - iwl_mei_cache.ops->rfkill(iwl_mei_cache.priv, false); + iwl_mei_cache.ops->rfkill(iwl_mei_cache.priv, false, false); } static void iwl_mei_handle_pldr_ack(struct mei_cl_device *cldev, @@ -1447,7 +1457,13 @@ int iwl_mei_get_ownership(void) ret = wait_event_timeout(mei->get_ownership_wq, mei->got_ownership, HZ / 2); - return (!ret) ? -ETIMEDOUT : 0; + if (!ret) { + schedule_delayed_work(&mei->ownership_dwork, + MEI_OWNERSHIP_RETAKE_TIMEOUT_MS); + return -ETIMEDOUT; + } + + return 0; out: mutex_unlock(&iwl_mei_mutex); return ret; @@ -1738,6 +1754,8 @@ void iwl_mei_device_state(bool up) iwl_mei_send_sap_msg(mei->cldev, SAP_MSG_NOTIF_CSME_OWNERSHIP_CONFIRMED); mei->csme_taking_ownership = false; + schedule_delayed_work(&mei->ownership_dwork, + MEI_OWNERSHIP_RETAKE_TIMEOUT_MS); out: mutex_unlock(&iwl_mei_mutex); } @@ -1773,7 +1791,8 @@ int iwl_mei_register(void *priv, const struct iwl_mei_ops *ops) if (iwl_mei_is_connected()) { if (mei->amt_enabled) iwl_mei_send_sap_msg(mei->cldev, - SAP_MSG_NOTIF_WIFIDR_UP); + SAP_MSG_NOTIF_WIFIDR_UP, + false); ops->rfkill(priv, mei->link_prot_state); } } @@ -1894,6 +1913,11 @@ static void iwl_mei_dbgfs_unregister(struct iwl_mei *mei) {} #endif /* CONFIG_DEBUG_FS */ +static void iwl_mei_ownership_dwork(struct work_struct *wk) +{ + iwl_mei_get_ownership(); +} + #define ALLOC_SHARED_MEM_RETRY_MAX_NUM 3 /* @@ -1923,6 +1947,7 @@ static int iwl_mei_probe(struct mei_cl_device *cldev, init_waitqueue_head(&mei->pldr_wq); spin_lock_init(&mei->data_q_lock); INIT_WORK(&mei->netdev_work, iwl_mei_netdev_work); + INIT_DELAYED_WORK(&mei->ownership_dwork, iwl_mei_ownership_dwork); mei_cldev_set_drvdata(cldev, mei); mei->cldev = cldev; @@ -2087,7 +2112,7 @@ static void iwl_mei_remove(struct mei_cl_device *cldev) spin_unlock_bh(&mei->data_q_lock); if (iwl_mei_cache.ops) - iwl_mei_cache.ops->rfkill(iwl_mei_cache.priv, false); + iwl_mei_cache.ops->rfkill(iwl_mei_cache.priv, false, false); /* * mei_cldev_disable will return only after all the MEI Rx is done. @@ -2105,6 +2130,7 @@ static void iwl_mei_remove(struct mei_cl_device *cldev) cancel_work_sync(&mei->send_csa_msg_wk); cancel_delayed_work_sync(&mei->csa_throttle_end_wk); cancel_work_sync(&mei->netdev_work); + cancel_delayed_work_sync(&mei->ownership_dwork); /* * If someone waits for the ownership, let him know that we are going diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/d3.c b/drivers/net/wireless/intel/iwlwifi/mvm/d3.c index 40adf789c8fc..37aa4676dc94 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/d3.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/d3.c @@ -564,6 +564,7 @@ static void iwl_mvm_wowlan_get_tkip_data(struct ieee80211_hw *hw, } for (i = 0; i < IWL_NUM_RSC; i++) { + ieee80211_get_key_rx_seq(key, i, &seq); /* wrapping isn't allowed, AP must rekey */ if (seq.tkip.iv32 > cur_rx_iv32) cur_rx_iv32 = seq.tkip.iv32; @@ -2017,6 +2018,12 @@ static void iwl_mvm_parse_wowlan_info_notif(struct iwl_mvm *mvm, { u32 i; + if (!data) { + IWL_ERR(mvm, "iwl_wowlan_info_notif data is NULL\n"); + status = NULL; + return; + } + if (len < sizeof(*data)) { IWL_ERR(mvm, "Invalid WoWLAN info notification!\n"); status = NULL; @@ -2705,10 +2712,15 @@ static bool iwl_mvm_wait_d3_notif(struct iwl_notif_wait_data *notif_wait, struct iwl_d3_data *d3_data = data; u32 len; int ret; + int wowlan_info_ver = iwl_fw_lookup_notif_ver(mvm->fw, + PROT_OFFLOAD_GROUP, + WOWLAN_INFO_NOTIFICATION, + IWL_FW_CMD_VER_UNKNOWN); + switch (WIDE_ID(pkt->hdr.group_id, pkt->hdr.cmd)) { case WIDE_ID(PROT_OFFLOAD_GROUP, WOWLAN_INFO_NOTIFICATION): { - struct iwl_wowlan_info_notif *notif = (void *)pkt->data; + struct iwl_wowlan_info_notif *notif; if (d3_data->notif_received & IWL_D3_NOTIF_WOWLAN_INFO) { /* We might get two notifications due to dual bss */ @@ -2717,10 +2729,32 @@ static bool iwl_mvm_wait_d3_notif(struct iwl_notif_wait_data *notif_wait, break; } + if (wowlan_info_ver < 2) { + struct iwl_wowlan_info_notif_v1 *notif_v1 = (void *)pkt->data; + + notif = kmemdup(notif_v1, + offsetofend(struct iwl_wowlan_info_notif, + received_beacons), + GFP_ATOMIC); + + if (!notif) + return false; + + notif->tid_tear_down = notif_v1->tid_tear_down; + notif->station_id = notif_v1->station_id; + + } else { + notif = (void *)pkt->data; + } + d3_data->notif_received |= IWL_D3_NOTIF_WOWLAN_INFO; len = iwl_rx_packet_payload_len(pkt); iwl_mvm_parse_wowlan_info_notif(mvm, notif, d3_data->status, len); + + if (wowlan_info_ver < 2) + kfree(notif); + if (d3_data->status && d3_data->status->wakeup_reasons & IWL_WOWLAN_WAKEUP_REASON_HAS_WAKEUP_PKT) /* We are supposed to get also wake packet notif */ diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/debugfs.c b/drivers/net/wireless/intel/iwlwifi/mvm/debugfs.c index 527daaf46f96..84a488538427 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/debugfs.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/debugfs.c @@ -340,6 +340,26 @@ static ssize_t iwl_dbgfs_sar_geo_profile_read(struct file *file, return simple_read_from_buffer(user_buf, count, ppos, buf, pos); } + +static ssize_t iwl_dbgfs_wifi_6e_enable_read(struct file *file, + char __user *user_buf, + size_t count, loff_t *ppos) +{ + struct iwl_mvm *mvm = file->private_data; + int err, pos; + char buf[12]; + u32 value; + + err = iwl_acpi_get_dsm_u32(mvm->fwrt.dev, 0, + DSM_FUNC_ENABLE_6E, + &iwl_guid, &value); + if (err) + return err; + + pos = sprintf(buf, "0x%08x\n", value); + + return simple_read_from_buffer(user_buf, count, ppos, buf, pos); +} #endif static ssize_t iwl_dbgfs_stations_read(struct file *file, char __user *user_buf, @@ -1898,6 +1918,7 @@ MVM_DEBUGFS_READ_FILE_OPS(uapsd_noagg_bssids); #ifdef CONFIG_ACPI MVM_DEBUGFS_READ_FILE_OPS(sar_geo_profile); +MVM_DEBUGFS_READ_FILE_OPS(wifi_6e_enable); #endif MVM_DEBUGFS_READ_WRITE_STA_FILE_OPS(amsdu_len, 16); @@ -1940,6 +1961,11 @@ static ssize_t iwl_dbgfs_mem_read(struct file *file, char __user *user_buf, if (ret < 0) return ret; + if (iwl_rx_packet_payload_len(hcmd.resp_pkt) < sizeof(*rsp)) { + ret = -EIO; + goto out; + } + rsp = (void *)hcmd.resp_pkt->data; if (le32_to_cpu(rsp->status) != DEBUG_MEM_STATUS_SUCCESS) { ret = -ENXIO; @@ -2016,6 +2042,11 @@ static ssize_t iwl_dbgfs_mem_write(struct file *file, if (ret < 0) return ret; + if (iwl_rx_packet_payload_len(hcmd.resp_pkt) < sizeof(*rsp)) { + ret = -EIO; + goto out; + } + rsp = (void *)hcmd.resp_pkt->data; if (rsp->status != DEBUG_MEM_STATUS_SUCCESS) { ret = -ENXIO; @@ -2092,6 +2123,7 @@ void iwl_mvm_dbgfs_register(struct iwl_mvm *mvm) MVM_DEBUGFS_ADD_FILE(tas_get_status, mvm->debugfs_dir, 0400); #ifdef CONFIG_ACPI MVM_DEBUGFS_ADD_FILE(sar_geo_profile, mvm->debugfs_dir, 0400); + MVM_DEBUGFS_ADD_FILE(wifi_6e_enable, mvm->debugfs_dir, 0400); #endif MVM_DEBUGFS_ADD_FILE(he_sniffer_params, mvm->debugfs_dir, 0600); diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/ftm-initiator.c b/drivers/net/wireless/intel/iwlwifi/mvm/ftm-initiator.c index 379da4bec5dd..3963a0d4ed04 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/ftm-initiator.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/ftm-initiator.c @@ -25,6 +25,10 @@ struct iwl_mvm_smooth_entry { u64 host_time; }; +enum iwl_mvm_pasn_flags { + IWL_MVM_PASN_FLAG_HAS_HLTK = BIT(0), +}; + struct iwl_mvm_ftm_pasn_entry { struct list_head list; u8 addr[ETH_ALEN]; @@ -33,6 +37,7 @@ struct iwl_mvm_ftm_pasn_entry { u8 cipher; u8 tx_pn[IEEE80211_CCMP_PN_LEN]; u8 rx_pn[IEEE80211_CCMP_PN_LEN]; + u32 flags; }; int iwl_mvm_ftm_add_pasn_sta(struct iwl_mvm *mvm, struct ieee80211_vif *vif, @@ -79,14 +84,24 @@ int iwl_mvm_ftm_add_pasn_sta(struct iwl_mvm *mvm, struct ieee80211_vif *vif, rcu_read_unlock(); } - if (tk_len != expected_tk_len || hltk_len != sizeof(pasn->hltk)) { + if (tk_len != expected_tk_len || + (hltk_len && hltk_len != sizeof(pasn->hltk))) { IWL_ERR(mvm, "Invalid key length: tk_len=%u hltk_len=%u\n", tk_len, hltk_len); goto out; } + if (!expected_tk_len && !hltk_len) { + IWL_ERR(mvm, "TK and HLTK not set\n"); + goto out; + } + memcpy(pasn->addr, addr, sizeof(pasn->addr)); - memcpy(pasn->hltk, hltk, sizeof(pasn->hltk)); + + if (hltk_len) { + memcpy(pasn->hltk, hltk, sizeof(pasn->hltk)); + pasn->flags |= IWL_MVM_PASN_FLAG_HAS_HLTK; + } if (tk && tk_len) memcpy(pasn->tk, tk, sizeof(pasn->tk)); @@ -691,7 +706,11 @@ iwl_mvm_ftm_set_secured_ranging(struct iwl_mvm *mvm, struct ieee80211_vif *vif, continue; target->cipher = entry->cipher; - memcpy(target->hltk, entry->hltk, sizeof(target->hltk)); + + if (entry->flags & IWL_MVM_PASN_FLAG_HAS_HLTK) + memcpy(target->hltk, entry->hltk, sizeof(target->hltk)); + else + memset(target->hltk, 0, sizeof(target->hltk)); if (vif->cfg.assoc && !memcmp(vif->bss_conf.bssid, target->bssid, diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/ftm-responder.c b/drivers/net/wireless/intel/iwlwifi/mvm/ftm-responder.c index c37d793d6a4a..1b6fb73ddfc7 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/ftm-responder.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/ftm-responder.c @@ -317,6 +317,8 @@ int iwl_mvm_ftm_respoder_add_pasn_sta(struct iwl_mvm *mvm, .addr = addr, .hltk = hltk, }; + struct iwl_mvm_pasn_hltk_data *hltk_data_ptr = NULL; + u8 cmd_ver = iwl_fw_lookup_cmd_ver(mvm->fw, WIDE_ID(LOCATION_GROUP, TOF_RESPONDER_DYN_CONFIG_CMD), 2); @@ -328,12 +330,21 @@ int iwl_mvm_ftm_respoder_add_pasn_sta(struct iwl_mvm *mvm, return -ENOTSUPP; } - hltk_data.cipher = iwl_mvm_cipher_to_location_cipher(cipher); - if (hltk_data.cipher == IWL_LOCATION_CIPHER_INVALID) { - IWL_ERR(mvm, "invalid cipher: %u\n", cipher); + if ((!hltk || !hltk_len) && (!tk || !tk_len)) { + IWL_ERR(mvm, "TK and HLTK not set\n"); return -EINVAL; } + if (hltk && hltk_len) { + hltk_data.cipher = iwl_mvm_cipher_to_location_cipher(cipher); + if (hltk_data.cipher == IWL_LOCATION_CIPHER_INVALID) { + IWL_ERR(mvm, "invalid cipher: %u\n", cipher); + return -EINVAL; + } + + hltk_data_ptr = &hltk_data; + } + if (tk && tk_len) { sta = kzalloc(sizeof(*sta), GFP_KERNEL); if (!sta) @@ -350,7 +361,7 @@ int iwl_mvm_ftm_respoder_add_pasn_sta(struct iwl_mvm *mvm, list_add_tail(&sta->list, &mvm->resp_pasn_list); } - ret = iwl_mvm_ftm_responder_dyn_cfg_v3(mvm, vif, NULL, &hltk_data); + ret = iwl_mvm_ftm_responder_dyn_cfg_v3(mvm, vif, NULL, hltk_data_ptr); if (ret && sta) iwl_mvm_resp_del_pasn_sta(mvm, vif, sta); diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/fw.c b/drivers/net/wireless/intel/iwlwifi/mvm/fw.c index 7fe733dcc748..b35c96cf7ad2 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/fw.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/fw.c @@ -321,6 +321,8 @@ static int iwl_mvm_load_ucode_wait_alive(struct iwl_mvm *mvm, static const u16 alive_cmd[] = { UCODE_ALIVE_NTFY }; bool run_in_rfkill = ucode_type == IWL_UCODE_INIT || iwl_mvm_has_unified_ucode(mvm); + u8 count; + struct iwl_pc_data *pc_data; if (ucode_type == IWL_UCODE_REGULAR && iwl_fw_dbg_conf_usniffer(mvm->fw, FW_DBG_START_FROM_ALIVE) && @@ -393,6 +395,14 @@ static int iwl_mvm_load_ucode_wait_alive(struct iwl_mvm *mvm, /* LMAC/UMAC PC info */ if (trans->trans_cfg->device_family >= + IWL_DEVICE_FAMILY_22000) { + pc_data = trans->dbg.pc_data; + for (count = 0; count < trans->dbg.num_pc; + count++, pc_data++) + IWL_ERR(mvm, "%s: 0x%x\n", + pc_data->pc_name, + pc_data->pc_address); + } else if (trans->trans_cfg->device_family >= IWL_DEVICE_FAMILY_9000) { IWL_ERR(mvm, "UMAC PC: 0x%x\n", iwl_read_umac_prph(trans, @@ -467,111 +477,6 @@ static int iwl_mvm_load_ucode_wait_alive(struct iwl_mvm *mvm, return 0; } -static int iwl_run_unified_mvm_ucode(struct iwl_mvm *mvm) -{ - struct iwl_notification_wait init_wait; - struct iwl_nvm_access_complete_cmd nvm_complete = {}; - struct iwl_init_extended_cfg_cmd init_cfg = { - .init_flags = cpu_to_le32(BIT(IWL_INIT_NVM)), - }; - static const u16 init_complete[] = { - INIT_COMPLETE_NOTIF, - }; - int ret; - - if (mvm->trans->cfg->tx_with_siso_diversity) - init_cfg.init_flags |= cpu_to_le32(BIT(IWL_INIT_PHY)); - - lockdep_assert_held(&mvm->mutex); - - mvm->rfkill_safe_init_done = false; - - iwl_init_notification_wait(&mvm->notif_wait, - &init_wait, - init_complete, - ARRAY_SIZE(init_complete), - iwl_wait_init_complete, - NULL); - - iwl_dbg_tlv_time_point(&mvm->fwrt, IWL_FW_INI_TIME_POINT_EARLY, NULL); - - /* Will also start the device */ - ret = iwl_mvm_load_ucode_wait_alive(mvm, IWL_UCODE_REGULAR); - if (ret) { - IWL_ERR(mvm, "Failed to start RT ucode: %d\n", ret); - goto error; - } - iwl_dbg_tlv_time_point(&mvm->fwrt, IWL_FW_INI_TIME_POINT_AFTER_ALIVE, - NULL); - - /* Send init config command to mark that we are sending NVM access - * commands - */ - ret = iwl_mvm_send_cmd_pdu(mvm, WIDE_ID(SYSTEM_GROUP, - INIT_EXTENDED_CFG_CMD), - CMD_SEND_IN_RFKILL, - sizeof(init_cfg), &init_cfg); - if (ret) { - IWL_ERR(mvm, "Failed to run init config command: %d\n", - ret); - goto error; - } - - /* Load NVM to NIC if needed */ - if (mvm->nvm_file_name) { - ret = iwl_read_external_nvm(mvm->trans, mvm->nvm_file_name, - mvm->nvm_sections); - if (ret) - goto error; - ret = iwl_mvm_load_nvm_to_nic(mvm); - if (ret) - goto error; - } - - if (IWL_MVM_PARSE_NVM && !mvm->nvm_data) { - ret = iwl_nvm_init(mvm); - if (ret) { - IWL_ERR(mvm, "Failed to read NVM: %d\n", ret); - goto error; - } - } - - ret = iwl_mvm_send_cmd_pdu(mvm, WIDE_ID(REGULATORY_AND_NVM_GROUP, - NVM_ACCESS_COMPLETE), - CMD_SEND_IN_RFKILL, - sizeof(nvm_complete), &nvm_complete); - if (ret) { - IWL_ERR(mvm, "Failed to run complete NVM access: %d\n", - ret); - goto error; - } - - /* We wait for the INIT complete notification */ - ret = iwl_wait_notification(&mvm->notif_wait, &init_wait, - MVM_UCODE_ALIVE_TIMEOUT); - if (ret) - return ret; - - /* Read the NVM only at driver load time, no need to do this twice */ - if (!IWL_MVM_PARSE_NVM && !mvm->nvm_data) { - mvm->nvm_data = iwl_get_nvm(mvm->trans, mvm->fw); - if (IS_ERR(mvm->nvm_data)) { - ret = PTR_ERR(mvm->nvm_data); - mvm->nvm_data = NULL; - IWL_ERR(mvm, "Failed to read NVM: %d\n", ret); - return ret; - } - } - - mvm->rfkill_safe_init_done = true; - - return 0; - -error: - iwl_remove_notification(&mvm->notif_wait, &init_wait); - return ret; -} - #ifdef CONFIG_ACPI static void iwl_mvm_phy_filter_init(struct iwl_mvm *mvm, struct iwl_phy_specific_cfg *phy_filters) @@ -698,6 +603,118 @@ static int iwl_send_phy_cfg_cmd(struct iwl_mvm *mvm) return iwl_mvm_send_cmd_pdu(mvm, cmd_id, 0, cmd_size, &phy_cfg_cmd); } +static int iwl_run_unified_mvm_ucode(struct iwl_mvm *mvm) +{ + struct iwl_notification_wait init_wait; + struct iwl_nvm_access_complete_cmd nvm_complete = {}; + struct iwl_init_extended_cfg_cmd init_cfg = { + .init_flags = cpu_to_le32(BIT(IWL_INIT_NVM)), + }; + static const u16 init_complete[] = { + INIT_COMPLETE_NOTIF, + }; + int ret; + + if (mvm->trans->cfg->tx_with_siso_diversity) + init_cfg.init_flags |= cpu_to_le32(BIT(IWL_INIT_PHY)); + + lockdep_assert_held(&mvm->mutex); + + mvm->rfkill_safe_init_done = false; + + iwl_init_notification_wait(&mvm->notif_wait, + &init_wait, + init_complete, + ARRAY_SIZE(init_complete), + iwl_wait_init_complete, + NULL); + + iwl_dbg_tlv_time_point(&mvm->fwrt, IWL_FW_INI_TIME_POINT_EARLY, NULL); + + /* Will also start the device */ + ret = iwl_mvm_load_ucode_wait_alive(mvm, IWL_UCODE_REGULAR); + if (ret) { + IWL_ERR(mvm, "Failed to start RT ucode: %d\n", ret); + goto error; + } + iwl_dbg_tlv_time_point(&mvm->fwrt, IWL_FW_INI_TIME_POINT_AFTER_ALIVE, + NULL); + + /* Send init config command to mark that we are sending NVM access + * commands + */ + ret = iwl_mvm_send_cmd_pdu(mvm, WIDE_ID(SYSTEM_GROUP, + INIT_EXTENDED_CFG_CMD), + CMD_SEND_IN_RFKILL, + sizeof(init_cfg), &init_cfg); + if (ret) { + IWL_ERR(mvm, "Failed to run init config command: %d\n", + ret); + goto error; + } + + /* Load NVM to NIC if needed */ + if (mvm->nvm_file_name) { + ret = iwl_read_external_nvm(mvm->trans, mvm->nvm_file_name, + mvm->nvm_sections); + if (ret) + goto error; + ret = iwl_mvm_load_nvm_to_nic(mvm); + if (ret) + goto error; + } + + if (IWL_MVM_PARSE_NVM && !mvm->nvm_data) { + ret = iwl_nvm_init(mvm); + if (ret) { + IWL_ERR(mvm, "Failed to read NVM: %d\n", ret); + goto error; + } + } + + ret = iwl_mvm_send_cmd_pdu(mvm, WIDE_ID(REGULATORY_AND_NVM_GROUP, + NVM_ACCESS_COMPLETE), + CMD_SEND_IN_RFKILL, + sizeof(nvm_complete), &nvm_complete); + if (ret) { + IWL_ERR(mvm, "Failed to run complete NVM access: %d\n", + ret); + goto error; + } + + ret = iwl_send_phy_cfg_cmd(mvm); + if (ret) { + IWL_ERR(mvm, "Failed to run PHY configuration: %d\n", + ret); + goto error; + } + + /* We wait for the INIT complete notification */ + ret = iwl_wait_notification(&mvm->notif_wait, &init_wait, + MVM_UCODE_ALIVE_TIMEOUT); + if (ret) + return ret; + + /* Read the NVM only at driver load time, no need to do this twice */ + if (!IWL_MVM_PARSE_NVM && !mvm->nvm_data) { + mvm->nvm_data = iwl_get_nvm(mvm->trans, mvm->fw); + if (IS_ERR(mvm->nvm_data)) { + ret = PTR_ERR(mvm->nvm_data); + mvm->nvm_data = NULL; + IWL_ERR(mvm, "Failed to read NVM: %d\n", ret); + return ret; + } + } + + mvm->rfkill_safe_init_done = true; + + return 0; + +error: + iwl_remove_notification(&mvm->notif_wait, &init_wait); + return ret; +} + int iwl_run_init_mvm_ucode(struct iwl_mvm *mvm) { struct iwl_notification_wait calib_wait; @@ -1538,12 +1555,11 @@ int iwl_mvm_up(struct iwl_mvm *mvm) ret = iwl_send_phy_db_data(mvm->phy_db); if (ret) goto error; + ret = iwl_send_phy_cfg_cmd(mvm); + if (ret) + goto error; } - ret = iwl_send_phy_cfg_cmd(mvm); - if (ret) - goto error; - ret = iwl_mvm_send_bt_init_conf(mvm); if (ret) goto error; @@ -1711,8 +1727,6 @@ int iwl_mvm_up(struct iwl_mvm *mvm) iwl_mvm_tas_init(mvm); iwl_mvm_leds_sync(mvm); - iwl_mvm_ftm_initiator_smooth_config(mvm); - if (fw_has_capa(&mvm->fw->ucode_capa, IWL_UCODE_TLV_CAPA_RFIM_SUPPORT)) { if (iwl_mvm_eval_dsm_rfi(mvm) == DSM_VALUE_RFI_ENABLE) diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mac-ctxt.c b/drivers/net/wireless/intel/iwlwifi/mvm/mac-ctxt.c index 82fad042a281..cc90f2884cff 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/mac-ctxt.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/mac-ctxt.c @@ -225,16 +225,20 @@ int iwl_mvm_mac_ctxt_init(struct iwl_mvm *mvm, struct ieee80211_vif *vif) * that we should share it with another interface. */ - /* Currently, MAC ID 0 should be used only for the managed/IBSS vif */ - switch (vif->type) { - case NL80211_IFTYPE_ADHOC: - break; - case NL80211_IFTYPE_STATION: - if (!vif->p2p) + /* MAC ID 0 should be used only for the managed/IBSS vif with non-MLO + * FW API + */ + if (!mvm->mld_api_is_used) { + switch (vif->type) { + case NL80211_IFTYPE_ADHOC: break; - fallthrough; - default: - __clear_bit(0, data.available_mac_ids); + case NL80211_IFTYPE_STATION: + if (!vif->p2p) + break; + fallthrough; + default: + __clear_bit(0, data.available_mac_ids); + } } ieee80211_iterate_active_interfaces_atomic( @@ -870,17 +874,44 @@ static u32 iwl_mvm_find_ie_offset(u8 *beacon, u8 eid, u32 frame_size) return ie - beacon; } -static u8 iwl_mvm_mac_ctxt_get_lowest_rate(struct iwl_mvm *mvm, - struct ieee80211_tx_info *info, - struct ieee80211_vif *vif) +u8 iwl_mvm_mac_ctxt_get_lowest_rate(struct iwl_mvm *mvm, + struct ieee80211_tx_info *info, + struct ieee80211_vif *vif) { + struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif); struct ieee80211_supported_band *sband; unsigned long basic = vif->bss_conf.basic_rates; u16 lowest_cck = IWL_RATE_COUNT, lowest_ofdm = IWL_RATE_COUNT; + u32 link_id = u32_get_bits(info->control.flags, + IEEE80211_TX_CTRL_MLO_LINK); + u8 band = info->band; u8 rate; u32 i; - sband = mvm->hw->wiphy->bands[info->band]; + if (link_id == IEEE80211_LINK_UNSPECIFIED && vif->valid_links) { + for (i = 0; i < ARRAY_SIZE(mvmvif->link); i++) { + if (!mvmvif->link[i]) + continue; + /* shouldn't do this when >1 link is active */ + WARN_ON_ONCE(link_id != IEEE80211_LINK_UNSPECIFIED); + link_id = i; + } + } + + if (link_id < IEEE80211_LINK_UNSPECIFIED) { + struct ieee80211_bss_conf *link_conf; + + rcu_read_lock(); + link_conf = rcu_dereference(vif->link_conf[link_id]); + if (link_conf) { + basic = link_conf->basic_rates; + if (link_conf->chandef.chan) + band = link_conf->chandef.chan->band; + } + rcu_read_unlock(); + } + + sband = mvm->hw->wiphy->bands[band]; for_each_set_bit(i, &basic, BITS_PER_LONG) { u16 hw = sband->bitrates[i].hw_value; @@ -892,7 +923,9 @@ static u8 iwl_mvm_mac_ctxt_get_lowest_rate(struct iwl_mvm *mvm, } } - if (info->band == NL80211_BAND_2GHZ && !vif->p2p) { + if (band == NL80211_BAND_2GHZ && !vif->p2p && + vif->type != NL80211_IFTYPE_P2P_DEVICE && + !(info->flags & IEEE80211_TX_CTL_NO_CCK_RATE)) { if (lowest_cck != IWL_RATE_COUNT) rate = lowest_cck; else if (lowest_ofdm != IWL_RATE_COUNT) @@ -1102,10 +1135,10 @@ static int iwl_mvm_mac_ctxt_send_beacon_v9(struct iwl_mvm *mvm, sizeof(beacon_cmd)); } -int iwl_mvm_mac_ctxt_send_beacon(struct iwl_mvm *mvm, - struct ieee80211_vif *vif, - struct sk_buff *beacon, - struct ieee80211_bss_conf *link_conf) +static int iwl_mvm_mac_ctxt_send_beacon(struct iwl_mvm *mvm, + struct ieee80211_vif *vif, + struct sk_buff *beacon, + struct ieee80211_bss_conf *link_conf) { if (WARN_ON(!beacon)) return -EINVAL; diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c b/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c index aaa7e3c561a0..0f01b62357c6 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c @@ -227,14 +227,18 @@ int iwl_mvm_init_fw_regd(struct iwl_mvm *mvm) static const u8 he_if_types_ext_capa_sta[] = { [0] = WLAN_EXT_CAPA1_EXT_CHANNEL_SWITCHING, [2] = WLAN_EXT_CAPA3_MULTI_BSSID_SUPPORT, - [7] = WLAN_EXT_CAPA8_OPMODE_NOTIF, + [7] = WLAN_EXT_CAPA8_OPMODE_NOTIF | + WLAN_EXT_CAPA8_MAX_MSDU_IN_AMSDU_LSB, + [8] = WLAN_EXT_CAPA9_MAX_MSDU_IN_AMSDU_MSB, }; static const u8 tm_if_types_ext_capa_sta[] = { [0] = WLAN_EXT_CAPA1_EXT_CHANNEL_SWITCHING, [2] = WLAN_EXT_CAPA3_MULTI_BSSID_SUPPORT | WLAN_EXT_CAPA3_TIMING_MEASUREMENT_SUPPORT, - [7] = WLAN_EXT_CAPA8_OPMODE_NOTIF, + [7] = WLAN_EXT_CAPA8_OPMODE_NOTIF | + WLAN_EXT_CAPA8_MAX_MSDU_IN_AMSDU_LSB, + [8] = WLAN_EXT_CAPA9_MAX_MSDU_IN_AMSDU_MSB, [9] = WLAN_EXT_CAPA10_TWT_REQUESTER_SUPPORT, }; @@ -301,6 +305,11 @@ int iwl_mvm_mac_setup_register(struct iwl_mvm *mvm) ieee80211_hw_set(hw, BUFF_MMPDU_TXQ); ieee80211_hw_set(hw, STA_MMPDU_TXQ); + /* Set this early since we need to have it for the check below */ + if (mvm->mld_api_is_used && + mvm->trans->trans_cfg->device_family >= IWL_DEVICE_FAMILY_BZ) + hw->wiphy->flags |= WIPHY_FLAG_SUPPORTS_MLO; + /* With MLD FW API, it tracks timing by itself, * no need for any timing from the host */ @@ -409,6 +418,8 @@ int iwl_mvm_mac_setup_register(struct iwl_mvm *mvm) wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_BEACON_RATE_LEGACY); + wiphy_ext_feature_set(hw->wiphy, + NL80211_EXT_FEATURE_SCAN_MIN_PREQ_CONTENT); if (fw_has_capa(&mvm->fw->ucode_capa, IWL_UCODE_TLV_CAPA_FTM_CALIBRATED)) { @@ -1061,6 +1072,9 @@ static void iwl_mvm_restart_cleanup(struct iwl_mvm *mvm) mvm->rx_ba_sessions = 0; mvm->fwrt.dump.conf = FW_DBG_INVALID; mvm->monitor_on = false; +#ifdef CONFIG_IWLWIFI_DEBUGFS + mvm->beacon_inject_active = false; +#endif /* keep statistics ticking */ iwl_mvm_accu_radio_stats(mvm); @@ -3305,8 +3319,8 @@ void iwl_mvm_sta_pre_rcu_remove(struct ieee80211_hw *hw, lockdep_is_held(&mvm->mutex)); sta_id = link_sta->sta_id; if (sta == rcu_access_pointer(mvm->fw_id_to_mac_id[sta_id])) { - rcu_assign_pointer(mvm->fw_id_to_mac_id[sta_id], - ERR_PTR(-ENOENT)); + RCU_INIT_POINTER(mvm->fw_id_to_mac_id[sta_id], + ERR_PTR(-ENOENT)); RCU_INIT_POINTER(mvm->fw_id_to_link_sta[sta_id], NULL); } } @@ -3568,24 +3582,22 @@ static int iwl_mvm_mac_sta_state(struct ieee80211_hw *hw, */ static void iwl_mvm_rs_rate_init_all_links(struct iwl_mvm *mvm, struct ieee80211_vif *vif, - struct ieee80211_sta *sta, - bool update) + struct ieee80211_sta *sta) { struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif); unsigned int link_id; for_each_mvm_vif_valid_link(mvmvif, link_id) { struct ieee80211_bss_conf *conf = - link_conf_dereference_protected(vif, link_id); + link_conf_dereference_check(vif, link_id); struct ieee80211_link_sta *link_sta = - link_sta_dereference_protected(sta, link_id); + link_sta_dereference_check(sta, link_id); if (!conf || !link_sta || !mvmvif->link[link_id]->phy_ctxt) continue; - iwl_mvm_rs_rate_init(mvm, sta, conf, link_sta, - mvmvif->link[link_id]->phy_ctxt->channel->band, - update); + iwl_mvm_rs_rate_init(mvm, vif, sta, conf, link_sta, + mvmvif->link[link_id]->phy_ctxt->channel->band); } } @@ -3662,6 +3674,7 @@ iwl_mvm_sta_state_notexist_to_none(struct iwl_mvm *mvm, struct ieee80211_sta *sta, struct iwl_mvm_sta_state_ops *callbacks) { + struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif); unsigned int i; int ret; @@ -3697,6 +3710,9 @@ iwl_mvm_sta_state_notexist_to_none(struct iwl_mvm *mvm, } ieee80211_sta_recalc_aggregates(sta); + if (vif->type == NL80211_IFTYPE_STATION && !sta->tdls) + mvmvif->ap_sta = sta; + return 0; } @@ -3753,7 +3769,7 @@ iwl_mvm_sta_state_auth_to_assoc(struct ieee80211_hw *hw, } out: - iwl_mvm_rs_rate_init_all_links(mvm, vif, sta, false); + iwl_mvm_rs_rate_init_all_links(mvm, vif, sta); return callbacks->update_sta(mvm, vif, sta); } @@ -3786,7 +3802,9 @@ iwl_mvm_sta_state_assoc_to_authorized(struct iwl_mvm *mvm, iwl_mvm_mei_host_associated(mvm, vif, mvm_sta); } - iwl_mvm_rs_rate_init_all_links(mvm, vif, sta, true); + mvm_sta->authorized = true; + + iwl_mvm_rs_rate_init_all_links(mvm, vif, sta); return 0; } @@ -3798,14 +3816,17 @@ iwl_mvm_sta_state_authorized_to_assoc(struct iwl_mvm *mvm, struct iwl_mvm_sta_state_ops *callbacks) { struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif); + struct iwl_mvm_sta *mvmsta = iwl_mvm_sta_from_mac80211(sta); int ret; lockdep_assert_held(&mvm->mutex); + mvmsta->authorized = false; + /* once we move into assoc state, need to update rate scale to * disable using wide bandwidth */ - iwl_mvm_rs_rate_init_all_links(mvm, vif, sta, false); + iwl_mvm_rs_rate_init_all_links(mvm, vif, sta); if (!sta->tdls) { /* Set this but don't call iwl_mvm_mac_ctxt_changed() @@ -3924,8 +3945,10 @@ int iwl_mvm_mac_sta_state_common(struct ieee80211_hw *hw, ret = 0; } else if (old_state == IEEE80211_STA_NONE && new_state == IEEE80211_STA_NOTEXIST) { - if (vif->type == NL80211_IFTYPE_STATION && !sta->tdls) + if (vif->type == NL80211_IFTYPE_STATION && !sta->tdls) { iwl_mvm_stop_session_protection(mvm, vif); + mvmvif->ap_sta = NULL; + } ret = callbacks->rm_sta(mvm, vif, sta); if (sta->tdls) { iwl_mvm_recalc_tdls_state(mvm, vif, false); @@ -3968,15 +3991,11 @@ void iwl_mvm_sta_rc_update(struct ieee80211_hw *hw, struct ieee80211_vif *vif, struct ieee80211_sta *sta, u32 changed) { struct iwl_mvm *mvm = IWL_MAC80211_GET_MVM(hw); - struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif); if (changed & (IEEE80211_RC_BW_CHANGED | IEEE80211_RC_SUPP_RATES_CHANGED | IEEE80211_RC_NSS_CHANGED)) - iwl_mvm_rs_rate_init(mvm, sta, - &vif->bss_conf, &sta->deflink, - mvmvif->deflink.phy_ctxt->channel->band, - true); + iwl_mvm_rs_rate_init_all_links(mvm, vif, sta); if (vif->type == NL80211_IFTYPE_STATION && changed & IEEE80211_RC_NSS_CHANGED) @@ -4094,7 +4113,7 @@ static int __iwl_mvm_mac_set_key(struct ieee80211_hw *hw, struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif); struct iwl_mvm *mvm = IWL_MAC80211_GET_MVM(hw); struct iwl_mvm_sta *mvmsta = NULL; - struct iwl_mvm_key_pn *ptk_pn; + struct iwl_mvm_key_pn *ptk_pn = NULL; int keyidx = key->keyidx; u32 sec_key_id = WIDE_ID(DATA_PATH_GROUP, SEC_KEY_CMD); u8 sec_key_ver = iwl_fw_lookup_cmd_ver(mvm->fw, sec_key_id, 0); @@ -4252,6 +4271,10 @@ static int __iwl_mvm_mac_set_key(struct ieee80211_hw *hw, if (ret) { IWL_WARN(mvm, "set key failed\n"); key->hw_key_idx = STA_KEY_IDX_INVALID; + if (ptk_pn) { + RCU_INIT_POINTER(mvmsta->ptk_pn[keyidx], NULL); + kfree(ptk_pn); + } /* * can't add key for RX, but we don't need it * in the device for TX so still return 0, @@ -5588,6 +5611,7 @@ void iwl_mvm_mac_flush(struct ieee80211_hw *hw, struct ieee80211_vif *vif, struct iwl_mvm_vif *mvmvif; struct iwl_mvm_sta *mvmsta; struct ieee80211_sta *sta; + bool ap_sta_done = false; int i; u32 msk = 0; @@ -5616,8 +5640,14 @@ void iwl_mvm_mac_flush(struct ieee80211_hw *hw, struct ieee80211_vif *vif, if (mvmsta->vif != vif) continue; + if (sta == mvmvif->ap_sta) { + if (ap_sta_done) + continue; + ap_sta_done = true; + } + /* make sure only TDLS peers or the AP are flushed */ - WARN_ON_ONCE(i != mvmvif->deflink.ap_sta_id && !sta->tdls); + WARN_ON_ONCE(sta != mvmvif->ap_sta && !sta->tdls); if (drop) { if (iwl_mvm_flush_sta(mvm, mvmsta, false)) @@ -6129,7 +6159,7 @@ static bool iwl_mvm_mac_can_aggregate(struct ieee80211_hw *hw, { struct iwl_mvm *mvm = IWL_MAC80211_GET_MVM(hw); - if (mvm->trans->trans_cfg->device_family >= IWL_DEVICE_FAMILY_BZ) + if (iwl_mvm_has_new_tx_csum(mvm)) return iwl_mvm_tx_csum_bz(mvm, head, true) == iwl_mvm_tx_csum_bz(mvm, skb, true); diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mld-key.c b/drivers/net/wireless/intel/iwlwifi/mvm/mld-key.c index f4785c0a0b84..8853821b3716 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/mld-key.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/mld-key.c @@ -14,23 +14,41 @@ static u32 iwl_mvm_get_sec_sta_mask(struct iwl_mvm *mvm, struct ieee80211_key_conf *keyconf) { struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif); + struct iwl_mvm_vif_link_info *link_info = &mvmvif->deflink; - if (vif->type == NL80211_IFTYPE_AP && - !(keyconf->flags & IEEE80211_KEY_FLAG_PAIRWISE)) - return BIT(mvmvif->deflink.mcast_sta.sta_id); + lockdep_assert_held(&mvm->mutex); - if (sta) { - struct iwl_mvm_sta *mvmsta = iwl_mvm_sta_from_mac80211(sta); - - return BIT(mvmsta->deflink.sta_id); + if (keyconf->link_id >= 0) { + link_info = mvmvif->link[keyconf->link_id]; + if (!link_info) + return 0; } - if (vif->type == NL80211_IFTYPE_STATION && - mvmvif->deflink.ap_sta_id != IWL_MVM_INVALID_STA) - return BIT(mvmvif->deflink.ap_sta_id); + /* AP group keys are per link and should be on the mcast STA */ + if (vif->type == NL80211_IFTYPE_AP && + !(keyconf->flags & IEEE80211_KEY_FLAG_PAIRWISE)) + return BIT(link_info->mcast_sta.sta_id); + + /* for client mode use the AP STA also for group keys */ + if (!sta && vif->type == NL80211_IFTYPE_STATION) + sta = mvmvif->ap_sta; + + /* During remove the STA was removed and the group keys come later + * (which sounds like a bad sequence, but remember that to mac80211 the + * group keys have no sta pointer), so we don't have a STA now. + * Since this happens for group keys only, just use the link_info as + * the group keys are per link; make sure that is the case by checking + * we do have a link_id or are not doing MLO. + * Of course the same can be done during add as well, but we must do + * it during remove, since we don't have the mvmvif->ap_sta pointer. + */ + if (!sta && (keyconf->link_id >= 0 || !vif->valid_links)) + return BIT(link_info->ap_sta_id); + + /* STA should be non-NULL now, but iwl_mvm_sta_fw_id_mask() checks */ - /* invalid */ - return 0; + /* pass link_id to filter by it if not -1 (GTK on client) */ + return iwl_mvm_sta_fw_id_mask(mvm, sta, keyconf->link_id); } static u32 iwl_mvm_get_sec_flags(struct iwl_mvm *mvm, @@ -41,6 +59,8 @@ static u32 iwl_mvm_get_sec_flags(struct iwl_mvm *mvm, struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif); u32 flags = 0; + lockdep_assert_held(&mvm->mutex); + if (!(keyconf->flags & IEEE80211_KEY_FLAG_PAIRWISE)) flags |= IWL_SEC_KEY_FLAG_MCAST_KEY; @@ -68,22 +88,68 @@ static u32 iwl_mvm_get_sec_flags(struct iwl_mvm *mvm, break; } - rcu_read_lock(); - if (!sta && vif->type == NL80211_IFTYPE_STATION && - mvmvif->deflink.ap_sta_id != IWL_MVM_INVALID_STA) { - u8 sta_id = mvmvif->deflink.ap_sta_id; - - sta = rcu_dereference_check(mvm->fw_id_to_mac_id[sta_id], - lockdep_is_held(&mvm->mutex)); - } + if (!sta && vif->type == NL80211_IFTYPE_STATION) + sta = mvmvif->ap_sta; if (!IS_ERR_OR_NULL(sta) && sta->mfp) flags |= IWL_SEC_KEY_FLAG_MFP; - rcu_read_unlock(); return flags; } +struct iwl_mvm_sta_key_update_data { + struct ieee80211_sta *sta; + u32 old_sta_mask; + u32 new_sta_mask; + int err; +}; + +static void iwl_mvm_mld_update_sta_key(struct ieee80211_hw *hw, + struct ieee80211_vif *vif, + struct ieee80211_sta *sta, + struct ieee80211_key_conf *key, + void *_data) +{ + u32 cmd_id = WIDE_ID(DATA_PATH_GROUP, SEC_KEY_CMD); + struct iwl_mvm_sta_key_update_data *data = _data; + struct iwl_mvm *mvm = IWL_MAC80211_GET_MVM(hw); + struct iwl_sec_key_cmd cmd = { + .action = cpu_to_le32(FW_CTXT_ACTION_MODIFY), + .u.modify.old_sta_mask = cpu_to_le32(data->old_sta_mask), + .u.modify.new_sta_mask = cpu_to_le32(data->new_sta_mask), + .u.modify.key_id = cpu_to_le32(key->keyidx), + .u.modify.key_flags = + cpu_to_le32(iwl_mvm_get_sec_flags(mvm, vif, sta, key)), + }; + int err; + + /* only need to do this for pairwise keys (link_id == -1) */ + if (sta != data->sta || key->link_id >= 0) + return; + + err = iwl_mvm_send_cmd_pdu(mvm, cmd_id, CMD_ASYNC, sizeof(cmd), &cmd); + + if (err) + data->err = err; +} + +int iwl_mvm_mld_update_sta_keys(struct iwl_mvm *mvm, + struct ieee80211_vif *vif, + struct ieee80211_sta *sta, + u32 old_sta_mask, + u32 new_sta_mask) +{ + struct iwl_mvm_sta_key_update_data data = { + .sta = sta, + .old_sta_mask = old_sta_mask, + .new_sta_mask = new_sta_mask, + }; + + ieee80211_iter_keys_rcu(mvm->hw, vif, iwl_mvm_mld_update_sta_key, + &data); + return data.err; +} + static int __iwl_mvm_sec_key_del(struct iwl_mvm *mvm, u32 sta_mask, u32 key_flags, u32 keyidx, u32 flags) { @@ -118,6 +184,9 @@ int iwl_mvm_sec_key_add(struct iwl_mvm *mvm, if (WARN_ON(keyconf->keylen > sizeof(cmd.u.add.key))) return -EINVAL; + if (WARN_ON(!sta_mask)) + return -EINVAL; + if (keyconf->cipher == WLAN_CIPHER_SUITE_WEP40 || keyconf->cipher == WLAN_CIPHER_SUITE_WEP104) memcpy(cmd.u.add.key + IWL_SEC_WEP_KEY_OFFSET, keyconf->key, @@ -164,6 +233,9 @@ static int _iwl_mvm_sec_key_del(struct iwl_mvm *mvm, u32 key_flags = iwl_mvm_get_sec_flags(mvm, vif, sta, keyconf); int ret; + if (WARN_ON(!sta_mask)) + return -EINVAL; + ret = __iwl_mvm_sec_key_del(mvm, sta_mask, key_flags, keyconf->keyidx, flags); if (ret) diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mld-mac.c b/drivers/net/wireless/intel/iwlwifi/mvm/mld-mac.c index ab0ba85936b4..1717fb52dc12 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/mld-mac.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/mld-mac.c @@ -4,6 +4,16 @@ */ #include "mvm.h" +static void iwl_mvm_mld_set_he_support(struct iwl_mvm *mvm, + struct ieee80211_vif *vif, + struct iwl_mac_config_cmd *cmd) +{ + if (vif->type == NL80211_IFTYPE_AP) + cmd->he_ap_support = cpu_to_le16(1); + else + cmd->he_support = cpu_to_le16(1); +} + static void iwl_mvm_mld_mac_ctxt_cmd_common(struct iwl_mvm *mvm, struct ieee80211_vif *vif, struct iwl_mac_config_cmd *cmd, @@ -41,7 +51,7 @@ static void iwl_mvm_mld_mac_ctxt_cmd_common(struct iwl_mvm *mvm, * and enable both when we have MLO. */ if (vif->valid_links) { - cmd->he_support = cpu_to_le32(1); + iwl_mvm_mld_set_he_support(mvm, vif, cmd); cmd->eht_support = cpu_to_le32(1); return; } @@ -53,7 +63,7 @@ static void iwl_mvm_mld_mac_ctxt_cmd_common(struct iwl_mvm *mvm, continue; if (link_conf->he_support) - cmd->he_support = cpu_to_le32(1); + iwl_mvm_mld_set_he_support(mvm, vif, cmd); /* it's not reasonable to have EHT without HE and FW API doesn't * support it. Ignore EHT in this case. @@ -157,7 +167,6 @@ static int iwl_mvm_mld_mac_ctxt_cmd_ibss(struct iwl_mvm *mvm, struct ieee80211_vif *vif, u32 action) { - struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif); struct iwl_mac_config_cmd cmd = {}; WARN_ON(vif->type != NL80211_IFTYPE_ADHOC); @@ -168,9 +177,6 @@ static int iwl_mvm_mld_mac_ctxt_cmd_ibss(struct iwl_mvm *mvm, MAC_CFG_FILTER_ACCEPT_PROBE_REQ | MAC_CFG_FILTER_ACCEPT_GRP); - /* TODO: Assumes that the beacon id == mac context id */ - cmd.go_ibss.beacon_template = cpu_to_le32(mvmvif->id); - return iwl_mvm_mld_mac_ctxt_send_cmd(mvm, &cmd); } @@ -210,9 +216,6 @@ static int iwl_mvm_mld_mac_ctxt_cmd_ap_go(struct iwl_mvm *mvm, MAC_CFG_FILTER_ACCEPT_PROBE_REQ, MAC_CFG_FILTER_ACCEPT_BEACON); - /* TODO: Assume that the beacon id == mac context id */ - cmd.go_ibss.beacon_template = cpu_to_le32(mvmvif->id); - return iwl_mvm_mld_mac_ctxt_send_cmd(mvm, &cmd); } diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mld-mac80211.c b/drivers/net/wireless/intel/iwlwifi/mvm/mld-mac80211.c index 203f2513e7ea..fbc2d5ed1006 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/mld-mac80211.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/mld-mac80211.c @@ -256,6 +256,30 @@ __iwl_mvm_mld_assign_vif_chanctx(struct iwl_mvm *mvm, if (ret) goto out; + /* Initialize rate control for the AP station, since we might be + * doing a link switch here - we cannot initialize it before since + * this needs the phy context assigned (and in FW?), and we cannot + * do it later because it needs to be initialized as soon as we're + * able to TX on the link, i.e. when active. + * + * Firmware restart isn't quite correct yet for MLO, but we don't + * need to do it in that case anyway since it will happen from the + * normal station state callback. + */ + if (mvmvif->ap_sta && + !test_bit(IWL_MVM_STATUS_IN_HW_RESTART, &mvm->status)) { + struct ieee80211_link_sta *link_sta; + + rcu_read_lock(); + link_sta = rcu_dereference(mvmvif->ap_sta->link[link_id]); + + if (!WARN_ON_ONCE(!link_sta)) + iwl_mvm_rs_rate_init(mvm, vif, mvmvif->ap_sta, + link_conf, link_sta, + phy_ctxt->channel->band); + rcu_read_unlock(); + } + /* then activate */ ret = iwl_mvm_link_changed(mvm, vif, link_conf, LINK_CONTEXT_MODIFY_ACTIVE | @@ -882,7 +906,10 @@ iwl_mvm_mld_change_vif_links(struct ieee80211_hw *hw, n_active++; } - if (n_active > 1) + if (vif->type == NL80211_IFTYPE_AP && + n_active > mvm->fw->ucode_capa.num_beacons) + return -EOPNOTSUPP; + else if (n_active > 1) return -EOPNOTSUPP; } diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mld-sta.c b/drivers/net/wireless/intel/iwlwifi/mvm/mld-sta.c index 78d4f186cd99..0bfdf4462755 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/mld-sta.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/mld-sta.c @@ -4,6 +4,43 @@ */ #include "mvm.h" #include "time-sync.h" +#include "sta.h" + +u32 iwl_mvm_sta_fw_id_mask(struct iwl_mvm *mvm, struct ieee80211_sta *sta, + int filter_link_id) +{ + struct iwl_mvm_sta *mvmsta; + unsigned int link_id; + u32 result = 0; + + if (!sta) + return 0; + + mvmsta = iwl_mvm_sta_from_mac80211(sta); + + /* it's easy when the STA is not an MLD */ + if (!sta->valid_links) + return BIT(mvmsta->deflink.sta_id); + + /* but if it is an MLD, get the mask of all the FW STAs it has ... */ + for (link_id = 0; link_id < ARRAY_SIZE(mvmsta->link); link_id++) { + struct iwl_mvm_link_sta *link_sta; + + /* unless we have a specific link in mind */ + if (filter_link_id >= 0 && link_id != filter_link_id) + continue; + + link_sta = + rcu_dereference_check(mvmsta->link[link_id], + lockdep_is_held(&mvm->mutex)); + if (!link_sta) + continue; + + result |= BIT(link_sta->sta_id); + } + + return result; +} static int iwl_mvm_mld_send_sta_cmd(struct iwl_mvm *mvm, struct iwl_mvm_sta_cfg_cmd *cmd) @@ -262,19 +299,22 @@ int iwl_mvm_mld_add_aux_sta(struct iwl_mvm *mvm, u32 lmac_id) IWL_MAX_TID_COUNT, NULL); } -static int iwl_mvm_mld_disable_txq(struct iwl_mvm *mvm, int sta_id, +static int iwl_mvm_mld_disable_txq(struct iwl_mvm *mvm, u32 sta_mask, u16 *queueptr, u8 tid) { int queue = *queueptr; int ret = 0; + if (tid == IWL_MAX_TID_COUNT) + tid = IWL_MGMT_TID; + if (mvm->sta_remove_requires_queue_remove) { u32 cmd_id = WIDE_ID(DATA_PATH_GROUP, SCD_QUEUE_CONFIG_CMD); struct iwl_scd_queue_cfg_cmd remove_cmd = { .operation = cpu_to_le32(IWL_SCD_QUEUE_REMOVE), .u.remove.tid = cpu_to_le32(tid), - .u.remove.sta_mask = cpu_to_le32(BIT(sta_id)), + .u.remove.sta_mask = cpu_to_le32(sta_mask), }; ret = iwl_mvm_send_cmd_pdu(mvm, cmd_id, 0, @@ -304,7 +344,7 @@ static int iwl_mvm_mld_rm_int_sta(struct iwl_mvm *mvm, if (flush) iwl_mvm_flush_sta(mvm, int_sta, true); - iwl_mvm_mld_disable_txq(mvm, int_sta->sta_id, queuptr, tid); + iwl_mvm_mld_disable_txq(mvm, BIT(int_sta->sta_id), queuptr, tid); ret = iwl_mvm_mld_rm_sta_from_fw(mvm, int_sta->sta_id); if (ret) @@ -387,7 +427,6 @@ static int iwl_mvm_mld_cfg_sta(struct iwl_mvm *mvm, struct ieee80211_sta *sta, struct iwl_mvm_sta_cfg_cmd cmd = { .sta_id = cpu_to_le32(mvm_link_sta->sta_id), .station_type = cpu_to_le32(mvm_sta->sta_type), - .mfp = cpu_to_le32(sta->mfp), }; u32 agg_size = 0, mpdu_dens = 0; @@ -720,6 +759,7 @@ static void iwl_mvm_mld_disable_sta_queues(struct iwl_mvm *mvm, struct ieee80211_sta *sta) { struct iwl_mvm_sta *mvm_sta = iwl_mvm_sta_from_mac80211(sta); + u32 sta_mask = iwl_mvm_sta_fw_id_mask(mvm, sta, -1); int i; lockdep_assert_held(&mvm->mutex); @@ -728,7 +768,7 @@ static void iwl_mvm_mld_disable_sta_queues(struct iwl_mvm *mvm, if (mvm_sta->tid_data[i].txq_id == IWL_MVM_INVALID_QUEUE) continue; - iwl_mvm_mld_disable_txq(mvm, mvm_sta->deflink.sta_id, + iwl_mvm_mld_disable_txq(mvm, sta_mask, &mvm_sta->tid_data[i].txq_id, i); mvm_sta->tid_data[i].txq_id = IWL_MVM_INVALID_QUEUE; } @@ -870,11 +910,12 @@ void iwl_mvm_mld_modify_all_sta_disable_tx(struct iwl_mvm *mvm, rcu_read_unlock(); } -static int iwl_mvm_mld_update_sta_queue(struct iwl_mvm *mvm, - struct iwl_mvm_sta *mvm_sta, - u32 old_sta_mask, - u32 new_sta_mask) +static int iwl_mvm_mld_update_sta_queues(struct iwl_mvm *mvm, + struct ieee80211_sta *sta, + u32 old_sta_mask, + u32 new_sta_mask) { + struct iwl_mvm_sta *mvm_sta = iwl_mvm_sta_from_mac80211(sta); struct iwl_scd_queue_cfg_cmd cmd = { .operation = cpu_to_le32(IWL_SCD_QUEUE_MODIFY), .u.modify.old_sta_mask = cpu_to_le32(old_sta_mask), @@ -910,6 +951,70 @@ static int iwl_mvm_mld_update_sta_queue(struct iwl_mvm *mvm, return 0; } +static int iwl_mvm_mld_update_sta_baids(struct iwl_mvm *mvm, + u32 old_sta_mask, + u32 new_sta_mask) +{ + struct iwl_rx_baid_cfg_cmd cmd = { + .action = cpu_to_le32(IWL_RX_BAID_ACTION_MODIFY), + .modify.old_sta_id_mask = cpu_to_le32(old_sta_mask), + .modify.new_sta_id_mask = cpu_to_le32(new_sta_mask), + }; + u32 cmd_id = WIDE_ID(DATA_PATH_GROUP, RX_BAID_ALLOCATION_CONFIG_CMD); + int baid; + + BUILD_BUG_ON(sizeof(struct iwl_rx_baid_cfg_resp) != sizeof(baid)); + + for (baid = 0; baid < ARRAY_SIZE(mvm->baid_map); baid++) { + struct iwl_mvm_baid_data *data; + int ret; + + data = rcu_dereference_protected(mvm->baid_map[baid], + lockdep_is_held(&mvm->mutex)); + if (!data) + continue; + + if (!(data->sta_mask & old_sta_mask)) + continue; + + WARN_ONCE(data->sta_mask != old_sta_mask, + "BAID data for %d corrupted - expected 0x%x found 0x%x\n", + baid, old_sta_mask, data->sta_mask); + + cmd.modify.tid = cpu_to_le32(data->tid); + + ret = iwl_mvm_send_cmd_pdu(mvm, cmd_id, 0, sizeof(cmd), &cmd); + data->sta_mask = new_sta_mask; + if (ret) + return ret; + } + + return 0; +} + +static int iwl_mvm_mld_update_sta_resources(struct iwl_mvm *mvm, + struct ieee80211_vif *vif, + struct ieee80211_sta *sta, + u32 old_sta_mask, + u32 new_sta_mask) +{ + int ret; + + ret = iwl_mvm_mld_update_sta_queues(mvm, sta, + old_sta_mask, + new_sta_mask); + if (ret) + return ret; + + ret = iwl_mvm_mld_update_sta_keys(mvm, vif, sta, + old_sta_mask, + new_sta_mask); + if (ret) + return ret; + + return iwl_mvm_mld_update_sta_baids(mvm, old_sta_mask, new_sta_mask); +} + int iwl_mvm_mld_update_sta_links(struct iwl_mvm *mvm, struct ieee80211_vif *vif, struct ieee80211_sta *sta, @@ -946,9 +1051,10 @@ int iwl_mvm_mld_update_sta_links(struct iwl_mvm *mvm, } if (sta_mask_to_rem) { - ret = iwl_mvm_mld_update_sta_queue(mvm, mvm_sta, - current_sta_mask, - current_sta_mask & ~sta_mask_to_rem); + ret = iwl_mvm_mld_update_sta_resources(mvm, vif, sta, + current_sta_mask, + current_sta_mask & + ~sta_mask_to_rem); if (WARN_ON(ret)) goto err; @@ -1020,12 +1126,15 @@ int iwl_mvm_mld_update_sta_links(struct iwl_mvm *mvm, goto err; link_sta_added_to_fw |= BIT(link_id); + + iwl_mvm_rs_add_sta_link(mvm, mvm_sta_link); } if (sta_mask_added) { - ret = iwl_mvm_mld_update_sta_queue(mvm, mvm_sta, - current_sta_mask, - current_sta_mask | sta_mask_added); + ret = iwl_mvm_mld_update_sta_resources(mvm, vif, sta, + current_sta_mask, + current_sta_mask | + sta_mask_added); if (WARN_ON(ret)) goto err; } diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h b/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h index e32ce876951f..6e7470d3a826 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h +++ b/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h @@ -434,6 +434,8 @@ struct iwl_mvm_vif { /* TCP Checksum Offload */ netdev_features_t features; + struct ieee80211_sta *ap_sta; + /* we can only have 2 GTK + 2 IGTK active at a time */ struct ieee80211_key_conf *ap_early_keys[4]; @@ -678,7 +680,7 @@ __aligned(roundup_pow_of_two(sizeof(struct _iwl_mvm_reorder_buf_entry))) /** * struct iwl_mvm_baid_data - BA session data - * @sta_id: station id + * @sta_mask: current station mask for the BAID * @tid: tid of the session * @baid baid of the session * @timeout: the timeout set in the addba request @@ -692,7 +694,7 @@ __aligned(roundup_pow_of_two(sizeof(struct _iwl_mvm_reorder_buf_entry))) */ struct iwl_mvm_baid_data { struct rcu_head rcu_head; - u8 sta_id; + u32 sta_mask; u8 tid; u8 baid; u16 timeout; @@ -824,6 +826,12 @@ struct iwl_time_sync_data { bool active; }; +struct iwl_mei_scan_filter { + bool is_mei_limited_scan; + struct sk_buff_head scan_res; + struct work_struct scan_work; +}; + struct iwl_mvm { /* for logger access */ struct device *dev; @@ -1175,6 +1183,8 @@ struct iwl_mvm { bool pldr_sync; struct iwl_time_sync_data time_sync; + + struct iwl_mei_scan_filter mei_scan_filter; }; /* Extract MVM priv from op_mode and _hw */ @@ -1401,24 +1411,8 @@ static inline bool iwl_mvm_has_new_rx_api(struct iwl_mvm *mvm) static inline bool iwl_mvm_has_mld_api(const struct iwl_fw *fw) { - return (iwl_fw_lookup_cmd_ver(fw, LINK_CONFIG_CMD, - IWL_FW_CMD_VER_UNKNOWN) != - IWL_FW_CMD_VER_UNKNOWN) && - (iwl_fw_lookup_cmd_ver(fw, MAC_CONFIG_CMD, - IWL_FW_CMD_VER_UNKNOWN) != - IWL_FW_CMD_VER_UNKNOWN) && - (iwl_fw_lookup_cmd_ver(fw, STA_CONFIG_CMD, - IWL_FW_CMD_VER_UNKNOWN) != - IWL_FW_CMD_VER_UNKNOWN) && - (iwl_fw_lookup_cmd_ver(fw, AUX_STA_CMD, - IWL_FW_CMD_VER_UNKNOWN) != - IWL_FW_CMD_VER_UNKNOWN) && - (iwl_fw_lookup_cmd_ver(fw, STA_REMOVE_CMD, - IWL_FW_CMD_VER_UNKNOWN) != - IWL_FW_CMD_VER_UNKNOWN) && - (iwl_fw_lookup_cmd_ver(fw, STA_DISABLE_TX_CMD, - IWL_FW_CMD_VER_UNKNOWN) != - IWL_FW_CMD_VER_UNKNOWN); + return fw_has_capa(&fw->ucode_capa, + IWL_UCODE_TLV_CAPA_MLD_API_SUPPORT); } static inline bool iwl_mvm_has_new_tx_api(struct iwl_mvm *mvm) @@ -1522,6 +1516,19 @@ static inline bool iwl_mvm_is_ctdp_supported(struct iwl_mvm *mvm) IWL_UCODE_TLV_CAPA_CTDP_SUPPORT); } +static inline bool iwl_mvm_has_new_tx_csum(struct iwl_mvm *mvm) +{ + if (mvm->trans->trans_cfg->device_family < IWL_DEVICE_FAMILY_BZ) + return false; + + if (mvm->trans->trans_cfg->device_family == IWL_DEVICE_FAMILY_BZ && + CSR_HW_REV_TYPE(mvm->trans->hw_rev) == IWL_CFG_MAC_TYPE_GL && + mvm->trans->hw_rev_step <= SILICON_B_STEP) + return false; + + return true; +} + extern const u8 iwl_mvm_ac_to_tx_fifo[]; extern const u8 iwl_mvm_ac_to_gen2_tx_fifo[]; @@ -1781,16 +1788,15 @@ int iwl_mvm_mac_ctxt_remove(struct iwl_mvm *mvm, struct ieee80211_vif *vif); int iwl_mvm_mac_ctxt_beacon_changed(struct iwl_mvm *mvm, struct ieee80211_vif *vif, struct ieee80211_bss_conf *link_conf); -int iwl_mvm_mac_ctxt_send_beacon(struct iwl_mvm *mvm, - struct ieee80211_vif *vif, - struct sk_buff *beacon, - struct ieee80211_bss_conf *link_conf); int iwl_mvm_mac_ctxt_send_beacon_cmd(struct iwl_mvm *mvm, struct sk_buff *beacon, void *data, int len); u8 iwl_mvm_mac_ctxt_get_beacon_rate(struct iwl_mvm *mvm, struct ieee80211_tx_info *info, struct ieee80211_vif *vif); +u8 iwl_mvm_mac_ctxt_get_lowest_rate(struct iwl_mvm *mvm, + struct ieee80211_tx_info *info, + struct ieee80211_vif *vif); u16 iwl_mvm_mac_ctxt_get_beacon_flags(const struct iwl_fw *fw, u8 rate_idx); void iwl_mvm_mac_ctxt_set_tim(struct iwl_mvm *mvm, @@ -2306,6 +2312,7 @@ void iwl_mvm_event_frame_timeout_callback(struct iwl_mvm *mvm, struct ieee80211_vif *vif, const struct ieee80211_sta *sta, u16 tid); +void iwl_mvm_mei_scan_filter_init(struct iwl_mei_scan_filter *mei_scan_filter); void iwl_mvm_ptp_init(struct iwl_mvm *mvm); void iwl_mvm_ptp_remove(struct iwl_mvm *mvm); @@ -2334,6 +2341,11 @@ void iwl_mvm_sec_key_remove_ap(struct iwl_mvm *mvm, struct ieee80211_vif *vif, struct iwl_mvm_vif_link_info *link, unsigned int link_id); +int iwl_mvm_mld_update_sta_keys(struct iwl_mvm *mvm, + struct ieee80211_vif *vif, + struct ieee80211_sta *sta, + u32 old_sta_mask, + u32 new_sta_mask); int iwl_rfi_send_config_cmd(struct iwl_mvm *mvm, struct iwl_rfi_lut_entry *rfi_table); @@ -2512,6 +2524,22 @@ static inline void iwl_mvm_mei_set_sw_rfkill_state(struct iwl_mvm *mvm) sw_rfkill); } +static inline bool iwl_mvm_mei_filter_scan(struct iwl_mvm *mvm, + struct sk_buff *skb) +{ + struct ieee80211_mgmt *mgmt = (void *)skb->data; + + if (mvm->mei_scan_filter.is_mei_limited_scan && + (ieee80211_is_probe_resp(mgmt->frame_control) || + ieee80211_is_beacon(mgmt->frame_control))) { + skb_queue_tail(&mvm->mei_scan_filter.scan_res, skb); + schedule_work(&mvm->mei_scan_filter.scan_work); + return true; + } + + return false; +} + void iwl_mvm_send_roaming_forbidden_event(struct iwl_mvm *mvm, struct ieee80211_vif *vif, bool forbidden); diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/ops.c b/drivers/net/wireless/intel/iwlwifi/mvm/ops.c index 56a4e9d6ae33..32625bfacaae 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/ops.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/ops.c @@ -472,8 +472,8 @@ static const struct iwl_hcmd_names iwl_mvm_legacy_names[] = { HCMD_NAME(SCAN_OFFLOAD_PROFILES_QUERY_CMD), HCMD_NAME(BT_COEX_UPDATE_REDUCED_TXP), HCMD_NAME(BT_COEX_CI), - HCMD_NAME(WNM_80211V_TIMING_MEASUREMENT_CONFIRM_NOTIFICATION), HCMD_NAME(WNM_80211V_TIMING_MEASUREMENT_NOTIFICATION), + HCMD_NAME(WNM_80211V_TIMING_MEASUREMENT_CONFIRM_NOTIFICATION), HCMD_NAME(PHY_CONFIGURATION_CMD), HCMD_NAME(CALIB_RES_NOTIF_PHY_DB), HCMD_NAME(PHY_DB_CMD), @@ -1020,10 +1020,14 @@ static void iwl_mvm_me_conn_status(void *priv, const struct iwl_mei_conn_info *c kfree_rcu(prev_conn_info, rcu_head); } -static void iwl_mvm_mei_rfkill(void *priv, bool blocked) +static void iwl_mvm_mei_rfkill(void *priv, bool blocked, + bool csme_taking_ownership) { struct iwl_mvm *mvm = priv; + if (blocked && !csme_taking_ownership) + return; + mvm->mei_rfkill_blocked = blocked; if (!mvm->hw_registered) return; @@ -1367,12 +1371,16 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg, else memset(&mvm->rx_stats, 0, sizeof(struct mvm_statistics_rx)); + iwl_mvm_ftm_initiator_smooth_config(mvm); + iwl_mvm_init_time_sync(&mvm->time_sync); mvm->debugfs_dir = dbgfs_dir; mvm->mei_registered = !iwl_mei_register(mvm, &mei_ops); + iwl_mvm_mei_scan_filter_init(&mvm->mei_scan_filter); + if (iwl_mvm_start_get_nvm(mvm)) { /* * Getting NVM failed while CSME is the owner, but we are diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/rs-fw.c b/drivers/net/wireless/intel/iwlwifi/mvm/rs-fw.c index c8ba2fe3e4a2..c3a00bfbeef2 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/rs-fw.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/rs-fw.c @@ -61,12 +61,14 @@ static u8 rs_fw_sgi_cw_support(struct ieee80211_link_sta *link_sta) } static u16 rs_fw_get_config_flags(struct iwl_mvm *mvm, + struct ieee80211_vif *vif, struct ieee80211_link_sta *link_sta, struct ieee80211_supported_band *sband) { struct ieee80211_sta_ht_cap *ht_cap = &link_sta->ht_cap; struct ieee80211_sta_vht_cap *vht_cap = &link_sta->vht_cap; struct ieee80211_sta_he_cap *he_cap = &link_sta->he_cap; + const struct ieee80211_sta_he_cap *sband_he_cap; bool vht_ena = vht_cap->vht_supported; u16 flags = 0; @@ -92,17 +94,19 @@ static u16 rs_fw_get_config_flags(struct iwl_mvm *mvm, IEEE80211_HE_PHY_CAP1_LDPC_CODING_IN_PAYLOAD)) flags |= IWL_TLC_MNG_CFG_FLAGS_LDPC_MSK; - if (sband->iftype_data && sband->iftype_data->he_cap.has_he && - !(sband->iftype_data->he_cap.he_cap_elem.phy_cap_info[1] & - IEEE80211_HE_PHY_CAP1_LDPC_CODING_IN_PAYLOAD)) + sband_he_cap = ieee80211_get_he_iftype_cap(sband, + ieee80211_vif_type_p2p(vif)); + if (sband_he_cap && + !(sband_he_cap->he_cap_elem.phy_cap_info[1] & + IEEE80211_HE_PHY_CAP1_LDPC_CODING_IN_PAYLOAD)) flags &= ~IWL_TLC_MNG_CFG_FLAGS_LDPC_MSK; if (he_cap->has_he && (he_cap->he_cap_elem.phy_cap_info[3] & IEEE80211_HE_PHY_CAP3_DCM_MAX_CONST_RX_MASK && - sband->iftype_data && - sband->iftype_data->he_cap.he_cap_elem.phy_cap_info[3] & - IEEE80211_HE_PHY_CAP3_DCM_MAX_CONST_TX_MASK)) + sband_he_cap && + sband_he_cap->he_cap_elem.phy_cap_info[3] & + IEEE80211_HE_PHY_CAP3_DCM_MAX_CONST_TX_MASK)) flags |= IWL_TLC_MNG_CFG_FLAGS_HE_DCM_NSS_1_MSK; return flags; @@ -283,7 +287,8 @@ rs_fw_rs_mcs2eht_mcs(enum IWL_TLC_MCS_PER_BW bw, } static void -rs_fw_eht_set_enabled_rates(const struct ieee80211_link_sta *link_sta, +rs_fw_eht_set_enabled_rates(struct ieee80211_vif *vif, + const struct ieee80211_link_sta *link_sta, struct ieee80211_supported_band *sband, struct iwl_tlc_config_cmd_v4 *cmd) { @@ -299,7 +304,8 @@ rs_fw_eht_set_enabled_rates(const struct ieee80211_link_sta *link_sta, struct ieee80211_eht_mcs_nss_supp_20mhz_only mcs_tx_20; /* peer is 20Mhz only */ - if (!(link_sta->he_cap.he_cap_elem.phy_cap_info[0] & + if (vif->type == NL80211_IFTYPE_AP && + !(link_sta->he_cap.he_cap_elem.phy_cap_info[0] & IEEE80211_HE_PHY_CAP0_CHANNEL_WIDTH_SET_MASK_ALL)) { mcs_rx_20 = eht_rx_mcs->only_20mhz; } else { @@ -361,7 +367,8 @@ rs_fw_eht_set_enabled_rates(const struct ieee80211_link_sta *link_sta, sizeof(cmd->ht_rates[IWL_TLC_NSS_2])); } -static void rs_fw_set_supp_rates(struct ieee80211_link_sta *link_sta, +static void rs_fw_set_supp_rates(struct ieee80211_vif *vif, + struct ieee80211_link_sta *link_sta, struct ieee80211_supported_band *sband, struct iwl_tlc_config_cmd_v4 *cmd) { @@ -383,7 +390,7 @@ static void rs_fw_set_supp_rates(struct ieee80211_link_sta *link_sta, /* HT/VHT rates */ if (link_sta->eht_cap.has_eht) { cmd->mode = IWL_TLC_MNG_MODE_EHT; - rs_fw_eht_set_enabled_rates(link_sta, sband, cmd); + rs_fw_eht_set_enabled_rates(vif, link_sta, sband, cmd); } else if (he_cap->has_he) { cmd->mode = IWL_TLC_MNG_MODE_HE; rs_fw_he_set_enabled_rates(link_sta, sband, cmd); @@ -557,10 +564,12 @@ u16 rs_fw_get_max_amsdu_len(struct ieee80211_sta *sta, return 0; } -void rs_fw_rate_init(struct iwl_mvm *mvm, struct ieee80211_sta *sta, - struct ieee80211_bss_conf *link_conf, - struct ieee80211_link_sta *link_sta, - enum nl80211_band band, bool update) +void iwl_mvm_rs_fw_rate_init(struct iwl_mvm *mvm, + struct ieee80211_vif *vif, + struct ieee80211_sta *sta, + struct ieee80211_bss_conf *link_conf, + struct ieee80211_link_sta *link_sta, + enum nl80211_band band) { struct ieee80211_hw *hw = mvm->hw; struct iwl_mvm_sta *mvmsta = iwl_mvm_sta_from_mac80211(sta); @@ -570,10 +579,9 @@ void rs_fw_rate_init(struct iwl_mvm *mvm, struct ieee80211_sta *sta, struct iwl_mvm_link_sta *mvm_link_sta; struct iwl_lq_sta_rs_fw *lq_sta; struct iwl_tlc_config_cmd_v4 cfg_cmd = { - .max_ch_width = update ? - rs_fw_bw_from_sta_bw(link_sta) : - IWL_TLC_MNG_CH_WIDTH_20MHZ, - .flags = cpu_to_le16(rs_fw_get_config_flags(mvm, link_sta, + .max_ch_width = mvmsta->authorized ? + rs_fw_bw_from_sta_bw(link_sta) : IWL_TLC_MNG_CH_WIDTH_20MHZ, + .flags = cpu_to_le16(rs_fw_get_config_flags(mvm, vif, link_sta, sband)), .chains = rs_fw_set_active_chains(iwl_mvm_get_valid_tx_ant(mvm)), .sgi_ch_width_supp = rs_fw_sgi_cw_support(link_sta), @@ -601,7 +609,7 @@ void rs_fw_rate_init(struct iwl_mvm *mvm, struct ieee80211_sta *sta, #ifdef CONFIG_IWLWIFI_DEBUGFS iwl_mvm_reset_frame_stats(mvm); #endif - rs_fw_set_supp_rates(link_sta, sband, &cfg_cmd); + rs_fw_set_supp_rates(vif, link_sta, sband, &cfg_cmd); /* * since TLC offload works with one mode we can assume @@ -671,6 +679,26 @@ int rs_fw_tx_protection(struct iwl_mvm *mvm, struct iwl_mvm_sta *mvmsta, return 0; } +void iwl_mvm_rs_add_sta_link(struct iwl_mvm *mvm, + struct iwl_mvm_link_sta *link_sta) +{ + struct iwl_lq_sta_rs_fw *lq_sta; + + lq_sta = &link_sta->lq_sta.rs_fw; + + lq_sta->pers.drv = mvm; + lq_sta->pers.sta_id = link_sta->sta_id; + lq_sta->pers.chains = 0; + memset(lq_sta->pers.chain_signal, 0, + sizeof(lq_sta->pers.chain_signal)); + lq_sta->pers.last_rssi = S8_MIN; + lq_sta->last_rate_n_flags = 0; + +#ifdef CONFIG_MAC80211_DEBUGFS + lq_sta->pers.dbg_fixed_rate = 0; +#endif +} + void iwl_mvm_rs_add_sta(struct iwl_mvm *mvm, struct iwl_mvm_sta *mvmsta) { unsigned int link_id; @@ -678,25 +706,12 @@ void iwl_mvm_rs_add_sta(struct iwl_mvm *mvm, struct iwl_mvm_sta *mvmsta) IWL_DEBUG_RATE(mvm, "create station rate scale window\n"); for (link_id = 0; link_id < ARRAY_SIZE(mvmsta->link); link_id++) { - struct iwl_lq_sta_rs_fw *lq_sta; struct iwl_mvm_link_sta *link = rcu_dereference_protected(mvmsta->link[link_id], lockdep_is_held(&mvm->mutex)); if (!link) continue; - lq_sta = &link->lq_sta.rs_fw; - - lq_sta->pers.drv = mvm; - lq_sta->pers.sta_id = link->sta_id; - lq_sta->pers.chains = 0; - memset(lq_sta->pers.chain_signal, 0, - sizeof(lq_sta->pers.chain_signal)); - lq_sta->pers.last_rssi = S8_MIN; - lq_sta->last_rate_n_flags = 0; - -#ifdef CONFIG_MAC80211_DEBUGFS - lq_sta->pers.dbg_fixed_rate = 0; -#endif + iwl_mvm_rs_add_sta_link(mvm, link); } } diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/rs.c b/drivers/net/wireless/intel/iwlwifi/mvm/rs.c index ab82965bc0f4..a4c1e3bf4ff1 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/rs.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/rs.c @@ -3015,9 +3015,9 @@ static void rs_drv_rate_update(void *mvm_r, for (tid = 0; tid < IWL_MAX_TID_COUNT; tid++) ieee80211_stop_tx_ba_session(sta, tid); - iwl_mvm_rs_rate_init(mvm, sta, + iwl_mvm_rs_rate_init(mvm, mvmsta->vif, sta, &mvmsta->vif->bss_conf, &sta->deflink, - sband->band, true); + sband->band); } static void __iwl_mvm_rs_tx_status(struct iwl_mvm *mvm, @@ -4101,13 +4101,16 @@ static const struct rate_control_ops rs_mvm_ops_drv = { .capa = RATE_CTRL_CAPA_VHT_EXT_NSS_BW, }; -void iwl_mvm_rs_rate_init(struct iwl_mvm *mvm, struct ieee80211_sta *sta, +void iwl_mvm_rs_rate_init(struct iwl_mvm *mvm, + struct ieee80211_vif *vif, + struct ieee80211_sta *sta, struct ieee80211_bss_conf *link_conf, struct ieee80211_link_sta *link_sta, - enum nl80211_band band, bool update) + enum nl80211_band band) { if (iwl_mvm_has_tlc_offload(mvm)) { - rs_fw_rate_init(mvm, sta, link_conf, link_sta, band, update); + iwl_mvm_rs_fw_rate_init(mvm, vif, sta, link_conf, + link_sta, band); } else { struct iwl_mvm_sta *mvmsta = iwl_mvm_sta_from_mac80211(sta); diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/rs.h b/drivers/net/wireless/intel/iwlwifi/mvm/rs.h index f99603b0f693..1ca375a5cf6b 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/rs.h +++ b/drivers/net/wireless/intel/iwlwifi/mvm/rs.h @@ -394,10 +394,12 @@ struct iwl_lq_sta { ((_c) << RS_DRV_DATA_LQ_COLOR_POS))) /* Initialize station's rate scaling information after adding station */ -void iwl_mvm_rs_rate_init(struct iwl_mvm *mvm, struct ieee80211_sta *sta, +void iwl_mvm_rs_rate_init(struct iwl_mvm *mvm, + struct ieee80211_vif *vif, + struct ieee80211_sta *sta, struct ieee80211_bss_conf *link_conf, struct ieee80211_link_sta *link_sta, - enum nl80211_band band, bool update); + enum nl80211_band band); /* Notify RS about Tx status */ void iwl_mvm_rs_tx_status(struct iwl_mvm *mvm, struct ieee80211_sta *sta, @@ -432,11 +434,18 @@ int iwl_mvm_tx_protection(struct iwl_mvm *mvm, struct iwl_mvm_sta *mvmsta, void iwl_mvm_reset_frame_stats(struct iwl_mvm *mvm); #endif +struct iwl_mvm_link_sta; + void iwl_mvm_rs_add_sta(struct iwl_mvm *mvm, struct iwl_mvm_sta *mvmsta); -void rs_fw_rate_init(struct iwl_mvm *mvm, struct ieee80211_sta *sta, - struct ieee80211_bss_conf *link_conf, - struct ieee80211_link_sta *link_sta, - enum nl80211_band band, bool update); +void iwl_mvm_rs_add_sta_link(struct iwl_mvm *mvm, + struct iwl_mvm_link_sta *link_sta); + +void iwl_mvm_rs_fw_rate_init(struct iwl_mvm *mvm, + struct ieee80211_vif *vif, + struct ieee80211_sta *sta, + struct ieee80211_bss_conf *link_conf, + struct ieee80211_link_sta *link_sta, + enum nl80211_band band); int rs_fw_tx_protection(struct iwl_mvm *mvm, struct iwl_mvm_sta *mvmsta, bool enable); void iwl_mvm_tlc_update_notif(struct iwl_mvm *mvm, diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/rx.c b/drivers/net/wireless/intel/iwlwifi/mvm/rx.c index e08dca8d2baa..b38b24246675 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/rx.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/rx.c @@ -383,9 +383,10 @@ void iwl_mvm_rx_rx_mpdu(struct iwl_mvm *mvm, struct napi_struct *napi, * Don't even try to decrypt a MCAST frame that was received * before the managed vif is authorized, we'd fail anyway. */ - if (vif->type == NL80211_IFTYPE_STATION && + if (is_multicast_ether_addr(hdr->addr1) && + vif->type == NL80211_IFTYPE_STATION && !mvmvif->authorized && - is_multicast_ether_addr(hdr->addr1)) { + ieee80211_has_protected(hdr->frame_control)) { IWL_DEBUG_DROP(mvm, "MCAST before the vif is authorized\n"); kfree_skb(skb); rcu_read_unlock(); diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c b/drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c index 5d803e537b00..e1d02c260e69 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c @@ -106,28 +106,12 @@ static int iwl_mvm_create_skb(struct iwl_mvm *mvm, struct sk_buff *skb, /* * For non monitor interface strip the bytes the RADA might not have - * removed. As monitor interface cannot exist with other interfaces - * this removal is safe. + * removed (it might be disabled, e.g. for mgmt frames). As a monitor + * interface cannot exist with other interfaces, this removal is safe + * and sufficient, in monitor mode there's no decryption being done. */ - if (mic_crc_len && !ieee80211_hw_check(mvm->hw, RX_INCLUDES_FCS)) { - u32 pkt_flags = le32_to_cpu(pkt->len_n_flags); - - /* - * If RADA was not enabled then decryption was not performed so - * the MIC cannot be removed. - */ - if (!(pkt_flags & FH_RSCSR_RADA_EN)) { - if (WARN_ON(crypt_len > mic_crc_len)) - return -EINVAL; - - mic_crc_len -= crypt_len; - } - - if (WARN_ON(mic_crc_len > len)) - return -EINVAL; - + if (len > mic_crc_len && !ieee80211_hw_check(mvm->hw, RX_INCLUDES_FCS)) len -= mic_crc_len; - } /* If frame is small enough to fit in skb->head, pull it completely. * If not, only pull ieee80211_hdr (including crypto if present, and @@ -172,8 +156,7 @@ static int iwl_mvm_create_skb(struct iwl_mvm *mvm, struct sk_buff *skb, * Starting from Bz hardware, it calculates starting directly after * the MAC header, so that matches mac80211's expectation. */ - if (skb->ip_summed == CHECKSUM_COMPLETE && - mvm->trans->trans_cfg->device_family < IWL_DEVICE_FAMILY_BZ) { + if (skb->ip_summed == CHECKSUM_COMPLETE) { struct { u8 hdr[6]; __be16 type; @@ -188,7 +171,7 @@ static int iwl_mvm_create_skb(struct iwl_mvm *mvm, struct sk_buff *skb, shdr->type != htons(ETH_P_PAE) && shdr->type != htons(ETH_P_TDLS)))) skb->ip_summed = CHECKSUM_NONE; - else + else if (mvm->trans->trans_cfg->device_family < IWL_DEVICE_FAMILY_BZ) /* mac80211 assumes full CSUM including SNAP header */ skb_postpush_rcsum(skb, shdr, sizeof(*shdr)); } @@ -412,9 +395,7 @@ static int iwl_mvm_rx_crypto(struct iwl_mvm *mvm, struct ieee80211_sta *sta, if (!(status & IWL_RX_MPDU_STATUS_MIC_OK)) return -1; - stats->flag |= RX_FLAG_DECRYPTED; - if (pkt_flags & FH_RSCSR_RADA_EN) - stats->flag |= RX_FLAG_MIC_STRIPPED; + stats->flag |= RX_FLAG_DECRYPTED | RX_FLAG_MIC_STRIPPED; *crypt_len = IEEE80211_CCMP_HDR_LEN; return 0; case IWL_RX_MPDU_STATUS_SEC_TKIP: @@ -706,7 +687,7 @@ void iwl_mvm_reorder_timer_expired(struct timer_list *t) if (expired) { struct ieee80211_sta *sta; struct iwl_mvm_sta *mvmsta; - u8 sta_id = baid_data->sta_id; + u8 sta_id = ffs(baid_data->sta_mask) - 1; rcu_read_lock(); sta = rcu_dereference(buf->mvm->fw_id_to_mac_id[sta_id]); @@ -741,6 +722,7 @@ static void iwl_mvm_del_ba(struct iwl_mvm *mvm, int queue, struct ieee80211_sta *sta; struct iwl_mvm_reorder_buffer *reorder_buf; u8 baid = data->baid; + u32 sta_id; if (WARN_ONCE(baid >= IWL_MAX_BAID, "invalid BAID: %x\n", baid)) return; @@ -751,7 +733,9 @@ static void iwl_mvm_del_ba(struct iwl_mvm *mvm, int queue, if (WARN_ON_ONCE(!ba_data)) goto out; - sta = rcu_dereference(mvm->fw_id_to_mac_id[ba_data->sta_id]); + /* pick any STA ID to find the pointer */ + sta_id = ffs(ba_data->sta_mask) - 1; + sta = rcu_dereference(mvm->fw_id_to_mac_id[sta_id]); if (WARN_ON_ONCE(IS_ERR_OR_NULL(sta))) goto out; @@ -778,6 +762,7 @@ static void iwl_mvm_release_frames_from_notif(struct iwl_mvm *mvm, struct ieee80211_sta *sta; struct iwl_mvm_reorder_buffer *reorder_buf; struct iwl_mvm_baid_data *ba_data; + u32 sta_id; IWL_DEBUG_HT(mvm, "Frame release notification for BAID %u, NSSN %d\n", baid, nssn); @@ -795,7 +780,9 @@ static void iwl_mvm_release_frames_from_notif(struct iwl_mvm *mvm, goto out; } - sta = rcu_dereference(mvm->fw_id_to_mac_id[ba_data->sta_id]); + /* pick any STA ID to find the pointer */ + sta_id = ffs(ba_data->sta_mask) - 1; + sta = rcu_dereference(mvm->fw_id_to_mac_id[sta_id]); if (WARN_ON_ONCE(IS_ERR_OR_NULL(sta))) goto out; @@ -936,7 +923,6 @@ static bool iwl_mvm_reorder(struct iwl_mvm *mvm, { struct ieee80211_rx_status *rx_status = IEEE80211_SKB_RXCB(skb); struct ieee80211_hdr *hdr = (void *)skb_mac_header(skb); - struct iwl_mvm_sta *mvm_sta; struct iwl_mvm_baid_data *baid_data; struct iwl_mvm_reorder_buffer *buffer; struct sk_buff *tail; @@ -948,6 +934,7 @@ static bool iwl_mvm_reorder(struct iwl_mvm *mvm, u8 sub_frame_idx = desc->amsdu_info & IWL_RX_MPDU_AMSDU_SUBFRAME_IDX_MASK; struct iwl_mvm_reorder_buf_entry *entries; + u32 sta_mask; int index; u16 nssn, sn; u8 baid; @@ -970,8 +957,6 @@ static bool iwl_mvm_reorder(struct iwl_mvm *mvm, "Got valid BAID without a valid station assigned\n")) return false; - mvm_sta = iwl_mvm_sta_from_mac80211(sta); - /* not a data packet or a bar */ if (!ieee80211_is_back_req(hdr->frame_control) && (!ieee80211_is_data_qos(hdr->frame_control) || @@ -989,11 +974,14 @@ static bool iwl_mvm_reorder(struct iwl_mvm *mvm, return false; } + rcu_read_lock(); + sta_mask = iwl_mvm_sta_fw_id_mask(mvm, sta, -1); + rcu_read_unlock(); + if (WARN(tid != baid_data->tid || - mvm_sta->deflink.sta_id != baid_data->sta_id, - "baid 0x%x is mapped to sta:%d tid:%d, but was received for sta:%d tid:%d\n", - baid, baid_data->sta_id, baid_data->tid, mvm_sta->deflink.sta_id, - tid)) + !(sta_mask & baid_data->sta_mask), + "baid 0x%x is mapped to sta_mask:0x%x tid:%d, but was received for sta_mask:0x%x tid:%d\n", + baid, baid_data->sta_mask, baid_data->tid, sta_mask, tid)) return false; nssn = reorder & IWL_RX_MPDU_REORDER_NSSN_MASK; @@ -2600,10 +2588,10 @@ void iwl_mvm_rx_mpdu_mq(struct iwl_mvm *mvm, struct napi_struct *napi, } if (!iwl_mvm_reorder(mvm, napi, queue, sta, skb, desc) && - likely(!iwl_mvm_time_sync_frame(mvm, skb, hdr->addr2))) + likely(!iwl_mvm_time_sync_frame(mvm, skb, hdr->addr2)) && + likely(!iwl_mvm_mei_filter_scan(mvm, skb))) iwl_mvm_pass_packet_to_mac80211(mvm, napi, skb, queue, sta, link_sta); - out: rcu_read_unlock(); } @@ -2777,9 +2765,10 @@ void iwl_mvm_rx_bar_frame_release(struct iwl_mvm *mvm, struct napi_struct *napi, goto out; } - if (WARN(tid != baid_data->tid || sta_id != baid_data->sta_id, - "baid 0x%x is mapped to sta:%d tid:%d, but BAR release received for sta:%d tid:%d\n", - baid, baid_data->sta_id, baid_data->tid, sta_id, + if (WARN(tid != baid_data->tid || sta_id > IWL_MVM_STATION_COUNT_MAX || + !(baid_data->sta_mask & BIT(sta_id)), + "baid 0x%x is mapped to sta_mask:0x%x tid:%d, but BAR release received for sta:%d tid:%d\n", + baid, baid_data->sta_mask, baid_data->tid, sta_id, tid)) goto out; diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/scan.c b/drivers/net/wireless/intel/iwlwifi/mvm/scan.c index 07045092c717..175615755d9d 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/scan.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/scan.c @@ -45,6 +45,9 @@ /* minimal number of 2GHz and 5GHz channels in the regular scan request */ #define IWL_MVM_6GHZ_PASSIVE_SCAN_MIN_CHANS 4 +/* Number of iterations on the channel for mei filtered scan */ +#define IWL_MEI_SCAN_NUM_ITER 5U + struct iwl_mvm_scan_timing_params { u32 suspend_time; u32 max_out_time; @@ -98,6 +101,7 @@ struct iwl_mvm_scan_params { bool scan_6ghz; bool enable_6ghz_passive; bool respect_p2p_go, respect_p2p_go_hb; + u8 bssid[ETH_ALEN] __aligned(2); }; static inline void *iwl_mvm_get_scan_req_umac_data(struct iwl_mvm *mvm) @@ -240,8 +244,9 @@ iwl_mvm_scan_type _iwl_mvm_get_scan_type(struct iwl_mvm *mvm, * set all scan requests as fast-balance scan */ if (vif && vif->type == NL80211_IFTYPE_STATION && - vif->bss_conf.dtim_period < 220 && - data.is_dcm_with_p2p_go) + data.is_dcm_with_p2p_go && + ((vif->bss_conf.beacon_int * + vif->bss_conf.dtim_period) < 220)) return IWL_SCAN_TYPE_FAST_BALANCE; } @@ -759,7 +764,7 @@ iwl_mvm_build_scan_probe(struct iwl_mvm *mvm, struct ieee80211_vif *vif, frame->frame_control = cpu_to_le16(IEEE80211_STYPE_PROBE_REQ); eth_broadcast_addr(frame->da); - eth_broadcast_addr(frame->bssid); + ether_addr_copy(frame->bssid, params->bssid); frame->seq_ctrl = 0; pos = frame->u.probe_req.variable; @@ -2081,6 +2086,11 @@ static u8 iwl_mvm_scan_umac_flags2(struct iwl_mvm *mvm, IWL_UMAC_SCAN_GEN_PARAMS_FLAGS2_RESPECT_P2P_GO_HB; } + if (params->scan_6ghz && + fw_has_capa(&mvm->fw->ucode_capa, + IWL_UCODE_TLV_CAPA_SCAN_DONT_TOGGLE_ANT)) + flags |= IWL_UMAC_SCAN_GEN_PARAMS_FLAGS2_DONT_TOGGLE_ANT; + return flags; } @@ -2292,7 +2302,7 @@ iwl_mvm_scan_umac_fill_general_p_v11(struct iwl_mvm *mvm, iwl_mvm_scan_umac_dwell_v11(mvm, gp, params); - IWL_DEBUG_SCAN(mvm, "Gerenal: flags=0x%x, flags2=0x%x\n", + IWL_DEBUG_SCAN(mvm, "General: flags=0x%x, flags2=0x%x\n", gen_flags, gen_flags2); gp->flags = cpu_to_le16(gen_flags); @@ -2617,6 +2627,89 @@ static const struct iwl_scan_umac_handler iwl_scan_umac_handlers[] = { IWL_SCAN_UMAC_HANDLER(12), }; +static void iwl_mvm_mei_scan_work(struct work_struct *wk) +{ + struct iwl_mei_scan_filter *scan_filter = + container_of(wk, struct iwl_mei_scan_filter, scan_work); + struct iwl_mvm *mvm = + container_of(scan_filter, struct iwl_mvm, mei_scan_filter); + struct iwl_mvm_csme_conn_info *info; + struct sk_buff *skb; + u8 bssid[ETH_ALEN]; + + mutex_lock(&mvm->mutex); + info = iwl_mvm_get_csme_conn_info(mvm); + memcpy(bssid, info->conn_info.bssid, ETH_ALEN); + mutex_unlock(&mvm->mutex); + + while ((skb = skb_dequeue(&scan_filter->scan_res))) { + struct ieee80211_mgmt *mgmt = (void *)skb->data; + + if (!memcmp(mgmt->bssid, bssid, ETH_ALEN)) + ieee80211_rx_irqsafe(mvm->hw, skb); + else + kfree_skb(skb); + } +} + +void iwl_mvm_mei_scan_filter_init(struct iwl_mei_scan_filter *mei_scan_filter) +{ + skb_queue_head_init(&mei_scan_filter->scan_res); + INIT_WORK(&mei_scan_filter->scan_work, iwl_mvm_mei_scan_work); +} + +/* In case CSME is connected and has link protection set, this function will + * override the scan request to scan only the associated channel and only for + * the associated SSID. + */ +static void iwl_mvm_mei_limited_scan(struct iwl_mvm *mvm, + struct iwl_mvm_scan_params *params) +{ + struct iwl_mvm_csme_conn_info *info = iwl_mvm_get_csme_conn_info(mvm); + struct iwl_mei_conn_info *conn_info; + struct ieee80211_channel *chan; + int scan_iters, i; + + if (!info) { + IWL_DEBUG_SCAN(mvm, "mei_limited_scan: no connection info\n"); + return; + } + + conn_info = &info->conn_info; + if (!info->conn_info.lp_state || !info->conn_info.ssid_len) + return; + + if (!params->n_channels || !params->n_ssids) + return; + + mvm->mei_scan_filter.is_mei_limited_scan = true; + + chan = ieee80211_get_channel(mvm->hw->wiphy, + ieee80211_channel_to_frequency(conn_info->channel, + conn_info->band)); + if (!chan) { + IWL_DEBUG_SCAN(mvm, + "Failed to get CSME channel (chan=%u band=%u)\n", + conn_info->channel, conn_info->band); + return; + } + + /* The mei filtered scan must find the AP, otherwise CSME will + * take the NIC ownership. Add several iterations on the channel to + * make the scan more robust. + */ + scan_iters = min(IWL_MEI_SCAN_NUM_ITER, params->n_channels); + params->n_channels = scan_iters; + for (i = 0; i < scan_iters; i++) + params->channels[i] = chan; + + IWL_DEBUG_SCAN(mvm, "Mei scan: num iterations=%u\n", scan_iters); + + params->n_ssids = 1; + params->ssids[0].ssid_len = conn_info->ssid_len; + memcpy(params->ssids[0].ssid, conn_info->ssid, conn_info->ssid_len); +} + static int iwl_mvm_build_scan_cmd(struct iwl_mvm *mvm, struct ieee80211_vif *vif, struct iwl_host_cmd *hcmd, @@ -2629,6 +2722,8 @@ static int iwl_mvm_build_scan_cmd(struct iwl_mvm *mvm, lockdep_assert_held(&mvm->mutex); memset(mvm->scan_cmd, 0, mvm->scan_cmd_size); + iwl_mvm_mei_limited_scan(mvm, params); + if (!fw_has_capa(&mvm->fw->ucode_capa, IWL_UCODE_TLV_CAPA_UMAC_SCAN)) { hcmd->id = SCAN_OFFLOAD_REQUEST_CMD; @@ -2795,6 +2890,7 @@ int iwl_mvm_reg_scan_start(struct iwl_mvm *mvm, struct ieee80211_vif *vif, params.pass_all = true; params.n_match_sets = 0; params.match_sets = NULL; + ether_addr_copy(params.bssid, req->bssid); params.scan_plans = &scan_plan; params.n_scan_plans = 1; @@ -2888,6 +2984,7 @@ int iwl_mvm_sched_scan_start(struct iwl_mvm *mvm, params.pass_all = iwl_mvm_scan_pass_all(mvm, req); params.n_match_sets = req->n_match_sets; params.match_sets = req->match_sets; + eth_broadcast_addr(params.bssid); if (!req->n_scan_plans) return -EINVAL; @@ -2983,6 +3080,8 @@ void iwl_mvm_rx_umac_scan_complete_notif(struct iwl_mvm *mvm, u32 uid = __le32_to_cpu(notif->uid); bool aborted = (notif->status == IWL_SCAN_OFFLOAD_ABORTED); + mvm->mei_scan_filter.is_mei_limited_scan = false; + if (WARN_ON(!(mvm->scan_uid_status[uid] & mvm->scan_status))) return; diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/sf.c b/drivers/net/wireless/intel/iwlwifi/mvm/sf.c index 7c5f41e40e7a..98f330fcf678 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/sf.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/sf.c @@ -8,7 +8,7 @@ /* For counting bound interfaces */ struct iwl_mvm_active_iface_iterator_data { struct ieee80211_vif *ignore_vif; - u8 sta_vif_ap_sta_id; + struct ieee80211_sta *sta_vif_ap_sta; enum iwl_sf_state sta_vif_state; u32 num_active_macs; }; @@ -30,7 +30,7 @@ static void iwl_mvm_bound_iface_iterator(void *_data, u8 *mac, data->num_active_macs++; if (vif->type == NL80211_IFTYPE_STATION) { - data->sta_vif_ap_sta_id = mvmvif->deflink.ap_sta_id; + data->sta_vif_ap_sta = mvmvif->ap_sta; if (vif->cfg.assoc) data->sta_vif_state = SF_FULL_ON; else @@ -172,13 +172,12 @@ static void iwl_mvm_fill_sf_command(struct iwl_mvm *mvm, } } -static int iwl_mvm_sf_config(struct iwl_mvm *mvm, u8 sta_id, +static int iwl_mvm_sf_config(struct iwl_mvm *mvm, struct ieee80211_sta *sta, enum iwl_sf_state new_state) { struct iwl_sf_cfg_cmd sf_cmd = { .state = cpu_to_le32(new_state), }; - struct ieee80211_sta *sta; int ret = 0; if (mvm->cfg->disable_dummy_notification) @@ -196,20 +195,12 @@ static int iwl_mvm_sf_config(struct iwl_mvm *mvm, u8 sta_id, iwl_mvm_fill_sf_command(mvm, &sf_cmd, NULL); break; case SF_FULL_ON: - if (sta_id == IWL_MVM_INVALID_STA) { + if (!sta) { IWL_ERR(mvm, "No station: Cannot switch SF to FULL_ON\n"); return -EINVAL; } - rcu_read_lock(); - sta = rcu_dereference(mvm->fw_id_to_mac_id[sta_id]); - if (IS_ERR_OR_NULL(sta)) { - IWL_ERR(mvm, "Invalid station id\n"); - rcu_read_unlock(); - return -EINVAL; - } iwl_mvm_fill_sf_command(mvm, &sf_cmd, sta); - rcu_read_unlock(); break; case SF_INIT_OFF: iwl_mvm_fill_sf_command(mvm, &sf_cmd, NULL); @@ -237,13 +228,12 @@ int iwl_mvm_sf_update(struct iwl_mvm *mvm, struct ieee80211_vif *changed_vif, bool remove_vif) { enum iwl_sf_state new_state; - u8 sta_id = IWL_MVM_INVALID_STA; struct iwl_mvm_vif *mvmvif = NULL; struct iwl_mvm_active_iface_iterator_data data = { .ignore_vif = changed_vif, .sta_vif_state = SF_UNINIT, - .sta_vif_ap_sta_id = IWL_MVM_INVALID_STA, }; + struct ieee80211_sta *sta = NULL; /* * Ignore the call if we are in HW Restart flow, or if the handled @@ -273,7 +263,7 @@ int iwl_mvm_sf_update(struct iwl_mvm *mvm, struct ieee80211_vif *changed_vif, * and we filled the relevant data during iteration */ new_state = data.sta_vif_state; - sta_id = data.sta_vif_ap_sta_id; + sta = data.sta_vif_ap_sta; } else { if (WARN_ON(!changed_vif)) return -EINVAL; @@ -282,7 +272,7 @@ int iwl_mvm_sf_update(struct iwl_mvm *mvm, struct ieee80211_vif *changed_vif, } else if (changed_vif->cfg.assoc && changed_vif->bss_conf.dtim_period) { mvmvif = iwl_mvm_vif_from_mac80211(changed_vif); - sta_id = mvmvif->deflink.ap_sta_id; + sta = mvmvif->ap_sta; new_state = SF_FULL_ON; } else { new_state = SF_INIT_OFF; @@ -294,8 +284,5 @@ int iwl_mvm_sf_update(struct iwl_mvm *mvm, struct ieee80211_vif *changed_vif, new_state = SF_UNINIT; } - /* For MLO it's ok to use deflink->sta_id as it's needed only to get - * a pointer to mac80211 sta - */ - return iwl_mvm_sf_config(mvm, sta_id, new_state); + return iwl_mvm_sf_config(mvm, sta, new_state); } diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/sta.c b/drivers/net/wireless/intel/iwlwifi/mvm/sta.c index 50e224883af0..5469d634e289 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/sta.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/sta.c @@ -251,6 +251,7 @@ static void iwl_mvm_rx_agg_session_expired(struct timer_list *t) struct ieee80211_sta *sta; struct iwl_mvm_sta *mvm_sta; unsigned long timeout; + unsigned int sta_id; rcu_read_lock(); @@ -269,7 +270,8 @@ static void iwl_mvm_rx_agg_session_expired(struct timer_list *t) } /* Timer expired */ - sta = rcu_dereference(ba_data->mvm->fw_id_to_mac_id[ba_data->sta_id]); + sta_id = ffs(ba_data->sta_mask) - 1; /* don't care which one */ + sta = rcu_dereference(ba_data->mvm->fw_id_to_mac_id[sta_id]); /* * sta should be valid unless the following happens: @@ -356,10 +358,14 @@ static int iwl_mvm_disable_txq(struct iwl_mvm *mvm, struct ieee80211_sta *sta, SCD_QUEUE_CONFIG_CMD); struct iwl_scd_queue_cfg_cmd remove_cmd = { .operation = cpu_to_le32(IWL_SCD_QUEUE_REMOVE), - .u.remove.tid = cpu_to_le32(tid), .u.remove.sta_mask = cpu_to_le32(BIT(sta_id)), }; + if (tid == IWL_MAX_TID_COUNT) + tid = IWL_MGMT_TID; + + remove_cmd.u.remove.tid = cpu_to_le32(tid); + ret = iwl_mvm_send_cmd_pdu(mvm, cmd_id, 0, sizeof(remove_cmd), &remove_cmd); @@ -2146,7 +2152,7 @@ int iwl_mvm_allocate_int_sta(struct iwl_mvm *mvm, sta->type = type; /* put a non-NULL value so iterating over the stations won't stop */ - rcu_assign_pointer(mvm->fw_id_to_mac_id[sta->sta_id], ERR_PTR(-EINVAL)); + RCU_INIT_POINTER(mvm->fw_id_to_mac_id[sta->sta_id], ERR_PTR(-EINVAL)); return 0; } @@ -2227,11 +2233,13 @@ static int iwl_mvm_add_int_sta_with_queue(struct iwl_mvm *mvm, int macidx, int iwl_mvm_add_aux_sta(struct iwl_mvm *mvm, u32 lmac_id) { int ret; + u32 qmask = mvm->aux_queue == IWL_MVM_INVALID_QUEUE ? 0 : + BIT(mvm->aux_queue); lockdep_assert_held(&mvm->mutex); /* Allocate aux station and assign to it the aux queue */ - ret = iwl_mvm_allocate_int_sta(mvm, &mvm->aux_sta, BIT(mvm->aux_queue), + ret = iwl_mvm_allocate_int_sta(mvm, &mvm->aux_sta, qmask, NL80211_IFTYPE_UNSPECIFIED, IWL_STA_AUX_ACTIVITY); if (ret) @@ -2750,10 +2758,11 @@ static void iwl_mvm_init_reorder_buffer(struct iwl_mvm *mvm, } static int iwl_mvm_fw_baid_op_sta(struct iwl_mvm *mvm, - struct iwl_mvm_sta *mvm_sta, + struct ieee80211_sta *sta, bool start, int tid, u16 ssn, u16 buf_size) { + struct iwl_mvm_sta *mvm_sta = iwl_mvm_sta_from_mac80211(sta); struct iwl_mvm_add_sta_cmd cmd = { .mac_id_n_color = cpu_to_le32(mvm_sta->mac_id_n_color), .sta_id = mvm_sta->deflink.sta_id, @@ -2798,7 +2807,7 @@ static int iwl_mvm_fw_baid_op_sta(struct iwl_mvm *mvm, } static int iwl_mvm_fw_baid_op_cmd(struct iwl_mvm *mvm, - struct iwl_mvm_sta *mvm_sta, + struct ieee80211_sta *sta, bool start, int tid, u16 ssn, u16 buf_size, int baid) { @@ -2813,7 +2822,7 @@ static int iwl_mvm_fw_baid_op_cmd(struct iwl_mvm *mvm, if (start) { cmd.alloc.sta_id_mask = - cpu_to_le32(BIT(mvm_sta->deflink.sta_id)); + cpu_to_le32(iwl_mvm_sta_fw_id_mask(mvm, sta, -1)); cmd.alloc.tid = tid; cmd.alloc.ssn = cpu_to_le16(ssn); cmd.alloc.win_size = cpu_to_le16(buf_size); @@ -2823,7 +2832,7 @@ static int iwl_mvm_fw_baid_op_cmd(struct iwl_mvm *mvm, BUILD_BUG_ON(sizeof(cmd.remove_v1) > sizeof(cmd.remove)); } else { cmd.remove.sta_id_mask = - cpu_to_le32(BIT(mvm_sta->deflink.sta_id)); + cpu_to_le32(iwl_mvm_sta_fw_id_mask(mvm, sta, -1)); cmd.remove.tid = cpu_to_le32(tid); } @@ -2846,16 +2855,16 @@ static int iwl_mvm_fw_baid_op_cmd(struct iwl_mvm *mvm, return baid; } -static int iwl_mvm_fw_baid_op(struct iwl_mvm *mvm, struct iwl_mvm_sta *mvm_sta, +static int iwl_mvm_fw_baid_op(struct iwl_mvm *mvm, struct ieee80211_sta *sta, bool start, int tid, u16 ssn, u16 buf_size, int baid) { if (fw_has_capa(&mvm->fw->ucode_capa, IWL_UCODE_TLV_CAPA_BAID_ML_SUPPORT)) - return iwl_mvm_fw_baid_op_cmd(mvm, mvm_sta, start, + return iwl_mvm_fw_baid_op_cmd(mvm, sta, start, tid, ssn, buf_size, baid); - return iwl_mvm_fw_baid_op_sta(mvm, mvm_sta, start, + return iwl_mvm_fw_baid_op_sta(mvm, sta, start, tid, ssn, buf_size); } @@ -2925,7 +2934,7 @@ int iwl_mvm_sta_rx_agg(struct iwl_mvm *mvm, struct ieee80211_sta *sta, /* Don't send command to remove (start=0) BAID during restart */ if (start || !test_bit(IWL_MVM_STATUS_IN_HW_RESTART, &mvm->status)) - baid = iwl_mvm_fw_baid_op(mvm, mvm_sta, start, tid, ssn, buf_size, + baid = iwl_mvm_fw_baid_op(mvm, sta, start, tid, ssn, buf_size, baid); if (baid < 0) { @@ -2947,7 +2956,7 @@ int iwl_mvm_sta_rx_agg(struct iwl_mvm *mvm, struct ieee80211_sta *sta, iwl_mvm_rx_agg_session_expired, 0); baid_data->mvm = mvm; baid_data->tid = tid; - baid_data->sta_id = mvm_sta->deflink.sta_id; + baid_data->sta_mask = iwl_mvm_sta_fw_id_mask(mvm, sta, -1); mvm_sta->tid_to_baid[tid] = baid; if (timeout) diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/sta.h b/drivers/net/wireless/intel/iwlwifi/mvm/sta.h index 7b9e91935aa0..a61d4f88125f 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/sta.h +++ b/drivers/net/wireless/intel/iwlwifi/mvm/sta.h @@ -356,6 +356,7 @@ struct iwl_mvm_link_sta { * @tid_disable_agg: bitmap: if bit(tid) is set, the fw won't send ampdus for * tid. * @sta_type: station type + * @authorized: indicates station is authorized * @sta_state: station state according to enum %ieee80211_sta_state * @bt_reduced_txpower: is reduced tx power enabled for this station * @next_status_eosp: the next reclaimed packet is a PS-Poll response and @@ -409,6 +410,7 @@ struct iwl_mvm_sta { enum ieee80211_sta_state sta_state; bool bt_reduced_txpower; bool next_status_eosp; + bool authorized; spinlock_t lock; struct iwl_mvm_tid_data tid_data[IWL_MAX_TID_COUNT + 1]; u8 tid_to_baid[IWL_MAX_TID_COUNT]; @@ -642,6 +644,8 @@ int iwl_mvm_mld_update_sta_links(struct iwl_mvm *mvm, struct ieee80211_vif *vif, struct ieee80211_sta *sta, u16 old_links, u16 new_links); +u32 iwl_mvm_sta_fw_id_mask(struct iwl_mvm *mvm, struct ieee80211_sta *sta, + int filter_link_id); /* Queues */ void iwl_mvm_mld_modify_all_sta_disable_tx(struct iwl_mvm *mvm, diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/tx.c b/drivers/net/wireless/intel/iwlwifi/mvm/tx.c index 2c842938656d..10d7178f1071 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/tx.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/tx.c @@ -184,10 +184,7 @@ static u32 iwl_mvm_tx_csum(struct iwl_mvm *mvm, struct sk_buff *skb, struct ieee80211_tx_info *info, bool amsdu) { - if (mvm->trans->trans_cfg->device_family < IWL_DEVICE_FAMILY_BZ || - (mvm->trans->trans_cfg->device_family == IWL_DEVICE_FAMILY_BZ && - CSR_HW_REV_TYPE(mvm->trans->hw_rev) == IWL_CFG_MAC_TYPE_GL && - mvm->trans->hw_rev_step == SILICON_A_STEP)) + if (!iwl_mvm_has_new_tx_csum(mvm)) return iwl_mvm_tx_csum_pre_bz(mvm, skb, info, amsdu); return iwl_mvm_tx_csum_bz(mvm, skb, amsdu); } @@ -332,22 +329,23 @@ static u32 iwl_mvm_get_tx_rate(struct iwl_mvm *mvm, sta ? iwl_mvm_sta_from_mac80211(sta)->sta_state : -1); rate_idx = info->control.rates[0].idx; + + /* For non 2 GHZ band, remap mac80211 rate indices into driver + * indices. + */ + if (info->band != NL80211_BAND_2GHZ || + (info->flags & IEEE80211_TX_CTL_NO_CCK_RATE)) + rate_idx += IWL_FIRST_OFDM_RATE; + + /* For 2.4 GHZ band, check that there is no need to remap */ + BUILD_BUG_ON(IWL_FIRST_CCK_RATE != 0); } /* if the rate isn't a well known legacy rate, take the lowest one */ if (rate_idx < 0 || rate_idx >= IWL_RATE_COUNT_LEGACY) - rate_idx = rate_lowest_index( - &mvm->nvm_data->bands[info->band], sta); - - /* - * For non 2 GHZ band, remap mac80211 rate - * indices into driver indices - */ - if (info->band != NL80211_BAND_2GHZ) - rate_idx += IWL_FIRST_OFDM_RATE; - - /* For 2.4 GHZ band, check that there is no need to remap */ - BUILD_BUG_ON(IWL_FIRST_CCK_RATE != 0); + rate_idx = iwl_mvm_mac_ctxt_get_lowest_rate(mvm, + info, + info->control.vif); /* Get PLCP rate for tx_cmd->rate_n_flags */ rate_plcp = iwl_mvm_mac80211_idx_to_hwrate(mvm->fw, rate_idx); @@ -606,8 +604,9 @@ static void iwl_mvm_skb_prepare_status(struct sk_buff *skb, static int iwl_mvm_get_ctrl_vif_queue(struct iwl_mvm *mvm, struct iwl_mvm_vif_link_info *link, struct ieee80211_tx_info *info, - struct ieee80211_hdr *hdr) + struct sk_buff *skb) { + struct ieee80211_hdr *hdr = (void *)skb->data; __le16 fc = hdr->frame_control; switch (info->control.vif->type) { @@ -624,7 +623,7 @@ static int iwl_mvm_get_ctrl_vif_queue(struct iwl_mvm *mvm, * reason 7 ("Class 3 frame received from nonassociated STA"). */ if (ieee80211_is_mgmt(fc) && - (!ieee80211_is_bufferable_mmpdu(fc) || + (!ieee80211_is_bufferable_mmpdu(skb) || ieee80211_is_deauth(fc) || ieee80211_is_disassoc(fc))) return link->mgmt_queue; @@ -757,7 +756,7 @@ int iwl_mvm_tx_skb_non_sta(struct iwl_mvm *mvm, struct sk_buff *skb) sta_id = link->mcast_sta.sta_id; queue = iwl_mvm_get_ctrl_vif_queue(mvm, link, &info, - hdr); + skb); } else if (info.control.vif->type == NL80211_IFTYPE_MONITOR) { queue = mvm->snif_queue; sta_id = mvm->snif_sta.sta_id; @@ -805,10 +804,11 @@ unsigned int iwl_mvm_max_amsdu_size(struct iwl_mvm *mvm, struct ieee80211_sta *sta, unsigned int tid) { struct iwl_mvm_sta *mvmsta = iwl_mvm_sta_from_mac80211(sta); - enum nl80211_band band = mvmsta->vif->bss_conf.chandef.chan->band; u8 ac = tid_to_mac80211_ac[tid]; + enum nl80211_band band; unsigned int txf; - int lmac = iwl_mvm_get_lmac_id(mvm->fw, band); + unsigned int val; + int lmac; /* For HE redirect to trigger based fifos */ if (sta->deflink.he_cap.has_he && !WARN_ON(!iwl_mvm_has_new_tx_api(mvm))) @@ -822,7 +822,37 @@ unsigned int iwl_mvm_max_amsdu_size(struct iwl_mvm *mvm, * We also want to have the start of the next packet inside the * fifo to be able to send bursts. */ - return min_t(unsigned int, mvmsta->max_amsdu_len, + val = mvmsta->max_amsdu_len; + + if (hweight16(sta->valid_links) <= 1) { + if (sta->valid_links) { + struct ieee80211_bss_conf *link_conf; + unsigned int link = ffs(sta->valid_links) - 1; + + rcu_read_lock(); + link_conf = rcu_dereference(mvmsta->vif->link_conf[link]); + if (WARN_ON(!link_conf)) + band = NL80211_BAND_2GHZ; + else + band = link_conf->chandef.chan->band; + rcu_read_unlock(); + } else { + band = mvmsta->vif->bss_conf.chandef.chan->band; + } + + lmac = iwl_mvm_get_lmac_id(mvm->fw, band); + } else if (fw_has_capa(&mvm->fw->ucode_capa, + IWL_UCODE_TLV_CAPA_CDB_SUPPORT)) { + /* for real MLO restrict to both LMACs if they exist */ + lmac = IWL_LMAC_5G_INDEX; + val = min_t(unsigned int, val, + mvm->fwrt.smem_cfg.lmac[lmac].txfifo_size[txf] - 256); + lmac = IWL_LMAC_24G_INDEX; + } else { + lmac = IWL_LMAC_24G_INDEX; + } + + return min_t(unsigned int, val, mvm->fwrt.smem_cfg.lmac[lmac].txfifo_size[txf] - 256); } @@ -1256,8 +1286,7 @@ int iwl_mvm_tx_skb_sta(struct iwl_mvm *mvm, struct sk_buff *skb, if (ret) return ret; - if (WARN_ON(skb_queue_empty(&mpdus_skbs))) - return ret; + WARN_ON(skb_queue_empty(&mpdus_skbs)); while (!skb_queue_empty(&mpdus_skbs)) { skb = __skb_dequeue(&mpdus_skbs); diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/drv.c b/drivers/net/wireless/intel/iwlwifi/pcie/drv.c index 6c935d73943a..dba112394838 100644 --- a/drivers/net/wireless/intel/iwlwifi/pcie/drv.c +++ b/drivers/net/wireless/intel/iwlwifi/pcie/drv.c @@ -504,6 +504,7 @@ static const struct pci_device_id iwl_hw_card_ids[] = { /* Bz devices */ {IWL_PCI_DEVICE(0x2727, PCI_ANY_ID, iwl_bz_trans_cfg)}, + {IWL_PCI_DEVICE(0x272b, PCI_ANY_ID, iwl_bz_trans_cfg)}, {IWL_PCI_DEVICE(0xA840, PCI_ANY_ID, iwl_bz_trans_cfg)}, {IWL_PCI_DEVICE(0x7740, PCI_ANY_ID, iwl_bz_trans_cfg)}, #endif /* CONFIG_IWLMVM */ @@ -513,16 +514,17 @@ static const struct pci_device_id iwl_hw_card_ids[] = { MODULE_DEVICE_TABLE(pci, iwl_hw_card_ids); #define _IWL_DEV_INFO(_device, _subdevice, _mac_type, _mac_step, _rf_type, \ - _rf_id, _no_160, _cores, _cdb, _jacket, _cfg, _name) \ - { .device = (_device), .subdevice = (_subdevice), .cfg = &(_cfg), \ - .name = _name, .mac_type = _mac_type, .rf_type = _rf_type, \ - .no_160 = _no_160, .cores = _cores, .rf_id = _rf_id, \ + _rf_id, _rf_step, _no_160, _cores, _cdb, _jacket, _cfg, \ + _name) \ + { .device = (_device), .subdevice = (_subdevice), .cfg = &(_cfg), \ + .name = _name, .mac_type = _mac_type, .rf_type = _rf_type, .rf_step = _rf_step, \ + .no_160 = _no_160, .cores = _cores, .rf_id = _rf_id, \ .mac_step = _mac_step, .cdb = _cdb, .jacket = _jacket } #define IWL_DEV_INFO(_device, _subdevice, _cfg, _name) \ - _IWL_DEV_INFO(_device, _subdevice, IWL_CFG_ANY, IWL_CFG_ANY, \ - IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, \ - IWL_CFG_ANY, IWL_CFG_ANY, _cfg, _name) + _IWL_DEV_INFO(_device, _subdevice, IWL_CFG_ANY, IWL_CFG_ANY, \ + IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, \ + IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, _cfg, _name) static const struct iwl_dev_info iwl_dev_info_table[] = { #if IS_ENABLED(CONFIG_IWLMVM) @@ -565,7 +567,6 @@ static const struct iwl_dev_info iwl_dev_info_table[] = { IWL_DEV_INFO(0x43F0, 0x1652, killer1650i_2ax_cfg_qu_b0_hr_b0, iwl_ax201_killer_1650i_name), IWL_DEV_INFO(0x43F0, 0x2074, iwl_ax201_cfg_qu_hr, NULL), IWL_DEV_INFO(0x43F0, 0x4070, iwl_ax201_cfg_qu_hr, NULL), - IWL_DEV_INFO(0x43F0, 0x1651, killer1650s_2ax_cfg_qu_b0_hr_b0, iwl_ax201_killer_1650s_name), IWL_DEV_INFO(0xA0F0, 0x0070, iwl_ax201_cfg_qu_hr, NULL), IWL_DEV_INFO(0xA0F0, 0x0074, iwl_ax201_cfg_qu_hr, NULL), IWL_DEV_INFO(0xA0F0, 0x0078, iwl_ax201_cfg_qu_hr, NULL), @@ -694,87 +695,87 @@ static const struct iwl_dev_info iwl_dev_info_table[] = { _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_PU, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_2ac_cfg_soc, iwl9461_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_PU, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_2ac_cfg_soc, iwl9461_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_PU, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_2ac_cfg_soc, iwl9462_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_PU, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_2ac_cfg_soc, iwl9462_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_PU, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_2ac_cfg_soc, iwl9560_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_PU, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_2ac_cfg_soc, iwl9560_name), _IWL_DEV_INFO(0x2526, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_PNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9260_2ac_cfg, iwl9461_160_name), _IWL_DEV_INFO(0x2526, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_PNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9260_2ac_cfg, iwl9461_name), _IWL_DEV_INFO(0x2526, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_PNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9260_2ac_cfg, iwl9462_160_name), _IWL_DEV_INFO(0x2526, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_PNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9260_2ac_cfg, iwl9462_name), _IWL_DEV_INFO(0x2526, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_TH, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_TH, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_TH, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT_GNSS, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9260_2ac_cfg, iwl9270_160_name), _IWL_DEV_INFO(0x2526, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_TH, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_TH, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_TH, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT_GNSS, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9260_2ac_cfg, iwl9270_name), _IWL_DEV_INFO(0x271B, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_TH, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_TH1, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_TH1, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9260_2ac_cfg, iwl9162_160_name), _IWL_DEV_INFO(0x271B, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_TH, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_TH1, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_TH1, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9260_2ac_cfg, iwl9162_name), _IWL_DEV_INFO(0x2526, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_TH, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_TH, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_TH, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9260_2ac_cfg, iwl9260_160_name), _IWL_DEV_INFO(0x2526, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_TH, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_TH, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_TH, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9260_2ac_cfg, iwl9260_name), @@ -782,176 +783,176 @@ static const struct iwl_dev_info iwl_dev_info_table[] = { /* Qu B step */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QU, SILICON_B_STEP, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qu_b0_jf_b0_cfg, iwl9461_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QU, SILICON_B_STEP, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qu_b0_jf_b0_cfg, iwl9461_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QU, SILICON_B_STEP, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qu_b0_jf_b0_cfg, iwl9462_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QU, SILICON_B_STEP, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qu_b0_jf_b0_cfg, iwl9462_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QU, SILICON_B_STEP, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qu_b0_jf_b0_cfg, iwl9560_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QU, SILICON_B_STEP, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qu_b0_jf_b0_cfg, iwl9560_name), _IWL_DEV_INFO(IWL_CFG_ANY, 0x1551, IWL_CFG_MAC_TYPE_QU, SILICON_B_STEP, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qu_b0_jf_b0_cfg, iwl9560_killer_1550s_name), _IWL_DEV_INFO(IWL_CFG_ANY, 0x1552, IWL_CFG_MAC_TYPE_QU, SILICON_B_STEP, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qu_b0_jf_b0_cfg, iwl9560_killer_1550i_name), /* Qu C step */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QU, SILICON_C_STEP, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qu_c0_jf_b0_cfg, iwl9461_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QU, SILICON_C_STEP, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qu_c0_jf_b0_cfg, iwl9461_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QU, SILICON_C_STEP, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qu_c0_jf_b0_cfg, iwl9462_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QU, SILICON_C_STEP, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qu_c0_jf_b0_cfg, iwl9462_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QU, SILICON_C_STEP, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qu_c0_jf_b0_cfg, iwl9560_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QU, SILICON_C_STEP, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qu_c0_jf_b0_cfg, iwl9560_name), _IWL_DEV_INFO(IWL_CFG_ANY, 0x1551, IWL_CFG_MAC_TYPE_QU, SILICON_C_STEP, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qu_c0_jf_b0_cfg, iwl9560_killer_1550s_name), _IWL_DEV_INFO(IWL_CFG_ANY, 0x1552, IWL_CFG_MAC_TYPE_QU, SILICON_C_STEP, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qu_c0_jf_b0_cfg, iwl9560_killer_1550i_name), /* QuZ */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QUZ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_quz_a0_jf_b0_cfg, iwl9461_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QUZ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_quz_a0_jf_b0_cfg, iwl9461_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QUZ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_quz_a0_jf_b0_cfg, iwl9462_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QUZ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_quz_a0_jf_b0_cfg, iwl9462_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QUZ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_quz_a0_jf_b0_cfg, iwl9560_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QUZ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_quz_a0_jf_b0_cfg, iwl9560_name), _IWL_DEV_INFO(IWL_CFG_ANY, 0x1551, IWL_CFG_MAC_TYPE_QUZ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_quz_a0_jf_b0_cfg, iwl9560_killer_1550s_name), _IWL_DEV_INFO(IWL_CFG_ANY, 0x1552, IWL_CFG_MAC_TYPE_QUZ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_quz_a0_jf_b0_cfg, iwl9560_killer_1550i_name), /* QnJ */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qnj_b0_jf_b0_cfg, iwl9461_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qnj_b0_jf_b0_cfg, iwl9461_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qnj_b0_jf_b0_cfg, iwl9462_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qnj_b0_jf_b0_cfg, iwl9462_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qnj_b0_jf_b0_cfg, iwl9560_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qnj_b0_jf_b0_cfg, iwl9560_name), _IWL_DEV_INFO(IWL_CFG_ANY, 0x1551, IWL_CFG_MAC_TYPE_QNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qnj_b0_jf_b0_cfg, iwl9560_killer_1550s_name), _IWL_DEV_INFO(IWL_CFG_ANY, 0x1552, IWL_CFG_MAC_TYPE_QNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl9560_qnj_b0_jf_b0_cfg, iwl9560_killer_1550i_name), @@ -959,397 +960,408 @@ static const struct iwl_dev_info iwl_dev_info_table[] = { /* Qu B step */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QU, SILICON_B_STEP, - IWL_CFG_RF_TYPE_HR1, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_HR1, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_qu_b0_hr1_b0, iwl_ax101_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QU, SILICON_B_STEP, - IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_qu_b0_hr_b0, iwl_ax203_name), /* Qu C step */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QU, SILICON_C_STEP, - IWL_CFG_RF_TYPE_HR1, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_HR1, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_qu_c0_hr1_b0, iwl_ax101_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QU, SILICON_C_STEP, - IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_qu_c0_hr_b0, iwl_ax203_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QU, SILICON_C_STEP, - IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_qu_c0_hr_b0, iwl_ax201_name), /* QuZ */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QUZ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_HR1, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_HR1, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_quz_a0_hr1_b0, iwl_ax101_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QUZ, SILICON_B_STEP, - IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_quz_a0_hr_b0, iwl_ax203_name), + _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_MAC_TYPE_QUZ, SILICON_B_STEP, + IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, + iwl_cfg_quz_a0_hr_b0, iwl_ax201_name), /* QnJ with Hr */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_QNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_qnj_b0_hr_b0_cfg, iwl_ax201_name), /* SnJ with Jf */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_snj_a0_jf_b0, iwl9461_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_snj_a0_jf_b0, iwl9461_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_snj_a0_jf_b0, iwl9462_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_snj_a0_jf_b0, iwl9462_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_snj_a0_jf_b0, iwl9560_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_snj_a0_jf_b0, iwl9560_name), /* SnJ with Hr */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_HR1, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_HR1, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_snj_hr_b0, iwl_ax101_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_snj_hr_b0, iwl_ax201_name), /* Ma */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, - IWL_CFG_MAC_TYPE_MA, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, + IWL_CFG_MAC_TYPE_MA, SILICON_A_STEP, + IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_ma_a0_hr_b0, iwl_ax201_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, - IWL_CFG_MAC_TYPE_MA, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, + IWL_CFG_MAC_TYPE_MA, SILICON_A_STEP, + IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_ma_a0_gf_a0, iwl_ax211_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, - IWL_CFG_MAC_TYPE_MA, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, + IWL_CFG_MAC_TYPE_MA, SILICON_A_STEP, + IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_CDB, IWL_CFG_ANY, iwl_cfg_ma_a0_gf4_a0, iwl_ax211_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, - IWL_CFG_MAC_TYPE_MA, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_MR, IWL_CFG_ANY, + IWL_CFG_MAC_TYPE_MA, SILICON_A_STEP, + IWL_CFG_RF_TYPE_MR, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_ma_a0_mr_a0, iwl_ax221_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, - IWL_CFG_MAC_TYPE_MA, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, + IWL_CFG_MAC_TYPE_MA, SILICON_A_STEP, + IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_ma_a0_fm_a0, iwl_ax231_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_MR, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_MR, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_snj_a0_mr_a0, iwl_ax221_name), + _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_MAC_TYPE_MA, SILICON_B_STEP, + IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, + iwl_cfg_ma_b0_hr_b0, iwl_ax201_name), + _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_MAC_TYPE_MA, SILICON_B_STEP, + IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, + iwl_cfg_ma_b0_gf_a0, iwl_ax211_name), + _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_MAC_TYPE_MA, SILICON_B_STEP, + IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_CDB, IWL_CFG_ANY, + iwl_cfg_ma_b0_gf4_a0, iwl_ax211_name), + _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_MAC_TYPE_MA, SILICON_B_STEP, + IWL_CFG_RF_TYPE_MR, IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, + iwl_cfg_ma_b0_mr_a0, iwl_ax221_name), + _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_MAC_TYPE_MA, SILICON_B_STEP, + IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, + iwl_cfg_ma_b0_fm_a0, iwl_ax231_name), /* So with Hr */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SO, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_so_a0_hr_a0, iwl_ax203_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SO, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_HR1, IWL_CFG_ANY, - IWL_CFG_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_HR1, IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_NO_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_so_a0_hr_a0, iwl_ax101_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SO, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_so_a0_hr_a0, iwl_ax201_name), /* So-F with Hr */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SOF, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_so_a0_hr_a0, iwl_ax203_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SOF, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_HR1, IWL_CFG_ANY, - IWL_CFG_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_HR1, IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_NO_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_so_a0_hr_a0, iwl_ax101_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SOF, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_so_a0_hr_a0, iwl_ax201_name), /* So-F with Gf */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SOF, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwlax211_2ax_cfg_so_gf_a0, iwl_ax211_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SOF, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_ANY, IWL_CFG_CDB, IWL_CFG_ANY, iwlax411_2ax_cfg_so_gf4_a0, iwl_ax411_name), /* Bz */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_BZ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_HR1, IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, + iwl_cfg_bz_a0_hr_a0, iwl_bz_name), + _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_MAC_TYPE_BZ, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_bz_a0_hr_b0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_BZ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_bz_a0_gf_a0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_BZ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_CDB, IWL_CFG_ANY, iwl_cfg_bz_a0_gf4_a0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_BZ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_MR, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_MR, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_bz_a0_mr_a0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, - IWL_CFG_MAC_TYPE_BZ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, + IWL_CFG_MAC_TYPE_BZ, SILICON_A_STEP, + IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, SILICON_A_STEP, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_bz_a0_fm_a0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_BZ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, SILICON_A_STEP, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_CDB, IWL_CFG_NO_JACKET, iwl_cfg_bz_a0_fm4_a0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_BZ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, SILICON_B_STEP, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_IS_JACKET, iwl_cfg_bz_a0_fm_b0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_BZ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, SILICON_B_STEP, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_CDB, IWL_CFG_IS_JACKET, iwl_cfg_bz_a0_fm4_b0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_GL, SILICON_A_STEP, - IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, SILICON_A_STEP, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_NO_JACKET, iwl_cfg_gl_a0_fm_a0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_GL, SILICON_B_STEP, - IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, SILICON_B_STEP, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_NO_JACKET, iwl_cfg_gl_b0_fm_b0, iwl_bz_name), /* BZ Z step */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_BZ, SILICON_Z_STEP, - IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_bz_z0_gf_a0, iwl_bz_name), /* BNJ */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_GL, SILICON_A_STEP, - IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, SILICON_A_STEP, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_IS_JACKET, iwl_cfg_bnj_a0_fm_a0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_GL, SILICON_B_STEP, - IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, SILICON_B_STEP, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_IS_JACKET, iwl_cfg_bnj_b0_fm_b0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, - IWL_CFG_MAC_TYPE_GL, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, + IWL_CFG_MAC_TYPE_GL, SILICON_A_STEP, + IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, SILICON_A_STEP, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_CDB, IWL_CFG_IS_JACKET, iwl_cfg_bnj_a0_fm4_a0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_GL, SILICON_B_STEP, - IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_FM, IWL_CFG_ANY, SILICON_B_STEP, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_CDB, IWL_CFG_IS_JACKET, iwl_cfg_bnj_b0_fm4_b0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_GL, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_IS_JACKET, iwl_cfg_bnj_a0_gf_a0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_GL, SILICON_B_STEP, - IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_IS_JACKET, iwl_cfg_bnj_b0_gf_a0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_GL, SILICON_A_STEP, - IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_CDB, IWL_CFG_IS_JACKET, iwl_cfg_bnj_a0_gf4_a0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_GL, SILICON_B_STEP, - IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_CDB, IWL_CFG_IS_JACKET, iwl_cfg_bnj_b0_gf4_a0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_GL, SILICON_A_STEP, - IWL_CFG_RF_TYPE_HR1, IWL_CFG_ANY, - IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_IS_JACKET, + IWL_CFG_RF_TYPE_HR1, IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, + iwl_cfg_bnj_a0_hr_a0, iwl_bz_name), + _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_MAC_TYPE_GL, SILICON_A_STEP, + IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_bnj_a0_hr_b0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_GL, SILICON_B_STEP, - IWL_CFG_RF_TYPE_HR1, IWL_CFG_ANY, - IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_NO_JACKET, - iwl_cfg_bnj_b0_hr_b0, iwl_bz_name), - -/* SoF with JF2 */ - _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, - IWL_CFG_MAC_TYPE_SOF, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, - IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, - iwlax210_2ax_cfg_so_jf_b0, iwl9560_160_name), - _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, - IWL_CFG_MAC_TYPE_SOF, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, - IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, - iwlax210_2ax_cfg_so_jf_b0, iwl9560_name), - -/* SoF with JF */ - _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, - IWL_CFG_MAC_TYPE_SOF, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, - IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, - iwlax210_2ax_cfg_so_jf_b0, iwl9461_160_name), - _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, - IWL_CFG_MAC_TYPE_SOF, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, - IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, - iwlax210_2ax_cfg_so_jf_b0, iwl9462_160_name), - _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, - IWL_CFG_MAC_TYPE_SOF, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, - IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, - iwlax210_2ax_cfg_so_jf_b0, iwl9461_name), + IWL_CFG_RF_TYPE_HR1, IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, + iwl_cfg_bnj_b0_hr_a0, iwl_bz_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, - IWL_CFG_MAC_TYPE_SOF, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, - IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, - iwlax210_2ax_cfg_so_jf_b0, iwl9462_name), + IWL_CFG_MAC_TYPE_GL, SILICON_B_STEP, + IWL_CFG_RF_TYPE_HR2, IWL_CFG_ANY, IWL_CFG_ANY, + IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, + iwl_cfg_bnj_b0_hr_b0, iwl_bz_name), /* SoF with JF2 */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SOF, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwlax210_2ax_cfg_so_jf_b0, iwl9560_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SOF, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwlax210_2ax_cfg_so_jf_b0, iwl9560_name), /* SoF with JF */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SOF, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwlax210_2ax_cfg_so_jf_b0, iwl9461_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SOF, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwlax210_2ax_cfg_so_jf_b0, iwl9462_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SOF, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwlax210_2ax_cfg_so_jf_b0, iwl9461_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SOF, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwlax210_2ax_cfg_so_jf_b0, iwl9462_name), /* So with GF */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SO, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwlax211_2ax_cfg_so_gf_a0, iwl_ax211_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SO, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_GF, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_ANY, IWL_CFG_CDB, IWL_CFG_ANY, iwlax411_2ax_cfg_so_gf4_a0, iwl_ax411_name), /* So with JF2 */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SO, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwlax210_2ax_cfg_so_jf_b0, iwl9560_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SO, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, + IWL_CFG_RF_TYPE_JF2, IWL_CFG_RF_ID_JF, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwlax210_2ax_cfg_so_jf_b0, iwl9560_name), /* So with JF */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SO, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwlax210_2ax_cfg_so_jf_b0, iwl9461_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SO, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwlax210_2ax_cfg_so_jf_b0, iwl9462_160_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SO, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwlax210_2ax_cfg_so_jf_b0, iwl9461_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SO, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, + IWL_CFG_RF_TYPE_JF1, IWL_CFG_RF_ID_JF1_DIV, IWL_CFG_ANY, IWL_CFG_NO_160, IWL_CFG_CORES_BT, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwlax210_2ax_cfg_so_jf_b0, iwl9462_name), @@ -1357,22 +1369,22 @@ static const struct iwl_dev_info iwl_dev_info_table[] = { /* For now we use the same FW as MR, but this will change in the future. */ _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SO, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_MS, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_MS, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_so_a0_ms_a0, iwl_ax204_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SOF, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_MS, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_MS, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_so_a0_ms_a0, iwl_ax204_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_MA, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_MS, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_MS, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_ma_a0_ms_a0, iwl_ax204_name), _IWL_DEV_INFO(IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_MAC_TYPE_SNJ, IWL_CFG_ANY, - IWL_CFG_RF_TYPE_MS, IWL_CFG_ANY, + IWL_CFG_RF_TYPE_MS, IWL_CFG_ANY, IWL_CFG_ANY, IWL_CFG_160, IWL_CFG_ANY, IWL_CFG_NO_CDB, IWL_CFG_ANY, iwl_cfg_snj_a0_ms_a0, iwl_ax204_name) @@ -1407,8 +1419,16 @@ static int get_crf_id(struct iwl_trans *iwl_trans) /* Read crf info */ iwl_trans->hw_crf_id = iwl_read_prph_no_grab(iwl_trans, sd_reg_ver_addr); + /* Read cnv info */ + iwl_trans->hw_cnv_id = + iwl_read_prph_no_grab(iwl_trans, CNVI_AUX_MISC_CHIP); + /* Read cdb info (also contains the jacket info if needed in the future */ - iwl_trans->hw_cdb_id = iwl_read_umac_prph_no_grab(iwl_trans, WFPM_OTP_CFG1_ADDR); + iwl_trans->hw_wfpm_id = + iwl_read_umac_prph_no_grab(iwl_trans, WFPM_OTP_CFG1_ADDR); + IWL_INFO(iwl_trans, "Detected crf-id 0x%x, cnv-id 0x%x wfpm id 0x%x\n", + iwl_trans->hw_crf_id, iwl_trans->hw_cnv_id, + iwl_trans->hw_wfpm_id); iwl_trans_release_nic_access(iwl_trans); @@ -1424,7 +1444,11 @@ static int map_crf_id(struct iwl_trans *iwl_trans) { int ret = 0; u32 val = iwl_trans->hw_crf_id; - u32 cdb = iwl_trans->hw_cdb_id; + u32 step_id = REG_CRF_ID_STEP(val); + u32 slave_id = REG_CRF_ID_SLAVE(val); + u32 jacket_id_cnv = REG_CRF_ID_SLAVE(iwl_trans->hw_cnv_id); + u32 jacket_id_wfpm = WFPM_OTP_CFG1_IS_JACKET(iwl_trans->hw_wfpm_id); + u32 cdb_id_wfpm = WFPM_OTP_CFG1_IS_CDB(iwl_trans->hw_wfpm_id); /* Map between crf id to rf id */ switch (REG_CRF_ID_TYPE(val)) { @@ -1434,9 +1458,12 @@ static int map_crf_id(struct iwl_trans *iwl_trans) case REG_CRF_ID_TYPE_JF_2: iwl_trans->hw_rf_id = (IWL_CFG_RF_TYPE_JF2 << 12); break; - case REG_CRF_ID_TYPE_HR_NONE_CDB: + case REG_CRF_ID_TYPE_HR_NONE_CDB_1X1: iwl_trans->hw_rf_id = (IWL_CFG_RF_TYPE_HR1 << 12); break; + case REG_CRF_ID_TYPE_HR_NONE_CDB: + iwl_trans->hw_rf_id = (IWL_CFG_RF_TYPE_HR2 << 12); + break; case REG_CRF_ID_TYPE_HR_CDB: iwl_trans->hw_rf_id = (IWL_CFG_RF_TYPE_HR2 << 12); break; @@ -1446,27 +1473,43 @@ static int map_crf_id(struct iwl_trans *iwl_trans) case REG_CRF_ID_TYPE_MR: iwl_trans->hw_rf_id = (IWL_CFG_RF_TYPE_MR << 12); break; - case REG_CRF_ID_TYPE_FM: - iwl_trans->hw_rf_id = (IWL_CFG_RF_TYPE_FM << 12); - break; + case REG_CRF_ID_TYPE_FM: + case REG_CRF_ID_TYPE_FMI: + case REG_CRF_ID_TYPE_FMR: + iwl_trans->hw_rf_id = (IWL_CFG_RF_TYPE_FM << 12); + break; default: ret = -EIO; IWL_ERR(iwl_trans, - "Can find a correct rfid for crf id 0x%x\n", + "Can't find a correct rfid for crf id 0x%x\n", REG_CRF_ID_TYPE(val)); goto out; } + /* Set Step-id */ + iwl_trans->hw_rf_id |= (step_id << 8); + /* Set CDB capabilities */ - if (cdb & BIT(4)) { + if (cdb_id_wfpm || slave_id) { iwl_trans->hw_rf_id += BIT(28); IWL_INFO(iwl_trans, "Adding cdb to rf id\n"); } - IWL_INFO(iwl_trans, "Detected RF 0x%x from crf id 0x%x\n", - iwl_trans->hw_rf_id, REG_CRF_ID_TYPE(val)); + /* Set Jacket capabilities */ + if (jacket_id_wfpm || jacket_id_cnv) { + iwl_trans->hw_rf_id += BIT(29); + IWL_INFO(iwl_trans, "Adding jacket to rf id\n"); + } + IWL_INFO(iwl_trans, + "Detected rf-type 0x%x step-id 0x%x slave-id 0x%x from crf id 0x%x\n", + REG_CRF_ID_TYPE(val), step_id, slave_id, iwl_trans->hw_rf_id); + IWL_INFO(iwl_trans, + "Detected cdb-id 0x%x jacket-id 0x%x from wfpm id 0x%x\n", + cdb_id_wfpm, jacket_id_wfpm, iwl_trans->hw_wfpm_id); + IWL_INFO(iwl_trans, "Detected jacket-id 0x%x from cnvi id 0x%x\n", + jacket_id_cnv, iwl_trans->hw_cnv_id); out: return ret; @@ -1477,8 +1520,8 @@ out: static const struct iwl_dev_info * iwl_pci_find_dev_info(u16 device, u16 subsystem_device, - u16 mac_type, u8 mac_step, - u16 rf_type, u8 cdb, u8 jacket, u8 rf_id, u8 no_160, u8 cores) + u16 mac_type, u8 mac_step, u16 rf_type, u8 cdb, + u8 jacket, u8 rf_id, u8 no_160, u8 cores, u8 rf_step) { int num_devices = ARRAY_SIZE(iwl_dev_info_table); int i; @@ -1529,6 +1572,10 @@ iwl_pci_find_dev_info(u16 device, u16 subsystem_device, dev_info->cores != cores) continue; + if (dev_info->rf_step != (u8)IWL_CFG_ANY && + dev_info->rf_step != rf_step) + continue; + return dev_info; } @@ -1600,6 +1647,10 @@ static int iwl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto out_free_trans; } + IWL_INFO(iwl_trans, "PCI dev %04x/%04x, rev=0x%x, rfid=0x%x\n", + pdev->device, pdev->subsystem_device, + iwl_trans->hw_rev, iwl_trans->hw_rf_id); + dev_info = iwl_pci_find_dev_info(pdev->device, pdev->subsystem_device, CSR_HW_REV_TYPE(iwl_trans->hw_rev), iwl_trans->hw_rev_step, @@ -1608,8 +1659,8 @@ static int iwl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent) CSR_HW_RFID_IS_JACKET(iwl_trans->hw_rf_id), IWL_SUBDEVICE_RF_ID(pdev->subsystem_device), IWL_SUBDEVICE_NO_160(pdev->subsystem_device), - IWL_SUBDEVICE_CORES(pdev->subsystem_device)); - + IWL_SUBDEVICE_CORES(pdev->subsystem_device), + CSR_HW_RFID_STEP(iwl_trans->hw_rf_id)); if (dev_info) { iwl_trans->cfg = dev_info->cfg; iwl_trans->name = dev_info->name; @@ -1729,6 +1780,9 @@ static void iwl_pci_remove(struct pci_dev *pdev) { struct iwl_trans *trans = pci_get_drvdata(pdev); + if (!trans) + return; + iwl_drv_stop(trans->drv); iwl_trans_pcie_free(trans); diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/internal.h b/drivers/net/wireless/intel/iwlwifi/pcie/internal.h index f7e4f868363d..69b95ad5993b 100644 --- a/drivers/net/wireless/intel/iwlwifi/pcie/internal.h +++ b/drivers/net/wireless/intel/iwlwifi/pcie/internal.h @@ -497,6 +497,7 @@ int iwl_pcie_rx_stop(struct iwl_trans *trans); void iwl_pcie_rx_free(struct iwl_trans *trans); void iwl_pcie_free_rbs_pool(struct iwl_trans *trans); void iwl_pcie_rx_init_rxb_lists(struct iwl_rxq *rxq); +void iwl_pcie_rx_napi_sync(struct iwl_trans *trans); void iwl_pcie_rxq_alloc_rbs(struct iwl_trans *trans, gfp_t priority, struct iwl_rxq *rxq); diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/rx.c b/drivers/net/wireless/intel/iwlwifi/pcie/rx.c index 9c9f87fe8377..0d7890f99a5f 100644 --- a/drivers/net/wireless/intel/iwlwifi/pcie/rx.c +++ b/drivers/net/wireless/intel/iwlwifi/pcie/rx.c @@ -1,6 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause /* - * Copyright (C) 2003-2014, 2018-2022 Intel Corporation + * Copyright (C) 2003-2014, 2018-2023 Intel Corporation * Copyright (C) 2013-2015 Intel Mobile Communications GmbH * Copyright (C) 2016-2017 Intel Deutschland GmbH */ @@ -1053,6 +1053,22 @@ static int iwl_pcie_napi_poll_msix(struct napi_struct *napi, int budget) return ret; } +void iwl_pcie_rx_napi_sync(struct iwl_trans *trans) +{ + struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans); + int i; + + if (unlikely(!trans_pcie->rxq)) + return; + + for (i = 0; i < trans->num_rx_queues; i++) { + struct iwl_rxq *rxq = &trans_pcie->rxq[i]; + + if (rxq && rxq->napi.poll) + napi_synchronize(&rxq->napi); + } +} + static int _iwl_pcie_rx_init(struct iwl_trans *trans) { struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans); diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/trans-gen2.c b/drivers/net/wireless/intel/iwlwifi/pcie/trans-gen2.c index 1e263154e9eb..73b395841ca8 100644 --- a/drivers/net/wireless/intel/iwlwifi/pcie/trans-gen2.c +++ b/drivers/net/wireless/intel/iwlwifi/pcie/trans-gen2.c @@ -156,6 +156,7 @@ void _iwl_trans_pcie_gen2_stop_device(struct iwl_trans *trans) if (test_and_clear_bit(STATUS_DEVICE_ENABLED, &trans->status)) { IWL_DEBUG_INFO(trans, "DEVICE_ENABLED bit was set and is now cleared\n"); + iwl_pcie_rx_napi_sync(trans); iwl_txq_gen2_tx_free(trans); iwl_pcie_rx_stop(trans); } @@ -350,7 +351,7 @@ void iwl_trans_pcie_gen2_fw_alive(struct iwl_trans *trans, u32 scd_addr) mutex_unlock(&trans_pcie->mutex); } -static void iwl_pcie_set_ltr(struct iwl_trans *trans) +static bool iwl_pcie_set_ltr(struct iwl_trans *trans) { u32 ltr_val = CSR_LTR_LONG_VAL_AD_NO_SNOOP_REQ | u32_encode_bits(CSR_LTR_LONG_VAL_AD_SCALE_USEC, @@ -371,18 +372,77 @@ static void iwl_pcie_set_ltr(struct iwl_trans *trans) trans->trans_cfg->device_family == IWL_DEVICE_FAMILY_22000) && !trans->trans_cfg->integrated) { iwl_write32(trans, CSR_LTR_LONG_VAL_AD, ltr_val); - } else if (trans->trans_cfg->integrated && - trans->trans_cfg->device_family == IWL_DEVICE_FAMILY_22000) { + return true; + } + + if (trans->trans_cfg->integrated && + trans->trans_cfg->device_family == IWL_DEVICE_FAMILY_22000) { iwl_write_prph(trans, HPM_MAC_LTR_CSR, HPM_MAC_LRT_ENABLE_ALL); iwl_write_prph(trans, HPM_UMAC_LTR, ltr_val); + return true; + } + + if (trans->trans_cfg->device_family == IWL_DEVICE_FAMILY_AX210) { + /* First clear the interrupt, just in case */ + iwl_write32(trans, CSR_MSIX_HW_INT_CAUSES_AD, + MSIX_HW_INT_CAUSES_REG_IML); + /* In this case, unfortunately the same ROM bug exists in the + * device (not setting LTR correctly), but we don't have control + * over the settings from the host due to some hardware security + * features. The only workaround we've been able to come up with + * so far is to try to keep the CPU and device busy by polling + * it and the IML (image loader) completed interrupt. + */ + return false; + } + + /* nothing needs to be done on other devices */ + return true; +} + +static void iwl_pcie_spin_for_iml(struct iwl_trans *trans) +{ +/* in practice, this seems to complete in around 20-30ms at most, wait 100 */ +#define IML_WAIT_TIMEOUT (HZ / 10) + struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans); + unsigned long end_time = jiffies + IML_WAIT_TIMEOUT; + u32 value, loops = 0; + bool irq = false; + + if (WARN_ON(!trans_pcie->iml)) + return; + + value = iwl_read32(trans, CSR_LTR_LAST_MSG); + IWL_DEBUG_INFO(trans, "Polling for IML load - CSR_LTR_LAST_MSG=0x%x\n", + value); + + while (time_before(jiffies, end_time)) { + if (iwl_read32(trans, CSR_MSIX_HW_INT_CAUSES_AD) & + MSIX_HW_INT_CAUSES_REG_IML) { + irq = true; + break; + } + /* Keep the CPU and device busy. */ + value = iwl_read32(trans, CSR_LTR_LAST_MSG); + loops++; } + + IWL_DEBUG_INFO(trans, + "Polled for IML load: irq=%d, loops=%d, CSR_LTR_LAST_MSG=0x%x\n", + irq, loops, value); + + /* We don't fail here even if we timed out - maybe we get lucky and the + * interrupt comes in later (and we get alive from firmware) and then + * we're all happy - but if not we'll fail on alive timeout or get some + * other error out. + */ } int iwl_trans_pcie_gen2_start_fw(struct iwl_trans *trans, const struct fw_img *fw, bool run_in_rfkill) { struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans); - bool hw_rfkill; + bool hw_rfkill, keep_ram_busy; int ret; /* This may fail if AMT took ownership of the device */ @@ -443,7 +503,7 @@ int iwl_trans_pcie_gen2_start_fw(struct iwl_trans *trans, if (ret) goto out; - iwl_pcie_set_ltr(trans); + keep_ram_busy = !iwl_pcie_set_ltr(trans); if (trans->trans_cfg->device_family >= IWL_DEVICE_FAMILY_BZ) { iwl_write32(trans, CSR_FUNC_SCRATCH, CSR_FUNC_SCRATCH_INIT_VALUE); @@ -455,6 +515,9 @@ int iwl_trans_pcie_gen2_start_fw(struct iwl_trans *trans, iwl_write_prph(trans, UREG_CPU_INIT_RUN, 1); } + if (keep_ram_busy) + iwl_pcie_spin_for_iml(trans); + /* re-check RF-Kill state since we may have missed the interrupt */ hw_rfkill = iwl_pcie_check_hw_rf_kill(trans); if (hw_rfkill && !run_in_rfkill) diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/trans.c b/drivers/net/wireless/intel/iwlwifi/pcie/trans.c index 0a9af1ad1f20..b281850fbf7a 100644 --- a/drivers/net/wireless/intel/iwlwifi/pcie/trans.c +++ b/drivers/net/wireless/intel/iwlwifi/pcie/trans.c @@ -599,7 +599,6 @@ static int iwl_pcie_set_hw_ready(struct iwl_trans *trans) int iwl_pcie_prepare_card_hw(struct iwl_trans *trans) { int ret; - int t = 0; int iter; IWL_DEBUG_INFO(trans, "iwl_trans_prepare_card_hw enter\n"); @@ -616,6 +615,8 @@ int iwl_pcie_prepare_card_hw(struct iwl_trans *trans) usleep_range(1000, 2000); for (iter = 0; iter < 10; iter++) { + int t = 0; + /* If HW is not ready, prepare the conditions to check again */ iwl_set_bit(trans, CSR_HW_IF_CONFIG_REG, CSR_HW_IF_CONFIG_REG_PREPARE); @@ -1260,6 +1261,7 @@ static void _iwl_trans_pcie_stop_device(struct iwl_trans *trans) if (test_and_clear_bit(STATUS_DEVICE_ENABLED, &trans->status)) { IWL_DEBUG_INFO(trans, "DEVICE_ENABLED bit was set and is now cleared\n"); + iwl_pcie_rx_napi_sync(trans); iwl_pcie_tx_stop(trans); iwl_pcie_rx_stop(trans); @@ -1522,19 +1524,16 @@ static int iwl_pcie_d3_handshake(struct iwl_trans *trans, bool suspend) struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans); int ret; - if (trans->trans_cfg->device_family == IWL_DEVICE_FAMILY_AX210) { + if (trans->trans_cfg->device_family == IWL_DEVICE_FAMILY_AX210) iwl_write_umac_prph(trans, UREG_DOORBELL_TO_ISR6, suspend ? UREG_DOORBELL_TO_ISR6_SUSPEND : UREG_DOORBELL_TO_ISR6_RESUME); - } else if (trans->trans_cfg->device_family >= IWL_DEVICE_FAMILY_BZ) { + else if (trans->trans_cfg->device_family >= IWL_DEVICE_FAMILY_BZ) iwl_write32(trans, CSR_IPC_SLEEP_CONTROL, suspend ? CSR_IPC_SLEEP_CONTROL_SUSPEND : CSR_IPC_SLEEP_CONTROL_RESUME); - iwl_write_umac_prph(trans, UREG_DOORBELL_TO_ISR6, - UREG_DOORBELL_TO_ISR6_SLEEP_CTRL); - } else { + else return 0; - } ret = wait_event_timeout(trans_pcie->sx_waitq, trans_pcie->sx_complete, 2 * HZ); @@ -2863,7 +2862,7 @@ static bool iwl_write_to_user_buf(char __user *user_buf, ssize_t count, void *buf, ssize_t *size, ssize_t *bytes_copied) { - int buf_size_left = count - *bytes_copied; + ssize_t buf_size_left = count - *bytes_copied; buf_size_left = buf_size_left - (buf_size_left % sizeof(u32)); if (*size > buf_size_left) diff --git a/drivers/net/wireless/intel/iwlwifi/queue/tx.c b/drivers/net/wireless/intel/iwlwifi/queue/tx.c index 726185d6fab8..d1c39c214f95 100644 --- a/drivers/net/wireless/intel/iwlwifi/queue/tx.c +++ b/drivers/net/wireless/intel/iwlwifi/queue/tx.c @@ -1554,14 +1554,18 @@ void iwl_txq_reclaim(struct iwl_trans *trans, int txq_id, int ssn, struct sk_buff_head *skbs) { struct iwl_txq *txq = trans->txqs.txq[txq_id]; - int tfd_num = iwl_txq_get_cmd_index(txq, ssn); - int read_ptr = iwl_txq_get_cmd_index(txq, txq->read_ptr); - int last_to_free; + int tfd_num, read_ptr, last_to_free; /* This function is not meant to release cmd queue*/ if (WARN_ON(txq_id == trans->txqs.cmd.q_id)) return; + if (WARN_ON(!txq)) + return; + + tfd_num = iwl_txq_get_cmd_index(txq, ssn); + read_ptr = iwl_txq_get_cmd_index(txq, txq->read_ptr); + spin_lock_bh(&txq->lock); if (!test_bit(txq_id, trans->txqs.queue_used)) { diff --git a/drivers/net/wireless/mediatek/mt76/dma.c b/drivers/net/wireless/mediatek/mt76/dma.c index da281cd1d36f..465190ebaf1c 100644 --- a/drivers/net/wireless/mediatek/mt76/dma.c +++ b/drivers/net/wireless/mediatek/mt76/dma.c @@ -402,8 +402,8 @@ mt76_dma_get_buf(struct mt76_dev *dev, struct mt76_queue *q, int idx, *info = le32_to_cpu(desc->info); if (mt76_queue_is_wed_rx(q)) { - u32 token = FIELD_GET(MT_DMA_CTL_TOKEN, - le32_to_cpu(desc->buf1)); + u32 buf1 = le32_to_cpu(desc->buf1); + u32 token = FIELD_GET(MT_DMA_CTL_TOKEN, buf1); struct mt76_txwi_cache *t = mt76_rx_token_release(dev, token); if (!t) @@ -424,6 +424,8 @@ mt76_dma_get_buf(struct mt76_dev *dev, struct mt76_queue *q, int idx, *drop = !!(ctrl & (MT_DMA_CTL_TO_HOST_A | MT_DMA_CTL_DROP)); + + *drop |= !!(buf1 & MT_DMA_CTL_WO_DROP); } } else { buf = e->buf; @@ -576,7 +578,9 @@ free: free_skb: status.skb = tx_info.skb; hw = mt76_tx_status_get_hw(dev, tx_info.skb); + spin_lock_bh(&dev->rx_lock); ieee80211_tx_status_ext(hw, &status); + spin_unlock_bh(&dev->rx_lock); return ret; } @@ -849,7 +853,7 @@ mt76_dma_rx_process(struct mt76_dev *dev, struct mt76_queue *q, int budget) !(dev->drv->rx_check(dev, data, len))) goto free_frag; - skb = build_skb(data, q->buf_size); + skb = napi_build_skb(data, q->buf_size); if (!skb) goto free_frag; diff --git a/drivers/net/wireless/mediatek/mt76/dma.h b/drivers/net/wireless/mediatek/mt76/dma.h index 4b9bc7f462b8..1b090d78cd05 100644 --- a/drivers/net/wireless/mediatek/mt76/dma.h +++ b/drivers/net/wireless/mediatek/mt76/dma.h @@ -19,6 +19,7 @@ #define MT_DMA_CTL_TO_HOST_A BIT(12) #define MT_DMA_CTL_DROP BIT(14) #define MT_DMA_CTL_TOKEN GENMASK(31, 16) +#define MT_DMA_CTL_WO_DROP BIT(8) #define MT_DMA_PPE_CPU_REASON GENMASK(15, 11) #define MT_DMA_PPE_ENTRY GENMASK(30, 16) diff --git a/drivers/net/wireless/mediatek/mt76/mac80211.c b/drivers/net/wireless/mediatek/mt76/mac80211.c index 34abf70f44af..467afef98ba2 100644 --- a/drivers/net/wireless/mediatek/mt76/mac80211.c +++ b/drivers/net/wireless/mediatek/mt76/mac80211.c @@ -418,7 +418,8 @@ mt76_phy_init(struct mt76_phy *phy, struct ieee80211_hw *hw) SET_IEEE80211_DEV(hw, dev->dev); SET_IEEE80211_PERM_ADDR(hw, phy->macaddr); - wiphy->features |= NL80211_FEATURE_ACTIVE_MONITOR; + wiphy->features |= NL80211_FEATURE_ACTIVE_MONITOR | + NL80211_FEATURE_AP_MODE_CHAN_WIDTH_CHANGE; wiphy->flags |= WIPHY_FLAG_HAS_CHANNEL_SWITCH | WIPHY_FLAG_SUPPORTS_TDLS | WIPHY_FLAG_AP_UAPSD; @@ -1066,9 +1067,14 @@ mt76_rx_convert(struct mt76_dev *dev, struct sk_buff *skb, status->enc_flags = mstat.enc_flags; status->encoding = mstat.encoding; status->bw = mstat.bw; - status->he_ru = mstat.he_ru; - status->he_gi = mstat.he_gi; - status->he_dcm = mstat.he_dcm; + if (status->encoding == RX_ENC_EHT) { + status->eht.ru = mstat.eht.ru; + status->eht.gi = mstat.eht.gi; + } else { + status->he_ru = mstat.he_ru; + status->he_gi = mstat.he_gi; + status->he_dcm = mstat.he_dcm; + } status->rate_idx = mstat.rate_idx; status->nss = mstat.nss; status->band = mstat.band; @@ -1303,7 +1309,8 @@ mt76_check_sta(struct mt76_dev *dev, struct sk_buff *skb) if (ps) set_bit(MT_WCID_FLAG_PS, &wcid->flags); - dev->drv->sta_ps(dev, sta, ps); + if (dev->drv->sta_ps) + dev->drv->sta_ps(dev, sta, ps); if (!ps) clear_bit(MT_WCID_FLAG_PS, &wcid->flags); diff --git a/drivers/net/wireless/mediatek/mt76/mt76.h b/drivers/net/wireless/mediatek/mt76/mt76.h index 183b0fc5a2d4..6b07b8fafec2 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76.h +++ b/drivers/net/wireless/mediatek/mt76/mt76.h @@ -621,12 +621,22 @@ struct mt76_rx_status { u16 freq; u32 flag; u8 enc_flags; - u8 encoding:2, bw:3, he_ru:3; - u8 he_gi:2, he_dcm:1; + u8 encoding:3, bw:4; + union { + struct { + u8 he_ru:3; + u8 he_gi:2; + u8 he_dcm:1; + }; + struct { + u8 ru:4; + u8 gi:2; + } eht; + }; + u8 amsdu:1, first_amsdu:1, last_amsdu:1; u8 rate_idx; - u8 nss; - u8 band; + u8 nss:5, band:3; s8 signal; u8 chains; s8 chain_signal[IEEE80211_MAX_CHAINS]; @@ -778,6 +788,7 @@ struct mt76_dev { spinlock_t rx_lock; struct napi_struct napi[__MT_RXQ_MAX]; struct sk_buff_head rx_skb[__MT_RXQ_MAX]; + struct tasklet_struct irq_tasklet; struct list_head txwi_cache; struct list_head rxwi_cache; diff --git a/drivers/net/wireless/mediatek/mt76/mt7603/mac.c b/drivers/net/wireless/mediatek/mt76/mt7603/mac.c index 70a7f84af028..12e0af52082a 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7603/mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt7603/mac.c @@ -1279,8 +1279,11 @@ void mt7603_mac_add_txs(struct mt7603_dev *dev, void *data) if (wcidx >= MT7603_WTBL_STA || !sta) goto out; - if (mt7603_fill_txs(dev, msta, &info, txs_data)) + if (mt7603_fill_txs(dev, msta, &info, txs_data)) { + spin_lock_bh(&dev->mt76.rx_lock); ieee80211_tx_status_noskb(mt76_hw(dev), sta, &info); + spin_unlock_bh(&dev->mt76.rx_lock); + } out: rcu_read_unlock(); diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/dma.c b/drivers/net/wireless/mediatek/mt76/mt7615/dma.c index f1914431ff7f..0ce01ccc5dce 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7615/dma.c +++ b/drivers/net/wireless/mediatek/mt76/mt7615/dma.c @@ -76,7 +76,8 @@ static int mt7615_poll_tx(struct napi_struct *napi, int budget) mt76_queue_tx_cleanup(dev, dev->mt76.q_mcu[MT_MCUQ_WM], false); if (napi_complete(napi)) - mt7615_irq_enable(dev, mt7615_tx_mcu_int_mask(dev)); + mt76_connac_irq_enable(&dev->mt76, + mt7615_tx_mcu_int_mask(dev)); mt76_connac_pm_unref(&dev->mphy, &dev->pm); @@ -297,7 +298,7 @@ int mt7615_dma_init(struct mt7615_dev *dev) else mask |= MT_INT_MCU_CMD; - mt7615_irq_enable(dev, mask); + mt76_connac_irq_enable(&dev->mt76, mask); mt7615_dma_start(dev); diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/eeprom.c b/drivers/net/wireless/mediatek/mt76/mt7615/eeprom.c index 6dbaaf95ee38..68e88224b8b1 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7615/eeprom.c +++ b/drivers/net/wireless/mediatek/mt76/mt7615/eeprom.c @@ -47,6 +47,9 @@ static int mt7615_efuse_init(struct mt7615_dev *dev, u32 base) void *buf; u32 val; + if (is_mt7663(&dev->mt76)) + len = MT7663_EEPROM_SIZE; + val = mt76_rr(dev, base + MT_EFUSE_BASE_CTRL); if (val & MT_EFUSE_BASE_CTRL_EMPTY) return 0; @@ -72,6 +75,8 @@ static int mt7615_eeprom_load(struct mt7615_dev *dev, u32 addr) { int ret; + BUILD_BUG_ON(MT7615_EEPROM_FULL_SIZE < MT7663_EEPROM_SIZE); + ret = mt76_eeprom_init(&dev->mt76, MT7615_EEPROM_FULL_SIZE); if (ret < 0) return ret; @@ -336,7 +341,7 @@ int mt7615_eeprom_init(struct mt7615_dev *dev, u32 addr) ret = mt7615_check_eeprom(&dev->mt76); if (ret && dev->mt76.otp.data) { memcpy(dev->mt76.eeprom.data, dev->mt76.otp.data, - MT7615_EEPROM_SIZE); + dev->mt76.otp.size); } else { dev->flash_eeprom = true; mt7615_cal_free_data(dev); diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/eeprom.h b/drivers/net/wireless/mediatek/mt76/mt7615/eeprom.h index a024dee10362..a67fbb90f5b3 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7615/eeprom.h +++ b/drivers/net/wireless/mediatek/mt76/mt7615/eeprom.h @@ -46,7 +46,7 @@ enum mt7615_eeprom_field { MT7615_EE_MAX = 0x3bf, MT7622_EE_MAX = 0x3db, - MT7663_EE_MAX = 0x400, + MT7663_EE_MAX = 0x600, }; #define MT_EE_RATE_POWER_MASK GENMASK(5, 0) diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/init.c b/drivers/net/wireless/mediatek/mt76/mt7615/init.c index 5fa6f097ec30..621e69f07e3c 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7615/init.c +++ b/drivers/net/wireless/mediatek/mt76/mt7615/init.c @@ -396,6 +396,7 @@ mt7615_init_wiphy(struct ieee80211_hw *hw) wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_SET_SCAN_DWELL); wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_VHT_IBSS); + wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_CAN_REPLACE_PTK0); ieee80211_hw_set(hw, SINGLE_SCAN_ON_ALL_BANDS); ieee80211_hw_set(hw, TX_STATUS_NO_AMPDU_LEN); diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/mac.c b/drivers/net/wireless/mediatek/mt76/mt7615/mac.c index 51a968a6afdc..da1d17b73a25 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7615/mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt7615/mac.c @@ -655,11 +655,6 @@ static int mt7615_mac_fill_rx(struct mt7615_dev *dev, struct sk_buff *skb) return 0; } -void mt7615_sta_ps(struct mt76_dev *mdev, struct ieee80211_sta *sta, bool ps) -{ -} -EXPORT_SYMBOL_GPL(mt7615_sta_ps); - static u16 mt7615_mac_tx_rate_val(struct mt7615_dev *dev, struct mt76_phy *mphy, @@ -1530,8 +1525,11 @@ static void mt7615_mac_add_txs(struct mt7615_dev *dev, void *data) if (wcid->phy_idx && dev->mt76.phys[MT_BAND1]) mphy = dev->mt76.phys[MT_BAND1]; - if (mt7615_fill_txs(dev, msta, &info, txs_data)) + if (mt7615_fill_txs(dev, msta, &info, txs_data)) { + spin_lock_bh(&dev->mt76.rx_lock); ieee80211_tx_status_noskb(mphy->hw, sta, &info); + spin_unlock_bh(&dev->mt76.rx_lock); + } out: rcu_read_unlock(); @@ -2352,7 +2350,7 @@ void mt7615_coredump_work(struct work_struct *work) break; skb_pull(skb, sizeof(struct mt7615_mcu_rxd)); - if (data + skb->len - dump > MT76_CONNAC_COREDUMP_SZ) { + if (!dump || data + skb->len - dump > MT76_CONNAC_COREDUMP_SZ) { dev_kfree_skb(skb); continue; } @@ -2362,6 +2360,8 @@ void mt7615_coredump_work(struct work_struct *work) dev_kfree_skb(skb); } - dev_coredumpv(dev->mt76.dev, dump, MT76_CONNAC_COREDUMP_SZ, - GFP_KERNEL); + + if (dump) + dev_coredumpv(dev->mt76.dev, dump, MT76_CONNAC_COREDUMP_SZ, + GFP_KERNEL); } diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/mac.h b/drivers/net/wireless/mediatek/mt76/mt7615/mac.h index 880c9f74a7f1..d08fbe64c262 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7615/mac.h +++ b/drivers/net/wireless/mediatek/mt76/mt7615/mac.h @@ -19,18 +19,6 @@ #define MT_RXD0_NORMAL_GROUP_3 BIT(27) #define MT_RXD0_NORMAL_GROUP_4 BIT(28) -enum rx_pkt_type { - PKT_TYPE_TXS, - PKT_TYPE_TXRXV, - PKT_TYPE_NORMAL, - PKT_TYPE_RX_DUP_RFB, - PKT_TYPE_RX_TMR, - PKT_TYPE_RETRIEVE, - PKT_TYPE_TXRX_NOTIFY, - PKT_TYPE_RX_EVENT, - PKT_TYPE_NORMAL_MCU, -}; - #define MT_RXD1_NORMAL_BSSID GENMASK(31, 26) #define MT_RXD1_NORMAL_PAYLOAD_FORMAT GENMASK(25, 24) #define MT_RXD1_FIRST_AMSDU_FRAME GENMASK(1, 0) diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/mcu.c b/drivers/net/wireless/mediatek/mt76/mt7615/mcu.c index eea398c79a98..8d745c9730c7 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7615/mcu.c +++ b/drivers/net/wireless/mediatek/mt76/mt7615/mcu.c @@ -163,16 +163,16 @@ int mt7615_mcu_parse_response(struct mt76_dev *mdev, int cmd, cmd == MCU_UNI_CMD(HIF_CTRL) || cmd == MCU_UNI_CMD(OFFLOAD) || cmd == MCU_UNI_CMD(SUSPEND)) { - struct mt7615_mcu_uni_event *event; + struct mt76_connac_mcu_uni_event *event; skb_pull(skb, sizeof(*rxd)); - event = (struct mt7615_mcu_uni_event *)skb->data; + event = (struct mt76_connac_mcu_uni_event *)skb->data; ret = le32_to_cpu(event->status); } else if (cmd == MCU_CE_QUERY(REG_READ)) { - struct mt7615_mcu_reg_event *event; + struct mt76_connac_mcu_reg_event *event; skb_pull(skb, sizeof(*rxd)); - event = (struct mt7615_mcu_reg_event *)skb->data; + event = (struct mt76_connac_mcu_reg_event *)skb->data; ret = (int)le32_to_cpu(event->val); } @@ -861,7 +861,8 @@ mt7615_mcu_wtbl_sta_add(struct mt7615_phy *phy, struct ieee80211_vif *vif, else mvif->sta_added = true; } - mt76_connac_mcu_sta_basic_tlv(sskb, vif, sta, enable, new_entry); + mt76_connac_mcu_sta_basic_tlv(&dev->mt76, sskb, vif, sta, enable, + new_entry); if (enable && sta) mt76_connac_mcu_sta_tlv(phy->mt76, sskb, sta, vif, 0, MT76_STA_INFO_STATE_ASSOC); diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/mcu.h b/drivers/net/wireless/mediatek/mt76/mt7615/mcu.h index 615956acc6b5..8e9604be0792 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7615/mcu.h +++ b/drivers/net/wireless/mediatek/mt76/mt7615/mcu.h @@ -206,17 +206,6 @@ enum { MCU_ATE_SET_TX_POWER_CONTROL = 0x15, }; -struct mt7615_mcu_uni_event { - u8 cid; - u8 pad[3]; - __le32 status; /* 0: success, others: fail */ -} __packed; - -struct mt7615_mcu_reg_event { - __le32 reg; - __le32 val; -} __packed; - struct mt7615_roc_tlv { u8 bss_idx; u8 token; diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/mmio.c b/drivers/net/wireless/mediatek/mt76/mt7615/mmio.c index 83173efb56dc..ac036a072439 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7615/mmio.c +++ b/drivers/net/wireless/mediatek/mt76/mt7615/mmio.c @@ -66,9 +66,7 @@ const u32 mt7663e_reg_map[] = { static void mt7615_rx_poll_complete(struct mt76_dev *mdev, enum mt76_rxq_id q) { - struct mt7615_dev *dev = container_of(mdev, struct mt7615_dev, mt76); - - mt7615_irq_enable(dev, MT_INT_RX_DONE(q)); + mt76_connac_irq_enable(mdev, MT_INT_RX_DONE(q)); } static irqreturn_t mt7615_irq_handler(int irq, void *dev_instance) @@ -80,14 +78,14 @@ static irqreturn_t mt7615_irq_handler(int irq, void *dev_instance) if (!test_bit(MT76_STATE_INITIALIZED, &dev->mphy.state)) return IRQ_NONE; - tasklet_schedule(&dev->irq_tasklet); + tasklet_schedule(&dev->mt76.irq_tasklet); return IRQ_HANDLED; } static void mt7615_irq_tasklet(struct tasklet_struct *t) { - struct mt7615_dev *dev = from_tasklet(dev, t, irq_tasklet); + struct mt7615_dev *dev = from_tasklet(dev, t, mt76.irq_tasklet); u32 intr, mask = 0, tx_mcu_mask = mt7615_tx_mcu_int_mask(dev); u32 mcu_int; @@ -181,7 +179,6 @@ int mt7615_mmio_probe(struct device *pdev, void __iomem *mem_base, .rx_check = mt7615_rx_check, .rx_skb = mt7615_queue_rx_skb, .rx_poll_complete = mt7615_rx_poll_complete, - .sta_ps = mt7615_sta_ps, .sta_add = mt7615_mac_sta_add, .sta_remove = mt7615_mac_sta_remove, .update_survey = mt7615_update_channel, @@ -202,7 +199,7 @@ int mt7615_mmio_probe(struct device *pdev, void __iomem *mem_base, dev = container_of(mdev, struct mt7615_dev, mt76); mt76_mmio_init(&dev->mt76, mem_base); - tasklet_setup(&dev->irq_tasklet, mt7615_irq_tasklet); + tasklet_setup(&mdev->irq_tasklet, mt7615_irq_tasklet); dev->reg_map = map; dev->ops = ops; diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/mt7615.h b/drivers/net/wireless/mediatek/mt76/mt7615/mt7615.h index 9e58f6924493..582d1b5b7cb3 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7615/mt7615.h +++ b/drivers/net/wireless/mediatek/mt76/mt7615/mt7615.h @@ -51,6 +51,7 @@ #define MT7663_FIRMWARE_N9 "mediatek/mt7663_n9_rebb.bin" #define MT7615_EEPROM_SIZE 1024 +#define MT7663_EEPROM_SIZE 1536 #define MT7615_TOKEN_SIZE 4096 #define MT_FRAC_SCALE 12 @@ -245,8 +246,6 @@ struct mt7615_dev { }; const struct mt76_bus_ops *bus_ops; - struct tasklet_struct irq_tasklet; - struct mt7615_phy phy; u64 omac_mask; @@ -412,13 +411,6 @@ void mt7615_mcu_rx_event(struct mt7615_dev *dev, struct sk_buff *skb); int mt7615_mcu_rdd_send_pattern(struct mt7615_dev *dev); int mt7615_mcu_fw_log_2_host(struct mt7615_dev *dev, u8 ctrl); -static inline void mt7615_irq_enable(struct mt7615_dev *dev, u32 mask) -{ - mt76_set_irq_mask(&dev->mt76, 0, 0, mask); - - tasklet_schedule(&dev->irq_tasklet); -} - static inline bool mt7615_firmware_offload(struct mt7615_dev *dev) { return dev->fw_ver > MT7615_FIRMWARE_V2; @@ -518,7 +510,6 @@ void mt7615_tx_token_put(struct mt7615_dev *dev); bool mt7615_rx_check(struct mt76_dev *mdev, void *data, int len); void mt7615_queue_rx_skb(struct mt76_dev *mdev, enum mt76_rxq_id q, struct sk_buff *skb, u32 *info); -void mt7615_sta_ps(struct mt76_dev *mdev, struct ieee80211_sta *sta, bool ps); int mt7615_mac_sta_add(struct mt76_dev *mdev, struct ieee80211_vif *vif, struct ieee80211_sta *sta); void mt7615_mac_sta_remove(struct mt76_dev *mdev, struct ieee80211_vif *vif, diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/pci.c b/drivers/net/wireless/mediatek/mt76/mt7615/pci.c index b808248943ea..9f43e673518b 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7615/pci.c +++ b/drivers/net/wireless/mediatek/mt76/mt7615/pci.c @@ -94,7 +94,7 @@ static int mt7615_pci_suspend(struct pci_dev *pdev, pm_message_t state) mt76_for_each_q_rx(mdev, i) { napi_disable(&mdev->napi[i]); } - tasklet_kill(&dev->irq_tasklet); + tasklet_kill(&mdev->irq_tasklet); mt7615_dma_reset(dev); diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/pci_init.c b/drivers/net/wireless/mediatek/mt76/mt7615/pci_init.c index 0680e002b981..f607eee3fb47 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7615/pci_init.c +++ b/drivers/net/wireless/mediatek/mt76/mt7615/pci_init.c @@ -122,7 +122,7 @@ void mt7615_unregister_device(struct mt7615_dev *dev) mt7615_tx_token_put(dev); mt7615_dma_cleanup(dev); - tasklet_disable(&dev->irq_tasklet); + tasklet_disable(&dev->mt76.irq_tasklet); mt76_free_device(&dev->mt76); } diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/sdio.c b/drivers/net/wireless/mediatek/mt76/mt7615/sdio.c index 304212f5f8da..fc547a0031ea 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7615/sdio.c +++ b/drivers/net/wireless/mediatek/mt76/mt7615/sdio.c @@ -84,7 +84,6 @@ static int mt7663s_probe(struct sdio_func *func, .tx_status_data = mt7663_usb_sdio_tx_status_data, .rx_skb = mt7615_queue_rx_skb, .rx_check = mt7615_rx_check, - .sta_ps = mt7615_sta_ps, .sta_add = mt7615_mac_sta_add, .sta_remove = mt7615_mac_sta_remove, .update_survey = mt7615_update_channel, diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/usb.c b/drivers/net/wireless/mediatek/mt76/mt7615/usb.c index f2d651d7adff..04963b9f7498 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7615/usb.c +++ b/drivers/net/wireless/mediatek/mt76/mt7615/usb.c @@ -120,7 +120,6 @@ static int mt7663u_probe(struct usb_interface *usb_intf, .tx_status_data = mt7663_usb_sdio_tx_status_data, .rx_skb = mt7615_queue_rx_skb, .rx_check = mt7615_rx_check, - .sta_ps = mt7615_sta_ps, .sta_add = mt7615_mac_sta_add, .sta_remove = mt7615_mac_sta_remove, .update_survey = mt7615_update_channel, diff --git a/drivers/net/wireless/mediatek/mt76/mt76_connac.h b/drivers/net/wireless/mediatek/mt76/mt76_connac.h index b339c50bff20..15653b274f83 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76_connac.h +++ b/drivers/net/wireless/mediatek/mt76/mt76_connac.h @@ -6,6 +6,20 @@ #include "mt76.h" +enum rx_pkt_type { + PKT_TYPE_TXS, + PKT_TYPE_TXRXV, + PKT_TYPE_NORMAL, + PKT_TYPE_RX_DUP_RFB, + PKT_TYPE_RX_TMR, + PKT_TYPE_RETRIEVE, + PKT_TYPE_TXRX_NOTIFY, + PKT_TYPE_RX_EVENT, + PKT_TYPE_NORMAL_MCU, + PKT_TYPE_RX_FW_MONITOR = 0x0c, + PKT_TYPE_TXRX_NOTIFY_V0 = 0x18, +}; + #define MT76_CONNAC_SCAN_IE_LEN 600 #define MT76_CONNAC_MAX_NUM_SCHED_SCAN_INTERVAL 10 #define MT76_CONNAC_MAX_TIME_SCHED_SCAN_INTERVAL U16_MAX @@ -279,6 +293,12 @@ static inline u8 mt76_connac_spe_idx(u8 antenna_mask) return ant_to_spe[antenna_mask]; } +static inline void mt76_connac_irq_enable(struct mt76_dev *dev, u32 mask) +{ + mt76_set_irq_mask(dev, 0, 0, mask); + tasklet_schedule(&dev->irq_tasklet); +} + int mt76_connac_pm_wake(struct mt76_phy *phy, struct mt76_connac_pm *pm); void mt76_connac_power_save_sched(struct mt76_phy *phy, struct mt76_connac_pm *pm); @@ -353,6 +373,7 @@ mt76_connac_mutex_release(struct mt76_dev *dev, struct mt76_connac_pm *pm) mutex_unlock(&dev->mutex); } +void mt76_connac_gen_ppe_thresh(u8 *he_ppet, int nss); int mt76_connac_init_tx_queues(struct mt76_phy *phy, int idx, int n_desc, int ring_base, u32 flags); void mt76_connac_write_hw_txp(struct mt76_dev *dev, diff --git a/drivers/net/wireless/mediatek/mt76/mt76_connac2_mac.h b/drivers/net/wireless/mediatek/mt76/mt76_connac2_mac.h index f33171bcd343..a5ec0f631385 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76_connac2_mac.h +++ b/drivers/net/wireless/mediatek/mt76/mt76_connac2_mac.h @@ -32,6 +32,16 @@ enum { MT_LMAC_PSMP0, }; +#define MT_TX_FREE_MSDU_CNT GENMASK(9, 0) +#define MT_TX_FREE_WLAN_ID GENMASK(23, 14) +#define MT_TX_FREE_LATENCY GENMASK(12, 0) +/* 0: success, others: dropped */ +#define MT_TX_FREE_STATUS GENMASK(14, 13) +#define MT_TX_FREE_MSDU_ID GENMASK(30, 16) +#define MT_TX_FREE_PAIR BIT(31) +/* will support this field in further revision */ +#define MT_TX_FREE_RATE GENMASK(13, 0) + #define MT_TXD0_Q_IDX GENMASK(31, 25) #define MT_TXD0_PKT_FMT GENMASK(24, 23) #define MT_TXD0_ETH_TYPE_OFFSET GENMASK(22, 16) @@ -166,6 +176,15 @@ enum { #define MT_TXS7_MPDU_RETRY_CNT GENMASK(31, 23) +/* RXD DW0 */ +#define MT_RXD0_LENGTH GENMASK(15, 0) +#define MT_RXD0_PKT_FLAG GENMASK(19, 16) +#define MT_RXD0_PKT_TYPE GENMASK(31, 27) + +#define MT_RXD0_NORMAL_ETH_TYPE_OFS GENMASK(22, 16) +#define MT_RXD0_NORMAL_IP_SUM BIT(23) +#define MT_RXD0_NORMAL_UDP_TCP_SUM BIT(24) + /* RXD DW1 */ #define MT_RXD1_NORMAL_WLAN_IDX GENMASK(9, 0) #define MT_RXD1_NORMAL_GROUP_1 BIT(11) @@ -308,6 +327,9 @@ enum { #define MT_CRXV_FOE_HI GENMASK(6, 0) #define MT_CRXV_FOE_SHIFT 13 +#define MT_CT_PARSE_LEN 72 +#define MT_CT_DMA_BUF_NUM 2 + #define MT_CT_INFO_APPLY_TXD BIT(0) #define MT_CT_INFO_COPY_HOST_TXD_ALL BIT(1) #define MT_CT_INFO_MGMT_FRAME BIT(2) diff --git a/drivers/net/wireless/mediatek/mt76/mt76_connac_mac.c b/drivers/net/wireless/mediatek/mt76/mt76_connac_mac.c index aed4ee95fb2e..ee0fbfcd07d6 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76_connac_mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt76_connac_mac.c @@ -9,6 +9,27 @@ #define HE_PREP(f, m, v) le16_encode_bits(le32_get_bits(v, MT_CRXV_HE_##m),\ IEEE80211_RADIOTAP_HE_##f) +void mt76_connac_gen_ppe_thresh(u8 *he_ppet, int nss) +{ + static const u8 ppet16_ppet8_ru3_ru0[] = { 0x1c, 0xc7, 0x71 }; + u8 i, ppet_bits, ppet_size, ru_bit_mask = 0x7; /* HE80 */ + + he_ppet[0] = FIELD_PREP(IEEE80211_PPE_THRES_NSS_MASK, nss - 1) | + FIELD_PREP(IEEE80211_PPE_THRES_RU_INDEX_BITMASK_MASK, + ru_bit_mask); + + ppet_bits = IEEE80211_PPE_THRES_INFO_PPET_SIZE * + nss * hweight8(ru_bit_mask) * 2; + ppet_size = DIV_ROUND_UP(ppet_bits, 8); + + for (i = 0; i < ppet_size - 1; i++) + he_ppet[i + 1] = ppet16_ppet8_ru3_ru0[i % 3]; + + he_ppet[i + 1] = ppet16_ppet8_ru3_ru0[i % 3] & + (0xff >> (8 - (ppet_bits - 1) % 8)); +} +EXPORT_SYMBOL_GPL(mt76_connac_gen_ppe_thresh); + int mt76_connac_pm_wake(struct mt76_phy *phy, struct mt76_connac_pm *pm) { struct mt76_dev *dev = phy->dev; @@ -267,11 +288,29 @@ int mt76_connac_init_tx_queues(struct mt76_phy *phy, int idx, int n_desc, } EXPORT_SYMBOL_GPL(mt76_connac_init_tx_queues); +#define __bitrate_mask_check(_mcs, _mode) \ +({ \ + u8 i = 0; \ + for (nss = 0; i < ARRAY_SIZE(mask->control[band]._mcs); i++) { \ + if (!mask->control[band]._mcs[i]) \ + continue; \ + if (hweight16(mask->control[band]._mcs[i]) == 1) { \ + mode = MT_PHY_TYPE_##_mode; \ + rateidx = ffs(mask->control[band]._mcs[i]) - 1; \ + if (mode == MT_PHY_TYPE_HT) \ + rateidx += 8 * i; \ + else \ + nss = i + 1; \ + goto out; \ + } \ + } \ +}) + u16 mt76_connac2_mac_tx_rate_val(struct mt76_phy *mphy, struct ieee80211_vif *vif, bool beacon, bool mcast) { - u8 mode = 0, band = mphy->chandef.chan->band; + u8 nss = 0, mode = 0, band = mphy->chandef.chan->band; int rateidx = 0, mcast_rate; if (!vif) @@ -286,19 +325,12 @@ u16 mt76_connac2_mac_tx_rate_val(struct mt76_phy *mphy, struct cfg80211_bitrate_mask *mask; mask = &vif->bss_conf.beacon_tx_rate; - if (hweight16(mask->control[band].he_mcs[0]) == 1) { - rateidx = ffs(mask->control[band].he_mcs[0]) - 1; - mode = MT_PHY_TYPE_HE_SU; - goto out; - } else if (hweight16(mask->control[band].vht_mcs[0]) == 1) { - rateidx = ffs(mask->control[band].vht_mcs[0]) - 1; - mode = MT_PHY_TYPE_VHT; - goto out; - } else if (hweight8(mask->control[band].ht_mcs[0]) == 1) { - rateidx = ffs(mask->control[band].ht_mcs[0]) - 1; - mode = MT_PHY_TYPE_HT; - goto out; - } else if (hweight32(mask->control[band].legacy) == 1) { + + __bitrate_mask_check(he_mcs, HE_SU); + __bitrate_mask_check(vht_mcs, VHT); + __bitrate_mask_check(ht_mcs, HT); + + if (hweight32(mask->control[band].legacy) == 1) { rateidx = ffs(mask->control[band].legacy) - 1; goto legacy; } @@ -314,9 +346,9 @@ legacy: rateidx = mt76_calculate_default_rate(mphy, rateidx); mode = rateidx >> 8; rateidx &= GENMASK(7, 0); - out: - return FIELD_PREP(MT_TX_RATE_IDX, rateidx) | + return FIELD_PREP(MT_TX_RATE_NSS, nss) | + FIELD_PREP(MT_TX_RATE_IDX, rateidx) | FIELD_PREP(MT_TX_RATE_MODE, mode); } EXPORT_SYMBOL_GPL(mt76_connac2_mac_tx_rate_val); @@ -537,7 +569,8 @@ void mt76_connac2_mac_write_txwi(struct mt76_dev *dev, __le32 *txwi, if (txwi[2] & cpu_to_le32(MT_TXD2_FIX_RATE)) { /* Fixed rata is available just for 802.11 txd */ struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data; - bool multicast = is_multicast_ether_addr(hdr->addr1); + bool multicast = ieee80211_is_data(hdr->frame_control) && + is_multicast_ether_addr(hdr->addr1); u16 rate = mt76_connac2_mac_tx_rate_val(mphy, vif, beacon, multicast); u32 val = MT_TXD6_FIXED_BW; @@ -582,6 +615,17 @@ bool mt76_connac2_mac_fill_txs(struct mt76_dev *dev, struct mt76_wcid *wcid, le32_get_bits(txs_data[6], MT_TXS6_MPDU_FAIL_CNT); stats->tx_retries += le32_get_bits(txs_data[7], MT_TXS7_MPDU_RETRY_CNT); + + if (wcid->sta) { + struct ieee80211_sta *sta; + u8 tid; + + sta = container_of((void *)wcid, struct ieee80211_sta, + drv_priv); + tid = FIELD_GET(MT_TXS0_TID, txs); + + ieee80211_refresh_tx_agg_session_timer(sta, tid); + } } txrate = FIELD_GET(MT_TXS0_TX_RATE, txs); diff --git a/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c b/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c index 008ece1b16f8..0f0a519f956f 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c +++ b/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c @@ -363,7 +363,7 @@ void mt76_connac_mcu_bss_omac_tlv(struct sk_buff *skb, } EXPORT_SYMBOL_GPL(mt76_connac_mcu_bss_omac_tlv); -void mt76_connac_mcu_sta_basic_tlv(struct sk_buff *skb, +void mt76_connac_mcu_sta_basic_tlv(struct mt76_dev *dev, struct sk_buff *skb, struct ieee80211_vif *vif, struct ieee80211_sta *sta, bool enable, bool newly) @@ -394,7 +394,7 @@ void mt76_connac_mcu_sta_basic_tlv(struct sk_buff *skb, switch (vif->type) { case NL80211_IFTYPE_MESH_POINT: case NL80211_IFTYPE_AP: - if (vif->p2p) + if (vif->p2p && !is_mt7921(dev)) conn_type = CONNECTION_P2P_GC; else conn_type = CONNECTION_INFRA_STA; @@ -402,7 +402,7 @@ void mt76_connac_mcu_sta_basic_tlv(struct sk_buff *skb, basic->aid = cpu_to_le16(sta->aid); break; case NL80211_IFTYPE_STATION: - if (vif->p2p) + if (vif->p2p && !is_mt7921(dev)) conn_type = CONNECTION_P2P_GO; else conn_type = CONNECTION_INFRA_AP; @@ -1029,7 +1029,7 @@ int mt76_connac_mcu_sta_cmd(struct mt76_phy *phy, return PTR_ERR(skb); if (info->sta || !info->offload_fw) - mt76_connac_mcu_sta_basic_tlv(skb, info->vif, info->sta, + mt76_connac_mcu_sta_basic_tlv(dev, skb, info->vif, info->sta, info->enable, info->newly); if (info->sta && info->enable) mt76_connac_mcu_sta_tlv(phy, skb, info->sta, @@ -1678,8 +1678,16 @@ int mt76_connac_mcu_hw_scan(struct mt76_phy *phy, struct ieee80211_vif *vif, req->channel_min_dwell_time = cpu_to_le16(duration); req->channel_dwell_time = cpu_to_le16(duration); - req->channels_num = min_t(u8, sreq->n_channels, 32); - req->ext_channels_num = min_t(u8, ext_channels_num, 32); + if (sreq->n_channels == 0 || sreq->n_channels > 64) { + req->channel_type = 0; + req->channels_num = 0; + req->ext_channels_num = 0; + } else { + req->channel_type = 4; + req->channels_num = min_t(u8, sreq->n_channels, 32); + req->ext_channels_num = min_t(u8, ext_channels_num, 32); + } + for (i = 0; i < req->channels_num + req->ext_channels_num; i++) { if (i >= 32) chan = &req->ext_channels[i - 32]; @@ -1699,7 +1707,6 @@ int mt76_connac_mcu_hw_scan(struct mt76_phy *phy, struct ieee80211_vif *vif, } chan->channel_num = scan_list[i]->hw_value; } - req->channel_type = sreq->n_channels ? 4 : 0; if (sreq->ie_len > 0) { memcpy(req->ies, sreq->ie, sreq->ie_len); diff --git a/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.h b/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.h index a5e6ee4daf92..ca1ce97a6d2f 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.h +++ b/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.h @@ -127,7 +127,7 @@ struct mt76_connac2_mcu_rxd { u8 rsv1[2]; u8 s2d_index; - u8 tlv[0]; + u8 tlv[]; }; struct mt76_connac2_patch_hdr { @@ -967,9 +967,6 @@ enum { DEV_INFO_MAX_NUM }; -#define MCU_UNI_CMD_EVENT BIT(1) -#define MCU_UNI_CMD_UNSOLICITED_EVENT BIT(2) - /* event table */ enum { MCU_EVENT_TARGET_ADDRESS_LEN = 0x01, @@ -1224,6 +1221,7 @@ enum { MCU_UNI_CMD_VOW = 0x37, MCU_UNI_CMD_RRO = 0x57, MCU_UNI_CMD_OFFCH_SCAN_CTRL = 0x58, + MCU_UNI_CMD_ASSERT_DUMP = 0x6f, }; enum { @@ -1692,6 +1690,17 @@ struct mt76_connac_config { u8 data[320]; } __packed; +struct mt76_connac_mcu_uni_event { + u8 cid; + u8 pad[3]; + __le32 status; /* 0: success, others: fail */ +} __packed; + +struct mt76_connac_mcu_reg_event { + __le32 reg; + __le32 val; +} __packed; + static inline enum mcu_cipher_type mt76_connac_mcu_get_cipher(int cipher) { @@ -1779,7 +1788,7 @@ mt76_connac_mcu_add_tlv(struct sk_buff *skb, int tag, int len) int mt76_connac_mcu_set_channel_domain(struct mt76_phy *phy); int mt76_connac_mcu_set_vif_ps(struct mt76_dev *dev, struct ieee80211_vif *vif); -void mt76_connac_mcu_sta_basic_tlv(struct sk_buff *skb, +void mt76_connac_mcu_sta_basic_tlv(struct mt76_dev *dev, struct sk_buff *skb, struct ieee80211_vif *vif, struct ieee80211_sta *sta, bool enable, bool newly); diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c b/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c index d3f74473e6fb..3e41d809ade3 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c @@ -631,8 +631,11 @@ void mt76x02_send_tx_status(struct mt76x02_dev *dev, mt76_tx_status_unlock(mdev, &list); - if (!status.skb) + if (!status.skb) { + spin_lock_bh(&dev->mt76.rx_lock); ieee80211_tx_status_ext(mt76_hw(dev), &status); + spin_unlock_bh(&dev->mt76.rx_lock); + } if (!len) goto out; diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/debugfs.c b/drivers/net/wireless/mediatek/mt76/mt7915/debugfs.c index 5a46813a59ea..879884ead660 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/debugfs.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/debugfs.c @@ -958,10 +958,10 @@ mt7915_xmit_queues_show(struct seq_file *file, void *data) DEFINE_SHOW_ATTRIBUTE(mt7915_xmit_queues); -#define mt7915_txpower_puts(prefix, rate) \ +#define mt7915_txpower_puts(rate) \ ({ \ - len += scnprintf(buf + len, sz - len, "%-16s:", #prefix " (tmac)"); \ - for (i = 0; i < mt7915_sku_group_len[rate]; i++, offs++) \ + len += scnprintf(buf + len, sz - len, "%-16s:", #rate " (TMAC)"); \ + for (i = 0; i < mt7915_sku_group_len[SKU_##rate]; i++, offs++) \ len += scnprintf(buf + len, sz - len, " %6d", txpwr[offs]); \ len += scnprintf(buf + len, sz - len, "\n"); \ }) @@ -1004,41 +1004,41 @@ mt7915_rate_txpower_get(struct file *file, char __user *user_buf, phy != &dev->phy, phy->mt76->chandef.chan->hw_value); len += scnprintf(buf + len, sz - len, "%-16s %6s %6s %6s %6s\n", " ", "1m", "2m", "5m", "11m"); - mt7915_txpower_puts(CCK, SKU_CCK); + mt7915_txpower_puts(CCK); len += scnprintf(buf + len, sz - len, "%-16s %6s %6s %6s %6s %6s %6s %6s %6s\n", " ", "6m", "9m", "12m", "18m", "24m", "36m", "48m", "54m"); - mt7915_txpower_puts(OFDM, SKU_OFDM); + mt7915_txpower_puts(OFDM); len += scnprintf(buf + len, sz - len, "%-16s %6s %6s %6s %6s %6s %6s %6s %6s\n", " ", "mcs0", "mcs1", "mcs2", "mcs3", "mcs4", "mcs5", "mcs6", "mcs7"); - mt7915_txpower_puts(HT20, SKU_HT_BW20); + mt7915_txpower_puts(HT_BW20); len += scnprintf(buf + len, sz - len, "%-16s %6s %6s %6s %6s %6s %6s %6s %6s %6s\n", " ", "mcs0", "mcs1", "mcs2", "mcs3", "mcs4", "mcs5", "mcs6", "mcs7", "mcs32"); - mt7915_txpower_puts(HT40, SKU_HT_BW40); + mt7915_txpower_puts(HT_BW40); len += scnprintf(buf + len, sz - len, "%-16s %6s %6s %6s %6s %6s %6s %6s %6s %6s %6s %6s %6s\n", " ", "mcs0", "mcs1", "mcs2", "mcs3", "mcs4", "mcs5", "mcs6", "mcs7", "mcs8", "mcs9", "mcs10", "mcs11"); - mt7915_txpower_puts(VHT20, SKU_VHT_BW20); - mt7915_txpower_puts(VHT40, SKU_VHT_BW40); - mt7915_txpower_puts(VHT80, SKU_VHT_BW80); - mt7915_txpower_puts(VHT160, SKU_VHT_BW160); - mt7915_txpower_puts(HE26, SKU_HE_RU26); - mt7915_txpower_puts(HE52, SKU_HE_RU52); - mt7915_txpower_puts(HE106, SKU_HE_RU106); - mt7915_txpower_puts(HE242, SKU_HE_RU242); - mt7915_txpower_puts(HE484, SKU_HE_RU484); - mt7915_txpower_puts(HE996, SKU_HE_RU996); - mt7915_txpower_puts(HE996x2, SKU_HE_RU2x996); + mt7915_txpower_puts(VHT_BW20); + mt7915_txpower_puts(VHT_BW40); + mt7915_txpower_puts(VHT_BW80); + mt7915_txpower_puts(VHT_BW160); + mt7915_txpower_puts(HE_RU26); + mt7915_txpower_puts(HE_RU52); + mt7915_txpower_puts(HE_RU106); + mt7915_txpower_puts(HE_RU242); + mt7915_txpower_puts(HE_RU484); + mt7915_txpower_puts(HE_RU996); + mt7915_txpower_puts(HE_RU2x996); reg = is_mt7915(&dev->mt76) ? MT_WF_PHY_TPC_CTRL_STAT(band) : MT_WF_PHY_TPC_CTRL_STAT_MT7916(band); diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/dma.c b/drivers/net/wireless/mediatek/mt76/mt7915/dma.c index abe17dac9996..43a5456d4b97 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/dma.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/dma.c @@ -87,8 +87,14 @@ static void mt7915_dma_config(struct mt7915_dev *dev) MT7916_RXQ_BAND0); RXQ_CONFIG(MT_RXQ_MCU_WA, WFDMA0, MT_INT_WED_RX_DONE_WA_MT7916, MT7916_RXQ_MCU_WA); - RXQ_CONFIG(MT_RXQ_BAND1, WFDMA0, MT_INT_WED_RX_DONE_BAND1_MT7916, - MT7916_RXQ_BAND1); + if (dev->hif2) + RXQ_CONFIG(MT_RXQ_BAND1, WFDMA0, + MT_INT_RX_DONE_BAND1_MT7916, + MT7916_RXQ_BAND1); + else + RXQ_CONFIG(MT_RXQ_BAND1, WFDMA0, + MT_INT_WED_RX_DONE_BAND1_MT7916, + MT7916_RXQ_BAND1); RXQ_CONFIG(MT_RXQ_MAIN_WA, WFDMA0, MT_INT_WED_RX_DONE_WA_MAIN_MT7916, MT7916_RXQ_MCU_WA_MAIN); TXQ_CONFIG(0, WFDMA0, MT_INT_WED_TX_DONE_BAND0, diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/init.c b/drivers/net/wireless/mediatek/mt76/mt7915/init.c index 5e288116b1b0..ac2049f49bb3 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/init.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/init.c @@ -89,6 +89,7 @@ static ssize_t mt7915_thermal_temp_store(struct device *dev, val < phy->throttle_temp[MT7915_CRIT_TEMP_IDX])) { dev_err(phy->dev->mt76.dev, "temp1_max shall be greater than temp1_crit."); + mutex_unlock(&phy->dev->mt76.mutex); return -EINVAL; } @@ -202,6 +203,10 @@ static int mt7915_thermal_init(struct mt7915_phy *phy) phy->cdev = cdev; } + /* initialize critical/maximum high temperature */ + phy->throttle_temp[MT7915_CRIT_TEMP_IDX] = MT7915_CRIT_TEMP; + phy->throttle_temp[MT7915_MAX_TEMP_IDX] = MT7915_MAX_TEMP; + if (!IS_REACHABLE(CONFIG_HWMON)) return 0; @@ -210,10 +215,6 @@ static int mt7915_thermal_init(struct mt7915_phy *phy) if (IS_ERR(hwmon)) return PTR_ERR(hwmon); - /* initialize critical/maximum high temperature */ - phy->throttle_temp[MT7915_CRIT_TEMP_IDX] = MT7915_CRIT_TEMP; - phy->throttle_temp[MT7915_MAX_TEMP_IDX] = MT7915_MAX_TEMP; - return 0; } @@ -368,6 +369,7 @@ mt7915_init_wiphy(struct mt7915_phy *phy) wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_UNSOL_BCAST_PROBE_RESP); wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_FILS_DISCOVERY); wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_ACK_SIGNAL_SUPPORT); + wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_CAN_REPLACE_PTK0); if (!is_mt7915(&dev->mt76)) wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_STA_TX_PWR); @@ -930,27 +932,6 @@ mt7915_set_stream_he_txbf_caps(struct mt7915_phy *phy, } } -static void -mt7915_gen_ppe_thresh(u8 *he_ppet, int nss) -{ - u8 i, ppet_bits, ppet_size, ru_bit_mask = 0x7; /* HE80 */ - static const u8 ppet16_ppet8_ru3_ru0[] = {0x1c, 0xc7, 0x71}; - - he_ppet[0] = FIELD_PREP(IEEE80211_PPE_THRES_NSS_MASK, nss - 1) | - FIELD_PREP(IEEE80211_PPE_THRES_RU_INDEX_BITMASK_MASK, - ru_bit_mask); - - ppet_bits = IEEE80211_PPE_THRES_INFO_PPET_SIZE * - nss * hweight8(ru_bit_mask) * 2; - ppet_size = DIV_ROUND_UP(ppet_bits, 8); - - for (i = 0; i < ppet_size - 1; i++) - he_ppet[i + 1] = ppet16_ppet8_ru3_ru0[i % 3]; - - he_ppet[i + 1] = ppet16_ppet8_ru3_ru0[i % 3] & - (0xff >> (8 - (ppet_bits - 1) % 8)); -} - static int mt7915_init_he_caps(struct mt7915_phy *phy, enum nl80211_band band, struct ieee80211_sband_iftype_data *data) @@ -1100,7 +1081,7 @@ mt7915_init_he_caps(struct mt7915_phy *phy, enum nl80211_band band, memset(he_cap->ppe_thres, 0, sizeof(he_cap->ppe_thres)); if (he_cap_elem->phy_cap_info[6] & IEEE80211_HE_PHY_CAP6_PPE_THRESHOLD_PRESENT) { - mt7915_gen_ppe_thresh(he_cap->ppe_thres, nss); + mt76_connac_gen_ppe_thresh(he_cap->ppe_thres, nss); } else { he_cap_elem->phy_cap_info[9] |= u8_encode_bits(IEEE80211_HE_PHY_CAP9_NOMINAL_PKT_PADDING_16US, @@ -1179,7 +1160,7 @@ static void mt7915_stop_hardware(struct mt7915_dev *dev) mt7915_mcu_exit(dev); mt7915_tx_token_put(dev); mt7915_dma_cleanup(dev); - tasklet_disable(&dev->irq_tasklet); + tasklet_disable(&dev->mt76.irq_tasklet); if (is_mt7986(&dev->mt76)) mt7986_wmac_disable(dev); diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/mac.c b/drivers/net/wireless/mediatek/mt76/mt7915/mac.c index 97ca55d283fb..7df8d95fc3fb 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/mac.c @@ -73,10 +73,6 @@ static struct mt76_wcid *mt7915_rx_get_wcid(struct mt7915_dev *dev, return &sta->vif->sta.wcid; } -void mt7915_sta_ps(struct mt76_dev *mdev, struct ieee80211_sta *sta, bool ps) -{ -} - bool mt7915_mac_wtbl_update(struct mt7915_dev *dev, int idx, u32 mask) { mt76_rmw(dev, MT_WTBL_UPDATE, MT_WTBL_UPDATE_WLAN_IDX, @@ -1627,7 +1623,7 @@ void mt7915_mac_reset_work(struct work_struct *work) } local_bh_enable(); - tasklet_schedule(&dev->irq_tasklet); + tasklet_schedule(&dev->mt76.irq_tasklet); mt76_wr(dev, MT_MCU_INT_EVENT, MT_MCU_INT_EVENT_RESET_DONE); mt7915_wait_reset_state(dev, MT_MCU_CMD_NORMAL_STATE); diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/mac.h b/drivers/net/wireless/mediatek/mt76/mt7915/mac.h index 6fa9c79f3e5f..ce94f87e2042 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/mac.h +++ b/drivers/net/wireless/mediatek/mt76/mt7915/mac.h @@ -6,43 +6,12 @@ #include "../mt76_connac2_mac.h" -#define MT_CT_PARSE_LEN 72 -#define MT_CT_DMA_BUF_NUM 2 - -#define MT_RXD0_LENGTH GENMASK(15, 0) -#define MT_RXD0_PKT_TYPE GENMASK(31, 27) - -#define MT_RXD0_NORMAL_ETH_TYPE_OFS GENMASK(22, 16) -#define MT_RXD0_NORMAL_IP_SUM BIT(23) -#define MT_RXD0_NORMAL_UDP_TCP_SUM BIT(24) - -enum rx_pkt_type { - PKT_TYPE_TXS, - PKT_TYPE_TXRXV, - PKT_TYPE_NORMAL, - PKT_TYPE_RX_DUP_RFB, - PKT_TYPE_RX_TMR, - PKT_TYPE_RETRIEVE, - PKT_TYPE_TXRX_NOTIFY, - PKT_TYPE_RX_EVENT, - PKT_TYPE_RX_FW_MONITOR = 0x0c, - PKT_TYPE_TXRX_NOTIFY_V0 = 0x18, -}; - #define MT_TX_FREE_VER GENMASK(18, 16) -#define MT_TX_FREE_MSDU_CNT GENMASK(9, 0) -#define MT_TX_FREE_MSDU_CNT_V0 GENMASK(6, 0) -#define MT_TX_FREE_WLAN_ID GENMASK(23, 14) -#define MT_TX_FREE_LATENCY GENMASK(12, 0) +#define MT_TX_FREE_MSDU_CNT_V0 GENMASK(6, 0) /* 0: success, others: dropped */ -#define MT_TX_FREE_MSDU_ID GENMASK(30, 16) -#define MT_TX_FREE_PAIR BIT(31) #define MT_TX_FREE_MPDU_HEADER BIT(30) #define MT_TX_FREE_MSDU_ID_V3 GENMASK(14, 0) -/* will support this field in further revision */ -#define MT_TX_FREE_RATE GENMASK(13, 0) - #define MT_TXS5_F0_FINAL_MPDU BIT(31) #define MT_TXS5_F0_QOS BIT(30) #define MT_TXS5_F0_TX_COUNT GENMASK(29, 25) diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/main.c b/drivers/net/wireless/mediatek/mt76/mt7915/main.c index 784191ec4802..1b361199c061 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/main.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/main.c @@ -269,7 +269,6 @@ static int mt7915_add_interface(struct ieee80211_hw *hw, vif->offload_flags |= IEEE80211_OFFLOAD_ENCAP_4ADDR; mt7915_init_bitrate_mask(vif); - memset(&mvif->cap, -1, sizeof(mvif->cap)); mt7915_mcu_add_bss_info(phy, vif, true); mt7915_mcu_add_sta(dev, vif, NULL, true); diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c b/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c index 5545a8bdf1d0..9fcb22fa1f97 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c @@ -706,7 +706,6 @@ static void mt7915_mcu_sta_he_tlv(struct sk_buff *skb, struct ieee80211_sta *sta, struct ieee80211_vif *vif) { - struct mt7915_vif *mvif = (struct mt7915_vif *)vif->drv_priv; struct ieee80211_he_cap_elem *elem = &sta->deflink.he_cap.he_cap_elem; struct ieee80211_he_mcs_nss_supp mcs_map; struct sta_rec_he *he; @@ -740,7 +739,7 @@ mt7915_mcu_sta_he_tlv(struct sk_buff *skb, struct ieee80211_sta *sta, IEEE80211_HE_PHY_CAP0_CHANNEL_WIDTH_SET_RU_MAPPING_IN_5G)) cap |= STA_REC_HE_CAP_BW20_RU242_SUPPORT; - if (mvif->cap.he_ldpc && + if (vif->bss_conf.he_ldpc && (elem->phy_cap_info[1] & IEEE80211_HE_PHY_CAP1_LDPC_CODING_IN_PAYLOAD)) cap |= STA_REC_HE_CAP_LDPC; @@ -849,7 +848,6 @@ static void mt7915_mcu_sta_muru_tlv(struct mt7915_dev *dev, struct sk_buff *skb, struct ieee80211_sta *sta, struct ieee80211_vif *vif) { - struct mt7915_vif *mvif = (struct mt7915_vif *)vif->drv_priv; struct ieee80211_he_cap_elem *elem = &sta->deflink.he_cap.he_cap_elem; struct sta_rec_muru *muru; struct tlv *tlv; @@ -862,9 +860,9 @@ mt7915_mcu_sta_muru_tlv(struct mt7915_dev *dev, struct sk_buff *skb, muru = (struct sta_rec_muru *)tlv; - muru->cfg.mimo_dl_en = mvif->cap.he_mu_ebfer || - mvif->cap.vht_mu_ebfer || - mvif->cap.vht_mu_ebfee; + muru->cfg.mimo_dl_en = vif->bss_conf.he_mu_beamformer || + vif->bss_conf.vht_mu_beamformer || + vif->bss_conf.vht_mu_beamformee; if (!is_mt7915(&dev->mt76)) muru->cfg.mimo_ul_en = true; muru->cfg.ofdma_dl_en = true; @@ -997,8 +995,8 @@ mt7915_mcu_sta_wtbl_tlv(struct mt7915_dev *dev, struct sk_buff *skb, mt76_connac_mcu_wtbl_hdr_trans_tlv(skb, vif, wcid, tlv, wtbl_hdr); if (sta) mt76_connac_mcu_wtbl_ht_tlv(&dev->mt76, skb, sta, tlv, - wtbl_hdr, mvif->cap.ht_ldpc, - mvif->cap.vht_ldpc); + wtbl_hdr, vif->bss_conf.ht_ldpc, + vif->bss_conf.vht_ldpc); return 0; } @@ -1007,7 +1005,6 @@ static inline bool mt7915_is_ebf_supported(struct mt7915_phy *phy, struct ieee80211_vif *vif, struct ieee80211_sta *sta, bool bfee) { - struct mt7915_vif *mvif = (struct mt7915_vif *)vif->drv_priv; int tx_ant = hweight8(phy->mt76->chainmask) - 1; if (vif->type != NL80211_IFTYPE_STATION && @@ -1021,10 +1018,10 @@ mt7915_is_ebf_supported(struct mt7915_phy *phy, struct ieee80211_vif *vif, struct ieee80211_he_cap_elem *pe = &sta->deflink.he_cap.he_cap_elem; if (bfee) - return mvif->cap.he_su_ebfee && + return vif->bss_conf.he_su_beamformee && HE_PHY(CAP3_SU_BEAMFORMER, pe->phy_cap_info[3]); else - return mvif->cap.he_su_ebfer && + return vif->bss_conf.he_su_beamformer && HE_PHY(CAP4_SU_BEAMFORMEE, pe->phy_cap_info[4]); } @@ -1032,10 +1029,10 @@ mt7915_is_ebf_supported(struct mt7915_phy *phy, struct ieee80211_vif *vif, u32 cap = sta->deflink.vht_cap.cap; if (bfee) - return mvif->cap.vht_su_ebfee && + return vif->bss_conf.vht_su_beamformee && (cap & IEEE80211_VHT_CAP_SU_BEAMFORMER_CAPABLE); else - return mvif->cap.vht_su_ebfer && + return vif->bss_conf.vht_su_beamformer && (cap & IEEE80211_VHT_CAP_SU_BEAMFORMEE_CAPABLE); } @@ -1530,7 +1527,7 @@ mt7915_mcu_sta_rate_ctrl_tlv(struct sk_buff *skb, struct mt7915_dev *dev, cap |= STA_CAP_TX_STBC; if (sta->deflink.ht_cap.cap & IEEE80211_HT_CAP_RX_STBC) cap |= STA_CAP_RX_STBC; - if (mvif->cap.ht_ldpc && + if (vif->bss_conf.ht_ldpc && (sta->deflink.ht_cap.cap & IEEE80211_HT_CAP_LDPC_CODING)) cap |= STA_CAP_LDPC; @@ -1556,7 +1553,7 @@ mt7915_mcu_sta_rate_ctrl_tlv(struct sk_buff *skb, struct mt7915_dev *dev, cap |= STA_CAP_VHT_TX_STBC; if (sta->deflink.vht_cap.cap & IEEE80211_VHT_CAP_RXSTBC_1) cap |= STA_CAP_VHT_RX_STBC; - if (mvif->cap.vht_ldpc && + if (vif->bss_conf.vht_ldpc && (sta->deflink.vht_cap.cap & IEEE80211_VHT_CAP_RXLDPC)) cap |= STA_CAP_VHT_LDPC; @@ -1657,8 +1654,8 @@ int mt7915_mcu_add_sta(struct mt7915_dev *dev, struct ieee80211_vif *vif, return PTR_ERR(skb); /* starec basic */ - mt76_connac_mcu_sta_basic_tlv(skb, vif, sta, enable, - !rcu_access_pointer(dev->mt76.wcid[msta->wcid.idx])); + mt76_connac_mcu_sta_basic_tlv(&dev->mt76, skb, vif, sta, enable, + !rcu_access_pointer(dev->mt76.wcid[msta->wcid.idx])); if (!enable) goto out; @@ -1876,84 +1873,6 @@ mt7915_mcu_beacon_cont(struct mt7915_dev *dev, struct ieee80211_vif *vif, } static void -mt7915_mcu_beacon_check_caps(struct mt7915_phy *phy, struct ieee80211_vif *vif, - struct sk_buff *skb) -{ - struct mt7915_vif *mvif = (struct mt7915_vif *)vif->drv_priv; - struct mt7915_vif_cap *vc = &mvif->cap; - const struct ieee80211_he_cap_elem *he; - const struct ieee80211_vht_cap *vht; - const struct ieee80211_ht_cap *ht; - struct ieee80211_mgmt *mgmt = (struct ieee80211_mgmt *)skb->data; - const u8 *ie; - u32 len, bc; - - /* Check missing configuration options to allow AP mode in mac80211 - * to remain in sync with hostapd settings, and get a subset of - * beacon and hardware capabilities. - */ - if (WARN_ON_ONCE(skb->len <= (mgmt->u.beacon.variable - skb->data))) - return; - - memset(vc, 0, sizeof(*vc)); - - len = skb->len - (mgmt->u.beacon.variable - skb->data); - - ie = cfg80211_find_ie(WLAN_EID_HT_CAPABILITY, mgmt->u.beacon.variable, - len); - if (ie && ie[1] >= sizeof(*ht)) { - ht = (void *)(ie + 2); - vc->ht_ldpc = !!(le16_to_cpu(ht->cap_info) & - IEEE80211_HT_CAP_LDPC_CODING); - } - - ie = cfg80211_find_ie(WLAN_EID_VHT_CAPABILITY, mgmt->u.beacon.variable, - len); - if (ie && ie[1] >= sizeof(*vht)) { - u32 pc = phy->mt76->sband_5g.sband.vht_cap.cap; - - vht = (void *)(ie + 2); - bc = le32_to_cpu(vht->vht_cap_info); - - vc->vht_ldpc = !!(bc & IEEE80211_VHT_CAP_RXLDPC); - vc->vht_su_ebfer = - (bc & IEEE80211_VHT_CAP_SU_BEAMFORMER_CAPABLE) && - (pc & IEEE80211_VHT_CAP_SU_BEAMFORMER_CAPABLE); - vc->vht_su_ebfee = - (bc & IEEE80211_VHT_CAP_SU_BEAMFORMEE_CAPABLE) && - (pc & IEEE80211_VHT_CAP_SU_BEAMFORMEE_CAPABLE); - vc->vht_mu_ebfer = - (bc & IEEE80211_VHT_CAP_MU_BEAMFORMER_CAPABLE) && - (pc & IEEE80211_VHT_CAP_MU_BEAMFORMER_CAPABLE); - vc->vht_mu_ebfee = - (bc & IEEE80211_VHT_CAP_MU_BEAMFORMEE_CAPABLE) && - (pc & IEEE80211_VHT_CAP_MU_BEAMFORMEE_CAPABLE); - } - - ie = cfg80211_find_ext_ie(WLAN_EID_EXT_HE_CAPABILITY, - mgmt->u.beacon.variable, len); - if (ie && ie[1] >= sizeof(*he) + 1) { - const struct ieee80211_sta_he_cap *pc = - mt76_connac_get_he_phy_cap(phy->mt76, vif); - const struct ieee80211_he_cap_elem *pe = &pc->he_cap_elem; - - he = (void *)(ie + 3); - - vc->he_ldpc = - HE_PHY(CAP1_LDPC_CODING_IN_PAYLOAD, pe->phy_cap_info[1]); - vc->he_su_ebfer = - HE_PHY(CAP3_SU_BEAMFORMER, he->phy_cap_info[3]) && - HE_PHY(CAP3_SU_BEAMFORMER, pe->phy_cap_info[3]); - vc->he_su_ebfee = - HE_PHY(CAP4_SU_BEAMFORMEE, he->phy_cap_info[4]) && - HE_PHY(CAP4_SU_BEAMFORMEE, pe->phy_cap_info[4]); - vc->he_mu_ebfer = - HE_PHY(CAP4_MU_BEAMFORMER, he->phy_cap_info[4]) && - HE_PHY(CAP4_MU_BEAMFORMER, pe->phy_cap_info[4]); - } -} - -static void mt7915_mcu_beacon_inband_discov(struct mt7915_dev *dev, struct ieee80211_vif *vif, struct sk_buff *rskb, struct bss_info_bcn *bcn, u32 changed) @@ -2063,8 +1982,6 @@ int mt7915_mcu_add_beacon(struct ieee80211_hw *hw, struct ieee80211_vif *vif, info = IEEE80211_SKB_CB(skb); info->hw_queue = FIELD_PREP(MT_TX_HW_QUEUE_PHY, ext_phy); - mt7915_mcu_beacon_check_caps(phy, vif, skb); - mt7915_mcu_beacon_cntdwn(vif, rskb, skb, bcn, &offs); mt7915_mcu_beacon_mbss(rskb, skb, vif, bcn, &offs); mt7915_mcu_beacon_cont(dev, vif, rskb, skb, bcn, &offs); @@ -2370,7 +2287,9 @@ int mt7915_mcu_init_firmware(struct mt7915_dev *dev) if (ret) return ret; - if (mtk_wed_device_active(&dev->mt76.mmio.wed) && is_mt7915(&dev->mt76)) + if ((mtk_wed_device_active(&dev->mt76.mmio.wed) && + is_mt7915(&dev->mt76)) || + !mtk_wed_get_rx_capa(&dev->mt76.mmio.wed)) mt7915_mcu_wa_cmd(dev, MCU_WA_PARAM_CMD(CAPABILITY), 0, 0, 0); ret = mt7915_mcu_set_mwds(dev, 1); diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/mmio.c b/drivers/net/wireless/mediatek/mt76/mt7915/mmio.c index 225a19604d3e..45f3558bf31c 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/mmio.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/mmio.c @@ -916,7 +916,7 @@ static void mt7915_rx_poll_complete(struct mt76_dev *mdev, /* TODO: support 2/4/6/8 MSI-X vectors */ static void mt7915_irq_tasklet(struct tasklet_struct *t) { - struct mt7915_dev *dev = from_tasklet(dev, t, irq_tasklet); + struct mt7915_dev *dev = from_tasklet(dev, t, mt76.irq_tasklet); struct mtk_wed_device *wed = &dev->mt76.mmio.wed; u32 intr, intr1, mask; @@ -989,18 +989,18 @@ irqreturn_t mt7915_irq_handler(int irq, void *dev_instance) struct mt7915_dev *dev = dev_instance; struct mtk_wed_device *wed = &dev->mt76.mmio.wed; - if (mtk_wed_device_active(wed)) { + if (mtk_wed_device_active(wed)) mtk_wed_device_irq_set_mask(wed, 0); - } else { + else mt76_wr(dev, MT_INT_MASK_CSR, 0); - if (dev->hif2) - mt76_wr(dev, MT_INT1_MASK_CSR, 0); - } + + if (dev->hif2) + mt76_wr(dev, MT_INT1_MASK_CSR, 0); if (!test_bit(MT76_STATE_INITIALIZED, &dev->mphy.state)) return IRQ_NONE; - tasklet_schedule(&dev->irq_tasklet); + tasklet_schedule(&dev->mt76.irq_tasklet); return IRQ_HANDLED; } @@ -1022,7 +1022,6 @@ struct mt7915_dev *mt7915_mmio_probe(struct device *pdev, .rx_skb = mt7915_queue_rx_skb, .rx_check = mt7915_rx_check, .rx_poll_complete = mt7915_rx_poll_complete, - .sta_ps = mt7915_sta_ps, .sta_add = mt7915_mac_sta_add, .sta_remove = mt7915_mac_sta_remove, .update_survey = mt7915_update_channel, @@ -1041,7 +1040,7 @@ struct mt7915_dev *mt7915_mmio_probe(struct device *pdev, if (ret) goto error; - tasklet_setup(&dev->irq_tasklet, mt7915_irq_tasklet); + tasklet_setup(&mdev->irq_tasklet, mt7915_irq_tasklet); return dev; diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h b/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h index 3cbfb9b6a305..b3ead3530740 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h +++ b/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h @@ -147,23 +147,9 @@ struct mt7915_sta { } twt; }; -struct mt7915_vif_cap { - bool ht_ldpc:1; - bool vht_ldpc:1; - bool he_ldpc:1; - bool vht_su_ebfer:1; - bool vht_su_ebfee:1; - bool vht_mu_ebfer:1; - bool vht_mu_ebfee:1; - bool he_su_ebfer:1; - bool he_su_ebfee:1; - bool he_mu_ebfer:1; -}; - struct mt7915_vif { struct mt76_vif mt76; /* must be first */ - struct mt7915_vif_cap cap; struct mt7915_sta sta; struct mt7915_phy *phy; @@ -308,7 +294,6 @@ struct mt7915_dev { u32 wfdma_mask; const struct mt76_bus_ops *bus_ops; - struct tasklet_struct irq_tasklet; struct mt7915_phy phy; /* monitor rx chain configured channel */ @@ -581,7 +566,7 @@ static inline void mt7915_irq_enable(struct mt7915_dev *dev, u32 mask) else mt76_set_irq_mask(&dev->mt76, 0, 0, mask); - tasklet_schedule(&dev->irq_tasklet); + tasklet_schedule(&dev->mt76.irq_tasklet); } static inline void mt7915_irq_disable(struct mt7915_dev *dev, u32 mask) @@ -631,7 +616,6 @@ void mt7915_tx_token_put(struct mt7915_dev *dev); void mt7915_queue_rx_skb(struct mt76_dev *mdev, enum mt76_rxq_id q, struct sk_buff *skb, u32 *info); bool mt7915_rx_check(struct mt76_dev *mdev, void *data, int len); -void mt7915_sta_ps(struct mt76_dev *mdev, struct ieee80211_sta *sta, bool ps); void mt7915_stats_work(struct work_struct *work); int mt76_dfs_start_rdd(struct mt7915_dev *dev, bool force); int mt7915_dfs_init_radar_detector(struct mt7915_phy *phy); diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/soc.c b/drivers/net/wireless/mediatek/mt76/mt7915/soc.c index 2ac0a0f2859c..32c137066e7f 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/soc.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/soc.c @@ -1239,6 +1239,8 @@ static const struct of_device_id mt7986_wmac_of_match[] = { {}, }; +MODULE_DEVICE_TABLE(of, mt7986_wmac_of_match); + struct platform_driver mt7986_wmac_driver = { .driver = { .name = "mt7986-wmac", diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/acpi_sar.h b/drivers/net/wireless/mediatek/mt76/mt7921/acpi_sar.h index 35268b0890ad..6f2c4a572572 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/acpi_sar.h +++ b/drivers/net/wireless/mediatek/mt76/mt7921/acpi_sar.h @@ -24,7 +24,7 @@ struct mt7921_asar_dyn { u8 names[4]; u8 enable; u8 nr_tbl; - struct mt7921_asar_dyn_limit tbl[0]; + DECLARE_FLEX_ARRAY(struct mt7921_asar_dyn_limit, tbl); } __packed; struct mt7921_asar_dyn_limit_v2 { @@ -37,7 +37,7 @@ struct mt7921_asar_dyn_v2 { u8 enable; u8 rsvd; u8 nr_tbl; - struct mt7921_asar_dyn_limit_v2 tbl[0]; + DECLARE_FLEX_ARRAY(struct mt7921_asar_dyn_limit_v2, tbl); } __packed; struct mt7921_asar_geo_band { @@ -55,7 +55,7 @@ struct mt7921_asar_geo { u8 names[4]; u8 version; u8 nr_tbl; - struct mt7921_asar_geo_limit tbl[0]; + DECLARE_FLEX_ARRAY(struct mt7921_asar_geo_limit, tbl); } __packed; struct mt7921_asar_geo_limit_v2 { @@ -69,7 +69,7 @@ struct mt7921_asar_geo_v2 { u8 version; u8 rsvd; u8 nr_tbl; - struct mt7921_asar_geo_limit_v2 tbl[0]; + DECLARE_FLEX_ARRAY(struct mt7921_asar_geo_limit_v2, tbl); } __packed; struct mt7921_asar_cl { @@ -85,7 +85,7 @@ struct mt7921_asar_fg { u8 rsvd; u8 nr_flag; u8 rsvd1; - u8 flag[0]; + u8 flag[]; } __packed; struct mt7921_acpi_sar { diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/debugfs.c b/drivers/net/wireless/mediatek/mt76/mt7921/debugfs.c index 29d8883268f6..d6b6edba2fec 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/debugfs.c +++ b/drivers/net/wireless/mediatek/mt76/mt7921/debugfs.c @@ -2,7 +2,6 @@ /* Copyright (C) 2020 MediaTek Inc. */ #include "mt7921.h" -#include "eeprom.h" static int mt7921_reg_set(void *data, u64 val) diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/dma.c b/drivers/net/wireless/mediatek/mt76/mt7921/dma.c index d1f10f6d9adc..f0a80c2b476a 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/dma.c +++ b/drivers/net/wireless/mediatek/mt76/mt7921/dma.c @@ -3,7 +3,7 @@ #include "mt7921.h" #include "../dma.h" -#include "mac.h" +#include "../mt76_connac2_mac.h" static int mt7921_poll_tx(struct napi_struct *napi, int budget) { @@ -19,7 +19,7 @@ static int mt7921_poll_tx(struct napi_struct *napi, int budget) mt76_connac_tx_cleanup(&dev->mt76); if (napi_complete(napi)) - mt7921_irq_enable(dev, MT_INT_TX_DONE_ALL); + mt76_connac_irq_enable(&dev->mt76, MT_INT_TX_DONE_ALL); mt76_connac_pm_unref(&dev->mphy, &dev->pm); return 0; @@ -66,6 +66,24 @@ static void mt7921_dma_prefetch(struct mt7921_dev *dev) static int mt7921_dma_disable(struct mt7921_dev *dev, bool force) { + /* disable WFDMA0 */ + mt76_clear(dev, MT_WFDMA0_GLO_CFG, + MT_WFDMA0_GLO_CFG_TX_DMA_EN | MT_WFDMA0_GLO_CFG_RX_DMA_EN | + MT_WFDMA0_GLO_CFG_CSR_DISP_BASE_PTR_CHAIN_EN | + MT_WFDMA0_GLO_CFG_OMIT_TX_INFO | + MT_WFDMA0_GLO_CFG_OMIT_RX_INFO | + MT_WFDMA0_GLO_CFG_OMIT_RX_INFO_PFET2); + + if (!mt76_poll_msec_tick(dev, MT_WFDMA0_GLO_CFG, + MT_WFDMA0_GLO_CFG_TX_DMA_BUSY | + MT_WFDMA0_GLO_CFG_RX_DMA_BUSY, 0, 100, 1)) + return -ETIMEDOUT; + + /* disable dmashdl */ + mt76_clear(dev, MT_WFDMA0_GLO_CFG_EXT0, + MT_WFDMA0_CSR_TX_DMASHDL_ENABLE); + mt76_set(dev, MT_DMASHDL_SW_CONTROL, MT_DMASHDL_DMASHDL_BYPASS); + if (force) { /* reset */ mt76_clear(dev, MT_WFDMA0_RST, @@ -77,24 +95,6 @@ static int mt7921_dma_disable(struct mt7921_dev *dev, bool force) MT_WFDMA0_RST_LOGIC_RST); } - /* disable dmashdl */ - mt76_clear(dev, MT_WFDMA0_GLO_CFG_EXT0, - MT_WFDMA0_CSR_TX_DMASHDL_ENABLE); - mt76_set(dev, MT_DMASHDL_SW_CONTROL, MT_DMASHDL_DMASHDL_BYPASS); - - /* disable WFDMA0 */ - mt76_clear(dev, MT_WFDMA0_GLO_CFG, - MT_WFDMA0_GLO_CFG_TX_DMA_EN | MT_WFDMA0_GLO_CFG_RX_DMA_EN | - MT_WFDMA0_GLO_CFG_CSR_DISP_BASE_PTR_CHAIN_EN | - MT_WFDMA0_GLO_CFG_OMIT_TX_INFO | - MT_WFDMA0_GLO_CFG_OMIT_RX_INFO | - MT_WFDMA0_GLO_CFG_OMIT_RX_INFO_PFET2); - - if (!mt76_poll(dev, MT_WFDMA0_GLO_CFG, - MT_WFDMA0_GLO_CFG_TX_DMA_BUSY | - MT_WFDMA0_GLO_CFG_RX_DMA_BUSY, 0, 1000)) - return -ETIMEDOUT; - return 0; } @@ -123,9 +123,9 @@ static int mt7921_dma_enable(struct mt7921_dev *dev) mt76_set(dev, MT_WFDMA_DUMMY_CR, MT_WFDMA_NEED_REINIT); /* enable interrupts for TX/RX rings */ - mt7921_irq_enable(dev, - MT_INT_RX_DONE_ALL | MT_INT_TX_DONE_ALL | - MT_INT_MCU_CMD); + mt76_connac_irq_enable(&dev->mt76, + MT_INT_RX_DONE_ALL | MT_INT_TX_DONE_ALL | + MT_INT_MCU_CMD); mt76_set(dev, MT_MCU2HOST_SW_INT_ENA, MT_MCU_CMD_WAKE_RX_PCIE); return 0; @@ -301,6 +301,10 @@ void mt7921_dma_cleanup(struct mt7921_dev *dev) MT_WFDMA0_GLO_CFG_OMIT_RX_INFO | MT_WFDMA0_GLO_CFG_OMIT_RX_INFO_PFET2); + mt76_poll_msec_tick(dev, MT_WFDMA0_GLO_CFG, + MT_WFDMA0_GLO_CFG_TX_DMA_BUSY | + MT_WFDMA0_GLO_CFG_RX_DMA_BUSY, 0, 100, 1); + /* reset */ mt76_clear(dev, MT_WFDMA0_RST, MT_WFDMA0_RST_DMASHDL_ALL_RST | diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/eeprom.h b/drivers/net/wireless/mediatek/mt76/mt7921/eeprom.h deleted file mode 100644 index 4b647278eb30..000000000000 --- a/drivers/net/wireless/mediatek/mt76/mt7921/eeprom.h +++ /dev/null @@ -1,30 +0,0 @@ -/* SPDX-License-Identifier: ISC */ -/* Copyright (C) 2020 MediaTek Inc. */ - -#ifndef __MT7921_EEPROM_H -#define __MT7921_EEPROM_H - -#include "mt7921.h" - -enum mt7921_eeprom_field { - MT_EE_CHIP_ID = 0x000, - MT_EE_VERSION = 0x002, - MT_EE_MAC_ADDR = 0x004, - MT_EE_WIFI_CONF = 0x07c, - MT_EE_HW_TYPE = 0x55b, - __MT_EE_MAX = 0x9ff -}; - -#define MT_EE_WIFI_CONF_TX_MASK BIT(0) -#define MT_EE_WIFI_CONF_BAND_SEL GENMASK(3, 2) - -#define MT_EE_HW_TYPE_ENCAP BIT(0) - -enum mt7921_eeprom_band { - MT_EE_NA, - MT_EE_5GHZ, - MT_EE_2GHZ, - MT_EE_DUAL_BAND, -}; - -#endif diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/init.c b/drivers/net/wireless/mediatek/mt76/mt7921/init.c index cc94531185da..bf1da9fddfab 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/init.c +++ b/drivers/net/wireless/mediatek/mt76/mt7921/init.c @@ -4,9 +4,8 @@ #include <linux/etherdevice.h> #include <linux/firmware.h> #include "mt7921.h" -#include "mac.h" +#include "../mt76_connac2_mac.h" #include "mcu.h" -#include "eeprom.h" static const struct ieee80211_iface_limit if_limits[] = { { @@ -32,11 +31,13 @@ static const struct ieee80211_iface_combination if_comb[] = { static const struct ieee80211_iface_limit if_limits_chanctx[] = { { .max = 2, - .types = BIT(NL80211_IFTYPE_STATION), + .types = BIT(NL80211_IFTYPE_STATION) | + BIT(NL80211_IFTYPE_P2P_CLIENT) }, { .max = 1, - .types = BIT(NL80211_IFTYPE_AP), + .types = BIT(NL80211_IFTYPE_AP) | + BIT(NL80211_IFTYPE_P2P_GO) } }; @@ -100,7 +101,9 @@ mt7921_init_wiphy(struct ieee80211_hw *hw) wiphy->flags &= ~(WIPHY_FLAG_IBSS_RSN | WIPHY_FLAG_4ADDR_AP | WIPHY_FLAG_4ADDR_STATION); wiphy->interface_modes = BIT(NL80211_IFTYPE_STATION) | - BIT(NL80211_IFTYPE_AP); + BIT(NL80211_IFTYPE_AP) | + BIT(NL80211_IFTYPE_P2P_CLIENT) | + BIT(NL80211_IFTYPE_P2P_GO); wiphy->max_remain_on_channel_duration = 5000; wiphy->max_scan_ie_len = MT76_CONNAC_SCAN_IE_LEN; wiphy->max_scan_ssids = 4; @@ -121,6 +124,7 @@ mt7921_init_wiphy(struct ieee80211_hw *hw) wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_BEACON_RATE_VHT); wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_BEACON_RATE_HE); wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_ACK_SIGNAL_SUPPORT); + wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_CAN_REPLACE_PTK0); ieee80211_hw_set(hw, SINGLE_SCAN_ON_ALL_BANDS); ieee80211_hw_set(hw, HAS_RATE_CONTROL); @@ -169,7 +173,8 @@ mt7921_mac_init_band(struct mt7921_dev *dev, u8 band) mt76_rmw(dev, MT_WTBLOFF_TOP_RSCR(band), mask, set); } -u8 mt7921_check_offload_capability(struct device *dev, const char *fw_wm) +static u8 +mt7921_get_offload_capability(struct device *dev, const char *fw_wm) { const struct mt76_connac2_fw_trailer *hdr; struct mt7921_realease_info *rel_info; @@ -223,7 +228,31 @@ out: return offload_caps; } -EXPORT_SYMBOL_GPL(mt7921_check_offload_capability); + +struct ieee80211_ops * +mt7921_get_mac80211_ops(struct device *dev, void *drv_data, u8 *fw_features) +{ + struct ieee80211_ops *ops; + + ops = devm_kmemdup(dev, &mt7921_ops, sizeof(mt7921_ops), GFP_KERNEL); + if (!ops) + return NULL; + + *fw_features = mt7921_get_offload_capability(dev, drv_data); + if (!(*fw_features & MT7921_FW_CAP_CNM)) { + ops->remain_on_channel = NULL; + ops->cancel_remain_on_channel = NULL; + ops->add_chanctx = NULL; + ops->remove_chanctx = NULL; + ops->change_chanctx = NULL; + ops->assign_vif_chanctx = NULL; + ops->unassign_vif_chanctx = NULL; + ops->mgd_prepare_tx = NULL; + ops->mgd_complete_tx = NULL; + } + return ops; +} +EXPORT_SYMBOL_GPL(mt7921_get_mac80211_ops); int mt7921_mac_init(struct mt7921_dev *dev) { diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/mac.c b/drivers/net/wireless/mediatek/mt76/mt7921/mac.c index 557c20190c2b..1675bf520481 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt7921/mac.c @@ -6,9 +6,20 @@ #include <linux/timekeeping.h> #include "mt7921.h" #include "../dma.h" -#include "mac.h" +#include "../mt76_connac2_mac.h" #include "mcu.h" +#define MT_WTBL_TXRX_CAP_RATE_OFFSET 7 +#define MT_WTBL_TXRX_RATE_G2_HE 24 +#define MT_WTBL_TXRX_RATE_G2 12 + +#define MT_WTBL_AC0_CTT_OFFSET 20 + +static u32 mt7921_mac_wtbl_lmac_addr(int idx, u8 offset) +{ + return MT_WTBL_LMAC_OFFS(idx, 0) + offset * 4; +} + static struct mt76_wcid *mt7921_rx_get_wcid(struct mt7921_dev *dev, u16 idx, bool unicast) { @@ -32,11 +43,6 @@ static struct mt76_wcid *mt7921_rx_get_wcid(struct mt7921_dev *dev, return &sta->vif->sta.wcid; } -void mt7921_sta_ps(struct mt76_dev *mdev, struct ieee80211_sta *sta, bool ps) -{ -} -EXPORT_SYMBOL_GPL(mt7921_sta_ps); - bool mt7921_mac_wtbl_update(struct mt7921_dev *dev, int idx, u32 mask) { mt76_rmw(dev, MT_WTBL_UPDATE, MT_WTBL_UPDATE_WLAN_IDX, diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/mac.h b/drivers/net/wireless/mediatek/mt76/mt7921/mac.h deleted file mode 100644 index 8afec600364f..000000000000 --- a/drivers/net/wireless/mediatek/mt76/mt7921/mac.h +++ /dev/null @@ -1,53 +0,0 @@ -/* SPDX-License-Identifier: ISC */ -/* Copyright (C) 2020 MediaTek Inc. */ - -#ifndef __MT7921_MAC_H -#define __MT7921_MAC_H - -#include "../mt76_connac2_mac.h" - -#define MT_CT_PARSE_LEN 72 -#define MT_CT_DMA_BUF_NUM 2 - -#define MT_RXD0_LENGTH GENMASK(15, 0) -#define MT_RXD0_PKT_FLAG GENMASK(19, 16) -#define MT_RXD0_PKT_TYPE GENMASK(31, 27) - -#define MT_RXD0_NORMAL_ETH_TYPE_OFS GENMASK(22, 16) -#define MT_RXD0_NORMAL_IP_SUM BIT(23) -#define MT_RXD0_NORMAL_UDP_TCP_SUM BIT(24) - -enum rx_pkt_type { - PKT_TYPE_TXS, - PKT_TYPE_TXRXV, - PKT_TYPE_NORMAL, - PKT_TYPE_RX_DUP_RFB, - PKT_TYPE_RX_TMR, - PKT_TYPE_RETRIEVE, - PKT_TYPE_TXRX_NOTIFY, - PKT_TYPE_RX_EVENT, - PKT_TYPE_NORMAL_MCU, -}; - -#define MT_TX_FREE_MSDU_CNT GENMASK(9, 0) -#define MT_TX_FREE_WLAN_ID GENMASK(23, 14) -#define MT_TX_FREE_LATENCY GENMASK(12, 0) -/* 0: success, others: dropped */ -#define MT_TX_FREE_STATUS GENMASK(14, 13) -#define MT_TX_FREE_MSDU_ID GENMASK(30, 16) -#define MT_TX_FREE_PAIR BIT(31) -/* will support this field in further revision */ -#define MT_TX_FREE_RATE GENMASK(13, 0) - -#define MT_WTBL_TXRX_CAP_RATE_OFFSET 7 -#define MT_WTBL_TXRX_RATE_G2_HE 24 -#define MT_WTBL_TXRX_RATE_G2 12 - -#define MT_WTBL_AC0_CTT_OFFSET 20 - -static inline u32 mt7921_mac_wtbl_lmac_addr(int idx, u8 offset) -{ - return MT_WTBL_LMAC_OFFS(idx, 0) + offset * 4; -} - -#endif diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/main.c b/drivers/net/wireless/mediatek/mt76/mt7921/main.c index 42933a6b7334..3b6adb29cbef 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/main.c +++ b/drivers/net/wireless/mediatek/mt76/mt7921/main.c @@ -9,27 +9,6 @@ #include "mt7921.h" #include "mcu.h" -static void -mt7921_gen_ppe_thresh(u8 *he_ppet, int nss) -{ - u8 i, ppet_bits, ppet_size, ru_bit_mask = 0x7; /* HE80 */ - static const u8 ppet16_ppet8_ru3_ru0[] = {0x1c, 0xc7, 0x71}; - - he_ppet[0] = FIELD_PREP(IEEE80211_PPE_THRES_NSS_MASK, nss - 1) | - FIELD_PREP(IEEE80211_PPE_THRES_RU_INDEX_BITMASK_MASK, - ru_bit_mask); - - ppet_bits = IEEE80211_PPE_THRES_INFO_PPET_SIZE * - nss * hweight8(ru_bit_mask) * 2; - ppet_size = DIV_ROUND_UP(ppet_bits, 8); - - for (i = 0; i < ppet_size - 1; i++) - he_ppet[i + 1] = ppet16_ppet8_ru3_ru0[i % 3]; - - he_ppet[i + 1] = ppet16_ppet8_ru3_ru0[i % 3] & - (0xff >> (8 - (ppet_bits - 1) % 8)); -} - static int mt7921_init_he_caps(struct mt7921_phy *phy, enum nl80211_band band, struct ieee80211_sband_iftype_data *data) @@ -168,7 +147,7 @@ mt7921_init_he_caps(struct mt7921_phy *phy, enum nl80211_band band, memset(he_cap->ppe_thres, 0, sizeof(he_cap->ppe_thres)); if (he_cap_elem->phy_cap_info[6] & IEEE80211_HE_PHY_CAP6_PPE_THRESHOLD_PRESENT) { - mt7921_gen_ppe_thresh(he_cap->ppe_thres, nss); + mt76_connac_gen_ppe_thresh(he_cap->ppe_thres, nss); } else { he_cap_elem->phy_cap_info[9] |= u8_encode_bits(IEEE80211_HE_PHY_CAP9_NOMINAL_PKT_PADDING_16US, @@ -702,10 +681,25 @@ static void mt7921_configure_filter(struct ieee80211_hw *hw, unsigned int *total_flags, u64 multicast) { +#define MT7921_FILTER_FCSFAIL BIT(2) +#define MT7921_FILTER_CONTROL BIT(5) +#define MT7921_FILTER_OTHER_BSS BIT(6) +#define MT7921_FILTER_ENABLE BIT(31) + struct mt7921_dev *dev = mt7921_hw_dev(hw); + u32 flags = MT7921_FILTER_ENABLE; + +#define MT7921_FILTER(_fif, _type) do { \ + if (*total_flags & (_fif)) \ + flags |= MT7921_FILTER_##_type; \ + } while (0) + + MT7921_FILTER(FIF_FCSFAIL, FCSFAIL); + MT7921_FILTER(FIF_CONTROL, CONTROL); + MT7921_FILTER(FIF_OTHER_BSS, OTHER_BSS); mt7921_mutex_acquire(dev); - mt7921_mcu_set_rxfilter(dev, *total_flags, 0, 0); + mt7921_mcu_set_rxfilter(dev, flags, 0, 0); mt7921_mutex_release(dev); *total_flags &= (FIF_OTHER_BSS | FIF_FCSFAIL | FIF_CONTROL); @@ -1694,7 +1688,7 @@ static void mt7921_ctx_iter(void *priv, u8 *mac, if (ctx != mvif->ctx) return; - if (vif->type & NL80211_IFTYPE_MONITOR) + if (vif->type == NL80211_IFTYPE_MONITOR) mt7921_mcu_config_sniffer(mvif, ctx); else mt76_connac_mcu_uni_set_chctx(mvif->phy->mt76, &mvif->mt76, ctx); diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/mcu.c b/drivers/net/wireless/mediatek/mt76/mt7921/mcu.c index c5e7ad06f877..c69ce6df4956 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/mcu.c +++ b/drivers/net/wireless/mediatek/mt76/mt7921/mcu.c @@ -5,9 +5,8 @@ #include <linux/firmware.h> #include "mt7921.h" #include "mt7921_trace.h" -#include "eeprom.h" #include "mcu.h" -#include "mac.h" +#include "../mt76_connac2_mac.h" #define MT_STA_BFER BIT(0) #define MT_STA_BFEE BIT(1) @@ -16,24 +15,6 @@ static bool mt7921_disable_clc; module_param_named(disable_clc, mt7921_disable_clc, bool, 0644); MODULE_PARM_DESC(disable_clc, "disable CLC support"); -static int -mt7921_mcu_parse_eeprom(struct mt76_dev *dev, struct sk_buff *skb) -{ - struct mt7921_mcu_eeprom_info *res; - u8 *buf; - - if (!skb) - return -EINVAL; - - skb_pull(skb, sizeof(struct mt76_connac2_mcu_rxd)); - - res = (struct mt7921_mcu_eeprom_info *)skb->data; - buf = dev->eeprom.data + le32_to_cpu(res->addr); - memcpy(buf, res->data, 16); - - return 0; -} - int mt7921_mcu_parse_response(struct mt76_dev *mdev, int cmd, struct sk_buff *skb, int seq) { @@ -60,27 +41,25 @@ int mt7921_mcu_parse_response(struct mt76_dev *mdev, int cmd, } else if (cmd == MCU_EXT_CMD(THERMAL_CTRL)) { skb_pull(skb, sizeof(*rxd) + 4); ret = le32_to_cpu(*(__le32 *)skb->data); - } else if (cmd == MCU_EXT_CMD(EFUSE_ACCESS)) { - ret = mt7921_mcu_parse_eeprom(mdev, skb); } else if (cmd == MCU_UNI_CMD(DEV_INFO_UPDATE) || cmd == MCU_UNI_CMD(BSS_INFO_UPDATE) || cmd == MCU_UNI_CMD(STA_REC_UPDATE) || cmd == MCU_UNI_CMD(HIF_CTRL) || cmd == MCU_UNI_CMD(OFFLOAD) || cmd == MCU_UNI_CMD(SUSPEND)) { - struct mt7921_mcu_uni_event *event; + struct mt76_connac_mcu_uni_event *event; skb_pull(skb, sizeof(*rxd)); - event = (struct mt7921_mcu_uni_event *)skb->data; + event = (struct mt76_connac_mcu_uni_event *)skb->data; ret = le32_to_cpu(event->status); /* skip invalid event */ if (mcu_cmd != event->cid) ret = -EAGAIN; } else if (cmd == MCU_CE_QUERY(REG_READ)) { - struct mt7921_mcu_reg_event *event; + struct mt76_connac_mcu_reg_event *event; skb_pull(skb, sizeof(*rxd)); - event = (struct mt7921_mcu_reg_event *)skb->data; + event = (struct mt76_connac_mcu_reg_event *)skb->data; ret = (int)le32_to_cpu(event->val); } else { skb_pull(skb, sizeof(struct mt76_connac2_mcu_rxd)); diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/mcu.h b/drivers/net/wireless/mediatek/mt76/mt7921/mcu.h index 96dc870fd35e..9b0aa3b70f0e 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/mcu.h +++ b/drivers/net/wireless/mediatek/mt76/mt7921/mcu.h @@ -50,22 +50,11 @@ struct mt7921_mcu_eeprom_info { #define MT_RA_RATE_DCM_EN BIT(4) #define MT_RA_RATE_BW GENMASK(14, 13) -struct mt7921_mcu_uni_event { - u8 cid; - u8 pad[3]; - __le32 status; /* 0: success, others: fail */ -} __packed; - enum { MT_EBF = BIT(0), /* explicit beamforming */ MT_IBF = BIT(1) /* implicit beamforming */ }; -struct mt7921_mcu_reg_event { - __le32 reg; - __le32 val; -} __packed; - struct mt7921_mcu_ant_id_config { u8 ant_id[4]; } __packed; diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/mt7921.h b/drivers/net/wireless/mediatek/mt76/mt7921/mt7921.h index 1af70dac723b..149acb1662d5 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/mt7921.h +++ b/drivers/net/wireless/mediatek/mt76/mt7921/mt7921.h @@ -266,6 +266,17 @@ struct mt7921_phy { bool roc_grant; }; +enum mt7921_eeprom_field { + MT_EE_CHIP_ID = 0x000, + MT_EE_VERSION = 0x002, + MT_EE_MAC_ADDR = 0x004, + MT_EE_WIFI_CONF = 0x07c, + MT_EE_HW_TYPE = 0x55b, + __MT_EE_MAX = 0x9ff +}; + +#define MT_EE_HW_TYPE_ENCAP BIT(0) + #define mt7921_init_reset(dev) ((dev)->hif_ops->init_reset(dev)) #define mt7921_dev_reset(dev) ((dev)->hif_ops->reset(dev)) #define mt7921_mcu_init(dev) ((dev)->hif_ops->mcu_init(dev)) @@ -287,7 +298,6 @@ struct mt7921_dev { const struct mt76_bus_ops *bus_ops; struct mt7921_phy phy; - struct tasklet_struct irq_tasklet; struct work_struct reset_work; bool hw_full_reset:1; @@ -391,13 +401,6 @@ void mt7921_mcu_rx_event(struct mt7921_dev *dev, struct sk_buff *skb); int mt7921_mcu_set_rxfilter(struct mt7921_dev *dev, u32 fif, u8 bit_op, u32 bit_map); -static inline void mt7921_irq_enable(struct mt7921_dev *dev, u32 mask) -{ - mt76_set_irq_mask(&dev->mt76, 0, 0, mask); - - tasklet_schedule(&dev->irq_tasklet); -} - static inline u32 mt7921_reg_map_l1(struct mt7921_dev *dev, u32 addr) { @@ -478,7 +481,6 @@ void mt7921_tx_token_put(struct mt7921_dev *dev); bool mt7921_rx_check(struct mt76_dev *mdev, void *data, int len); void mt7921_queue_rx_skb(struct mt76_dev *mdev, enum mt76_rxq_id q, struct sk_buff *skb, u32 *info); -void mt7921_sta_ps(struct mt76_dev *mdev, struct ieee80211_sta *sta, bool ps); void mt7921_stats_work(struct work_struct *work); void mt7921_set_stream_he_caps(struct mt7921_phy *phy); void mt7921_update_channel(struct mt76_phy *mphy); @@ -593,5 +595,6 @@ int mt7921_mcu_set_roc(struct mt7921_phy *phy, struct mt7921_vif *vif, enum mt7921_roc_req type, u8 token_id); int mt7921_mcu_abort_roc(struct mt7921_phy *phy, struct mt7921_vif *vif, u8 token_id); -u8 mt7921_check_offload_capability(struct device *dev, const char *fw_wm); +struct ieee80211_ops *mt7921_get_mac80211_ops(struct device *dev, + void *drv_data, u8 *fw_features); #endif diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/pci.c b/drivers/net/wireless/mediatek/mt76/mt7921/pci.c index 5c23c827abe4..ddb1fa4ee01d 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/pci.c +++ b/drivers/net/wireless/mediatek/mt76/mt7921/pci.c @@ -8,7 +8,7 @@ #include <linux/pci.h> #include "mt7921.h" -#include "mac.h" +#include "../mt76_connac2_mac.h" #include "mcu.h" #include "../trace.h" @@ -31,14 +31,12 @@ MODULE_PARM_DESC(disable_aspm, "disable PCI ASPM support"); static void mt7921_rx_poll_complete(struct mt76_dev *mdev, enum mt76_rxq_id q) { - struct mt7921_dev *dev = container_of(mdev, struct mt7921_dev, mt76); - if (q == MT_RXQ_MAIN) - mt7921_irq_enable(dev, MT_INT_RX_DONE_DATA); + mt76_connac_irq_enable(mdev, MT_INT_RX_DONE_DATA); else if (q == MT_RXQ_MCU_WA) - mt7921_irq_enable(dev, MT_INT_RX_DONE_WM2); + mt76_connac_irq_enable(mdev, MT_INT_RX_DONE_WM2); else - mt7921_irq_enable(dev, MT_INT_RX_DONE_WM); + mt76_connac_irq_enable(mdev, MT_INT_RX_DONE_WM); } static irqreturn_t mt7921_irq_handler(int irq, void *dev_instance) @@ -50,7 +48,7 @@ static irqreturn_t mt7921_irq_handler(int irq, void *dev_instance) if (!test_bit(MT76_STATE_INITIALIZED, &dev->mphy.state)) return IRQ_NONE; - tasklet_schedule(&dev->irq_tasklet); + tasklet_schedule(&dev->mt76.irq_tasklet); return IRQ_HANDLED; } @@ -115,14 +113,15 @@ static void mt7921e_unregister_device(struct mt7921_dev *dev) napi_disable(&dev->mt76.napi[i]); cancel_delayed_work_sync(&pm->ps_work); cancel_work_sync(&pm->wake_work); + cancel_work_sync(&dev->reset_work); mt7921_tx_token_put(dev); - mt7921_mcu_drv_pmctrl(dev); + __mt7921_mcu_drv_pmctrl(dev); mt7921_dma_cleanup(dev); mt7921_wfsys_reset(dev); skb_queue_purge(&dev->mt76.mcu.res_q); - tasklet_disable(&dev->irq_tasklet); + tasklet_disable(&dev->mt76.irq_tasklet); } static u32 __mt7921_reg_addr(struct mt7921_dev *dev, u32 addr) @@ -243,7 +242,6 @@ static int mt7921_pci_probe(struct pci_dev *pdev, .rx_check = mt7921_rx_check, .rx_skb = mt7921_queue_rx_skb, .rx_poll_complete = mt7921_rx_poll_complete, - .sta_ps = mt7921_sta_ps, .sta_add = mt7921_mac_sta_add, .sta_assoc = mt7921_mac_sta_assoc, .sta_remove = mt7921_mac_sta_remove, @@ -256,13 +254,13 @@ static int mt7921_pci_probe(struct pci_dev *pdev, .drv_own = mt7921e_mcu_drv_pmctrl, .fw_own = mt7921e_mcu_fw_pmctrl, }; - struct ieee80211_ops *ops; struct mt76_bus_ops *bus_ops; struct mt7921_dev *dev; struct mt76_dev *mdev; u8 features; int ret; + u16 cmd; ret = pcim_enable_device(pdev); if (ret) @@ -272,6 +270,11 @@ static int mt7921_pci_probe(struct pci_dev *pdev, if (ret) return ret; + pci_read_config_word(pdev, PCI_COMMAND, &cmd); + if (!(cmd & PCI_COMMAND_MEMORY)) { + cmd |= PCI_COMMAND_MEMORY; + pci_write_config_word(pdev, PCI_COMMAND, cmd); + } pci_set_master(pdev); ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_ALL_TYPES); @@ -285,27 +288,13 @@ static int mt7921_pci_probe(struct pci_dev *pdev, if (mt7921_disable_aspm) mt76_pci_disable_aspm(pdev); - features = mt7921_check_offload_capability(&pdev->dev, (const char *) - id->driver_data); - ops = devm_kmemdup(&pdev->dev, &mt7921_ops, sizeof(mt7921_ops), - GFP_KERNEL); + ops = mt7921_get_mac80211_ops(&pdev->dev, (void *)id->driver_data, + &features); if (!ops) { ret = -ENOMEM; goto err_free_pci_vec; } - if (!(features & MT7921_FW_CAP_CNM)) { - ops->remain_on_channel = NULL; - ops->cancel_remain_on_channel = NULL; - ops->add_chanctx = NULL; - ops->remove_chanctx = NULL; - ops->change_chanctx = NULL; - ops->assign_vif_chanctx = NULL; - ops->unassign_vif_chanctx = NULL; - ops->mgd_prepare_tx = NULL; - ops->mgd_complete_tx = NULL; - } - mdev = mt76_alloc_device(&pdev->dev, sizeof(*dev), ops, &drv_ops); if (!mdev) { ret = -ENOMEM; @@ -318,7 +307,7 @@ static int mt7921_pci_probe(struct pci_dev *pdev, dev->fw_features = features; dev->hif_ops = &mt7921_pcie_ops; mt76_mmio_init(&dev->mt76, pcim_iomap_table(pdev)[0]); - tasklet_init(&dev->irq_tasklet, mt7921_irq_tasklet, (unsigned long)dev); + tasklet_init(&mdev->irq_tasklet, mt7921_irq_tasklet, (unsigned long)dev); dev->phy.dev = dev; dev->phy.mt76 = &dev->mt76.phy; @@ -430,7 +419,7 @@ static int mt7921_pci_suspend(struct device *device) mt76_wr(dev, MT_WFDMA0_HOST_INT_ENA, 0); mt76_wr(dev, MT_PCIE_MAC_INT_ENABLE, 0x0); synchronize_irq(pdev->irq); - tasklet_kill(&dev->irq_tasklet); + tasklet_kill(&mdev->irq_tasklet); err = mt7921_mcu_fw_pmctrl(dev); if (err) @@ -474,8 +463,9 @@ static int mt7921_pci_resume(struct device *device) /* enable interrupt */ mt76_wr(dev, MT_PCIE_MAC_INT_ENABLE, 0xff); - mt7921_irq_enable(dev, MT_INT_RX_DONE_ALL | MT_INT_TX_DONE_ALL | - MT_INT_MCU_CMD); + mt76_connac_irq_enable(&dev->mt76, + MT_INT_RX_DONE_ALL | MT_INT_TX_DONE_ALL | + MT_INT_MCU_CMD); mt76_set(dev, MT_MCU2HOST_SW_INT_ENA, MT_MCU_CMD_WAKE_RX_PCIE); /* put dma enabled */ @@ -509,17 +499,7 @@ failed: static void mt7921_pci_shutdown(struct pci_dev *pdev) { - struct mt76_dev *mdev = pci_get_drvdata(pdev); - struct mt7921_dev *dev = container_of(mdev, struct mt7921_dev, mt76); - struct mt76_connac_pm *pm = &dev->pm; - - cancel_delayed_work_sync(&pm->ps_work); - cancel_work_sync(&pm->wake_work); - - /* chip cleanup before reboot */ - mt7921_mcu_drv_pmctrl(dev); - mt7921_dma_cleanup(dev); - mt7921_wfsys_reset(dev); + mt7921_pci_remove(pdev); } static DEFINE_SIMPLE_DEV_PM_OPS(mt7921_pm_ops, mt7921_pci_suspend, mt7921_pci_resume); diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/pci_mac.c b/drivers/net/wireless/mediatek/mt76/mt7921/pci_mac.c index 8dd60408b117..6053a2556c20 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/pci_mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt7921/pci_mac.c @@ -3,7 +3,7 @@ #include "mt7921.h" #include "../dma.h" -#include "mac.h" +#include "../mt76_connac2_mac.h" int mt7921e_tx_prepare_skb(struct mt76_dev *mdev, void *txwi_ptr, enum mt76_txq_id qid, struct mt76_wcid *wcid, diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/sdio.c b/drivers/net/wireless/mediatek/mt76/mt7921/sdio.c index 8ce4252b8ae7..a77a309c0d60 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/sdio.c +++ b/drivers/net/wireless/mediatek/mt76/mt7921/sdio.c @@ -13,7 +13,7 @@ #include "mt7921.h" #include "../sdio.h" -#include "mac.h" +#include "../mt76_connac2_mac.h" #include "mcu.h" static const struct sdio_device_id mt7921s_table[] = { @@ -99,7 +99,6 @@ static int mt7921s_probe(struct sdio_func *func, .tx_status_data = mt7921_usb_sdio_tx_status_data, .rx_skb = mt7921_queue_rx_skb, .rx_check = mt7921_rx_check, - .sta_ps = mt7921_sta_ps, .sta_add = mt7921_mac_sta_add, .sta_assoc = mt7921_mac_sta_assoc, .sta_remove = mt7921_mac_sta_remove, @@ -122,33 +121,17 @@ static int mt7921s_probe(struct sdio_func *func, .drv_own = mt7921s_mcu_drv_pmctrl, .fw_own = mt7921s_mcu_fw_pmctrl, }; - struct ieee80211_ops *ops; struct mt7921_dev *dev; struct mt76_dev *mdev; u8 features; int ret; - features = mt7921_check_offload_capability(&func->dev, (const char *) - id->driver_data); - - ops = devm_kmemdup(&func->dev, &mt7921_ops, sizeof(mt7921_ops), - GFP_KERNEL); + ops = mt7921_get_mac80211_ops(&func->dev, (void *)id->driver_data, + &features); if (!ops) return -ENOMEM; - if (!(features & MT7921_FW_CAP_CNM)) { - ops->remain_on_channel = NULL; - ops->cancel_remain_on_channel = NULL; - ops->add_chanctx = NULL; - ops->remove_chanctx = NULL; - ops->change_chanctx = NULL; - ops->assign_vif_chanctx = NULL; - ops->unassign_vif_chanctx = NULL; - ops->mgd_prepare_tx = NULL; - ops->mgd_complete_tx = NULL; - } - mdev = mt76_alloc_device(&func->dev, sizeof(*dev), ops, &drv_ops); if (!mdev) return -ENOMEM; diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/sdio_mac.c b/drivers/net/wireless/mediatek/mt76/mt7921/sdio_mac.c index 1b3adb3d91e8..cff9925c41ea 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/sdio_mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt7921/sdio_mac.c @@ -4,7 +4,7 @@ #include <linux/iopoll.h> #include <linux/mmc/sdio_func.h> #include "mt7921.h" -#include "mac.h" +#include "../mt76_connac2_mac.h" #include "../sdio.h" static void mt7921s_enable_irq(struct mt76_dev *dev) diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/sdio_mcu.c b/drivers/net/wireless/mediatek/mt76/mt7921/sdio_mcu.c index 5c1489766d9f..177679ce1c80 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/sdio_mcu.c +++ b/drivers/net/wireless/mediatek/mt76/mt7921/sdio_mcu.c @@ -8,7 +8,7 @@ #include "mt7921.h" #include "../sdio.h" -#include "mac.h" +#include "../mt76_connac2_mac.h" #include "mcu.h" #include "regs.h" diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/usb.c b/drivers/net/wireless/mediatek/mt76/mt7921/usb.c index 8fef09ed29c9..1f302c430339 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/usb.c +++ b/drivers/net/wireless/mediatek/mt76/mt7921/usb.c @@ -10,7 +10,7 @@ #include "mt7921.h" #include "mcu.h" -#include "mac.h" +#include "../mt76_connac2_mac.h" static const struct usb_device_id mt7921u_device_table[] = { { USB_DEVICE_AND_INTERFACE_INFO(0x0e8d, 0x7961, 0xff, 0xff, 0xff), @@ -18,6 +18,9 @@ static const struct usb_device_id mt7921u_device_table[] = { /* Comfast CF-952AX */ { USB_DEVICE_AND_INTERFACE_INFO(0x3574, 0x6211, 0xff, 0xff, 0xff), .driver_info = (kernel_ulong_t)MT7921_FIRMWARE_WM }, + /* Netgear, Inc. [A8000,AXE3000] */ + { USB_DEVICE_AND_INTERFACE_INFO(0x0846, 0x9060, 0xff, 0xff, 0xff), + .driver_info = (kernel_ulong_t)MT7921_FIRMWARE_WM }, { }, }; @@ -183,7 +186,6 @@ static int mt7921u_probe(struct usb_interface *usb_intf, .tx_status_data = mt7921_usb_sdio_tx_status_data, .rx_skb = mt7921_queue_rx_skb, .rx_check = mt7921_rx_check, - .sta_ps = mt7921_sta_ps, .sta_add = mt7921_mac_sta_add, .sta_assoc = mt7921_mac_sta_assoc, .sta_remove = mt7921_mac_sta_remove, @@ -210,27 +212,12 @@ static int mt7921u_probe(struct usb_interface *usb_intf, u8 features; int ret; - features = mt7921_check_offload_capability(&usb_intf->dev, (const char *) - id->driver_info); - ops = devm_kmemdup(&usb_intf->dev, &mt7921_ops, sizeof(mt7921_ops), - GFP_KERNEL); + ops = mt7921_get_mac80211_ops(&usb_intf->dev, (void *)id->driver_info, + &features); if (!ops) return -ENOMEM; - if (!(features & MT7921_FW_CAP_CNM)) { - ops->remain_on_channel = NULL; - ops->cancel_remain_on_channel = NULL; - ops->add_chanctx = NULL; - ops->remove_chanctx = NULL; - ops->change_chanctx = NULL; - ops->assign_vif_chanctx = NULL; - ops->unassign_vif_chanctx = NULL; - ops->mgd_prepare_tx = NULL; - ops->mgd_complete_tx = NULL; - } - ops->stop = mt7921u_stop; - mdev = mt76_alloc_device(&usb_intf->dev, sizeof(*dev), ops, &drv_ops); if (!mdev) return -ENOMEM; @@ -272,7 +259,7 @@ static int mt7921u_probe(struct usb_interface *usb_intf, ret = mt7921u_dma_init(dev, false); if (ret) - return ret; + goto error; hw = mt76_hw(dev); /* check hw sg support in order to enable AMSDU */ diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/usb_mac.c b/drivers/net/wireless/mediatek/mt76/mt7921/usb_mac.c index efbd3954c883..50eb6e7fd6b5 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/usb_mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt7921/usb_mac.c @@ -10,7 +10,7 @@ #include "mt7921.h" #include "mcu.h" -#include "mac.h" +#include "../mt76_connac2_mac.h" static u32 mt7921u_uhw_rr(struct mt76_dev *dev, u32 addr) { diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/Kconfig b/drivers/net/wireless/mediatek/mt76/mt7996/Kconfig index 79fb47a73c91..1afa2f662e47 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/Kconfig +++ b/drivers/net/wireless/mediatek/mt76/mt7996/Kconfig @@ -2,6 +2,7 @@ config MT7996E tristate "MediaTek MT7996 (PCIe) support" select MT76_CONNAC_LIB + select WANT_DEV_COREDUMP select RELAY depends on MAC80211 depends on PCI diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/Makefile b/drivers/net/wireless/mediatek/mt76/mt7996/Makefile index bcb9a3c53149..07c8b555c1ac 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/Makefile +++ b/drivers/net/wireless/mediatek/mt76/mt7996/Makefile @@ -4,3 +4,5 @@ obj-$(CONFIG_MT7996E) += mt7996e.o mt7996e-y := pci.o init.o dma.o eeprom.o main.o mcu.o mac.o \ debugfs.o mmio.o + +mt7996e-$(CONFIG_DEV_COREDUMP) += coredump.o diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/coredump.c b/drivers/net/wireless/mediatek/mt76/mt7996/coredump.c new file mode 100644 index 000000000000..ccab0d7b9be4 --- /dev/null +++ b/drivers/net/wireless/mediatek/mt76/mt7996/coredump.c @@ -0,0 +1,268 @@ +// SPDX-License-Identifier: ISC +/* Copyright (C) 2023 MediaTek Inc. */ + +#include <linux/devcoredump.h> +#include <linux/kernel.h> +#include <linux/types.h> +#include <linux/utsname.h> +#include "coredump.h" + +static bool coredump_memdump; +module_param(coredump_memdump, bool, 0644); +MODULE_PARM_DESC(coredump_memdump, "Optional ability to dump firmware memory"); + +static const struct mt7996_mem_region mt7996_mem_regions[] = { + { + .start = 0x00800000, + .len = 0x0004ffff, + .name = "ULM0", + }, + { + .start = 0x00900000, + .len = 0x00037fff, + .name = "ULM1", + }, + { + .start = 0x02200000, + .len = 0x0003ffff, + .name = "ULM2", + }, + { + .start = 0x00400000, + .len = 0x00067fff, + .name = "SRAM", + }, + { + .start = 0xe0000000, + .len = 0x0015ffff, + .name = "CRAM0", + }, + { + .start = 0xe0160000, + .len = 0x0011bfff, + .name = "CRAM1", + }, +}; + +const struct mt7996_mem_region* +mt7996_coredump_get_mem_layout(struct mt7996_dev *dev, u32 *num) +{ + switch (mt76_chip(&dev->mt76)) { + case 0x7990: + case 0x7991: + *num = ARRAY_SIZE(mt7996_mem_regions); + return &mt7996_mem_regions[0]; + default: + return NULL; + } +} + +static int mt7996_coredump_get_mem_size(struct mt7996_dev *dev) +{ + const struct mt7996_mem_region *mem_region; + size_t size = 0; + u32 num; + int i; + + mem_region = mt7996_coredump_get_mem_layout(dev, &num); + if (!mem_region) + return 0; + + for (i = 0; i < num; i++) { + size += mem_region->len; + mem_region++; + } + + /* reserve space for the headers */ + size += num * sizeof(struct mt7996_mem_hdr); + /* make sure it is aligned 4 bytes for debug message print out */ + size = ALIGN(size, 4); + + return size; +} + +struct mt7996_crash_data *mt7996_coredump_new(struct mt7996_dev *dev) +{ + struct mt7996_crash_data *crash_data = dev->coredump.crash_data; + + lockdep_assert_held(&dev->dump_mutex); + + if (coredump_memdump && + !mt76_poll_msec(dev, MT_FW_DUMP_STATE, 0x3, 0x2, 500)) + return NULL; + + guid_gen(&crash_data->guid); + ktime_get_real_ts64(&crash_data->timestamp); + + return crash_data; +} + +static void +mt7996_coredump_fw_state(struct mt7996_dev *dev, struct mt7996_coredump *dump, + bool *exception) +{ + u32 count; + + count = mt76_rr(dev, MT_FW_ASSERT_CNT); + + /* normal mode: driver can manually trigger assert for detail info */ + if (!count) + strscpy(dump->fw_state, "normal", sizeof(dump->fw_state)); + else + strscpy(dump->fw_state, "exception", sizeof(dump->fw_state)); + + *exception = !!count; +} + +static void +mt7996_coredump_fw_stack(struct mt7996_dev *dev, struct mt7996_coredump *dump, + bool exception) +{ + u32 oldest, i, idx; + + strscpy(dump->pc_current, "program counter", sizeof(dump->pc_current)); + + /* 0: WM PC log output */ + mt76_wr(dev, MT_CONN_DBG_CTL_OUT_SEL, 0); + /* choose 33th PC log buffer to read current PC index */ + mt76_wr(dev, MT_CONN_DBG_CTL_PC_LOG_SEL, 0x3f); + + /* read current PC */ + dump->pc_stack[0] = mt76_rr(dev, MT_CONN_DBG_CTL_PC_LOG); + + /* stop call stack record */ + if (!exception) { + mt76_clear(dev, MT_MCU_WM_EXCP_PC_CTRL, BIT(0)); + mt76_clear(dev, MT_MCU_WM_EXCP_LR_CTRL, BIT(0)); + } + + oldest = (u32)mt76_get_field(dev, MT_MCU_WM_EXCP_PC_CTRL, + GENMASK(20, 16)) + 2; + for (i = 0; i < 16; i++) { + idx = ((oldest + 2 * i + 1) % 32); + dump->pc_stack[i + 1] = + mt76_rr(dev, MT_MCU_WM_EXCP_PC_LOG + idx * 4); + } + + oldest = (u32)mt76_get_field(dev, MT_MCU_WM_EXCP_LR_CTRL, + GENMASK(20, 16)) + 2; + for (i = 0; i < 16; i++) { + idx = ((oldest + 2 * i + 1) % 32); + dump->lr_stack[i] = + mt76_rr(dev, MT_MCU_WM_EXCP_LR_LOG + idx * 4); + } + + /* start call stack record */ + if (!exception) { + mt76_set(dev, MT_MCU_WM_EXCP_PC_CTRL, BIT(0)); + mt76_set(dev, MT_MCU_WM_EXCP_LR_CTRL, BIT(0)); + } +} + +static struct mt7996_coredump *mt7996_coredump_build(struct mt7996_dev *dev) +{ + struct mt7996_crash_data *crash_data = dev->coredump.crash_data; + struct mt7996_coredump *dump; + struct mt7996_coredump_mem *dump_mem; + size_t len, sofar = 0, hdr_len = sizeof(*dump); + unsigned char *buf; + bool exception; + + len = hdr_len; + + if (coredump_memdump && crash_data->memdump_buf_len) + len += sizeof(*dump_mem) + crash_data->memdump_buf_len; + + sofar += hdr_len; + + /* this is going to get big when we start dumping memory and such, + * so go ahead and use vmalloc. + */ + buf = vzalloc(len); + if (!buf) + return NULL; + + mutex_lock(&dev->dump_mutex); + + dump = (struct mt7996_coredump *)(buf); + dump->len = len; + + /* plain text */ + strscpy(dump->magic, "mt76-crash-dump", sizeof(dump->magic)); + strscpy(dump->kernel, init_utsname()->release, sizeof(dump->kernel)); + strscpy(dump->fw_ver, dev->mt76.hw->wiphy->fw_version, + sizeof(dump->fw_ver)); + + guid_copy(&dump->guid, &crash_data->guid); + dump->tv_sec = crash_data->timestamp.tv_sec; + dump->tv_nsec = crash_data->timestamp.tv_nsec; + dump->device_id = mt76_chip(&dev->mt76); + + mt7996_coredump_fw_state(dev, dump, &exception); + mt7996_coredump_fw_stack(dev, dump, exception); + + /* gather memory content */ + dump_mem = (struct mt7996_coredump_mem *)(buf + sofar); + dump_mem->len = crash_data->memdump_buf_len; + if (coredump_memdump && crash_data->memdump_buf_len) + memcpy(dump_mem->data, crash_data->memdump_buf, + crash_data->memdump_buf_len); + + mutex_unlock(&dev->dump_mutex); + + return dump; +} + +int mt7996_coredump_submit(struct mt7996_dev *dev) +{ + struct mt7996_coredump *dump; + + dump = mt7996_coredump_build(dev); + if (!dump) { + dev_warn(dev->mt76.dev, "no crash dump data found\n"); + return -ENODATA; + } + + dev_coredumpv(dev->mt76.dev, dump, dump->len, GFP_KERNEL); + + return 0; +} + +int mt7996_coredump_register(struct mt7996_dev *dev) +{ + struct mt7996_crash_data *crash_data; + + crash_data = vzalloc(sizeof(*dev->coredump.crash_data)); + if (!crash_data) + return -ENOMEM; + + dev->coredump.crash_data = crash_data; + + if (coredump_memdump) { + crash_data->memdump_buf_len = mt7996_coredump_get_mem_size(dev); + if (!crash_data->memdump_buf_len) + /* no memory content */ + return 0; + + crash_data->memdump_buf = vzalloc(crash_data->memdump_buf_len); + if (!crash_data->memdump_buf) { + vfree(crash_data); + return -ENOMEM; + } + } + + return 0; +} + +void mt7996_coredump_unregister(struct mt7996_dev *dev) +{ + if (dev->coredump.crash_data->memdump_buf) { + vfree(dev->coredump.crash_data->memdump_buf); + dev->coredump.crash_data->memdump_buf = NULL; + dev->coredump.crash_data->memdump_buf_len = 0; + } + + vfree(dev->coredump.crash_data); + dev->coredump.crash_data = NULL; +} + diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/coredump.h b/drivers/net/wireless/mediatek/mt76/mt7996/coredump.h new file mode 100644 index 000000000000..af2ba219b1b5 --- /dev/null +++ b/drivers/net/wireless/mediatek/mt76/mt7996/coredump.h @@ -0,0 +1,97 @@ +/* SPDX-License-Identifier: ISC */ +/* Copyright (C) 2023 MediaTek Inc. */ + +#ifndef _COREDUMP_H_ +#define _COREDUMP_H_ + +#include "mt7996.h" + +struct mt7996_coredump { + char magic[16]; + + u32 len; + + guid_t guid; + + /* time-of-day stamp */ + u64 tv_sec; + /* time-of-day stamp, nano-seconds */ + u64 tv_nsec; + /* kernel version */ + char kernel[64]; + /* firmware version */ + char fw_ver[ETHTOOL_FWVERS_LEN]; + + u32 device_id; + + /* exception state */ + char fw_state[12]; + + /* program counters */ + char pc_current[16]; + u32 pc_stack[17]; + /* link registers */ + u32 lr_stack[16]; + + /* memory content */ + u8 data[]; +} __packed; + +struct mt7996_coredump_mem { + u32 len; + u8 data[]; +} __packed; + +struct mt7996_mem_hdr { + u32 start; + u32 len; + u8 data[]; +}; + +struct mt7996_mem_region { + u32 start; + size_t len; + + const char *name; +}; + +#ifdef CONFIG_DEV_COREDUMP + +const struct mt7996_mem_region * +mt7996_coredump_get_mem_layout(struct mt7996_dev *dev, u32 *num); +struct mt7996_crash_data *mt7996_coredump_new(struct mt7996_dev *dev); +int mt7996_coredump_submit(struct mt7996_dev *dev); +int mt7996_coredump_register(struct mt7996_dev *dev); +void mt7996_coredump_unregister(struct mt7996_dev *dev); + +#else /* CONFIG_DEV_COREDUMP */ + +static inline const struct mt7996_mem_region * +mt7996_coredump_get_mem_layout(struct mt7996_dev *dev, u32 *num) +{ + return NULL; +} + +static inline int mt7996_coredump_submit(struct mt7996_dev *dev) +{ + return 0; +} + +static inline struct +mt7996_crash_data *mt7996_coredump_new(struct mt7996_dev *dev) +{ + return NULL; +} + +static inline int mt7996_coredump_register(struct mt7996_dev *dev) +{ + return 0; +} + +static inline void mt7996_coredump_unregister(struct mt7996_dev *dev) +{ +} + +#endif /* CONFIG_DEV_COREDUMP */ + +#endif /* _COREDUMP_H_ */ diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/debugfs.c b/drivers/net/wireless/mediatek/mt76/mt7996/debugfs.c index 9c5e9ac1c335..513ab4ba41c9 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/debugfs.c +++ b/drivers/net/wireless/mediatek/mt76/mt7996/debugfs.c @@ -48,12 +48,12 @@ DEFINE_DEBUGFS_ATTRIBUTE(fops_implicit_txbf, mt7996_implicit_txbf_get, /* test knob of system error recovery */ static ssize_t -mt7996_fw_ser_set(struct file *file, const char __user *user_buf, - size_t count, loff_t *ppos) +mt7996_sys_recovery_set(struct file *file, const char __user *user_buf, + size_t count, loff_t *ppos) { struct mt7996_phy *phy = file->private_data; struct mt7996_dev *dev = phy->dev; - u8 band_idx = phy->mt76->band_idx; + bool band = phy->mt76->band_idx; char buf[16]; int ret = 0; u16 val; @@ -73,17 +73,47 @@ mt7996_fw_ser_set(struct file *file, const char __user *user_buf, return -EINVAL; switch (val) { - case SER_SET_RECOVER_L1: - case SER_SET_RECOVER_L2: - case SER_SET_RECOVER_L3_RX_ABORT: - case SER_SET_RECOVER_L3_TX_ABORT: - case SER_SET_RECOVER_L3_TX_DISABLE: - case SER_SET_RECOVER_L3_BF: - ret = mt7996_mcu_set_ser(dev, SER_ENABLE, BIT(val), band_idx); + /* + * 0: grab firmware current SER state. + * 1: trigger & enable system error L1 recovery. + * 2: trigger & enable system error L2 recovery. + * 3: trigger & enable system error L3 rx abort. + * 4: trigger & enable system error L3 tx abort + * 5: trigger & enable system error L3 tx disable. + * 6: trigger & enable system error L3 bf recovery. + * 7: trigger & enable system error L4 mdp recovery. + * 8: trigger & enable system error full recovery. + * 9: trigger firmware crash. + */ + case UNI_CMD_SER_QUERY: + ret = mt7996_mcu_set_ser(dev, UNI_CMD_SER_QUERY, 0, band); + break; + case UNI_CMD_SER_SET_RECOVER_L1: + case UNI_CMD_SER_SET_RECOVER_L2: + case UNI_CMD_SER_SET_RECOVER_L3_RX_ABORT: + case UNI_CMD_SER_SET_RECOVER_L3_TX_ABORT: + case UNI_CMD_SER_SET_RECOVER_L3_TX_DISABLE: + case UNI_CMD_SER_SET_RECOVER_L3_BF: + case UNI_CMD_SER_SET_RECOVER_L4_MDP: + ret = mt7996_mcu_set_ser(dev, UNI_CMD_SER_SET, BIT(val), band); if (ret) return ret; - ret = mt7996_mcu_set_ser(dev, SER_RECOVER, val, band_idx); + ret = mt7996_mcu_set_ser(dev, UNI_CMD_SER_TRIGGER, val, band); + break; + + /* enable full chip reset */ + case UNI_CMD_SER_SET_RECOVER_FULL: + mt76_set(dev, MT_WFDMA0_MCU_HOST_INT_ENA, MT_MCU_CMD_WDT_MASK); + dev->recovery.state |= MT_MCU_CMD_WDT_MASK; + mt7996_reset(dev); + break; + + /* WARNING: trigger firmware crash */ + case UNI_CMD_SER_SET_SYSTEM_ASSERT: + ret = mt7996_mcu_trigger_assert(dev); + if (ret) + return ret; break; default: break; @@ -92,9 +122,97 @@ mt7996_fw_ser_set(struct file *file, const char __user *user_buf, return ret ? ret : count; } -static const struct file_operations mt7996_fw_ser_ops = { - .write = mt7996_fw_ser_set, - /* TODO: ser read */ +static ssize_t +mt7996_sys_recovery_get(struct file *file, char __user *user_buf, + size_t count, loff_t *ppos) +{ + struct mt7996_phy *phy = file->private_data; + struct mt7996_dev *dev = phy->dev; + char *buff; + int desc = 0; + ssize_t ret; + static const size_t bufsz = 1024; + + buff = kmalloc(bufsz, GFP_KERNEL); + if (!buff) + return -ENOMEM; + + /* HELP */ + desc += scnprintf(buff + desc, bufsz - desc, + "Please echo the correct value ...\n"); + desc += scnprintf(buff + desc, bufsz - desc, + "0: grab firmware transient SER state\n"); + desc += scnprintf(buff + desc, bufsz - desc, + "1: trigger system error L1 recovery\n"); + desc += scnprintf(buff + desc, bufsz - desc, + "2: trigger system error L2 recovery\n"); + desc += scnprintf(buff + desc, bufsz - desc, + "3: trigger system error L3 rx abort\n"); + desc += scnprintf(buff + desc, bufsz - desc, + "4: trigger system error L3 tx abort\n"); + desc += scnprintf(buff + desc, bufsz - desc, + "5: trigger system error L3 tx disable\n"); + desc += scnprintf(buff + desc, bufsz - desc, + "6: trigger system error L3 bf recovery\n"); + desc += scnprintf(buff + desc, bufsz - desc, + "7: trigger system error L4 mdp recovery\n"); + desc += scnprintf(buff + desc, bufsz - desc, + "8: trigger system error full recovery\n"); + desc += scnprintf(buff + desc, bufsz - desc, + "9: trigger firmware crash\n"); + + /* SER statistics */ + desc += scnprintf(buff + desc, bufsz - desc, + "\nlet's dump firmware SER statistics...\n"); + desc += scnprintf(buff + desc, bufsz - desc, + "::E R , SER_STATUS = 0x%08x\n", + mt76_rr(dev, MT_SWDEF_SER_STATS)); + desc += scnprintf(buff + desc, bufsz - desc, + "::E R , SER_PLE_ERR = 0x%08x\n", + mt76_rr(dev, MT_SWDEF_PLE_STATS)); + desc += scnprintf(buff + desc, bufsz - desc, + "::E R , SER_PLE_ERR_1 = 0x%08x\n", + mt76_rr(dev, MT_SWDEF_PLE1_STATS)); + desc += scnprintf(buff + desc, bufsz - desc, + "::E R , SER_PLE_ERR_AMSDU = 0x%08x\n", + mt76_rr(dev, MT_SWDEF_PLE_AMSDU_STATS)); + desc += scnprintf(buff + desc, bufsz - desc, + "::E R , SER_PSE_ERR = 0x%08x\n", + mt76_rr(dev, MT_SWDEF_PSE_STATS)); + desc += scnprintf(buff + desc, bufsz - desc, + "::E R , SER_PSE_ERR_1 = 0x%08x\n", + mt76_rr(dev, MT_SWDEF_PSE1_STATS)); + desc += scnprintf(buff + desc, bufsz - desc, + "::E R , SER_LMAC_WISR6_B0 = 0x%08x\n", + mt76_rr(dev, MT_SWDEF_LAMC_WISR6_BN0_STATS)); + desc += scnprintf(buff + desc, bufsz - desc, + "::E R , SER_LMAC_WISR6_B1 = 0x%08x\n", + mt76_rr(dev, MT_SWDEF_LAMC_WISR6_BN1_STATS)); + desc += scnprintf(buff + desc, bufsz - desc, + "::E R , SER_LMAC_WISR6_B2 = 0x%08x\n", + mt76_rr(dev, MT_SWDEF_LAMC_WISR6_BN2_STATS)); + desc += scnprintf(buff + desc, bufsz - desc, + "::E R , SER_LMAC_WISR7_B0 = 0x%08x\n", + mt76_rr(dev, MT_SWDEF_LAMC_WISR7_BN0_STATS)); + desc += scnprintf(buff + desc, bufsz - desc, + "::E R , SER_LMAC_WISR7_B1 = 0x%08x\n", + mt76_rr(dev, MT_SWDEF_LAMC_WISR7_BN1_STATS)); + desc += scnprintf(buff + desc, bufsz - desc, + "::E R , SER_LMAC_WISR7_B2 = 0x%08x\n", + mt76_rr(dev, MT_SWDEF_LAMC_WISR7_BN2_STATS)); + desc += scnprintf(buff + desc, bufsz - desc, + "\nSYS_RESET_COUNT: WM %d, WA %d\n", + dev->recovery.wm_reset_count, + dev->recovery.wa_reset_count); + + ret = simple_read_from_buffer(user_buf, count, ppos, buff, desc); + kfree(buff); + return ret; +} + +static const struct file_operations mt7996_sys_recovery_ops = { + .write = mt7996_sys_recovery_set, + .read = mt7996_sys_recovery_get, .open = simple_open, .llseek = default_llseek, }; @@ -674,6 +792,8 @@ int mt7996_init_debugfs(struct mt7996_phy *phy) debugfs_create_file("xmit-queues", 0400, dir, phy, &mt7996_xmit_queues_fops); debugfs_create_file("tx_stats", 0400, dir, phy, &mt7996_tx_stats_fops); + debugfs_create_file("sys_recovery", 0600, dir, phy, + &mt7996_sys_recovery_ops); debugfs_create_file("fw_debug_wm", 0600, dir, dev, &fops_fw_debug_wm); debugfs_create_file("fw_debug_wa", 0600, dir, dev, &fops_fw_debug_wa); debugfs_create_file("fw_debug_bin", 0600, dir, dev, &fops_fw_debug_bin); @@ -684,7 +804,6 @@ int mt7996_init_debugfs(struct mt7996_phy *phy) &fops_implicit_txbf); debugfs_create_devm_seqfile(dev->mt76.dev, "twt_stats", dir, mt7996_twt_stats); - debugfs_create_file("fw_ser", 0600, dir, phy, &mt7996_fw_ser_ops); debugfs_create_file("rf_regval", 0600, dir, dev, &fops_rf_regval); if (phy->mt76->cap.has_5ghz) { diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/dma.c b/drivers/net/wireless/mediatek/mt76/mt7996/dma.c index c09fe4274935..534143465d9b 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/dma.c +++ b/drivers/net/wireless/mediatek/mt76/mt7996/dma.c @@ -352,6 +352,70 @@ int mt7996_dma_init(struct mt7996_dev *dev) return 0; } +void mt7996_dma_reset(struct mt7996_dev *dev, bool force) +{ + struct mt76_phy *phy2 = dev->mt76.phys[MT_BAND1]; + struct mt76_phy *phy3 = dev->mt76.phys[MT_BAND2]; + u32 hif1_ofs = MT_WFDMA0_PCIE1(0) - MT_WFDMA0(0); + int i; + + mt76_clear(dev, MT_WFDMA0_GLO_CFG, + MT_WFDMA0_GLO_CFG_TX_DMA_EN | + MT_WFDMA0_GLO_CFG_RX_DMA_EN); + + if (dev->hif2) + mt76_clear(dev, MT_WFDMA0_GLO_CFG + hif1_ofs, + MT_WFDMA0_GLO_CFG_TX_DMA_EN | + MT_WFDMA0_GLO_CFG_RX_DMA_EN); + + usleep_range(1000, 2000); + + for (i = 0; i < __MT_TXQ_MAX; i++) { + mt76_queue_tx_cleanup(dev, dev->mphy.q_tx[i], true); + if (phy2) + mt76_queue_tx_cleanup(dev, phy2->q_tx[i], true); + if (phy3) + mt76_queue_tx_cleanup(dev, phy3->q_tx[i], true); + } + + for (i = 0; i < __MT_MCUQ_MAX; i++) + mt76_queue_tx_cleanup(dev, dev->mt76.q_mcu[i], true); + + mt76_for_each_q_rx(&dev->mt76, i) + mt76_queue_rx_cleanup(dev, &dev->mt76.q_rx[i]); + + mt76_tx_status_check(&dev->mt76, true); + + /* reset wfsys */ + if (force) + mt7996_wfsys_reset(dev); + + mt7996_dma_disable(dev, force); + + /* reset hw queues */ + for (i = 0; i < __MT_TXQ_MAX; i++) { + mt76_queue_reset(dev, dev->mphy.q_tx[i]); + if (phy2) + mt76_queue_reset(dev, phy2->q_tx[i]); + if (phy3) + mt76_queue_reset(dev, phy3->q_tx[i]); + } + + for (i = 0; i < __MT_MCUQ_MAX; i++) + mt76_queue_reset(dev, dev->mt76.q_mcu[i]); + + mt76_for_each_q_rx(&dev->mt76, i) { + mt76_queue_reset(dev, &dev->mt76.q_rx[i]); + } + + mt76_tx_status_check(&dev->mt76, true); + + mt76_for_each_q_rx(&dev->mt76, i) + mt76_queue_rx_reset(dev, i); + + mt7996_dma_enable(dev); +} + void mt7996_dma_cleanup(struct mt7996_dev *dev) { mt7996_dma_disable(dev, true); diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/eeprom.c b/drivers/net/wireless/mediatek/mt76/mt7996/eeprom.c index 2e48c5a40f81..544b6c6f1ea3 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/eeprom.c +++ b/drivers/net/wireless/mediatek/mt76/mt7996/eeprom.c @@ -138,10 +138,6 @@ static int mt7996_eeprom_parse_band_config(struct mt7996_phy *phy) case MT_EE_BAND_SEL_6GHZ: phy->mt76->cap.has_6ghz = true; break; - case MT_EE_BAND_SEL_5GHZ_6GHZ: - phy->mt76->cap.has_5ghz = true; - phy->mt76->cap.has_6ghz = true; - break; default: ret = -EINVAL; break; diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/eeprom.h b/drivers/net/wireless/mediatek/mt76/mt7996/eeprom.h index 8da599e0abea..0c749774f6b1 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/eeprom.h +++ b/drivers/net/wireless/mediatek/mt76/mt7996/eeprom.h @@ -31,11 +31,11 @@ enum mt7996_eeprom_field { #define MT_EE_WIFI_CONF2_BAND_SEL GENMASK(2, 0) #define MT_EE_WIFI_CONF1_TX_PATH_BAND0 GENMASK(5, 3) -#define MT_EE_WIFI_CONF2_TX_PATH_BAND1 GENMASK(5, 3) -#define MT_EE_WIFI_CONF2_TX_PATH_BAND2 GENMASK(2, 0) +#define MT_EE_WIFI_CONF2_TX_PATH_BAND1 GENMASK(2, 0) +#define MT_EE_WIFI_CONF2_TX_PATH_BAND2 GENMASK(5, 3) #define MT_EE_WIFI_CONF4_STREAM_NUM_BAND0 GENMASK(5, 3) -#define MT_EE_WIFI_CONF5_STREAM_NUM_BAND1 GENMASK(5, 3) -#define MT_EE_WIFI_CONF5_STREAM_NUM_BAND2 GENMASK(2, 0) +#define MT_EE_WIFI_CONF5_STREAM_NUM_BAND1 GENMASK(2, 0) +#define MT_EE_WIFI_CONF5_STREAM_NUM_BAND2 GENMASK(5, 3) #define MT_EE_RATE_DELTA_MASK GENMASK(5, 0) #define MT_EE_RATE_DELTA_SIGN BIT(6) @@ -46,7 +46,6 @@ enum mt7996_eeprom_band { MT_EE_BAND_SEL_2GHZ, MT_EE_BAND_SEL_5GHZ, MT_EE_BAND_SEL_6GHZ, - MT_EE_BAND_SEL_5GHZ_6GHZ, }; static inline int diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/init.c b/drivers/net/wireless/mediatek/mt76/mt7996/init.c index 946da93eed32..f1b48cdda58f 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/init.c +++ b/drivers/net/wireless/mediatek/mt76/mt7996/init.c @@ -8,6 +8,7 @@ #include "mt7996.h" #include "mac.h" #include "mcu.h" +#include "coredump.h" #include "eeprom.h" static const struct ieee80211_iface_limit if_limits[] = { @@ -99,9 +100,8 @@ static void mt7996_led_set_brightness(struct led_classdev *led_cdev, mt7996_led_set_config(led_cdev, 0xff, 0); } -static void -mt7996_init_txpower(struct mt7996_dev *dev, - struct ieee80211_supported_band *sband) +void mt7996_init_txpower(struct mt7996_dev *dev, + struct ieee80211_supported_band *sband) { int i, nss = hweight8(dev->mphy.antenna_mask); int nss_delta = mt76_tx_power_nss_delta(nss); @@ -182,6 +182,7 @@ mt7996_init_wiphy(struct ieee80211_hw *hw) wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_UNSOL_BCAST_PROBE_RESP); wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_FILS_DISCOVERY); wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_ACK_SIGNAL_SUPPORT); + wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_CAN_REPLACE_PTK0); if (!mdev->dev->of_node || !of_property_read_bool(mdev->dev->of_node, @@ -196,10 +197,13 @@ mt7996_init_wiphy(struct ieee80211_hw *hw) hw->max_tx_fragments = 4; - if (phy->mt76->cap.has_2ghz) + if (phy->mt76->cap.has_2ghz) { phy->mt76->sband_2g.sband.ht_cap.cap |= IEEE80211_HT_CAP_LDPC_CODING | IEEE80211_HT_CAP_MAX_AMSDU; + phy->mt76->sband_2g.sband.ht_cap.ampdu_density = + IEEE80211_HT_MPDU_DENSITY_2; + } if (phy->mt76->cap.has_5ghz) { phy->mt76->sband_5g.sband.ht_cap.cap |= @@ -211,6 +215,8 @@ mt7996_init_wiphy(struct ieee80211_hw *hw) IEEE80211_VHT_CAP_MAX_A_MPDU_LENGTH_EXPONENT_MASK | IEEE80211_VHT_CAP_SHORT_GI_160 | IEEE80211_VHT_CAP_SUPP_CHAN_WIDTH_160MHZ; + phy->mt76->sband_5g.sband.ht_cap.ampdu_density = + IEEE80211_HT_MPDU_DENSITY_1; } mt76_set_stream_caps(phy->mt76, true); @@ -250,7 +256,21 @@ mt7996_mac_init_band(struct mt7996_dev *dev, u8 band) mt76_rmw(dev, MT_WTBLOFF_RSCR(band), mask, set); } -static void mt7996_mac_init(struct mt7996_dev *dev) +static void mt7996_mac_init_basic_rates(struct mt7996_dev *dev) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(mt76_rates); i++) { + u16 rate = mt76_rates[i].hw_value; + u16 idx = MT7996_BASIC_RATES_TBL + i; + + rate = FIELD_PREP(MT_TX_RATE_MODE, rate >> 8) | + FIELD_PREP(MT_TX_RATE_IDX, rate & GENMASK(7, 0)); + mt7996_mac_set_fixed_rate_table(dev, idx, rate); + } +} + +void mt7996_mac_init(struct mt7996_dev *dev) { #define HIF_TXD_V2_1 4 int i; @@ -282,9 +302,11 @@ static void mt7996_mac_init(struct mt7996_dev *dev) for (i = MT_BAND0; i <= MT_BAND2; i++) mt7996_mac_init_band(dev, i); + + mt7996_mac_init_basic_rates(dev); } -static int mt7996_txbf_init(struct mt7996_dev *dev) +int mt7996_txbf_init(struct mt7996_dev *dev) { int ret; @@ -553,27 +575,6 @@ mt7996_set_stream_he_txbf_caps(struct mt7996_phy *phy, } static void -mt7996_gen_ppe_thresh(u8 *he_ppet, int nss) -{ - u8 i, ppet_bits, ppet_size, ru_bit_mask = 0x7; /* HE80 */ - static const u8 ppet16_ppet8_ru3_ru0[] = {0x1c, 0xc7, 0x71}; - - he_ppet[0] = FIELD_PREP(IEEE80211_PPE_THRES_NSS_MASK, nss - 1) | - FIELD_PREP(IEEE80211_PPE_THRES_RU_INDEX_BITMASK_MASK, - ru_bit_mask); - - ppet_bits = IEEE80211_PPE_THRES_INFO_PPET_SIZE * - nss * hweight8(ru_bit_mask) * 2; - ppet_size = DIV_ROUND_UP(ppet_bits, 8); - - for (i = 0; i < ppet_size - 1; i++) - he_ppet[i + 1] = ppet16_ppet8_ru3_ru0[i % 3]; - - he_ppet[i + 1] = ppet16_ppet8_ru3_ru0[i % 3] & - (0xff >> (8 - (ppet_bits - 1) % 8)); -} - -static void mt7996_init_he_caps(struct mt7996_phy *phy, enum nl80211_band band, struct ieee80211_sband_iftype_data *data, enum nl80211_iftype iftype) @@ -678,7 +679,7 @@ mt7996_init_he_caps(struct mt7996_phy *phy, enum nl80211_band band, memset(he_cap->ppe_thres, 0, sizeof(he_cap->ppe_thres)); if (he_cap_elem->phy_cap_info[6] & IEEE80211_HE_PHY_CAP6_PPE_THRESHOLD_PRESENT) { - mt7996_gen_ppe_thresh(he_cap->ppe_thres, nss); + mt76_connac_gen_ppe_thresh(he_cap->ppe_thres, nss); } else { he_cap_elem->phy_cap_info[9] |= u8_encode_bits(IEEE80211_HE_PHY_CAP9_NOMINAL_PKT_PADDING_16US, @@ -689,7 +690,7 @@ mt7996_init_he_caps(struct mt7996_phy *phy, enum nl80211_band band, u16 cap = IEEE80211_HE_6GHZ_CAP_TX_ANTPAT_CONS | IEEE80211_HE_6GHZ_CAP_RX_ANTPAT_CONS; - cap |= u16_encode_bits(IEEE80211_HT_MPDU_DENSITY_2, + cap |= u16_encode_bits(IEEE80211_HT_MPDU_DENSITY_0_5, IEEE80211_HE_6GHZ_CAP_MIN_MPDU_START) | u16_encode_bits(IEEE80211_VHT_MAX_AMPDU_1024K, IEEE80211_HE_6GHZ_CAP_MAX_AMPDU_LEN_EXP) | @@ -858,6 +859,8 @@ int mt7996_register_device(struct mt7996_dev *dev) init_waitqueue_head(&dev->reset_wait); INIT_WORK(&dev->reset_work, mt7996_mac_reset_work); + INIT_WORK(&dev->dump_work, mt7996_mac_dump_work); + mutex_init(&dev->dump_mutex); ret = mt7996_init_hardware(dev); if (ret) @@ -886,18 +889,25 @@ int mt7996_register_device(struct mt7996_dev *dev) if (ret) return ret; - return mt7996_init_debugfs(&dev->phy); + dev->recovery.hw_init_done = true; + + ret = mt7996_init_debugfs(&dev->phy); + if (ret) + return ret; + + return mt7996_coredump_register(dev); } void mt7996_unregister_device(struct mt7996_dev *dev) { mt7996_unregister_phy(mt7996_phy3(dev), MT_BAND2); mt7996_unregister_phy(mt7996_phy2(dev), MT_BAND1); + mt7996_coredump_unregister(dev); mt76_unregister_device(&dev->mt76); mt7996_mcu_exit(dev); mt7996_tx_token_put(dev); mt7996_dma_cleanup(dev); - tasklet_disable(&dev->irq_tasklet); + tasklet_disable(&dev->mt76.irq_tasklet); mt76_free_device(&dev->mt76); } diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mac.c b/drivers/net/wireless/mediatek/mt76/mt7996/mac.c index c9a9f0e31771..130eb7b4fd91 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt7996/mac.c @@ -5,6 +5,7 @@ #include <linux/etherdevice.h> #include <linux/timekeeping.h> +#include "coredump.h" #include "mt7996.h" #include "../dma.h" #include "mac.h" @@ -78,10 +79,6 @@ static struct mt76_wcid *mt7996_rx_get_wcid(struct mt7996_dev *dev, return &sta->vif->sta.wcid; } -void mt7996_sta_ps(struct mt76_dev *mdev, struct ieee80211_sta *sta, bool ps) -{ -} - bool mt7996_mac_wtbl_update(struct mt7996_dev *dev, int idx, u32 mask) { mt76_rmw(dev, MT_WTBL_UPDATE, MT_WTBL_UPDATE_WLAN_IDX, @@ -255,17 +252,25 @@ void mt7996_mac_enable_rtscts(struct mt7996_dev *dev, mt76_clear(dev, addr, BIT(5)); } +void mt7996_mac_set_fixed_rate_table(struct mt7996_dev *dev, + u8 tbl_idx, u16 rate_idx) +{ + u32 ctrl = MT_WTBL_ITCR_WR | MT_WTBL_ITCR_EXEC | tbl_idx; + + mt76_wr(dev, MT_WTBL_ITDR0, rate_idx); + /* use wtbl spe idx */ + mt76_wr(dev, MT_WTBL_ITDR1, MT_WTBL_SPE_IDX_SEL); + mt76_wr(dev, MT_WTBL_ITCR, ctrl); +} + static void mt7996_mac_decode_he_radiotap_ru(struct mt76_rx_status *status, struct ieee80211_radiotap_he *he, __le32 *rxv) { - u32 ru_h, ru_l; - u8 ru, offs = 0; + u32 ru, offs = 0; - ru_l = le32_get_bits(rxv[0], MT_PRXV_HE_RU_ALLOC_L); - ru_h = le32_get_bits(rxv[1], MT_PRXV_HE_RU_ALLOC_H); - ru = (u8)(ru_l | ru_h << 4); + ru = le32_get_bits(rxv[0], MT_PRXV_HE_RU_ALLOC); status->bw = RATE_INFO_BW_HE_RU; @@ -330,18 +335,23 @@ mt7996_mac_decode_he_mu_radiotap(struct sk_buff *skb, __le32 *rxv) he_mu->flags2 |= MU_PREP(FLAGS2_BW_FROM_SIG_A_BW, status->bw) | MU_PREP(FLAGS2_SIG_B_SYMS_USERS, - le32_get_bits(rxv[2], MT_CRXV_HE_NUM_USER)); + le32_get_bits(rxv[4], MT_CRXV_HE_NUM_USER)); - he_mu->ru_ch1[0] = le32_get_bits(rxv[3], MT_CRXV_HE_RU0); + he_mu->ru_ch1[0] = le32_get_bits(rxv[16], MT_CRXV_HE_RU0) & 0xff; if (status->bw >= RATE_INFO_BW_40) { he_mu->flags1 |= HE_BITS(MU_FLAGS1_CH2_RU_KNOWN); - he_mu->ru_ch2[0] = le32_get_bits(rxv[3], MT_CRXV_HE_RU1); + he_mu->ru_ch2[0] = le32_get_bits(rxv[16], MT_CRXV_HE_RU1) & 0xff; } if (status->bw >= RATE_INFO_BW_80) { - he_mu->ru_ch1[1] = le32_get_bits(rxv[3], MT_CRXV_HE_RU2); - he_mu->ru_ch2[1] = le32_get_bits(rxv[3], MT_CRXV_HE_RU3); + u32 ru_h, ru_l; + + he_mu->ru_ch1[1] = le32_get_bits(rxv[16], MT_CRXV_HE_RU2) & 0xff; + + ru_l = le32_get_bits(rxv[16], MT_CRXV_HE_RU3_L); + ru_h = le32_get_bits(rxv[17], MT_CRXV_HE_RU3_H) & 0x7; + he_mu->ru_ch2[1] = (u8)(ru_l | ru_h << 4); } } @@ -364,23 +374,23 @@ mt7996_mac_decode_he_radiotap(struct sk_buff *skb, __le32 *rxv, u8 mode) HE_BITS(DATA2_TXOP_KNOWN), }; struct ieee80211_radiotap_he *he = NULL; - u32 ltf_size = le32_get_bits(rxv[2], MT_CRXV_HE_LTF_SIZE) + 1; + u32 ltf_size = le32_get_bits(rxv[4], MT_CRXV_HE_LTF_SIZE) + 1; status->flag |= RX_FLAG_RADIOTAP_HE; he = skb_push(skb, sizeof(known)); memcpy(he, &known, sizeof(known)); - he->data3 = HE_PREP(DATA3_BSS_COLOR, BSS_COLOR, rxv[14]) | - HE_PREP(DATA3_LDPC_XSYMSEG, LDPC_EXT_SYM, rxv[2]); - he->data4 = HE_PREP(DATA4_SU_MU_SPTL_REUSE, SR_MASK, rxv[11]); - he->data5 = HE_PREP(DATA5_PE_DISAMBIG, PE_DISAMBIG, rxv[2]) | + he->data3 = HE_PREP(DATA3_BSS_COLOR, BSS_COLOR, rxv[9]) | + HE_PREP(DATA3_LDPC_XSYMSEG, LDPC_EXT_SYM, rxv[4]); + he->data4 = HE_PREP(DATA4_SU_MU_SPTL_REUSE, SR_MASK, rxv[13]); + he->data5 = HE_PREP(DATA5_PE_DISAMBIG, PE_DISAMBIG, rxv[5]) | le16_encode_bits(ltf_size, IEEE80211_RADIOTAP_HE_DATA5_LTF_SIZE); if (le32_to_cpu(rxv[0]) & MT_PRXV_TXBF) he->data5 |= HE_BITS(DATA5_TXBF); - he->data6 = HE_PREP(DATA6_TXOP, TXOP_DUR, rxv[14]) | - HE_PREP(DATA6_DOPPLER, DOPPLER, rxv[14]); + he->data6 = HE_PREP(DATA6_TXOP, TXOP_DUR, rxv[9]) | + HE_PREP(DATA6_DOPPLER, DOPPLER, rxv[9]); switch (mode) { case MT_PHY_TYPE_HE_SU: @@ -389,22 +399,22 @@ mt7996_mac_decode_he_radiotap(struct sk_buff *skb, __le32 *rxv, u8 mode) HE_BITS(DATA1_BEAM_CHANGE_KNOWN) | HE_BITS(DATA1_BW_RU_ALLOC_KNOWN); - he->data3 |= HE_PREP(DATA3_BEAM_CHANGE, BEAM_CHNG, rxv[14]) | - HE_PREP(DATA3_UL_DL, UPLINK, rxv[2]); + he->data3 |= HE_PREP(DATA3_BEAM_CHANGE, BEAM_CHNG, rxv[8]) | + HE_PREP(DATA3_UL_DL, UPLINK, rxv[5]); break; case MT_PHY_TYPE_HE_EXT_SU: he->data1 |= HE_BITS(DATA1_FORMAT_EXT_SU) | HE_BITS(DATA1_UL_DL_KNOWN) | HE_BITS(DATA1_BW_RU_ALLOC_KNOWN); - he->data3 |= HE_PREP(DATA3_UL_DL, UPLINK, rxv[2]); + he->data3 |= HE_PREP(DATA3_UL_DL, UPLINK, rxv[5]); break; case MT_PHY_TYPE_HE_MU: he->data1 |= HE_BITS(DATA1_FORMAT_MU) | HE_BITS(DATA1_UL_DL_KNOWN); - he->data3 |= HE_PREP(DATA3_UL_DL, UPLINK, rxv[2]); - he->data4 |= HE_PREP(DATA4_MU_STA_ID, MU_AID, rxv[7]); + he->data3 |= HE_PREP(DATA3_UL_DL, UPLINK, rxv[5]); + he->data4 |= HE_PREP(DATA4_MU_STA_ID, MU_AID, rxv[8]); mt7996_mac_decode_he_radiotap_ru(status, he, rxv); mt7996_mac_decode_he_mu_radiotap(skb, rxv); @@ -415,10 +425,10 @@ mt7996_mac_decode_he_radiotap(struct sk_buff *skb, __le32 *rxv, u8 mode) HE_BITS(DATA1_SPTL_REUSE3_KNOWN) | HE_BITS(DATA1_SPTL_REUSE4_KNOWN); - he->data4 |= HE_PREP(DATA4_TB_SPTL_REUSE1, SR_MASK, rxv[11]) | - HE_PREP(DATA4_TB_SPTL_REUSE2, SR1_MASK, rxv[11]) | - HE_PREP(DATA4_TB_SPTL_REUSE3, SR2_MASK, rxv[11]) | - HE_PREP(DATA4_TB_SPTL_REUSE4, SR3_MASK, rxv[11]); + he->data4 |= HE_PREP(DATA4_TB_SPTL_REUSE1, SR_MASK, rxv[13]) | + HE_PREP(DATA4_TB_SPTL_REUSE2, SR1_MASK, rxv[13]) | + HE_PREP(DATA4_TB_SPTL_REUSE3, SR2_MASK, rxv[13]) | + HE_PREP(DATA4_TB_SPTL_REUSE4, SR3_MASK, rxv[13]); mt7996_mac_decode_he_radiotap_ru(status, he, rxv); break; @@ -570,11 +580,12 @@ mt7996_mac_fill_rx_rate(struct mt7996_dev *dev, case MT_PHY_TYPE_EHT_SU: case MT_PHY_TYPE_EHT_TRIG: case MT_PHY_TYPE_EHT_MU: - /* TODO: currently report rx rate with HE rate */ status->nss = nss; - status->encoding = RX_ENC_HE; - bw = min_t(int, bw, IEEE80211_STA_RX_BW_160); - i = min_t(int, i & 0xf, 11); + status->encoding = RX_ENC_EHT; + i &= GENMASK(3, 0); + + if (gi <= NL80211_RATE_INFO_EHT_GI_3_2) + status->eht.gi = gi; break; default: return -EINVAL; @@ -630,6 +641,8 @@ mt7996_mac_fill_rx(struct mt7996_dev *dev, struct sk_buff *skb) u32 rxd4 = le32_to_cpu(rxd[4]); u32 csum_mask = MT_RXD0_NORMAL_IP_SUM | MT_RXD0_NORMAL_UDP_TCP_SUM; u32 csum_status = *(u32 *)skb->cb; + u32 mesh_mask = MT_RXD0_MESH | MT_RXD0_MHCP; + bool is_mesh = (rxd0 & mesh_mask) == mesh_mask; bool unicast, insert_ccmp_hdr = false; u8 remove_pad, amsdu_info, band_idx; u8 mode = 0, qos_ctl = 0; @@ -821,19 +834,16 @@ mt7996_mac_fill_rx(struct mt7996_dev *dev, struct sk_buff *skb) int pad_start = 0; skb_pull(skb, hdr_gap); - if (!hdr_trans && status->amsdu) { + if (!hdr_trans && status->amsdu && !(ieee80211_has_a4(fc) && is_mesh)) { pad_start = ieee80211_get_hdrlen_from_skb(skb); - } else if (hdr_trans && (rxd2 & MT_RXD2_NORMAL_HDR_TRANS_ERROR)) { + } else if (hdr_trans && (rxd2 & MT_RXD2_NORMAL_HDR_TRANS_ERROR) && + get_unaligned_be16(skb->data + pad_start) == ETH_P_8021Q) { /* When header translation failure is indicated, * the hardware will insert an extra 2-byte field * containing the data length after the protocol * type field. */ - pad_start = 12; - if (get_unaligned_be16(skb->data + pad_start) == ETH_P_8021Q) - pad_start += 4; - else - pad_start = 0; + pad_start = 16; } if (pad_start) { @@ -854,8 +864,17 @@ mt7996_mac_fill_rx(struct mt7996_dev *dev, struct sk_buff *skb) hdr = mt76_skb_get_hdr(skb); fc = hdr->frame_control; if (ieee80211_is_data_qos(fc)) { + u8 *qos = ieee80211_get_qos_ctl(hdr); + seq_ctrl = le16_to_cpu(hdr->seq_ctrl); - qos_ctl = *ieee80211_get_qos_ctl(hdr); + qos_ctl = *qos; + + /* Mesh DA/SA/Length will be stripped after hardware + * de-amsdu, so here needs to clear amsdu present bit + * to mark it as a normal mesh frame. + */ + if (ieee80211_has_a4(fc) && is_mesh && status->amsdu) + *qos &= ~IEEE80211_QOS_CTL_A_MSDU_PRESENT; } } else { status->flag |= RX_FLAG_8023; @@ -979,12 +998,13 @@ mt7996_mac_write_txwi_80211(struct mt7996_dev *dev, __le32 *txwi, } void mt7996_mac_write_txwi(struct mt7996_dev *dev, __le32 *txwi, - struct sk_buff *skb, struct mt76_wcid *wcid, int pid, - struct ieee80211_key_conf *key, u32 changed) + struct sk_buff *skb, struct mt76_wcid *wcid, + struct ieee80211_key_conf *key, int pid, + enum mt76_txq_id qid, u32 changed) { struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb); struct ieee80211_vif *vif = info->control.vif; - struct mt76_phy *mphy = &dev->mphy; + struct mt7996_vif *mvif = (struct mt7996_vif *)vif->drv_priv; u8 band_idx = (info->hw_queue & MT_TX_HW_QUEUE_PHY) >> 2; u8 p_fmt, q_idx, omac_idx = 0, wmm_idx = 0; bool is_8023 = info->flags & IEEE80211_TX_CTL_HW_80211_ENCAP; @@ -996,22 +1016,18 @@ void mt7996_mac_write_txwi(struct mt7996_dev *dev, __le32 *txwi, BSS_CHANGED_FILS_DISCOVERY)); if (vif) { - struct mt7996_vif *mvif = (struct mt7996_vif *)vif->drv_priv; - omac_idx = mvif->mt76.omac_idx; wmm_idx = mvif->mt76.wmm_idx; band_idx = mvif->mt76.band_idx; } - mphy = mt76_dev_phy(&dev->mt76, band_idx); - if (inband_disc) { p_fmt = MT_TX_TYPE_FW; q_idx = MT_LMAC_ALTX0; } else if (beacon) { p_fmt = MT_TX_TYPE_FW; q_idx = MT_LMAC_BCN0; - } else if (skb_get_queue_mapping(skb) >= MT_TXQ_PSD) { + } else if (qid >= MT_TXQ_PSD) { p_fmt = MT_TX_TYPE_CT; q_idx = MT_LMAC_ALTX0; } else { @@ -1062,18 +1078,17 @@ void mt7996_mac_write_txwi(struct mt7996_dev *dev, __le32 *txwi, mt7996_mac_write_txwi_80211(dev, txwi, skb, key); if (txwi[1] & cpu_to_le32(MT_TXD1_FIXED_RATE)) { - /* Fixed rata is available just for 802.11 txd */ struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data; - bool multicast = is_multicast_ether_addr(hdr->addr1); - u16 rate = mt76_connac2_mac_tx_rate_val(mphy, vif, beacon, - multicast); + bool mcast = ieee80211_is_data(hdr->frame_control) && + is_multicast_ether_addr(hdr->addr1); + u8 idx = mvif->basic_rates_idx; - /* fix to bw 20 */ - val = MT_TXD6_FIXED_BW | - FIELD_PREP(MT_TXD6_BW, 0) | - FIELD_PREP(MT_TXD6_TX_RATE, rate); + if (mcast && mvif->mcast_rates_idx) + idx = mvif->mcast_rates_idx; + else if (beacon && mvif->beacon_rates_idx) + idx = mvif->beacon_rates_idx; - txwi[6] |= cpu_to_le32(val); + txwi[6] |= FIELD_PREP(MT_TXD6_TX_RATE, idx); txwi[3] |= cpu_to_le32(MT_TXD3_BA_DISABLE); } } @@ -1117,11 +1132,8 @@ int mt7996_tx_prepare_skb(struct mt76_dev *mdev, void *txwi_ptr, return id; pid = mt76_tx_status_skb_add(mdev, wcid, tx_info->skb); - memset(txwi_ptr, 0, MT_TXD_SIZE); - /* Transmit non qos data by 802.11 header and need to fill txd by host*/ - if (!is_8023 || pid >= MT_PACKET_ID_FIRST) - mt7996_mac_write_txwi(dev, txwi_ptr, tx_info->skb, wcid, pid, - key, 0); + mt7996_mac_write_txwi(dev, txwi_ptr, tx_info->skb, wcid, key, + pid, qid, 0); txp = (struct mt76_connac_txp_common *)(txwi + MT_TXD_SIZE); for (i = 0; i < nbuf; i++) { @@ -1130,10 +1142,8 @@ int mt7996_tx_prepare_skb(struct mt76_dev *mdev, void *txwi_ptr, } txp->fw.nbuf = nbuf; - txp->fw.flags = cpu_to_le16(MT_CT_INFO_FROM_HOST); - - if (!is_8023 || pid >= MT_PACKET_ID_FIRST) - txp->fw.flags |= cpu_to_le16(MT_CT_INFO_APPLY_TXD); + txp->fw.flags = + cpu_to_le16(MT_CT_INFO_FROM_HOST | MT_CT_INFO_APPLY_TXD); if (!key) txp->fw.flags |= cpu_to_le16(MT_CT_INFO_NONE_CIPHER_FRAME); @@ -1704,7 +1714,7 @@ mt7996_wait_reset_state(struct mt7996_dev *dev, u32 state) bool ret; ret = wait_event_timeout(dev->reset_wait, - (READ_ONCE(dev->reset_state) & state), + (READ_ONCE(dev->recovery.state) & state), MT7996_RESET_TIMEOUT); WARN(!ret, "Timeout waiting for MCU reset state %x\n", state); @@ -1753,68 +1763,207 @@ mt7996_update_beacons(struct mt7996_dev *dev) mt7996_update_vif_beacon, phy3->hw); } -static void -mt7996_dma_reset(struct mt7996_dev *dev) +void mt7996_tx_token_put(struct mt7996_dev *dev) { - struct mt76_phy *phy2 = dev->mt76.phys[MT_BAND1]; - struct mt76_phy *phy3 = dev->mt76.phys[MT_BAND2]; - u32 hif1_ofs = MT_WFDMA0_PCIE1(0) - MT_WFDMA0(0); - int i; + struct mt76_txwi_cache *txwi; + int id; - mt76_clear(dev, MT_WFDMA0_GLO_CFG, - MT_WFDMA0_GLO_CFG_TX_DMA_EN | - MT_WFDMA0_GLO_CFG_RX_DMA_EN); + spin_lock_bh(&dev->mt76.token_lock); + idr_for_each_entry(&dev->mt76.token, txwi, id) { + mt7996_txwi_free(dev, txwi, NULL, NULL); + dev->mt76.token_count--; + } + spin_unlock_bh(&dev->mt76.token_lock); + idr_destroy(&dev->mt76.token); +} - if (dev->hif2) - mt76_clear(dev, MT_WFDMA0_GLO_CFG + hif1_ofs, - MT_WFDMA0_GLO_CFG_TX_DMA_EN | - MT_WFDMA0_GLO_CFG_RX_DMA_EN); +static int +mt7996_mac_restart(struct mt7996_dev *dev) +{ + struct mt7996_phy *phy2, *phy3; + struct mt76_dev *mdev = &dev->mt76; + int i, ret; - usleep_range(1000, 2000); + phy2 = mt7996_phy2(dev); + phy3 = mt7996_phy3(dev); - for (i = 0; i < __MT_TXQ_MAX; i++) { - mt76_queue_tx_cleanup(dev, dev->mphy.q_tx[i], true); - if (phy2) - mt76_queue_tx_cleanup(dev, phy2->q_tx[i], true); - if (phy3) - mt76_queue_tx_cleanup(dev, phy3->q_tx[i], true); + if (dev->hif2) { + mt76_wr(dev, MT_INT1_MASK_CSR, 0x0); + mt76_wr(dev, MT_INT1_SOURCE_CSR, ~0); } - for (i = 0; i < __MT_MCUQ_MAX; i++) - mt76_queue_tx_cleanup(dev, dev->mt76.q_mcu[i], true); + if (dev_is_pci(mdev->dev)) { + mt76_wr(dev, MT_PCIE_MAC_INT_ENABLE, 0x0); + if (dev->hif2) + mt76_wr(dev, MT_PCIE1_MAC_INT_ENABLE, 0x0); + } - mt76_for_each_q_rx(&dev->mt76, i) - mt76_queue_rx_reset(dev, i); + set_bit(MT76_RESET, &dev->mphy.state); + set_bit(MT76_MCU_RESET, &dev->mphy.state); + wake_up(&dev->mt76.mcu.wait); + if (phy2) { + set_bit(MT76_RESET, &phy2->mt76->state); + set_bit(MT76_MCU_RESET, &phy2->mt76->state); + } + if (phy3) { + set_bit(MT76_RESET, &phy3->mt76->state); + set_bit(MT76_MCU_RESET, &phy3->mt76->state); + } + + /* lock/unlock all queues to ensure that no tx is pending */ + mt76_txq_schedule_all(&dev->mphy); + if (phy2) + mt76_txq_schedule_all(phy2->mt76); + if (phy3) + mt76_txq_schedule_all(phy3->mt76); + + /* disable all tx/rx napi */ + mt76_worker_disable(&dev->mt76.tx_worker); + mt76_for_each_q_rx(mdev, i) { + if (mdev->q_rx[i].ndesc) + napi_disable(&dev->mt76.napi[i]); + } + napi_disable(&dev->mt76.tx_napi); + + /* token reinit */ + mt7996_tx_token_put(dev); + idr_init(&dev->mt76.token); + + mt7996_dma_reset(dev, true); - mt76_tx_status_check(&dev->mt76, true); + local_bh_disable(); + mt76_for_each_q_rx(mdev, i) { + if (mdev->q_rx[i].ndesc) { + napi_enable(&dev->mt76.napi[i]); + napi_schedule(&dev->mt76.napi[i]); + } + } + local_bh_enable(); + clear_bit(MT76_MCU_RESET, &dev->mphy.state); + clear_bit(MT76_STATE_MCU_RUNNING, &dev->mphy.state); + + mt76_wr(dev, MT_INT_MASK_CSR, dev->mt76.mmio.irqmask); + mt76_wr(dev, MT_INT_SOURCE_CSR, ~0); + if (dev->hif2) { + mt76_wr(dev, MT_INT1_MASK_CSR, dev->mt76.mmio.irqmask); + mt76_wr(dev, MT_INT1_SOURCE_CSR, ~0); + } + if (dev_is_pci(mdev->dev)) { + mt76_wr(dev, MT_PCIE_MAC_INT_ENABLE, 0xff); + if (dev->hif2) + mt76_wr(dev, MT_PCIE1_MAC_INT_ENABLE, 0xff); + } + + /* load firmware */ + ret = mt7996_mcu_init_firmware(dev); + if (ret) + goto out; - /* re-init prefetch settings after reset */ - mt7996_dma_prefetch(dev); + /* set the necessary init items */ + ret = mt7996_mcu_set_eeprom(dev); + if (ret) + goto out; - mt76_set(dev, MT_WFDMA0_GLO_CFG, - MT_WFDMA0_GLO_CFG_TX_DMA_EN | MT_WFDMA0_GLO_CFG_RX_DMA_EN); + mt7996_mac_init(dev); + mt7996_init_txpower(dev, &dev->mphy.sband_2g.sband); + mt7996_init_txpower(dev, &dev->mphy.sband_5g.sband); + mt7996_init_txpower(dev, &dev->mphy.sband_6g.sband); + ret = mt7996_txbf_init(dev); - if (dev->hif2) - mt76_set(dev, MT_WFDMA0_GLO_CFG + hif1_ofs, - MT_WFDMA0_GLO_CFG_TX_DMA_EN | - MT_WFDMA0_GLO_CFG_RX_DMA_EN); + if (test_bit(MT76_STATE_RUNNING, &dev->mphy.state)) { + ret = mt7996_run(dev->mphy.hw); + if (ret) + goto out; + } + + if (phy2 && test_bit(MT76_STATE_RUNNING, &phy2->mt76->state)) { + ret = mt7996_run(phy2->mt76->hw); + if (ret) + goto out; + } + + if (phy3 && test_bit(MT76_STATE_RUNNING, &phy3->mt76->state)) { + ret = mt7996_run(phy3->mt76->hw); + if (ret) + goto out; + } + +out: + /* reset done */ + clear_bit(MT76_RESET, &dev->mphy.state); + if (phy2) + clear_bit(MT76_RESET, &phy2->mt76->state); + if (phy3) + clear_bit(MT76_RESET, &phy3->mt76->state); + + local_bh_disable(); + napi_enable(&dev->mt76.tx_napi); + napi_schedule(&dev->mt76.tx_napi); + local_bh_enable(); + + mt76_worker_enable(&dev->mt76.tx_worker); + return ret; } -void mt7996_tx_token_put(struct mt7996_dev *dev) +static void +mt7996_mac_full_reset(struct mt7996_dev *dev) { - struct mt76_txwi_cache *txwi; - int id; + struct mt7996_phy *phy2, *phy3; + int i; - spin_lock_bh(&dev->mt76.token_lock); - idr_for_each_entry(&dev->mt76.token, txwi, id) { - mt7996_txwi_free(dev, txwi, NULL, NULL); - dev->mt76.token_count--; + phy2 = mt7996_phy2(dev); + phy3 = mt7996_phy3(dev); + dev->recovery.hw_full_reset = true; + + wake_up(&dev->mt76.mcu.wait); + ieee80211_stop_queues(mt76_hw(dev)); + if (phy2) + ieee80211_stop_queues(phy2->mt76->hw); + if (phy3) + ieee80211_stop_queues(phy3->mt76->hw); + + cancel_delayed_work_sync(&dev->mphy.mac_work); + if (phy2) + cancel_delayed_work_sync(&phy2->mt76->mac_work); + if (phy3) + cancel_delayed_work_sync(&phy3->mt76->mac_work); + + mutex_lock(&dev->mt76.mutex); + for (i = 0; i < 10; i++) { + if (!mt7996_mac_restart(dev)) + break; } - spin_unlock_bh(&dev->mt76.token_lock); - idr_destroy(&dev->mt76.token); + mutex_unlock(&dev->mt76.mutex); + + if (i == 10) + dev_err(dev->mt76.dev, "chip full reset failed\n"); + + ieee80211_restart_hw(mt76_hw(dev)); + if (phy2) + ieee80211_restart_hw(phy2->mt76->hw); + if (phy3) + ieee80211_restart_hw(phy3->mt76->hw); + + ieee80211_wake_queues(mt76_hw(dev)); + if (phy2) + ieee80211_wake_queues(phy2->mt76->hw); + if (phy3) + ieee80211_wake_queues(phy3->mt76->hw); + + dev->recovery.hw_full_reset = false; + ieee80211_queue_delayed_work(mt76_hw(dev), + &dev->mphy.mac_work, + MT7996_WATCHDOG_TIME); + if (phy2) + ieee80211_queue_delayed_work(phy2->mt76->hw, + &phy2->mt76->mac_work, + MT7996_WATCHDOG_TIME); + if (phy3) + ieee80211_queue_delayed_work(phy3->mt76->hw, + &phy3->mt76->mac_work, + MT7996_WATCHDOG_TIME); } -/* system error recovery */ void mt7996_mac_reset_work(struct work_struct *work) { struct mt7996_phy *phy2, *phy3; @@ -1825,9 +1974,36 @@ void mt7996_mac_reset_work(struct work_struct *work) phy2 = mt7996_phy2(dev); phy3 = mt7996_phy3(dev); - if (!(READ_ONCE(dev->reset_state) & MT_MCU_CMD_STOP_DMA)) + /* chip full reset */ + if (dev->recovery.restart) { + /* disable WA/WM WDT */ + mt76_clear(dev, MT_WFDMA0_MCU_HOST_INT_ENA, + MT_MCU_CMD_WDT_MASK); + + if (READ_ONCE(dev->recovery.state) & MT_MCU_CMD_WA_WDT) + dev->recovery.wa_reset_count++; + else + dev->recovery.wm_reset_count++; + + mt7996_mac_full_reset(dev); + + /* enable mcu irq */ + mt7996_irq_enable(dev, MT_INT_MCU_CMD); + mt7996_irq_disable(dev, 0); + + /* enable WA/WM WDT */ + mt76_set(dev, MT_WFDMA0_MCU_HOST_INT_ENA, MT_MCU_CMD_WDT_MASK); + + dev->recovery.state = MT_MCU_CMD_NORMAL_STATE; + dev->recovery.restart = false; + return; + } + + if (!(READ_ONCE(dev->recovery.state) & MT_MCU_CMD_STOP_DMA)) return; + dev_info(dev->mt76.dev,"\n%s L1 SER recovery start.", + wiphy_name(dev->mt76.hw->wiphy)); ieee80211_stop_queues(mt76_hw(dev)); if (phy2) ieee80211_stop_queues(phy2->mt76->hw); @@ -1856,7 +2032,7 @@ void mt7996_mac_reset_work(struct work_struct *work) mt76_wr(dev, MT_MCU_INT_EVENT, MT_MCU_INT_EVENT_DMA_STOPPED); if (mt7996_wait_reset_state(dev, MT_MCU_CMD_RESET_DONE)) { - mt7996_dma_reset(dev); + mt7996_dma_reset(dev, false); mt7996_tx_token_put(dev); idr_init(&dev->mt76.token); @@ -1879,7 +2055,7 @@ void mt7996_mac_reset_work(struct work_struct *work) } local_bh_enable(); - tasklet_schedule(&dev->irq_tasklet); + tasklet_schedule(&dev->mt76.irq_tasklet); mt76_wr(dev, MT_MCU_INT_EVENT, MT_MCU_INT_EVENT_RESET_DONE); mt7996_wait_reset_state(dev, MT_MCU_CMD_NORMAL_STATE); @@ -1911,6 +2087,101 @@ void mt7996_mac_reset_work(struct work_struct *work) ieee80211_queue_delayed_work(phy3->mt76->hw, &phy3->mt76->mac_work, MT7996_WATCHDOG_TIME); + dev_info(dev->mt76.dev,"\n%s L1 SER recovery completed.", + wiphy_name(dev->mt76.hw->wiphy)); +} + +/* firmware coredump */ +void mt7996_mac_dump_work(struct work_struct *work) +{ + const struct mt7996_mem_region *mem_region; + struct mt7996_crash_data *crash_data; + struct mt7996_dev *dev; + struct mt7996_mem_hdr *hdr; + size_t buf_len; + int i; + u32 num; + u8 *buf; + + dev = container_of(work, struct mt7996_dev, dump_work); + + mutex_lock(&dev->dump_mutex); + + crash_data = mt7996_coredump_new(dev); + if (!crash_data) { + mutex_unlock(&dev->dump_mutex); + goto skip_coredump; + } + + mem_region = mt7996_coredump_get_mem_layout(dev, &num); + if (!mem_region || !crash_data->memdump_buf_len) { + mutex_unlock(&dev->dump_mutex); + goto skip_memdump; + } + + buf = crash_data->memdump_buf; + buf_len = crash_data->memdump_buf_len; + + /* dumping memory content... */ + memset(buf, 0, buf_len); + for (i = 0; i < num; i++) { + if (mem_region->len > buf_len) { + dev_warn(dev->mt76.dev, "%s len %zu is too large\n", + mem_region->name, mem_region->len); + break; + } + + /* reserve space for the header */ + hdr = (void *)buf; + buf += sizeof(*hdr); + buf_len -= sizeof(*hdr); + + mt7996_memcpy_fromio(dev, buf, mem_region->start, + mem_region->len); + + hdr->start = mem_region->start; + hdr->len = mem_region->len; + + if (!mem_region->len) + /* note: the header remains, just with zero length */ + break; + + buf += mem_region->len; + buf_len -= mem_region->len; + + mem_region++; + } + + mutex_unlock(&dev->dump_mutex); + +skip_memdump: + mt7996_coredump_submit(dev); +skip_coredump: + queue_work(dev->mt76.wq, &dev->reset_work); +} + +void mt7996_reset(struct mt7996_dev *dev) +{ + if (!dev->recovery.hw_init_done) + return; + + if (dev->recovery.hw_full_reset) + return; + + /* wm/wa exception: do full recovery */ + if (READ_ONCE(dev->recovery.state) & MT_MCU_CMD_WDT_MASK) { + dev->recovery.restart = true; + dev_info(dev->mt76.dev, + "%s indicated firmware crash, attempting recovery\n", + wiphy_name(dev->mt76.hw->wiphy)); + + mt7996_irq_disable(dev, MT_INT_MCU_CMD); + queue_work(dev->mt76.wq, &dev->dump_work); + return; + } + + queue_work(dev->mt76.wq, &dev->reset_work); + wake_up(&dev->reset_wait); } void mt7996_mac_update_stats(struct mt7996_phy *phy) diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mac.h b/drivers/net/wireless/mediatek/mt76/mt7996/mac.h index 27184cbac619..bc4e6c55373e 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/mac.h +++ b/drivers/net/wireless/mediatek/mt76/mt7996/mac.h @@ -12,6 +12,8 @@ #define MT_RXD0_LENGTH GENMASK(15, 0) #define MT_RXD0_PKT_TYPE GENMASK(31, 27) +#define MT_RXD0_MESH BIT(18) +#define MT_RXD0_MHCP BIT(19) #define MT_RXD0_NORMAL_ETH_TYPE_OFS GENMASK(22, 16) #define MT_RXD0_NORMAL_IP_SUM BIT(23) #define MT_RXD0_NORMAL_UDP_TCP_SUM BIT(24) @@ -20,18 +22,6 @@ #define MT_RXD0_SW_PKT_TYPE_MAP 0x380F #define MT_RXD0_SW_PKT_TYPE_FRAME 0x3801 -enum rx_pkt_type { - PKT_TYPE_TXS, - PKT_TYPE_TXRXV, - PKT_TYPE_NORMAL, - PKT_TYPE_RX_DUP_RFB, - PKT_TYPE_RX_TMR, - PKT_TYPE_RETRIEVE, - PKT_TYPE_TXRX_NOTIFY, - PKT_TYPE_RX_EVENT, - PKT_TYPE_RX_FW_MONITOR = 0x0c, -}; - /* RXD DW1 */ #define MT_RXD1_NORMAL_WLAN_IDX GENMASK(11, 0) #define MT_RXD1_NORMAL_GROUP_1 BIT(16) @@ -102,8 +92,7 @@ enum rx_pkt_type { #define MT_PRXV_NSTS GENMASK(10, 7) #define MT_PRXV_TXBF BIT(11) #define MT_PRXV_HT_AD_CODE BIT(12) -#define MT_PRXV_HE_RU_ALLOC_L GENMASK(31, 28) -#define MT_PRXV_HE_RU_ALLOC_H GENMASK(3, 0) +#define MT_PRXV_HE_RU_ALLOC GENMASK(30, 22) #define MT_PRXV_RCPI3 GENMASK(31, 24) #define MT_PRXV_RCPI2 GENMASK(23, 16) #define MT_PRXV_RCPI1 GENMASK(15, 8) @@ -113,34 +102,32 @@ enum rx_pkt_type { #define MT_PRXV_TX_MODE GENMASK(14, 11) #define MT_PRXV_FRAME_MODE GENMASK(2, 0) #define MT_PRXV_DCM BIT(5) -#define MT_PRXV_NUM_RX BIT(8, 6) /* C-RXV */ -#define MT_CRXV_HT_STBC GENMASK(1, 0) -#define MT_CRXV_TX_MODE GENMASK(7, 4) -#define MT_CRXV_FRAME_MODE GENMASK(10, 8) -#define MT_CRXV_HT_SHORT_GI GENMASK(14, 13) -#define MT_CRXV_HE_LTF_SIZE GENMASK(18, 17) -#define MT_CRXV_HE_LDPC_EXT_SYM BIT(20) -#define MT_CRXV_HE_PE_DISAMBIG BIT(23) -#define MT_CRXV_HE_NUM_USER GENMASK(30, 24) -#define MT_CRXV_HE_UPLINK BIT(31) -#define MT_CRXV_HE_RU0 GENMASK(7, 0) -#define MT_CRXV_HE_RU1 GENMASK(15, 8) -#define MT_CRXV_HE_RU2 GENMASK(23, 16) -#define MT_CRXV_HE_RU3 GENMASK(31, 24) - -#define MT_CRXV_HE_MU_AID GENMASK(30, 20) +#define MT_CRXV_HE_NUM_USER GENMASK(26, 20) +#define MT_CRXV_HE_LTF_SIZE GENMASK(28, 27) +#define MT_CRXV_HE_LDPC_EXT_SYM BIT(30) + +#define MT_CRXV_HE_PE_DISAMBIG BIT(1) +#define MT_CRXV_HE_UPLINK BIT(2) + +#define MT_CRXV_HE_MU_AID GENMASK(27, 17) +#define MT_CRXV_HE_BEAM_CHNG BIT(29) + +#define MT_CRXV_HE_DOPPLER BIT(0) +#define MT_CRXV_HE_BSS_COLOR GENMASK(15, 10) +#define MT_CRXV_HE_TXOP_DUR GENMASK(19, 17) #define MT_CRXV_HE_SR_MASK GENMASK(11, 8) #define MT_CRXV_HE_SR1_MASK GENMASK(16, 12) #define MT_CRXV_HE_SR2_MASK GENMASK(20, 17) #define MT_CRXV_HE_SR3_MASK GENMASK(24, 21) -#define MT_CRXV_HE_BSS_COLOR GENMASK(5, 0) -#define MT_CRXV_HE_TXOP_DUR GENMASK(12, 6) -#define MT_CRXV_HE_BEAM_CHNG BIT(13) -#define MT_CRXV_HE_DOPPLER BIT(16) +#define MT_CRXV_HE_RU0 GENMASK(8, 0) +#define MT_CRXV_HE_RU1 GENMASK(17, 9) +#define MT_CRXV_HE_RU2 GENMASK(26, 18) +#define MT_CRXV_HE_RU3_L GENMASK(31, 27) +#define MT_CRXV_HE_RU3_H GENMASK(3, 0) enum tx_header_format { MT_HDR_FORMAT_802_3, @@ -239,14 +226,11 @@ enum tx_mgnt_type { #define MT_TXD6_TX_SRC GENMASK(31, 30) #define MT_TXD6_VTA BIT(28) -#define MT_TXD6_FIXED_BW BIT(25) -#define MT_TXD6_BW GENMASK(24, 22) +#define MT_TXD6_BW GENMASK(25, 22) #define MT_TXD6_TX_RATE GENMASK(21, 16) #define MT_TXD6_TIMESTAMP_OFS_EN BIT(15) #define MT_TXD6_TIMESTAMP_OFS_IDX GENMASK(14, 10) #define MT_TXD6_MSDU_CNT GENMASK(9, 4) -#define MT_TXD6_SPE_ID_IDX BIT(10) -#define MT_TXD6_ANT_ID GENMASK(7, 4) #define MT_TXD6_DIS_MAT BIT(3) #define MT_TXD6_DAS BIT(2) #define MT_TXD6_AMSDU_CAP BIT(1) @@ -260,7 +244,7 @@ enum tx_mgnt_type { #define MT_TXD7_UDP_TCP_SUM BIT(15) #define MT_TXD7_TX_TIME GENMASK(9, 0) -#define MT_TX_RATE_STBC BIT(13) +#define MT_TX_RATE_STBC BIT(14) #define MT_TX_RATE_NSS GENMASK(13, 10) #define MT_TX_RATE_MODE GENMASK(9, 6) #define MT_TX_RATE_SU_EXT_TONE BIT(5) diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/main.c b/drivers/net/wireless/mediatek/mt76/mt7996/main.c index 1ba22d147949..f306e9c50ea3 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/main.c +++ b/drivers/net/wireless/mediatek/mt76/mt7996/main.c @@ -5,6 +5,7 @@ #include "mt7996.h" #include "mcu.h" +#include "mac.h" static bool mt7996_dev_running(struct mt7996_dev *dev) { @@ -22,17 +23,13 @@ static bool mt7996_dev_running(struct mt7996_dev *dev) return phy && test_bit(MT76_STATE_RUNNING, &phy->mt76->state); } -static int mt7996_start(struct ieee80211_hw *hw) +int mt7996_run(struct ieee80211_hw *hw) { struct mt7996_dev *dev = mt7996_hw_dev(hw); struct mt7996_phy *phy = mt7996_hw_phy(hw); bool running; int ret; - flush_work(&dev->init_work); - - mutex_lock(&dev->mt76.mutex); - running = mt7996_dev_running(dev); if (!running) { ret = mt7996_mcu_set_hdr_trans(dev, true); @@ -52,10 +49,6 @@ static int mt7996_start(struct ieee80211_hw *hw) set_bit(MT76_STATE_RUNNING, &phy->mt76->state); - ieee80211_iterate_interfaces(dev->mt76.hw, - IEEE80211_IFACE_ITER_RESUME_ALL, - mt7996_mcu_set_pm, dev->mt76.hw); - ieee80211_queue_delayed_work(hw, &phy->mt76->mac_work, MT7996_WATCHDOG_TIME); @@ -63,6 +56,18 @@ static int mt7996_start(struct ieee80211_hw *hw) mt7996_mac_reset_counters(phy); out: + return ret; +} + +static int mt7996_start(struct ieee80211_hw *hw) +{ + struct mt7996_dev *dev = mt7996_hw_dev(hw); + int ret; + + flush_work(&dev->init_work); + + mutex_lock(&dev->mt76.mutex); + ret = mt7996_run(hw); mutex_unlock(&dev->mt76.mutex); return ret; @@ -79,10 +84,6 @@ static void mt7996_stop(struct ieee80211_hw *hw) clear_bit(MT76_STATE_RUNNING, &phy->mt76->state); - ieee80211_iterate_interfaces(dev->mt76.hw, - IEEE80211_IFACE_ITER_RESUME_ALL, - mt7996_mcu_set_pm, dev->mt76.hw); - mutex_unlock(&dev->mt76.mutex); } @@ -219,8 +220,12 @@ static int mt7996_add_interface(struct ieee80211_hw *hw, vif->offload_flags = 0; vif->offload_flags |= IEEE80211_OFFLOAD_ENCAP_4ADDR; + if (phy->mt76->chandef.chan->band != NL80211_BAND_2GHZ) + mvif->basic_rates_idx = MT7996_BASIC_RATES_TBL + 4; + else + mvif->basic_rates_idx = MT7996_BASIC_RATES_TBL; + mt7996_init_bitrate_mask(vif); - memset(&mvif->cap, -1, sizeof(mvif->cap)); mt7996_mcu_add_bss_info(phy, vif, true); mt7996_mcu_add_sta(dev, vif, NULL, true); @@ -496,11 +501,41 @@ mt7996_update_bss_color(struct ieee80211_hw *hw, } } +static u8 +mt7996_get_rates_table(struct ieee80211_hw *hw, struct ieee80211_vif *vif, + bool beacon, bool mcast) +{ + struct mt7996_vif *mvif = (struct mt7996_vif *)vif->drv_priv; + struct mt76_phy *mphy = hw->priv; + u16 rate; + u8 i, idx, ht; + + rate = mt76_connac2_mac_tx_rate_val(mphy, vif, beacon, mcast); + ht = FIELD_GET(MT_TX_RATE_MODE, rate) > MT_PHY_TYPE_OFDM; + + if (beacon && ht) { + struct mt7996_dev *dev = mt7996_hw_dev(hw); + + /* must odd index */ + idx = MT7996_BEACON_RATES_TBL + 2 * (mvif->mt76.idx % 20); + mt7996_mac_set_fixed_rate_table(dev, idx, rate); + return idx; + } + + idx = FIELD_GET(MT_TX_RATE_IDX, rate); + for (i = 0; i < ARRAY_SIZE(mt76_rates); i++) + if ((mt76_rates[i].hw_value & GENMASK(7, 0)) == idx) + return MT7996_BASIC_RATES_TBL + i; + + return mvif->basic_rates_idx; +} + static void mt7996_bss_info_changed(struct ieee80211_hw *hw, struct ieee80211_vif *vif, struct ieee80211_bss_conf *info, u64 changed) { + struct mt7996_vif *mvif = (struct mt7996_vif *)vif->drv_priv; struct mt7996_phy *phy = mt7996_hw_phy(hw); struct mt7996_dev *dev = mt7996_hw_dev(hw); @@ -532,6 +567,14 @@ static void mt7996_bss_info_changed(struct ieee80211_hw *hw, } } + if (changed & BSS_CHANGED_MCAST_RATE) + mvif->mcast_rates_idx = + mt7996_get_rates_table(hw, vif, false, true); + + if (changed & BSS_CHANGED_BASIC_RATES) + mvif->basic_rates_idx = + mt7996_get_rates_table(hw, vif, false, false); + if (changed & BSS_CHANGED_BEACON_ENABLED && info->enable_beacon) { mt7996_mcu_add_bss_info(phy, vif, true); mt7996_mcu_add_sta(dev, vif, NULL, true); @@ -548,8 +591,12 @@ static void mt7996_bss_info_changed(struct ieee80211_hw *hw, mt7996_update_bss_color(hw, vif, &info->he_bss_color); if (changed & (BSS_CHANGED_BEACON | - BSS_CHANGED_BEACON_ENABLED)) + BSS_CHANGED_BEACON_ENABLED)) { + mvif->beacon_rates_idx = + mt7996_get_rates_table(hw, vif, true, false); + mt7996_mcu_add_beacon(hw, vif, info->enable_beacon); + } if (changed & BSS_CHANGED_UNSOL_BCAST_PROBE_RESP || changed & BSS_CHANGED_FILS_DISCOVERY) @@ -891,6 +938,7 @@ mt7996_set_antenna(struct ieee80211_hw *hw, u32 tx_ant, u32 rx_ant) mt7996_set_stream_vht_txbf_caps(phy); mt7996_set_stream_he_eht_caps(phy); + /* TODO: update bmc_wtbl spe_idx when antenna changes */ mutex_unlock(&dev->mt76.mutex); return 0; diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mcu.c b/drivers/net/wireless/mediatek/mt76/mt7996/mcu.c index dbe30832fd88..88e2f9d0e513 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/mcu.c +++ b/drivers/net/wireless/mediatek/mt76/mt7996/mcu.c @@ -422,7 +422,8 @@ mt7996_mcu_ie_countdown(struct mt7996_dev *dev, struct sk_buff *skb) if (hdr->band && dev->mt76.phys[hdr->band]) mphy = dev->mt76.phys[hdr->band]; - tail = skb->data + le16_to_cpu(rxd->len); + tail = skb->data + skb->len; + data += sizeof(struct header); while (data + sizeof(struct tlv) < tail && le16_to_cpu(tlv->len)) { switch (le16_to_cpu(tlv->tag)) { case UNI_EVENT_IE_COUNTDOWN_CSA: @@ -596,25 +597,24 @@ mt7996_mcu_bss_he_tlv(struct sk_buff *skb, struct ieee80211_vif *vif, } static void -mt7996_mcu_bss_bmc_tlv(struct sk_buff *skb, struct mt7996_phy *phy) +mt7996_mcu_bss_bmc_tlv(struct sk_buff *skb, struct ieee80211_vif *vif, + struct mt7996_phy *phy) { + struct mt7996_vif *mvif = (struct mt7996_vif *)vif->drv_priv; struct bss_rate_tlv *bmc; struct cfg80211_chan_def *chandef = &phy->mt76->chandef; enum nl80211_band band = chandef->chan->band; struct tlv *tlv; + u8 idx = mvif->mcast_rates_idx ? + mvif->mcast_rates_idx : mvif->basic_rates_idx; tlv = mt7996_mcu_add_uni_tlv(skb, UNI_BSS_INFO_RATE, sizeof(*bmc)); bmc = (struct bss_rate_tlv *)tlv; - if (band == NL80211_BAND_2GHZ) { - bmc->short_preamble = true; - } else { - bmc->bc_trans = cpu_to_le16(0x8080); - bmc->mc_trans = cpu_to_le16(0x8080); - bmc->bc_fixed_rate = 1; - bmc->mc_fixed_rate = 1; - bmc->short_preamble = 1; - } + + bmc->short_preamble = (band == NL80211_BAND_2GHZ); + bmc->bc_fixed_rate = idx; + bmc->mc_fixed_rate = idx; } static void @@ -822,7 +822,7 @@ int mt7996_mcu_add_bss_info(struct mt7996_phy *phy, if (enable) { mt7996_mcu_bss_rfch_tlv(skb, vif, phy); - mt7996_mcu_bss_bmc_tlv(skb, phy); + mt7996_mcu_bss_bmc_tlv(skb, vif, phy); mt7996_mcu_bss_ra_tlv(skb, vif, phy); mt7996_mcu_bss_txcmd_tlv(skb, true); @@ -1022,6 +1022,7 @@ mt7996_mcu_sta_amsdu_tlv(struct mt7996_dev *dev, struct sk_buff *skb, struct tlv *tlv; if (vif->type != NL80211_IFTYPE_STATION && + vif->type != NL80211_IFTYPE_MESH_POINT && vif->type != NL80211_IFTYPE_AP) return; @@ -1053,7 +1054,6 @@ static inline bool mt7996_is_ebf_supported(struct mt7996_phy *phy, struct ieee80211_vif *vif, struct ieee80211_sta *sta, bool bfee) { - struct mt7996_vif *mvif = (struct mt7996_vif *)vif->drv_priv; int sts = hweight16(phy->mt76->chainmask); if (vif->type != NL80211_IFTYPE_STATION && @@ -1068,10 +1068,10 @@ mt7996_is_ebf_supported(struct mt7996_phy *phy, struct ieee80211_vif *vif, struct ieee80211_eht_cap_elem_fixed *pe = &pc->eht_cap_elem; if (bfee) - return mvif->cap.eht_su_ebfee && + return vif->bss_conf.eht_su_beamformee && EHT_PHY(CAP0_SU_BEAMFORMEE, pe->phy_cap_info[0]); else - return mvif->cap.eht_su_ebfer && + return vif->bss_conf.eht_su_beamformer && EHT_PHY(CAP0_SU_BEAMFORMER, pe->phy_cap_info[0]); } @@ -1079,10 +1079,10 @@ mt7996_is_ebf_supported(struct mt7996_phy *phy, struct ieee80211_vif *vif, struct ieee80211_he_cap_elem *pe = &sta->deflink.he_cap.he_cap_elem; if (bfee) - return mvif->cap.he_su_ebfee && + return vif->bss_conf.he_su_beamformee && HE_PHY(CAP3_SU_BEAMFORMER, pe->phy_cap_info[3]); else - return mvif->cap.he_su_ebfer && + return vif->bss_conf.he_su_beamformer && HE_PHY(CAP4_SU_BEAMFORMEE, pe->phy_cap_info[4]); } @@ -1090,10 +1090,10 @@ mt7996_is_ebf_supported(struct mt7996_phy *phy, struct ieee80211_vif *vif, u32 cap = sta->deflink.vht_cap.cap; if (bfee) - return mvif->cap.vht_su_ebfee && + return vif->bss_conf.vht_su_beamformee && (cap & IEEE80211_VHT_CAP_SU_BEAMFORMER_CAPABLE); else - return mvif->cap.vht_su_ebfer && + return vif->bss_conf.vht_su_beamformer && (cap & IEEE80211_VHT_CAP_SU_BEAMFORMEE_CAPABLE); } @@ -1471,6 +1471,12 @@ mt7996_mcu_sta_hdr_trans_tlv(struct mt7996_dev *dev, struct sk_buff *skb, hdr_trans->to_ds = true; hdr_trans->from_ds = true; } + + if (vif->type == NL80211_IFTYPE_MESH_POINT) { + hdr_trans->to_ds = true; + hdr_trans->from_ds = true; + hdr_trans->mesh = true; + } } static enum mcu_mmps_mode @@ -1572,7 +1578,7 @@ mt7996_mcu_sta_rate_ctrl_tlv(struct sk_buff *skb, struct mt7996_dev *dev, cap |= STA_CAP_TX_STBC; if (sta->deflink.ht_cap.cap & IEEE80211_HT_CAP_RX_STBC) cap |= STA_CAP_RX_STBC; - if (mvif->cap.ht_ldpc && + if (vif->bss_conf.ht_ldpc && (sta->deflink.ht_cap.cap & IEEE80211_HT_CAP_LDPC_CODING)) cap |= STA_CAP_LDPC; @@ -1598,7 +1604,7 @@ mt7996_mcu_sta_rate_ctrl_tlv(struct sk_buff *skb, struct mt7996_dev *dev, cap |= STA_CAP_VHT_TX_STBC; if (sta->deflink.vht_cap.cap & IEEE80211_VHT_CAP_RXSTBC_1) cap |= STA_CAP_VHT_RX_STBC; - if (mvif->cap.vht_ldpc && + if (vif->bss_conf.vht_ldpc && (sta->deflink.vht_cap.cap & IEEE80211_VHT_CAP_RXLDPC)) cap |= STA_CAP_VHT_LDPC; @@ -1694,8 +1700,8 @@ int mt7996_mcu_add_sta(struct mt7996_dev *dev, struct ieee80211_vif *vif, return PTR_ERR(skb); /* starec basic */ - mt76_connac_mcu_sta_basic_tlv(skb, vif, sta, enable, - !rcu_access_pointer(dev->mt76.wcid[msta->wcid.idx])); + mt76_connac_mcu_sta_basic_tlv(&dev->mt76, skb, vif, sta, enable, + !rcu_access_pointer(dev->mt76.wcid[msta->wcid.idx])); if (!enable) goto out; @@ -1906,105 +1912,10 @@ mt7996_mcu_beacon_cont(struct mt7996_dev *dev, struct ieee80211_vif *vif, } buf = (u8 *)bcn + sizeof(*bcn) - MAX_BEACON_SIZE; - mt7996_mac_write_txwi(dev, (__le32 *)buf, skb, wcid, 0, NULL, + mt7996_mac_write_txwi(dev, (__le32 *)buf, skb, wcid, NULL, 0, 0, BSS_CHANGED_BEACON); - memcpy(buf + MT_TXD_SIZE, skb->data, skb->len); -} -static void -mt7996_mcu_beacon_check_caps(struct mt7996_phy *phy, struct ieee80211_vif *vif, - struct sk_buff *skb) -{ - struct mt7996_vif *mvif = (struct mt7996_vif *)vif->drv_priv; - struct mt7996_vif_cap *vc = &mvif->cap; - const struct ieee80211_eht_cap_elem_fixed *eht; - const struct ieee80211_he_cap_elem *he; - const struct ieee80211_vht_cap *vht; - const struct ieee80211_ht_cap *ht; - struct ieee80211_mgmt *mgmt = (struct ieee80211_mgmt *)skb->data; - const u8 *ie; - u32 len, bc; - - /* Check missing configuration options to allow AP mode in mac80211 - * to remain in sync with hostapd settings, and get a subset of - * beacon and hardware capabilities. - */ - if (WARN_ON_ONCE(skb->len <= (mgmt->u.beacon.variable - skb->data))) - return; - - memset(vc, 0, sizeof(*vc)); - - len = skb->len - (mgmt->u.beacon.variable - skb->data); - - ie = cfg80211_find_ie(WLAN_EID_HT_CAPABILITY, mgmt->u.beacon.variable, - len); - if (ie && ie[1] >= sizeof(*ht)) { - ht = (void *)(ie + 2); - vc->ht_ldpc |= !!(le16_to_cpu(ht->cap_info) & - IEEE80211_HT_CAP_LDPC_CODING); - } - - ie = cfg80211_find_ie(WLAN_EID_VHT_CAPABILITY, mgmt->u.beacon.variable, - len); - if (ie && ie[1] >= sizeof(*vht)) { - u32 pc = phy->mt76->sband_5g.sband.vht_cap.cap; - - vht = (void *)(ie + 2); - bc = le32_to_cpu(vht->vht_cap_info); - - vc->vht_ldpc |= !!(bc & IEEE80211_VHT_CAP_RXLDPC); - vc->vht_su_ebfer = - (bc & IEEE80211_VHT_CAP_SU_BEAMFORMER_CAPABLE) && - (pc & IEEE80211_VHT_CAP_SU_BEAMFORMER_CAPABLE); - vc->vht_su_ebfee = - (bc & IEEE80211_VHT_CAP_SU_BEAMFORMEE_CAPABLE) && - (pc & IEEE80211_VHT_CAP_SU_BEAMFORMEE_CAPABLE); - vc->vht_mu_ebfer = - (bc & IEEE80211_VHT_CAP_MU_BEAMFORMER_CAPABLE) && - (pc & IEEE80211_VHT_CAP_MU_BEAMFORMER_CAPABLE); - vc->vht_mu_ebfee = - (bc & IEEE80211_VHT_CAP_MU_BEAMFORMEE_CAPABLE) && - (pc & IEEE80211_VHT_CAP_MU_BEAMFORMEE_CAPABLE); - } - - ie = cfg80211_find_ext_ie(WLAN_EID_EXT_HE_CAPABILITY, - mgmt->u.beacon.variable, len); - if (ie && ie[1] >= sizeof(*he) + 1) { - const struct ieee80211_sta_he_cap *pc = - mt76_connac_get_he_phy_cap(phy->mt76, vif); - const struct ieee80211_he_cap_elem *pe = &pc->he_cap_elem; - - he = (void *)(ie + 3); - - vc->he_ldpc = - HE_PHY(CAP1_LDPC_CODING_IN_PAYLOAD, pe->phy_cap_info[1]); - vc->he_su_ebfer = - HE_PHY(CAP3_SU_BEAMFORMER, he->phy_cap_info[3]) && - HE_PHY(CAP3_SU_BEAMFORMER, pe->phy_cap_info[3]); - vc->he_su_ebfee = - HE_PHY(CAP4_SU_BEAMFORMEE, he->phy_cap_info[4]) && - HE_PHY(CAP4_SU_BEAMFORMEE, pe->phy_cap_info[4]); - vc->he_mu_ebfer = - HE_PHY(CAP4_MU_BEAMFORMER, he->phy_cap_info[4]) && - HE_PHY(CAP4_MU_BEAMFORMER, pe->phy_cap_info[4]); - } - - ie = cfg80211_find_ext_ie(WLAN_EID_EXT_EHT_CAPABILITY, - mgmt->u.beacon.variable, len); - if (ie && ie[1] >= sizeof(*eht) + 1) { - const struct ieee80211_sta_eht_cap *pc = - mt76_connac_get_eht_phy_cap(phy->mt76, vif); - const struct ieee80211_eht_cap_elem_fixed *pe = &pc->eht_cap_elem; - - eht = (void *)(ie + 3); - - vc->eht_su_ebfer = - EHT_PHY(CAP0_SU_BEAMFORMER, eht->phy_cap_info[0]) && - EHT_PHY(CAP0_SU_BEAMFORMER, pe->phy_cap_info[0]); - vc->eht_su_ebfee = - EHT_PHY(CAP0_SU_BEAMFORMEE, eht->phy_cap_info[0]) && - EHT_PHY(CAP0_SU_BEAMFORMEE, pe->phy_cap_info[0]); - } + memcpy(buf + MT_TXD_SIZE, skb->data, skb->len); } int mt7996_mcu_add_beacon(struct ieee80211_hw *hw, @@ -2045,8 +1956,6 @@ int mt7996_mcu_add_beacon(struct ieee80211_hw *hw, info = IEEE80211_SKB_CB(skb); info->hw_queue |= FIELD_PREP(MT_TX_HW_QUEUE_PHY, phy->mt76->band_idx); - mt7996_mcu_beacon_check_caps(phy, vif, skb); - mt7996_mcu_beacon_cont(dev, vif, rskb, skb, bcn, &offs); /* TODO: subtag - 11v MBSSID */ mt7996_mcu_beacon_cntdwn(vif, rskb, skb, &offs); @@ -2115,8 +2024,7 @@ int mt7996_mcu_beacon_inband_discov(struct mt7996_dev *dev, buf = (u8 *)tlv + sizeof(*discov) - MAX_INBAND_FRAME_SIZE; - mt7996_mac_write_txwi(dev, (__le32 *)buf, skb, wcid, 0, NULL, - changed); + mt7996_mac_write_txwi(dev, (__le32 *)buf, skb, wcid, NULL, 0, 0, changed); memcpy(buf + MT_TXD_SIZE, skb->data, skb->len); @@ -2523,17 +2431,10 @@ mt7996_mcu_init_rx_airtime(struct mt7996_dev *dev) MCU_WM_UNI_CMD(VOW), true); } -int mt7996_mcu_init(struct mt7996_dev *dev) +int mt7996_mcu_init_firmware(struct mt7996_dev *dev) { - static const struct mt76_mcu_ops mt7996_mcu_ops = { - .headroom = sizeof(struct mt76_connac2_mcu_txd), /* reuse */ - .mcu_skb_send_msg = mt7996_mcu_send_message, - .mcu_parse_response = mt7996_mcu_parse_response, - }; int ret; - dev->mt76.mcu_ops = &mt7996_mcu_ops; - /* force firmware operation mode into normal state, * which should be set before firmware download stage. */ @@ -2574,6 +2475,19 @@ int mt7996_mcu_init(struct mt7996_dev *dev) MCU_WA_PARAM_RED, 0, 0); } +int mt7996_mcu_init(struct mt7996_dev *dev) +{ + static const struct mt76_mcu_ops mt7996_mcu_ops = { + .headroom = sizeof(struct mt76_connac2_mcu_txd), /* reuse */ + .mcu_skb_send_msg = mt7996_mcu_send_message, + .mcu_parse_response = mt7996_mcu_parse_response, + }; + + dev->mt76.mcu_ops = &mt7996_mcu_ops; + + return mt7996_mcu_init_firmware(dev); +} + void mt7996_mcu_exit(struct mt7996_dev *dev) { mt7996_mcu_restart(&dev->mt76); @@ -3133,7 +3047,7 @@ int mt7996_mcu_get_chip_config(struct mt7996_dev *dev, u32 *cap) break; default: break; - }; + } buf += le16_to_cpu(tlv->len); } @@ -3576,32 +3490,6 @@ int mt7996_mcu_twt_agrt_update(struct mt7996_dev *dev, &req, sizeof(req), true); } -void mt7996_mcu_set_pm(void *priv, u8 *mac, struct ieee80211_vif *vif) -{ -#define EXIT_PM_STATE 0 -#define ENTER_PM_STATE 1 - struct ieee80211_hw *hw = priv; - struct mt7996_dev *dev = mt7996_hw_dev(hw); - struct mt7996_phy *phy = mt7996_hw_phy(hw); - struct mt7996_vif *mvif = (struct mt7996_vif *)vif->drv_priv; - struct bss_power_save *ps; - struct sk_buff *skb; - struct tlv *tlv; - bool running = test_bit(MT76_STATE_RUNNING, &phy->mt76->state); - - skb = __mt7996_mcu_alloc_bss_req(&dev->mt76, &mvif->mt76, - MT7996_BSS_UPDATE_MAX_SIZE); - if (IS_ERR(skb)) - return; - - tlv = mt7996_mcu_add_uni_tlv(skb, UNI_BSS_INFO_PS, sizeof(*ps)); - ps = (struct bss_power_save *)tlv; - ps->profile = running ? EXIT_PM_STATE : ENTER_PM_STATE; - - mt76_mcu_skb_send_msg(&dev->mt76, skb, - MCU_WMWA_UNI_CMD(BSS_INFO_UPDATE), true); -} - int mt7996_mcu_set_rts_thresh(struct mt7996_phy *phy, u32 val) { struct { @@ -3733,6 +3621,22 @@ int mt7996_mcu_rf_regval(struct mt7996_dev *dev, u32 regidx, u32 *val, bool set) return 0; } +int mt7996_mcu_trigger_assert(struct mt7996_dev *dev) +{ + struct { + __le16 tag; + __le16 len; + u8 enable; + u8 rsv[3]; + } __packed req = { + .len = cpu_to_le16(sizeof(req) - 4), + .enable = true, + }; + + return mt76_mcu_send_msg(&dev->mt76, MCU_WM_UNI_CMD(ASSERT_DUMP), + &req, sizeof(req), false); +} + int mt7996_mcu_set_rro(struct mt7996_dev *dev, u16 tag, u8 val) { struct { diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mcu.h b/drivers/net/wireless/mediatek/mt76/mt7996/mcu.h index dd0c5ac52703..d7075a4d0667 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/mcu.h +++ b/drivers/net/wireless/mediatek/mt76/mt7996/mcu.h @@ -396,7 +396,7 @@ struct sta_rec_hdr_trans { u8 from_ds; u8 to_ds; u8 dis_rx_hdr_tran; - u8 rsv; + u8 mesh; } __packed; struct hdr_trans_en { @@ -648,23 +648,21 @@ enum { }; enum { - UNI_CMD_SER_QUERY = 0x0, - UNI_CMD_SER_SET = 0x2, - UNI_CMD_SER_TRIGGER = 0x3, -}; - -enum { - SER_QUERY, + UNI_CMD_SER_QUERY, /* recovery */ - SER_SET_RECOVER_L1, - SER_SET_RECOVER_L2, - SER_SET_RECOVER_L3_RX_ABORT, - SER_SET_RECOVER_L3_TX_ABORT, - SER_SET_RECOVER_L3_TX_DISABLE, - SER_SET_RECOVER_L3_BF, + UNI_CMD_SER_SET_RECOVER_L1, + UNI_CMD_SER_SET_RECOVER_L2, + UNI_CMD_SER_SET_RECOVER_L3_RX_ABORT, + UNI_CMD_SER_SET_RECOVER_L3_TX_ABORT, + UNI_CMD_SER_SET_RECOVER_L3_TX_DISABLE, + UNI_CMD_SER_SET_RECOVER_L3_BF, + UNI_CMD_SER_SET_RECOVER_L4_MDP, + UNI_CMD_SER_SET_RECOVER_FULL, + UNI_CMD_SER_SET_SYSTEM_ASSERT, /* action */ - SER_ENABLE = 2, - SER_RECOVER + UNI_CMD_SER_ENABLE = 1, + UNI_CMD_SER_SET, + UNI_CMD_SER_TRIGGER }; enum { diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mmio.c b/drivers/net/wireless/mediatek/mt76/mt7996/mmio.c index 902370a2a639..3a591a7b47ae 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/mmio.c +++ b/drivers/net/wireless/mediatek/mt76/mt7996/mmio.c @@ -162,6 +162,14 @@ static u32 __mt7996_reg_addr(struct mt7996_dev *dev, u32 addr) return mt7996_reg_map_l2(dev, addr); } +void mt7996_memcpy_fromio(struct mt7996_dev *dev, void *buf, u32 offset, + size_t len) +{ + u32 addr = __mt7996_reg_addr(dev, offset); + + memcpy_fromio(buf, dev->mt76.mmio.regs + addr, len); +} + static u32 mt7996_rr(struct mt76_dev *mdev, u32 offset) { struct mt7996_dev *dev = container_of(mdev, struct mt7996_dev, mt76); @@ -251,7 +259,7 @@ static void mt7996_rx_poll_complete(struct mt76_dev *mdev, /* TODO: support 2/4/6/8 MSI-X vectors */ static void mt7996_irq_tasklet(struct tasklet_struct *t) { - struct mt7996_dev *dev = from_tasklet(dev, t, irq_tasklet); + struct mt7996_dev *dev = from_tasklet(dev, t, mt76.irq_tasklet); u32 i, intr, mask, intr1; mt76_wr(dev, MT_INT_MASK_CSR, 0); @@ -289,10 +297,9 @@ static void mt7996_irq_tasklet(struct tasklet_struct *t) u32 val = mt76_rr(dev, MT_MCU_CMD); mt76_wr(dev, MT_MCU_CMD, val); - if (val & MT_MCU_CMD_ERROR_MASK) { - dev->reset_state = val; - ieee80211_queue_work(mt76_hw(dev), &dev->reset_work); - wake_up(&dev->reset_wait); + if (val & (MT_MCU_CMD_ERROR_MASK | MT_MCU_CMD_WDT_MASK)) { + dev->recovery.state = val; + mt7996_reset(dev); } } } @@ -308,7 +315,7 @@ irqreturn_t mt7996_irq_handler(int irq, void *dev_instance) if (!test_bit(MT76_STATE_INITIALIZED, &dev->mphy.state)) return IRQ_NONE; - tasklet_schedule(&dev->irq_tasklet); + tasklet_schedule(&dev->mt76.irq_tasklet); return IRQ_HANDLED; } @@ -320,6 +327,7 @@ struct mt7996_dev *mt7996_mmio_probe(struct device *pdev, /* txwi_size = txd size + txp size */ .txwi_size = MT_TXD_SIZE + sizeof(struct mt76_connac_fw_txp), .drv_flags = MT_DRV_TXWI_NO_FREE | + MT_DRV_AMSDU_OFFLOAD | MT_DRV_HW_MGMT_TXQ, .survey_flags = SURVEY_INFO_TIME_TX | SURVEY_INFO_TIME_RX | @@ -330,7 +338,6 @@ struct mt7996_dev *mt7996_mmio_probe(struct device *pdev, .rx_skb = mt7996_queue_rx_skb, .rx_check = mt7996_rx_check, .rx_poll_complete = mt7996_rx_poll_complete, - .sta_ps = mt7996_sta_ps, .sta_add = mt7996_mac_sta_add, .sta_remove = mt7996_mac_sta_remove, .update_survey = mt7996_update_channel, @@ -349,7 +356,7 @@ struct mt7996_dev *mt7996_mmio_probe(struct device *pdev, if (ret) goto error; - tasklet_setup(&dev->irq_tasklet, mt7996_irq_tasklet); + tasklet_setup(&mdev->irq_tasklet, mt7996_irq_tasklet); mt76_wr(dev, MT_INT_MASK_CSR, 0); diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mt7996.h b/drivers/net/wireless/mediatek/mt76/mt7996/mt7996.h index 018dfd2b36b0..4d7dcb95a620 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/mt7996.h +++ b/drivers/net/wireless/mediatek/mt76/mt7996/mt7996.h @@ -43,6 +43,10 @@ #define MT7996_MAX_STA_TWT_AGRT 8 #define MT7996_MAX_QUEUE (__MT_RXQ_MAX + __MT_MCUQ_MAX + 3) +/* NOTE: used to map mt76_rates. idx may change if firmware expands table */ +#define MT7996_BASIC_RATES_TBL 11 +#define MT7996_BEACON_RATES_TBL 25 + struct mt7996_vif; struct mt7996_sta; struct mt7996_dfs_pulse; @@ -112,30 +116,18 @@ struct mt7996_sta { } twt; }; -struct mt7996_vif_cap { - bool ht_ldpc:1; - bool vht_ldpc:1; - bool he_ldpc:1; - bool vht_su_ebfer:1; - bool vht_su_ebfee:1; - bool vht_mu_ebfer:1; - bool vht_mu_ebfee:1; - bool he_su_ebfer:1; - bool he_su_ebfee:1; - bool he_mu_ebfer:1; - bool eht_su_ebfer:1; - bool eht_su_ebfee:1; -}; - struct mt7996_vif { struct mt76_vif mt76; /* must be first */ - struct mt7996_vif_cap cap; struct mt7996_sta sta; struct mt7996_phy *phy; struct ieee80211_tx_queue_params queue_params[IEEE80211_NUM_ACS]; struct cfg80211_bitrate_mask bitrate_mask; + + u8 basic_rates_idx; + u8 mcast_rates_idx; + u8 beacon_rates_idx; }; /* per-phy stats. */ @@ -192,6 +184,15 @@ struct mib_stats { u32 tx_amsdu_cnt; }; +/* crash-dump */ +struct mt7996_crash_data { + guid_t guid; + struct timespec64 timestamp; + + u8 *memdump_buf; + size_t memdump_buf_len; +}; + struct mt7996_hif { struct list_head list; @@ -238,7 +239,6 @@ struct mt7996_dev { u32 q_wfdma_mask; const struct mt76_bus_ops *bus_ops; - struct tasklet_struct irq_tasklet; struct mt7996_phy phy; /* monitor rx chain configured channel */ @@ -251,9 +251,25 @@ struct mt7996_dev { struct work_struct init_work; struct work_struct rc_work; + struct work_struct dump_work; struct work_struct reset_work; wait_queue_head_t reset_wait; - u32 reset_state; + struct { + u32 state; + u32 wa_reset_count; + u32 wm_reset_count; + bool hw_full_reset:1; + bool hw_init_done:1; + bool restart:1; + } recovery; + + /* protects coredump data */ + struct mutex dump_mutex; +#ifdef CONFIG_DEV_COREDUMP + struct { + struct mt7996_crash_data *crash_data; + } coredump; +#endif struct list_head sta_rc_list; struct list_head sta_poll_list; @@ -386,9 +402,16 @@ int mt7996_eeprom_get_target_power(struct mt7996_dev *dev, struct ieee80211_channel *chan); s8 mt7996_eeprom_get_power_delta(struct mt7996_dev *dev, int band); int mt7996_dma_init(struct mt7996_dev *dev); +void mt7996_dma_reset(struct mt7996_dev *dev, bool force); void mt7996_dma_prefetch(struct mt7996_dev *dev); void mt7996_dma_cleanup(struct mt7996_dev *dev); +void mt7996_init_txpower(struct mt7996_dev *dev, + struct ieee80211_supported_band *sband); +int mt7996_txbf_init(struct mt7996_dev *dev); +void mt7996_reset(struct mt7996_dev *dev); +int mt7996_run(struct ieee80211_hw *hw); int mt7996_mcu_init(struct mt7996_dev *dev); +int mt7996_mcu_init_firmware(struct mt7996_dev *dev); int mt7996_mcu_twt_agrt_update(struct mt7996_dev *dev, struct mt7996_vif *mvif, struct mt7996_twt_flow *flow, @@ -432,7 +455,6 @@ int mt7996_mcu_set_pulse_th(struct mt7996_dev *dev, int mt7996_mcu_set_radar_th(struct mt7996_dev *dev, int index, const struct mt7996_dfs_pattern *pattern); int mt7996_mcu_set_radio_en(struct mt7996_phy *phy, bool enable); -void mt7996_mcu_set_pm(void *priv, u8 *mac, struct ieee80211_vif *vif); int mt7996_mcu_set_rts_thresh(struct mt7996_phy *phy, u32 val); int mt7996_mcu_get_chan_mib_info(struct mt7996_phy *phy, bool chan_switch); int mt7996_mcu_rdd_cmd(struct mt7996_dev *dev, int cmd, u8 index, @@ -445,6 +467,7 @@ int mt7996_mcu_set_rro(struct mt7996_dev *dev, u16 tag, u8 val); int mt7996_mcu_wa_cmd(struct mt7996_dev *dev, int cmd, u32 a1, u32 a2, u32 a3); int mt7996_mcu_fw_log_2_host(struct mt7996_dev *dev, u8 type, u8 ctrl); int mt7996_mcu_fw_dbg_ctrl(struct mt7996_dev *dev, u32 module, u8 level); +int mt7996_mcu_trigger_assert(struct mt7996_dev *dev); void mt7996_mcu_rx_event(struct mt7996_dev *dev, struct sk_buff *skb); void mt7996_mcu_exit(struct mt7996_dev *dev); @@ -468,7 +491,7 @@ static inline void mt7996_irq_enable(struct mt7996_dev *dev, u32 mask) else mt76_set_irq_mask(&dev->mt76, 0, 0, mask); - tasklet_schedule(&dev->irq_tasklet); + tasklet_schedule(&dev->mt76.irq_tasklet); } static inline void mt7996_irq_disable(struct mt7996_dev *dev, u32 mask) @@ -479,6 +502,10 @@ static inline void mt7996_irq_disable(struct mt7996_dev *dev, u32 mask) mt76_set_irq_mask(&dev->mt76, MT_INT_MASK_CSR, mask, 0); } +void mt7996_memcpy_fromio(struct mt7996_dev *dev, void *buf, u32 offset, + size_t len); + +void mt7996_mac_init(struct mt7996_dev *dev); u32 mt7996_mac_wtbl_lmac_addr(struct mt7996_dev *dev, u16 wcid, u8 dw); bool mt7996_mac_wtbl_update(struct mt7996_dev *dev, int idx, u32 mask); void mt7996_mac_reset_counters(struct mt7996_phy *phy); @@ -486,9 +513,12 @@ void mt7996_mac_cca_stats_reset(struct mt7996_phy *phy); void mt7996_mac_enable_nf(struct mt7996_dev *dev, u8 band); void mt7996_mac_enable_rtscts(struct mt7996_dev *dev, struct ieee80211_vif *vif, bool enable); +void mt7996_mac_set_fixed_rate_table(struct mt7996_dev *dev, + u8 tbl_idx, u16 rate_idx); void mt7996_mac_write_txwi(struct mt7996_dev *dev, __le32 *txwi, - struct sk_buff *skb, struct mt76_wcid *wcid, int pid, - struct ieee80211_key_conf *key, u32 changed); + struct sk_buff *skb, struct mt76_wcid *wcid, + struct ieee80211_key_conf *key, int pid, + enum mt76_txq_id qid, u32 changed); void mt7996_mac_set_timing(struct mt7996_phy *phy); int mt7996_mac_sta_add(struct mt76_dev *mdev, struct ieee80211_vif *vif, struct ieee80211_sta *sta); @@ -496,6 +526,7 @@ void mt7996_mac_sta_remove(struct mt76_dev *mdev, struct ieee80211_vif *vif, struct ieee80211_sta *sta); void mt7996_mac_work(struct work_struct *work); void mt7996_mac_reset_work(struct work_struct *work); +void mt7996_mac_dump_work(struct work_struct *work); void mt7996_mac_sta_rc_work(struct work_struct *work); void mt7996_mac_update_stats(struct mt7996_phy *phy); void mt7996_mac_twt_teardown_flow(struct mt7996_dev *dev, @@ -512,7 +543,6 @@ void mt7996_tx_token_put(struct mt7996_dev *dev); void mt7996_queue_rx_skb(struct mt76_dev *mdev, enum mt76_rxq_id q, struct sk_buff *skb, u32 *info); bool mt7996_rx_check(struct mt76_dev *mdev, void *data, int len); -void mt7996_sta_ps(struct mt76_dev *mdev, struct ieee80211_sta *sta, bool ps); void mt7996_stats_work(struct work_struct *work); int mt76_dfs_start_rdd(struct mt7996_dev *dev, bool force); int mt7996_dfs_init_radar_detector(struct mt7996_phy *phy); diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/regs.h b/drivers/net/wireless/mediatek/mt76/mt7996/regs.h index 7a28cae34e34..d1d3d154195d 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/regs.h +++ b/drivers/net/wireless/mediatek/mt76/mt7996/regs.h @@ -228,6 +228,13 @@ enum base_rev { #define MT_WTBL_UPDATE_ADM_COUNT_CLEAR BIT(14) #define MT_WTBL_UPDATE_BUSY BIT(31) +#define MT_WTBL_ITCR MT_WTBLON_TOP(0x3b0) +#define MT_WTBL_ITCR_WR BIT(16) +#define MT_WTBL_ITCR_EXEC BIT(31) +#define MT_WTBL_ITDR0 MT_WTBLON_TOP(0x3b8) +#define MT_WTBL_ITDR1 MT_WTBLON_TOP(0x3bc) +#define MT_WTBL_SPE_IDX_SEL BIT(6) + /* WTBL */ #define MT_WTBL_BASE 0x820d8000 #define MT_WTBL_LMAC_ID GENMASK(14, 8) @@ -317,6 +324,8 @@ enum base_rev { #define MT_WFDMA0_RX_INT_PCIE_SEL MT_WFDMA0(0x154) #define MT_WFDMA0_RX_INT_SEL_RING3 BIT(3) +#define MT_WFDMA0_MCU_HOST_INT_ENA MT_WFDMA0(0x1f4) + #define MT_WFDMA0_GLO_CFG MT_WFDMA0(0x208) #define MT_WFDMA0_GLO_CFG_TX_DMA_EN BIT(0) #define MT_WFDMA0_GLO_CFG_RX_DMA_EN BIT(2) @@ -444,6 +453,10 @@ enum base_rev { #define MT_MCU_CMD_NORMAL_STATE BIT(5) #define MT_MCU_CMD_ERROR_MASK GENMASK(5, 1) +#define MT_MCU_CMD_WA_WDT BIT(31) +#define MT_MCU_CMD_WM_WDT BIT(30) +#define MT_MCU_CMD_WDT_MASK GENMASK(31, 30) + /* l1/l2 remap */ #define MT_HIF_REMAP_L1 0x155024 #define MT_HIF_REMAP_L1_MASK GENMASK(31, 16) @@ -468,9 +481,28 @@ enum base_rev { #define MT_INFRA_MCU_END 0x7c3fffff /* FW MODE SYNC */ -#define MT_SWDEF_MODE 0x9143c +#define MT_FW_ASSERT_CNT 0x02208274 +#define MT_FW_DUMP_STATE 0x02209e90 + +#define MT_SWDEF_BASE 0x00401400 + +#define MT_SWDEF(ofs) (MT_SWDEF_BASE + (ofs)) +#define MT_SWDEF_MODE MT_SWDEF(0x3c) #define MT_SWDEF_NORMAL_MODE 0 +#define MT_SWDEF_SER_STATS MT_SWDEF(0x040) +#define MT_SWDEF_PLE_STATS MT_SWDEF(0x044) +#define MT_SWDEF_PLE1_STATS MT_SWDEF(0x048) +#define MT_SWDEF_PLE_AMSDU_STATS MT_SWDEF(0x04c) +#define MT_SWDEF_PSE_STATS MT_SWDEF(0x050) +#define MT_SWDEF_PSE1_STATS MT_SWDEF(0x054) +#define MT_SWDEF_LAMC_WISR6_BN0_STATS MT_SWDEF(0x058) +#define MT_SWDEF_LAMC_WISR6_BN1_STATS MT_SWDEF(0x05c) +#define MT_SWDEF_LAMC_WISR6_BN2_STATS MT_SWDEF(0x060) +#define MT_SWDEF_LAMC_WISR7_BN0_STATS MT_SWDEF(0x064) +#define MT_SWDEF_LAMC_WISR7_BN1_STATS MT_SWDEF(0x068) +#define MT_SWDEF_LAMC_WISR7_BN2_STATS MT_SWDEF(0x06c) + /* LED */ #define MT_LED_TOP_BASE 0x18013000 #define MT_LED_PHYS(_n) (MT_LED_TOP_BASE + (_n)) @@ -486,6 +518,13 @@ enum base_rev { #define MT_LED_EN(_n) MT_LED_PHYS(0x40 + ((_n) * 4)) +/* CONN DBG */ +#define MT_CONN_DBG_CTL_BASE 0x18023000 +#define MT_CONN_DBG_CTL(ofs) (MT_CONN_DBG_CTL_BASE + (ofs)) +#define MT_CONN_DBG_CTL_OUT_SEL MT_CONN_DBG_CTL(0x604) +#define MT_CONN_DBG_CTL_PC_LOG_SEL MT_CONN_DBG_CTL(0x60c) +#define MT_CONN_DBG_CTL_PC_LOG MT_CONN_DBG_CTL(0x610) + #define MT_LED_GPIO_MUX2 0x70005058 /* GPIO 18 */ #define MT_LED_GPIO_MUX3 0x7000505C /* GPIO 26 */ #define MT_LED_GPIO_SEL_MASK GENMASK(11, 8) @@ -506,7 +545,7 @@ enum base_rev { #define MT_TOP_MISC_FW_STATE GENMASK(2, 0) #define MT_HW_REV 0x70010204 -#define MT_WF_SUBSYS_RST 0x70002600 +#define MT_WF_SUBSYS_RST 0x70028600 /* PCIE MAC */ #define MT_PCIE_MAC_BASE 0x74030000 @@ -539,4 +578,12 @@ enum base_rev { #define MT_WF_PHYRX_CSD_BAND_RXTD12_IRPI_SW_CLR_ONLY BIT(18) #define MT_WF_PHYRX_CSD_BAND_RXTD12_IRPI_SW_CLR BIT(29) +/* CONN MCU EXCP CON */ +#define MT_MCU_WM_EXCP_BASE 0x89050000 +#define MT_MCU_WM_EXCP(ofs) (MT_MCU_WM_EXCP_BASE + (ofs)) +#define MT_MCU_WM_EXCP_PC_CTRL MT_MCU_WM_EXCP(0x100) +#define MT_MCU_WM_EXCP_PC_LOG MT_MCU_WM_EXCP(0x104) +#define MT_MCU_WM_EXCP_LR_CTRL MT_MCU_WM_EXCP(0x200) +#define MT_MCU_WM_EXCP_LR_LOG MT_MCU_WM_EXCP(0x204) + #endif diff --git a/drivers/net/wireless/mediatek/mt76/tx.c b/drivers/net/wireless/mediatek/mt76/tx.c index 1f309d05380a..72b3ec715e47 100644 --- a/drivers/net/wireless/mediatek/mt76/tx.c +++ b/drivers/net/wireless/mediatek/mt76/tx.c @@ -77,7 +77,9 @@ mt76_tx_status_unlock(struct mt76_dev *dev, struct sk_buff_head *list) } hw = mt76_tx_status_get_hw(dev, skb); + spin_lock_bh(&dev->rx_lock); ieee80211_tx_status_ext(hw, &status); + spin_unlock_bh(&dev->rx_lock); } rcu_read_unlock(); } @@ -263,7 +265,9 @@ void __mt76_tx_complete_skb(struct mt76_dev *dev, u16 wcid_idx, struct sk_buff * if (cb->pktid < MT_PACKET_ID_FIRST) { hw = mt76_tx_status_get_hw(dev, skb); status.sta = wcid_to_sta(wcid); + spin_lock_bh(&dev->rx_lock); ieee80211_tx_status_ext(hw, &status); + spin_unlock_bh(&dev->rx_lock); goto out; } @@ -330,7 +334,7 @@ mt76_tx(struct mt76_phy *phy, struct ieee80211_sta *sta, if ((dev->drv->drv_flags & MT_DRV_HW_MGMT_TXQ) && !(info->flags & IEEE80211_TX_CTL_HW_80211_ENCAP) && !ieee80211_is_data(hdr->frame_control) && - !ieee80211_is_bufferable_mmpdu(hdr->frame_control)) { + !ieee80211_is_bufferable_mmpdu(skb)) { qid = MT_TXQ_PSD; } diff --git a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu.h b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu.h index 9d48c69ffece..8eafbf1cee71 100644 --- a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu.h +++ b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu.h @@ -27,7 +27,7 @@ #define RTL8XXXU_MAX_REG_POLL 500 #define USB_INTR_CONTENT_LENGTH 56 -#define RTL8XXXU_OUT_ENDPOINTS 4 +#define RTL8XXXU_OUT_ENDPOINTS 6 #define REALTEK_USB_READ 0xc0 #define REALTEK_USB_WRITE 0x40 @@ -1923,6 +1923,11 @@ struct rtl8xxxu_fileops { u8 has_tx_report:1; u8 gen2_thermal_meter:1; u8 needs_full_init:1; + u8 init_reg_rxfltmap:1; + u8 init_reg_pkt_life_time:1; + u8 init_reg_hmtfr:1; + u8 ampdu_max_time; + u8 ustime_tsf_edca; u32 adda_1t_init; u32 adda_1t_path_on; u32 adda_2t_path_on_a; @@ -1948,10 +1953,22 @@ u32 rtl8xxxu_read32(struct rtl8xxxu_priv *priv, u16 addr); int rtl8xxxu_write8(struct rtl8xxxu_priv *priv, u16 addr, u8 val); int rtl8xxxu_write16(struct rtl8xxxu_priv *priv, u16 addr, u16 val); int rtl8xxxu_write32(struct rtl8xxxu_priv *priv, u16 addr, u32 val); +int rtl8xxxu_write8_set(struct rtl8xxxu_priv *priv, u16 addr, u8 bits); +int rtl8xxxu_write8_clear(struct rtl8xxxu_priv *priv, u16 addr, u8 bits); +int rtl8xxxu_write16_set(struct rtl8xxxu_priv *priv, u16 addr, u16 bits); +int rtl8xxxu_write16_clear(struct rtl8xxxu_priv *priv, u16 addr, u16 bits); +int rtl8xxxu_write32_set(struct rtl8xxxu_priv *priv, u16 addr, u32 bits); +int rtl8xxxu_write32_clear(struct rtl8xxxu_priv *priv, u16 addr, u32 bits); +int rtl8xxxu_write32_mask(struct rtl8xxxu_priv *priv, u16 addr, + u32 mask, u32 val); + u32 rtl8xxxu_read_rfreg(struct rtl8xxxu_priv *priv, enum rtl8xxxu_rfpath path, u8 reg); int rtl8xxxu_write_rfreg(struct rtl8xxxu_priv *priv, enum rtl8xxxu_rfpath path, u8 reg, u32 data); +int rtl8xxxu_write_rfreg_mask(struct rtl8xxxu_priv *priv, + enum rtl8xxxu_rfpath path, u8 reg, + u32 mask, u32 val); void rtl8xxxu_save_regs(struct rtl8xxxu_priv *priv, const u32 *regs, u32 *backup, int count); void rtl8xxxu_restore_regs(struct rtl8xxxu_priv *priv, const u32 *regs, diff --git a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8188e.c b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8188e.c index 6a82ec47568e..8986783ae8fa 100644 --- a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8188e.c +++ b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8188e.c @@ -568,10 +568,6 @@ static int rtl8188eu_parse_efuse(struct rtl8xxxu_priv *priv) priv->default_crystal_cap = efuse->xtal_k & 0x3f; - dev_info(&priv->udev->dev, "Vendor: %.7s\n", efuse->vendor_name); - dev_info(&priv->udev->dev, "Product: %.11s\n", efuse->device_name); - dev_info(&priv->udev->dev, "Serial: %.11s\n", efuse->serial); - return 0; } @@ -1883,6 +1879,7 @@ struct rtl8xxxu_fileops rtl8188eu_fops = { .rx_desc_size = sizeof(struct rtl8xxxu_rxdesc16), .tx_desc_size = sizeof(struct rtl8xxxu_txdesc32), .has_tx_report = 1, + .init_reg_pkt_life_time = 1, .gen2_thermal_meter = 1, .adda_1t_init = 0x0b1b25a0, .adda_1t_path_on = 0x0bdb25a0, diff --git a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8188f.c b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8188f.c index 82dee1fed477..dbdfd7787465 100644 --- a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8188f.c +++ b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8188f.c @@ -734,9 +734,6 @@ static int rtl8188fu_parse_efuse(struct rtl8xxxu_priv *priv) priv->default_crystal_cap = efuse->xtal_k & 0x3f; - dev_info(&priv->udev->dev, "Vendor: %.7s\n", efuse->vendor_name); - dev_info(&priv->udev->dev, "Product: %.7s\n", efuse->device_name); - return 0; } @@ -1746,6 +1743,11 @@ struct rtl8xxxu_fileops rtl8188fu_fops = { .has_tx_report = 1, .gen2_thermal_meter = 1, .needs_full_init = 1, + .init_reg_rxfltmap = 1, + .init_reg_pkt_life_time = 1, + .init_reg_hmtfr = 1, + .ampdu_max_time = 0x70, + .ustime_tsf_edca = 0x28, .adda_1t_init = 0x03c00014, .adda_1t_path_on = 0x03c00014, .trxff_boundary = 0x3f7f, diff --git a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8192c.c b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8192c.c index caeba56241fc..b30a9a513cb8 100644 --- a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8192c.c +++ b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8192c.c @@ -441,11 +441,6 @@ static int rtl8192cu_parse_efuse(struct rtl8xxxu_priv *priv) efuse->ht20_max_power_offset, sizeof(efuse->ht20_max_power_offset)); - dev_info(&priv->udev->dev, "Vendor: %.7s\n", - efuse->vendor_name); - dev_info(&priv->udev->dev, "Product: %.20s\n", - efuse->device_name); - priv->power_base = &rtl8192c_power_base; if (efuse->rf_regulatory & 0x20) { diff --git a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8192e.c b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8192e.c index 4498748164af..fcc2926ea938 100644 --- a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8192e.c +++ b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8192e.c @@ -601,43 +601,9 @@ rtl8192e_set_tx_power(struct rtl8xxxu_priv *priv, int channel, bool ht40) } } -static void rtl8192eu_log_next_device_info(struct rtl8xxxu_priv *priv, - char *record_name, - char *device_info, - unsigned int *record_offset) -{ - char *record = device_info + *record_offset; - - /* A record is [ total length | 0x03 | value ] */ - unsigned char l = record[0]; - - /* - * The whole device info section seems to be 80 characters, make sure - * we don't read further. - */ - if (*record_offset + l > 80) { - dev_warn(&priv->udev->dev, - "invalid record length %d while parsing \"%s\" at offset %u.\n", - l, record_name, *record_offset); - return; - } - - if (l >= 2) { - char value[80]; - - memcpy(value, &record[2], l - 2); - value[l - 2] = '\0'; - dev_info(&priv->udev->dev, "%s: %s\n", record_name, value); - *record_offset = *record_offset + l; - } else { - dev_info(&priv->udev->dev, "%s not available.\n", record_name); - } -} - static int rtl8192eu_parse_efuse(struct rtl8xxxu_priv *priv) { struct rtl8192eu_efuse *efuse = &priv->efuse_wifi.efuse8192eu; - unsigned int record_offset; int i; if (efuse->rtl_id != cpu_to_le16(0x8129)) @@ -684,26 +650,6 @@ static int rtl8192eu_parse_efuse(struct rtl8xxxu_priv *priv) priv->default_crystal_cap = priv->efuse_wifi.efuse8192eu.xtal_k & 0x3f; - /* - * device_info section seems to be laid out as records - * [ total length | 0x03 | value ] so: - * - vendor length + 2 - * - 0x03 - * - vendor string (not null terminated) - * - product length + 2 - * - 0x03 - * - product string (not null terminated) - * Then there is one or 2 0x00 on all the 4 devices I own or found - * dumped online. - * As previous version of the code handled an optional serial - * string, I now assume there may be a third record if the - * length is not 0. - */ - record_offset = 0; - rtl8192eu_log_next_device_info(priv, "Vendor", efuse->device_info, &record_offset); - rtl8192eu_log_next_device_info(priv, "Product", efuse->device_info, &record_offset); - rtl8192eu_log_next_device_info(priv, "Serial", efuse->device_info, &record_offset); - return 0; } diff --git a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8710b.c b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8710b.c index 920466e39604..22d4704dd31e 100644 --- a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8710b.c +++ b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8710b.c @@ -1865,6 +1865,15 @@ struct rtl8xxxu_fileops rtl8710bu_fops = { .has_tx_report = 1, .gen2_thermal_meter = 1, .needs_full_init = 1, + .init_reg_rxfltmap = 1, + .init_reg_pkt_life_time = 1, + .init_reg_hmtfr = 1, + .ampdu_max_time = 0x5e, + /* + * The RTL8710BU vendor driver uses 0x50 here and it works fine, + * but in rtl8xxxu 0x50 causes slow upload and random packet loss. Why? + */ + .ustime_tsf_edca = 0x28, .adda_1t_init = 0x03c00016, .adda_1t_path_on = 0x03c00016, .trxff_boundary = 0x3f7f, diff --git a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8723a.c b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8723a.c index d219be19d07f..15a30e496221 100644 --- a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8723a.c +++ b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8723a.c @@ -222,10 +222,6 @@ static int rtl8723au_parse_efuse(struct rtl8xxxu_priv *priv) priv->power_base = &rtl8723a_power_base; - dev_info(&priv->udev->dev, "Vendor: %.7s\n", - efuse->vendor_name); - dev_info(&priv->udev->dev, "Product: %.41s\n", - efuse->device_name); return 0; } diff --git a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8723b.c b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8723b.c index d99538eb8398..abc56c7de6f7 100644 --- a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8723b.c +++ b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_8723b.c @@ -494,9 +494,6 @@ static int rtl8723bu_parse_efuse(struct rtl8xxxu_priv *priv) priv->default_crystal_cap = priv->efuse_wifi.efuse8723bu.xtal_k & 0x3f; - dev_info(&priv->udev->dev, "Vendor: %.7s\n", efuse->vendor_name); - dev_info(&priv->udev->dev, "Product: %.41s\n", efuse->device_name); - return 0; } @@ -1741,6 +1738,9 @@ struct rtl8xxxu_fileops rtl8723bu_fops = { .has_tx_report = 1, .gen2_thermal_meter = 1, .needs_full_init = 1, + .init_reg_hmtfr = 1, + .ampdu_max_time = 0x5e, + .ustime_tsf_edca = 0x50, .adda_1t_init = 0x01c00014, .adda_1t_path_on = 0x01c00014, .adda_2t_path_on_a = 0x01c00014, diff --git a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_core.c b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_core.c index c152b228606f..fd8c8c6d53d6 100644 --- a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_core.c +++ b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_core.c @@ -786,6 +786,85 @@ int rtl8xxxu_write32(struct rtl8xxxu_priv *priv, u16 addr, u32 val) return ret; } +int rtl8xxxu_write8_set(struct rtl8xxxu_priv *priv, u16 addr, u8 bits) +{ + u8 val8; + + val8 = rtl8xxxu_read8(priv, addr); + val8 |= bits; + return rtl8xxxu_write8(priv, addr, val8); +} + +int rtl8xxxu_write8_clear(struct rtl8xxxu_priv *priv, u16 addr, u8 bits) +{ + u8 val8; + + val8 = rtl8xxxu_read8(priv, addr); + val8 &= ~bits; + return rtl8xxxu_write8(priv, addr, val8); +} + +int rtl8xxxu_write16_set(struct rtl8xxxu_priv *priv, u16 addr, u16 bits) +{ + u16 val16; + + val16 = rtl8xxxu_read16(priv, addr); + val16 |= bits; + return rtl8xxxu_write16(priv, addr, val16); +} + +int rtl8xxxu_write16_clear(struct rtl8xxxu_priv *priv, u16 addr, u16 bits) +{ + u16 val16; + + val16 = rtl8xxxu_read16(priv, addr); + val16 &= ~bits; + return rtl8xxxu_write16(priv, addr, val16); +} + +int rtl8xxxu_write32_set(struct rtl8xxxu_priv *priv, u16 addr, u32 bits) +{ + u32 val32; + + val32 = rtl8xxxu_read32(priv, addr); + val32 |= bits; + return rtl8xxxu_write32(priv, addr, val32); +} + +int rtl8xxxu_write32_clear(struct rtl8xxxu_priv *priv, u16 addr, u32 bits) +{ + u32 val32; + + val32 = rtl8xxxu_read32(priv, addr); + val32 &= ~bits; + return rtl8xxxu_write32(priv, addr, val32); +} + +int rtl8xxxu_write32_mask(struct rtl8xxxu_priv *priv, u16 addr, + u32 mask, u32 val) +{ + u32 orig, new, shift; + + shift = __ffs(mask); + + orig = rtl8xxxu_read32(priv, addr); + new = (orig & ~mask) | ((val << shift) & mask); + return rtl8xxxu_write32(priv, addr, new); +} + +int rtl8xxxu_write_rfreg_mask(struct rtl8xxxu_priv *priv, + enum rtl8xxxu_rfpath path, u8 reg, + u32 mask, u32 val) +{ + u32 orig, new, shift; + + shift = __ffs(mask); + + orig = rtl8xxxu_read_rfreg(priv, path, reg); + new = (orig & ~mask) | ((val << shift) & mask); + return rtl8xxxu_write_rfreg(priv, path, reg, new); +} + static int rtl8xxxu_writeN(struct rtl8xxxu_priv *priv, u16 addr, u8 *buf, u16 len) { @@ -1663,6 +1742,8 @@ int rtl8xxxu_config_endpoints_no_sie(struct rtl8xxxu_priv *priv) struct device *dev = &priv->udev->dev; switch (priv->nr_out_eps) { + case 6: + case 5: case 4: case 3: priv->ep_tx_low_queue = 1; @@ -1916,7 +1997,7 @@ static int rtl8xxxu_start_firmware(struct rtl8xxxu_priv *priv) /* * Init H2C command */ - if (priv->rtl_chip == RTL8723B || priv->rtl_chip == RTL8188F || priv->rtl_chip == RTL8710B) + if (priv->fops->init_reg_hmtfr) rtl8xxxu_write8(priv, REG_HMTFR, 0x0f); exit: return ret; @@ -3864,11 +3945,8 @@ void rtl8xxxu_init_burst(struct rtl8xxxu_priv *priv) rtl8xxxu_write8(priv, REG_HT_SINGLE_AMPDU_8723B, val8); rtl8xxxu_write16(priv, REG_MAX_AGGR_NUM, 0x0c14); - if (priv->rtl_chip == RTL8723B || priv->rtl_chip == RTL8710B) - val8 = 0x5e; - else if (priv->rtl_chip == RTL8188F) - val8 = 0x70; /* 0x5e would make it very slow */ - rtl8xxxu_write8(priv, REG_AMPDU_MAX_TIME_8723B, val8); + rtl8xxxu_write8(priv, REG_AMPDU_MAX_TIME_8723B, + priv->fops->ampdu_max_time); rtl8xxxu_write32(priv, REG_AGGLEN_LMT, 0xffffffff); rtl8xxxu_write8(priv, REG_RX_PKT_LIMIT, 0x18); rtl8xxxu_write8(priv, REG_PIFS, 0x00); @@ -3876,16 +3954,8 @@ void rtl8xxxu_init_burst(struct rtl8xxxu_priv *priv) rtl8xxxu_write8(priv, REG_FWHW_TXQ_CTRL, FWHW_TXQ_CTRL_AMPDU_RETRY); rtl8xxxu_write32(priv, REG_FAST_EDCA_CTRL, 0x03086666); } - /* - * The RTL8710BU vendor driver uses 0x50 here and it works fine, - * but in rtl8xxxu 0x50 causes slow upload and random packet loss. Why? - */ - if (priv->rtl_chip == RTL8723B) - val8 = 0x50; - else if (priv->rtl_chip == RTL8188F || priv->rtl_chip == RTL8710B) - val8 = 0x28; /* 0x50 would make the upload slow */ - rtl8xxxu_write8(priv, REG_USTIME_TSF_8723B, val8); - rtl8xxxu_write8(priv, REG_USTIME_EDCA, val8); + rtl8xxxu_write8(priv, REG_USTIME_TSF_8723B, priv->fops->ustime_tsf_edca); + rtl8xxxu_write8(priv, REG_USTIME_EDCA, priv->fops->ustime_tsf_edca); /* to prevent mac is reseted by bus. */ val8 = rtl8xxxu_read8(priv, REG_RSV_CTRL); @@ -4102,7 +4172,7 @@ static int rtl8xxxu_init_device(struct ieee80211_hw *hw) RCR_APPEND_PHYSTAT | RCR_APPEND_ICV | RCR_APPEND_MIC; rtl8xxxu_write32(priv, REG_RCR, val32); - if (priv->rtl_chip == RTL8188F || priv->rtl_chip == RTL8710B) { + if (fops->init_reg_rxfltmap) { /* Accept all data frames */ rtl8xxxu_write16(priv, REG_RXFLTMAP2, 0xffff); @@ -4187,8 +4257,7 @@ static int rtl8xxxu_init_device(struct ieee80211_hw *hw) if (fops->init_aggregation) fops->init_aggregation(priv); - if (priv->rtl_chip == RTL8188F || priv->rtl_chip == RTL8188E || - priv->rtl_chip == RTL8710B) { + if (fops->init_reg_pkt_life_time) { rtl8xxxu_write16(priv, REG_PKT_VO_VI_LIFE_TIME, 0x0400); /* unit: 256us. 256ms */ rtl8xxxu_write16(priv, REG_PKT_BE_BK_LIFE_TIME, 0x0400); /* unit: 256us. 256ms */ } @@ -6965,10 +7034,8 @@ exit: rtl8xxxu_write16(priv, REG_RXFLTMAP2, 0xffff); rtl8xxxu_write16(priv, REG_RXFLTMAP0, 0xffff); - if (priv->rtl_chip == RTL8188E) - rtl8xxxu_write32(priv, REG_OFDM0_XA_AGC_CORE1, 0x6955341e); - else - rtl8xxxu_write32(priv, REG_OFDM0_XA_AGC_CORE1, 0x6954341e); + rtl8xxxu_write32_mask(priv, REG_OFDM0_XA_AGC_CORE1, + OFDM0_X_AGC_CORE1_IGI_MASK, 0x1e); return ret; diff --git a/drivers/net/wireless/realtek/rtw88/Kconfig b/drivers/net/wireless/realtek/rtw88/Kconfig index 651ab56d9c6b..29eb2f8e0eb7 100644 --- a/drivers/net/wireless/realtek/rtw88/Kconfig +++ b/drivers/net/wireless/realtek/rtw88/Kconfig @@ -16,6 +16,9 @@ config RTW88_CORE config RTW88_PCI tristate +config RTW88_SDIO + tristate + config RTW88_USB tristate @@ -42,6 +45,17 @@ config RTW88_8822BE 802.11ac PCIe wireless network adapter +config RTW88_8822BS + tristate "Realtek 8822BS SDIO wireless network adapter" + depends on MMC + select RTW88_CORE + select RTW88_SDIO + select RTW88_8822B + help + Select this option will enable support for 8822BS chipset + + 802.11ac SDIO wireless network adapter + config RTW88_8822BU tristate "Realtek 8822BU USB wireless network adapter" depends on USB @@ -64,6 +78,17 @@ config RTW88_8822CE 802.11ac PCIe wireless network adapter +config RTW88_8822CS + tristate "Realtek 8822CS SDIO wireless network adapter" + depends on MMC + select RTW88_CORE + select RTW88_SDIO + select RTW88_8822C + help + Select this option will enable support for 8822CS chipset + + 802.11ac SDIO wireless network adapter + config RTW88_8822CU tristate "Realtek 8822CU USB wireless network adapter" depends on USB @@ -108,6 +133,17 @@ config RTW88_8821CE 802.11ac PCIe wireless network adapter +config RTW88_8821CS + tristate "Realtek 8821CS SDIO wireless network adapter" + depends on MMC + select RTW88_CORE + select RTW88_SDIO + select RTW88_8821C + help + Select this option will enable support for 8821CS chipset + + 802.11ac SDIO wireless network adapter + config RTW88_8821CU tristate "Realtek 8821CU USB wireless network adapter" depends on USB diff --git a/drivers/net/wireless/realtek/rtw88/Makefile b/drivers/net/wireless/realtek/rtw88/Makefile index fe7293ee87b4..82979b30ae8d 100644 --- a/drivers/net/wireless/realtek/rtw88/Makefile +++ b/drivers/net/wireless/realtek/rtw88/Makefile @@ -26,6 +26,9 @@ rtw88_8822b-objs := rtw8822b.o rtw8822b_table.o obj-$(CONFIG_RTW88_8822BE) += rtw88_8822be.o rtw88_8822be-objs := rtw8822be.o +obj-$(CONFIG_RTW88_8822BS) += rtw88_8822bs.o +rtw88_8822bs-objs := rtw8822bs.o + obj-$(CONFIG_RTW88_8822BU) += rtw88_8822bu.o rtw88_8822bu-objs := rtw8822bu.o @@ -35,6 +38,9 @@ rtw88_8822c-objs := rtw8822c.o rtw8822c_table.o obj-$(CONFIG_RTW88_8822CE) += rtw88_8822ce.o rtw88_8822ce-objs := rtw8822ce.o +obj-$(CONFIG_RTW88_8822CS) += rtw88_8822cs.o +rtw88_8822cs-objs := rtw8822cs.o + obj-$(CONFIG_RTW88_8822CU) += rtw88_8822cu.o rtw88_8822cu-objs := rtw8822cu.o @@ -53,11 +59,17 @@ rtw88_8821c-objs := rtw8821c.o rtw8821c_table.o obj-$(CONFIG_RTW88_8821CE) += rtw88_8821ce.o rtw88_8821ce-objs := rtw8821ce.o +obj-$(CONFIG_RTW88_8821CS) += rtw88_8821cs.o +rtw88_8821cs-objs := rtw8821cs.o + obj-$(CONFIG_RTW88_8821CU) += rtw88_8821cu.o rtw88_8821cu-objs := rtw8821cu.o obj-$(CONFIG_RTW88_PCI) += rtw88_pci.o rtw88_pci-objs := pci.o +obj-$(CONFIG_RTW88_SDIO) += rtw88_sdio.o +rtw88_sdio-objs := sdio.o + obj-$(CONFIG_RTW88_USB) += rtw88_usb.o rtw88_usb-objs := usb.o diff --git a/drivers/net/wireless/realtek/rtw88/debug.h b/drivers/net/wireless/realtek/rtw88/debug.h index 066792dd96af..a9149c6c2b48 100644 --- a/drivers/net/wireless/realtek/rtw88/debug.h +++ b/drivers/net/wireless/realtek/rtw88/debug.h @@ -24,6 +24,7 @@ enum rtw_debug_mask { RTW_DBG_ADAPTIVITY = 0x00008000, RTW_DBG_HW_SCAN = 0x00010000, RTW_DBG_STATE = 0x00020000, + RTW_DBG_SDIO = 0x00040000, RTW_DBG_ALL = 0xffffffff }; diff --git a/drivers/net/wireless/realtek/rtw88/fw.c b/drivers/net/wireless/realtek/rtw88/fw.c index 82295ac6402e..2a8ccc8a7f60 100644 --- a/drivers/net/wireless/realtek/rtw88/fw.c +++ b/drivers/net/wireless/realtek/rtw88/fw.c @@ -1393,6 +1393,10 @@ static void rtw_build_rsvd_page_iter(void *data, u8 *mac, struct rtw_vif *rtwvif = (struct rtw_vif *)vif->drv_priv; struct rtw_rsvd_page *rsvd_pkt; + /* AP not yet started, don't gather its rsvd pages */ + if (vif->type == NL80211_IFTYPE_AP && !rtwdev->ap_active) + return; + list_for_each_entry(rsvd_pkt, &rtwvif->rsvd_page_list, vif_list) { if (rsvd_pkt->type == RSVD_BEACON) list_add(&rsvd_pkt->build_list, @@ -1614,6 +1618,7 @@ void rtw_fw_update_beacon_work(struct work_struct *work) mutex_lock(&rtwdev->mutex); rtw_fw_download_rsvd_page(rtwdev); + rtw_send_rsvd_page_h2c(rtwdev); mutex_unlock(&rtwdev->mutex); } @@ -2155,11 +2160,19 @@ int rtw_hw_scan_offload(struct rtw_dev *rtwdev, struct ieee80211_vif *vif, } rtw_fw_set_scan_offload(rtwdev, &cs_option, rtwvif, &chan_list); out: + if (rtwdev->ap_active) { + ret = rtw_download_beacon(rtwdev); + if (ret) + rtw_err(rtwdev, "HW scan download beacon failed\n"); + } + return ret; } -void rtw_hw_scan_abort(struct rtw_dev *rtwdev, struct ieee80211_vif *vif) +void rtw_hw_scan_abort(struct rtw_dev *rtwdev) { + struct ieee80211_vif *vif = rtwdev->scan_info.scanning_vif; + if (!rtw_fw_feature_check(&rtwdev->fw, FW_FEATURE_SCAN_OFFLOAD)) return; @@ -2244,6 +2257,7 @@ void rtw_hw_scan_chan_switch(struct rtw_dev *rtwdev, struct sk_buff *skb) if (rtw_is_op_chan(rtwdev, chan)) { rtw_store_op_chan(rtwdev, false); ieee80211_wake_queues(rtwdev->hw); + rtw_core_enable_beacon(rtwdev, true); } } else if (id == RTW_SCAN_NOTIFY_ID_PRESWITCH) { if (IS_CH_5G_BAND(chan)) { @@ -2262,8 +2276,10 @@ void rtw_hw_scan_chan_switch(struct rtw_dev *rtwdev, struct sk_buff *skb) * if next channel is non-op channel. */ if (!rtw_is_op_chan(rtwdev, chan) && - rtw_is_op_chan(rtwdev, hal->current_channel)) + rtw_is_op_chan(rtwdev, hal->current_channel)) { + rtw_core_enable_beacon(rtwdev, false); ieee80211_stop_queues(rtwdev->hw); + } } rtw_dbg(rtwdev, RTW_DBG_HW_SCAN, diff --git a/drivers/net/wireless/realtek/rtw88/fw.h b/drivers/net/wireless/realtek/rtw88/fw.h index 0a386e6d6e0d..397cbc3f6af6 100644 --- a/drivers/net/wireless/realtek/rtw88/fw.h +++ b/drivers/net/wireless/realtek/rtw88/fw.h @@ -868,5 +868,5 @@ int rtw_hw_scan_offload(struct rtw_dev *rtwdev, struct ieee80211_vif *vif, bool enable); void rtw_hw_scan_status_report(struct rtw_dev *rtwdev, struct sk_buff *skb); void rtw_hw_scan_chan_switch(struct rtw_dev *rtwdev, struct sk_buff *skb); -void rtw_hw_scan_abort(struct rtw_dev *rtwdev, struct ieee80211_vif *vif); +void rtw_hw_scan_abort(struct rtw_dev *rtwdev); #endif diff --git a/drivers/net/wireless/realtek/rtw88/mac.c b/drivers/net/wireless/realtek/rtw88/mac.c index f3a566cf979b..a168f36c38ec 100644 --- a/drivers/net/wireless/realtek/rtw88/mac.c +++ b/drivers/net/wireless/realtek/rtw88/mac.c @@ -7,6 +7,7 @@ #include "reg.h" #include "fw.h" #include "debug.h" +#include "sdio.h" void rtw_set_channel_mac(struct rtw_dev *rtwdev, u8 channel, u8 bw, u8 primary_ch_idx) @@ -60,6 +61,7 @@ EXPORT_SYMBOL(rtw_set_channel_mac); static int rtw_mac_pre_system_cfg(struct rtw_dev *rtwdev) { + unsigned int retry; u32 value32; u8 value8; @@ -77,6 +79,28 @@ static int rtw_mac_pre_system_cfg(struct rtw_dev *rtwdev) case RTW_HCI_TYPE_PCIE: rtw_write32_set(rtwdev, REG_HCI_OPT_CTRL, BIT_USB_SUS_DIS); break; + case RTW_HCI_TYPE_SDIO: + rtw_write8_clr(rtwdev, REG_SDIO_HSUS_CTRL, BIT_HCI_SUS_REQ); + + for (retry = 0; retry < RTW_PWR_POLLING_CNT; retry++) { + if (rtw_read8(rtwdev, REG_SDIO_HSUS_CTRL) & BIT_HCI_RESUME_RDY) + break; + + usleep_range(10, 50); + } + + if (retry == RTW_PWR_POLLING_CNT) { + rtw_err(rtwdev, "failed to poll REG_SDIO_HSUS_CTRL[1]"); + return -ETIMEDOUT; + } + + if (rtw_sdio_is_sdio30_supported(rtwdev)) + rtw_write8_set(rtwdev, REG_HCI_OPT_CTRL + 2, + BIT_SDIO_PAD_E5 >> 16); + else + rtw_write8_clr(rtwdev, REG_HCI_OPT_CTRL + 2, + BIT_SDIO_PAD_E5 >> 16); + break; case RTW_HCI_TYPE_USB: break; default: @@ -248,6 +272,7 @@ static int rtw_mac_power_switch(struct rtw_dev *rtwdev, bool pwr_on) { const struct rtw_chip_info *chip = rtwdev->chip; const struct rtw_pwr_seq_cmd **pwr_seq; + u32 imr = 0; u8 rpwm; bool cur_pwr; int ret; @@ -273,17 +298,24 @@ static int rtw_mac_power_switch(struct rtw_dev *rtwdev, bool pwr_on) if (pwr_on == cur_pwr) return -EALREADY; + if (rtw_hci_type(rtwdev) == RTW_HCI_TYPE_SDIO) { + imr = rtw_read32(rtwdev, REG_SDIO_HIMR); + rtw_write32(rtwdev, REG_SDIO_HIMR, 0); + } + + if (!pwr_on) + clear_bit(RTW_FLAG_POWERON, rtwdev->flags); + pwr_seq = pwr_on ? chip->pwr_on_seq : chip->pwr_off_seq; ret = rtw_pwr_seq_parser(rtwdev, pwr_seq); - if (ret) - return ret; - if (pwr_on) + if (rtw_hci_type(rtwdev) == RTW_HCI_TYPE_SDIO) + rtw_write32(rtwdev, REG_SDIO_HIMR, imr); + + if (!ret && pwr_on) set_bit(RTW_FLAG_POWERON, rtwdev->flags); - else - clear_bit(RTW_FLAG_POWERON, rtwdev->flags); - return 0; + return ret; } static int __rtw_mac_init_system_cfg(struct rtw_dev *rtwdev) @@ -454,6 +486,9 @@ static void download_firmware_reg_backup(struct rtw_dev *rtwdev, rtw_write16(rtwdev, REG_FIFOPAGE_INFO_1, 0x200); rtw_write32(rtwdev, REG_RQPN_CTRL_2, bckp[bckp_idx - 1].val); + if (rtw_hci_type(rtwdev) == RTW_HCI_TYPE_SDIO) + rtw_read32(rtwdev, REG_SDIO_FREE_TXPG); + /* Disable beacon related functions */ tmp = rtw_read8(rtwdev, REG_BCN_CTRL); bckp[bckp_idx].len = 1; @@ -1066,8 +1101,12 @@ static int txdma_queue_mapping(struct rtw_dev *rtwdev) if (rtw_chip_wcpu_11ac(rtwdev)) rtw_write32(rtwdev, REG_H2CQ_CSR, BIT_H2CQ_FULL); - if (rtw_hci_type(rtwdev) == RTW_HCI_TYPE_USB) + if (rtw_hci_type(rtwdev) == RTW_HCI_TYPE_SDIO) { + rtw_read32(rtwdev, REG_SDIO_FREE_TXPG); + rtw_write32(rtwdev, REG_SDIO_TX_CTRL, 0); + } else if (rtw_hci_type(rtwdev) == RTW_HCI_TYPE_USB) { rtw_write8_set(rtwdev, REG_TXDMA_PQ_MAP, BIT_RXDMA_ARBBW_EN); + } return 0; } @@ -1080,7 +1119,7 @@ static int set_trx_fifo_info(struct rtw_dev *rtwdev) u8 csi_buf_pg_num = chip->csi_buf_pg_num; /* config rsvd page num */ - fifo->rsvd_drv_pg_num = 8; + fifo->rsvd_drv_pg_num = chip->rsvd_drv_pg_num; fifo->txff_pg_num = chip->txff_size >> 7; if (rtw_chip_wcpu_11n(rtwdev)) fifo->rsvd_pg_num = fifo->rsvd_drv_pg_num; diff --git a/drivers/net/wireless/realtek/rtw88/mac.h b/drivers/net/wireless/realtek/rtw88/mac.h index 3172aa5ac4de..58c3dccc14bb 100644 --- a/drivers/net/wireless/realtek/rtw88/mac.h +++ b/drivers/net/wireless/realtek/rtw88/mac.h @@ -7,7 +7,6 @@ #define RTW_HW_PORT_NUM 5 #define cut_version_to_mask(cut) (0x1 << ((cut) + 1)) -#define SDIO_LOCAL_OFFSET 0x10250000 #define DDMA_POLLING_COUNT 1000 #define C2H_PKT_BUF 256 #define REPORT_BUF 128 diff --git a/drivers/net/wireless/realtek/rtw88/mac80211.c b/drivers/net/wireless/realtek/rtw88/mac80211.c index 3b92ac611d3f..7aa6edad0d01 100644 --- a/drivers/net/wireless/realtek/rtw88/mac80211.c +++ b/drivers/net/wireless/realtek/rtw88/mac80211.c @@ -155,25 +155,30 @@ static int rtw_ops_add_interface(struct ieee80211_hw *hw, struct rtw_vif *rtwvif = (struct rtw_vif *)vif->drv_priv; enum rtw_net_type net_type; u32 config = 0; - u8 port = 0; + u8 port; u8 bcn_ctrl = 0; if (rtw_fw_feature_check(&rtwdev->fw, FW_FEATURE_BCN_FILTER)) vif->driver_flags |= IEEE80211_VIF_BEACON_FILTER | IEEE80211_VIF_SUPPORTS_CQM_RSSI; - rtwvif->port = port; rtwvif->stats.tx_unicast = 0; rtwvif->stats.rx_unicast = 0; rtwvif->stats.tx_cnt = 0; rtwvif->stats.rx_cnt = 0; rtwvif->scan_req = NULL; memset(&rtwvif->bfee, 0, sizeof(struct rtw_bfee)); - rtwvif->conf = &rtw_vif_port[port]; rtw_txq_init(rtwdev, vif->txq); INIT_LIST_HEAD(&rtwvif->rsvd_page_list); mutex_lock(&rtwdev->mutex); + port = find_first_zero_bit(rtwdev->hw_port, RTW_PORT_NUM); + if (port >= RTW_PORT_NUM) + return -EINVAL; + set_bit(port, rtwdev->hw_port); + + rtwvif->port = port; + rtwvif->conf = &rtw_vif_port[port]; rtw_leave_lps_deep(rtwdev); switch (vif->type) { @@ -195,6 +200,7 @@ static int rtw_ops_add_interface(struct ieee80211_hw *hw, break; default: WARN_ON(1); + clear_bit(rtwvif->port, rtwdev->hw_port); mutex_unlock(&rtwdev->mutex); return -EINVAL; } @@ -206,6 +212,7 @@ static int rtw_ops_add_interface(struct ieee80211_hw *hw, rtwvif->bcn_ctrl = bcn_ctrl; config |= PORT_SET_BCN_CTRL; rtw_vif_port_config(rtwdev, rtwvif, config); + rtw_core_port_switch(rtwdev, vif); mutex_unlock(&rtwdev->mutex); @@ -236,6 +243,7 @@ static void rtw_ops_remove_interface(struct ieee80211_hw *hw, rtwvif->bcn_ctrl = 0; config |= PORT_SET_BCN_CTRL; rtw_vif_port_config(rtwdev, rtwvif, config); + clear_bit(rtwvif->port, rtwdev->hw_port); mutex_unlock(&rtwdev->mutex); } @@ -385,7 +393,8 @@ static void rtw_ops_bss_info_changed(struct ieee80211_hw *hw, * when disconnected by peer */ if (test_bit(RTW_FLAG_SCANNING, rtwdev->flags)) - rtw_hw_scan_abort(rtwdev, vif); + rtw_hw_scan_abort(rtwdev); + } config |= PORT_SET_NET_TYPE; @@ -395,7 +404,7 @@ static void rtw_ops_bss_info_changed(struct ieee80211_hw *hw, if (changed & BSS_CHANGED_BSSID) { ether_addr_copy(rtwvif->bssid, conf->bssid); config |= PORT_SET_BSSID; - if (is_zero_ether_addr(rtwvif->bssid)) + if (!rtw_core_check_sta_active(rtwdev)) rtw_clear_op_chan(rtwdev); else rtw_store_op_chan(rtwdev, true); @@ -409,6 +418,7 @@ static void rtw_ops_bss_info_changed(struct ieee80211_hw *hw, if (changed & BSS_CHANGED_BEACON) { rtw_set_dtim_period(rtwdev, conf->dtim_period); rtw_fw_download_rsvd_page(rtwdev); + rtw_send_rsvd_page_h2c(rtwdev); } if (changed & BSS_CHANGED_BEACON_ENABLED) { @@ -441,12 +451,27 @@ static int rtw_ops_start_ap(struct ieee80211_hw *hw, const struct rtw_chip_info *chip = rtwdev->chip; mutex_lock(&rtwdev->mutex); + rtwdev->ap_active = true; + rtw_store_op_chan(rtwdev, true); chip->ops->phy_calibration(rtwdev); mutex_unlock(&rtwdev->mutex); return 0; } +static void rtw_ops_stop_ap(struct ieee80211_hw *hw, + struct ieee80211_vif *vif, + struct ieee80211_bss_conf *link_conf) +{ + struct rtw_dev *rtwdev = hw->priv; + + mutex_lock(&rtwdev->mutex); + rtwdev->ap_active = false; + if (!rtw_core_check_sta_active(rtwdev)) + rtw_clear_op_chan(rtwdev); + mutex_unlock(&rtwdev->mutex); +} + static int rtw_ops_conf_tx(struct ieee80211_hw *hw, struct ieee80211_vif *vif, unsigned int link_id, u16 ac, @@ -849,7 +874,7 @@ static int rtw_ops_hw_scan(struct ieee80211_hw *hw, struct ieee80211_vif *vif, rtw_hw_scan_start(rtwdev, vif, req); ret = rtw_hw_scan_offload(rtwdev, vif, true); if (ret) { - rtw_hw_scan_abort(rtwdev, vif); + rtw_hw_scan_abort(rtwdev); rtw_err(rtwdev, "HW scan failed with status: %d\n", ret); } mutex_unlock(&rtwdev->mutex); @@ -869,7 +894,7 @@ static void rtw_ops_cancel_hw_scan(struct ieee80211_hw *hw, return; mutex_lock(&rtwdev->mutex); - rtw_hw_scan_abort(rtwdev, vif); + rtw_hw_scan_abort(rtwdev); mutex_unlock(&rtwdev->mutex); } @@ -908,6 +933,7 @@ const struct ieee80211_ops rtw_ops = { .configure_filter = rtw_ops_configure_filter, .bss_info_changed = rtw_ops_bss_info_changed, .start_ap = rtw_ops_start_ap, + .stop_ap = rtw_ops_stop_ap, .conf_tx = rtw_ops_conf_tx, .sta_add = rtw_ops_sta_add, .sta_remove = rtw_ops_sta_remove, diff --git a/drivers/net/wireless/realtek/rtw88/main.c b/drivers/net/wireless/realtek/rtw88/main.c index b2e78737bd5d..5bf6b4581557 100644 --- a/drivers/net/wireless/realtek/rtw88/main.c +++ b/drivers/net/wireless/realtek/rtw88/main.c @@ -18,6 +18,7 @@ #include "debug.h" #include "bf.h" #include "sar.h" +#include "sdio.h" bool rtw_disable_lps_deep_mode; EXPORT_SYMBOL(rtw_disable_lps_deep_mode); @@ -102,6 +103,26 @@ static struct ieee80211_rate rtw_ratetable[] = { {.bitrate = 540, .hw_value = 0x0b,}, }; +static const struct ieee80211_iface_limit rtw_iface_limits[] = { + { + .max = 1, + .types = BIT(NL80211_IFTYPE_STATION), + }, + { + .max = 1, + .types = BIT(NL80211_IFTYPE_AP), + } +}; + +static const struct ieee80211_iface_combination rtw_iface_combs[] = { + { + .limits = rtw_iface_limits, + .n_limits = ARRAY_SIZE(rtw_iface_limits), + .max_interfaces = 2, + .num_different_channels = 1, + } +}; + u16 rtw_desc_to_bitrate(u8 desc_rate) { struct ieee80211_rate rate; @@ -256,7 +277,7 @@ static void rtw_watch_dog_work(struct work_struct *work) * threshold. */ if (rtwdev->ps_enabled && data.rtwvif && !ps_active && - !rtwdev->beacon_loss) + !rtwdev->beacon_loss && !rtwdev->ap_active) rtw_enter_lps(rtwdev, data.rtwvif->port); rtwdev->watch_dog_cnt++; @@ -609,6 +630,7 @@ free: rcu_read_unlock(); rtw_iterate_stas_atomic(rtwdev, rtw_reset_sta_iter, rtwdev); rtw_iterate_vifs_atomic(rtwdev, rtw_reset_vif_iter, rtwdev); + bitmap_zero(rtwdev->hw_port, RTW_PORT_NUM); rtw_enter_ips(rtwdev); } @@ -828,6 +850,9 @@ void rtw_set_channel(struct rtw_dev *rtwdev) rtw_update_channel(rtwdev, center_chan, primary_chan, band, bandwidth); + if (rtwdev->scan_info.op_chan) + rtw_store_op_chan(rtwdev, true); + chip->ops->set_channel(rtwdev, center_chan, bandwidth, hal->current_primary_channel_index); @@ -1785,6 +1810,10 @@ static int rtw_chip_parameter_setup(struct rtw_dev *rtwdev) rtwdev->hci.rpwm_addr = 0x03d9; rtwdev->hci.cpwm_addr = 0x03da; break; + case RTW_HCI_TYPE_SDIO: + rtwdev->hci.rpwm_addr = REG_SDIO_HRPWM1; + rtwdev->hci.cpwm_addr = REG_SDIO_HCPWM1_V2; + break; case RTW_HCI_TYPE_USB: rtwdev->hci.rpwm_addr = 0xfe58; rtwdev->hci.cpwm_addr = 0xfe57; @@ -1979,7 +2008,7 @@ static int rtw_chip_board_info_setup(struct rtw_dev *rtwdev) if (!rfe_def) return -ENODEV; - rtw_phy_setup_phy_cond(rtwdev, 0); + rtw_phy_setup_phy_cond(rtwdev, hal->pkg_type); rtw_phy_init_tx_power(rtwdev); if (rfe_def->agc_btg_tbl) @@ -2158,9 +2187,11 @@ int rtw_register_hw(struct rtw_dev *rtwdev, struct ieee80211_hw *hw) int max_tx_headroom = 0; int ret; - /* TODO: USB & SDIO may need extra room? */ max_tx_headroom = rtwdev->chip->tx_pkt_desc_sz; + if (rtw_hci_type(rtwdev) == RTW_HCI_TYPE_SDIO) + max_tx_headroom += RTW_SDIO_DATA_PTR_ALIGN; + hw->extra_tx_headroom = max_tx_headroom; hw->queues = IEEE80211_NUM_ACS; hw->txq_data_size = sizeof(struct rtw_txq); @@ -2194,6 +2225,11 @@ int rtw_register_hw(struct rtw_dev *rtwdev, struct ieee80211_hw *hw) hw->wiphy->max_scan_ssids = RTW_SCAN_MAX_SSIDS; hw->wiphy->max_scan_ie_len = rtw_get_max_scan_ie_len(rtwdev); + if (rtwdev->chip->id == RTW_CHIP_TYPE_8822C) { + hw->wiphy->iface_combinations = rtw_iface_combs; + hw->wiphy->n_iface_combinations = ARRAY_SIZE(rtw_iface_combs); + } + wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_CAN_REPLACE_PTK0); wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_SCAN_RANDOM_SN); wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_SET_SCAN_DWELL); @@ -2243,6 +2279,121 @@ void rtw_unregister_hw(struct rtw_dev *rtwdev, struct ieee80211_hw *hw) } EXPORT_SYMBOL(rtw_unregister_hw); +static +void rtw_swap_reg_nbytes(struct rtw_dev *rtwdev, const struct rtw_hw_reg *reg1, + const struct rtw_hw_reg *reg2, u8 nbytes) +{ + u8 i; + + for (i = 0; i < nbytes; i++) { + u8 v1 = rtw_read8(rtwdev, reg1->addr + i); + u8 v2 = rtw_read8(rtwdev, reg2->addr + i); + + rtw_write8(rtwdev, reg1->addr + i, v2); + rtw_write8(rtwdev, reg2->addr + i, v1); + } +} + +static +void rtw_swap_reg_mask(struct rtw_dev *rtwdev, const struct rtw_hw_reg *reg1, + const struct rtw_hw_reg *reg2) +{ + u32 v1, v2; + + v1 = rtw_read32_mask(rtwdev, reg1->addr, reg1->mask); + v2 = rtw_read32_mask(rtwdev, reg2->addr, reg2->mask); + rtw_write32_mask(rtwdev, reg2->addr, reg2->mask, v1); + rtw_write32_mask(rtwdev, reg1->addr, reg1->mask, v2); +} + +struct rtw_iter_port_switch_data { + struct rtw_dev *rtwdev; + struct rtw_vif *rtwvif_ap; +}; + +static void rtw_port_switch_iter(void *data, u8 *mac, struct ieee80211_vif *vif) +{ + struct rtw_iter_port_switch_data *iter_data = data; + struct rtw_dev *rtwdev = iter_data->rtwdev; + struct rtw_vif *rtwvif_target = (struct rtw_vif *)vif->drv_priv; + struct rtw_vif *rtwvif_ap = iter_data->rtwvif_ap; + const struct rtw_hw_reg *reg1, *reg2; + + if (rtwvif_target->port != RTW_PORT_0) + return; + + rtw_dbg(rtwdev, RTW_DBG_STATE, "AP port switch from %d -> %d\n", + rtwvif_ap->port, rtwvif_target->port); + + reg1 = &rtwvif_ap->conf->net_type; + reg2 = &rtwvif_target->conf->net_type; + rtw_swap_reg_mask(rtwdev, reg1, reg2); + + reg1 = &rtwvif_ap->conf->mac_addr; + reg2 = &rtwvif_target->conf->mac_addr; + rtw_swap_reg_nbytes(rtwdev, reg1, reg2, ETH_ALEN); + + reg1 = &rtwvif_ap->conf->bssid; + reg2 = &rtwvif_target->conf->bssid; + rtw_swap_reg_nbytes(rtwdev, reg1, reg2, ETH_ALEN); + + reg1 = &rtwvif_ap->conf->bcn_ctrl; + reg2 = &rtwvif_target->conf->bcn_ctrl; + rtw_swap_reg_nbytes(rtwdev, reg1, reg2, 1); + + swap(rtwvif_target->port, rtwvif_ap->port); + swap(rtwvif_target->conf, rtwvif_ap->conf); +} + +void rtw_core_port_switch(struct rtw_dev *rtwdev, struct ieee80211_vif *vif) +{ + struct rtw_vif *rtwvif = (struct rtw_vif *)vif->drv_priv; + struct rtw_iter_port_switch_data iter_data; + + if (vif->type != NL80211_IFTYPE_AP || rtwvif->port == RTW_PORT_0) + return; + + iter_data.rtwdev = rtwdev; + iter_data.rtwvif_ap = rtwvif; + rtw_iterate_vifs(rtwdev, rtw_port_switch_iter, &iter_data); +} + +static void rtw_check_sta_active_iter(void *data, u8 *mac, + struct ieee80211_vif *vif) +{ + struct rtw_vif *rtwvif = (struct rtw_vif *)vif->drv_priv; + bool *active = data; + + if (*active) + return; + + if (vif->type != NL80211_IFTYPE_STATION) + return; + + if (vif->cfg.assoc || !is_zero_ether_addr(rtwvif->bssid)) + *active = true; +} + +bool rtw_core_check_sta_active(struct rtw_dev *rtwdev) +{ + bool sta_active = false; + + rtw_iterate_vifs(rtwdev, rtw_check_sta_active_iter, &sta_active); + + return rtwdev->ap_active || sta_active; +} + +void rtw_core_enable_beacon(struct rtw_dev *rtwdev, bool enable) +{ + if (!rtwdev->ap_active) + return; + + if (enable) + rtw_write32_set(rtwdev, REG_BCN_CTRL, BIT_EN_BCN_FUNCTION); + else + rtw_write32_clr(rtwdev, REG_BCN_CTRL, BIT_EN_BCN_FUNCTION); +} + MODULE_AUTHOR("Realtek Corporation"); MODULE_DESCRIPTION("Realtek 802.11ac wireless core module"); MODULE_LICENSE("Dual BSD/GPL"); diff --git a/drivers/net/wireless/realtek/rtw88/main.h b/drivers/net/wireless/realtek/rtw88/main.h index d4a53d556745..a563285e90ed 100644 --- a/drivers/net/wireless/realtek/rtw88/main.h +++ b/drivers/net/wireless/realtek/rtw88/main.h @@ -88,7 +88,7 @@ enum rtw_supported_band { RTW_BAND_60G = BIT(NL80211_BAND_60GHZ), }; -/* now, support upto 80M bw */ +/* now, support up to 80M bw */ #define RTW_MAX_CHANNEL_WIDTH RTW_CHANNEL_WIDTH_80 enum rtw_bandwidth { @@ -395,6 +395,15 @@ enum rtw_snr { RTW_SNR_NUM }; +enum rtw_port { + RTW_PORT_0 = 0, + RTW_PORT_1 = 1, + RTW_PORT_2 = 2, + RTW_PORT_3 = 3, + RTW_PORT_4 = 4, + RTW_PORT_NUM +}; + enum rtw_wow_flags { RTW_WOW_FLAG_EN_MAGIC_PKT, RTW_WOW_FLAG_EN_REKEY_PKT, @@ -1168,6 +1177,7 @@ struct rtw_chip_info { u32 txff_size; u32 rxff_size; u32 fw_rxff_size; + u16 rsvd_drv_pg_num; u8 band; u8 page_size; u8 csi_buf_pg_num; @@ -1871,7 +1881,7 @@ enum rtw_sar_bands { RTW_SAR_BAND_NR, }; -/* the union is reserved for other knids of SAR sources +/* the union is reserved for other kinds of SAR sources * which might not re-use same format with array common. */ union rtw_sar_cfg { @@ -1890,7 +1900,9 @@ struct rtw_hal { u8 cut_version; u8 mp_chip; u8 oem_id; + u8 pkg_type; struct rtw_phy_cond phy_cond; + bool rfe_btg; u8 ps_mode; u8 current_channel; @@ -2020,7 +2032,7 @@ struct rtw_dev { struct rtw_tx_report tx_report; struct { - /* incicate the mail box to use with fw */ + /* indicate the mail box to use with fw */ u8 last_box_num; u32 seq; } h2c; @@ -2036,6 +2048,7 @@ struct rtw_dev { u8 sta_cnt; u32 rts_threshold; + DECLARE_BITMAP(hw_port, RTW_PORT_NUM); DECLARE_BITMAP(mac_id_map, RTW_MAX_MAC_ID_NUM); DECLARE_BITMAP(flags, NUM_OF_RTW_FLAGS); @@ -2047,6 +2060,7 @@ struct rtw_dev { bool need_rfk; struct completion fw_scan_density; + bool ap_active; /* hci related data, must be last */ u8 priv[] __aligned(sizeof(void *)); @@ -2188,4 +2202,7 @@ void rtw_set_txrx_1ss(struct rtw_dev *rtwdev, bool config_1ss); void rtw_update_channel(struct rtw_dev *rtwdev, u8 center_channel, u8 primary_channel, enum rtw_supported_band band, enum rtw_bandwidth bandwidth); +void rtw_core_port_switch(struct rtw_dev *rtwdev, struct ieee80211_vif *vif); +bool rtw_core_check_sta_active(struct rtw_dev *rtwdev); +void rtw_core_enable_beacon(struct rtw_dev *rtwdev, bool enable); #endif diff --git a/drivers/net/wireless/realtek/rtw88/reg.h b/drivers/net/wireless/realtek/rtw88/reg.h index 8852b24d6c2a..2a2ae2081f34 100644 --- a/drivers/net/wireless/realtek/rtw88/reg.h +++ b/drivers/net/wireless/realtek/rtw88/reg.h @@ -87,6 +87,7 @@ #define BIT_LTE_MUX_CTRL_PATH BIT(26) #define REG_HCI_OPT_CTRL 0x0074 #define BIT_USB_SUS_DIS BIT(8) +#define BIT_SDIO_PAD_E5 BIT(18) #define REG_AFE_CTRL_4 0x0078 #define BIT_CK320M_AFE_EN BIT(4) @@ -185,6 +186,9 @@ (((x) & BIT_MASK_TXDMA_VIQ_MAP) << BIT_SHIFT_TXDMA_VIQ_MAP) #define REG_TXDMA_PQ_MAP 0x010C #define BIT_RXDMA_ARBBW_EN BIT(0) +#define BIT_RXSHFT_EN BIT(1) +#define BIT_RXDMA_AGG_EN BIT(2) +#define BIT_TXDMA_BW_EN BIT(3) #define BIT_SHIFT_TXDMA_BEQ_MAP 8 #define BIT_MASK_TXDMA_BEQ_MAP 0x3 #define BIT_TXDMA_BEQ_MAP(x) \ @@ -283,10 +287,18 @@ #define REG_H2C_TAIL 0x0248 #define REG_H2C_READ_ADDR 0x024C #define REG_H2C_INFO 0x0254 +#define REG_RXDMA_AGG_PG_TH 0x0280 +#define BIT_RXDMA_AGG_PG_TH GENMASK(7, 0) +#define BIT_DMA_AGG_TO_V1 GENMASK(15, 8) +#define BIT_EN_PRE_CALC BIT(29) #define REG_RXPKT_NUM 0x0284 #define BIT_RXDMA_REQ BIT(19) #define BIT_RW_RELEASE BIT(18) #define BIT_RXDMA_IDLE BIT(17) +#define REG_RXDMA_STATUS 0x0288 +#define REG_RXDMA_DPR 0x028C +#define REG_RXDMA_MODE 0x0290 +#define BIT_DMA_MODE BIT(1) #define REG_RXPKTNUM 0x02B0 #define REG_INT_MIG 0x0304 diff --git a/drivers/net/wireless/realtek/rtw88/rtw8723d.c b/drivers/net/wireless/realtek/rtw88/rtw8723d.c index 2d2f768bae2e..06e7454c9ca6 100644 --- a/drivers/net/wireless/realtek/rtw88/rtw8723d.c +++ b/drivers/net/wireless/realtek/rtw88/rtw8723d.c @@ -2743,6 +2743,7 @@ const struct rtw_chip_info rtw8723d_hw_spec = { .ptct_efuse_size = 96 + 1, .txff_size = 32768, .rxff_size = 16384, + .rsvd_drv_pg_num = 8, .txgi_factor = 1, .is_pwr_by_rate_dec = true, .max_power_index = 0x3f, diff --git a/drivers/net/wireless/realtek/rtw88/rtw8821c.c b/drivers/net/wireless/realtek/rtw88/rtw8821c.c index 7ae0541d7b99..adf224618a2a 100644 --- a/drivers/net/wireless/realtek/rtw88/rtw8821c.c +++ b/drivers/net/wireless/realtek/rtw88/rtw8821c.c @@ -47,13 +47,14 @@ enum rtw8821ce_rf_set { static int rtw8821c_read_efuse(struct rtw_dev *rtwdev, u8 *log_map) { + struct rtw_hal *hal = &rtwdev->hal; struct rtw_efuse *efuse = &rtwdev->efuse; struct rtw8821c_efuse *map; int i; map = (struct rtw8821c_efuse *)log_map; - efuse->rfe_option = map->rfe_option; + efuse->rfe_option = map->rfe_option & 0x1f; efuse->rf_board_option = map->rf_board_option; efuse->crystal_cap = map->xtal_k; efuse->pa_type_2g = map->pa_type; @@ -70,6 +71,19 @@ static int rtw8821c_read_efuse(struct rtw_dev *rtwdev, u8 *log_map) efuse->tx_bb_swing_setting_2g = map->tx_bb_swing_setting_2g; efuse->tx_bb_swing_setting_5g = map->tx_bb_swing_setting_5g; + hal->pkg_type = map->rfe_option & BIT(5) ? 1 : 0; + + switch (efuse->rfe_option) { + case 0x2: + case 0x4: + case 0x7: + case 0xa: + case 0xc: + case 0xf: + hal->rfe_btg = true; + break; + } + for (i = 0; i < 4; i++) efuse->txpwr_idx_table[i] = map->txpwr_idx_table[i]; @@ -295,6 +309,7 @@ static void rtw8821c_switch_rf_set(struct rtw_dev *rtwdev, u8 rf_set) static void rtw8821c_set_channel_rf(struct rtw_dev *rtwdev, u8 channel, u8 bw) { + struct rtw_hal *hal = &rtwdev->hal; u32 rf_reg18; rf_reg18 = rtw_read_rf(rtwdev, RF_PATH_A, 0x18, RFREG_MASK); @@ -326,11 +341,10 @@ static void rtw8821c_set_channel_rf(struct rtw_dev *rtwdev, u8 channel, u8 bw) } if (channel <= 14) { - if (rtwdev->efuse.rfe_option == 0) - rtw8821c_switch_rf_set(rtwdev, SWITCH_TO_WLG); - else if (rtwdev->efuse.rfe_option == 2 || - rtwdev->efuse.rfe_option == 4) + if (hal->rfe_btg) rtw8821c_switch_rf_set(rtwdev, SWITCH_TO_BTG); + else + rtw8821c_switch_rf_set(rtwdev, SWITCH_TO_WLG); rtw_write_rf(rtwdev, RF_PATH_A, RF_LUTDBG, BIT(6), 0x1); rtw_write_rf(rtwdev, RF_PATH_A, 0x64, 0xf, 0xf); } else { @@ -1546,7 +1560,6 @@ static const struct rtw_rfe_def rtw8821c_rfe_defs[] = { [2] = RTW_DEF_RFE_EXT(8821c, 0, 0, 2), [4] = RTW_DEF_RFE_EXT(8821c, 0, 0, 2), [6] = RTW_DEF_RFE(8821c, 0, 0), - [34] = RTW_DEF_RFE(8821c, 0, 0), }; static struct rtw_hw_reg rtw8821c_dig[] = { @@ -1920,6 +1933,7 @@ const struct rtw_chip_info rtw8821c_hw_spec = { .ptct_efuse_size = 96, .txff_size = 65536, .rxff_size = 16384, + .rsvd_drv_pg_num = 8, .txgi_factor = 1, .is_pwr_by_rate_dec = true, .max_power_index = 0x3f, diff --git a/drivers/net/wireless/realtek/rtw88/rtw8821cs.c b/drivers/net/wireless/realtek/rtw88/rtw8821cs.c new file mode 100644 index 000000000000..a359413369a4 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/rtw8821cs.c @@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) Martin Blumenstingl <[email protected]> + */ + +#include <linux/mmc/sdio_func.h> +#include <linux/mmc/sdio_ids.h> +#include <linux/module.h> +#include "main.h" +#include "rtw8821c.h" +#include "sdio.h" + +static const struct sdio_device_id rtw_8821cs_id_table[] = { + { + SDIO_DEVICE(SDIO_VENDOR_ID_REALTEK, + SDIO_DEVICE_ID_REALTEK_RTW8821CS), + .driver_data = (kernel_ulong_t)&rtw8821c_hw_spec, + }, + {} +}; +MODULE_DEVICE_TABLE(sdio, rtw_8821cs_id_table); + +static struct sdio_driver rtw_8821cs_driver = { + .name = "rtw_8821cs", + .probe = rtw_sdio_probe, + .remove = rtw_sdio_remove, + .id_table = rtw_8821cs_id_table, + .drv = { + .pm = &rtw_sdio_pm_ops, + .shutdown = rtw_sdio_shutdown, + } +}; +module_sdio_driver(rtw_8821cs_driver); + +MODULE_AUTHOR("Martin Blumenstingl <[email protected]>"); +MODULE_DESCRIPTION("Realtek 802.11ac wireless 8821cs driver"); +MODULE_LICENSE("Dual BSD/GPL"); diff --git a/drivers/net/wireless/realtek/rtw88/rtw8822b.c b/drivers/net/wireless/realtek/rtw88/rtw8822b.c index 531b67787e2e..3017a9760da8 100644 --- a/drivers/net/wireless/realtek/rtw88/rtw8822b.c +++ b/drivers/net/wireless/realtek/rtw88/rtw8822b.c @@ -2540,6 +2540,7 @@ const struct rtw_chip_info rtw8822b_hw_spec = { .txff_size = 262144, .rxff_size = 24576, .fw_rxff_size = 12288, + .rsvd_drv_pg_num = 8, .txgi_factor = 1, .is_pwr_by_rate_dec = true, .max_power_index = 0x3f, diff --git a/drivers/net/wireless/realtek/rtw88/rtw8822bs.c b/drivers/net/wireless/realtek/rtw88/rtw8822bs.c new file mode 100644 index 000000000000..31d8645f83bd --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/rtw8822bs.c @@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) Jernej Skrabec <[email protected]> + */ + +#include <linux/mmc/sdio_func.h> +#include <linux/mmc/sdio_ids.h> +#include <linux/module.h> +#include "main.h" +#include "rtw8822b.h" +#include "sdio.h" + +static const struct sdio_device_id rtw_8822bs_id_table[] = { + { + SDIO_DEVICE(SDIO_VENDOR_ID_REALTEK, + SDIO_DEVICE_ID_REALTEK_RTW8822BS), + .driver_data = (kernel_ulong_t)&rtw8822b_hw_spec, + }, + {} +}; +MODULE_DEVICE_TABLE(sdio, rtw_8822bs_id_table); + +static struct sdio_driver rtw_8822bs_driver = { + .name = "rtw_8822bs", + .probe = rtw_sdio_probe, + .remove = rtw_sdio_remove, + .id_table = rtw_8822bs_id_table, + .drv = { + .pm = &rtw_sdio_pm_ops, + .shutdown = rtw_sdio_shutdown, + } +}; +module_sdio_driver(rtw_8822bs_driver); + +MODULE_AUTHOR("Jernej Skrabec <[email protected]>"); +MODULE_DESCRIPTION("Realtek 802.11ac wireless 8822bs driver"); +MODULE_LICENSE("Dual BSD/GPL"); diff --git a/drivers/net/wireless/realtek/rtw88/rtw8822c.c b/drivers/net/wireless/realtek/rtw88/rtw8822c.c index 5a2c004b12df..cd965edc29ce 100644 --- a/drivers/net/wireless/realtek/rtw88/rtw8822c.c +++ b/drivers/net/wireless/realtek/rtw88/rtw8822c.c @@ -5358,6 +5358,7 @@ const struct rtw_chip_info rtw8822c_hw_spec = { .txff_size = 262144, .rxff_size = 24576, .fw_rxff_size = 12288, + .rsvd_drv_pg_num = 16, .txgi_factor = 2, .is_pwr_by_rate_dec = false, .max_power_index = 0x7f, diff --git a/drivers/net/wireless/realtek/rtw88/rtw8822cs.c b/drivers/net/wireless/realtek/rtw88/rtw8822cs.c new file mode 100644 index 000000000000..975e81c824f2 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/rtw8822cs.c @@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) Martin Blumenstingl <[email protected]> + */ + +#include <linux/mmc/sdio_func.h> +#include <linux/mmc/sdio_ids.h> +#include <linux/module.h> +#include "main.h" +#include "rtw8822c.h" +#include "sdio.h" + +static const struct sdio_device_id rtw_8822cs_id_table[] = { + { + SDIO_DEVICE(SDIO_VENDOR_ID_REALTEK, + SDIO_DEVICE_ID_REALTEK_RTW8822CS), + .driver_data = (kernel_ulong_t)&rtw8822c_hw_spec, + }, + {} +}; +MODULE_DEVICE_TABLE(sdio, rtw_8822cs_id_table); + +static struct sdio_driver rtw_8822cs_driver = { + .name = "rtw_8822cs", + .probe = rtw_sdio_probe, + .remove = rtw_sdio_remove, + .id_table = rtw_8822cs_id_table, + .drv = { + .pm = &rtw_sdio_pm_ops, + .shutdown = rtw_sdio_shutdown, + } +}; +module_sdio_driver(rtw_8822cs_driver); + +MODULE_AUTHOR("Martin Blumenstingl <[email protected]>"); +MODULE_DESCRIPTION("Realtek 802.11ac wireless 8822cs driver"); +MODULE_LICENSE("Dual BSD/GPL"); diff --git a/drivers/net/wireless/realtek/rtw88/sdio.c b/drivers/net/wireless/realtek/rtw88/sdio.c new file mode 100644 index 000000000000..af0459a79899 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/sdio.c @@ -0,0 +1,1394 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright (C) 2021 Martin Blumenstingl <[email protected]> + * Copyright (C) 2021 Jernej Skrabec <[email protected]> + * + * Based on rtw88/pci.c: + * Copyright(c) 2018-2019 Realtek Corporation + */ + +#include <linux/module.h> +#include <linux/mmc/host.h> +#include <linux/mmc/sdio_func.h> +#include "main.h" +#include "debug.h" +#include "fw.h" +#include "ps.h" +#include "reg.h" +#include "rx.h" +#include "sdio.h" +#include "tx.h" + +#define RTW_SDIO_INDIRECT_RW_RETRIES 50 + +static bool rtw_sdio_is_bus_addr(u32 addr) +{ + return !!(addr & RTW_SDIO_BUS_MSK); +} + +static bool rtw_sdio_bus_claim_needed(struct rtw_sdio *rtwsdio) +{ + return !rtwsdio->irq_thread || + rtwsdio->irq_thread != current; +} + +static u32 rtw_sdio_to_bus_offset(struct rtw_dev *rtwdev, u32 addr) +{ + switch (addr & RTW_SDIO_BUS_MSK) { + case WLAN_IOREG_OFFSET: + addr &= WLAN_IOREG_REG_MSK; + addr |= FIELD_PREP(REG_SDIO_CMD_ADDR_MSK, + REG_SDIO_CMD_ADDR_MAC_REG); + break; + case SDIO_LOCAL_OFFSET: + addr &= SDIO_LOCAL_REG_MSK; + addr |= FIELD_PREP(REG_SDIO_CMD_ADDR_MSK, + REG_SDIO_CMD_ADDR_SDIO_REG); + break; + default: + rtw_warn(rtwdev, "Cannot convert addr 0x%08x to bus offset", + addr); + } + + return addr; +} + +static bool rtw_sdio_use_memcpy_io(struct rtw_dev *rtwdev, u32 addr, + u8 alignment) +{ + return IS_ALIGNED(addr, alignment) && + test_bit(RTW_FLAG_POWERON, rtwdev->flags); +} + +static void rtw_sdio_writel(struct rtw_dev *rtwdev, u32 val, u32 addr, + int *err_ret) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + u8 buf[4]; + int i; + + if (rtw_sdio_use_memcpy_io(rtwdev, addr, 4)) { + sdio_writel(rtwsdio->sdio_func, val, addr, err_ret); + return; + } + + *(__le32 *)buf = cpu_to_le32(val); + + for (i = 0; i < 4; i++) { + sdio_writeb(rtwsdio->sdio_func, buf[i], addr + i, err_ret); + if (*err_ret) + return; + } +} + +static void rtw_sdio_writew(struct rtw_dev *rtwdev, u16 val, u32 addr, + int *err_ret) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + u8 buf[2]; + int i; + + if (rtw_sdio_use_memcpy_io(rtwdev, addr, 2)) { + sdio_writew(rtwsdio->sdio_func, val, addr, err_ret); + return; + } + + *(__le16 *)buf = cpu_to_le16(val); + + for (i = 0; i < 2; i++) { + sdio_writeb(rtwsdio->sdio_func, buf[i], addr + i, err_ret); + if (*err_ret) + return; + } +} + +static u32 rtw_sdio_readl(struct rtw_dev *rtwdev, u32 addr, int *err_ret) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + u8 buf[4]; + int i; + + if (rtw_sdio_use_memcpy_io(rtwdev, addr, 4)) + return sdio_readl(rtwsdio->sdio_func, addr, err_ret); + + for (i = 0; i < 4; i++) { + buf[i] = sdio_readb(rtwsdio->sdio_func, addr + i, err_ret); + if (*err_ret) + return 0; + } + + return le32_to_cpu(*(__le32 *)buf); +} + +static u16 rtw_sdio_readw(struct rtw_dev *rtwdev, u32 addr, int *err_ret) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + u8 buf[2]; + int i; + + if (rtw_sdio_use_memcpy_io(rtwdev, addr, 2)) + return sdio_readw(rtwsdio->sdio_func, addr, err_ret); + + for (i = 0; i < 2; i++) { + buf[i] = sdio_readb(rtwsdio->sdio_func, addr + i, err_ret); + if (*err_ret) + return 0; + } + + return le16_to_cpu(*(__le16 *)buf); +} + +static u32 rtw_sdio_to_io_address(struct rtw_dev *rtwdev, u32 addr, + bool direct) +{ + if (!direct) + return addr; + + if (!rtw_sdio_is_bus_addr(addr)) + addr |= WLAN_IOREG_OFFSET; + + return rtw_sdio_to_bus_offset(rtwdev, addr); +} + +static bool rtw_sdio_use_direct_io(struct rtw_dev *rtwdev, u32 addr) +{ + return !rtw_sdio_is_sdio30_supported(rtwdev) || + rtw_sdio_is_bus_addr(addr); +} + +static int rtw_sdio_indirect_reg_cfg(struct rtw_dev *rtwdev, u32 addr, u32 cfg) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + unsigned int retry; + u32 reg_cfg; + int ret; + u8 tmp; + + reg_cfg = rtw_sdio_to_bus_offset(rtwdev, REG_SDIO_INDIRECT_REG_CFG); + + rtw_sdio_writel(rtwdev, addr | cfg | BIT_SDIO_INDIRECT_REG_CFG_UNK20, + reg_cfg, &ret); + if (ret) + return ret; + + for (retry = 0; retry < RTW_SDIO_INDIRECT_RW_RETRIES; retry++) { + tmp = sdio_readb(rtwsdio->sdio_func, reg_cfg + 2, &ret); + if (!ret && (tmp & BIT(4))) + return 0; + } + + return -ETIMEDOUT; +} + +static u8 rtw_sdio_indirect_read8(struct rtw_dev *rtwdev, u32 addr, + int *err_ret) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + u32 reg_data; + + *err_ret = rtw_sdio_indirect_reg_cfg(rtwdev, addr, + BIT_SDIO_INDIRECT_REG_CFG_READ); + if (*err_ret) + return 0; + + reg_data = rtw_sdio_to_bus_offset(rtwdev, REG_SDIO_INDIRECT_REG_DATA); + return sdio_readb(rtwsdio->sdio_func, reg_data, err_ret); +} + +static int rtw_sdio_indirect_read_bytes(struct rtw_dev *rtwdev, u32 addr, + u8 *buf, int count) +{ + int i, ret = 0; + + for (i = 0; i < count; i++) { + buf[i] = rtw_sdio_indirect_read8(rtwdev, addr + i, &ret); + if (ret) + break; + } + + return ret; +} + +static u16 rtw_sdio_indirect_read16(struct rtw_dev *rtwdev, u32 addr, + int *err_ret) +{ + u32 reg_data; + u8 buf[2]; + + if (!IS_ALIGNED(addr, 2)) { + *err_ret = rtw_sdio_indirect_read_bytes(rtwdev, addr, buf, 2); + if (*err_ret) + return 0; + + return le16_to_cpu(*(__le16 *)buf); + } + + *err_ret = rtw_sdio_indirect_reg_cfg(rtwdev, addr, + BIT_SDIO_INDIRECT_REG_CFG_READ); + if (*err_ret) + return 0; + + reg_data = rtw_sdio_to_bus_offset(rtwdev, REG_SDIO_INDIRECT_REG_DATA); + return rtw_sdio_readw(rtwdev, reg_data, err_ret); +} + +static u32 rtw_sdio_indirect_read32(struct rtw_dev *rtwdev, u32 addr, + int *err_ret) +{ + u32 reg_data; + u8 buf[4]; + + if (!IS_ALIGNED(addr, 4)) { + *err_ret = rtw_sdio_indirect_read_bytes(rtwdev, addr, buf, 4); + if (*err_ret) + return 0; + + return le32_to_cpu(*(__le32 *)buf); + } + + *err_ret = rtw_sdio_indirect_reg_cfg(rtwdev, addr, + BIT_SDIO_INDIRECT_REG_CFG_READ); + if (*err_ret) + return 0; + + reg_data = rtw_sdio_to_bus_offset(rtwdev, REG_SDIO_INDIRECT_REG_DATA); + return rtw_sdio_readl(rtwdev, reg_data, err_ret); +} + +static u8 rtw_sdio_read8(struct rtw_dev *rtwdev, u32 addr) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + bool direct, bus_claim; + int ret; + u8 val; + + direct = rtw_sdio_use_direct_io(rtwdev, addr); + addr = rtw_sdio_to_io_address(rtwdev, addr, direct); + bus_claim = rtw_sdio_bus_claim_needed(rtwsdio); + + if (bus_claim) + sdio_claim_host(rtwsdio->sdio_func); + + if (direct) + val = sdio_readb(rtwsdio->sdio_func, addr, &ret); + else + val = rtw_sdio_indirect_read8(rtwdev, addr, &ret); + + if (bus_claim) + sdio_release_host(rtwsdio->sdio_func); + + if (ret) + rtw_warn(rtwdev, "sdio read8 failed (0x%x): %d", addr, ret); + + return val; +} + +static u16 rtw_sdio_read16(struct rtw_dev *rtwdev, u32 addr) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + bool direct, bus_claim; + int ret; + u16 val; + + direct = rtw_sdio_use_direct_io(rtwdev, addr); + addr = rtw_sdio_to_io_address(rtwdev, addr, direct); + bus_claim = rtw_sdio_bus_claim_needed(rtwsdio); + + if (bus_claim) + sdio_claim_host(rtwsdio->sdio_func); + + if (direct) + val = rtw_sdio_readw(rtwdev, addr, &ret); + else + val = rtw_sdio_indirect_read16(rtwdev, addr, &ret); + + if (bus_claim) + sdio_release_host(rtwsdio->sdio_func); + + if (ret) + rtw_warn(rtwdev, "sdio read16 failed (0x%x): %d", addr, ret); + + return val; +} + +static u32 rtw_sdio_read32(struct rtw_dev *rtwdev, u32 addr) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + bool direct, bus_claim; + u32 val; + int ret; + + direct = rtw_sdio_use_direct_io(rtwdev, addr); + addr = rtw_sdio_to_io_address(rtwdev, addr, direct); + bus_claim = rtw_sdio_bus_claim_needed(rtwsdio); + + if (bus_claim) + sdio_claim_host(rtwsdio->sdio_func); + + if (direct) + val = rtw_sdio_readl(rtwdev, addr, &ret); + else + val = rtw_sdio_indirect_read32(rtwdev, addr, &ret); + + if (bus_claim) + sdio_release_host(rtwsdio->sdio_func); + + if (ret) + rtw_warn(rtwdev, "sdio read32 failed (0x%x): %d", addr, ret); + + return val; +} + +static void rtw_sdio_indirect_write8(struct rtw_dev *rtwdev, u8 val, u32 addr, + int *err_ret) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + u32 reg_data; + + reg_data = rtw_sdio_to_bus_offset(rtwdev, REG_SDIO_INDIRECT_REG_DATA); + sdio_writeb(rtwsdio->sdio_func, val, reg_data, err_ret); + if (*err_ret) + return; + + *err_ret = rtw_sdio_indirect_reg_cfg(rtwdev, addr, + BIT_SDIO_INDIRECT_REG_CFG_WRITE); +} + +static void rtw_sdio_indirect_write16(struct rtw_dev *rtwdev, u16 val, u32 addr, + int *err_ret) +{ + u32 reg_data; + + if (!IS_ALIGNED(addr, 2)) { + addr = rtw_sdio_to_io_address(rtwdev, addr, true); + rtw_sdio_writew(rtwdev, val, addr, err_ret); + return; + } + + reg_data = rtw_sdio_to_bus_offset(rtwdev, REG_SDIO_INDIRECT_REG_DATA); + rtw_sdio_writew(rtwdev, val, reg_data, err_ret); + if (*err_ret) + return; + + *err_ret = rtw_sdio_indirect_reg_cfg(rtwdev, addr, + BIT_SDIO_INDIRECT_REG_CFG_WRITE | + BIT_SDIO_INDIRECT_REG_CFG_WORD); +} + +static void rtw_sdio_indirect_write32(struct rtw_dev *rtwdev, u32 val, + u32 addr, int *err_ret) +{ + u32 reg_data; + + if (!IS_ALIGNED(addr, 4)) { + addr = rtw_sdio_to_io_address(rtwdev, addr, true); + rtw_sdio_writel(rtwdev, val, addr, err_ret); + return; + } + + reg_data = rtw_sdio_to_bus_offset(rtwdev, REG_SDIO_INDIRECT_REG_DATA); + rtw_sdio_writel(rtwdev, val, reg_data, err_ret); + + *err_ret = rtw_sdio_indirect_reg_cfg(rtwdev, addr, + BIT_SDIO_INDIRECT_REG_CFG_WRITE | + BIT_SDIO_INDIRECT_REG_CFG_DWORD); +} + +static void rtw_sdio_write8(struct rtw_dev *rtwdev, u32 addr, u8 val) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + bool direct, bus_claim; + int ret; + + direct = rtw_sdio_use_direct_io(rtwdev, addr); + addr = rtw_sdio_to_io_address(rtwdev, addr, direct); + bus_claim = rtw_sdio_bus_claim_needed(rtwsdio); + + if (bus_claim) + sdio_claim_host(rtwsdio->sdio_func); + + if (direct) + sdio_writeb(rtwsdio->sdio_func, val, addr, &ret); + else + rtw_sdio_indirect_write8(rtwdev, val, addr, &ret); + + if (bus_claim) + sdio_release_host(rtwsdio->sdio_func); + + if (ret) + rtw_warn(rtwdev, "sdio write8 failed (0x%x): %d", addr, ret); +} + +static void rtw_sdio_write16(struct rtw_dev *rtwdev, u32 addr, u16 val) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + bool direct, bus_claim; + int ret; + + direct = rtw_sdio_use_direct_io(rtwdev, addr); + addr = rtw_sdio_to_io_address(rtwdev, addr, direct); + bus_claim = rtw_sdio_bus_claim_needed(rtwsdio); + + if (bus_claim) + sdio_claim_host(rtwsdio->sdio_func); + + if (direct) + rtw_sdio_writew(rtwdev, val, addr, &ret); + else + rtw_sdio_indirect_write16(rtwdev, val, addr, &ret); + + if (bus_claim) + sdio_release_host(rtwsdio->sdio_func); + + if (ret) + rtw_warn(rtwdev, "sdio write16 failed (0x%x): %d", addr, ret); +} + +static void rtw_sdio_write32(struct rtw_dev *rtwdev, u32 addr, u32 val) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + bool direct, bus_claim; + int ret; + + direct = rtw_sdio_use_direct_io(rtwdev, addr); + addr = rtw_sdio_to_io_address(rtwdev, addr, direct); + bus_claim = rtw_sdio_bus_claim_needed(rtwsdio); + + if (bus_claim) + sdio_claim_host(rtwsdio->sdio_func); + + if (direct) + rtw_sdio_writel(rtwdev, val, addr, &ret); + else + rtw_sdio_indirect_write32(rtwdev, val, addr, &ret); + + if (bus_claim) + sdio_release_host(rtwsdio->sdio_func); + + if (ret) + rtw_warn(rtwdev, "sdio write32 failed (0x%x): %d", addr, ret); +} + +static u32 rtw_sdio_get_tx_addr(struct rtw_dev *rtwdev, size_t size, + enum rtw_tx_queue_type queue) +{ + u32 txaddr; + + switch (queue) { + case RTW_TX_QUEUE_BCN: + case RTW_TX_QUEUE_H2C: + case RTW_TX_QUEUE_HI0: + txaddr = FIELD_PREP(REG_SDIO_CMD_ADDR_MSK, + REG_SDIO_CMD_ADDR_TXFF_HIGH); + break; + case RTW_TX_QUEUE_VI: + case RTW_TX_QUEUE_VO: + txaddr = FIELD_PREP(REG_SDIO_CMD_ADDR_MSK, + REG_SDIO_CMD_ADDR_TXFF_NORMAL); + break; + case RTW_TX_QUEUE_BE: + case RTW_TX_QUEUE_BK: + txaddr = FIELD_PREP(REG_SDIO_CMD_ADDR_MSK, + REG_SDIO_CMD_ADDR_TXFF_LOW); + break; + case RTW_TX_QUEUE_MGMT: + txaddr = FIELD_PREP(REG_SDIO_CMD_ADDR_MSK, + REG_SDIO_CMD_ADDR_TXFF_EXTRA); + break; + default: + rtw_warn(rtwdev, "Unsupported queue for TX addr: 0x%02x\n", + queue); + return 0; + } + + txaddr += DIV_ROUND_UP(size, 4); + + return txaddr; +}; + +static int rtw_sdio_read_port(struct rtw_dev *rtwdev, u8 *buf, size_t count) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + bool bus_claim = rtw_sdio_bus_claim_needed(rtwsdio); + u32 rxaddr = rtwsdio->rx_addr++; + int ret; + + if (bus_claim) + sdio_claim_host(rtwsdio->sdio_func); + + ret = sdio_memcpy_fromio(rtwsdio->sdio_func, buf, + RTW_SDIO_ADDR_RX_RX0FF_GEN(rxaddr), count); + if (ret) + rtw_warn(rtwdev, + "Failed to read %zu byte(s) from SDIO port 0x%08x", + count, rxaddr); + + if (bus_claim) + sdio_release_host(rtwsdio->sdio_func); + + return ret; +} + +static int rtw_sdio_check_free_txpg(struct rtw_dev *rtwdev, u8 queue, + size_t count) +{ + unsigned int pages_free, pages_needed; + + if (rtw_chip_wcpu_11n(rtwdev)) { + u32 free_txpg; + + free_txpg = rtw_sdio_read32(rtwdev, REG_SDIO_FREE_TXPG); + + switch (queue) { + case RTW_TX_QUEUE_BCN: + case RTW_TX_QUEUE_H2C: + case RTW_TX_QUEUE_HI0: + case RTW_TX_QUEUE_MGMT: + /* high */ + pages_free = free_txpg & 0xff; + break; + case RTW_TX_QUEUE_VI: + case RTW_TX_QUEUE_VO: + /* normal */ + pages_free = (free_txpg >> 8) & 0xff; + break; + case RTW_TX_QUEUE_BE: + case RTW_TX_QUEUE_BK: + /* low */ + pages_free = (free_txpg >> 16) & 0xff; + break; + default: + rtw_warn(rtwdev, "Unknown mapping for queue %u\n", queue); + return -EINVAL; + } + + /* add the pages from the public queue */ + pages_free += (free_txpg >> 24) & 0xff; + } else { + u32 free_txpg[3]; + + free_txpg[0] = rtw_sdio_read32(rtwdev, REG_SDIO_FREE_TXPG); + free_txpg[1] = rtw_sdio_read32(rtwdev, REG_SDIO_FREE_TXPG + 4); + free_txpg[2] = rtw_sdio_read32(rtwdev, REG_SDIO_FREE_TXPG + 8); + + switch (queue) { + case RTW_TX_QUEUE_BCN: + case RTW_TX_QUEUE_H2C: + case RTW_TX_QUEUE_HI0: + /* high */ + pages_free = free_txpg[0] & 0xfff; + break; + case RTW_TX_QUEUE_VI: + case RTW_TX_QUEUE_VO: + /* normal */ + pages_free = (free_txpg[0] >> 16) & 0xfff; + break; + case RTW_TX_QUEUE_BE: + case RTW_TX_QUEUE_BK: + /* low */ + pages_free = free_txpg[1] & 0xfff; + break; + case RTW_TX_QUEUE_MGMT: + /* extra */ + pages_free = free_txpg[2] & 0xfff; + break; + default: + rtw_warn(rtwdev, "Unknown mapping for queue %u\n", queue); + return -EINVAL; + } + + /* add the pages from the public queue */ + pages_free += (free_txpg[1] >> 16) & 0xfff; + } + + pages_needed = DIV_ROUND_UP(count, rtwdev->chip->page_size); + + if (pages_needed > pages_free) { + rtw_dbg(rtwdev, RTW_DBG_SDIO, + "Not enough free pages (%u needed, %u free) in queue %u for %zu bytes\n", + pages_needed, pages_free, queue, count); + return -EBUSY; + } + + return 0; +} + +static int rtw_sdio_write_port(struct rtw_dev *rtwdev, struct sk_buff *skb, + enum rtw_tx_queue_type queue) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + bool bus_claim; + size_t txsize; + u32 txaddr; + int ret; + + txaddr = rtw_sdio_get_tx_addr(rtwdev, skb->len, queue); + if (!txaddr) + return -EINVAL; + + txsize = sdio_align_size(rtwsdio->sdio_func, skb->len); + + ret = rtw_sdio_check_free_txpg(rtwdev, queue, txsize); + if (ret) + return ret; + + if (!IS_ALIGNED((unsigned long)skb->data, RTW_SDIO_DATA_PTR_ALIGN)) + rtw_warn(rtwdev, "Got unaligned SKB in %s() for queue %u\n", + __func__, queue); + + bus_claim = rtw_sdio_bus_claim_needed(rtwsdio); + + if (bus_claim) + sdio_claim_host(rtwsdio->sdio_func); + + ret = sdio_memcpy_toio(rtwsdio->sdio_func, txaddr, skb->data, txsize); + + if (bus_claim) + sdio_release_host(rtwsdio->sdio_func); + + if (ret) + rtw_warn(rtwdev, + "Failed to write %zu byte(s) to SDIO port 0x%08x", + txsize, txaddr); + + return ret; +} + +static void rtw_sdio_init(struct rtw_dev *rtwdev) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + + rtwsdio->irq_mask = REG_SDIO_HIMR_RX_REQUEST | REG_SDIO_HIMR_CPWM1; +} + +static void rtw_sdio_enable_rx_aggregation(struct rtw_dev *rtwdev) +{ + u8 size, timeout; + + if (rtw_chip_wcpu_11n(rtwdev)) { + size = 0x6; + timeout = 0x6; + } else { + size = 0xff; + timeout = 0x1; + } + + /* Make the firmware honor the size limit configured below */ + rtw_write32_set(rtwdev, REG_RXDMA_AGG_PG_TH, BIT_EN_PRE_CALC); + + rtw_write8_set(rtwdev, REG_TXDMA_PQ_MAP, BIT_RXDMA_AGG_EN); + + rtw_write16(rtwdev, REG_RXDMA_AGG_PG_TH, + FIELD_PREP(BIT_RXDMA_AGG_PG_TH, size) | + FIELD_PREP(BIT_DMA_AGG_TO_V1, timeout)); + + rtw_write8_set(rtwdev, REG_RXDMA_MODE, BIT_DMA_MODE); +} + +static void rtw_sdio_enable_interrupt(struct rtw_dev *rtwdev) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + + rtw_write32(rtwdev, REG_SDIO_HIMR, rtwsdio->irq_mask); +} + +static void rtw_sdio_disable_interrupt(struct rtw_dev *rtwdev) +{ + rtw_write32(rtwdev, REG_SDIO_HIMR, 0x0); +} + +static u8 rtw_sdio_get_tx_qsel(struct rtw_dev *rtwdev, struct sk_buff *skb, + u8 queue) +{ + switch (queue) { + case RTW_TX_QUEUE_BCN: + return TX_DESC_QSEL_BEACON; + case RTW_TX_QUEUE_H2C: + return TX_DESC_QSEL_H2C; + case RTW_TX_QUEUE_MGMT: + if (rtw_chip_wcpu_11n(rtwdev)) + return TX_DESC_QSEL_HIGH; + else + return TX_DESC_QSEL_MGMT; + case RTW_TX_QUEUE_HI0: + return TX_DESC_QSEL_HIGH; + default: + return skb->priority; + } +} + +static int rtw_sdio_setup(struct rtw_dev *rtwdev) +{ + /* nothing to do */ + return 0; +} + +static int rtw_sdio_start(struct rtw_dev *rtwdev) +{ + rtw_sdio_enable_rx_aggregation(rtwdev); + rtw_sdio_enable_interrupt(rtwdev); + + return 0; +} + +static void rtw_sdio_stop(struct rtw_dev *rtwdev) +{ + rtw_sdio_disable_interrupt(rtwdev); +} + +static void rtw_sdio_deep_ps_enter(struct rtw_dev *rtwdev) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + bool tx_empty = true; + u8 queue; + + if (!rtw_fw_feature_check(&rtwdev->fw, FW_FEATURE_TX_WAKE)) { + /* Deep PS state is not allowed to TX-DMA */ + for (queue = 0; queue < RTK_MAX_TX_QUEUE_NUM; queue++) { + /* BCN queue is rsvd page, does not have DMA interrupt + * H2C queue is managed by firmware + */ + if (queue == RTW_TX_QUEUE_BCN || + queue == RTW_TX_QUEUE_H2C) + continue; + + /* check if there is any skb DMAing */ + if (skb_queue_len(&rtwsdio->tx_queue[queue])) { + tx_empty = false; + break; + } + } + } + + if (!tx_empty) { + rtw_dbg(rtwdev, RTW_DBG_PS, + "TX path not empty, cannot enter deep power save state\n"); + return; + } + + set_bit(RTW_FLAG_LEISURE_PS_DEEP, rtwdev->flags); + rtw_power_mode_change(rtwdev, true); +} + +static void rtw_sdio_deep_ps_leave(struct rtw_dev *rtwdev) +{ + if (test_and_clear_bit(RTW_FLAG_LEISURE_PS_DEEP, rtwdev->flags)) + rtw_power_mode_change(rtwdev, false); +} + +static void rtw_sdio_deep_ps(struct rtw_dev *rtwdev, bool enter) +{ + if (enter && !test_bit(RTW_FLAG_LEISURE_PS_DEEP, rtwdev->flags)) + rtw_sdio_deep_ps_enter(rtwdev); + + if (!enter && test_bit(RTW_FLAG_LEISURE_PS_DEEP, rtwdev->flags)) + rtw_sdio_deep_ps_leave(rtwdev); +} + +static void rtw_sdio_tx_kick_off(struct rtw_dev *rtwdev) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + + queue_work(rtwsdio->txwq, &rtwsdio->tx_handler_data->work); +} + +static void rtw_sdio_link_ps(struct rtw_dev *rtwdev, bool enter) +{ + /* nothing to do */ +} + +static void rtw_sdio_interface_cfg(struct rtw_dev *rtwdev) +{ + u32 val; + + rtw_read32(rtwdev, REG_SDIO_FREE_TXPG); + + val = rtw_read32(rtwdev, REG_SDIO_TX_CTRL); + val &= 0xfff8; + rtw_write32(rtwdev, REG_SDIO_TX_CTRL, val); +} + +static struct rtw_sdio_tx_data *rtw_sdio_get_tx_data(struct sk_buff *skb) +{ + struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb); + + BUILD_BUG_ON(sizeof(struct rtw_sdio_tx_data) > + sizeof(info->status.status_driver_data)); + + return (struct rtw_sdio_tx_data *)info->status.status_driver_data; +} + +static void rtw_sdio_tx_skb_prepare(struct rtw_dev *rtwdev, + struct rtw_tx_pkt_info *pkt_info, + struct sk_buff *skb, + enum rtw_tx_queue_type queue) +{ + const struct rtw_chip_info *chip = rtwdev->chip; + unsigned long data_addr, aligned_addr; + size_t offset; + u8 *pkt_desc; + + pkt_desc = skb_push(skb, chip->tx_pkt_desc_sz); + + data_addr = (unsigned long)pkt_desc; + aligned_addr = ALIGN(data_addr, RTW_SDIO_DATA_PTR_ALIGN); + + if (data_addr != aligned_addr) { + /* Ensure that the start of the pkt_desc is always aligned at + * RTW_SDIO_DATA_PTR_ALIGN. + */ + offset = RTW_SDIO_DATA_PTR_ALIGN - (aligned_addr - data_addr); + + pkt_desc = skb_push(skb, offset); + + /* By inserting padding to align the start of the pkt_desc we + * need to inform the firmware that the actual data starts at + * a different offset than normal. + */ + pkt_info->offset += offset; + } + + memset(pkt_desc, 0, chip->tx_pkt_desc_sz); + + pkt_info->qsel = rtw_sdio_get_tx_qsel(rtwdev, skb, queue); + + rtw_tx_fill_tx_desc(pkt_info, skb); + rtw_tx_fill_txdesc_checksum(rtwdev, pkt_info, pkt_desc); +} + +static int rtw_sdio_write_data(struct rtw_dev *rtwdev, + struct rtw_tx_pkt_info *pkt_info, + struct sk_buff *skb, + enum rtw_tx_queue_type queue) +{ + int ret; + + rtw_sdio_tx_skb_prepare(rtwdev, pkt_info, skb, queue); + + ret = rtw_sdio_write_port(rtwdev, skb, queue); + dev_kfree_skb_any(skb); + + return ret; +} + +static int rtw_sdio_write_data_rsvd_page(struct rtw_dev *rtwdev, u8 *buf, + u32 size) +{ + struct rtw_tx_pkt_info pkt_info = {}; + struct sk_buff *skb; + + skb = rtw_tx_write_data_rsvd_page_get(rtwdev, &pkt_info, buf, size); + if (!skb) + return -ENOMEM; + + return rtw_sdio_write_data(rtwdev, &pkt_info, skb, RTW_TX_QUEUE_BCN); +} + +static int rtw_sdio_write_data_h2c(struct rtw_dev *rtwdev, u8 *buf, u32 size) +{ + struct rtw_tx_pkt_info pkt_info = {}; + struct sk_buff *skb; + + skb = rtw_tx_write_data_h2c_get(rtwdev, &pkt_info, buf, size); + if (!skb) + return -ENOMEM; + + return rtw_sdio_write_data(rtwdev, &pkt_info, skb, RTW_TX_QUEUE_H2C); +} + +static int rtw_sdio_tx_write(struct rtw_dev *rtwdev, + struct rtw_tx_pkt_info *pkt_info, + struct sk_buff *skb) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + enum rtw_tx_queue_type queue = rtw_tx_queue_mapping(skb); + struct rtw_sdio_tx_data *tx_data; + + rtw_sdio_tx_skb_prepare(rtwdev, pkt_info, skb, queue); + + tx_data = rtw_sdio_get_tx_data(skb); + tx_data->sn = pkt_info->sn; + + skb_queue_tail(&rtwsdio->tx_queue[queue], skb); + + return 0; +} + +static void rtw_sdio_tx_err_isr(struct rtw_dev *rtwdev) +{ + u32 val = rtw_read32(rtwdev, REG_TXDMA_STATUS); + + rtw_write32(rtwdev, REG_TXDMA_STATUS, val); +} + +static void rtw_sdio_rx_skb(struct rtw_dev *rtwdev, struct sk_buff *skb, + u32 pkt_offset, struct rtw_rx_pkt_stat *pkt_stat, + struct ieee80211_rx_status *rx_status) +{ + *IEEE80211_SKB_RXCB(skb) = *rx_status; + + if (pkt_stat->is_c2h) { + skb_put(skb, pkt_stat->pkt_len + pkt_offset); + rtw_fw_c2h_cmd_rx_irqsafe(rtwdev, pkt_offset, skb); + return; + } + + skb_put(skb, pkt_stat->pkt_len); + skb_reserve(skb, pkt_offset); + + rtw_rx_stats(rtwdev, pkt_stat->vif, skb); + + ieee80211_rx_irqsafe(rtwdev->hw, skb); +} + +static void rtw_sdio_rxfifo_recv(struct rtw_dev *rtwdev, u32 rx_len) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + const struct rtw_chip_info *chip = rtwdev->chip; + u32 pkt_desc_sz = chip->rx_pkt_desc_sz; + struct ieee80211_rx_status rx_status; + struct rtw_rx_pkt_stat pkt_stat; + struct sk_buff *skb, *split_skb; + u32 pkt_offset, curr_pkt_len; + size_t bufsz; + u8 *rx_desc; + int ret; + + bufsz = sdio_align_size(rtwsdio->sdio_func, rx_len); + + skb = dev_alloc_skb(bufsz); + if (!skb) + return; + + ret = rtw_sdio_read_port(rtwdev, skb->data, bufsz); + if (ret) { + dev_kfree_skb_any(skb); + return; + } + + while (true) { + rx_desc = skb->data; + chip->ops->query_rx_desc(rtwdev, rx_desc, &pkt_stat, + &rx_status); + pkt_offset = pkt_desc_sz + pkt_stat.drv_info_sz + + pkt_stat.shift; + + curr_pkt_len = ALIGN(pkt_offset + pkt_stat.pkt_len, + RTW_SDIO_DATA_PTR_ALIGN); + + if ((curr_pkt_len + pkt_desc_sz) >= rx_len) { + /* Use the original skb (with it's adjusted offset) + * when processing the last (or even the only) entry to + * have it's memory freed automatically. + */ + rtw_sdio_rx_skb(rtwdev, skb, pkt_offset, &pkt_stat, + &rx_status); + break; + } + + split_skb = dev_alloc_skb(curr_pkt_len); + if (!split_skb) { + rtw_sdio_rx_skb(rtwdev, skb, pkt_offset, &pkt_stat, + &rx_status); + break; + } + + skb_copy_header(split_skb, skb); + memcpy(split_skb->data, skb->data, curr_pkt_len); + + rtw_sdio_rx_skb(rtwdev, split_skb, pkt_offset, &pkt_stat, + &rx_status); + + /* Move to the start of the next RX descriptor */ + skb_reserve(skb, curr_pkt_len); + rx_len -= curr_pkt_len; + } +} + +static void rtw_sdio_rx_isr(struct rtw_dev *rtwdev) +{ + u32 rx_len, total_rx_bytes = 0; + + while (total_rx_bytes < SZ_64K) { + if (rtw_chip_wcpu_11n(rtwdev)) + rx_len = rtw_read16(rtwdev, REG_SDIO_RX0_REQ_LEN); + else + rx_len = rtw_read32(rtwdev, REG_SDIO_RX0_REQ_LEN); + + if (!rx_len) + break; + + rtw_sdio_rxfifo_recv(rtwdev, rx_len); + + total_rx_bytes += rx_len; + } +} + +static void rtw_sdio_handle_interrupt(struct sdio_func *sdio_func) +{ + struct ieee80211_hw *hw = sdio_get_drvdata(sdio_func); + struct rtw_sdio *rtwsdio; + struct rtw_dev *rtwdev; + u32 hisr; + + rtwdev = hw->priv; + rtwsdio = (struct rtw_sdio *)rtwdev->priv; + + rtwsdio->irq_thread = current; + + hisr = rtw_read32(rtwdev, REG_SDIO_HISR); + + if (hisr & REG_SDIO_HISR_TXERR) + rtw_sdio_tx_err_isr(rtwdev); + if (hisr & REG_SDIO_HISR_RX_REQUEST) { + hisr &= ~REG_SDIO_HISR_RX_REQUEST; + rtw_sdio_rx_isr(rtwdev); + } + + rtw_write32(rtwdev, REG_SDIO_HISR, hisr); + + rtwsdio->irq_thread = NULL; +} + +static int __maybe_unused rtw_sdio_suspend(struct device *dev) +{ + struct sdio_func *func = dev_to_sdio_func(dev); + struct ieee80211_hw *hw = dev_get_drvdata(dev); + struct rtw_dev *rtwdev = hw->priv; + int ret; + + ret = sdio_set_host_pm_flags(func, MMC_PM_KEEP_POWER); + if (ret) + rtw_err(rtwdev, "Failed to host PM flag MMC_PM_KEEP_POWER"); + + return ret; +} + +static int __maybe_unused rtw_sdio_resume(struct device *dev) +{ + return 0; +} + +SIMPLE_DEV_PM_OPS(rtw_sdio_pm_ops, rtw_sdio_suspend, rtw_sdio_resume); +EXPORT_SYMBOL(rtw_sdio_pm_ops); + +static int rtw_sdio_claim(struct rtw_dev *rtwdev, struct sdio_func *sdio_func) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + int ret; + + sdio_claim_host(sdio_func); + + ret = sdio_enable_func(sdio_func); + if (ret) { + rtw_err(rtwdev, "Failed to enable SDIO func"); + goto err_release_host; + } + + ret = sdio_set_block_size(sdio_func, RTW_SDIO_BLOCK_SIZE); + if (ret) { + rtw_err(rtwdev, "Failed to set SDIO block size to 512"); + goto err_disable_func; + } + + rtwsdio->sdio_func = sdio_func; + + rtwsdio->sdio3_bus_mode = mmc_card_uhs(sdio_func->card); + + sdio_set_drvdata(sdio_func, rtwdev->hw); + SET_IEEE80211_DEV(rtwdev->hw, &sdio_func->dev); + + sdio_release_host(sdio_func); + + return 0; + +err_disable_func: + sdio_disable_func(sdio_func); +err_release_host: + sdio_release_host(sdio_func); + return ret; +} + +static void rtw_sdio_declaim(struct rtw_dev *rtwdev, + struct sdio_func *sdio_func) +{ + sdio_claim_host(sdio_func); + sdio_disable_func(sdio_func); + sdio_release_host(sdio_func); +} + +static struct rtw_hci_ops rtw_sdio_ops = { + .tx_write = rtw_sdio_tx_write, + .tx_kick_off = rtw_sdio_tx_kick_off, + .setup = rtw_sdio_setup, + .start = rtw_sdio_start, + .stop = rtw_sdio_stop, + .deep_ps = rtw_sdio_deep_ps, + .link_ps = rtw_sdio_link_ps, + .interface_cfg = rtw_sdio_interface_cfg, + + .read8 = rtw_sdio_read8, + .read16 = rtw_sdio_read16, + .read32 = rtw_sdio_read32, + .write8 = rtw_sdio_write8, + .write16 = rtw_sdio_write16, + .write32 = rtw_sdio_write32, + .write_data_rsvd_page = rtw_sdio_write_data_rsvd_page, + .write_data_h2c = rtw_sdio_write_data_h2c, +}; + +static int rtw_sdio_request_irq(struct rtw_dev *rtwdev, + struct sdio_func *sdio_func) +{ + int ret; + + sdio_claim_host(sdio_func); + ret = sdio_claim_irq(sdio_func, &rtw_sdio_handle_interrupt); + sdio_release_host(sdio_func); + + if (ret) { + rtw_err(rtwdev, "failed to claim SDIO IRQ"); + return ret; + } + + return 0; +} + +static void rtw_sdio_indicate_tx_status(struct rtw_dev *rtwdev, + struct sk_buff *skb) +{ + struct rtw_sdio_tx_data *tx_data = rtw_sdio_get_tx_data(skb); + struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb); + struct ieee80211_hw *hw = rtwdev->hw; + + /* enqueue to wait for tx report */ + if (info->flags & IEEE80211_TX_CTL_REQ_TX_STATUS) { + rtw_tx_report_enqueue(rtwdev, skb, tx_data->sn); + return; + } + + /* always ACK for others, then they won't be marked as drop */ + ieee80211_tx_info_clear_status(info); + if (info->flags & IEEE80211_TX_CTL_NO_ACK) + info->flags |= IEEE80211_TX_STAT_NOACK_TRANSMITTED; + else + info->flags |= IEEE80211_TX_STAT_ACK; + + ieee80211_tx_status_irqsafe(hw, skb); +} + +static void rtw_sdio_process_tx_queue(struct rtw_dev *rtwdev, + enum rtw_tx_queue_type queue) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + struct sk_buff *skb; + int ret; + + skb = skb_dequeue(&rtwsdio->tx_queue[queue]); + if (!skb) + return; + + ret = rtw_sdio_write_port(rtwdev, skb, queue); + if (ret) { + skb_queue_head(&rtwsdio->tx_queue[queue], skb); + return; + } + + if (queue <= RTW_TX_QUEUE_VO) + rtw_sdio_indicate_tx_status(rtwdev, skb); + else + dev_kfree_skb_any(skb); +} + +static void rtw_sdio_tx_handler(struct work_struct *work) +{ + struct rtw_sdio_work_data *work_data = + container_of(work, struct rtw_sdio_work_data, work); + struct rtw_sdio *rtwsdio; + struct rtw_dev *rtwdev; + int limit, queue; + + rtwdev = work_data->rtwdev; + rtwsdio = (struct rtw_sdio *)rtwdev->priv; + + if (!rtw_fw_feature_check(&rtwdev->fw, FW_FEATURE_TX_WAKE)) + rtw_sdio_deep_ps_leave(rtwdev); + + for (queue = RTK_MAX_TX_QUEUE_NUM - 1; queue >= 0; queue--) { + for (limit = 0; limit < 1000; limit++) { + rtw_sdio_process_tx_queue(rtwdev, queue); + + if (skb_queue_empty(&rtwsdio->tx_queue[queue])) + break; + } + } +} + +static void rtw_sdio_free_irq(struct rtw_dev *rtwdev, + struct sdio_func *sdio_func) +{ + sdio_claim_host(sdio_func); + sdio_release_irq(sdio_func); + sdio_release_host(sdio_func); +} + +static int rtw_sdio_init_tx(struct rtw_dev *rtwdev) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + int i; + + rtwsdio->txwq = create_singlethread_workqueue("rtw88_sdio: tx wq"); + if (!rtwsdio->txwq) { + rtw_err(rtwdev, "failed to create TX work queue\n"); + return -ENOMEM; + } + + for (i = 0; i < RTK_MAX_TX_QUEUE_NUM; i++) + skb_queue_head_init(&rtwsdio->tx_queue[i]); + rtwsdio->tx_handler_data = kmalloc(sizeof(*rtwsdio->tx_handler_data), + GFP_KERNEL); + if (!rtwsdio->tx_handler_data) + goto err_destroy_wq; + + rtwsdio->tx_handler_data->rtwdev = rtwdev; + INIT_WORK(&rtwsdio->tx_handler_data->work, rtw_sdio_tx_handler); + + return 0; + +err_destroy_wq: + destroy_workqueue(rtwsdio->txwq); + return -ENOMEM; +} + +static void rtw_sdio_deinit_tx(struct rtw_dev *rtwdev) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + int i; + + for (i = 0; i < RTK_MAX_TX_QUEUE_NUM; i++) + skb_queue_purge(&rtwsdio->tx_queue[i]); + + flush_workqueue(rtwsdio->txwq); + destroy_workqueue(rtwsdio->txwq); + kfree(rtwsdio->tx_handler_data); +} + +int rtw_sdio_probe(struct sdio_func *sdio_func, + const struct sdio_device_id *id) +{ + struct ieee80211_hw *hw; + struct rtw_dev *rtwdev; + int drv_data_size; + int ret; + + drv_data_size = sizeof(struct rtw_dev) + sizeof(struct rtw_sdio); + hw = ieee80211_alloc_hw(drv_data_size, &rtw_ops); + if (!hw) { + dev_err(&sdio_func->dev, "failed to allocate hw"); + return -ENOMEM; + } + + rtwdev = hw->priv; + rtwdev->hw = hw; + rtwdev->dev = &sdio_func->dev; + rtwdev->chip = (struct rtw_chip_info *)id->driver_data; + rtwdev->hci.ops = &rtw_sdio_ops; + rtwdev->hci.type = RTW_HCI_TYPE_SDIO; + + ret = rtw_core_init(rtwdev); + if (ret) + goto err_release_hw; + + rtw_dbg(rtwdev, RTW_DBG_SDIO, + "rtw88 SDIO probe: vendor=0x%04x device=%04x class=%02x", + id->vendor, id->device, id->class); + + ret = rtw_sdio_claim(rtwdev, sdio_func); + if (ret) { + rtw_err(rtwdev, "failed to claim SDIO device"); + goto err_deinit_core; + } + + rtw_sdio_init(rtwdev); + + ret = rtw_sdio_init_tx(rtwdev); + if (ret) { + rtw_err(rtwdev, "failed to init SDIO TX queue\n"); + goto err_sdio_declaim; + } + + ret = rtw_chip_info_setup(rtwdev); + if (ret) { + rtw_err(rtwdev, "failed to setup chip information"); + goto err_destroy_txwq; + } + + ret = rtw_sdio_request_irq(rtwdev, sdio_func); + if (ret) + goto err_destroy_txwq; + + ret = rtw_register_hw(rtwdev, hw); + if (ret) { + rtw_err(rtwdev, "failed to register hw"); + goto err_free_irq; + } + + return 0; + +err_free_irq: + rtw_sdio_free_irq(rtwdev, sdio_func); +err_destroy_txwq: + rtw_sdio_deinit_tx(rtwdev); +err_sdio_declaim: + rtw_sdio_declaim(rtwdev, sdio_func); +err_deinit_core: + rtw_core_deinit(rtwdev); +err_release_hw: + ieee80211_free_hw(hw); + + return ret; +} +EXPORT_SYMBOL(rtw_sdio_probe); + +void rtw_sdio_remove(struct sdio_func *sdio_func) +{ + struct ieee80211_hw *hw = sdio_get_drvdata(sdio_func); + struct rtw_dev *rtwdev; + + if (!hw) + return; + + rtwdev = hw->priv; + + rtw_unregister_hw(rtwdev, hw); + rtw_sdio_disable_interrupt(rtwdev); + rtw_sdio_free_irq(rtwdev, sdio_func); + rtw_sdio_declaim(rtwdev, sdio_func); + rtw_sdio_deinit_tx(rtwdev); + rtw_core_deinit(rtwdev); + ieee80211_free_hw(hw); +} +EXPORT_SYMBOL(rtw_sdio_remove); + +void rtw_sdio_shutdown(struct device *dev) +{ + struct sdio_func *sdio_func = dev_to_sdio_func(dev); + const struct rtw_chip_info *chip; + struct ieee80211_hw *hw; + struct rtw_dev *rtwdev; + + hw = sdio_get_drvdata(sdio_func); + if (!hw) + return; + + rtwdev = hw->priv; + chip = rtwdev->chip; + + if (chip->ops->shutdown) + chip->ops->shutdown(rtwdev); +} +EXPORT_SYMBOL(rtw_sdio_shutdown); + +MODULE_AUTHOR("Martin Blumenstingl"); +MODULE_AUTHOR("Jernej Skrabec"); +MODULE_DESCRIPTION("Realtek 802.11ac wireless SDIO driver"); +MODULE_LICENSE("Dual BSD/GPL"); diff --git a/drivers/net/wireless/realtek/rtw88/sdio.h b/drivers/net/wireless/realtek/rtw88/sdio.h new file mode 100644 index 000000000000..3c659ed180f0 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/sdio.h @@ -0,0 +1,178 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright (C) 2021 Martin Blumenstingl <[email protected]> + * Copyright (C) 2021 Jernej Skrabec <[email protected]> + */ + +#ifndef __REG_SDIO_H_ +#define __REG_SDIO_H_ + +/* I/O bus domain address mapping */ +#define SDIO_LOCAL_OFFSET 0x10250000 +#define WLAN_IOREG_OFFSET 0x10260000 +#define FIRMWARE_FIFO_OFFSET 0x10270000 +#define TX_HIQ_OFFSET 0x10310000 +#define TX_MIQ_OFFSET 0x10320000 +#define TX_LOQ_OFFSET 0x10330000 +#define TX_EPQ_OFFSET 0x10350000 +#define RX_RX0FF_OFFSET 0x10340000 + +#define RTW_SDIO_BUS_MSK 0xffff0000 +#define SDIO_LOCAL_REG_MSK 0x00000fff +#define WLAN_IOREG_REG_MSK 0x0000ffff + +/* SDIO Tx Control */ +#define REG_SDIO_TX_CTRL (SDIO_LOCAL_OFFSET + 0x0000) + +/*SDIO status timeout*/ +#define REG_SDIO_TIMEOUT (SDIO_LOCAL_OFFSET + 0x0002) + +/* SDIO Host Interrupt Mask */ +#define REG_SDIO_HIMR (SDIO_LOCAL_OFFSET + 0x0014) +#define REG_SDIO_HIMR_RX_REQUEST BIT(0) +#define REG_SDIO_HIMR_AVAL BIT(1) +#define REG_SDIO_HIMR_TXERR BIT(2) +#define REG_SDIO_HIMR_RXERR BIT(3) +#define REG_SDIO_HIMR_TXFOVW BIT(4) +#define REG_SDIO_HIMR_RXFOVW BIT(5) +#define REG_SDIO_HIMR_TXBCNOK BIT(6) +#define REG_SDIO_HIMR_TXBCNERR BIT(7) +#define REG_SDIO_HIMR_BCNERLY_INT BIT(16) +#define REG_SDIO_HIMR_C2HCMD BIT(17) +#define REG_SDIO_HIMR_CPWM1 BIT(18) +#define REG_SDIO_HIMR_CPWM2 BIT(19) +#define REG_SDIO_HIMR_HSISR_IND BIT(20) +#define REG_SDIO_HIMR_GTINT3_IND BIT(21) +#define REG_SDIO_HIMR_GTINT4_IND BIT(22) +#define REG_SDIO_HIMR_PSTIMEOUT BIT(23) +#define REG_SDIO_HIMR_OCPINT BIT(24) +#define REG_SDIO_HIMR_ATIMEND BIT(25) +#define REG_SDIO_HIMR_ATIMEND_E BIT(26) +#define REG_SDIO_HIMR_CTWEND BIT(27) +/* the following two are RTL8188 SDIO Specific */ +#define REG_SDIO_HIMR_MCU_ERR BIT(28) +#define REG_SDIO_HIMR_TSF_BIT32_TOGGLE BIT(29) + +/* SDIO Host Interrupt Service Routine */ +#define REG_SDIO_HISR (SDIO_LOCAL_OFFSET + 0x0018) +#define REG_SDIO_HISR_RX_REQUEST BIT(0) +#define REG_SDIO_HISR_AVAL BIT(1) +#define REG_SDIO_HISR_TXERR BIT(2) +#define REG_SDIO_HISR_RXERR BIT(3) +#define REG_SDIO_HISR_TXFOVW BIT(4) +#define REG_SDIO_HISR_RXFOVW BIT(5) +#define REG_SDIO_HISR_TXBCNOK BIT(6) +#define REG_SDIO_HISR_TXBCNERR BIT(7) +#define REG_SDIO_HISR_BCNERLY_INT BIT(16) +#define REG_SDIO_HISR_C2HCMD BIT(17) +#define REG_SDIO_HISR_CPWM1 BIT(18) +#define REG_SDIO_HISR_CPWM2 BIT(19) +#define REG_SDIO_HISR_HSISR_IND BIT(20) +#define REG_SDIO_HISR_GTINT3_IND BIT(21) +#define REG_SDIO_HISR_GTINT4_IND BIT(22) +#define REG_SDIO_HISR_PSTIMEOUT BIT(23) +#define REG_SDIO_HISR_OCPINT BIT(24) +#define REG_SDIO_HISR_ATIMEND BIT(25) +#define REG_SDIO_HISR_ATIMEND_E BIT(26) +#define REG_SDIO_HISR_CTWEND BIT(27) +/* the following two are RTL8188 SDIO Specific */ +#define REG_SDIO_HISR_MCU_ERR BIT(28) +#define REG_SDIO_HISR_TSF_BIT32_TOGGLE BIT(29) + +/* HCI Current Power Mode */ +#define REG_SDIO_HCPWM (SDIO_LOCAL_OFFSET + 0x0019) +/* RXDMA Request Length */ +#define REG_SDIO_RX0_REQ_LEN (SDIO_LOCAL_OFFSET + 0x001C) +/* OQT Free Page */ +#define REG_SDIO_OQT_FREE_PG (SDIO_LOCAL_OFFSET + 0x001E) +/* Free Tx Buffer Page */ +#define REG_SDIO_FREE_TXPG (SDIO_LOCAL_OFFSET + 0x0020) +/* HCI Current Power Mode 1 */ +#define REG_SDIO_HCPWM1 (SDIO_LOCAL_OFFSET + 0x0024) +/* HCI Current Power Mode 2 */ +#define REG_SDIO_HCPWM2 (SDIO_LOCAL_OFFSET + 0x0026) +/* Free Tx Page Sequence */ +#define REG_SDIO_FREE_TXPG_SEQ (SDIO_LOCAL_OFFSET + 0x0028) +/* HTSF Information */ +#define REG_SDIO_HTSFR_INFO (SDIO_LOCAL_OFFSET + 0x0030) +#define REG_SDIO_HCPWM1_V2 (SDIO_LOCAL_OFFSET + 0x0038) +/* H2C */ +#define REG_SDIO_H2C (SDIO_LOCAL_OFFSET + 0x0060) +/* HCI Request Power Mode 1 */ +#define REG_SDIO_HRPWM1 (SDIO_LOCAL_OFFSET + 0x0080) +/* HCI Request Power Mode 2 */ +#define REG_SDIO_HRPWM2 (SDIO_LOCAL_OFFSET + 0x0082) +/* HCI Power Save Clock */ +#define REG_SDIO_HPS_CLKR (SDIO_LOCAL_OFFSET + 0x0084) +/* SDIO HCI Suspend Control */ +#define REG_SDIO_HSUS_CTRL (SDIO_LOCAL_OFFSET + 0x0086) +#define BIT_HCI_SUS_REQ BIT(0) +#define BIT_HCI_RESUME_RDY BIT(1) +/* SDIO Host Extension Interrupt Mask Always */ +#define REG_SDIO_HIMR_ON (SDIO_LOCAL_OFFSET + 0x0090) +/* SDIO Host Extension Interrupt Status Always */ +#define REG_SDIO_HISR_ON (SDIO_LOCAL_OFFSET + 0x0091) + +#define REG_SDIO_INDIRECT_REG_CFG (SDIO_LOCAL_OFFSET + 0x0040) +#define BIT_SDIO_INDIRECT_REG_CFG_WORD BIT(16) +#define BIT_SDIO_INDIRECT_REG_CFG_DWORD BIT(17) +#define BIT_SDIO_INDIRECT_REG_CFG_WRITE BIT(18) +#define BIT_SDIO_INDIRECT_REG_CFG_READ BIT(19) +#define BIT_SDIO_INDIRECT_REG_CFG_UNK20 BIT(20) +#define REG_SDIO_INDIRECT_REG_DATA (SDIO_LOCAL_OFFSET + 0x0044) + +/* Sdio Address for SDIO Local Reg, TRX FIFO, MAC Reg */ +#define REG_SDIO_CMD_ADDR_MSK GENMASK(16, 13) +#define REG_SDIO_CMD_ADDR_SDIO_REG 0 +#define REG_SDIO_CMD_ADDR_MAC_REG 8 +#define REG_SDIO_CMD_ADDR_TXFF_HIGH 4 +#define REG_SDIO_CMD_ADDR_TXFF_LOW 6 +#define REG_SDIO_CMD_ADDR_TXFF_NORMAL 5 +#define REG_SDIO_CMD_ADDR_TXFF_EXTRA 7 +#define REG_SDIO_CMD_ADDR_RXFF 7 + +#define RTW_SDIO_BLOCK_SIZE 512 +#define RTW_SDIO_ADDR_RX_RX0FF_GEN(_id) (0x0e000 | ((_id) & 0x3)) + +#define RTW_SDIO_DATA_PTR_ALIGN 8 + +struct sdio_func; +struct sdio_device_id; + +struct rtw_sdio_tx_data { + u8 sn; +}; + +struct rtw_sdio_work_data { + struct work_struct work; + struct rtw_dev *rtwdev; +}; + +struct rtw_sdio { + struct sdio_func *sdio_func; + + u32 irq_mask; + u8 rx_addr; + bool sdio3_bus_mode; + + void *irq_thread; + + struct workqueue_struct *txwq; + struct rtw_sdio_work_data *tx_handler_data; + struct sk_buff_head tx_queue[RTK_MAX_TX_QUEUE_NUM]; +}; + +extern const struct dev_pm_ops rtw_sdio_pm_ops; + +int rtw_sdio_probe(struct sdio_func *sdio_func, + const struct sdio_device_id *id); +void rtw_sdio_remove(struct sdio_func *sdio_func); +void rtw_sdio_shutdown(struct device *dev); + +static inline bool rtw_sdio_is_sdio30_supported(struct rtw_dev *rtwdev) +{ + struct rtw_sdio *rtwsdio = (struct rtw_sdio *)rtwdev->priv; + + return rtwsdio->sdio3_bus_mode; +} + +#endif diff --git a/drivers/net/wireless/realtek/rtw88/usb.c b/drivers/net/wireless/realtek/rtw88/usb.c index 68e1b782d199..44a5fafb9905 100644 --- a/drivers/net/wireless/realtek/rtw88/usb.c +++ b/drivers/net/wireless/realtek/rtw88/usb.c @@ -118,6 +118,22 @@ static void rtw_usb_write32(struct rtw_dev *rtwdev, u32 addr, u32 val) rtw_usb_write(rtwdev, addr, val, 4); } +static int dma_mapping_to_ep(enum rtw_dma_mapping dma_mapping) +{ + switch (dma_mapping) { + case RTW_DMA_MAPPING_HIGH: + return 0; + case RTW_DMA_MAPPING_NORMAL: + return 1; + case RTW_DMA_MAPPING_LOW: + return 2; + case RTW_DMA_MAPPING_EXTRA: + return 3; + default: + return -EINVAL; + } +} + static int rtw_usb_parse(struct rtw_dev *rtwdev, struct usb_interface *interface) { @@ -129,6 +145,8 @@ static int rtw_usb_parse(struct rtw_dev *rtwdev, int num_out_pipes = 0; int i; u8 num; + const struct rtw_chip_info *chip = rtwdev->chip; + const struct rtw_rqpn *rqpn; for (i = 0; i < interface_desc->bNumEndpoints; i++) { endpoint = &host_interface->endpoint[i].desc; @@ -183,31 +201,34 @@ static int rtw_usb_parse(struct rtw_dev *rtwdev, rtwdev->hci.bulkout_num = num_out_pipes; - switch (num_out_pipes) { - case 4: - case 3: - rtwusb->qsel_to_ep[TX_DESC_QSEL_TID0] = 2; - rtwusb->qsel_to_ep[TX_DESC_QSEL_TID1] = 2; - rtwusb->qsel_to_ep[TX_DESC_QSEL_TID2] = 2; - rtwusb->qsel_to_ep[TX_DESC_QSEL_TID3] = 2; - rtwusb->qsel_to_ep[TX_DESC_QSEL_TID4] = 1; - rtwusb->qsel_to_ep[TX_DESC_QSEL_TID5] = 1; - rtwusb->qsel_to_ep[TX_DESC_QSEL_TID6] = 0; - rtwusb->qsel_to_ep[TX_DESC_QSEL_TID7] = 0; - break; - case 2: - rtwusb->qsel_to_ep[TX_DESC_QSEL_TID0] = 1; - rtwusb->qsel_to_ep[TX_DESC_QSEL_TID1] = 1; - rtwusb->qsel_to_ep[TX_DESC_QSEL_TID2] = 1; - rtwusb->qsel_to_ep[TX_DESC_QSEL_TID3] = 1; - break; - case 1: - break; - default: - rtw_err(rtwdev, "failed to get out_pipes(%d)\n", num_out_pipes); + if (num_out_pipes < 1 || num_out_pipes > 4) { + rtw_err(rtwdev, "invalid number of endpoints %d\n", num_out_pipes); return -EINVAL; } + rqpn = &chip->rqpn_table[num_out_pipes]; + + rtwusb->qsel_to_ep[TX_DESC_QSEL_TID0] = dma_mapping_to_ep(rqpn->dma_map_be); + rtwusb->qsel_to_ep[TX_DESC_QSEL_TID1] = dma_mapping_to_ep(rqpn->dma_map_bk); + rtwusb->qsel_to_ep[TX_DESC_QSEL_TID2] = dma_mapping_to_ep(rqpn->dma_map_bk); + rtwusb->qsel_to_ep[TX_DESC_QSEL_TID3] = dma_mapping_to_ep(rqpn->dma_map_be); + rtwusb->qsel_to_ep[TX_DESC_QSEL_TID4] = dma_mapping_to_ep(rqpn->dma_map_vi); + rtwusb->qsel_to_ep[TX_DESC_QSEL_TID5] = dma_mapping_to_ep(rqpn->dma_map_vi); + rtwusb->qsel_to_ep[TX_DESC_QSEL_TID6] = dma_mapping_to_ep(rqpn->dma_map_vo); + rtwusb->qsel_to_ep[TX_DESC_QSEL_TID7] = dma_mapping_to_ep(rqpn->dma_map_vo); + rtwusb->qsel_to_ep[TX_DESC_QSEL_TID8] = -EINVAL; + rtwusb->qsel_to_ep[TX_DESC_QSEL_TID9] = -EINVAL; + rtwusb->qsel_to_ep[TX_DESC_QSEL_TID10] = -EINVAL; + rtwusb->qsel_to_ep[TX_DESC_QSEL_TID11] = -EINVAL; + rtwusb->qsel_to_ep[TX_DESC_QSEL_TID12] = -EINVAL; + rtwusb->qsel_to_ep[TX_DESC_QSEL_TID13] = -EINVAL; + rtwusb->qsel_to_ep[TX_DESC_QSEL_TID14] = -EINVAL; + rtwusb->qsel_to_ep[TX_DESC_QSEL_TID15] = -EINVAL; + rtwusb->qsel_to_ep[TX_DESC_QSEL_BEACON] = dma_mapping_to_ep(rqpn->dma_map_hi); + rtwusb->qsel_to_ep[TX_DESC_QSEL_HIGH] = dma_mapping_to_ep(rqpn->dma_map_hi); + rtwusb->qsel_to_ep[TX_DESC_QSEL_MGMT] = dma_mapping_to_ep(rqpn->dma_map_mg); + rtwusb->qsel_to_ep[TX_DESC_QSEL_H2C] = dma_mapping_to_ep(rqpn->dma_map_hi); + return 0; } @@ -250,7 +271,7 @@ static void rtw_usb_write_port_tx_complete(struct urb *urb) static int qsel_to_ep(struct rtw_usb *rtwusb, unsigned int qsel) { if (qsel >= ARRAY_SIZE(rtwusb->qsel_to_ep)) - return 0; + return -EINVAL; return rtwusb->qsel_to_ep[qsel]; } @@ -265,6 +286,9 @@ static int rtw_usb_write_port(struct rtw_dev *rtwdev, u8 qsel, struct sk_buff *s int ret; int ep = qsel_to_ep(rtwusb, qsel); + if (ep < 0) + return ep; + pipe = usb_sndbulkpipe(usbd, rtwusb->out_ep[ep]); urb = usb_alloc_urb(0, GFP_ATOMIC); if (!urb) @@ -780,6 +804,7 @@ static void rtw_usb_intf_deinit(struct rtw_dev *rtwdev, struct rtw_usb *rtwusb = rtw_get_usb_priv(rtwdev); usb_put_dev(rtwusb->udev); + kfree(rtwusb->usb_data); usb_set_intfdata(intf, NULL); } diff --git a/drivers/net/wireless/realtek/rtw89/chan.c b/drivers/net/wireless/realtek/rtw89/chan.c index 90596806bc93..4663db4ce2f6 100644 --- a/drivers/net/wireless/realtek/rtw89/chan.c +++ b/drivers/net/wireless/realtek/rtw89/chan.c @@ -141,6 +141,38 @@ void rtw89_config_entity_chandef(struct rtw89_dev *rtwdev, __rtw89_config_entity_chandef(rtwdev, idx, chandef, true); } +void rtw89_config_roc_chandef(struct rtw89_dev *rtwdev, + enum rtw89_sub_entity_idx idx, + const struct cfg80211_chan_def *chandef) +{ + struct rtw89_hal *hal = &rtwdev->hal; + enum rtw89_sub_entity_idx cur; + + if (chandef) { + cur = atomic_cmpxchg(&hal->roc_entity_idx, + RTW89_SUB_ENTITY_IDLE, idx); + if (cur != RTW89_SUB_ENTITY_IDLE) { + rtw89_debug(rtwdev, RTW89_DBG_TXRX, + "ROC still processing on entity %d\n", idx); + return; + } + + hal->roc_chandef = *chandef; + } else { + cur = atomic_cmpxchg(&hal->roc_entity_idx, idx, + RTW89_SUB_ENTITY_IDLE); + if (cur == idx) + return; + + if (cur == RTW89_SUB_ENTITY_IDLE) + rtw89_debug(rtwdev, RTW89_DBG_TXRX, + "ROC already finished on entity %d\n", idx); + else + rtw89_debug(rtwdev, RTW89_DBG_TXRX, + "ROC is processing on entity %d\n", cur); + } +} + static void rtw89_config_default_chandef(struct rtw89_dev *rtwdev) { struct cfg80211_chan_def chandef = {0}; @@ -154,6 +186,7 @@ void rtw89_entity_init(struct rtw89_dev *rtwdev) struct rtw89_hal *hal = &rtwdev->hal; bitmap_zero(hal->entity_map, NUM_OF_RTW89_SUB_ENTITY); + atomic_set(&hal->roc_entity_idx, RTW89_SUB_ENTITY_IDLE); rtw89_config_default_chandef(rtwdev); } @@ -229,6 +262,8 @@ void rtw89_chanctx_ops_remove(struct rtw89_dev *rtwdev, rtwvif->sub_entity_idx = RTW89_SUB_ENTITY_0; } + atomic_cmpxchg(&hal->roc_entity_idx, roll, RTW89_SUB_ENTITY_0); + drop = roll; out: diff --git a/drivers/net/wireless/realtek/rtw89/chan.h b/drivers/net/wireless/realtek/rtw89/chan.h index ecbd4503bead..bdf369db5041 100644 --- a/drivers/net/wireless/realtek/rtw89/chan.h +++ b/drivers/net/wireless/realtek/rtw89/chan.h @@ -45,6 +45,9 @@ bool rtw89_assign_entity_chan(struct rtw89_dev *rtwdev, void rtw89_config_entity_chandef(struct rtw89_dev *rtwdev, enum rtw89_sub_entity_idx idx, const struct cfg80211_chan_def *chandef); +void rtw89_config_roc_chandef(struct rtw89_dev *rtwdev, + enum rtw89_sub_entity_idx idx, + const struct cfg80211_chan_def *chandef); void rtw89_entity_init(struct rtw89_dev *rtwdev); enum rtw89_entity_mode rtw89_entity_recalc(struct rtw89_dev *rtwdev); int rtw89_chanctx_ops_add(struct rtw89_dev *rtwdev, diff --git a/drivers/net/wireless/realtek/rtw89/core.c b/drivers/net/wireless/realtek/rtw89/core.c index 56a13be2e283..7fc0a26a4d73 100644 --- a/drivers/net/wireless/realtek/rtw89/core.c +++ b/drivers/net/wireless/realtek/rtw89/core.c @@ -156,6 +156,28 @@ static struct ieee80211_rate rtw89_bitrates[] = { { .bitrate = 540, .hw_value = 0x0b, }, }; +static const struct ieee80211_iface_limit rtw89_iface_limits[] = { + { + .max = 1, + .types = BIT(NL80211_IFTYPE_STATION), + }, + { + .max = 1, + .types = BIT(NL80211_IFTYPE_P2P_CLIENT) | + BIT(NL80211_IFTYPE_P2P_GO) | + BIT(NL80211_IFTYPE_AP), + }, +}; + +static const struct ieee80211_iface_combination rtw89_iface_combs[] = { + { + .limits = rtw89_iface_limits, + .n_limits = ARRAY_SIZE(rtw89_iface_limits), + .max_interfaces = 2, + .num_different_channels = 1, + } +}; + bool rtw89_ra_report_to_bitrate(struct rtw89_dev *rtwdev, u8 rpt_rate, u16 *bitrate) { struct ieee80211_rate rate; @@ -360,6 +382,15 @@ void rtw89_set_channel(struct rtw89_dev *rtwdev) rtw89_set_entity_state(rtwdev, true); } +void rtw89_get_channel(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif, + struct rtw89_chan *chan) +{ + const struct cfg80211_chan_def *chandef; + + chandef = rtw89_chandef_get(rtwdev, rtwvif->sub_entity_idx); + rtw89_get_channel_params(chandef, chan); +} + static enum rtw89_core_tx_type rtw89_core_get_tx_type(struct rtw89_dev *rtwdev, struct sk_buff *skb) @@ -707,7 +738,7 @@ static u16 rtw89_core_get_data_rate(struct rtw89_dev *rtwdev, else lowest_rate = RTW89_HW_RATE_OFDM6; - if (!sta->deflink.supp_rates[chan->band_type]) + if (!sta || !sta->deflink.supp_rates[chan->band_type]) return lowest_rate; return __ffs(sta->deflink.supp_rates[chan->band_type]) + lowest_rate; @@ -867,6 +898,37 @@ void rtw89_core_tx_kick_off(struct rtw89_dev *rtwdev, u8 qsel) rtw89_hci_tx_kick_off(rtwdev, ch_dma); } +int rtw89_core_tx_kick_off_and_wait(struct rtw89_dev *rtwdev, struct sk_buff *skb, + int qsel, unsigned int timeout) +{ + struct rtw89_tx_skb_data *skb_data = RTW89_TX_SKB_CB(skb); + struct rtw89_tx_wait_info *wait; + unsigned long time_left; + int ret = 0; + + wait = kzalloc(sizeof(*wait), GFP_KERNEL); + if (!wait) { + rtw89_core_tx_kick_off(rtwdev, qsel); + return 0; + } + + init_completion(&wait->completion); + rcu_assign_pointer(skb_data->wait, wait); + + rtw89_core_tx_kick_off(rtwdev, qsel); + time_left = wait_for_completion_timeout(&wait->completion, + msecs_to_jiffies(timeout)); + if (time_left == 0) + ret = -ETIMEDOUT; + else if (!wait->tx_done) + ret = -EAGAIN; + + rcu_assign_pointer(skb_data->wait, NULL); + kfree_rcu(wait, rcu_head); + + return ret; +} + int rtw89_h2c_tx(struct rtw89_dev *rtwdev, struct sk_buff *skb, bool fwdl) { @@ -1457,6 +1519,7 @@ static void rtw89_vif_rx_stats_iter(void *data, u8 *mac, struct rtw89_rx_desc_info *desc_info = iter_data->desc_info; struct sk_buff *skb = iter_data->skb; struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data; + struct rtw89_rx_phy_ppdu *phy_ppdu = iter_data->phy_ppdu; const u8 *bssid = iter_data->bssid; if (rtwdev->scanning && @@ -1475,8 +1538,11 @@ static void rtw89_vif_rx_stats_iter(void *data, u8 *mac, if (!ether_addr_equal(vif->bss_conf.bssid, bssid)) return; - if (ieee80211_is_beacon(hdr->frame_control)) + if (ieee80211_is_beacon(hdr->frame_control)) { + if (vif->type == NL80211_IFTYPE_STATION) + rtw89_fw_h2c_rssi_offload(rtwdev, phy_ppdu); pkt_stat->beacon_nr++; + } if (!ether_addr_equal(vif->addr, hdr->addr1)) return; @@ -1979,6 +2045,18 @@ static void rtw89_core_free_sta_pending_forbid_ba(struct rtw89_dev *rtwdev, spin_unlock_bh(&rtwdev->ba_lock); } +static void rtw89_core_free_sta_pending_roc_tx(struct rtw89_dev *rtwdev, + struct ieee80211_sta *sta) +{ + struct rtw89_sta *rtwsta = (struct rtw89_sta *)sta->drv_priv; + struct sk_buff *skb, *tmp; + + skb_queue_walk_safe(&rtwsta->roc_queue, skb, tmp) { + skb_unlink(skb, &rtwsta->roc_queue); + dev_kfree_skb_any(skb); + } +} + static void rtw89_core_stop_tx_ba_session(struct rtw89_dev *rtwdev, struct rtw89_txq *rtwtxq) { @@ -2118,6 +2196,7 @@ static void rtw89_core_txq_schedule(struct rtw89_dev *rtwdev, u8 ac, bool *reinv { struct ieee80211_hw *hw = rtwdev->hw; struct ieee80211_txq *txq; + struct rtw89_vif *rtwvif; struct rtw89_txq *rtwtxq; unsigned long frame_cnt; unsigned long byte_cnt; @@ -2127,6 +2206,12 @@ static void rtw89_core_txq_schedule(struct rtw89_dev *rtwdev, u8 ac, bool *reinv ieee80211_txq_schedule_start(hw, ac); while ((txq = ieee80211_next_txq(hw, ac))) { rtwtxq = (struct rtw89_txq *)txq->drv_priv; + rtwvif = (struct rtw89_vif *)txq->vif->drv_priv; + + if (rtwvif->offchan) { + ieee80211_return_txq(hw, txq, true); + continue; + } tx_resource = rtw89_check_and_reclaim_tx_resource(rtwdev, txq->tid); sched_txq = false; @@ -2153,8 +2238,7 @@ static void rtw89_ips_work(struct work_struct *work) struct rtw89_dev *rtwdev = container_of(work, struct rtw89_dev, ips_work); mutex_lock(&rtwdev->mutex); - if (rtwdev->hw->conf.flags & IEEE80211_CONF_IDLE) - rtw89_enter_ips(rtwdev); + rtw89_enter_ips_by_hwflags(rtwdev); mutex_unlock(&rtwdev->mutex); } @@ -2195,6 +2279,187 @@ static void rtw89_forbid_ba_work(struct work_struct *w) spin_unlock_bh(&rtwdev->ba_lock); } +static void rtw89_core_sta_pending_tx_iter(void *data, + struct ieee80211_sta *sta) +{ + struct rtw89_sta *rtwsta = (struct rtw89_sta *)sta->drv_priv; + struct rtw89_vif *rtwvif_target = data, *rtwvif = rtwsta->rtwvif; + struct rtw89_dev *rtwdev = rtwvif->rtwdev; + struct ieee80211_vif *vif = rtwvif_to_vif(rtwvif); + struct sk_buff *skb, *tmp; + int qsel, ret; + + if (rtwvif->sub_entity_idx != rtwvif_target->sub_entity_idx) + return; + + if (skb_queue_len(&rtwsta->roc_queue) == 0) + return; + + skb_queue_walk_safe(&rtwsta->roc_queue, skb, tmp) { + skb_unlink(skb, &rtwsta->roc_queue); + + ret = rtw89_core_tx_write(rtwdev, vif, sta, skb, &qsel); + if (ret) { + rtw89_warn(rtwdev, "pending tx failed with %d\n", ret); + dev_kfree_skb_any(skb); + } else { + rtw89_core_tx_kick_off(rtwdev, qsel); + } + } +} + +static void rtw89_core_handle_sta_pending_tx(struct rtw89_dev *rtwdev, + struct rtw89_vif *rtwvif) +{ + ieee80211_iterate_stations_atomic(rtwdev->hw, + rtw89_core_sta_pending_tx_iter, + rtwvif); +} + +static int rtw89_core_send_nullfunc(struct rtw89_dev *rtwdev, + struct rtw89_vif *rtwvif, bool qos, bool ps) +{ + struct ieee80211_vif *vif = rtwvif_to_vif(rtwvif); + struct ieee80211_sta *sta; + struct ieee80211_hdr *hdr; + struct sk_buff *skb; + int ret, qsel; + + if (vif->type != NL80211_IFTYPE_STATION || !vif->cfg.assoc) + return 0; + + rcu_read_lock(); + sta = ieee80211_find_sta(vif, vif->bss_conf.bssid); + if (!sta) { + ret = -EINVAL; + goto out; + } + + skb = ieee80211_nullfunc_get(rtwdev->hw, vif, -1, qos); + if (!skb) { + ret = -ENOMEM; + goto out; + } + + hdr = (struct ieee80211_hdr *)skb->data; + if (ps) + hdr->frame_control |= cpu_to_le16(IEEE80211_FCTL_PM); + + ret = rtw89_core_tx_write(rtwdev, vif, sta, skb, &qsel); + if (ret) { + rtw89_warn(rtwdev, "nullfunc transmit failed: %d\n", ret); + dev_kfree_skb_any(skb); + goto out; + } + + rcu_read_unlock(); + + return rtw89_core_tx_kick_off_and_wait(rtwdev, skb, qsel, + RTW89_ROC_TX_TIMEOUT); +out: + rcu_read_unlock(); + + return ret; +} + +void rtw89_roc_start(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif) +{ + struct ieee80211_hw *hw = rtwdev->hw; + struct rtw89_roc *roc = &rtwvif->roc; + struct cfg80211_chan_def roc_chan; + struct rtw89_vif *tmp; + int ret; + + lockdep_assert_held(&rtwdev->mutex); + + ieee80211_queue_delayed_work(hw, &rtwvif->roc.roc_work, + msecs_to_jiffies(rtwvif->roc.duration)); + + rtw89_leave_ips_by_hwflags(rtwdev); + rtw89_leave_lps(rtwdev); + + ret = rtw89_core_send_nullfunc(rtwdev, rtwvif, true, true); + if (ret) + rtw89_debug(rtwdev, RTW89_DBG_TXRX, + "roc send null-1 failed: %d\n", ret); + + rtw89_for_each_rtwvif(rtwdev, tmp) + if (tmp->sub_entity_idx == rtwvif->sub_entity_idx) + tmp->offchan = true; + + cfg80211_chandef_create(&roc_chan, &roc->chan, NL80211_CHAN_NO_HT); + rtw89_config_roc_chandef(rtwdev, rtwvif->sub_entity_idx, &roc_chan); + rtw89_set_channel(rtwdev); + rtw89_write32_clr(rtwdev, + rtw89_mac_reg_by_idx(R_AX_RX_FLTR_OPT, RTW89_MAC_0), + B_AX_A_UC_CAM_MATCH | B_AX_A_BC_CAM_MATCH); + + ieee80211_ready_on_channel(hw); +} + +void rtw89_roc_end(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif) +{ + struct ieee80211_hw *hw = rtwdev->hw; + struct rtw89_roc *roc = &rtwvif->roc; + struct rtw89_vif *tmp; + int ret; + + lockdep_assert_held(&rtwdev->mutex); + + ieee80211_remain_on_channel_expired(hw); + + rtw89_leave_ips_by_hwflags(rtwdev); + rtw89_leave_lps(rtwdev); + + rtw89_write32_mask(rtwdev, + rtw89_mac_reg_by_idx(R_AX_RX_FLTR_OPT, RTW89_MAC_0), + B_AX_RX_FLTR_CFG_MASK, + rtwdev->hal.rx_fltr); + + roc->state = RTW89_ROC_IDLE; + rtw89_config_roc_chandef(rtwdev, rtwvif->sub_entity_idx, NULL); + rtw89_set_channel(rtwdev); + ret = rtw89_core_send_nullfunc(rtwdev, rtwvif, true, false); + if (ret) + rtw89_debug(rtwdev, RTW89_DBG_TXRX, + "roc send null-0 failed: %d\n", ret); + + rtw89_for_each_rtwvif(rtwdev, tmp) + if (tmp->sub_entity_idx == rtwvif->sub_entity_idx) + tmp->offchan = false; + + rtw89_core_handle_sta_pending_tx(rtwdev, rtwvif); + queue_work(rtwdev->txq_wq, &rtwdev->txq_work); + + if (hw->conf.flags & IEEE80211_CONF_IDLE) + ieee80211_queue_delayed_work(hw, &roc->roc_work, + RTW89_ROC_IDLE_TIMEOUT); +} + +void rtw89_roc_work(struct work_struct *work) +{ + struct rtw89_vif *rtwvif = container_of(work, struct rtw89_vif, + roc.roc_work.work); + struct rtw89_dev *rtwdev = rtwvif->rtwdev; + struct rtw89_roc *roc = &rtwvif->roc; + + mutex_lock(&rtwdev->mutex); + + switch (roc->state) { + case RTW89_ROC_IDLE: + rtw89_enter_ips_by_hwflags(rtwdev); + break; + case RTW89_ROC_MGMT: + case RTW89_ROC_NORMAL: + rtw89_roc_end(rtwdev, rtwvif); + break; + default: + break; + } + + mutex_unlock(&rtwdev->mutex); +} + static enum rtw89_tfc_lv rtw89_get_traffic_level(struct rtw89_dev *rtwdev, u32 throughput, u64 cnt) { @@ -2251,8 +2516,10 @@ static bool rtw89_traffic_stats_track(struct rtw89_dev *rtwdev) bool tfc_changed; tfc_changed = rtw89_traffic_stats_calc(rtwdev, &rtwdev->stats); - rtw89_for_each_rtwvif(rtwdev, rtwvif) + rtw89_for_each_rtwvif(rtwdev, rtwvif) { rtw89_traffic_stats_calc(rtwdev, &rtwvif->stats); + rtw89_fw_h2c_tp_offload(rtwdev, rtwvif); + } return tfc_changed; } @@ -2264,9 +2531,15 @@ static void rtw89_vif_enter_lps(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwv rtwvif->tdls_peer) return; + if (rtwdev->total_sta_assoc > 1) + return; + + if (rtwvif->offchan) + return; + if (rtwvif->stats.tx_tfc_lv == RTW89_TFC_IDLE && rtwvif->stats.rx_tfc_lv == RTW89_TFC_IDLE) - rtw89_enter_lps(rtwdev, rtwvif); + rtw89_enter_lps(rtwdev, rtwvif, true); } static void rtw89_enter_lps_track(struct rtw89_dev *rtwdev) @@ -2493,6 +2766,7 @@ int rtw89_core_sta_add(struct rtw89_dev *rtwdev, rtwsta->rtwvif = rtwvif; rtwsta->prev_rssi = 0; INIT_LIST_HEAD(&rtwsta->ba_cam_list); + skb_queue_head_init(&rtwsta->roc_queue); for (i = 0; i < ARRAY_SIZE(sta->txq); i++) rtw89_core_txq_init(rtwdev, sta->txq[i]); @@ -2539,6 +2813,9 @@ int rtw89_core_sta_disassoc(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif = (struct rtw89_vif *)vif->drv_priv; struct rtw89_sta *rtwsta = (struct rtw89_sta *)sta->drv_priv; + if (vif->type == NL80211_IFTYPE_STATION) + rtw89_fw_h2c_set_bcn_fltr_cfg(rtwdev, vif, false); + rtwdev->total_sta_assoc--; if (sta->tdls) rtwvif->tdls_peer--; @@ -2559,6 +2836,8 @@ int rtw89_core_sta_disconnect(struct rtw89_dev *rtwdev, rtw89_mac_bf_disassoc(rtwdev, vif, sta); rtw89_core_free_sta_pending_ba(rtwdev, sta); rtw89_core_free_sta_pending_forbid_ba(rtwdev, sta); + rtw89_core_free_sta_pending_roc_tx(rtwdev, sta); + if (vif->type == NL80211_IFTYPE_AP || sta->tdls) rtw89_cam_deinit_addr_cam(rtwdev, &rtwsta->addr_cam); if (sta->tdls) @@ -3180,7 +3459,6 @@ void rtw89_core_stop(struct rtw89_dev *rtwdev) int rtw89_core_init(struct rtw89_dev *rtwdev) { struct rtw89_btc *btc = &rtwdev->btc; - int ret; u8 band; INIT_LIST_HEAD(&rtwdev->ba_list); @@ -3214,6 +3492,8 @@ int rtw89_core_init(struct rtw89_dev *rtwdev) INIT_WORK(&rtwdev->c2h_work, rtw89_fw_c2h_work); INIT_WORK(&rtwdev->ips_work, rtw89_ips_work); + INIT_WORK(&rtwdev->load_firmware_work, rtw89_load_firmware_work); + skb_queue_head_init(&rtwdev->c2h_queue); rtw89_core_ppdu_sts_init(rtwdev); rtw89_traffic_stats_init(rtwdev, &rtwdev->stats); @@ -3225,12 +3505,10 @@ int rtw89_core_init(struct rtw89_dev *rtwdev) INIT_WORK(&btc->dhcp_notify_work, rtw89_btc_ntfy_dhcp_packet_work); INIT_WORK(&btc->icmp_notify_work, rtw89_btc_ntfy_icmp_packet_work); - ret = rtw89_load_firmware(rtwdev); - if (ret) { - rtw89_warn(rtwdev, "no firmware loaded\n"); - destroy_workqueue(rtwdev->txq_wq); - return ret; - } + init_completion(&rtwdev->fw.req.completion); + + schedule_work(&rtwdev->load_firmware_work); + rtw89_ser_init(rtwdev); rtw89_entity_init(rtwdev); @@ -3257,8 +3535,8 @@ void rtw89_core_scan_start(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif, rtwdev->scanning = true; rtw89_leave_lps(rtwdev); - if (hw_scan && (rtwdev->hw->conf.flags & IEEE80211_CONF_IDLE)) - rtw89_leave_ips(rtwdev); + if (hw_scan) + rtw89_leave_ips_by_hwflags(rtwdev); ether_addr_copy(rtwvif->mac_addr, mac_addr); rtw89_btc_ntfy_scan_start(rtwdev, RTW89_PHY_0, chan->band_type); @@ -3293,6 +3571,8 @@ void rtw89_core_scan_complete(struct rtw89_dev *rtwdev, static void rtw89_read_chip_ver(struct rtw89_dev *rtwdev) { const struct rtw89_chip_info *chip = rtwdev->chip; + int ret; + u8 val; u8 cv; cv = rtw89_read32_mask(rtwdev, R_AX_SYS_CFG1, B_AX_CHIP_VER_MASK); @@ -3304,6 +3584,14 @@ static void rtw89_read_chip_ver(struct rtw89_dev *rtwdev) } rtwdev->hal.cv = cv; + + if (chip->chip_id == RTL8852B || chip->chip_id == RTL8851B) { + ret = rtw89_mac_read_xtal_si(rtwdev, XTAL_SI_CV, &val); + if (!ret) + return; + + rtwdev->hal.acv = u8_get_bits(val, XTAL_SI_ACV_MASK); + } } static void rtw89_core_setup_phycap(struct rtw89_dev *rtwdev) @@ -3315,6 +3603,28 @@ static void rtw89_core_setup_phycap(struct rtw89_dev *rtwdev) rtwdev->chip->chip_id == RTL8852A && rtwdev->hal.cv <= CHIP_CBV; } +static void rtw89_core_setup_rfe_parms(struct rtw89_dev *rtwdev) +{ + const struct rtw89_chip_info *chip = rtwdev->chip; + const struct rtw89_rfe_parms_conf *conf = chip->rfe_parms_conf; + struct rtw89_efuse *efuse = &rtwdev->efuse; + u8 rfe_type = efuse->rfe_type; + + if (!conf) + goto out; + + while (conf->rfe_parms) { + if (rfe_type == conf->rfe_type) { + rtwdev->rfe_parms = conf->rfe_parms; + return; + } + conf++; + } + +out: + rtwdev->rfe_parms = chip->dflt_parms; +} + static int rtw89_chip_efuse_info_setup(struct rtw89_dev *rtwdev) { int ret; @@ -3336,6 +3646,7 @@ static int rtw89_chip_efuse_info_setup(struct rtw89_dev *rtwdev) return ret; rtw89_core_setup_phycap(rtwdev); + rtw89_core_setup_rfe_parms(rtwdev); rtw89_mac_pwr_off(rtwdev); @@ -3415,6 +3726,8 @@ static int rtw89_core_register_hw(struct rtw89_dev *rtwdev) ieee80211_hw_set(hw, SINGLE_SCAN_ON_ALL_BANDS); ieee80211_hw_set(hw, SUPPORTS_MULTI_BSSID); ieee80211_hw_set(hw, WANT_MONITOR_VIF); + if (RTW89_CHK_FW_FEATURE(BEACON_FILTER, &rtwdev->fw)) + ieee80211_hw_set(hw, CONNECTION_MONITOR); hw->wiphy->interface_modes = BIT(NL80211_IFTYPE_STATION) | BIT(NL80211_IFTYPE_AP) | @@ -3440,6 +3753,7 @@ static int rtw89_core_register_hw(struct rtw89_dev *rtwdev) hw->wiphy->tid_config_support.peer |= BIT(NL80211_TID_CONFIG_ATTR_AMPDU_CTRL); hw->wiphy->tid_config_support.vif |= BIT(NL80211_TID_CONFIG_ATTR_AMSDU_CTRL); hw->wiphy->tid_config_support.peer |= BIT(NL80211_TID_CONFIG_ATTR_AMSDU_CTRL); + hw->wiphy->max_remain_on_channel_duration = 1000; wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_CAN_REPLACE_PTK0); @@ -3508,22 +3822,24 @@ struct rtw89_dev *rtw89_alloc_ieee80211_hw(struct device *device, u32 bus_data_size, const struct rtw89_chip_info *chip) { + struct rtw89_fw_info early_fw = {}; const struct firmware *firmware; struct ieee80211_hw *hw; struct rtw89_dev *rtwdev; struct ieee80211_ops *ops; u32 driver_data_size; - u32 early_feat_map = 0; + int fw_format = -1; bool no_chanctx; - firmware = rtw89_early_fw_feature_recognize(device, chip, &early_feat_map); + firmware = rtw89_early_fw_feature_recognize(device, chip, &early_fw, &fw_format); ops = kmemdup(&rtw89_ops, sizeof(rtw89_ops), GFP_KERNEL); if (!ops) goto err; no_chanctx = chip->support_chanctx_num == 0 || - !(early_feat_map & BIT(RTW89_FW_FEATURE_SCAN_OFFLOAD)); + !RTW89_CHK_FW_FEATURE(SCAN_OFFLOAD, &early_fw) || + !RTW89_CHK_FW_FEATURE(BEACON_FILTER, &early_fw); if (no_chanctx) { ops->add_chanctx = NULL; @@ -3531,6 +3847,8 @@ struct rtw89_dev *rtw89_alloc_ieee80211_hw(struct device *device, ops->change_chanctx = NULL; ops->assign_vif_chanctx = NULL; ops->unassign_vif_chanctx = NULL; + ops->remain_on_channel = NULL; + ops->cancel_remain_on_channel = NULL; } driver_data_size = sizeof(struct rtw89_dev) + bus_data_size; @@ -3538,12 +3856,16 @@ struct rtw89_dev *rtw89_alloc_ieee80211_hw(struct device *device, if (!hw) goto err; + hw->wiphy->iface_combinations = rtw89_iface_combs; + hw->wiphy->n_iface_combinations = ARRAY_SIZE(rtw89_iface_combs); + rtwdev = hw->priv; rtwdev->hw = hw; rtwdev->dev = device; rtwdev->ops = ops; rtwdev->chip = chip; - rtwdev->fw.firmware = firmware; + rtwdev->fw.req.firmware = firmware; + rtwdev->fw.fw_format = fw_format; rtw89_debug(rtwdev, RTW89_DBG_FW, "probe driver %s chanctx\n", no_chanctx ? "without" : "with"); @@ -3560,7 +3882,7 @@ EXPORT_SYMBOL(rtw89_alloc_ieee80211_hw); void rtw89_free_ieee80211_hw(struct rtw89_dev *rtwdev) { kfree(rtwdev->ops); - release_firmware(rtwdev->fw.firmware); + release_firmware(rtwdev->fw.req.firmware); ieee80211_free_hw(rtwdev->hw); } EXPORT_SYMBOL(rtw89_free_ieee80211_hw); diff --git a/drivers/net/wireless/realtek/rtw89/core.h b/drivers/net/wireless/realtek/rtw89/core.h index 40fb18b613d9..6df386a38fb4 100644 --- a/drivers/net/wireless/realtek/rtw89/core.h +++ b/drivers/net/wireless/realtek/rtw89/core.h @@ -108,6 +108,7 @@ enum rtw89_core_chip_id { RTL8852A, RTL8852B, RTL8852C, + RTL8851B, }; enum rtw89_cv { @@ -569,6 +570,7 @@ enum rtw89_sub_entity_idx { RTW89_SUB_ENTITY_0 = 0, NUM_OF_RTW89_SUB_ENTITY, + RTW89_SUB_ENTITY_IDLE = NUM_OF_RTW89_SUB_ENTITY, }; enum rtw89_rf_path { @@ -957,6 +959,8 @@ struct rtw89_btc_ant_info { u8 single_pos: 1;/* Single antenna at S0 or S1 */ u8 diversity: 1; + u8 btg_pos: 2; + u8 stream_cnt: 4; }; enum rtw89_tfc_dir { @@ -1413,8 +1417,9 @@ struct rtw89_btc_module { u8 bt_solo: 1; u8 bt_pos: 1; u8 switch_type: 1; + u8 wa_type: 3; - u8 rsvd; + u8 kt_ver_adie; }; #define RTW89_BTC_DM_MAXSTEP 30 @@ -2597,6 +2602,7 @@ struct rtw89_sta { struct rtw89_addr_cam_entry addr_cam; /* AP mode or TDLS peer only */ struct rtw89_bssid_cam_entry bssid_cam; /* TDLS peer only */ struct list_head ba_cam_list; + struct sk_buff_head roc_queue; bool use_cfg_mask; struct cfg80211_bitrate_mask mask; @@ -2623,11 +2629,39 @@ struct rtw89_phy_rate_pattern { bool enable; }; +struct rtw89_tx_wait_info { + struct rcu_head rcu_head; + struct completion completion; + bool tx_done; +}; + +struct rtw89_tx_skb_data { + struct rtw89_tx_wait_info __rcu *wait; + u8 hci_priv[]; +}; + +#define RTW89_ROC_IDLE_TIMEOUT 500 +#define RTW89_ROC_TX_TIMEOUT 30 +enum rtw89_roc_state { + RTW89_ROC_IDLE, + RTW89_ROC_NORMAL, + RTW89_ROC_MGMT, +}; + +struct rtw89_roc { + struct ieee80211_channel chan; + struct delayed_work roc_work; + enum ieee80211_roc_type type; + enum rtw89_roc_state state; + int duration; +}; + #define RTW89_P2P_MAX_NOA_NUM 2 struct rtw89_vif { struct list_head list; struct rtw89_dev *rtwdev; + struct rtw89_roc roc; enum rtw89_sub_entity_idx sub_entity_idx; u8 mac_id; @@ -2643,6 +2677,7 @@ struct rtw89_vif { u8 bcn_hit_cond; u8 hit_rule; u8 last_noa_nr; + bool offchan; bool trigger; bool lsig_txop; u8 tgt_ind; @@ -2959,6 +2994,41 @@ struct rtw89_txpwr_table { const struct rtw89_txpwr_table *tbl); }; +struct rtw89_txpwr_rule_2ghz { + const s8 (*lmt)[RTW89_2G_BW_NUM][RTW89_NTX_NUM] + [RTW89_RS_LMT_NUM][RTW89_BF_NUM] + [RTW89_REGD_NUM][RTW89_2G_CH_NUM]; + const s8 (*lmt_ru)[RTW89_RU_NUM][RTW89_NTX_NUM] + [RTW89_REGD_NUM][RTW89_2G_CH_NUM]; +}; + +struct rtw89_txpwr_rule_5ghz { + const s8 (*lmt)[RTW89_5G_BW_NUM][RTW89_NTX_NUM] + [RTW89_RS_LMT_NUM][RTW89_BF_NUM] + [RTW89_REGD_NUM][RTW89_5G_CH_NUM]; + const s8 (*lmt_ru)[RTW89_RU_NUM][RTW89_NTX_NUM] + [RTW89_REGD_NUM][RTW89_5G_CH_NUM]; +}; + +struct rtw89_txpwr_rule_6ghz { + const s8 (*lmt)[RTW89_6G_BW_NUM][RTW89_NTX_NUM] + [RTW89_RS_LMT_NUM][RTW89_BF_NUM] + [RTW89_REGD_NUM][RTW89_6G_CH_NUM]; + const s8 (*lmt_ru)[RTW89_RU_NUM][RTW89_NTX_NUM] + [RTW89_REGD_NUM][RTW89_6G_CH_NUM]; +}; + +struct rtw89_rfe_parms { + struct rtw89_txpwr_rule_2ghz rule_2ghz; + struct rtw89_txpwr_rule_5ghz rule_5ghz; + struct rtw89_txpwr_rule_6ghz rule_6ghz; +}; + +struct rtw89_rfe_parms_conf { + const struct rtw89_rfe_parms *rfe_parms; + u8 rfe_type; +}; + struct rtw89_page_regs { u32 hci_fc_ctrl; u32 ch_page_ctrl; @@ -3049,7 +3119,8 @@ struct rtw89_phy_ul_tb_info { struct rtw89_chip_info { enum rtw89_core_chip_id chip_id; const struct rtw89_chip_ops *ops; - const char *fw_name; + const char *fw_basename; + u8 fw_format_max; bool try_ce_fw; u32 fifo_size; u32 dle_scc_rsvd_size; @@ -3095,21 +3166,10 @@ struct rtw89_chip_info { const struct rtw89_phy_dig_gain_table *dig_table; const struct rtw89_dig_regs *dig_regs; const struct rtw89_phy_tssi_dbw_table *tssi_dbw_table; - const s8 (*txpwr_lmt_2g)[RTW89_2G_BW_NUM][RTW89_NTX_NUM] - [RTW89_RS_LMT_NUM][RTW89_BF_NUM] - [RTW89_REGD_NUM][RTW89_2G_CH_NUM]; - const s8 (*txpwr_lmt_5g)[RTW89_5G_BW_NUM][RTW89_NTX_NUM] - [RTW89_RS_LMT_NUM][RTW89_BF_NUM] - [RTW89_REGD_NUM][RTW89_5G_CH_NUM]; - const s8 (*txpwr_lmt_6g)[RTW89_6G_BW_NUM][RTW89_NTX_NUM] - [RTW89_RS_LMT_NUM][RTW89_BF_NUM] - [RTW89_REGD_NUM][RTW89_6G_CH_NUM]; - const s8 (*txpwr_lmt_ru_2g)[RTW89_RU_NUM][RTW89_NTX_NUM] - [RTW89_REGD_NUM][RTW89_2G_CH_NUM]; - const s8 (*txpwr_lmt_ru_5g)[RTW89_RU_NUM][RTW89_NTX_NUM] - [RTW89_REGD_NUM][RTW89_5G_CH_NUM]; - const s8 (*txpwr_lmt_ru_6g)[RTW89_RU_NUM][RTW89_NTX_NUM] - [RTW89_REGD_NUM][RTW89_6G_CH_NUM]; + + /* NULL if no rfe-specific, or a null-terminated array by rfe_parms */ + const struct rtw89_rfe_parms_conf *rfe_parms_conf; + const struct rtw89_rfe_parms *dflt_parms; u8 txpwr_factor_rf; u8 txpwr_factor_mac; @@ -3146,6 +3206,7 @@ struct rtw89_chip_info { struct rtw89_reg_def c2h_counter_reg; const struct rtw89_page_regs *page_regs; bool cfo_src_fd; + bool cfo_hw_comp; const struct rtw89_reg_def *dcfo_comp; u8 dcfo_comp_sft; const struct rtw89_imr_info *imr_info; @@ -3231,6 +3292,7 @@ enum rtw89_fw_feature { RTW89_FW_FEATURE_NO_PACKET_DROP, RTW89_FW_FEATURE_NO_DEEP_PS, RTW89_FW_FEATURE_NO_LPS_PG, + RTW89_FW_FEATURE_BEACON_FILTER, }; struct rtw89_fw_suit { @@ -3265,10 +3327,14 @@ struct rtw89_fw_suit { GET_FW_HDR_SUBVERSION(fw_hdr), \ GET_FW_HDR_SUBINDEX(fw_hdr)) -struct rtw89_fw_info { +struct rtw89_fw_req_info { const struct firmware *firmware; - struct rtw89_dev *rtwdev; struct completion completion; +}; + +struct rtw89_fw_info { + struct rtw89_fw_req_info req; + int fw_format; u8 h2c_seq; u8 rec_seq; u8 h2c_counter; @@ -3350,6 +3416,7 @@ struct rtw89_sub_entity { struct rtw89_hal { u32 rx_fltr; u8 cv; + u8 acv; u32 sw_amsdu_max_size; u32 antenna_tx; u32 antenna_rx; @@ -3358,9 +3425,11 @@ struct rtw89_hal { bool tx_path_diversity; bool support_cckpd; bool support_igi; + atomic_t roc_entity_idx; DECLARE_BITMAP(entity_map, NUM_OF_RTW89_SUB_ENTITY); struct rtw89_sub_entity sub[NUM_OF_RTW89_SUB_ENTITY]; + struct cfg80211_chan_def roc_chandef; bool entity_active; enum rtw89_entity_mode entity_mode; @@ -3621,6 +3690,8 @@ struct rtw89_cfo_tracking_info { s32 cfo_avg_pre; s32 cfo_avg[CFO_TRACK_MAX_USER]; s32 pre_cfo_avg[CFO_TRACK_MAX_USER]; + s32 dcfo_avg; + s32 dcfo_avg_pre; u32 packet_count; u32 packet_count_pre; s32 residual_cfo_acc; @@ -3865,10 +3936,7 @@ struct rtw89_early_h2c { struct rtw89_hw_scan_info { struct ieee80211_vif *scanning_vif; struct list_head pkt_list[NUM_NL80211_BANDS]; - u8 op_pri_ch; - u8 op_chan; - u8 op_bw; - u8 op_band; + struct rtw89_chan op_chan; u32 last_chan_idx; }; @@ -3953,6 +4021,7 @@ struct rtw89_dev { struct rtw89_hw_scan_info scan_info; const struct rtw89_chip_info *chip; const struct rtw89_pci_info *pci_info; + const struct rtw89_rfe_parms *rfe_parms; struct rtw89_hal hal; struct rtw89_mcc_info mcc; struct rtw89_mac_info mac; @@ -3984,6 +4053,7 @@ struct rtw89_dev { struct sk_buff_head c2h_queue; struct work_struct c2h_work; struct work_struct ips_work; + struct work_struct load_firmware_work; struct list_head early_h2c_list; @@ -4023,6 +4093,7 @@ struct rtw89_dev { struct delayed_work coex_rfk_chk_work; struct delayed_work cfo_track_work; struct delayed_work forbid_ba_work; + struct delayed_work roc_work; struct rtw89_ppdu_sts_info ppdu_sts; u8 total_sta_assoc; bool scanning; @@ -4178,6 +4249,14 @@ static inline void rtw89_hci_clear(struct rtw89_dev *rtwdev, struct pci_dev *pde rtwdev->hci.ops->clear(rtwdev, pdev); } +static inline +struct rtw89_tx_skb_data *RTW89_TX_SKB_CB(struct sk_buff *skb) +{ + struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb); + + return (struct rtw89_tx_skb_data *)info->status.status_driver_data; +} + static inline u8 rtw89_read8(struct rtw89_dev *rtwdev, u32 addr) { return rtwdev->hci.ops->read8(rtwdev, addr); @@ -4530,6 +4609,10 @@ const struct cfg80211_chan_def *rtw89_chandef_get(struct rtw89_dev *rtwdev, enum rtw89_sub_entity_idx idx) { struct rtw89_hal *hal = &rtwdev->hal; + enum rtw89_sub_entity_idx roc_idx = atomic_read(&hal->roc_entity_idx); + + if (roc_idx == idx) + return &hal->roc_chandef; return &hal->sub[idx].chandef; } @@ -4821,11 +4904,32 @@ static inline struct sk_buff *rtw89_alloc_skb_for_rx(struct rtw89_dev *rtwdev, return dev_alloc_skb(length); } +static inline void rtw89_core_tx_wait_complete(struct rtw89_dev *rtwdev, + struct rtw89_tx_skb_data *skb_data, + bool tx_done) +{ + struct rtw89_tx_wait_info *wait; + + rcu_read_lock(); + + wait = rcu_dereference(skb_data->wait); + if (!wait) + goto out; + + wait->tx_done = tx_done; + complete(&wait->completion); + +out: + rcu_read_unlock(); +} + int rtw89_core_tx_write(struct rtw89_dev *rtwdev, struct ieee80211_vif *vif, struct ieee80211_sta *sta, struct sk_buff *skb, int *qsel); int rtw89_h2c_tx(struct rtw89_dev *rtwdev, struct sk_buff *skb, bool fwdl); void rtw89_core_tx_kick_off(struct rtw89_dev *rtwdev, u8 qsel); +int rtw89_core_tx_kick_off_and_wait(struct rtw89_dev *rtwdev, struct sk_buff *skb, + int qsel, unsigned int timeout); void rtw89_core_fill_txdesc(struct rtw89_dev *rtwdev, struct rtw89_tx_desc_info *desc_info, void *txdesc); @@ -4874,6 +4978,8 @@ void rtw89_free_ieee80211_hw(struct rtw89_dev *rtwdev); void rtw89_core_set_chip_txpwr(struct rtw89_dev *rtwdev); void rtw89_get_default_chandef(struct cfg80211_chan_def *chandef); void rtw89_set_channel(struct rtw89_dev *rtwdev); +void rtw89_get_channel(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif, + struct rtw89_chan *chan); u8 rtw89_core_acquire_bit_map(unsigned long *addr, unsigned long size); void rtw89_core_release_bit_map(unsigned long *addr, u8 bit); void rtw89_core_release_all_bits_map(unsigned long *addr, unsigned int nbits); @@ -4895,6 +5001,9 @@ void rtw89_complete_cond(struct rtw89_wait_info *wait, unsigned int cond, int rtw89_core_start(struct rtw89_dev *rtwdev); void rtw89_core_stop(struct rtw89_dev *rtwdev); void rtw89_core_update_beacon_work(struct work_struct *work); +void rtw89_roc_work(struct work_struct *work); +void rtw89_roc_start(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif); +void rtw89_roc_end(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif); void rtw89_core_scan_start(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif, const u8 *mac_addr, bool hw_scan); void rtw89_core_scan_complete(struct rtw89_dev *rtwdev, diff --git a/drivers/net/wireless/realtek/rtw89/debug.c b/drivers/net/wireless/realtek/rtw89/debug.c index 0e0e1483c099..1e5b7a998716 100644 --- a/drivers/net/wireless/realtek/rtw89/debug.c +++ b/drivers/net/wireless/realtek/rtw89/debug.c @@ -3069,18 +3069,13 @@ static int rtw89_dbg_trigger_ctrl_error(struct rtw89_dev *rtwdev) { struct rtw89_cpuio_ctrl ctrl_para = {0}; u16 pkt_id; + int ret; rtw89_leave_ps_mode(rtwdev); - pkt_id = rtw89_mac_dle_buf_req(rtwdev, 0x20, true); - switch (pkt_id) { - case 0xffff: - return -ETIMEDOUT; - case 0xfff: - return -ENOMEM; - default: - break; - } + ret = rtw89_mac_dle_buf_req(rtwdev, 0x20, true, &pkt_id); + if (ret) + return ret; /* intentionally, enqueue two pkt, but has only one pkt id */ ctrl_para.cmd_type = CPUIO_OP_CMD_ENQ_TO_HEAD; diff --git a/drivers/net/wireless/realtek/rtw89/fw.c b/drivers/net/wireless/realtek/rtw89/fw.c index 5fa6863d36b3..b9b675bf9d05 100644 --- a/drivers/net/wireless/realtek/rtw89/fw.c +++ b/drivers/net/wireless/realtek/rtw89/fw.c @@ -155,8 +155,9 @@ int rtw89_mfw_recognize(struct rtw89_dev *rtwdev, enum rtw89_fw_type type, struct rtw89_fw_suit *fw_suit, bool nowarn) { struct rtw89_fw_info *fw_info = &rtwdev->fw; - const u8 *mfw = fw_info->firmware->data; - u32 mfw_len = fw_info->firmware->size; + const struct firmware *firmware = fw_info->req.firmware; + const u8 *mfw = firmware->data; + u32 mfw_len = firmware->size; const struct rtw89_mfw_hdr *mfw_hdr = (const struct rtw89_mfw_hdr *)mfw; const struct rtw89_mfw_info *mfw_info; int i; @@ -266,40 +267,51 @@ static const struct __fw_feat_cfg fw_feat_tbl[] = { __CFG_FW_FEAT(RTL8852C, ge, 0, 27, 34, 0, TX_WAKE), __CFG_FW_FEAT(RTL8852C, ge, 0, 27, 36, 0, SCAN_OFFLOAD), __CFG_FW_FEAT(RTL8852C, ge, 0, 27, 40, 0, CRASH_TRIGGER), + __CFG_FW_FEAT(RTL8852C, ge, 0, 27, 56, 10, BEACON_FILTER), }; +static void rtw89_fw_iterate_feature_cfg(struct rtw89_fw_info *fw, + const struct rtw89_chip_info *chip, + u32 ver_code) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(fw_feat_tbl); i++) { + const struct __fw_feat_cfg *ent = &fw_feat_tbl[i]; + + if (chip->chip_id != ent->chip_id) + continue; + + if (ent->cond(ver_code, ent->ver_code)) + RTW89_SET_FW_FEATURE(ent->feature, fw); + } +} + static void rtw89_fw_recognize_features(struct rtw89_dev *rtwdev) { const struct rtw89_chip_info *chip = rtwdev->chip; - const struct __fw_feat_cfg *ent; const struct rtw89_fw_suit *fw_suit; u32 suit_ver_code; - int i; fw_suit = rtw89_fw_suit_get(rtwdev, RTW89_FW_NORMAL); suit_ver_code = RTW89_FW_SUIT_VER_CODE(fw_suit); - for (i = 0; i < ARRAY_SIZE(fw_feat_tbl); i++) { - ent = &fw_feat_tbl[i]; - if (chip->chip_id != ent->chip_id) - continue; - - if (ent->cond(suit_ver_code, ent->ver_code)) - RTW89_SET_FW_FEATURE(ent->feature, &rtwdev->fw); - } + rtw89_fw_iterate_feature_cfg(&rtwdev->fw, chip, suit_ver_code); } const struct firmware * rtw89_early_fw_feature_recognize(struct device *device, const struct rtw89_chip_info *chip, - u32 *early_feat_map) + struct rtw89_fw_info *early_fw, + int *used_fw_format) { union rtw89_compat_fw_hdr buf = {}; const struct firmware *firmware; bool full_req = false; + char fw_name[64]; + int fw_format; u32 ver_code; int ret; - int i; /* If SECURITY_LOADPIN_ENFORCE is enabled, reading partial files will * be denied (-EPERM). Then, we don't get right firmware things as @@ -308,12 +320,22 @@ rtw89_early_fw_feature_recognize(struct device *device, if (IS_ENABLED(CONFIG_SECURITY_LOADPIN_ENFORCE)) full_req = true; - if (full_req) - ret = request_firmware(&firmware, chip->fw_name, device); - else - ret = request_partial_firmware_into_buf(&firmware, chip->fw_name, - device, &buf, sizeof(buf), - 0); + for (fw_format = chip->fw_format_max; fw_format >= 0; fw_format--) { + rtw89_fw_get_filename(fw_name, sizeof(fw_name), + chip->fw_basename, fw_format); + + if (full_req) + ret = request_firmware(&firmware, fw_name, device); + else + ret = request_partial_firmware_into_buf(&firmware, fw_name, + device, &buf, sizeof(buf), + 0); + if (!ret) { + dev_info(device, "loaded firmware %s\n", fw_name); + *used_fw_format = fw_format; + break; + } + } if (ret) { dev_err(device, "failed to early request firmware: %d\n", ret); @@ -328,15 +350,7 @@ rtw89_early_fw_feature_recognize(struct device *device, if (!ver_code) goto out; - for (i = 0; i < ARRAY_SIZE(fw_feat_tbl); i++) { - const struct __fw_feat_cfg *ent = &fw_feat_tbl[i]; - - if (chip->chip_id != ent->chip_id) - continue; - - if (ent->cond(ver_code, ent->ver_code)) - *early_feat_map |= BIT(ent->feature); - } + rtw89_fw_iterate_feature_cfg(early_fw, chip, ver_code); out: if (full_req) @@ -631,67 +645,62 @@ int rtw89_wait_firmware_completion(struct rtw89_dev *rtwdev) { struct rtw89_fw_info *fw = &rtwdev->fw; - wait_for_completion(&fw->completion); - if (!fw->firmware) + wait_for_completion(&fw->req.completion); + if (!fw->req.firmware) return -EINVAL; return 0; } -static void rtw89_load_firmware_cb(const struct firmware *firmware, void *context) +static int rtw89_load_firmware_req(struct rtw89_dev *rtwdev, + struct rtw89_fw_req_info *req, + const char *fw_name, bool nowarn) { - struct rtw89_fw_info *fw = context; - struct rtw89_dev *rtwdev = fw->rtwdev; - - if (!firmware || !firmware->data) { - rtw89_err(rtwdev, "failed to request firmware\n"); - complete_all(&fw->completion); - return; - } - - fw->firmware = firmware; - complete_all(&fw->completion); -} - -int rtw89_load_firmware(struct rtw89_dev *rtwdev) -{ - struct rtw89_fw_info *fw = &rtwdev->fw; - const char *fw_name = rtwdev->chip->fw_name; int ret; - fw->rtwdev = rtwdev; - init_completion(&fw->completion); - - if (fw->firmware) { + if (req->firmware) { rtw89_debug(rtwdev, RTW89_DBG_FW, "full firmware has been early requested\n"); - complete_all(&fw->completion); + complete_all(&req->completion); return 0; } - ret = request_firmware_nowait(THIS_MODULE, true, fw_name, rtwdev->dev, - GFP_KERNEL, fw, rtw89_load_firmware_cb); - if (ret) { - rtw89_err(rtwdev, "failed to async firmware request\n"); - return ret; - } + if (nowarn) + ret = firmware_request_nowarn(&req->firmware, fw_name, rtwdev->dev); + else + ret = request_firmware(&req->firmware, fw_name, rtwdev->dev); - return 0; + complete_all(&req->completion); + + return ret; +} + +void rtw89_load_firmware_work(struct work_struct *work) +{ + struct rtw89_dev *rtwdev = + container_of(work, struct rtw89_dev, load_firmware_work); + const struct rtw89_chip_info *chip = rtwdev->chip; + char fw_name[64]; + + rtw89_fw_get_filename(fw_name, sizeof(fw_name), + chip->fw_basename, rtwdev->fw.fw_format); + + rtw89_load_firmware_req(rtwdev, &rtwdev->fw.req, fw_name, false); } void rtw89_unload_firmware(struct rtw89_dev *rtwdev) { struct rtw89_fw_info *fw = &rtwdev->fw; - rtw89_wait_firmware_completion(rtwdev); + cancel_work_sync(&rtwdev->load_firmware_work); - if (fw->firmware) { - release_firmware(fw->firmware); + if (fw->req.firmware) { + release_firmware(fw->req.firmware); /* assign NULL back in case rtw89_free_ieee80211_hw() * try to release the same one again. */ - fw->firmware = NULL; + fw->req.firmware = NULL; } } @@ -1152,9 +1161,18 @@ fail: static void __rtw89_fw_h2c_set_tx_path(struct rtw89_dev *rtwdev, struct sk_buff *skb) { + const struct rtw89_chip_info *chip = rtwdev->chip; struct rtw89_hal *hal = &rtwdev->hal; - u8 ntx_path = hal->antenna_tx ? hal->antenna_tx : RF_B; - u8 map_b = hal->antenna_tx == RF_AB ? 1 : 0; + u8 ntx_path; + u8 map_b; + + if (chip->rf_path_num == 1) { + ntx_path = RF_A; + map_b = 0; + } else { + ntx_path = hal->antenna_tx ? hal->antenna_tx : RF_B; + map_b = hal->antenna_tx == RF_AB ? 1 : 0; + } SET_CMC_TBL_NTX_PATH_EN(skb->data, ntx_path); SET_CMC_TBL_PATH_MAP_A(skb->data, 0); @@ -1737,6 +1755,147 @@ fail: return ret; } +int rtw89_fw_h2c_set_bcn_fltr_cfg(struct rtw89_dev *rtwdev, + struct ieee80211_vif *vif, + bool connect) +{ + struct rtw89_vif *rtwvif = vif_to_rtwvif_safe(vif); + struct ieee80211_bss_conf *bss_conf = vif ? &vif->bss_conf : NULL; + struct rtw89_h2c_bcnfltr *h2c; + u32 len = sizeof(*h2c); + struct sk_buff *skb; + int ret; + + if (!RTW89_CHK_FW_FEATURE(BEACON_FILTER, &rtwdev->fw)) + return -EINVAL; + + if (!rtwvif || !bss_conf || rtwvif->net_type != RTW89_NET_TYPE_INFRA) + return -EINVAL; + + skb = rtw89_fw_h2c_alloc_skb_with_hdr(rtwdev, len); + if (!skb) { + rtw89_err(rtwdev, "failed to alloc skb for h2c bcn filter\n"); + return -ENOMEM; + } + + skb_put(skb, len); + h2c = (struct rtw89_h2c_bcnfltr *)skb->data; + + h2c->w0 = le32_encode_bits(connect, RTW89_H2C_BCNFLTR_W0_MON_RSSI) | + le32_encode_bits(connect, RTW89_H2C_BCNFLTR_W0_MON_BCN) | + le32_encode_bits(connect, RTW89_H2C_BCNFLTR_W0_MON_EN) | + le32_encode_bits(RTW89_BCN_FLTR_OFFLOAD_MODE_DEFAULT, + RTW89_H2C_BCNFLTR_W0_MODE) | + le32_encode_bits(RTW89_BCN_LOSS_CNT, RTW89_H2C_BCNFLTR_W0_BCN_LOSS_CNT) | + le32_encode_bits(bss_conf->cqm_rssi_hyst, RTW89_H2C_BCNFLTR_W0_RSSI_HYST) | + le32_encode_bits(bss_conf->cqm_rssi_thold + MAX_RSSI, + RTW89_H2C_BCNFLTR_W0_RSSI_THRESHOLD) | + le32_encode_bits(rtwvif->mac_id, RTW89_H2C_BCNFLTR_W0_MAC_ID); + + rtw89_h2c_pkt_set_hdr(rtwdev, skb, FWCMD_TYPE_H2C, + H2C_CAT_MAC, H2C_CL_MAC_FW_OFLD, + H2C_FUNC_CFG_BCNFLTR, 0, 1, len); + + ret = rtw89_h2c_tx(rtwdev, skb, false); + if (ret) { + rtw89_err(rtwdev, "failed to send h2c\n"); + goto fail; + } + + return 0; +fail: + dev_kfree_skb_any(skb); + + return ret; +} + +int rtw89_fw_h2c_rssi_offload(struct rtw89_dev *rtwdev, + struct rtw89_rx_phy_ppdu *phy_ppdu) +{ + struct rtw89_h2c_ofld_rssi *h2c; + u32 len = sizeof(*h2c); + struct sk_buff *skb; + s8 rssi; + int ret; + + if (!RTW89_CHK_FW_FEATURE(BEACON_FILTER, &rtwdev->fw)) + return -EINVAL; + + if (!phy_ppdu) + return -EINVAL; + + skb = rtw89_fw_h2c_alloc_skb_with_hdr(rtwdev, len); + if (!skb) { + rtw89_err(rtwdev, "failed to alloc skb for h2c rssi\n"); + return -ENOMEM; + } + + rssi = phy_ppdu->rssi_avg >> RSSI_FACTOR; + skb_put(skb, len); + h2c = (struct rtw89_h2c_ofld_rssi *)skb->data; + + h2c->w0 = le32_encode_bits(phy_ppdu->mac_id, RTW89_H2C_OFLD_RSSI_W0_MACID) | + le32_encode_bits(1, RTW89_H2C_OFLD_RSSI_W0_NUM); + h2c->w1 = le32_encode_bits(rssi, RTW89_H2C_OFLD_RSSI_W1_VAL); + + rtw89_h2c_pkt_set_hdr(rtwdev, skb, FWCMD_TYPE_H2C, + H2C_CAT_MAC, H2C_CL_MAC_FW_OFLD, + H2C_FUNC_OFLD_RSSI, 0, 1, len); + + ret = rtw89_h2c_tx(rtwdev, skb, false); + if (ret) { + rtw89_err(rtwdev, "failed to send h2c\n"); + goto fail; + } + + return 0; +fail: + dev_kfree_skb_any(skb); + + return ret; +} + +int rtw89_fw_h2c_tp_offload(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif) +{ + struct rtw89_traffic_stats *stats = &rtwvif->stats; + struct rtw89_h2c_ofld *h2c; + u32 len = sizeof(*h2c); + struct sk_buff *skb; + int ret; + + if (rtwvif->net_type != RTW89_NET_TYPE_INFRA) + return -EINVAL; + + skb = rtw89_fw_h2c_alloc_skb_with_hdr(rtwdev, len); + if (!skb) { + rtw89_err(rtwdev, "failed to alloc skb for h2c tp\n"); + return -ENOMEM; + } + + skb_put(skb, len); + h2c = (struct rtw89_h2c_ofld *)skb->data; + + h2c->w0 = le32_encode_bits(rtwvif->mac_id, RTW89_H2C_OFLD_W0_MAC_ID) | + le32_encode_bits(stats->tx_throughput, RTW89_H2C_OFLD_W0_TX_TP) | + le32_encode_bits(stats->rx_throughput, RTW89_H2C_OFLD_W0_RX_TP); + + rtw89_h2c_pkt_set_hdr(rtwdev, skb, FWCMD_TYPE_H2C, + H2C_CAT_MAC, H2C_CL_MAC_FW_OFLD, + H2C_FUNC_OFLD_TP, 0, 1, len); + + ret = rtw89_h2c_tx(rtwdev, skb, false); + if (ret) { + rtw89_err(rtwdev, "failed to send h2c\n"); + goto fail; + } + + return 0; +fail: + dev_kfree_skb_any(skb); + + return ret; +} + #define H2C_RA_LEN 16 int rtw89_fw_h2c_ra(struct rtw89_dev *rtwdev, struct rtw89_ra_info *ra, bool csi) { @@ -1806,8 +1965,6 @@ fail: return ret; } -#define H2C_LEN_CXDRVHDR 2 -#define H2C_LEN_CXDRVINFO_INIT (12 + H2C_LEN_CXDRVHDR) int rtw89_fw_h2c_cxdrv_init(struct rtw89_dev *rtwdev) { struct rtw89_btc *btc = &rtwdev->btc; @@ -1815,44 +1972,52 @@ int rtw89_fw_h2c_cxdrv_init(struct rtw89_dev *rtwdev) struct rtw89_btc_init_info *init_info = &dm->init_info; struct rtw89_btc_module *module = &init_info->module; struct rtw89_btc_ant_info *ant = &module->ant; + struct rtw89_h2c_cxinit *h2c; + u32 len = sizeof(*h2c); struct sk_buff *skb; - u8 *cmd; int ret; - skb = rtw89_fw_h2c_alloc_skb_with_hdr(rtwdev, H2C_LEN_CXDRVINFO_INIT); + skb = rtw89_fw_h2c_alloc_skb_with_hdr(rtwdev, len); if (!skb) { rtw89_err(rtwdev, "failed to alloc skb for h2c cxdrv_init\n"); return -ENOMEM; } - skb_put(skb, H2C_LEN_CXDRVINFO_INIT); - cmd = skb->data; - - RTW89_SET_FWCMD_CXHDR_TYPE(cmd, CXDRVINFO_INIT); - RTW89_SET_FWCMD_CXHDR_LEN(cmd, H2C_LEN_CXDRVINFO_INIT - H2C_LEN_CXDRVHDR); - - RTW89_SET_FWCMD_CXINIT_ANT_TYPE(cmd, ant->type); - RTW89_SET_FWCMD_CXINIT_ANT_NUM(cmd, ant->num); - RTW89_SET_FWCMD_CXINIT_ANT_ISO(cmd, ant->isolation); - RTW89_SET_FWCMD_CXINIT_ANT_POS(cmd, ant->single_pos); - RTW89_SET_FWCMD_CXINIT_ANT_DIVERSITY(cmd, ant->diversity); - - RTW89_SET_FWCMD_CXINIT_MOD_RFE(cmd, module->rfe_type); - RTW89_SET_FWCMD_CXINIT_MOD_CV(cmd, module->cv); - RTW89_SET_FWCMD_CXINIT_MOD_BT_SOLO(cmd, module->bt_solo); - RTW89_SET_FWCMD_CXINIT_MOD_BT_POS(cmd, module->bt_pos); - RTW89_SET_FWCMD_CXINIT_MOD_SW_TYPE(cmd, module->switch_type); - - RTW89_SET_FWCMD_CXINIT_WL_GCH(cmd, init_info->wl_guard_ch); - RTW89_SET_FWCMD_CXINIT_WL_ONLY(cmd, init_info->wl_only); - RTW89_SET_FWCMD_CXINIT_WL_INITOK(cmd, init_info->wl_init_ok); - RTW89_SET_FWCMD_CXINIT_DBCC_EN(cmd, init_info->dbcc_en); - RTW89_SET_FWCMD_CXINIT_CX_OTHER(cmd, init_info->cx_other); - RTW89_SET_FWCMD_CXINIT_BT_ONLY(cmd, init_info->bt_only); + skb_put(skb, len); + h2c = (struct rtw89_h2c_cxinit *)skb->data; + + h2c->hdr.type = CXDRVINFO_INIT; + h2c->hdr.len = len - H2C_LEN_CXDRVHDR; + + h2c->ant_type = ant->type; + h2c->ant_num = ant->num; + h2c->ant_iso = ant->isolation; + h2c->ant_info = + u8_encode_bits(ant->single_pos, RTW89_H2C_CXINIT_ANT_INFO_POS) | + u8_encode_bits(ant->diversity, RTW89_H2C_CXINIT_ANT_INFO_DIVERSITY) | + u8_encode_bits(ant->btg_pos, RTW89_H2C_CXINIT_ANT_INFO_BTG_POS) | + u8_encode_bits(ant->stream_cnt, RTW89_H2C_CXINIT_ANT_INFO_STREAM_CNT); + + h2c->mod_rfe = module->rfe_type; + h2c->mod_cv = module->cv; + h2c->mod_info = + u8_encode_bits(module->bt_solo, RTW89_H2C_CXINIT_MOD_INFO_BT_SOLO) | + u8_encode_bits(module->bt_pos, RTW89_H2C_CXINIT_MOD_INFO_BT_POS) | + u8_encode_bits(module->switch_type, RTW89_H2C_CXINIT_MOD_INFO_SW_TYPE) | + u8_encode_bits(module->wa_type, RTW89_H2C_CXINIT_MOD_INFO_WA_TYPE); + h2c->mod_adie_kt = module->kt_ver_adie; + h2c->wl_gch = init_info->wl_guard_ch; + + h2c->info = + u8_encode_bits(init_info->wl_only, RTW89_H2C_CXINIT_INFO_WL_ONLY) | + u8_encode_bits(init_info->wl_init_ok, RTW89_H2C_CXINIT_INFO_WL_INITOK) | + u8_encode_bits(init_info->dbcc_en, RTW89_H2C_CXINIT_INFO_DBCC_EN) | + u8_encode_bits(init_info->cx_other, RTW89_H2C_CXINIT_INFO_CX_OTHER) | + u8_encode_bits(init_info->bt_only, RTW89_H2C_CXINIT_INFO_BT_ONLY); rtw89_h2c_pkt_set_hdr(rtwdev, skb, FWCMD_TYPE_H2C, H2C_CAT_OUTSRC, BTFC_SET, SET_DRV_INFO, 0, 0, - H2C_LEN_CXDRVINFO_INIT); + len); ret = rtw89_h2c_tx(rtwdev, skb, false); if (ret) { @@ -2422,46 +2587,51 @@ fail: return ret; } -#define H2C_LEN_SCAN_OFFLOAD 28 int rtw89_fw_h2c_scan_offload(struct rtw89_dev *rtwdev, struct rtw89_scan_option *option, struct rtw89_vif *rtwvif) { - struct rtw89_hw_scan_info *scan_info = &rtwdev->scan_info; + struct rtw89_chan *op = &rtwdev->scan_info.op_chan; + struct rtw89_h2c_scanofld *h2c; + u32 len = sizeof(*h2c); struct sk_buff *skb; - u8 *cmd; int ret; - skb = rtw89_fw_h2c_alloc_skb_with_hdr(rtwdev, H2C_LEN_SCAN_OFFLOAD); + skb = rtw89_fw_h2c_alloc_skb_with_hdr(rtwdev, len); if (!skb) { rtw89_err(rtwdev, "failed to alloc skb for h2c scan offload\n"); return -ENOMEM; } - skb_put(skb, H2C_LEN_SCAN_OFFLOAD); - cmd = skb->data; + skb_put(skb, len); + h2c = (struct rtw89_h2c_scanofld *)skb->data; + + h2c->w0 = le32_encode_bits(rtwvif->mac_id, RTW89_H2C_SCANOFLD_W0_MACID) | + le32_encode_bits(rtwvif->port, RTW89_H2C_SCANOFLD_W0_PORT_ID) | + le32_encode_bits(RTW89_PHY_0, RTW89_H2C_SCANOFLD_W0_BAND) | + le32_encode_bits(option->enable, RTW89_H2C_SCANOFLD_W0_OPERATION); + + h2c->w1 = le32_encode_bits(true, RTW89_H2C_SCANOFLD_W1_NOTIFY_END) | + le32_encode_bits(option->target_ch_mode, + RTW89_H2C_SCANOFLD_W1_TARGET_CH_MODE) | + le32_encode_bits(RTW89_SCAN_IMMEDIATE, + RTW89_H2C_SCANOFLD_W1_START_MODE) | + le32_encode_bits(RTW89_SCAN_ONCE, RTW89_H2C_SCANOFLD_W1_SCAN_TYPE); - RTW89_SET_FWCMD_SCANOFLD_MACID(cmd, rtwvif->mac_id); - RTW89_SET_FWCMD_SCANOFLD_PORT_ID(cmd, rtwvif->port); - RTW89_SET_FWCMD_SCANOFLD_BAND(cmd, RTW89_PHY_0); - RTW89_SET_FWCMD_SCANOFLD_OPERATION(cmd, option->enable); - RTW89_SET_FWCMD_SCANOFLD_NOTIFY_END(cmd, true); - RTW89_SET_FWCMD_SCANOFLD_TARGET_CH_MODE(cmd, option->target_ch_mode); - RTW89_SET_FWCMD_SCANOFLD_START_MODE(cmd, RTW89_SCAN_IMMEDIATE); - RTW89_SET_FWCMD_SCANOFLD_SCAN_TYPE(cmd, RTW89_SCAN_ONCE); if (option->target_ch_mode) { - RTW89_SET_FWCMD_SCANOFLD_TARGET_CH_BW(cmd, scan_info->op_bw); - RTW89_SET_FWCMD_SCANOFLD_TARGET_PRI_CH(cmd, - scan_info->op_pri_ch); - RTW89_SET_FWCMD_SCANOFLD_TARGET_CENTRAL_CH(cmd, - scan_info->op_chan); - RTW89_SET_FWCMD_SCANOFLD_TARGET_CH_BAND(cmd, - scan_info->op_band); + h2c->w1 |= le32_encode_bits(op->band_width, + RTW89_H2C_SCANOFLD_W1_TARGET_CH_BW) | + le32_encode_bits(op->primary_channel, + RTW89_H2C_SCANOFLD_W1_TARGET_PRI_CH) | + le32_encode_bits(op->channel, + RTW89_H2C_SCANOFLD_W1_TARGET_CENTRAL_CH); + h2c->w0 |= le32_encode_bits(op->band_type, + RTW89_H2C_SCANOFLD_W0_TARGET_CH_BAND); } rtw89_h2c_pkt_set_hdr(rtwdev, skb, FWCMD_TYPE_H2C, H2C_CAT_MAC, H2C_CL_MAC_FW_OFLD, H2C_FUNC_SCANOFLD, 1, 1, - H2C_LEN_SCAN_OFFLOAD); + len); ret = rtw89_h2c_tx(rtwdev, skb, false); if (ret) { @@ -3034,6 +3204,7 @@ static void rtw89_hw_scan_add_chan(struct rtw89_dev *rtwdev, int chan_type, struct ieee80211_vif *vif = rtwdev->scan_info.scanning_vif; struct rtw89_vif *rtwvif = (struct rtw89_vif *)vif->drv_priv; struct cfg80211_scan_request *req = rtwvif->scan_req; + struct rtw89_chan *op = &rtwdev->scan_info.op_chan; struct rtw89_pktofld_info *info; u8 band, probe_count = 0; int ret; @@ -3077,10 +3248,10 @@ static void rtw89_hw_scan_add_chan(struct rtw89_dev *rtwdev, int chan_type, switch (chan_type) { case RTW89_CHAN_OPERATE: - ch_info->central_ch = scan_info->op_chan; - ch_info->pri_ch = scan_info->op_pri_ch; - ch_info->ch_band = scan_info->op_band; - ch_info->bw = scan_info->op_bw; + ch_info->central_ch = op->channel; + ch_info->pri_ch = op->primary_channel; + ch_info->ch_band = op->band_type; + ch_info->bw = op->band_width; ch_info->tx_null = true; ch_info->num_pkt = 0; break; @@ -3098,7 +3269,7 @@ static void rtw89_hw_scan_add_chan(struct rtw89_dev *rtwdev, int chan_type, } static int rtw89_hw_scan_add_chan_list(struct rtw89_dev *rtwdev, - struct rtw89_vif *rtwvif) + struct rtw89_vif *rtwvif, bool connected) { struct cfg80211_scan_request *req = rtwvif->scan_req; struct rtw89_mac_chinfo *ch_info, *tmp; @@ -3142,7 +3313,7 @@ static int rtw89_hw_scan_add_chan_list(struct rtw89_dev *rtwdev, type = RTW89_CHAN_ACTIVE; rtw89_hw_scan_add_chan(rtwdev, type, req->n_ssids, ch_info); - if (rtwvif->net_type != RTW89_NET_TYPE_NO_LINK && + if (connected && off_chan_time + ch_info->period > RTW89_OFF_CHAN_TIME) { tmp = kzalloc(sizeof(*tmp), GFP_KERNEL); if (!tmp) { @@ -3175,7 +3346,7 @@ out: } static int rtw89_hw_scan_prehandle(struct rtw89_dev *rtwdev, - struct rtw89_vif *rtwvif) + struct rtw89_vif *rtwvif, bool connected) { int ret; @@ -3184,7 +3355,7 @@ static int rtw89_hw_scan_prehandle(struct rtw89_dev *rtwdev, rtw89_err(rtwdev, "Update probe request failed\n"); goto out; } - ret = rtw89_hw_scan_add_chan_list(rtwdev, rtwvif); + ret = rtw89_hw_scan_add_chan_list(rtwdev, rtwvif, connected); out: return ret; } @@ -3197,6 +3368,7 @@ void rtw89_hw_scan_start(struct rtw89_dev *rtwdev, struct ieee80211_vif *vif, u32 rx_fltr = rtwdev->hal.rx_fltr; u8 mac_addr[ETH_ALEN]; + rtw89_get_channel(rtwdev, rtwvif, &rtwdev->scan_info.op_chan); rtwdev->scan_info.scanning_vif = vif; rtwdev->scan_info.last_chan_idx = 0; rtwvif->scan_ies = &scan_req->ies; @@ -3222,6 +3394,7 @@ void rtw89_hw_scan_start(struct rtw89_dev *rtwdev, struct ieee80211_vif *vif, void rtw89_hw_scan_complete(struct rtw89_dev *rtwdev, struct ieee80211_vif *vif, bool aborted) { + struct rtw89_hw_scan_info *scan_info = &rtwdev->scan_info; struct cfg80211_scan_info info = { .aborted = aborted, }; @@ -3243,11 +3416,9 @@ void rtw89_hw_scan_complete(struct rtw89_dev *rtwdev, struct ieee80211_vif *vif, rtwvif = (struct rtw89_vif *)vif->drv_priv; rtwvif->scan_req = NULL; rtwvif->scan_ies = NULL; - rtwdev->scan_info.last_chan_idx = 0; - rtwdev->scan_info.scanning_vif = NULL; + scan_info->last_chan_idx = 0; + scan_info->scanning_vif = NULL; - if (rtwvif->net_type != RTW89_NET_TYPE_NO_LINK) - rtw89_store_op_chan(rtwdev, false); rtw89_set_channel(rtwdev); } @@ -3262,16 +3433,19 @@ int rtw89_hw_scan_offload(struct rtw89_dev *rtwdev, struct ieee80211_vif *vif, { struct rtw89_scan_option opt = {0}; struct rtw89_vif *rtwvif; + bool connected; int ret = 0; rtwvif = vif ? (struct rtw89_vif *)vif->drv_priv : NULL; if (!rtwvif) return -EINVAL; + /* This variable implies connected or during attempt to connect */ + connected = !is_zero_ether_addr(rtwvif->bssid); opt.enable = enable; - opt.target_ch_mode = rtwvif->net_type != RTW89_NET_TYPE_NO_LINK; + opt.target_ch_mode = connected; if (enable) { - ret = rtw89_hw_scan_prehandle(rtwdev, rtwvif); + ret = rtw89_hw_scan_prehandle(rtwdev, rtwvif, connected); if (ret) goto out; } @@ -3280,24 +3454,6 @@ out: return ret; } -void rtw89_store_op_chan(struct rtw89_dev *rtwdev, bool backup) -{ - struct rtw89_hw_scan_info *scan_info = &rtwdev->scan_info; - const struct rtw89_chan *cur = rtw89_chan_get(rtwdev, RTW89_SUB_ENTITY_0); - struct rtw89_chan new; - - if (backup) { - scan_info->op_pri_ch = cur->primary_channel; - scan_info->op_chan = cur->channel; - scan_info->op_bw = cur->band_width; - scan_info->op_band = cur->band_type; - } else { - rtw89_chan_create(&new, scan_info->op_chan, scan_info->op_pri_ch, - scan_info->op_band, scan_info->op_bw); - rtw89_assign_entity_chan(rtwdev, RTW89_SUB_ENTITY_0, &new); - } -} - #define H2C_FW_CPU_EXCEPTION_LEN 4 #define H2C_FW_CPU_EXCEPTION_TYPE_DEF 0x5566 int rtw89_fw_h2c_trigger_cpu_exception(struct rtw89_dev *rtwdev) diff --git a/drivers/net/wireless/realtek/rtw89/fw.h b/drivers/net/wireless/realtek/rtw89/fw.h index c3c67ddf61a2..675f85c41471 100644 --- a/drivers/net/wireless/realtek/rtw89/fw.h +++ b/drivers/net/wireless/realtek/rtw89/fw.h @@ -162,6 +162,27 @@ enum rtw89_p2pps_action { RTW89_P2P_ACT_TERMINATE = 3, }; +enum rtw89_bcn_fltr_offload_mode { + RTW89_BCN_FLTR_OFFLOAD_MODE_0 = 0, + RTW89_BCN_FLTR_OFFLOAD_MODE_1, + RTW89_BCN_FLTR_OFFLOAD_MODE_2, + RTW89_BCN_FLTR_OFFLOAD_MODE_3, + + RTW89_BCN_FLTR_OFFLOAD_MODE_DEFAULT = RTW89_BCN_FLTR_OFFLOAD_MODE_0, +}; + +enum rtw89_bcn_fltr_type { + RTW89_BCN_FLTR_BEACON_LOSS, + RTW89_BCN_FLTR_RSSI, + RTW89_BCN_FLTR_NOTIFY, +}; + +enum rtw89_bcn_fltr_rssi_event { + RTW89_BCN_FLTR_RSSI_NOT_CHANGED, + RTW89_BCN_FLTR_RSSI_HIGH, + RTW89_BCN_FLTR_RSSI_LOW, +}; + #define FWDL_SECTION_MAX_NUM 10 #define FWDL_SECTION_CHKSUM_LEN 8 #define FWDL_SECTION_PER_PKT_LEN 2020 @@ -216,6 +237,8 @@ struct rtw89_h2creg_sch_tx_en { #define RTW89_SCAN_LIST_LIMIT \ ((RTW89_H2C_MAX_SIZE / RTW89_MAC_CHINFO_SIZE) - RTW89_SCAN_LIST_GUARD) +#define RTW89_BCN_LOSS_CNT 10 + struct rtw89_mac_chinfo { u8 period; u8 dwell_time; @@ -2174,85 +2197,44 @@ static inline void RTW89_SET_FWCMD_CXHDR_LEN(void *cmd, u8 val) u8p_replace_bits((u8 *)(cmd) + 1, val, GENMASK(7, 0)); } -static inline void RTW89_SET_FWCMD_CXINIT_ANT_TYPE(void *cmd, u8 val) -{ - u8p_replace_bits((u8 *)(cmd) + 2, val, GENMASK(7, 0)); -} - -static inline void RTW89_SET_FWCMD_CXINIT_ANT_NUM(void *cmd, u8 val) -{ - u8p_replace_bits((u8 *)(cmd) + 3, val, GENMASK(7, 0)); -} - -static inline void RTW89_SET_FWCMD_CXINIT_ANT_ISO(void *cmd, u8 val) -{ - u8p_replace_bits((u8 *)(cmd) + 4, val, GENMASK(7, 0)); -} - -static inline void RTW89_SET_FWCMD_CXINIT_ANT_POS(void *cmd, u8 val) -{ - u8p_replace_bits((u8 *)(cmd) + 5, val, BIT(0)); -} - -static inline void RTW89_SET_FWCMD_CXINIT_ANT_DIVERSITY(void *cmd, u8 val) -{ - u8p_replace_bits((u8 *)(cmd) + 5, val, BIT(1)); -} - -static inline void RTW89_SET_FWCMD_CXINIT_MOD_RFE(void *cmd, u8 val) -{ - u8p_replace_bits((u8 *)(cmd) + 6, val, GENMASK(7, 0)); -} - -static inline void RTW89_SET_FWCMD_CXINIT_MOD_CV(void *cmd, u8 val) -{ - u8p_replace_bits((u8 *)(cmd) + 7, val, GENMASK(7, 0)); -} - -static inline void RTW89_SET_FWCMD_CXINIT_MOD_BT_SOLO(void *cmd, u8 val) -{ - u8p_replace_bits((u8 *)(cmd) + 8, val, BIT(0)); -} - -static inline void RTW89_SET_FWCMD_CXINIT_MOD_BT_POS(void *cmd, u8 val) -{ - u8p_replace_bits((u8 *)(cmd) + 8, val, BIT(1)); -} - -static inline void RTW89_SET_FWCMD_CXINIT_MOD_SW_TYPE(void *cmd, u8 val) -{ - u8p_replace_bits((u8 *)(cmd) + 8, val, BIT(2)); -} - -static inline void RTW89_SET_FWCMD_CXINIT_WL_GCH(void *cmd, u8 val) -{ - u8p_replace_bits((u8 *)(cmd) + 10, val, GENMASK(7, 0)); -} - -static inline void RTW89_SET_FWCMD_CXINIT_WL_ONLY(void *cmd, u8 val) -{ - u8p_replace_bits((u8 *)(cmd) + 11, val, BIT(0)); -} +struct rtw89_h2c_cxhdr { + u8 type; + u8 len; +} __packed; -static inline void RTW89_SET_FWCMD_CXINIT_WL_INITOK(void *cmd, u8 val) -{ - u8p_replace_bits((u8 *)(cmd) + 11, val, BIT(1)); -} +#define H2C_LEN_CXDRVHDR sizeof(struct rtw89_h2c_cxhdr) + +struct rtw89_h2c_cxinit { + struct rtw89_h2c_cxhdr hdr; + u8 ant_type; + u8 ant_num; + u8 ant_iso; + u8 ant_info; + u8 mod_rfe; + u8 mod_cv; + u8 mod_info; + u8 mod_adie_kt; + u8 wl_gch; + u8 info; + u8 rsvd; + u8 rsvd1; +} __packed; -static inline void RTW89_SET_FWCMD_CXINIT_DBCC_EN(void *cmd, u8 val) -{ - u8p_replace_bits((u8 *)(cmd) + 11, val, BIT(2)); -} +#define RTW89_H2C_CXINIT_ANT_INFO_POS BIT(0) +#define RTW89_H2C_CXINIT_ANT_INFO_DIVERSITY BIT(1) +#define RTW89_H2C_CXINIT_ANT_INFO_BTG_POS GENMASK(3, 2) +#define RTW89_H2C_CXINIT_ANT_INFO_STREAM_CNT GENMASK(7, 4) -static inline void RTW89_SET_FWCMD_CXINIT_CX_OTHER(void *cmd, u8 val) -{ - u8p_replace_bits((u8 *)(cmd) + 11, val, BIT(3)); -} +#define RTW89_H2C_CXINIT_MOD_INFO_BT_SOLO BIT(0) +#define RTW89_H2C_CXINIT_MOD_INFO_BT_POS BIT(1) +#define RTW89_H2C_CXINIT_MOD_INFO_SW_TYPE BIT(2) +#define RTW89_H2C_CXINIT_MOD_INFO_WA_TYPE GENMASK(5, 3) -static inline void RTW89_SET_FWCMD_CXINIT_BT_ONLY(void *cmd, u8 val) -{ - u8p_replace_bits((u8 *)(cmd) + 11, val, BIT(4)); -} +#define RTW89_H2C_CXINIT_INFO_WL_ONLY BIT(0) +#define RTW89_H2C_CXINIT_INFO_WL_INITOK BIT(1) +#define RTW89_H2C_CXINIT_INFO_DBCC_EN BIT(2) +#define RTW89_H2C_CXINIT_INFO_CX_OTHER BIT(3) +#define RTW89_H2C_CXINIT_INFO_BT_ONLY BIT(4) static inline void RTW89_SET_FWCMD_CXROLE_CONNECT_CNT(void *cmd, u8 val) { @@ -2749,96 +2731,32 @@ static inline void RTW89_SET_FWCMD_CHINFO_POWER_IDX(void *cmd, u32 val) le32p_replace_bits((__le32 *)((u8 *)(cmd) + 16), val, GENMASK(15, 0)); } -static inline void RTW89_SET_FWCMD_SCANOFLD_MACID(void *cmd, u32 val) -{ - le32p_replace_bits((__le32 *)((u8 *)(cmd)), val, GENMASK(7, 0)); -} - -static inline void RTW89_SET_FWCMD_SCANOFLD_NORM_CY(void *cmd, u32 val) -{ - le32p_replace_bits((__le32 *)((u8 *)(cmd)), val, GENMASK(15, 8)); -} - -static inline void RTW89_SET_FWCMD_SCANOFLD_PORT_ID(void *cmd, u32 val) -{ - le32p_replace_bits((__le32 *)((u8 *)(cmd)), val, GENMASK(18, 16)); -} - -static inline void RTW89_SET_FWCMD_SCANOFLD_BAND(void *cmd, u32 val) -{ - le32p_replace_bits((__le32 *)((u8 *)(cmd)), val, BIT(19)); -} - -static inline void RTW89_SET_FWCMD_SCANOFLD_OPERATION(void *cmd, u32 val) -{ - le32p_replace_bits((__le32 *)((u8 *)(cmd)), val, GENMASK(21, 20)); -} - -static inline void RTW89_SET_FWCMD_SCANOFLD_TARGET_CH_BAND(void *cmd, u32 val) -{ - le32p_replace_bits((__le32 *)((u8 *)(cmd)), val, GENMASK(23, 22)); -} - -static inline void RTW89_SET_FWCMD_SCANOFLD_NOTIFY_END(void *cmd, u32 val) -{ - le32p_replace_bits((__le32 *)((u8 *)(cmd) + 4), val, BIT(0)); -} - -static inline void RTW89_SET_FWCMD_SCANOFLD_TARGET_CH_MODE(void *cmd, u32 val) -{ - le32p_replace_bits((__le32 *)((u8 *)(cmd) + 4), val, BIT(1)); -} - -static inline void RTW89_SET_FWCMD_SCANOFLD_START_MODE(void *cmd, u32 val) -{ - le32p_replace_bits((__le32 *)((u8 *)(cmd) + 4), val, BIT(2)); -} - -static inline void RTW89_SET_FWCMD_SCANOFLD_SCAN_TYPE(void *cmd, u32 val) -{ - le32p_replace_bits((__le32 *)((u8 *)(cmd) + 4), val, GENMASK(4, 3)); -} - -static inline void RTW89_SET_FWCMD_SCANOFLD_TARGET_CH_BW(void *cmd, u32 val) -{ - le32p_replace_bits((__le32 *)((u8 *)(cmd) + 4), val, GENMASK(7, 5)); -} - -static inline void RTW89_SET_FWCMD_SCANOFLD_TARGET_PRI_CH(void *cmd, u32 val) -{ - le32p_replace_bits((__le32 *)((u8 *)(cmd) + 4), val, GENMASK(15, 8)); -} - -static inline void RTW89_SET_FWCMD_SCANOFLD_TARGET_CENTRAL_CH(void *cmd, - u32 val) -{ - le32p_replace_bits((__le32 *)((u8 *)(cmd) + 4), val, GENMASK(23, 16)); -} - -static inline void RTW89_SET_FWCMD_SCANOFLD_PROBE_REQ_PKT_ID(void *cmd, u32 val) -{ - le32p_replace_bits((__le32 *)((u8 *)(cmd) + 4), val, GENMASK(31, 24)); -} - -static inline void RTW89_SET_FWCMD_SCANOFLD_NORM_PD(void *cmd, u32 val) -{ - le32p_replace_bits((__le32 *)((u8 *)(cmd) + 8), val, GENMASK(15, 0)); -} - -static inline void RTW89_SET_FWCMD_SCANOFLD_SLOW_PD(void *cmd, u32 val) -{ - le32p_replace_bits((__le32 *)((u8 *)(cmd) + 8), val, GENMASK(23, 16)); -} - -static inline void RTW89_SET_FWCMD_SCANOFLD_TSF_HIGH(void *cmd, u32 val) -{ - le32p_replace_bits((__le32 *)((u8 *)(cmd) + 12), val, GENMASK(31, 0)); -} +struct rtw89_h2c_scanofld { + __le32 w0; + __le32 w1; + __le32 w2; + __le32 tsf_high; + __le32 tsf_low; + __le32 w5; + __le32 w6; +} __packed; -static inline void RTW89_SET_FWCMD_SCANOFLD_TSF_SLOW(void *cmd, u32 val) -{ - le32p_replace_bits((__le32 *)((u8 *)(cmd) + 16), val, GENMASK(31, 0)); -} +#define RTW89_H2C_SCANOFLD_W0_MACID GENMASK(7, 0) +#define RTW89_H2C_SCANOFLD_W0_NORM_CY GENMASK(15, 8) +#define RTW89_H2C_SCANOFLD_W0_PORT_ID GENMASK(18, 16) +#define RTW89_H2C_SCANOFLD_W0_BAND BIT(19) +#define RTW89_H2C_SCANOFLD_W0_OPERATION GENMASK(21, 20) +#define RTW89_H2C_SCANOFLD_W0_TARGET_CH_BAND GENMASK(23, 22) +#define RTW89_H2C_SCANOFLD_W1_NOTIFY_END BIT(0) +#define RTW89_H2C_SCANOFLD_W1_TARGET_CH_MODE BIT(1) +#define RTW89_H2C_SCANOFLD_W1_START_MODE BIT(2) +#define RTW89_H2C_SCANOFLD_W1_SCAN_TYPE GENMASK(4, 3) +#define RTW89_H2C_SCANOFLD_W1_TARGET_CH_BW GENMASK(7, 5) +#define RTW89_H2C_SCANOFLD_W1_TARGET_PRI_CH GENMASK(15, 8) +#define RTW89_H2C_SCANOFLD_W1_TARGET_CENTRAL_CH GENMASK(23, 16) +#define RTW89_H2C_SCANOFLD_W1_PROBE_REQ_PKT_ID GENMASK(31, 24) +#define RTW89_H2C_SCANOFLD_W2_NORM_PD GENMASK(15, 0) +#define RTW89_H2C_SCANOFLD_W2_SLOW_PD GENMASK(23, 16) static inline void RTW89_SET_FWCMD_P2P_MACID(void *cmd, u32 val) { @@ -3317,6 +3235,17 @@ static inline struct rtw89_fw_c2h_attr *RTW89_SKB_C2H_CB(struct sk_buff *skb) #define RTW89_GET_MAC_C2H_REV_ACK_H2C_SEQ(c2h) \ le32_get_bits(*((const __le32 *)(c2h) + 2), GENMASK(23, 16)) +struct rtw89_c2h_mac_bcnfltr_rpt { + __le32 w0; + __le32 w1; + __le32 w2; +} __packed; + +#define RTW89_C2H_MAC_BCNFLTR_RPT_W2_MACID GENMASK(7, 0) +#define RTW89_C2H_MAC_BCNFLTR_RPT_W2_TYPE GENMASK(9, 8) +#define RTW89_C2H_MAC_BCNFLTR_RPT_W2_EVENT GENMASK(11, 10) +#define RTW89_C2H_MAC_BCNFLTR_RPT_W2_MA GENMASK(23, 16) + #define RTW89_GET_PHY_C2H_RA_RPT_MACID(c2h) \ le32_get_bits(*((const __le32 *)(c2h) + 2), GENMASK(15, 0)) #define RTW89_GET_PHY_C2H_RA_RPT_RETRY_RATIO(c2h) \ @@ -3410,6 +3339,36 @@ static_assert(sizeof(struct rtw89_mac_mcc_tsf_rpt) <= RTW89_COMPLETION_BUF_SIZE) #define RTW89_GET_MAC_C2H_MCC_STATUS_RPT_TSF_HIGH(c2h) \ le32_get_bits(*((const __le32 *)(c2h) + 4), GENMASK(31, 0)) +struct rtw89_h2c_bcnfltr { + __le32 w0; +} __packed; + +#define RTW89_H2C_BCNFLTR_W0_MON_RSSI BIT(0) +#define RTW89_H2C_BCNFLTR_W0_MON_BCN BIT(1) +#define RTW89_H2C_BCNFLTR_W0_MON_EN BIT(2) +#define RTW89_H2C_BCNFLTR_W0_MODE GENMASK(4, 3) +#define RTW89_H2C_BCNFLTR_W0_BCN_LOSS_CNT GENMASK(11, 8) +#define RTW89_H2C_BCNFLTR_W0_RSSI_HYST GENMASK(15, 12) +#define RTW89_H2C_BCNFLTR_W0_RSSI_THRESHOLD GENMASK(23, 16) +#define RTW89_H2C_BCNFLTR_W0_MAC_ID GENMASK(31, 24) + +struct rtw89_h2c_ofld_rssi { + __le32 w0; + __le32 w1; +} __packed; + +#define RTW89_H2C_OFLD_RSSI_W0_MACID GENMASK(7, 0) +#define RTW89_H2C_OFLD_RSSI_W0_NUM GENMASK(15, 8) +#define RTW89_H2C_OFLD_RSSI_W1_VAL GENMASK(7, 0) + +struct rtw89_h2c_ofld { + __le32 w0; +} __packed; + +#define RTW89_H2C_OFLD_W0_MAC_ID GENMASK(7, 0) +#define RTW89_H2C_OFLD_W0_TX_TP GENMASK(17, 8) +#define RTW89_H2C_OFLD_W0_RX_TP GENMASK(27, 18) + #define RTW89_FW_HDR_SIZE 32 #define RTW89_FW_SECTION_HDR_SIZE 16 @@ -3459,6 +3418,15 @@ static inline u32 rtw89_compat_fw_hdr_ver_code(const void *fw_buf) return RTW89_FW_HDR_VER_CODE(&compat->fw_hdr); } +static inline void rtw89_fw_get_filename(char *buf, size_t size, + const char *fw_basename, int fw_format) +{ + if (fw_format <= 0) + snprintf(buf, size, "%s.bin", fw_basename); + else + snprintf(buf, size, "%s-%d.bin", fw_basename, fw_format); +} + #define RTW89_H2C_RF_PAGE_SIZE 500 #define RTW89_H2C_RF_PAGE_NUM 3 struct rtw89_fw_h2c_rf_reg_info { @@ -3537,6 +3505,9 @@ struct rtw89_fw_h2c_rf_reg_info { #define H2C_FUNC_ADD_SCANOFLD_CH 0x16 #define H2C_FUNC_SCANOFLD 0x17 #define H2C_FUNC_PKT_DROP 0x1b +#define H2C_FUNC_CFG_BCNFLTR 0x1e +#define H2C_FUNC_OFLD_RSSI 0x1f +#define H2C_FUNC_OFLD_TP 0x20 /* CLASS 10 - Security CAM */ #define H2C_CL_MAC_SEC_CAM 0xa @@ -3600,9 +3571,10 @@ int rtw89_fw_recognize(struct rtw89_dev *rtwdev); const struct firmware * rtw89_early_fw_feature_recognize(struct device *device, const struct rtw89_chip_info *chip, - u32 *early_feat_map); + struct rtw89_fw_info *early_fw, + int *used_fw_format); int rtw89_fw_download(struct rtw89_dev *rtwdev, enum rtw89_fw_type type); -int rtw89_load_firmware(struct rtw89_dev *rtwdev); +void rtw89_load_firmware_work(struct work_struct *work); void rtw89_unload_firmware(struct rtw89_dev *rtwdev); int rtw89_wait_firmware_completion(struct rtw89_dev *rtwdev); void rtw89_h2c_pkt_set_hdr(struct rtw89_dev *rtwdev, struct sk_buff *skb, @@ -3637,6 +3609,12 @@ int rtw89_fw_h2c_macid_pause(struct rtw89_dev *rtwdev, u8 sh, u8 grp, int rtw89_fw_h2c_set_edca(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif, u8 ac, u32 val); int rtw89_fw_h2c_set_ofld_cfg(struct rtw89_dev *rtwdev); +int rtw89_fw_h2c_set_bcn_fltr_cfg(struct rtw89_dev *rtwdev, + struct ieee80211_vif *vif, + bool connect); +int rtw89_fw_h2c_rssi_offload(struct rtw89_dev *rtwdev, + struct rtw89_rx_phy_ppdu *phy_ppdu); +int rtw89_fw_h2c_tp_offload(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif); int rtw89_fw_h2c_ra(struct rtw89_dev *rtwdev, struct rtw89_ra_info *ra, bool csi); int rtw89_fw_h2c_cxdrv_init(struct rtw89_dev *rtwdev); int rtw89_fw_h2c_cxdrv_role(struct rtw89_dev *rtwdev); @@ -3681,7 +3659,6 @@ int rtw89_fw_msg_reg(struct rtw89_dev *rtwdev, struct rtw89_mac_c2h_info *c2h_info); int rtw89_fw_h2c_fw_log(struct rtw89_dev *rtwdev, bool enable); void rtw89_fw_st_dbg_dump(struct rtw89_dev *rtwdev); -void rtw89_store_op_chan(struct rtw89_dev *rtwdev, bool backup); void rtw89_hw_scan_start(struct rtw89_dev *rtwdev, struct ieee80211_vif *vif, struct ieee80211_scan_request *req); void rtw89_hw_scan_complete(struct rtw89_dev *rtwdev, struct ieee80211_vif *vif, diff --git a/drivers/net/wireless/realtek/rtw89/mac.c b/drivers/net/wireless/realtek/rtw89/mac.c index d0e138f8b880..b8019cfc11b2 100644 --- a/drivers/net/wireless/realtek/rtw89/mac.c +++ b/drivers/net/wireless/realtek/rtw89/mac.c @@ -1473,6 +1473,8 @@ const struct rtw89_mac_size_set rtw89_mac_size = { .ple_qt58 = {147, 0, 16, 20, 157, 13, 229, 0, 172, 14, 24, 0,}, /* 8852A PCIE WOW */ .ple_qt_52a_wow = {264, 0, 32, 20, 64, 13, 1005, 0, 64, 128, 120,}, + /* 8852B PCIE WOW */ + .ple_qt_52b_wow = {147, 0, 16, 20, 157, 13, 133, 0, 172, 14, 24, 0,}, }; EXPORT_SYMBOL(rtw89_mac_size); @@ -1584,12 +1586,15 @@ static void dle_func_en(struct rtw89_dev *rtwdev, bool enable) static void dle_clk_en(struct rtw89_dev *rtwdev, bool enable) { - if (enable) - rtw89_write32_set(rtwdev, R_AX_DMAC_CLK_EN, - B_AX_DLE_WDE_CLK_EN | B_AX_DLE_PLE_CLK_EN); - else - rtw89_write32_clr(rtwdev, R_AX_DMAC_CLK_EN, - B_AX_DLE_WDE_CLK_EN | B_AX_DLE_PLE_CLK_EN); + u32 val = B_AX_DLE_WDE_CLK_EN | B_AX_DLE_PLE_CLK_EN; + + if (enable) { + if (rtwdev->chip->chip_id == RTL8851B) + val |= B_AX_AXIDMA_CLK_EN; + rtw89_write32_set(rtwdev, R_AX_DMAC_CLK_EN, val); + } else { + rtw89_write32_clr(rtwdev, R_AX_DMAC_CLK_EN, val); + } } static int dle_mix_cfg(struct rtw89_dev *rtwdev, const struct rtw89_dle_mem *cfg) @@ -1854,7 +1859,8 @@ static int preload_init(struct rtw89_dev *rtwdev, enum rtw89_mac_idx mac_idx, { const struct rtw89_chip_info *chip = rtwdev->chip; - if (chip->chip_id == RTL8852A || chip->chip_id == RTL8852B || !is_qta_poh(rtwdev)) + if (chip->chip_id == RTL8852A || chip->chip_id == RTL8852B || + chip->chip_id == RTL8851B || !is_qta_poh(rtwdev)) return 0; return preload_init_set(rtwdev, mac_idx, mode); @@ -1890,7 +1896,8 @@ static void _patch_ss2f_path(struct rtw89_dev *rtwdev) { const struct rtw89_chip_info *chip = rtwdev->chip; - if (chip->chip_id == RTL8852A || chip->chip_id == RTL8852B) + if (chip->chip_id == RTL8852A || chip->chip_id == RTL8852B || + chip->chip_id == RTL8851B) return; rtw89_write32_mask(rtwdev, R_AX_SS2FINFO_PATH, B_AX_SS_DEST_QUEUE_MASK, @@ -1959,7 +1966,8 @@ static int sec_eng_init(struct rtw89_dev *rtwdev) /* init TX encryption */ val |= (B_AX_SEC_TX_ENC | B_AX_SEC_RX_DEC); val |= (B_AX_MC_DEC | B_AX_BC_DEC); - if (chip->chip_id == RTL8852A || chip->chip_id == RTL8852B) + if (chip->chip_id == RTL8852A || chip->chip_id == RTL8852B || + chip->chip_id == RTL8851B) val &= ~B_AX_TX_PARTIAL_MODE; rtw89_write32(rtwdev, R_AX_SEC_ENG_CTRL, val); @@ -2065,7 +2073,7 @@ static int scheduler_init(struct rtw89_dev *rtwdev, u8 mac_idx) rtw89_write32_mask(rtwdev, reg, B_AX_SIFS_MACTXEN_T1_MASK, SIFS_MACTXEN_T1); - if (rtwdev->chip->chip_id == RTL8852B) { + if (rtwdev->chip->chip_id == RTL8852B || rtwdev->chip->chip_id == RTL8851B) { reg = rtw89_mac_reg_by_idx(R_AX_SCH_EXT_CTRL, mac_idx); rtw89_write32_set(rtwdev, reg, B_AX_PORT_RST_TSF_ADV); } @@ -2805,7 +2813,7 @@ int rtw89_mac_resume_sch_tx_v1(struct rtw89_dev *rtwdev, u8 mac_idx, u32 tx_en) } EXPORT_SYMBOL(rtw89_mac_resume_sch_tx_v1); -u16 rtw89_mac_dle_buf_req(struct rtw89_dev *rtwdev, u16 buf_len, bool wd) +int rtw89_mac_dle_buf_req(struct rtw89_dev *rtwdev, u16 buf_len, bool wd, u16 *pkt_id) { u32 val, reg; int ret; @@ -2820,9 +2828,13 @@ u16 rtw89_mac_dle_buf_req(struct rtw89_dev *rtwdev, u16 buf_len, bool wd) ret = read_poll_timeout(rtw89_read32, val, val & B_AX_WD_BUF_STAT_DONE, 1, 2000, false, rtwdev, reg); if (ret) - return 0xffff; + return ret; + + *pkt_id = FIELD_GET(B_AX_WD_BUF_STAT_PKTID_MASK, val); + if (*pkt_id == S_WD_BUF_STAT_PKTID_INVALID) + return -ENOENT; - return FIELD_GET(B_AX_WD_BUF_STAT_PKTID_MASK, val); + return 0; } int rtw89_mac_set_cpuio(struct rtw89_dev *rtwdev, @@ -2899,10 +2911,10 @@ static int dle_quota_change(struct rtw89_dev *rtwdev, enum rtw89_qta_mode mode) dle_quota_cfg(rtwdev, cfg, INVALID_QT_WCPU); - pkt_id = rtw89_mac_dle_buf_req(rtwdev, 0x20, true); - if (pkt_id == 0xffff) { + ret = rtw89_mac_dle_buf_req(rtwdev, 0x20, true, &pkt_id); + if (ret) { rtw89_err(rtwdev, "[ERR]WDE DLE buf req\n"); - return -ENOMEM; + return ret; } ctrl_para.cmd_type = CPUIO_OP_CMD_ENQ_TO_HEAD; @@ -2917,10 +2929,10 @@ static int dle_quota_change(struct rtw89_dev *rtwdev, enum rtw89_qta_mode mode) return -EFAULT; } - pkt_id = rtw89_mac_dle_buf_req(rtwdev, 0x20, false); - if (pkt_id == 0xffff) { + ret = rtw89_mac_dle_buf_req(rtwdev, 0x20, false, &pkt_id); + if (ret) { rtw89_err(rtwdev, "[ERR]PLE DLE buf req\n"); - return -ENOMEM; + return ret; } ctrl_para.cmd_type = CPUIO_OP_CMD_ENQ_TO_HEAD; @@ -3364,8 +3376,15 @@ static int rtw89_mac_trx_init(struct rtw89_dev *rtwdev) static void rtw89_disable_fw_watchdog(struct rtw89_dev *rtwdev) { + enum rtw89_core_chip_id chip_id = rtwdev->chip->chip_id; u32 val32; + if (chip_id == RTL8852B || chip_id == RTL8851B) { + rtw89_write32_clr(rtwdev, R_AX_PLATFORM_ENABLE, B_AX_APB_WRAP_EN); + rtw89_write32_set(rtwdev, R_AX_PLATFORM_ENABLE, B_AX_APB_WRAP_EN); + return; + } + rtw89_mac_mem_write(rtwdev, R_AX_WDT_CTRL, WDT_CTRL_ALL_DIS, RTW89_MAC_MEM_CPU_LOCAL); @@ -3450,7 +3469,10 @@ static int rtw89_mac_dmac_pre_init(struct rtw89_dev *rtwdev) B_AX_PKT_BUF_EN; rtw89_write32(rtwdev, R_AX_DMAC_FUNC_EN, val); - val = B_AX_DISPATCHER_CLK_EN; + if (chip_id == RTL8851B) + val = B_AX_DISPATCHER_CLK_EN | B_AX_AXIDMA_CLK_EN; + else + val = B_AX_DISPATCHER_CLK_EN; rtw89_write32(rtwdev, R_AX_DMAC_CLK_EN, val); if (chip_id != RTL8852C) @@ -4176,9 +4198,9 @@ rtw89_mac_c2h_macid_pause(struct rtw89_dev *rtwdev, struct sk_buff *c2h, u32 len static bool rtw89_is_op_chan(struct rtw89_dev *rtwdev, u8 band, u8 channel) { - struct rtw89_hw_scan_info *scan_info = &rtwdev->scan_info; + const struct rtw89_chan *op = &rtwdev->scan_info.op_chan; - return band == scan_info->op_band && channel == scan_info->op_pri_ch; + return band == op->band_type && channel == op->primary_channel; } static void @@ -4193,6 +4215,9 @@ rtw89_mac_c2h_scanofld_rsp(struct rtw89_dev *rtwdev, struct sk_buff *c2h, u16 chan; int ret; + if (!rtwvif) + return; + tx_fail = RTW89_GET_MAC_C2H_SCANOFLD_TX_FAIL(c2h->data); status = RTW89_GET_MAC_C2H_SCANOFLD_STATUS(c2h->data); chan = RTW89_GET_MAC_C2H_SCANOFLD_PRI_CH(c2h->data); @@ -4225,11 +4250,15 @@ rtw89_mac_c2h_scanofld_rsp(struct rtw89_dev *rtwdev, struct sk_buff *c2h, } break; case RTW89_SCAN_ENTER_CH_NOTIFY: - rtw89_chan_create(&new, chan, chan, band, RTW89_CHANNEL_WIDTH_20); - rtw89_assign_entity_chan(rtwdev, RTW89_SUB_ENTITY_0, &new); if (rtw89_is_op_chan(rtwdev, band, chan)) { - rtw89_store_op_chan(rtwdev, false); + rtw89_assign_entity_chan(rtwdev, rtwvif->sub_entity_idx, + &rtwdev->scan_info.op_chan); ieee80211_wake_queues(rtwdev->hw); + } else { + rtw89_chan_create(&new, chan, chan, band, + RTW89_CHANNEL_WIDTH_20); + rtw89_assign_entity_chan(rtwdev, rtwvif->sub_entity_idx, + &new); } break; default: @@ -4238,6 +4267,64 @@ rtw89_mac_c2h_scanofld_rsp(struct rtw89_dev *rtwdev, struct sk_buff *c2h, } static void +rtw89_mac_bcn_fltr_rpt(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif, + struct sk_buff *skb) +{ + struct ieee80211_vif *vif = rtwvif_to_vif_safe(rtwvif); + enum nl80211_cqm_rssi_threshold_event nl_event; + const struct rtw89_c2h_mac_bcnfltr_rpt *c2h = + (const struct rtw89_c2h_mac_bcnfltr_rpt *)skb->data; + u8 type, event, mac_id; + s8 sig; + + type = le32_get_bits(c2h->w2, RTW89_C2H_MAC_BCNFLTR_RPT_W2_TYPE); + sig = le32_get_bits(c2h->w2, RTW89_C2H_MAC_BCNFLTR_RPT_W2_MA) - MAX_RSSI; + event = le32_get_bits(c2h->w2, RTW89_C2H_MAC_BCNFLTR_RPT_W2_EVENT); + mac_id = le32_get_bits(c2h->w2, RTW89_C2H_MAC_BCNFLTR_RPT_W2_MACID); + + if (mac_id != rtwvif->mac_id) + return; + + rtw89_debug(rtwdev, RTW89_DBG_FW, + "C2H bcnfltr rpt macid: %d, type: %d, ma: %d, event: %d\n", + mac_id, type, sig, event); + + switch (type) { + case RTW89_BCN_FLTR_BEACON_LOSS: + if (!rtwdev->scanning && !rtwvif->offchan) + ieee80211_connection_loss(vif); + else + rtw89_fw_h2c_set_bcn_fltr_cfg(rtwdev, vif, true); + return; + case RTW89_BCN_FLTR_NOTIFY: + nl_event = NL80211_CQM_RSSI_THRESHOLD_EVENT_HIGH; + break; + case RTW89_BCN_FLTR_RSSI: + if (event == RTW89_BCN_FLTR_RSSI_LOW) + nl_event = NL80211_CQM_RSSI_THRESHOLD_EVENT_LOW; + else if (event == RTW89_BCN_FLTR_RSSI_HIGH) + nl_event = NL80211_CQM_RSSI_THRESHOLD_EVENT_HIGH; + else + return; + break; + default: + return; + } + + ieee80211_cqm_rssi_notify(vif, nl_event, sig, GFP_KERNEL); +} + +static void +rtw89_mac_c2h_bcn_fltr_rpt(struct rtw89_dev *rtwdev, struct sk_buff *c2h, + u32 len) +{ + struct rtw89_vif *rtwvif; + + rtw89_for_each_rtwvif(rtwdev, rtwvif) + rtw89_mac_bcn_fltr_rpt(rtwdev, rtwvif, c2h); +} + +static void rtw89_mac_c2h_rec_ack(struct rtw89_dev *rtwdev, struct sk_buff *c2h, u32 len) { rtw89_debug(rtwdev, RTW89_DBG_FW, @@ -4457,6 +4544,7 @@ void (* const rtw89_mac_c2h_ofld_handler[])(struct rtw89_dev *rtwdev, [RTW89_MAC_C2H_FUNC_MACID_PAUSE] = rtw89_mac_c2h_macid_pause, [RTW89_MAC_C2H_FUNC_SCANOFLD_RSP] = rtw89_mac_c2h_scanofld_rsp, [RTW89_MAC_C2H_FUNC_TSF32_TOGL_RPT] = rtw89_mac_c2h_tsf32_toggle_rpt, + [RTW89_MAC_C2H_FUNC_BCNFLTR_RPT] = rtw89_mac_c2h_bcn_fltr_rpt, }; static @@ -4629,11 +4717,13 @@ int rtw89_mac_coex_init(struct rtw89_dev *rtwdev, const struct rtw89_mac_ax_coex int ret; rtw89_write8_set(rtwdev, R_AX_GPIO_MUXCFG, B_AX_ENBT); - rtw89_write8_set(rtwdev, R_AX_BTC_FUNC_EN, B_AX_PTA_WL_TX_EN); + if (rtwdev->chip->chip_id != RTL8851B) + rtw89_write8_set(rtwdev, R_AX_BTC_FUNC_EN, B_AX_PTA_WL_TX_EN); rtw89_write8_set(rtwdev, R_AX_BT_COEX_CFG_2 + 1, B_AX_GNT_BT_POLARITY >> 8); rtw89_write8_set(rtwdev, R_AX_CSR_MODE, B_AX_STATIS_BT_EN | B_AX_WL_ACT_MSK); rtw89_write8_set(rtwdev, R_AX_CSR_MODE + 2, B_AX_BT_CNT_RST >> 16); - rtw89_write8_clr(rtwdev, R_AX_TRXPTCL_RESP_0 + 3, B_AX_RSP_CHK_BTCCA >> 24); + if (rtwdev->chip->chip_id != RTL8851B) + rtw89_write8_clr(rtwdev, R_AX_TRXPTCL_RESP_0 + 3, B_AX_RSP_CHK_BTCCA >> 24); val16 = rtw89_read16(rtwdev, R_AX_CCA_CFG_0); val16 = (val16 | B_AX_BTCCA_EN) & ~B_AX_BTCCA_BRK_TXOP_EN; diff --git a/drivers/net/wireless/realtek/rtw89/mac.h b/drivers/net/wireless/realtek/rtw89/mac.h index 8064d3953d7f..a8d9847ef0b4 100644 --- a/drivers/net/wireless/realtek/rtw89/mac.h +++ b/drivers/net/wireless/realtek/rtw89/mac.h @@ -359,6 +359,7 @@ enum rtw89_mac_c2h_ofld_func { RTW89_MAC_C2H_FUNC_MACID_PAUSE, RTW89_MAC_C2H_FUNC_TSF32_TOGL_RPT = 0x6, RTW89_MAC_C2H_FUNC_SCANOFLD_RSP = 0x9, + RTW89_MAC_C2H_FUNC_BCNFLTR_RPT = 0xd, RTW89_MAC_C2H_FUNC_OFLD_MAX, }; @@ -815,6 +816,7 @@ struct rtw89_mac_size_set { const struct rtw89_ple_quota ple_qt47; const struct rtw89_ple_quota ple_qt58; const struct rtw89_ple_quota ple_qt_52a_wow; + const struct rtw89_ple_quota ple_qt_52b_wow; }; extern const struct rtw89_mac_size_set rtw89_mac_size; @@ -1116,6 +1118,7 @@ enum rtw89_mac_xtal_si_offset { XTAL_SI_XTAL_XMD_4 = 0x26, #define XTAL_SI_LPS_CAP GENMASK(3, 0) XTAL_SI_CV = 0x41, +#define XTAL_SI_ACV_MASK GENMASK(3, 0) XTAL_SI_LOW_ADDR = 0x62, #define XTAL_SI_LOW_ADDR_MASK GENMASK(7, 0) XTAL_SI_CTRL = 0x63, @@ -1146,7 +1149,7 @@ enum rtw89_mac_xtal_si_offset { int rtw89_mac_write_xtal_si(struct rtw89_dev *rtwdev, u8 offset, u8 val, u8 mask); int rtw89_mac_read_xtal_si(struct rtw89_dev *rtwdev, u8 offset, u8 *val); void rtw89_mac_pkt_drop_vif(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif); -u16 rtw89_mac_dle_buf_req(struct rtw89_dev *rtwdev, u16 buf_len, bool wd); +int rtw89_mac_dle_buf_req(struct rtw89_dev *rtwdev, u16 buf_len, bool wd, u16 *pkt_id); int rtw89_mac_set_cpuio(struct rtw89_dev *rtwdev, struct rtw89_cpuio_ctrl *ctrl_para, bool wd); int rtw89_mac_typ_fltr_opt(struct rtw89_dev *rtwdev, diff --git a/drivers/net/wireless/realtek/rtw89/mac80211.c b/drivers/net/wireless/realtek/rtw89/mac80211.c index 367a7bf319da..ee4588b61b8f 100644 --- a/drivers/net/wireless/realtek/rtw89/mac80211.c +++ b/drivers/net/wireless/realtek/rtw89/mac80211.c @@ -23,9 +23,19 @@ static void rtw89_ops_tx(struct ieee80211_hw *hw, struct rtw89_dev *rtwdev = hw->priv; struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb); struct ieee80211_vif *vif = info->control.vif; + struct rtw89_vif *rtwvif = (struct rtw89_vif *)vif->drv_priv; struct ieee80211_sta *sta = control->sta; + u32 flags = IEEE80211_SKB_CB(skb)->flags; int ret, qsel; + if (rtwvif->offchan && !(flags & IEEE80211_TX_CTL_TX_OFFCHAN) && sta) { + struct rtw89_sta *rtwsta = (struct rtw89_sta *)sta->drv_priv; + + rtw89_debug(rtwdev, RTW89_DBG_TXRX, "ops_tx during offchan\n"); + skb_queue_tail(&rtwsta->roc_queue, skb); + return; + } + ret = rtw89_core_tx_write(rtwdev, vif, sta, skb, &qsel); if (ret) { rtw89_err(rtwdev, "failed to transmit skb: %d\n", ret); @@ -95,7 +105,8 @@ static int rtw89_ops_config(struct ieee80211_hw *hw, u32 changed) } if ((changed & IEEE80211_CONF_CHANGE_IDLE) && - (hw->conf.flags & IEEE80211_CONF_IDLE)) + (hw->conf.flags & IEEE80211_CONF_IDLE) && + !rtwdev->scanning) rtw89_enter_ips(rtwdev); mutex_unlock(&rtwdev->mutex); @@ -114,9 +125,19 @@ static int rtw89_ops_add_interface(struct ieee80211_hw *hw, vif->addr, vif->type, vif->p2p); mutex_lock(&rtwdev->mutex); + + rtw89_leave_ips_by_hwflags(rtwdev); + + if (RTW89_CHK_FW_FEATURE(BEACON_FILTER, &rtwdev->fw)) + vif->driver_flags |= IEEE80211_VIF_BEACON_FILTER | + IEEE80211_VIF_SUPPORTS_CQM_RSSI; + rtwvif->rtwdev = rtwdev; + rtwvif->roc.state = RTW89_ROC_IDLE; + rtwvif->offchan = false; list_add_tail(&rtwvif->list, &rtwdev->rtwvifs_list); INIT_WORK(&rtwvif->update_beacon_work, rtw89_core_update_beacon_work); + INIT_DELAYED_WORK(&rtwvif->roc.roc_work, rtw89_roc_work); rtw89_leave_ps_mode(rtwdev); rtw89_traffic_stats_init(rtwdev, &rtwvif->stats); @@ -163,6 +184,7 @@ static void rtw89_ops_remove_interface(struct ieee80211_hw *hw, vif->addr, vif->type, vif->p2p); cancel_work_sync(&rtwvif->update_beacon_work); + cancel_delayed_work_sync(&rtwvif->roc.roc_work); mutex_lock(&rtwdev->mutex); rtw89_leave_ps_mode(rtwdev); @@ -170,6 +192,8 @@ static void rtw89_ops_remove_interface(struct ieee80211_hw *hw, rtw89_mac_remove_vif(rtwdev, rtwvif); rtw89_core_release_bit_map(rtwdev->hw_port, rtwvif->port); list_del_init(&rtwvif->list); + rtw89_enter_ips_by_hwflags(rtwdev); + mutex_unlock(&rtwdev->mutex); } @@ -394,7 +418,6 @@ static void rtw89_ops_bss_info_changed(struct ieee80211_hw *hw, rtw89_chip_cfg_txpwr_ul_tb_offset(rtwdev, vif); rtw89_mac_port_update(rtwdev, rtwvif); rtw89_mac_set_he_obss_narrow_bw_ru(rtwdev, vif); - rtw89_store_op_chan(rtwdev, true); } else { /* Abort ongoing scan if cancel_scan isn't issued * when disconnected by peer @@ -425,6 +448,9 @@ static void rtw89_ops_bss_info_changed(struct ieee80211_hw *hw, if (changed & BSS_CHANGED_P2P_PS) rtw89_process_p2p_ps(rtwdev, vif); + if (changed & BSS_CHANGED_CQM) + rtw89_fw_h2c_set_bcn_fltr_cfg(rtwdev, vif, true); + mutex_unlock(&rtwdev->mutex); } @@ -795,12 +821,13 @@ static int rtw89_ops_hw_scan(struct ieee80211_hw *hw, struct ieee80211_vif *vif, struct ieee80211_scan_request *req) { struct rtw89_dev *rtwdev = hw->priv; + struct rtw89_vif *rtwvif = vif_to_rtwvif_safe(vif); int ret = 0; if (!RTW89_CHK_FW_FEATURE(SCAN_OFFLOAD, &rtwdev->fw)) return 1; - if (rtwdev->scanning) + if (rtwdev->scanning || rtwvif->offchan) return -EBUSY; mutex_lock(&rtwdev->mutex); @@ -903,6 +930,63 @@ static void rtw89_ops_unassign_vif_chanctx(struct ieee80211_hw *hw, mutex_unlock(&rtwdev->mutex); } +static int rtw89_ops_remain_on_channel(struct ieee80211_hw *hw, + struct ieee80211_vif *vif, + struct ieee80211_channel *chan, + int duration, + enum ieee80211_roc_type type) +{ + struct rtw89_dev *rtwdev = hw->priv; + struct rtw89_vif *rtwvif = vif_to_rtwvif_safe(vif); + struct rtw89_roc *roc = &rtwvif->roc; + + if (!vif) + return -EINVAL; + + mutex_lock(&rtwdev->mutex); + + if (roc->state != RTW89_ROC_IDLE) { + mutex_unlock(&rtwdev->mutex); + return -EBUSY; + } + + if (rtwdev->scanning) + rtw89_hw_scan_abort(rtwdev, vif); + + if (type == IEEE80211_ROC_TYPE_MGMT_TX) + roc->state = RTW89_ROC_MGMT; + else + roc->state = RTW89_ROC_NORMAL; + + roc->duration = duration; + roc->chan = *chan; + roc->type = type; + + rtw89_roc_start(rtwdev, rtwvif); + + mutex_unlock(&rtwdev->mutex); + + return 0; +} + +static int rtw89_ops_cancel_remain_on_channel(struct ieee80211_hw *hw, + struct ieee80211_vif *vif) +{ + struct rtw89_dev *rtwdev = hw->priv; + struct rtw89_vif *rtwvif = vif_to_rtwvif_safe(vif); + + if (!rtwvif) + return -EINVAL; + + cancel_delayed_work_sync(&rtwvif->roc.roc_work); + + mutex_lock(&rtwdev->mutex); + rtw89_roc_end(rtwdev, rtwvif); + mutex_unlock(&rtwdev->mutex); + + return 0; +} + static void rtw89_set_tid_config_iter(void *data, struct ieee80211_sta *sta) { struct cfg80211_tid_config *tid_config = data; @@ -1014,6 +1098,8 @@ const struct ieee80211_ops rtw89_ops = { .change_chanctx = rtw89_ops_change_chanctx, .assign_vif_chanctx = rtw89_ops_assign_vif_chanctx, .unassign_vif_chanctx = rtw89_ops_unassign_vif_chanctx, + .remain_on_channel = rtw89_ops_remain_on_channel, + .cancel_remain_on_channel = rtw89_ops_cancel_remain_on_channel, .set_sar_specs = rtw89_ops_set_sar_specs, .sta_rc_update = rtw89_ops_sta_rc_update, .set_tid_config = rtw89_ops_set_tid_config, diff --git a/drivers/net/wireless/realtek/rtw89/pci.c b/drivers/net/wireless/realtek/rtw89/pci.c index 68f0fed6d31e..70b4754667c9 100644 --- a/drivers/net/wireless/realtek/rtw89/pci.c +++ b/drivers/net/wireless/realtek/rtw89/pci.c @@ -364,8 +364,11 @@ static void rtw89_pci_tx_status(struct rtw89_dev *rtwdev, struct rtw89_pci_tx_ring *tx_ring, struct sk_buff *skb, u8 tx_status) { + struct rtw89_tx_skb_data *skb_data = RTW89_TX_SKB_CB(skb); struct ieee80211_tx_info *info; + rtw89_core_tx_wait_complete(rtwdev, skb_data, tx_status == RTW89_TX_DONE); + info = IEEE80211_SKB_CB(skb); ieee80211_tx_info_clear_status(info); @@ -1203,6 +1206,7 @@ static int rtw89_pci_txwd_submit(struct rtw89_dev *rtwdev, struct pci_dev *pdev = rtwpci->pdev; struct sk_buff *skb = tx_req->skb; struct rtw89_pci_tx_data *tx_data = RTW89_PCI_TX_SKB_CB(skb); + struct rtw89_tx_skb_data *skb_data = RTW89_TX_SKB_CB(skb); bool en_wd_info = desc_info->en_wd_info; u32 txwd_len; u32 txwp_len; @@ -1218,6 +1222,7 @@ static int rtw89_pci_txwd_submit(struct rtw89_dev *rtwdev, } tx_data->dma = dma; + rcu_assign_pointer(skb_data->wait, NULL); txwp_len = sizeof(*txwp_info); txwd_len = chip->txwd_body_size; @@ -1912,9 +1917,10 @@ __get_target(struct rtw89_dev *rtwdev, u16 *target, enum rtw89_pcie_phy phy_rate static int rtw89_pci_autok_x(struct rtw89_dev *rtwdev) { + enum rtw89_core_chip_id chip_id = rtwdev->chip->chip_id; int ret; - if (rtwdev->chip->chip_id != RTL8852B) + if (chip_id != RTL8852B && chip_id != RTL8851B) return 0; ret = rtw89_write16_mdio_mask(rtwdev, RAC_REG_FLD_0, BAC_AUTOK_N_MASK, @@ -1924,13 +1930,14 @@ static int rtw89_pci_autok_x(struct rtw89_dev *rtwdev) static int rtw89_pci_auto_refclk_cal(struct rtw89_dev *rtwdev, bool autook_en) { + enum rtw89_core_chip_id chip_id = rtwdev->chip->chip_id; enum rtw89_pcie_phy phy_rate; u16 val16, mgn_set, div_set, tar; u8 val8, bdr_ori; bool l1_flag = false; int ret = 0; - if (rtwdev->chip->chip_id != RTL8852B) + if (chip_id != RTL8852B && chip_id != RTL8851B) return 0; ret = rtw89_pci_read_config_byte(rtwdev, RTW89_PCIE_PHY_RATE, &val8); @@ -2107,7 +2114,9 @@ static void rtw89_pci_rxdma_prefth(struct rtw89_dev *rtwdev) static void rtw89_pci_l1off_pwroff(struct rtw89_dev *rtwdev) { - if (rtwdev->chip->chip_id != RTL8852A && rtwdev->chip->chip_id != RTL8852B) + enum rtw89_core_chip_id chip_id = rtwdev->chip->chip_id; + + if (chip_id != RTL8852A && chip_id != RTL8852B && chip_id != RTL8851B) return; rtw89_write32_clr(rtwdev, R_AX_PCIE_PS_CTRL, B_AX_L1OFF_PWR_OFF_EN); @@ -2135,7 +2144,9 @@ static u32 rtw89_pci_l2_rxen_lat(struct rtw89_dev *rtwdev) static void rtw89_pci_aphy_pwrcut(struct rtw89_dev *rtwdev) { - if (rtwdev->chip->chip_id != RTL8852A && rtwdev->chip->chip_id != RTL8852B) + enum rtw89_core_chip_id chip_id = rtwdev->chip->chip_id; + + if (chip_id != RTL8852A && chip_id != RTL8852B && chip_id != RTL8851B) return; rtw89_write32_clr(rtwdev, R_AX_SYS_PW_CTRL, B_AX_PSUS_OFF_CAPC_EN); @@ -2143,8 +2154,9 @@ static void rtw89_pci_aphy_pwrcut(struct rtw89_dev *rtwdev) static void rtw89_pci_hci_ldo(struct rtw89_dev *rtwdev) { - if (rtwdev->chip->chip_id == RTL8852A || - rtwdev->chip->chip_id == RTL8852B) { + enum rtw89_core_chip_id chip_id = rtwdev->chip->chip_id; + + if (chip_id == RTL8852A || chip_id == RTL8852B || chip_id == RTL8851B) { rtw89_write32_set(rtwdev, R_AX_SYS_SDIO_CTRL, B_AX_PCIE_DIS_L2_CTRL_LDO_HCI); rtw89_write32_clr(rtwdev, R_AX_SYS_SDIO_CTRL, @@ -2157,7 +2169,9 @@ static void rtw89_pci_hci_ldo(struct rtw89_dev *rtwdev) static int rtw89_pci_dphy_delay(struct rtw89_dev *rtwdev) { - if (rtwdev->chip->chip_id != RTL8852B) + enum rtw89_core_chip_id chip_id = rtwdev->chip->chip_id; + + if (chip_id != RTL8852B && chip_id != RTL8851B) return 0; return rtw89_write16_mdio_mask(rtwdev, RAC_REG_REV2, BAC_CMU_EN_DLY_MASK, @@ -3397,7 +3411,7 @@ static void rtw89_pci_clkreq_set(struct rtw89_dev *rtwdev, bool enable) if (ret) rtw89_err(rtwdev, "failed to set CLKREQ Delay\n"); - if (chip_id == RTL8852A || chip_id == RTL8852B) { + if (chip_id == RTL8852A || chip_id == RTL8852B || chip_id == RTL8851B) { if (enable) ret = rtw89_pci_config_byte_set(rtwdev, RTW89_PCIE_L1_CTRL, @@ -3442,7 +3456,7 @@ static void rtw89_pci_aspm_set(struct rtw89_dev *rtwdev, bool enable) if (ret) rtw89_err(rtwdev, "failed to read ASPM Delay\n"); - if (chip_id == RTL8852A || chip_id == RTL8852B) { + if (chip_id == RTL8852A || chip_id == RTL8852B || chip_id == RTL8851B) { if (enable) ret = rtw89_pci_config_byte_set(rtwdev, RTW89_PCIE_L1_CTRL, @@ -3522,7 +3536,7 @@ static void rtw89_pci_l1ss_set(struct rtw89_dev *rtwdev, bool enable) enum rtw89_core_chip_id chip_id = rtwdev->chip->chip_id; int ret; - if (chip_id == RTL8852A || chip_id == RTL8852B) { + if (chip_id == RTL8852A || chip_id == RTL8852B || chip_id == RTL8851B) { if (enable) ret = rtw89_pci_config_byte_set(rtwdev, RTW89_PCIE_TIMER_CTRL, @@ -3721,7 +3735,7 @@ static int __maybe_unused rtw89_pci_suspend(struct device *dev) rtw89_write32_set(rtwdev, R_AX_RSV_CTRL, B_AX_WLOCK_1C_BIT6); rtw89_write32_set(rtwdev, R_AX_RSV_CTRL, B_AX_R_DIS_PRST); rtw89_write32_clr(rtwdev, R_AX_RSV_CTRL, B_AX_WLOCK_1C_BIT6); - if (chip_id == RTL8852A || chip_id == RTL8852B) { + if (chip_id == RTL8852A || chip_id == RTL8852B || chip_id == RTL8851B) { rtw89_write32_clr(rtwdev, R_AX_SYS_SDIO_CTRL, B_AX_PCIE_DIS_L2_CTRL_LDO_HCI); rtw89_write32_set(rtwdev, R_AX_PCIE_INIT_CFG1, @@ -3755,7 +3769,7 @@ static int __maybe_unused rtw89_pci_resume(struct device *dev) rtw89_write32_set(rtwdev, R_AX_RSV_CTRL, B_AX_WLOCK_1C_BIT6); rtw89_write32_clr(rtwdev, R_AX_RSV_CTRL, B_AX_R_DIS_PRST); rtw89_write32_clr(rtwdev, R_AX_RSV_CTRL, B_AX_WLOCK_1C_BIT6); - if (chip_id == RTL8852A || chip_id == RTL8852B) { + if (chip_id == RTL8852A || chip_id == RTL8852B || chip_id == RTL8851B) { rtw89_write32_set(rtwdev, R_AX_SYS_SDIO_CTRL, B_AX_PCIE_DIS_L2_CTRL_LDO_HCI); rtw89_write32_clr(rtwdev, R_AX_PCIE_INIT_CFG1, diff --git a/drivers/net/wireless/realtek/rtw89/pci.h b/drivers/net/wireless/realtek/rtw89/pci.h index 1e19740db8c5..0e4bd210b100 100644 --- a/drivers/net/wireless/realtek/rtw89/pci.h +++ b/drivers/net/wireless/realtek/rtw89/pci.h @@ -1004,9 +1004,9 @@ rtw89_pci_rxbd_increase(struct rtw89_pci_rx_ring *rx_ring, u32 cnt) static inline struct rtw89_pci_tx_data *RTW89_PCI_TX_SKB_CB(struct sk_buff *skb) { - struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb); + struct rtw89_tx_skb_data *data = RTW89_TX_SKB_CB(skb); - return (struct rtw89_pci_tx_data *)info->status.status_driver_data; + return (struct rtw89_pci_tx_data *)data->hci_priv; } static inline struct rtw89_pci_tx_bd_32 * diff --git a/drivers/net/wireless/realtek/rtw89/phy.c b/drivers/net/wireless/realtek/rtw89/phy.c index cb0f6cc51d6b..c7e906123416 100644 --- a/drivers/net/wireless/realtek/rtw89/phy.c +++ b/drivers/net/wireless/realtek/rtw89/phy.c @@ -1617,29 +1617,35 @@ static u8 rtw89_channel_to_idx(struct rtw89_dev *rtwdev, u8 band, u8 channel) s8 rtw89_phy_read_txpwr_limit(struct rtw89_dev *rtwdev, u8 band, u8 bw, u8 ntx, u8 rs, u8 bf, u8 ch) { - const struct rtw89_chip_info *chip = rtwdev->chip; + const struct rtw89_rfe_parms *rfe_parms = rtwdev->rfe_parms; + const struct rtw89_txpwr_rule_2ghz *rule_2ghz = &rfe_parms->rule_2ghz; + const struct rtw89_txpwr_rule_5ghz *rule_5ghz = &rfe_parms->rule_5ghz; + const struct rtw89_txpwr_rule_6ghz *rule_6ghz = &rfe_parms->rule_6ghz; u8 ch_idx = rtw89_channel_to_idx(rtwdev, band, ch); u8 regd = rtw89_regd_get(rtwdev, band); s8 lmt = 0, sar; switch (band) { case RTW89_BAND_2G: - lmt = (*chip->txpwr_lmt_2g)[bw][ntx][rs][bf][regd][ch_idx]; - if (!lmt) - lmt = (*chip->txpwr_lmt_2g)[bw][ntx][rs][bf] - [RTW89_WW][ch_idx]; + lmt = (*rule_2ghz->lmt)[bw][ntx][rs][bf][regd][ch_idx]; + if (lmt) + break; + + lmt = (*rule_2ghz->lmt)[bw][ntx][rs][bf][RTW89_WW][ch_idx]; break; case RTW89_BAND_5G: - lmt = (*chip->txpwr_lmt_5g)[bw][ntx][rs][bf][regd][ch_idx]; - if (!lmt) - lmt = (*chip->txpwr_lmt_5g)[bw][ntx][rs][bf] - [RTW89_WW][ch_idx]; + lmt = (*rule_5ghz->lmt)[bw][ntx][rs][bf][regd][ch_idx]; + if (lmt) + break; + + lmt = (*rule_5ghz->lmt)[bw][ntx][rs][bf][RTW89_WW][ch_idx]; break; case RTW89_BAND_6G: - lmt = (*chip->txpwr_lmt_6g)[bw][ntx][rs][bf][regd][ch_idx]; - if (!lmt) - lmt = (*chip->txpwr_lmt_6g)[bw][ntx][rs][bf] - [RTW89_WW][ch_idx]; + lmt = (*rule_6ghz->lmt)[bw][ntx][rs][bf][regd][ch_idx]; + if (lmt) + break; + + lmt = (*rule_6ghz->lmt)[bw][ntx][rs][bf][RTW89_WW][ch_idx]; break; default: rtw89_warn(rtwdev, "unknown band type: %d\n", band); @@ -1862,29 +1868,35 @@ void rtw89_phy_fill_txpwr_limit(struct rtw89_dev *rtwdev, static s8 rtw89_phy_read_txpwr_limit_ru(struct rtw89_dev *rtwdev, u8 band, u8 ru, u8 ntx, u8 ch) { - const struct rtw89_chip_info *chip = rtwdev->chip; + const struct rtw89_rfe_parms *rfe_parms = rtwdev->rfe_parms; + const struct rtw89_txpwr_rule_2ghz *rule_2ghz = &rfe_parms->rule_2ghz; + const struct rtw89_txpwr_rule_5ghz *rule_5ghz = &rfe_parms->rule_5ghz; + const struct rtw89_txpwr_rule_6ghz *rule_6ghz = &rfe_parms->rule_6ghz; u8 ch_idx = rtw89_channel_to_idx(rtwdev, band, ch); u8 regd = rtw89_regd_get(rtwdev, band); s8 lmt_ru = 0, sar; switch (band) { case RTW89_BAND_2G: - lmt_ru = (*chip->txpwr_lmt_ru_2g)[ru][ntx][regd][ch_idx]; - if (!lmt_ru) - lmt_ru = (*chip->txpwr_lmt_ru_2g)[ru][ntx] - [RTW89_WW][ch_idx]; + lmt_ru = (*rule_2ghz->lmt_ru)[ru][ntx][regd][ch_idx]; + if (lmt_ru) + break; + + lmt_ru = (*rule_2ghz->lmt_ru)[ru][ntx][RTW89_WW][ch_idx]; break; case RTW89_BAND_5G: - lmt_ru = (*chip->txpwr_lmt_ru_5g)[ru][ntx][regd][ch_idx]; - if (!lmt_ru) - lmt_ru = (*chip->txpwr_lmt_ru_5g)[ru][ntx] - [RTW89_WW][ch_idx]; + lmt_ru = (*rule_5ghz->lmt_ru)[ru][ntx][regd][ch_idx]; + if (lmt_ru) + break; + + lmt_ru = (*rule_5ghz->lmt_ru)[ru][ntx][RTW89_WW][ch_idx]; break; case RTW89_BAND_6G: - lmt_ru = (*chip->txpwr_lmt_ru_6g)[ru][ntx][regd][ch_idx]; - if (!lmt_ru) - lmt_ru = (*chip->txpwr_lmt_ru_6g)[ru][ntx] - [RTW89_WW][ch_idx]; + lmt_ru = (*rule_6ghz->lmt_ru)[ru][ntx][regd][ch_idx]; + if (lmt_ru) + break; + + lmt_ru = (*rule_6ghz->lmt_ru)[ru][ntx][RTW89_WW][ch_idx]; break; default: rtw89_warn(rtwdev, "unknown band type: %d\n", band); @@ -2405,7 +2417,6 @@ static void rtw89_dcfo_comp(struct rtw89_dev *rtwdev, s32 curr_cfo) bool is_linked = rtwdev->total_sta_assoc > 0; s32 cfo_avg_312; s32 dcfo_comp_val; - u8 dcfo_comp_sft = rtwdev->chip->dcfo_comp_sft; int sign; if (!is_linked) { @@ -2418,8 +2429,8 @@ static void rtw89_dcfo_comp(struct rtw89_dev *rtwdev, s32 curr_cfo) return; dcfo_comp_val = rtw89_phy_read32_mask(rtwdev, R_DCFO, B_DCFO); sign = curr_cfo > 0 ? 1 : -1; - cfo_avg_312 = (curr_cfo << dcfo_comp_sft) / 5 + sign * dcfo_comp_val; - rtw89_debug(rtwdev, RTW89_DBG_CFO, "DCFO: avg_cfo=%d\n", cfo_avg_312); + cfo_avg_312 = curr_cfo / 625 + sign * dcfo_comp_val; + rtw89_debug(rtwdev, RTW89_DBG_CFO, "avg_cfo_312=%d step\n", cfo_avg_312); if (rtwdev->chip->chip_id == RTL8852A && rtwdev->hal.cv == CHIP_CBV) cfo_avg_312 = -cfo_avg_312; rtw89_phy_set_phy_regs(rtwdev, dcfo_comp->addr, dcfo_comp->mask, @@ -2428,9 +2439,16 @@ static void rtw89_dcfo_comp(struct rtw89_dev *rtwdev, s32 curr_cfo) static void rtw89_dcfo_comp_init(struct rtw89_dev *rtwdev) { + const struct rtw89_chip_info *chip = rtwdev->chip; + rtw89_phy_set_phy_regs(rtwdev, R_DCFO_OPT, B_DCFO_OPT_EN, 1); rtw89_phy_set_phy_regs(rtwdev, R_DCFO_WEIGHT, B_DCFO_WEIGHT_MSK, 8); - rtw89_write32_clr(rtwdev, R_AX_PWR_UL_CTRL2, B_AX_PWR_UL_CFO_MASK); + + if (chip->cfo_hw_comp) + rtw89_write32_mask(rtwdev, R_AX_PWR_UL_CTRL2, + B_AX_PWR_UL_CFO_MASK, 0x6); + else + rtw89_write32_clr(rtwdev, R_AX_PWR_UL_CTRL2, B_AX_PWR_UL_CFO_MASK); } static void rtw89_phy_cfo_init(struct rtw89_dev *rtwdev) @@ -2500,6 +2518,7 @@ static void rtw89_phy_cfo_crystal_cap_adjust(struct rtw89_dev *rtwdev, static s32 rtw89_phy_average_cfo_calc(struct rtw89_dev *rtwdev) { + const struct rtw89_chip_info *chip = rtwdev->chip; struct rtw89_cfo_tracking_info *cfo = &rtwdev->cfo_tracking; s32 cfo_khz_all = 0; s32 cfo_cnt_all = 0; @@ -2516,6 +2535,8 @@ static s32 rtw89_phy_average_cfo_calc(struct rtw89_dev *rtwdev) cfo_cnt_all += cfo->cfo_cnt[i]; cfo_all_avg = phy_div(cfo_khz_all, cfo_cnt_all); cfo->pre_cfo_avg[i] = cfo->cfo_avg[i]; + cfo->dcfo_avg = phy_div(cfo_khz_all << chip->dcfo_comp_sft, + cfo_cnt_all); } rtw89_debug(rtwdev, RTW89_DBG_CFO, "CFO track for macid = %d\n", i); @@ -2642,7 +2663,9 @@ static void rtw89_phy_cfo_dm(struct rtw89_dev *rtwdev) s32 new_cfo = 0; bool x_cap_update = false; u8 pre_x_cap = cfo->crystal_cap; + u8 dcfo_comp_sft = rtwdev->chip->dcfo_comp_sft; + cfo->dcfo_avg = 0; rtw89_debug(rtwdev, RTW89_DBG_CFO, "CFO:total_sta_assoc=%d\n", rtwdev->total_sta_assoc); if (rtwdev->total_sta_assoc == 0) { @@ -2684,18 +2707,19 @@ static void rtw89_phy_cfo_dm(struct rtw89_dev *rtwdev) rtw89_phy_cfo_crystal_cap_adjust(rtwdev, new_cfo); cfo->cfo_avg_pre = new_cfo; + cfo->dcfo_avg_pre = cfo->dcfo_avg; x_cap_update = cfo->crystal_cap != pre_x_cap; rtw89_debug(rtwdev, RTW89_DBG_CFO, "Xcap_up=%d\n", x_cap_update); rtw89_debug(rtwdev, RTW89_DBG_CFO, "Xcap: D:%x C:%x->%x, ofst=%d\n", cfo->def_x_cap, pre_x_cap, cfo->crystal_cap, cfo->x_cap_ofst); if (x_cap_update) { - if (new_cfo > 0) - new_cfo -= CFO_SW_COMP_FINE_TUNE; + if (cfo->dcfo_avg > 0) + cfo->dcfo_avg -= CFO_SW_COMP_FINE_TUNE << dcfo_comp_sft; else - new_cfo += CFO_SW_COMP_FINE_TUNE; + cfo->dcfo_avg += CFO_SW_COMP_FINE_TUNE << dcfo_comp_sft; } - rtw89_dcfo_comp(rtwdev, new_cfo); + rtw89_dcfo_comp(rtwdev, cfo->dcfo_avg); rtw89_phy_cfo_statistics_reset(rtwdev); } diff --git a/drivers/net/wireless/realtek/rtw89/ps.c b/drivers/net/wireless/realtek/rtw89/ps.c index 40498812205e..fa94335f699a 100644 --- a/drivers/net/wireless/realtek/rtw89/ps.c +++ b/drivers/net/wireless/realtek/rtw89/ps.c @@ -114,7 +114,8 @@ void rtw89_leave_ps_mode(struct rtw89_dev *rtwdev) __rtw89_leave_ps_mode(rtwdev); } -void rtw89_enter_lps(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif) +void rtw89_enter_lps(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif, + bool ps_mode) { lockdep_assert_held(&rtwdev->mutex); @@ -122,7 +123,8 @@ void rtw89_enter_lps(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif) return; __rtw89_enter_lps(rtwdev, rtwvif->mac_id); - __rtw89_enter_ps_mode(rtwdev, rtwvif); + if (ps_mode) + __rtw89_enter_ps_mode(rtwdev, rtwvif); } static void rtw89_leave_lps_vif(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif) @@ -155,6 +157,9 @@ void rtw89_enter_ips(struct rtw89_dev *rtwdev) set_bit(RTW89_FLAG_INACTIVE_PS, rtwdev->flags); + if (!test_bit(RTW89_FLAG_POWERON, rtwdev->flags)) + return; + rtw89_for_each_rtwvif(rtwdev, rtwvif) rtw89_mac_vif_deinit(rtwdev, rtwvif); @@ -166,6 +171,9 @@ void rtw89_leave_ips(struct rtw89_dev *rtwdev) struct rtw89_vif *rtwvif; int ret; + if (test_bit(RTW89_FLAG_POWERON, rtwdev->flags)) + return; + ret = rtw89_core_start(rtwdev); if (ret) rtw89_err(rtwdev, "failed to leave idle state\n"); diff --git a/drivers/net/wireless/realtek/rtw89/ps.h b/drivers/net/wireless/realtek/rtw89/ps.h index 6ac1f7ea5339..73c008db0426 100644 --- a/drivers/net/wireless/realtek/rtw89/ps.h +++ b/drivers/net/wireless/realtek/rtw89/ps.h @@ -5,7 +5,8 @@ #ifndef __RTW89_PS_H_ #define __RTW89_PS_H_ -void rtw89_enter_lps(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif); +void rtw89_enter_lps(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif, + bool ps_mode); void rtw89_leave_lps(struct rtw89_dev *rtwdev); void __rtw89_leave_ps_mode(struct rtw89_dev *rtwdev); void __rtw89_enter_ps_mode(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif); @@ -15,4 +16,20 @@ void rtw89_leave_ips(struct rtw89_dev *rtwdev); void rtw89_set_coex_ctrl_lps(struct rtw89_dev *rtwdev, bool btc_ctrl); void rtw89_process_p2p_ps(struct rtw89_dev *rtwdev, struct ieee80211_vif *vif); +static inline void rtw89_leave_ips_by_hwflags(struct rtw89_dev *rtwdev) +{ + struct ieee80211_hw *hw = rtwdev->hw; + + if (hw->conf.flags & IEEE80211_CONF_IDLE) + rtw89_leave_ips(rtwdev); +} + +static inline void rtw89_enter_ips_by_hwflags(struct rtw89_dev *rtwdev) +{ + struct ieee80211_hw *hw = rtwdev->hw; + + if (hw->conf.flags & IEEE80211_CONF_IDLE) + rtw89_enter_ips(rtwdev); +} + #endif diff --git a/drivers/net/wireless/realtek/rtw89/reg.h b/drivers/net/wireless/realtek/rtw89/reg.h index 37a1777a25a3..266e4231b5f3 100644 --- a/drivers/net/wireless/realtek/rtw89/reg.h +++ b/drivers/net/wireless/realtek/rtw89/reg.h @@ -129,6 +129,7 @@ #define R_AX_PLATFORM_ENABLE 0x0088 #define B_AX_AXIDMA_EN BIT(3) +#define B_AX_APB_WRAP_EN BIT(2) #define B_AX_WCPU_EN BIT(1) #define B_AX_PLATFORM_EN BIT(0) @@ -488,6 +489,7 @@ #define B_AX_DISPATCHER_CLK_EN BIT(18) #define B_AX_BBRPT_CLK_EN BIT(17) #define B_AX_MAC_SEC_CLK_EN BIT(16) +#define B_AX_AXIDMA_CLK_EN BIT(9) #define PCI_LTR_IDLE_TIMER_1US 0 #define PCI_LTR_IDLE_TIMER_10US 1 @@ -1586,6 +1588,7 @@ #define R_AX_PL_BUF_STATUS 0x9824 #define B_AX_WD_BUF_STAT_DONE BIT(31) #define B_AX_WD_BUF_STAT_PKTID_MASK GENMASK(11, 0) +#define S_WD_BUF_STAT_PKTID_INVALID GENMASK(11, 0) #define R_AX_WD_CPUQ_OP_0 0x9810 #define R_AX_PL_CPUQ_OP_0 0x9830 diff --git a/drivers/net/wireless/realtek/rtw89/rtw8851b_rfk_table.c b/drivers/net/wireless/realtek/rtw89/rtw8851b_rfk_table.c new file mode 100644 index 000000000000..0abf7978ccab --- /dev/null +++ b/drivers/net/wireless/realtek/rtw89/rtw8851b_rfk_table.c @@ -0,0 +1,534 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2022-2023 Realtek Corporation + */ + +#include "rtw8851b_rfk_table.h" + +static const struct rtw89_reg5_def rtw8851b_dadck_setup_defs[] = { + RTW89_DECL_RFK_WM(0xc210, 0x003fc000, 0x80), + RTW89_DECL_RFK_WM(0xc224, 0x003fc000, 0x80), + RTW89_DECL_RFK_WM(0xc0f8, 0x30000000, 0x3), + RTW89_DECL_RFK_WM(0x12b8, BIT(30), 0x1), + RTW89_DECL_RFK_WM(0x030c, 0x1f000000, 0x1f), + RTW89_DECL_RFK_WM(0x032c, 0xc0000000, 0x0), + RTW89_DECL_RFK_WM(0x032c, BIT(22), 0x0), + RTW89_DECL_RFK_WM(0x032c, BIT(22), 0x1), + RTW89_DECL_RFK_WM(0x032c, BIT(16), 0x0), + RTW89_DECL_RFK_WM(0x032c, BIT(20), 0x1), + RTW89_DECL_RFK_WM(0x030c, 0x0f000000, 0x3), + RTW89_DECL_RFK_WM(0xc0f4, BIT(2), 0x0), + RTW89_DECL_RFK_WM(0xc0f4, BIT(4), 0x0), + RTW89_DECL_RFK_WM(0xc0f4, BIT(11), 0x1), + RTW89_DECL_RFK_WM(0xc0f4, BIT(11), 0x0), + RTW89_DECL_RFK_DELAY(1), + RTW89_DECL_RFK_WM(0xc0f4, 0x300, 0x1), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_dadck_setup_defs); + +static const struct rtw89_reg5_def rtw8851b_dadck_post_defs[] = { + RTW89_DECL_RFK_WM(0x032c, BIT(16), 0x1), + RTW89_DECL_RFK_WM(0x032c, BIT(20), 0x0), + RTW89_DECL_RFK_WM(0x030c, 0x1f000000, 0xc), + RTW89_DECL_RFK_WM(0x032c, 0xc0000000, 0x1), + RTW89_DECL_RFK_WM(0x12b8, BIT(30), 0x0), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_dadck_post_defs); + +static const struct rtw89_reg5_def rtw8851b_dack_s0_1_defs[] = { + RTW89_DECL_RFK_WM(0x12a0, BIT(15), 0x1), + RTW89_DECL_RFK_WM(0x12a0, 0x7000, 0x3), + RTW89_DECL_RFK_WM(0x12b8, BIT(30), 0x1), + RTW89_DECL_RFK_WM(0x030c, BIT(28), 0x1), + RTW89_DECL_RFK_WM(0x032c, 0x80000000, 0x0), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_dack_s0_1_defs); + +static const struct rtw89_reg5_def rtw8851b_dack_s0_2_defs[] = { + RTW89_DECL_RFK_WM(0xc004, BIT(0), 0x0), + RTW89_DECL_RFK_WM(0x12a0, BIT(15), 0x0), + RTW89_DECL_RFK_WM(0x12a0, 0x7000, 0x7), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_dack_s0_2_defs); + +static const struct rtw89_reg5_def rtw8851b_dack_manual_off_defs[] = { + RTW89_DECL_RFK_WM(0xc0f8, 0x30000000, 0x0), + RTW89_DECL_RFK_WM(0xc210, BIT(0), 0x0), + RTW89_DECL_RFK_WM(0xc224, BIT(0), 0x0), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_dack_manual_off_defs); + +static const struct rtw89_reg5_def rtw8851b_iqk_rxclk_80_defs[] = { + RTW89_DECL_RFK_WM(0x20fc, 0xffff0000, 0x0101), + RTW89_DECL_RFK_WM(0x5670, 0x00002000, 0x1), + RTW89_DECL_RFK_WM(0x12a0, 0x00080000, 0x1), + RTW89_DECL_RFK_WM(0x12a0, 0x00070000, 0x2), + RTW89_DECL_RFK_WM(0x5670, 0x60000000, 0x1), + RTW89_DECL_RFK_WM(0xc0d4, 0x00000780, 0x8), + RTW89_DECL_RFK_WM(0xc0d4, 0x00007800, 0x2), + RTW89_DECL_RFK_WM(0xc0d4, 0x0c000000, 0x2), + RTW89_DECL_RFK_WM(0xc0d8, 0x000001e0, 0x5), + RTW89_DECL_RFK_WM(0xc0c4, 0x003e0000, 0xf), + RTW89_DECL_RFK_WM(0xc0ec, 0x00006000, 0x0), + RTW89_DECL_RFK_WM(0x12b8, 0x40000000, 0x1), + RTW89_DECL_RFK_WM(0x030c, 0xff000000, 0x0f), + RTW89_DECL_RFK_WM(0x030c, 0xff000000, 0x03), + RTW89_DECL_RFK_WM(0x032c, 0xffff0000, 0x0001), + RTW89_DECL_RFK_WM(0x032c, 0xffff0000, 0x0041), + RTW89_DECL_RFK_WM(0x20fc, 0xffff0000, 0x1101), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_iqk_rxclk_80_defs); + +static const struct rtw89_reg5_def rtw8851b_iqk_rxclk_others_defs[] = { + RTW89_DECL_RFK_WM(0x20fc, 0xffff0000, 0x0101), + RTW89_DECL_RFK_WM(0x5670, 0x00002000, 0x1), + RTW89_DECL_RFK_WM(0x12a0, 0x00080000, 0x1), + RTW89_DECL_RFK_WM(0x12a0, 0x00070000, 0x2), + RTW89_DECL_RFK_WM(0x5670, 0x60000000, 0x0), + RTW89_DECL_RFK_WM(0xc0d4, 0x00000780, 0x8), + RTW89_DECL_RFK_WM(0xc0d4, 0x00007800, 0x2), + RTW89_DECL_RFK_WM(0xc0d4, 0x0c000000, 0x2), + RTW89_DECL_RFK_WM(0xc0d8, 0x000001e0, 0x5), + RTW89_DECL_RFK_WM(0xc0c4, 0x003e0000, 0xf), + RTW89_DECL_RFK_WM(0xc0ec, 0x00006000, 0x2), + RTW89_DECL_RFK_WM(0x12b8, 0x40000000, 0x1), + RTW89_DECL_RFK_WM(0x030c, 0xff000000, 0x0f), + RTW89_DECL_RFK_WM(0x030c, 0xff000000, 0x03), + RTW89_DECL_RFK_WM(0x032c, 0xffff0000, 0x0001), + RTW89_DECL_RFK_WM(0x032c, 0xffff0000, 0x0041), + RTW89_DECL_RFK_WM(0x20fc, 0xffff0000, 0x1101), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_iqk_rxclk_others_defs); + +static const struct rtw89_reg5_def rtw8851b_iqk_txk_2ghz_defs[] = { + RTW89_DECL_RFK_WRF(RF_PATH_A, 0x51, 0x80000, 0x0), + RTW89_DECL_RFK_WRF(RF_PATH_A, 0x51, 0x00800, 0x0), + RTW89_DECL_RFK_WRF(RF_PATH_A, 0x52, 0x00800, 0x0), + RTW89_DECL_RFK_WRF(RF_PATH_A, 0x55, 0x0001f, 0x4), + RTW89_DECL_RFK_WRF(RF_PATH_A, 0xef, 0x00004, 0x1), + RTW89_DECL_RFK_WRF(RF_PATH_A, 0x00, 0xffff0, 0x403e), + RTW89_DECL_RFK_WRF(RF_PATH_A, 0x11, 0x00003, 0x0), + RTW89_DECL_RFK_WRF(RF_PATH_A, 0x11, 0x00070, 0x6), + RTW89_DECL_RFK_WRF(RF_PATH_A, 0x11, 0x1f000, 0x10), + RTW89_DECL_RFK_DELAY(1), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_iqk_txk_2ghz_defs); + +static const struct rtw89_reg5_def rtw8851b_iqk_txk_5ghz_defs[] = { + RTW89_DECL_RFK_WRF(RF_PATH_A, 0x60, 0x00007, 0x0), + RTW89_DECL_RFK_WRF(RF_PATH_A, 0x55, 0x0001f, 0x4), + RTW89_DECL_RFK_WRF(RF_PATH_A, 0xef, 0x00004, 0x1), + RTW89_DECL_RFK_WRF(RF_PATH_A, 0x00, 0xffff0, 0x403e), + RTW89_DECL_RFK_WRF(RF_PATH_A, 0x11, 0x00003, 0x0), + RTW89_DECL_RFK_WRF(RF_PATH_A, 0x11, 0x00070, 0x7), + RTW89_DECL_RFK_WRF(RF_PATH_A, 0x11, 0x1f000, 0x7), + RTW89_DECL_RFK_DELAY(1), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_iqk_txk_5ghz_defs); + +static const struct rtw89_reg5_def rtw8851b_iqk_afebb_restore_defs[] = { + RTW89_DECL_RFK_WM(0x12b8, 0x40000000, 0x0), + RTW89_DECL_RFK_WM(0x20fc, 0x00010000, 0x1), + RTW89_DECL_RFK_WM(0x20fc, 0x00100000, 0x0), + RTW89_DECL_RFK_WM(0x20fc, 0x01000000, 0x1), + RTW89_DECL_RFK_WM(0x20fc, 0x10000000, 0x0), + RTW89_DECL_RFK_WM(0x5670, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x12a0, 0x000ff000, 0x00), + RTW89_DECL_RFK_WM(0x20fc, 0x00010000, 0x0), + RTW89_DECL_RFK_WM(0x20fc, 0x01000000, 0x0), + RTW89_DECL_RFK_WRF(RF_PATH_A, 0x10005, 0x00001, 0x1), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_iqk_afebb_restore_defs); + +static const struct rtw89_reg5_def rtw8851b_iqk_macbb_defs[] = { + RTW89_DECL_RFK_WRF(RF_PATH_A, 0x10005, 0x00001, 0x0), + RTW89_DECL_RFK_WM(0x20fc, 0x00010000, 0x1), + RTW89_DECL_RFK_WM(0x20fc, 0x00100000, 0x0), + RTW89_DECL_RFK_WM(0x20fc, 0x01000000, 0x1), + RTW89_DECL_RFK_WM(0x20fc, 0x10000000, 0x0), + RTW89_DECL_RFK_WM(0x5670, MASKDWORD, 0xf801fffd), + RTW89_DECL_RFK_WM(0x5670, 0x00004000, 0x1), + RTW89_DECL_RFK_WM(0x12a0, 0x00008000, 0x1), + RTW89_DECL_RFK_WM(0x5670, 0x80000000, 0x1), + RTW89_DECL_RFK_WM(0x12a0, 0x00007000, 0x7), + RTW89_DECL_RFK_WM(0x5670, 0x00002000, 0x1), + RTW89_DECL_RFK_WM(0x12a0, 0x00080000, 0x1), + RTW89_DECL_RFK_WM(0x12a0, 0x00070000, 0x3), + RTW89_DECL_RFK_WM(0x5670, 0x60000000, 0x2), + RTW89_DECL_RFK_WM(0xc0d4, 0x00000780, 0x9), + RTW89_DECL_RFK_WM(0xc0d4, 0x00007800, 0x1), + RTW89_DECL_RFK_WM(0xc0d4, 0x0c000000, 0x0), + RTW89_DECL_RFK_WM(0xc0d8, 0x000001e0, 0x3), + RTW89_DECL_RFK_WM(0xc0c4, 0x003e0000, 0xa), + RTW89_DECL_RFK_WM(0xc0ec, 0x00006000, 0x0), + RTW89_DECL_RFK_WM(0xc0e8, 0x00000040, 0x1), + RTW89_DECL_RFK_WM(0x12b8, 0x40000000, 0x1), + RTW89_DECL_RFK_WM(0x030c, 0xff000000, 0x1f), + RTW89_DECL_RFK_WM(0x030c, 0xff000000, 0x13), + RTW89_DECL_RFK_WM(0x032c, 0xffff0000, 0x0001), + RTW89_DECL_RFK_WM(0x032c, 0xffff0000, 0x0041), + RTW89_DECL_RFK_WM(0x20fc, 0x00100000, 0x1), + RTW89_DECL_RFK_WM(0x20fc, 0x10000000, 0x1), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_iqk_macbb_defs); + +static const struct rtw89_reg5_def rtw8851b_iqk_bb_afe_defs[] = { + RTW89_DECL_RFK_WM(0x5670, 0x00004000, 0x1), + RTW89_DECL_RFK_WM(0x12a0, 0x00008000, 0x1), + RTW89_DECL_RFK_WM(0x5670, 0x80000000, 0x1), + RTW89_DECL_RFK_WM(0x12a0, 0x00007000, 0x7), + RTW89_DECL_RFK_WM(0x5670, 0x00002000, 0x1), + RTW89_DECL_RFK_WM(0x12a0, 0x00080000, 0x1), + RTW89_DECL_RFK_WM(0x12a0, 0x00070000, 0x3), + RTW89_DECL_RFK_WM(0x5670, 0x60000000, 0x2), + RTW89_DECL_RFK_WM(0xc0d4, 0x00000780, 0x9), + RTW89_DECL_RFK_WM(0xc0d4, 0x00007800, 0x1), + RTW89_DECL_RFK_WM(0xc0d4, 0x0c000000, 0x0), + RTW89_DECL_RFK_WM(0xc0d8, 0x000001e0, 0x3), + RTW89_DECL_RFK_WM(0xc0c4, 0x003e0000, 0xa), + RTW89_DECL_RFK_WM(0xc0ec, 0x00006000, 0x0), + RTW89_DECL_RFK_WM(0xc0e8, 0x00000040, 0x1), + RTW89_DECL_RFK_WM(0x12b8, 0x40000000, 0x1), + RTW89_DECL_RFK_WM(0x030c, MASKBYTE3, 0x1f), + RTW89_DECL_RFK_WM(0x030c, MASKBYTE3, 0x13), + RTW89_DECL_RFK_WM(0x032c, MASKHWORD, 0x0001), + RTW89_DECL_RFK_WM(0x032c, MASKHWORD, 0x0041), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_iqk_bb_afe_defs); + +static const struct rtw89_reg5_def rtw8851b_tssi_sys_defs[] = { + RTW89_DECL_RFK_WM(0x12bc, 0x000ffff0, 0xb5b5), + RTW89_DECL_RFK_WM(0x32bc, 0x000ffff0, 0xb5b5), + RTW89_DECL_RFK_WM(0x0300, 0xff000000, 0x16), + RTW89_DECL_RFK_WM(0x0304, 0x0000ffff, 0x1f19), + RTW89_DECL_RFK_WM(0x0308, 0xff000000, 0x1c), + RTW89_DECL_RFK_WM(0x0314, 0xffff0000, 0x2041), + RTW89_DECL_RFK_WM(0x0318, 0xffffffff, 0x20012041), + RTW89_DECL_RFK_WM(0x0324, 0xffff0000, 0x2001), + RTW89_DECL_RFK_WM(0x0020, 0x00006000, 0x3), + RTW89_DECL_RFK_WM(0x0024, 0x00006000, 0x3), + RTW89_DECL_RFK_WM(0x0704, 0xffff0000, 0x601e), + RTW89_DECL_RFK_WM(0x2704, 0xffff0000, 0x601e), + RTW89_DECL_RFK_WM(0x0700, 0xf0000000, 0x4), + RTW89_DECL_RFK_WM(0x2700, 0xf0000000, 0x4), + RTW89_DECL_RFK_WM(0x0650, 0x3c000000, 0x0), + RTW89_DECL_RFK_WM(0x2650, 0x3c000000, 0x0), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_tssi_sys_defs); + +static const struct rtw89_reg5_def rtw8851b_tssi_sys_a_defs_2g[] = { + RTW89_DECL_RFK_WM(0x120c, 0x000000ff, 0x33), + RTW89_DECL_RFK_WM(0x12c0, 0x0ff00000, 0x33), + RTW89_DECL_RFK_WM(0x58f8, 0x40000000, 0x1), + RTW89_DECL_RFK_WM(0x5814, 0x20000000, 0x0), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_tssi_sys_a_defs_2g); + +static const struct rtw89_reg5_def rtw8851b_tssi_sys_a_defs_5g[] = { + RTW89_DECL_RFK_WM(0x120c, 0x000000ff, 0x44), + RTW89_DECL_RFK_WM(0x12c0, 0x0ff00000, 0x44), + RTW89_DECL_RFK_WM(0x58f8, 0x40000000, 0x0), + RTW89_DECL_RFK_WM(0x5814, 0x20000000, 0x0), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_tssi_sys_a_defs_5g); + +static const struct rtw89_reg5_def rtw8851b_tssi_init_txpwr_defs_a[] = { + RTW89_DECL_RFK_WM(0x566c, 0x00001000, 0x0), + RTW89_DECL_RFK_WM(0x5800, 0xffffffff, 0x003f807f), + RTW89_DECL_RFK_WM(0x580c, 0x0000007f, 0x40), + RTW89_DECL_RFK_WM(0x580c, 0x0fffff00, 0x00040), + RTW89_DECL_RFK_WM(0x5810, 0xffffffff, 0x59010000), + RTW89_DECL_RFK_WM(0x5814, 0x01ffffff, 0x026d000), + RTW89_DECL_RFK_WM(0x5814, 0xf8000000, 0x00), + RTW89_DECL_RFK_WM(0x5818, 0x00ffffff, 0x2c18e8), + RTW89_DECL_RFK_WM(0x5818, 0x07000000, 0x0), + RTW89_DECL_RFK_WM(0x5818, 0xf0000000, 0x0), + RTW89_DECL_RFK_WM(0x581c, 0x3fffffff, 0x3dc80280), + RTW89_DECL_RFK_WM(0x5820, 0xffffffff, 0x00000080), + RTW89_DECL_RFK_WM(0x58e8, 0x0000003f, 0x04), + RTW89_DECL_RFK_WM(0x580c, 0x10000000, 0x1), + RTW89_DECL_RFK_WM(0x580c, 0x40000000, 0x1), + RTW89_DECL_RFK_WM(0x5834, 0x3fffffff, 0x000115f2), + RTW89_DECL_RFK_WM(0x5838, 0x7fffffff, 0x0000121), + RTW89_DECL_RFK_WM(0x5854, 0x3fffffff, 0x000115f2), + RTW89_DECL_RFK_WM(0x5858, 0x7fffffff, 0x0000121), + RTW89_DECL_RFK_WM(0x5860, 0x80000000, 0x0), + RTW89_DECL_RFK_WM(0x5864, 0x07ffffff, 0x00801ff), + RTW89_DECL_RFK_WM(0x5898, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x589c, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x58a4, 0x000000ff, 0x16), + RTW89_DECL_RFK_WM(0x58b0, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x58b4, 0x7fffffff, 0x0a002000), + RTW89_DECL_RFK_WM(0x58b8, 0x7fffffff, 0x00007628), + RTW89_DECL_RFK_WM(0x58bc, 0x07ffffff, 0x7a7807f), + RTW89_DECL_RFK_WM(0x58c0, 0xfffe0000, 0x003f), + RTW89_DECL_RFK_WM(0x58c4, 0xffffffff, 0x0003ffff), + RTW89_DECL_RFK_WM(0x58c8, 0x00ffffff, 0x000000), + RTW89_DECL_RFK_WM(0x58c8, 0xf0000000, 0x0), + RTW89_DECL_RFK_WM(0x58cc, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x58d0, 0x07ffffff, 0x2008101), + RTW89_DECL_RFK_WM(0x58d4, 0x000000ff, 0x00), + RTW89_DECL_RFK_WM(0x58d4, 0x0003fe00, 0x0ff), + RTW89_DECL_RFK_WM(0x58d4, 0x07fc0000, 0x100), + RTW89_DECL_RFK_WM(0x58d8, 0xffffffff, 0x8008016c), + RTW89_DECL_RFK_WM(0x58dc, 0x0001ffff, 0x0807f), + RTW89_DECL_RFK_WM(0x58dc, 0xfff00000, 0x800), + RTW89_DECL_RFK_WM(0x58f0, 0x0003ffff, 0x001ff), + RTW89_DECL_RFK_WM(0x58f4, 0x000fffff, 0x00000), + RTW89_DECL_RFK_WM(0x58f8, 0x000fffff, 0x00000), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_tssi_init_txpwr_defs_a); + +static const struct rtw89_reg5_def rtw8851b_tssi_init_txpwr_he_tb_defs_a[] = { + RTW89_DECL_RFK_WM(0x58a0, MASKDWORD, 0x000000fe), + RTW89_DECL_RFK_WM(0x58e4, 0x0000007f, 0x1f), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_tssi_init_txpwr_he_tb_defs_a); + +static const struct rtw89_reg5_def rtw8851b_tssi_dck_defs_a[] = { + RTW89_DECL_RFK_WM(0x580c, 0x0fff0000, 0x000), + RTW89_DECL_RFK_WM(0x5814, 0x00001000, 0x1), + RTW89_DECL_RFK_WM(0x5814, 0x00002000, 0x1), + RTW89_DECL_RFK_WM(0x5814, 0x00004000, 0x1), + RTW89_DECL_RFK_WM(0x5814, 0x00038000, 0x3), + RTW89_DECL_RFK_WM(0x5814, 0x003c0000, 0x5), + RTW89_DECL_RFK_WM(0x5814, 0x18000000, 0x0), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_tssi_dck_defs_a); + +static const struct rtw89_reg5_def rtw8851b_tssi_dac_gain_defs_a[] = { + RTW89_DECL_RFK_WM(0x58b0, 0x00000fff, 0x000), + RTW89_DECL_RFK_WM(0x5a00, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a04, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a08, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a0c, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a10, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a14, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a18, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a1c, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a20, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a24, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a28, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a2c, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a30, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a34, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a38, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a3c, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a40, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a44, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a48, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a4c, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a50, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a54, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a58, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a5c, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a60, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a64, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a68, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a6c, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a70, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a74, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a78, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a7c, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a80, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a84, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a88, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a8c, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a90, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a94, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a98, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5a9c, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5aa0, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5aa4, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5aa8, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5aac, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5ab0, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5ab4, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5ab8, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5abc, MASKDWORD, 0x00000000), + RTW89_DECL_RFK_WM(0x5ac0, MASKDWORD, 0x00000000), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_tssi_dac_gain_defs_a); + +static const struct rtw89_reg5_def rtw8851b_tssi_slope_a_defs_2g[] = { + RTW89_DECL_RFK_WM(0x5608, 0x07ffffff, 0x0201008), + RTW89_DECL_RFK_WM(0x560c, 0x07ffffff, 0x0201008), + RTW89_DECL_RFK_WM(0x5610, 0x07ffffff, 0x0200e08), + RTW89_DECL_RFK_WM(0x5614, 0x07ffffff, 0x0201008), + RTW89_DECL_RFK_WM(0x5618, 0x07ffffff, 0x0201008), + RTW89_DECL_RFK_WM(0x561c, 0x000001ff, 0x007), + RTW89_DECL_RFK_WM(0x561c, 0xffff0000, 0x0808), + RTW89_DECL_RFK_WM(0x5620, 0xffffffff, 0x08080808), + RTW89_DECL_RFK_WM(0x5624, 0xffffffff, 0x08080808), + RTW89_DECL_RFK_WM(0x5628, 0xffffffff, 0x08080808), + RTW89_DECL_RFK_WM(0x562c, 0x0000ffff, 0x0808), + RTW89_DECL_RFK_WM(0x581c, 0x00100000, 0x1), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_tssi_slope_a_defs_2g); + +static const struct rtw89_reg5_def rtw8851b_tssi_slope_a_defs_5g[] = { + RTW89_DECL_RFK_WM(0x5608, 0x07ffffff, 0x0201008), + RTW89_DECL_RFK_WM(0x560c, 0x07ffffff, 0x0341a08), + RTW89_DECL_RFK_WM(0x5610, 0x07ffffff, 0x0201417), + RTW89_DECL_RFK_WM(0x5614, 0x07ffffff, 0x0201008), + RTW89_DECL_RFK_WM(0x5618, 0x07ffffff, 0x0201008), + RTW89_DECL_RFK_WM(0x561c, 0x000001ff, 0x008), + RTW89_DECL_RFK_WM(0x561c, 0xffff0000, 0x0808), + RTW89_DECL_RFK_WM(0x5620, 0xffffffff, 0x0e0e0808), + RTW89_DECL_RFK_WM(0x5624, 0xffffffff, 0x08080d18), + RTW89_DECL_RFK_WM(0x5628, 0xffffffff, 0x08080808), + RTW89_DECL_RFK_WM(0x562c, 0x0000ffff, 0x0808), + RTW89_DECL_RFK_WM(0x581c, 0x00100000, 0x1), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_tssi_slope_a_defs_5g); + +static const struct rtw89_reg5_def rtw8851b_tssi_align_a_2g_defs[] = { + RTW89_DECL_RFK_WM(0x5604, 0x80000000, 0x1), + RTW89_DECL_RFK_WM(0x5600, 0x3fffffff, 0x000000), + RTW89_DECL_RFK_WM(0x5604, 0x003fffff, 0x2d2400), + RTW89_DECL_RFK_WM(0x5630, 0x3fffffff, 0x00000000), + RTW89_DECL_RFK_WM(0x5634, 0x000003ff, 0x000), + RTW89_DECL_RFK_WM(0x5634, 0x000ffc00, 0x000), + RTW89_DECL_RFK_WM(0x5634, 0x3ff00000, 0x3fa), + RTW89_DECL_RFK_WM(0x5638, 0x000003ff, 0x02e), + RTW89_DECL_RFK_WM(0x5638, 0x000ffc00, 0x09c), + RTW89_DECL_RFK_WM(0x563c, 0x3fffffff, 0x00000000), + RTW89_DECL_RFK_WM(0x5640, 0x3fffffff, 0x3fb00000), + RTW89_DECL_RFK_WM(0x5644, 0x000003ff, 0x02f), + RTW89_DECL_RFK_WM(0x5644, 0x000ffc00, 0x09c), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_tssi_align_a_2g_defs); + +static const struct rtw89_reg5_def rtw8851b_tssi_align_a_5g_defs[] = { + RTW89_DECL_RFK_WM(0x5604, 0x80000000, 0x1), + RTW89_DECL_RFK_WM(0x5600, 0x3fffffff, 0x000000), + RTW89_DECL_RFK_WM(0x5604, 0x003fffff, 0x3b2d24), + RTW89_DECL_RFK_WM(0x5630, 0x3fffffff, 0x00000000), + RTW89_DECL_RFK_WM(0x5634, 0x000003ff, 0x000), + RTW89_DECL_RFK_WM(0x5634, 0x000ffc00, 0x3cb), + RTW89_DECL_RFK_WM(0x5634, 0x3ff00000, 0x030), + RTW89_DECL_RFK_WM(0x5638, 0x000003ff, 0x73), + RTW89_DECL_RFK_WM(0x5638, 0x000ffc00, 0xd4), + RTW89_DECL_RFK_WM(0x563c, 0x3fffffff, 0x00000000), + RTW89_DECL_RFK_WM(0x5640, 0x3fffffff, 0x00000000), + RTW89_DECL_RFK_WM(0x5644, 0x000fffff, 0x00000), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_tssi_align_a_5g_defs); + +static const struct rtw89_reg5_def rtw8851b_tssi_slope_defs_a[] = { + RTW89_DECL_RFK_WM(0x5820, 0x80000000, 0x0), + RTW89_DECL_RFK_WM(0x5818, 0x10000000, 0x0), + RTW89_DECL_RFK_WM(0x5814, 0x00000800, 0x1), + RTW89_DECL_RFK_WM(0x581c, 0x20000000, 0x1), + RTW89_DECL_RFK_WM(0x5820, 0x0000f000, 0xf), + RTW89_DECL_RFK_WM(0x581c, 0x000003ff, 0x280), + RTW89_DECL_RFK_WM(0x581c, 0x000ffc00, 0x200), + RTW89_DECL_RFK_WM(0x58b8, 0x007f0000, 0x00), + RTW89_DECL_RFK_WM(0x58b8, 0x7f000000, 0x00), + RTW89_DECL_RFK_WM(0x58b4, 0x7f000000, 0x0a), + RTW89_DECL_RFK_WM(0x58b8, 0x0000007f, 0x28), + RTW89_DECL_RFK_WM(0x58b8, 0x00007f00, 0x76), + RTW89_DECL_RFK_WM(0x5810, 0x20000000, 0x0), + RTW89_DECL_RFK_WM(0x580c, 0x10000000, 0x1), + RTW89_DECL_RFK_WM(0x580c, 0x40000000, 0x1), + RTW89_DECL_RFK_WM(0x5834, 0x0003ffff, 0x115f2), + RTW89_DECL_RFK_WM(0x5834, 0x3ffc0000, 0x000), + RTW89_DECL_RFK_WM(0x5838, 0x00000fff, 0x121), + RTW89_DECL_RFK_WM(0x5838, 0x003ff000, 0x000), + RTW89_DECL_RFK_WM(0x5854, 0x0003ffff, 0x115f2), + RTW89_DECL_RFK_WM(0x5854, 0x3ffc0000, 0x000), + RTW89_DECL_RFK_WM(0x5858, 0x00000fff, 0x121), + RTW89_DECL_RFK_WM(0x5858, 0x003ff000, 0x000), + RTW89_DECL_RFK_WM(0x5824, 0x0003ffff, 0x115f2), + RTW89_DECL_RFK_WM(0x5824, 0x3ffc0000, 0x000), + RTW89_DECL_RFK_WM(0x5828, 0x00000fff, 0x121), + RTW89_DECL_RFK_WM(0x5828, 0x003ff000, 0x000), + RTW89_DECL_RFK_WM(0x582c, 0x0003ffff, 0x115f2), + RTW89_DECL_RFK_WM(0x582c, 0x3ffc0000, 0x000), + RTW89_DECL_RFK_WM(0x5830, 0x00000fff, 0x121), + RTW89_DECL_RFK_WM(0x5830, 0x003ff000, 0x000), + RTW89_DECL_RFK_WM(0x583c, 0x0003ffff, 0x115f2), + RTW89_DECL_RFK_WM(0x583c, 0x3ffc0000, 0x000), + RTW89_DECL_RFK_WM(0x5840, 0x00000fff, 0x121), + RTW89_DECL_RFK_WM(0x5840, 0x003ff000, 0x000), + RTW89_DECL_RFK_WM(0x5844, 0x0003ffff, 0x115f2), + RTW89_DECL_RFK_WM(0x5844, 0x3ffc0000, 0x000), + RTW89_DECL_RFK_WM(0x5848, 0x00000fff, 0x121), + RTW89_DECL_RFK_WM(0x5848, 0x003ff000, 0x000), + RTW89_DECL_RFK_WM(0x584c, 0x0003ffff, 0x115f2), + RTW89_DECL_RFK_WM(0x584c, 0x3ffc0000, 0x000), + RTW89_DECL_RFK_WM(0x5850, 0x00000fff, 0x121), + RTW89_DECL_RFK_WM(0x5850, 0x003ff000, 0x000), + RTW89_DECL_RFK_WM(0x585c, 0x0003ffff, 0x115f2), + RTW89_DECL_RFK_WM(0x585c, 0x3ffc0000, 0x000), + RTW89_DECL_RFK_WM(0x5860, 0x00000fff, 0x121), + RTW89_DECL_RFK_WM(0x5860, 0x003ff000, 0x000), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_tssi_slope_defs_a); + +static const struct rtw89_reg5_def rtw8851b_tssi_track_defs_a[] = { + RTW89_DECL_RFK_WM(0x5820, 0x80000000, 0x0), + RTW89_DECL_RFK_WM(0x5818, 0x10000000, 0x0), + RTW89_DECL_RFK_WM(0x5814, 0x00000800, 0x0), + RTW89_DECL_RFK_WM(0x581c, 0x20000000, 0x1), + RTW89_DECL_RFK_WM(0x5864, 0x000003ff, 0x1ff), + RTW89_DECL_RFK_WM(0x5864, 0x000ffc00, 0x200), + RTW89_DECL_RFK_WM(0x5820, 0x00000fff, 0x080), + RTW89_DECL_RFK_WM(0x5814, 0x01000000, 0x0), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_tssi_track_defs_a); + +static const struct rtw89_reg5_def rtw8851b_tssi_mv_avg_defs_a[] = { + RTW89_DECL_RFK_WM(0x58e4, 0x00003800, 0x1), + RTW89_DECL_RFK_WM(0x58e4, 0x00004000, 0x0), + RTW89_DECL_RFK_WM(0x58e4, 0x00008000, 0x1), + RTW89_DECL_RFK_WM(0x58e4, 0x000f0000, 0x0), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_tssi_mv_avg_defs_a); + +static const struct rtw89_reg5_def rtw8851b_nctl_post_defs[] = { + RTW89_DECL_RFK_WM(0x5864, 0x18000000, 0x3), + RTW89_DECL_RFK_WM(0x7864, 0x18000000, 0x3), + RTW89_DECL_RFK_WM(0x030c, 0xff000000, 0x13), + RTW89_DECL_RFK_WM(0x032c, 0xffff0000, 0x0041), + RTW89_DECL_RFK_WM(0x12b8, 0x10000000, 0x1), + RTW89_DECL_RFK_WM(0x2008, 0x01ffffff, 0x00fffff), + RTW89_DECL_RFK_WM(0x0c60, 0x00000003, 0x3), + RTW89_DECL_RFK_WM(0x0c6c, 0x00000001, 0x1), + RTW89_DECL_RFK_WM(0x58ac, 0x08000000, 0x1), + RTW89_DECL_RFK_WM(0x78ac, 0x08000000, 0x1), + RTW89_DECL_RFK_WM(0x0730, 0x00003800, 0x7), + RTW89_DECL_RFK_WM(0x2730, 0x00003800, 0x7), + RTW89_DECL_RFK_WM(0x0c7c, 0x00e00000, 0x1), + RTW89_DECL_RFK_WM(0x58c0, 0x0001ffff, 0x00000), + RTW89_DECL_RFK_WM(0x78c0, 0x0001ffff, 0x00000), + RTW89_DECL_RFK_WM(0x58fc, 0x3f000000, 0x00), + RTW89_DECL_RFK_WM(0x78fc, 0x3f000000, 0x00), +}; + +RTW89_DECLARE_RFK_TBL(rtw8851b_nctl_post_defs); diff --git a/drivers/net/wireless/realtek/rtw89/rtw8851b_rfk_table.h b/drivers/net/wireless/realtek/rtw89/rtw8851b_rfk_table.h new file mode 100644 index 000000000000..febfbecb691c --- /dev/null +++ b/drivers/net/wireless/realtek/rtw89/rtw8851b_rfk_table.h @@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2022-2023 Realtek Corporation + */ + +#ifndef __RTW89_8851B_RFK_TABLE_H__ +#define __RTW89_8851B_RFK_TABLE_H__ + +#include "phy.h" + +extern const struct rtw89_rfk_tbl rtw8851b_dadck_setup_defs_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_dadck_post_defs_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_dack_s0_1_defs_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_dack_s0_2_defs_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_dack_manual_off_defs_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_iqk_rxclk_80_defs_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_iqk_rxclk_others_defs_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_iqk_txk_2ghz_defs_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_iqk_txk_5ghz_defs_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_iqk_afebb_restore_defs_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_iqk_bb_afe_defs_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_iqk_macbb_defs_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_tssi_sys_defs_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_tssi_sys_a_defs_2g_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_tssi_sys_a_defs_5g_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_tssi_init_txpwr_defs_a_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_tssi_init_txpwr_he_tb_defs_a_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_tssi_dck_defs_a_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_tssi_dac_gain_defs_a_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_tssi_slope_a_defs_2g_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_tssi_slope_a_defs_5g_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_tssi_align_a_2g_defs_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_tssi_align_a_5g_defs_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_tssi_slope_defs_a_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_tssi_track_defs_a_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_tssi_mv_avg_defs_a_tbl; +extern const struct rtw89_rfk_tbl rtw8851b_nctl_post_defs_tbl; + +#endif diff --git a/drivers/net/wireless/realtek/rtw89/rtw8851b_table.c b/drivers/net/wireless/realtek/rtw89/rtw8851b_table.c new file mode 100644 index 000000000000..bb724140df4f --- /dev/null +++ b/drivers/net/wireless/realtek/rtw89/rtw8851b_table.c @@ -0,0 +1,14824 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2022-2023 Realtek Corporation + */ + +#include "phy.h" +#include "reg.h" +#include "rtw8851b_table.h" + +static const struct rtw89_reg2_def rtw89_8851b_phy_bb_regs[] = { + {0x704, 0x601E0500}, + {0x4000, 0x00000000}, + {0x4004, 0xCA014000}, + {0x4008, 0xC751D4F0}, + {0x400C, 0x44511475}, + {0x4010, 0x00000000}, + {0x4014, 0x00000000}, + {0x47BC, 0x00000380}, + {0x4018, 0x4F4C084B}, + {0x401C, 0x084A4E52}, + {0x4020, 0x4D504E4B}, + {0x4024, 0x4F4C0849}, + {0x4028, 0x08484C50}, + {0x402C, 0x4C50504C}, + {0x4030, 0x5454084A}, + {0x4034, 0x084B5654}, + {0x4038, 0x6A6C605A}, + {0x403C, 0x4C4C084C}, + {0x4040, 0x084B4E4D}, + {0x4044, 0x4E4C4B4B}, + {0x4048, 0x4B4B084A}, + {0x404C, 0x084A4E4C}, + {0x4050, 0x514F4C4A}, + {0x4054, 0x524E084A}, + {0x4058, 0x084A5154}, + {0x405C, 0x53555554}, + {0x4060, 0x45450845}, + {0x4064, 0x08454144}, + {0x4068, 0x40434445}, + {0x406C, 0x44450845}, + {0x4070, 0x08444043}, + {0x4074, 0x42434444}, + {0x4078, 0x46450844}, + {0x407C, 0x08444843}, + {0x4080, 0x4B4E4A47}, + {0x4084, 0x4F4C084B}, + {0x4088, 0x084A4E52}, + {0x408C, 0x4D504E4B}, + {0x4090, 0x4F4C0849}, + {0x4094, 0x08484C50}, + {0x4098, 0x4C50504C}, + {0x409C, 0x5454084A}, + {0x40A0, 0x084B5654}, + {0x40A4, 0x6A6C605A}, + {0x40A8, 0x4C4C084C}, + {0x40AC, 0x084B4E4D}, + {0x40B0, 0x4E4C4B4B}, + {0x40B4, 0x4B4B084A}, + {0x40B8, 0x084A4E4C}, + {0x40BC, 0x514F4C4A}, + {0x40C0, 0x524E084A}, + {0x40C4, 0x084A5154}, + {0x40C8, 0x53555554}, + {0x40CC, 0x45450845}, + {0x40D0, 0x08454144}, + {0x40D4, 0x40434445}, + {0x40D8, 0x44450845}, + {0x40DC, 0x08444043}, + {0x40E0, 0x42434444}, + {0x40E4, 0x46450844}, + {0x40E8, 0x08444843}, + {0x40EC, 0x4B4E4A47}, + {0x40F0, 0x00000000}, + {0x4A38, 0x00000000}, + {0x40F4, 0x00000006}, + {0x40F8, 0x00000000}, + {0x40FC, 0x8C30C30C}, + {0x4100, 0x4C30C30C}, + {0x4104, 0x0C30C30C}, + {0x4108, 0x0C30C30C}, + {0x410C, 0x0C30C30C}, + {0x4110, 0x0C30C30C}, + {0x4114, 0x28A28A28}, + {0x4118, 0x28A28A28}, + {0x411C, 0x28A28A28}, + {0x4120, 0x28A28A28}, + {0x4124, 0x28A28A28}, + {0x4128, 0x28A28A28}, + {0x412C, 0x06666666}, + {0x4130, 0x33333333}, + {0x4134, 0x33333333}, + {0x4138, 0x33333333}, + {0x413C, 0x00000031}, + {0x4140, 0x5100600A}, + {0x4144, 0x18363113}, + {0x4148, 0x1D976DDC}, + {0x414C, 0x1C072DD7}, + {0x4150, 0x1127CDF4}, + {0x4154, 0x1E37BDF1}, + {0x4158, 0x1FB7F1D6}, + {0x415C, 0x1EA7DDF9}, + {0x4160, 0x1FE445DD}, + {0x4164, 0x1F97F1FE}, + {0x4168, 0x1FF781ED}, + {0x416C, 0x1FA7F5FE}, + {0x4170, 0x1E07B913}, + {0x4174, 0x1FD7FDFF}, + {0x4178, 0x1E17B9FA}, + {0x417C, 0x19A66914}, + {0x4180, 0x10F65598}, + {0x4184, 0x14A5A111}, + {0x4188, 0x1D3765DB}, + {0x418C, 0x17C685CA}, + {0x4190, 0x1107C5F3}, + {0x4194, 0x1B5785EB}, + {0x4198, 0x1F97ED8F}, + {0x419C, 0x1BC7A5F3}, + {0x41A0, 0x1FE43595}, + {0x41A4, 0x1EB7D9FC}, + {0x41A8, 0x1FE65DBE}, + {0x41AC, 0x1EC7D9FC}, + {0x41B0, 0x1976FCFF}, + {0x41B4, 0x1F77F5FF}, + {0x41B8, 0x1976FDEC}, + {0x41BC, 0x198664EF}, + {0x41C0, 0x11062D93}, + {0x41C4, 0x10C4E910}, + {0x41C8, 0x1CA759DB}, + {0x41CC, 0x1335A9B5}, + {0x41D0, 0x1097B9F3}, + {0x41D4, 0x17B72DE1}, + {0x41D8, 0x1F67ED42}, + {0x41DC, 0x18074DE9}, + {0x41E0, 0x1FD40547}, + {0x41E4, 0x1D57ADF9}, + {0x41E8, 0x1FE52182}, + {0x41EC, 0x1D67B1F9}, + {0x41F0, 0x14860CE1}, + {0x41F4, 0x1EC7E9FE}, + {0x41F8, 0x14860DD6}, + {0x41FC, 0x195664C7}, + {0x4200, 0x0005E58A}, + {0x4204, 0x00000000}, + {0x4208, 0x00000000}, + {0x420C, 0x7A000000}, + {0x4210, 0x0F9F3D7A}, + {0x4214, 0x0040817C}, + {0x4218, 0x00E10204}, + {0x421C, 0x227D94CD}, + {0x4220, 0x08028A28}, + {0x4224, 0x00000200}, + {0x4228, 0x04688000}, + {0x47C0, 0x00000001}, + {0x4A48, 0x00000002}, + {0x4B04, 0x00000000}, + {0x4B08, 0x00000000}, + {0x422C, 0x0060B002}, + {0x4230, 0x9A8249A8}, + {0x4234, 0x26A1469E}, + {0x4238, 0x2099A824}, + {0x423C, 0x2359461C}, + {0x4240, 0x1631A675}, + {0x4244, 0x2C6B1D63}, + {0x4248, 0x0000000E}, + {0x424C, 0x00000001}, + {0x4250, 0x00000001}, + {0x4254, 0x00000000}, + {0x4258, 0x00000000}, + {0x425C, 0x00000000}, + {0x4260, 0x0020000C}, + {0x4A30, 0x00000000}, + {0x4264, 0x00000000}, + {0x4268, 0x00000000}, + {0x426C, 0x0418317C}, + {0x4270, 0x2B33135C}, + {0x4274, 0x00000002}, + {0x4278, 0x00000000}, + {0x427C, 0x00000000}, + {0x4280, 0x00000000}, + {0x4284, 0x00000000}, + {0x4288, 0x00000000}, + {0x428C, 0x00000000}, + {0x4290, 0x00000000}, + {0x4294, 0x00000000}, + {0x4298, 0x00000000}, + {0x429C, 0x84026000}, + {0x42A0, 0x0051AC20}, + {0x4A24, 0x0010C040}, + {0x42A4, 0x02024008}, + {0x42A8, 0x00000000}, + {0x42AC, 0x00000000}, + {0x42B0, 0x22CE803C}, + {0x42B4, 0xD8000000}, + {0x42B8, 0x596FD67E}, + {0x42BC, 0x7D67D67D}, + {0x42C0, 0x7D67D65B}, + {0x42C4, 0x28029F59}, + {0x42C8, 0x00280280}, + {0x4AF4, 0x00000000}, + {0x42CC, 0x00000000}, + {0x42D0, 0x00000000}, + {0x42D4, 0x00000003}, + {0x4AF8, 0x00280000}, + {0x42D8, 0x00000001}, + {0x42DC, 0x69AEC800}, + {0x42E0, 0x8B4CD3D1}, + {0x42E4, 0xC514534F}, + {0x42E8, 0x85145145}, + {0x42EC, 0x45145145}, + {0x42F0, 0x05145145}, + {0x42F4, 0x05145145}, + {0x42F8, 0x05145145}, + {0x42FC, 0x17659145}, + {0x4300, 0x176DD5D9}, + {0x4304, 0x0F65765B}, + {0x4308, 0x0F3CF3CF}, + {0x430C, 0x0F3CF3CF}, + {0x4310, 0x0F3CF3CF}, + {0x4314, 0x0F3CF3CF}, + {0x4318, 0x0F3CF3CF}, + {0x431C, 0x0F3CF3CF}, + {0x4320, 0x0F3CF3CF}, + {0x4324, 0x0F44F351}, + {0x4328, 0x192D7547}, + {0x432C, 0x0F5CF5CF}, + {0x4330, 0x051593D9}, + {0x4334, 0x05145145}, + {0x4338, 0x05145145}, + {0x433C, 0x05145145}, + {0x4340, 0x05145145}, + {0x4344, 0x05145145}, + {0x4348, 0x19545145}, + {0x434C, 0x1B65B5DB}, + {0x4350, 0x1965965B}, + {0x4354, 0x0F3CF3CF}, + {0x4358, 0x0F3CF3CF}, + {0x435C, 0x0F3CF1CF}, + {0x4360, 0x0F3CF3CF}, + {0x4364, 0x0F3CF3CF}, + {0x4368, 0x0F3CF3CF}, + {0x436C, 0x0F3CF3CF}, + {0x4370, 0x0934D2CF}, + {0x4374, 0x112CB3CF}, + {0x4378, 0x9777A777}, + {0x437C, 0xBB7BAC95}, + {0x4380, 0xB667B889}, + {0x4384, 0x7B9B8899}, + {0x4388, 0x7A5567C8}, + {0x438C, 0x2278CCCC}, + {0x4390, 0x7C222222}, + {0x4394, 0x0000029B}, + {0x4398, 0x001CCCCC}, + {0x4AAC, 0xCCCCC88C}, + {0x4AB0, 0x0000AACC}, + {0x439C, 0x00000000}, + {0x43A0, 0x00000008}, + {0x43A4, 0x00000000}, + {0x43A8, 0x00000000}, + {0x43AC, 0x00000000}, + {0x43B0, 0x10000000}, + {0x43B4, 0x00401001}, + {0x43B8, 0x00061003}, + {0x43BC, 0x000024D8}, + {0x43C0, 0x00000000}, + {0x43C4, 0x10000020}, + {0x43C8, 0x20000200}, + {0x43CC, 0x00000000}, + {0x43D0, 0x04000000}, + {0x43D4, 0x44000100}, + {0x43D8, 0x60804060}, + {0x43DC, 0x44204210}, + {0x43E0, 0x82108082}, + {0x43E4, 0x82108402}, + {0x43E8, 0xC8082108}, + {0x43EC, 0xC8202084}, + {0x43F0, 0x44208208}, + {0x43F4, 0x84108204}, + {0x43F8, 0xD0108104}, + {0x43FC, 0xF8210108}, + {0x4400, 0x6431E930}, + {0x4404, 0x02309468}, + {0x4408, 0x10C61C22}, + {0x440C, 0x02109469}, + {0x4410, 0x10C61C22}, + {0x4414, 0x00041049}, + {0x4A4C, 0x00060581}, + {0x4418, 0x00000000}, + {0x441C, 0x00000000}, + {0x4420, 0xEC000000}, + {0x4424, 0xB0200020}, + {0x4428, 0x00001FF0}, + {0x4AC8, 0x00000001}, + {0x4B0C, 0x00000000}, + {0x4CDC, 0x00000000}, + {0x442C, 0x00000000}, + {0x4430, 0x00000000}, + {0x4434, 0x00000000}, + {0x4438, 0x00000000}, + {0x443C, 0x190642D0}, + {0x4440, 0xA80668A0}, + {0x4444, 0x60900820}, + {0x4448, 0x9F28518C}, + {0x444C, 0x32488A62}, + {0x4450, 0x9C6E36DC}, + {0x4454, 0x0000F52B}, + {0x4A34, 0x00000007}, + {0x4CE0, 0x68120000}, + {0x4CE4, 0x1A0681E0}, + {0x4CE8, 0x94060180}, + {0x4CEC, 0x000603FF}, + {0x4CF0, 0xA0502000}, + {0x4CF4, 0x00001000}, + {0x4D00, 0x00000044}, + {0x4B14, 0x00000000}, + {0x4458, 0x00000000}, + {0x445C, 0x4801442E}, + {0x4460, 0x0051A0FA}, + {0x4B18, 0x0000011F}, + {0x4B1C, 0x0000011F}, + {0x4464, 0x00000000}, + {0x4468, 0x00000000}, + {0x446C, 0x00000000}, + {0x4470, 0x00000000}, + {0x4474, 0x00000000}, + {0x4478, 0x00000000}, + {0x447C, 0x00000000}, + {0x4480, 0x2A0A6040}, + {0x4484, 0x0A0A6829}, + {0x4488, 0x00000004}, + {0x448C, 0x00000000}, + {0x4490, 0x80000000}, + {0x4494, 0x10000000}, + {0x4498, 0xE0000000}, + {0x4A28, 0x000ED877}, + {0x4AB4, 0x00000000}, + {0x4B20, 0x00000000}, + {0x4B24, 0x00000000}, + {0x4B28, 0x00000000}, + {0x4B2C, 0x00000000}, + {0x449C, 0x0000001E}, + {0x44A0, 0x02B2C394}, + {0x44A4, 0x00000400}, + {0x4A2C, 0x0050240E}, + {0x4B30, 0x7FFFFD20}, + {0x4B34, 0x920823FF}, + {0x4B38, 0x7FFFFFFF}, + {0x4B3C, 0x01773773}, + {0x44A8, 0x00000001}, + {0x44AC, 0x000190C0}, + {0x44B0, 0x00000000}, + {0x44B4, 0x00000000}, + {0x44B8, 0x00000000}, + {0x44BC, 0x00000000}, + {0x44C0, 0x00000000}, + {0x44C4, 0x00000000}, + {0x44C8, 0x00000000}, + {0x44CC, 0x00000000}, + {0x44D0, 0x00000000}, + {0x44D4, 0x00000000}, + {0x44D8, 0x00000000}, + {0x44DC, 0x00000000}, + {0x44E0, 0x00000000}, + {0x44E4, 0x00000000}, + {0x44E8, 0x00000000}, + {0x44EC, 0x00000000}, + {0x44F0, 0x00000000}, + {0x44F4, 0x00000000}, + {0x44F8, 0x00000000}, + {0x44FC, 0x00000000}, + {0x4500, 0x00000000}, + {0x4504, 0x00000000}, + {0x4508, 0x00000000}, + {0x450C, 0x00000000}, + {0x4510, 0x00000000}, + {0x4514, 0x00000000}, + {0x4518, 0x00000000}, + {0x451C, 0x00000000}, + {0x4520, 0x00000000}, + {0x4524, 0x00000000}, + {0x4528, 0x00000000}, + {0x452C, 0x00000000}, + {0x4530, 0x4E830171}, + {0x4534, 0x00000870}, + {0x4538, 0x000000FF}, + {0x453C, 0x00000000}, + {0x4540, 0x00000000}, + {0x4544, 0x00000000}, + {0x4548, 0x00000000}, + {0x454C, 0x00000000}, + {0x4550, 0x00000000}, + {0x4554, 0x00000000}, + {0x4558, 0x00000000}, + {0x455C, 0x00000000}, + {0x4560, 0x40000000}, + {0x4564, 0x40000000}, + {0x4568, 0x00000000}, + {0x456C, 0x20000000}, + {0x4570, 0x04F040BB}, + {0x4574, 0x000E53FF}, + {0x4578, 0x000205CB}, + {0x457C, 0x00200000}, + {0x4580, 0x00000040}, + {0x4584, 0x00000000}, + {0x4588, 0x00000017}, + {0x458C, 0x30000000}, + {0x4590, 0x00000000}, + {0x4594, 0x00000000}, + {0x4598, 0x00000001}, + {0x459C, 0x0003FE00}, + {0x45A0, 0x00000086}, + {0x45A4, 0x00000000}, + {0x45A8, 0xC00001C0}, + {0x45AC, 0x78038000}, + {0x45B0, 0x8000004A}, + {0x45B4, 0x04094800}, + {0x45B8, 0x00280002}, + {0x45BC, 0x06748790}, + {0x45C0, 0x80000000}, + {0x45C4, 0x00000000}, + {0x45C8, 0x00000000}, + {0x45CC, 0x00558670}, + {0x45D0, 0x002883F0}, + {0x45D4, 0x00090120}, + {0x45D8, 0x00000000}, + {0x45E0, 0xA3A6D3C4}, + {0x45E4, 0xAB27B126}, + {0x45E8, 0x00006778}, + {0x45F4, 0x000001B5}, + {0x45EC, 0x11110F0A}, + {0x45F0, 0x00000003}, + {0x4A0C, 0x0000000A}, + {0x45F8, 0x0058BC3F}, + {0x45FC, 0x00000003}, + {0x462C, 0x00000020}, + {0x4600, 0x000003D9}, + {0x45F0, 0x00000004}, + {0x4604, 0x002B1CB0}, + {0x4A50, 0xC0000000}, + {0x4A54, 0x00001000}, + {0x4A58, 0x00000000}, + {0x4A18, 0x00000024}, + {0x4608, 0x00000001}, + {0x460C, 0x00000000}, + {0x4A10, 0x00000001}, + {0x4610, 0x00000001}, + {0x4614, 0x16E5298F}, + {0x4618, 0x18C6294A}, + {0x461C, 0x0E06318A}, + {0x4620, 0x0E539CE5}, + {0x4624, 0x00019287}, + {0x4A14, 0x000000BF}, + {0x4628, 0x00000001}, + {0x4630, 0x000001AA}, + {0x4A18, 0x00001900}, + {0x4A1C, 0x000002A6}, + {0x4634, 0x000000A3}, + {0x4A20, 0x00000086}, + {0x4638, 0x00045656}, + {0x49F8, 0x00000000}, + {0x463C, 0x00000000}, + {0x4640, 0x00000000}, + {0x4644, 0x00C8CC00}, + {0x4648, 0xC400B6B6}, + {0x464C, 0xDC400FC0}, + {0x4A44, 0x00000000}, + {0x4A8C, 0x00000110}, + {0x4BC4, 0x00000001}, + {0x4650, 0x08882550}, + {0x4654, 0x08CC2660}, + {0x4658, 0x09102660}, + {0x465C, 0x00000154}, + {0x45DC, 0xC39E38E8}, + {0x4660, 0x452607E6}, + {0x4664, 0x6750DC65}, + {0x4668, 0xF3F0F1ED}, + {0x466C, 0x30141506}, + {0x4670, 0x2C2B2B2B}, + {0x4674, 0x2C2C2C2C}, + {0x4678, 0xDDB738E8}, + {0x467C, 0x543618FB}, + {0x4680, 0x4F31DC6F}, + {0x4684, 0xFBEBDA00}, + {0x4688, 0x1A10FF04}, + {0x468C, 0x282A3000}, + {0x4690, 0x2A29292A}, + {0x4694, 0x04FA2A2A}, + {0x4698, 0xEE0F04D1}, + {0x469C, 0x99E91436}, + {0x46A0, 0x0701E79E}, + {0x46A4, 0x08D77CFF}, + {0x46A8, 0x321AFF14}, + {0x46AC, 0x60313447}, + {0x46B0, 0x63666666}, + {0x46B4, 0x35374425}, + {0x46B8, 0x35883042}, + {0x46BC, 0x5177C252}, + {0x4720, 0x7FFFFD63}, + {0x4724, 0xB58D11FF}, + {0x4728, 0x47FFFFFF}, + {0x472C, 0x0E7893B6}, + {0x4730, 0xE0391201}, + {0x4734, 0x00000020}, + {0x4738, 0x8325C500}, + {0x473C, 0x00000B7F}, + {0x46C0, 0x00000000}, + {0x46C4, 0x00000000}, + {0x46C8, 0x00000219}, + {0x4BDC, 0x00002020}, + {0x46CC, 0x00000000}, + {0x46D0, 0x00000000}, + {0x4A3C, 0x00000002}, + {0x46D4, 0x00000001}, + {0x46D8, 0x00000001}, + {0x46DC, 0x00000000}, + {0x46E0, 0x00000000}, + {0x46E4, 0x00000151}, + {0x46E8, 0x00000498}, + {0x46EC, 0x00000498}, + {0x46F0, 0x00000000}, + {0x46F4, 0x00000000}, + {0x46F8, 0x00001146}, + {0x46FC, 0x00000000}, + {0x4700, 0x00000000}, + {0x4704, 0x00C8CC00}, + {0x4708, 0xC400B6B6}, + {0x470C, 0xDC400FC0}, + {0x4A90, 0x00000110}, + {0x4B10, 0x00000000}, + {0x4BE0, 0x00000001}, + {0x4710, 0x08882550}, + {0x4714, 0x08CC2660}, + {0x4718, 0x09102660}, + {0x471C, 0x00000154}, + {0x4740, 0xC69F38E8}, + {0x4744, 0x462709E9}, + {0x4748, 0x6750DC67}, + {0x474C, 0xF3F0F1ED}, + {0x4750, 0x30141506}, + {0x4754, 0x2C2B2B2B}, + {0x4758, 0x2C2C2C2C}, + {0x475C, 0xE0B738E8}, + {0x4760, 0x52381BFE}, + {0x4764, 0x5031DC6C}, + {0x4768, 0xFBEBDA00}, + {0x476C, 0x1A10FF04}, + {0x4770, 0x282A3000}, + {0x4774, 0x2A29292A}, + {0x4778, 0x04FA2A2A}, + {0x477C, 0xEE0F04D1}, + {0x47C4, 0x00000000}, + {0x47C8, 0xA32103FE}, + {0x47CC, 0xB20A5328}, + {0x47D0, 0xC686314F}, + {0x47D4, 0x000004D7}, + {0x4BFC, 0x00000000}, + {0x4C00, 0x0C442416}, + {0x4C04, 0x00000000}, + {0x47D8, 0x009B902A}, + {0x47DC, 0x009B902A}, + {0x47E0, 0x98682C18}, + {0x47E4, 0x6318C4C1}, + {0x47E8, 0x6248C631}, + {0x47EC, 0x922A8253}, + {0x47F0, 0x00000005}, + {0x47F4, 0x00001759}, + {0x47F8, 0x4BB01800}, + {0x47FC, 0x831408BE}, + {0x4A84, 0x000000E9}, + {0x4C08, 0x0F801404}, + {0x4C0C, 0x00A2B404}, + {0x4800, 0x9ABBCACB}, + {0x4804, 0x56867578}, + {0x4808, 0xBCCBBB13}, + {0x480C, 0x7889989B}, + {0x4810, 0xBBB0F455}, + {0x4814, 0x777BBBBB}, + {0x4818, 0x15277777}, + {0x481C, 0x27039CE9}, + {0x4820, 0x42424432}, + {0x4824, 0x36058342}, + {0x4828, 0x00000006}, + {0x482C, 0x00000005}, + {0x4830, 0x00000005}, + {0x4834, 0xC7013016}, + {0x4838, 0x84413016}, + {0x483C, 0x84413016}, + {0x4840, 0x8C413016}, + {0x4844, 0x8C40B028}, + {0x4848, 0x3140B028}, + {0x484C, 0x2940B028}, + {0x4850, 0x8440B028}, + {0x4854, 0x2318C610}, + {0x4858, 0x45344753}, + {0x485C, 0x236A6A88}, + {0x4860, 0xAC8DF814}, + {0x4864, 0x08877ACB}, + {0x4868, 0x000107AA}, + {0x4A94, 0x00000000}, + {0x486C, 0xBCEB4A14}, + {0x4870, 0x000A3A4A}, + {0x4874, 0xBCEB4A14}, + {0x4878, 0x000A3A4A}, + {0x487C, 0xBCBDBD85}, + {0x4880, 0x0CABB99A}, + {0x4884, 0x38384242}, + {0x4888, 0x0086102E}, + {0x488C, 0xCA24C82A}, + {0x4AFC, 0x00000000}, + {0x4C14, 0x0000349D}, + {0x4CF8, 0x00000007}, + {0x4890, 0x00008A62}, + {0x4894, 0x00000008}, + {0x4958, 0x80040000}, + {0x495C, 0x80040000}, + {0x4960, 0xFE800000}, + {0x4964, 0x834C0000}, + {0x4968, 0x00000000}, + {0x496C, 0x00000000}, + {0x4970, 0x00000000}, + {0x4974, 0x00000000}, + {0x4978, 0x00000000}, + {0x497C, 0x00000000}, + {0x4980, 0x40000000}, + {0x4984, 0x00000000}, + {0x4988, 0x00000000}, + {0x498C, 0x00000000}, + {0x4990, 0x00000000}, + {0x4994, 0x04065800}, + {0x4998, 0x02004080}, + {0x499C, 0x0E1E3E05}, + {0x49A0, 0x0A163068}, + {0x49A4, 0x00206040}, + {0x49A8, 0x02020202}, + {0x49AC, 0x00002020}, + {0x49B0, 0xF8F8F418}, + {0x49B4, 0xF8E8F8F8}, + {0x49B8, 0xF80808E8}, + {0x4A00, 0xF8F8FA00}, + {0x4A04, 0xFAFAFAF8}, + {0x4A08, 0xFAFAFAFA}, + {0x49BC, 0x00000000}, + {0x49C0, 0x800C562D}, + {0x49C4, 0x00000101}, + {0x49C8, 0x00000000}, + {0x49CC, 0x00000000}, + {0x49D0, 0x00000000}, + {0x49D4, 0x00000000}, + {0x49D8, 0x00000000}, + {0x49DC, 0x00000000}, + {0x49E0, 0x00000000}, + {0x49E4, 0x00000000}, + {0x49E8, 0x00000000}, + {0x49EC, 0x00000000}, + {0x4C28, 0x00000000}, + {0x4C2C, 0x00000000}, + {0x4C30, 0x00000000}, + {0x4C34, 0x00000000}, + {0x4C38, 0x00000000}, + {0x4C3C, 0x00000000}, + {0x4C40, 0x00000000}, + {0x4C44, 0x01C0C832}, + {0x4C48, 0x03207032}, + {0x4C4C, 0x0320701C}, + {0x4C50, 0x03207032}, + {0x4C54, 0x01C0C81C}, + {0x4C58, 0x00A0281C}, + {0x4C5C, 0x0320C80A}, + {0x4C60, 0x00A0C832}, + {0x4C64, 0x01C0C832}, + {0x4C68, 0x03207032}, + {0x4C6C, 0x0320701C}, + {0x4C70, 0x03207032}, + {0x4C74, 0x01C0C81C}, + {0x4C78, 0x00A0281C}, + {0x4C7C, 0x0321A80A}, + {0x4C80, 0x0320C86A}, + {0x4C84, 0x12B02832}, + {0x4C88, 0x12B3292B}, + {0x4C8C, 0x0CA4ACCA}, + {0x4C90, 0x12B4AC6A}, + {0x4C94, 0x0CA4ACCA}, + {0x4C98, 0x06A3292B}, + {0x4C9C, 0x06A0280A}, + {0x4CA0, 0x0CA0286A}, + {0x4CA4, 0x0CA1A8CA}, + {0x4CA8, 0x06A3286A}, + {0x4CAC, 0x0000000A}, + {0x4CB0, 0x01209C27}, + {0x4CB4, 0x02704800}, + {0x4CB8, 0x02704812}, + {0x4CBC, 0x00004827}, + {0x4CC0, 0x01209C12}, + {0x4CC4, 0x00000012}, + {0x4CC8, 0x02718000}, + {0x4CCC, 0x02709C60}, + {0x4CD0, 0x00000027}, + {0x4CD4, 0x00000000}, + {0x4CD8, 0x0000014A}, + {0x994, 0x00000010}, + {0x904, 0x00000005}, + {0x708, 0x00000000}, + {0x884, 0x0043F01D}, + {0x710, 0xEF810000}, + {0x718, 0x1333233F}, + {0x604, 0x041E1E1E}, + {0x714, 0x00010000}, + {0x586C, 0x000000F0}, + {0x586C, 0x000000E0}, + {0x586C, 0x000000D0}, + {0x586C, 0x000000C0}, + {0x586C, 0x000000B0}, + {0x586C, 0x000000A0}, + {0x586C, 0x00000090}, + {0x586C, 0x00000080}, + {0x586C, 0x00000070}, + {0x586C, 0x00000060}, + {0x586C, 0x00000050}, + {0x586C, 0x00000040}, + {0x586C, 0x00000030}, + {0x586C, 0x00000020}, + {0x586C, 0x00000010}, + {0x586C, 0x00000000}, + {0xC0D4, 0xABA41460}, + {0xC0D8, 0xC43A7E87}, + {0xC0DC, 0x30C194B8}, + {0xC0E0, 0x75008138}, + {0xC0E4, 0x0000272B}, + {0xC0E8, 0x000A0C81}, + {0xC0EC, 0x00030003}, + {0xC0F0, 0x00000024}, + {0xC0C4, 0x005E3A00}, + {0xC004, 0x45800000}, + {0xC024, 0x45800000}, + {0x334, 0xFFFFFFFF}, + {0x33C, 0x55000000}, + {0x340, 0x00005555}, + {0x724, 0x00111200}, + {0x5868, 0xA9550000}, + {0x5870, 0x33221100}, + {0x5874, 0x77665544}, + {0x5878, 0xBBAA9988}, + {0x587C, 0xFFEEDDCC}, + {0x5880, 0x76543210}, + {0x5884, 0xFEDCBA98}, + {0x5888, 0x00000000}, + {0x588C, 0x00000000}, + {0x5894, 0x00000008}, + {0x650, 0x00200888}, + {0x710, 0xF3810000}, + {0x020, 0x0000F381}, + {0x024, 0x0000F381}, + {0x000, 0xC580801E}, + {0x980, 0x10002250}, + {0x988, 0x3C3C4107}, + {0x994, 0x00000010}, + {0x000, 0x0580801F}, + {0x240C, 0x00000000}, + {0x640, 0x210A141E}, + {0x640, 0x2114141E}, + {0x640, 0x2114141E}, + {0x644, 0x3414283C}, + {0x644, 0x3425283C}, + {0x644, 0x3426283C}, + {0x2640, 0x140A141E}, + {0x2640, 0x1414141E}, + {0x2640, 0x1414141E}, + {0x2644, 0x3414283C}, + {0x2644, 0x3425283C}, + {0x2644, 0x3425183C}, + {0x2300, 0x02748790}, + {0x2304, 0x00558670}, + {0x2308, 0x002883F0}, + {0x230C, 0x00090120}, + {0x2310, 0x00000000}, + {0x2314, 0x06000000}, + {0x2318, 0x00000000}, + {0x231C, 0x00000000}, + {0x2320, 0x03020100}, + {0x2324, 0x07060504}, + {0x2328, 0x0B0A0908}, + {0x232C, 0x0F0E0D0C}, + {0x2330, 0x13121110}, + {0x2334, 0x17161514}, + {0x2338, 0x0C700022}, + {0x233C, 0x0A0529D0}, + {0x2340, 0x000529D0}, + {0x2344, 0x0006318A}, + {0x2348, 0xB7E6318A}, + {0x234C, 0x80039C00}, + {0x2350, 0x80039C00}, + {0x2354, 0x0005298F}, + {0x2358, 0x0015296E}, + {0x235C, 0x0C07FC31}, + {0x2360, 0x0219AAAE}, + {0x2364, 0xE4F624C3}, + {0x2368, 0x53626F15}, + {0x236C, 0x48000000}, + {0x2370, 0x48000000}, + {0x2374, 0x07540000}, + {0x2378, 0x202401B9}, + {0x237C, 0x00F7000E}, + {0x2380, 0x0F0A1111}, + {0x2384, 0x30D9000F}, + {0x2388, 0x0200EA02}, + {0x238C, 0x003CB061}, + {0x2390, 0x69C00000}, + {0x2394, 0x00000000}, + {0x2398, 0x000000F0}, + {0x239C, 0x0001FFFF}, + {0x23A0, 0x00C80064}, + {0x23A4, 0x0190012C}, + {0x23A8, 0x001917BE}, + {0x23AC, 0x0B30880C}, + {0x23B0, 0x9281CE00}, + {0x23B4, 0x7F027C00}, + {0x704, 0x601E0502}, + {0x5600, 0x00000000}, + {0x5604, 0x802D2721}, + {0x5610, 0x00201020}, + {0x5618, 0x00801008}, + {0x5624, 0x0808081E}, + {0x562C, 0x0000081D}, + {0x5634, 0x3D2EE000}, + {0x5638, 0x0001AC42}, + {0x5640, 0x3D6EF000}, + {0x5644, 0x0001AC3E}, + {0x566C, 0x00210005}, + {0x5680, 0x20500010}, + {0x5684, 0x00020001}, + {0x56A0, 0x0034C000}, + {0x56BC, 0x04000000}, + {0x56C0, 0x00000688}, + {0x56C4, 0x00000010}, + {0x56C8, 0x0E800400}, + {0x56CC, 0x01E400FF}, + {0x5800, 0x003F807F}, + {0x5810, 0x59008400}, + {0x5814, 0x201AF000}, + {0x5818, 0x182C18E8}, + {0x581C, 0x3DD80280}, + {0x5820, 0x80000080}, + {0x5828, 0x023F8121}, + {0x5830, 0x023F8121}, + {0x5838, 0x003F8121}, + {0x5840, 0x023F8121}, + {0x5848, 0x023F8121}, + {0x5850, 0x023F8121}, + {0x5858, 0x003F7121}, + {0x5860, 0x023F7121}, + {0x5864, 0x1A1801FF}, + {0x5868, 0xA9A90002}, + {0x5880, 0x77777777}, + {0x5884, 0x77777777}, + {0x5894, 0x01080604}, + {0x5898, 0x00000000}, + {0x589C, 0x00000000}, + {0x58A0, 0x000000FE}, + {0x58B0, 0x00000800}, + {0x58BC, 0x07A7807F}, + {0x58C0, 0x007E0000}, + {0x58C4, 0x0003FFFF}, + {0x58D4, 0x7401FE00}, + {0x58D8, 0x8008016C}, + {0x58DC, 0xC000807F}, + {0x58E4, 0x3000881F}, + {0x58E8, 0x00000003}, + {0x58F0, 0x400401FF}, + {0x58F4, 0x80000000}, + {0x58F8, 0xC0000000}, + {0x58FC, 0x00000000}, + {0x700, 0x40000030}, + {0x704, 0x601E0502}, + {0x704, 0x601E0500}, + {0x704, 0x601E0502}, + {0x20FC, 0x00000000}, + {0x20F8, 0x00000000}, + {0x20F0, 0x00000000}, + {0x9C0, 0x00000001}, + {0x9C0, 0x00000000}, + {0x9C0, 0x00000001}, + {0x9C0, 0x00000000}, + {0x4AE8, 0x00000744}, + {0x4AD4, 0x00000040}, + {0x4AE4, 0x0079E99E}, + {0x4BC8, 0xFBD5B89F}, + {0x4BCC, 0x99563918}, + {0x4BD0, 0x12EED5B8}, + {0x4BD4, 0x6F7D542F}, + {0x4BD8, 0x0000001D}, + {0x300, 0xF30CE31C}, + {0x304, 0x13EF1F19}, + {0x308, 0x0C0CF3F3}, + {0x30C, 0x0CE30C0C}, + {0x310, 0x80496000}, + {0x314, 0x0041E000}, + {0x318, 0x20022042}, + {0x31C, 0x20448009}, + {0x320, 0x00010031}, + {0x324, 0xE000E000}, + {0x328, 0xE000E000}, + {0x32C, 0xE0008049}, + {0x12BC, 0x10104041}, + {0x12C0, 0x13311111}, + {0x12E4, 0x30D52A68}, + {0x010, 0x0005FFFF}, + {0x028, 0x0000F381}, + {0x02C, 0x0000F381}, + {0x620, 0x00141230}, + {0x70C, 0x00000020}, + {0x720, 0x20000000}, + {0x730, 0x00000002}, + {0x738, 0x004100C0}, + {0x73C, 0x00000002}, + {0x748, 0x01000002}, + {0x74C, 0x00000001}, + {0xA08, 0x00007800}, + {0xC14, 0x25010000}, + {0xC3C, 0x2840E1BF}, + {0xC40, 0x00000000}, + {0xC44, 0x00000007}, + {0xC48, 0x410E4000}, + {0xC54, 0x1EE14368}, + {0xC58, 0x41000000}, + {0xC5C, 0x80558000}, + {0xC60, 0x017FFFF2}, + {0xC64, 0x0010A130}, + {0xC68, 0x90000050}, + {0xC6C, 0x10201021}, + {0xC70, 0x071B0660}, + {0xC74, 0x00000000}, + {0xC78, 0x80000000}, + {0xC7C, 0x0020BFE0}, + {0xC88, 0xC2AC8000}, + {0xC8C, 0x02F2FC08}, + {0xD00, 0x77777777}, + {0xD04, 0xBBBBBBBB}, + {0xD08, 0xBBBBBBBB}, + {0xD0C, 0x000B2070}, + {0xD10, 0x20110FFF}, + {0xD18, 0x50009800}, + {0xD20, 0x01900000}, + {0xD30, 0x03FF8000}, + {0xD40, 0xF64FA0F7}, + {0xD44, 0x0401463F}, + {0xD48, 0x0003FF7F}, + {0xD4C, 0x00000000}, + {0xD50, 0xF64FA0F7}, + {0xD54, 0x04100437}, + {0xD58, 0x0000FF7F}, + {0xD5C, 0x00000000}, + {0xD60, 0x00000000}, + {0xD64, 0x00000000}, + {0xD70, 0x00000015}, + {0xD78, 0x00000001}, + {0xD7C, 0x001D050E}, + {0xD80, 0x00000100}, + {0xD84, 0x00006607}, + {0xD90, 0x000003FF}, + {0xD94, 0x00000000}, + {0xD98, 0x0000003F}, + {0xD9C, 0x00000000}, + {0xDA0, 0x000003FE}, + {0xDA4, 0x00000000}, + {0xDA8, 0x0000003F}, + {0xDAC, 0x00000000}, + {0xDD4, 0x00000000}, + {0x1010, 0x00000000}, + {0x2000, 0x50BBBF04}, + {0x2008, 0x000FFFFF}, + {0x5800, 0x03FF807F}, + {0x5804, 0x04237040}, + {0x5808, 0x04237040}, + {0x5818, 0x082C1800}, + {0x624, 0x0101030A}, + {0x241C, 0x00000001}, + {0xC0F8, 0x00000001}, + {0x35C, 0x000004C4}, + {0x1200, 0x00010142}, + {0x120C, 0x00012233}, + {0x1210, 0x8049E304}, + {0x12A0, 0x49107056}, + {0x12A8, 0x33337025}, + {0x12AC, 0x12333121}, + {0x12B8, 0x30020000}, + {0x0F0, 0x00000001}, + {0x0F4, 0x00000011}, + {0x0F8, 0x20230307}, +}; + +static const struct rtw89_reg2_def rtw89_8851b_phy_bb_reg_gain[] = { + {0xF00100FF, 0x00000000}, + {0xF00200FF, 0x00000001}, + {0xF00300FF, 0x00000002}, + {0xF00400FF, 0x00000003}, + {0xF00500FF, 0x00000004}, + {0xF00600FF, 0x00000005}, + {0x800100ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x000, 0x13F6D7B6}, + {0x001, 0x00725132}, + {0x002, 0x00005A38}, + {0x900200ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x000, 0x13F6D7B6}, + {0x001, 0x00725132}, + {0x002, 0x00005A38}, + {0x900300ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x000, 0x13F6D7B6}, + {0x001, 0x00725132}, + {0x002, 0x00005A38}, + {0x900400ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x000, 0x19FADCBC}, + {0x001, 0x007A5A3A}, + {0x002, 0x00005838}, + {0x900500ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x000, 0x19FADCBC}, + {0x001, 0x007A5A3A}, + {0x002, 0x00005838}, + {0x900600ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x000, 0x19FADCBC}, + {0x001, 0x007A5A3A}, + {0x002, 0x00005838}, + {0xA0000000, 0x00000000}, + {0x000, 0x13F6D7B6}, + {0x001, 0x00725132}, + {0x002, 0x00005A38}, + {0xB0000000, 0x00000000}, + {0x800100ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x100, 0x1BFEE0B7}, + {0x101, 0x006C5238}, + {0x102, 0x00005031}, + {0x900200ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x100, 0x1BFEE0B7}, + {0x101, 0x006C5238}, + {0x102, 0x00005031}, + {0x900300ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x100, 0x1BFEE0B7}, + {0x101, 0x006C5238}, + {0x102, 0x00005031}, + {0x900400ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x100, 0x1BFEE0B7}, + {0x101, 0x006C5238}, + {0x102, 0x00005031}, + {0x900500ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x100, 0x1BFEE0B7}, + {0x101, 0x006C5238}, + {0x102, 0x00005031}, + {0x900600ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x100, 0x1BFEE0B7}, + {0x101, 0x006C5238}, + {0x102, 0x00005031}, + {0xA0000000, 0x00000000}, + {0x100, 0x1BFEE0B7}, + {0x101, 0x006C5238}, + {0x102, 0x00005031}, + {0xB0000000, 0x00000000}, + {0x800100ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x10000, 0x19F8D8C1}, + {0x10001, 0x006F4F31}, + {0x10002, 0x00006F58}, + {0x900200ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x10000, 0x19F8D8C1}, + {0x10001, 0x006F4F31}, + {0x10002, 0x00006F58}, + {0x900300ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x10000, 0x19F8D8C1}, + {0x10001, 0x006F4F31}, + {0x10002, 0x00006F58}, + {0x900400ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x10000, 0x1DF8DAC1}, + {0x10001, 0x00755437}, + {0x10002, 0x00007058}, + {0x900500ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x10000, 0x1DF8DAC1}, + {0x10001, 0x00755437}, + {0x10002, 0x00007058}, + {0x900600ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x10000, 0x1DF8DAC1}, + {0x10001, 0x00755437}, + {0x10002, 0x00007058}, + {0xA0000000, 0x00000000}, + {0x10000, 0x19F8D8C1}, + {0x10001, 0x006F4F31}, + {0x10002, 0x00006F58}, + {0xB0000000, 0x00000000}, + {0x800100ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x10100, 0x09E9C69F}, + {0x10101, 0x00674627}, + {0x10102, 0x00006750}, + {0x900200ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x10100, 0x09E9C69F}, + {0x10101, 0x00674627}, + {0x10102, 0x00006750}, + {0x900300ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x10100, 0x09E9C69F}, + {0x10101, 0x00674627}, + {0x10102, 0x00006750}, + {0x900400ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x10100, 0x09E9C69F}, + {0x10101, 0x00674627}, + {0x10102, 0x00006750}, + {0x900500ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x10100, 0x09E9C69F}, + {0x10101, 0x00674627}, + {0x10102, 0x00006750}, + {0x900600ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x10100, 0x09E9C69F}, + {0x10101, 0x00674627}, + {0x10102, 0x00006750}, + {0xA0000000, 0x00000000}, + {0x10100, 0x09E9C69F}, + {0x10101, 0x00674627}, + {0x10102, 0x00006750}, + {0xB0000000, 0x00000000}, + {0x800100ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x20000, 0x1AF0D2B8}, + {0x20001, 0x00755334}, + {0x20002, 0x00006F58}, + {0x900200ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x20000, 0x1AF0D2B8}, + {0x20001, 0x00755334}, + {0x20002, 0x00006F58}, + {0x900300ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x20000, 0x1AF0D2B8}, + {0x20001, 0x00755334}, + {0x20002, 0x00006F58}, + {0x900400ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x20000, 0x1D00E2C8}, + {0x20001, 0x00775336}, + {0x20002, 0x00006D58}, + {0x900500ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x20000, 0x1D00E2C8}, + {0x20001, 0x00775336}, + {0x20002, 0x00006D58}, + {0x900600ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x20000, 0x1D00E2C8}, + {0x20001, 0x00775336}, + {0x20002, 0x00006D58}, + {0xA0000000, 0x00000000}, + {0x20000, 0x1AF0D2B8}, + {0x20001, 0x00755334}, + {0x20002, 0x00006F58}, + {0xB0000000, 0x00000000}, + {0x800100ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x20100, 0x07E9C6A0}, + {0x20101, 0x00674728}, + {0x20102, 0x00006850}, + {0x900200ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x20100, 0x07E9C6A0}, + {0x20101, 0x00674728}, + {0x20102, 0x00006850}, + {0x900300ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x20100, 0x07E9C6A0}, + {0x20101, 0x00674728}, + {0x20102, 0x00006850}, + {0x900400ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x20100, 0x07E9C6A0}, + {0x20101, 0x00674728}, + {0x20102, 0x00006850}, + {0x900500ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x20100, 0x07E9C6A0}, + {0x20101, 0x00674728}, + {0x20102, 0x00006850}, + {0x900600ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x20100, 0x07E9C6A0}, + {0x20101, 0x00674728}, + {0x20102, 0x00006850}, + {0xA0000000, 0x00000000}, + {0x20100, 0x07E9C6A0}, + {0x20101, 0x00674728}, + {0x20102, 0x00006850}, + {0xB0000000, 0x00000000}, + {0x800100ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x30000, 0x15EED2B6}, + {0x30001, 0x006F4D2F}, + {0x30002, 0x00006F58}, + {0x900200ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x30000, 0x15EED2B6}, + {0x30001, 0x006F4D2F}, + {0x30002, 0x00006F58}, + {0x900300ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x30000, 0x15EED2B6}, + {0x30001, 0x006F4D2F}, + {0x30002, 0x00006F58}, + {0x900400ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x30000, 0x1F00E2C6}, + {0x30001, 0x00795739}, + {0x30002, 0x00006F58}, + {0x900500ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x30000, 0x1F00E2C6}, + {0x30001, 0x00795739}, + {0x30002, 0x00006F58}, + {0x900600ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x30000, 0x1F00E2C6}, + {0x30001, 0x00795739}, + {0x30002, 0x00006F58}, + {0xA0000000, 0x00000000}, + {0x30000, 0x15EED2B6}, + {0x30001, 0x006F4D2F}, + {0x30002, 0x00006F58}, + {0xB0000000, 0x00000000}, + {0x800100ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x30100, 0x06E9C69F}, + {0x30101, 0x00654527}, + {0x30102, 0x00006750}, + {0x900200ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x30100, 0x06E9C69F}, + {0x30101, 0x00654527}, + {0x30102, 0x00006750}, + {0x900300ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x30100, 0x06E9C69F}, + {0x30101, 0x00654527}, + {0x30102, 0x00006750}, + {0x900400ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x30100, 0x06E9C69F}, + {0x30101, 0x00654527}, + {0x30102, 0x00006750}, + {0x900500ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x30100, 0x06E9C69F}, + {0x30101, 0x00654527}, + {0x30102, 0x00006750}, + {0x900600ff, 0x00000000}, {0x40000000, 0x00000000}, + {0x30100, 0x06E9C69F}, + {0x30101, 0x00654527}, + {0x30102, 0x00006750}, + {0xA0000000, 0x00000000}, + {0x30100, 0x06E9C69F}, + {0x30101, 0x00654527}, + {0x30102, 0x00006750}, + {0xB0000000, 0x00000000}, + {0x1000000, 0x000000F4}, + {0x1000010, 0x000000F8}, + {0x1000011, 0x0000F8F8}, + {0x1000100, 0x000000F8}, + {0x1000110, 0x00000000}, + {0x1000111, 0x00000000}, + {0x1010000, 0x000000F4}, + {0x1010010, 0x000000F8}, + {0x1010011, 0x0000F8F8}, + {0x1010020, 0x000000F8}, + {0x1010021, 0x0808E8E8}, + {0x1010029, 0x0000F8F8}, + {0x1010100, 0x000000F4}, + {0x1010110, 0x000000F8}, + {0x1010111, 0x0000F8F8}, + {0x1010120, 0x000000F8}, + {0x1010121, 0x0808E8E8}, + {0x1010129, 0x0000F8F8}, + {0x1020000, 0x000000F4}, + {0x1020010, 0x000000F8}, + {0x1020011, 0x0000F8F8}, + {0x1020020, 0x000000F8}, + {0x1020021, 0x0808E8E8}, + {0x1020029, 0x0000F8F8}, + {0x1020100, 0x000000F4}, + {0x1020110, 0x000000F8}, + {0x1020111, 0x0000F8F8}, + {0x1020120, 0x000000F8}, + {0x1020121, 0x0808E8E8}, + {0x1020129, 0x0000F8F8}, + {0x1030000, 0x000000F4}, + {0x1030010, 0x000000F8}, + {0x1030011, 0x0000F8F8}, + {0x1030020, 0x000000F8}, + {0x1030021, 0x0808E8E8}, + {0x1030029, 0x0000F8F8}, + {0x1030100, 0x000000F4}, + {0x1030110, 0x000000F8}, + {0x1030111, 0x0000F8F8}, + {0x1030120, 0x000000F8}, + {0x1030121, 0x0808E8E8}, + {0x1030129, 0x0000F8F8}, +}; + +static const struct rtw89_reg2_def rtw89_8851b_phy_radioa_regs[] = { + {0xF0010000, 0x00000000}, + {0xF0020000, 0x00000001}, + {0xF0030000, 0x00000002}, + {0x000, 0x00030000}, + {0x018, 0x00013124}, + {0x0EF, 0x00080000}, + {0x033, 0x00000008}, + {0x03E, 0x00000110}, + {0x03F, 0x0000D39C}, + {0x033, 0x0000000C}, + {0x03E, 0x00000110}, + {0x03F, 0x0000F79E}, + {0x0EF, 0x00000000}, + {0x01B, 0x00003A40}, + {0x08F, 0x000C170E}, + {0x08E, 0x00005160}, + {0x002, 0x00000600}, + {0x0EE, 0x00000002}, + {0x033, 0x00000002}, + {0x03F, 0x0000003F}, + {0x033, 0x00000003}, + {0x03F, 0x0000003F}, + {0x033, 0x00000004}, + {0x03F, 0x0000003F}, + {0x033, 0x00000005}, + {0x03F, 0x0000003F}, + {0x033, 0x00000006}, + {0x03F, 0x0000003F}, + {0x033, 0x00000007}, + {0x03F, 0x0000003F}, + {0x033, 0x00000008}, + {0x03F, 0x0000003F}, + {0x033, 0x0000000C}, + {0x03F, 0x0000003F}, + {0x033, 0x0000000D}, + {0x03F, 0x0000003F}, + {0x033, 0x0000000E}, + {0x03F, 0x0000003F}, + {0x0EE, 0x00000000}, + {0x0EF, 0x00004000}, + {0x033, 0x00000007}, + {0x03E, 0x00000000}, + {0x03F, 0x00000707}, + {0x033, 0x00000006}, + {0x03E, 0x00000000}, + {0x03F, 0x00000704}, + {0x033, 0x00000005}, + {0x03E, 0x00000000}, + {0x03F, 0x00020500}, + {0x033, 0x00000004}, + {0x03E, 0x00000000}, + {0x03F, 0x00010404}, + {0x033, 0x00000003}, + {0x03E, 0x00000000}, + {0x03F, 0x00099B04}, + {0x033, 0x00000002}, + {0x03E, 0x00000000}, + {0x03F, 0x00092B04}, + {0x033, 0x00000001}, + {0x03E, 0x00000000}, + {0x03F, 0x000B3204}, + {0x033, 0x00000000}, + {0x03E, 0x00000000}, + {0x03F, 0x00003000}, + {0x033, 0x00000017}, + {0x03E, 0x00000000}, + {0x03F, 0x00000787}, + {0x033, 0x00000016}, + {0x03E, 0x00000000}, + {0x03F, 0x00000784}, + {0x033, 0x00000015}, + {0x03E, 0x00000000}, + {0x03F, 0x00020580}, + {0x033, 0x00000014}, + {0x03E, 0x00000000}, + {0x03F, 0x00010484}, + {0x033, 0x00000013}, + {0x03E, 0x00000000}, + {0x03F, 0x00099B84}, + {0x033, 0x00000012}, + {0x03E, 0x00000000}, + {0x03F, 0x00092B84}, + {0x033, 0x00000011}, + {0x03E, 0x00000000}, + {0x03F, 0x000B3284}, + {0x033, 0x00000010}, + {0x03E, 0x00000000}, + {0x03F, 0x00003080}, + {0x0EF, 0x00000000}, + {0x0EE, 0x00000010}, + {0x033, 0x00000006}, + {0x03F, 0x00000003}, + {0x033, 0x00000007}, + {0x03F, 0x00000003}, + {0x0EE, 0x00000000}, + {0x0EF, 0x00001000}, + {0x033, 0x00000000}, + {0x03F, 0x00000034}, + {0x033, 0x00000001}, + {0x03F, 0x00000037}, + {0x033, 0x00000002}, + {0x03F, 0x00000034}, + {0x033, 0x00000003}, + {0x03F, 0x00000024}, + {0x033, 0x00000004}, + {0x03F, 0x00000037}, + {0x033, 0x00000005}, + {0x03F, 0x00000027}, + {0x0EF, 0x00000000}, + {0x0EC, 0x00000400}, + {0x033, 0x00000001}, + {0x03F, 0x00000022}, + {0x033, 0x00000003}, + {0x03F, 0x00000022}, + {0x033, 0x00000009}, + {0x03F, 0x00000022}, + {0x0EC, 0x00000000}, + {0x0EC, 0x00000004}, + {0x033, 0x00000000}, + {0x03F, 0x000000AE}, + {0x033, 0x00000001}, + {0x03F, 0x0000008C}, + {0x033, 0x00000002}, + {0x03F, 0x0000006A}, + {0x033, 0x00000003}, + {0x03F, 0x00000048}, + {0x033, 0x00000004}, + {0x03F, 0x00000026}, + {0x033, 0x00000005}, + {0x03F, 0x00000004}, + {0x033, 0x00000006}, + {0x03F, 0x00000002}, + {0x033, 0x00000007}, + {0x03F, 0x00000000}, + {0x0EC, 0x00000000}, + {0x0EF, 0x00008000}, + {0x033, 0x00000007}, + {0x03E, 0x00000003}, + {0x03F, 0x00001FB0}, + {0x033, 0x00000006}, + {0x03E, 0x00000003}, + {0x03F, 0x00001FB0}, + {0x033, 0x00000005}, + {0x03E, 0x00000003}, + {0x03F, 0x00001DB0}, + {0x033, 0x00000004}, + {0x03E, 0x00000003}, + {0x03F, 0x00001CB0}, + {0x033, 0x00000003}, + {0x03E, 0x00000003}, + {0x03F, 0x00001BB0}, + {0x033, 0x00000002}, + {0x03E, 0x00000003}, + {0x03F, 0x00001AB0}, + {0x033, 0x00000001}, + {0x03E, 0x00000003}, + {0x03F, 0x0000D9BC}, + {0x033, 0x00000000}, + {0x03E, 0x00000003}, + {0x03F, 0x0000D4BC}, + {0x033, 0x00000017}, + {0x03E, 0x00000003}, + {0x03F, 0x00001FB0}, + {0x033, 0x00000016}, + {0x03E, 0x00000003}, + {0x03F, 0x00001FB0}, + {0x033, 0x00000015}, + {0x03E, 0x00000003}, + {0x03F, 0x00001DB0}, + {0x033, 0x00000014}, + {0x03E, 0x00000003}, + {0x03F, 0x00001CB0}, + {0x033, 0x00000013}, + {0x03E, 0x00000003}, + {0x03F, 0x00001BB0}, + {0x033, 0x00000012}, + {0x03E, 0x00000003}, + {0x03F, 0x00001AB0}, + {0x033, 0x00000011}, + {0x03E, 0x00000003}, + {0x03F, 0x0000D9BC}, + {0x033, 0x00000010}, + {0x03E, 0x00000003}, + {0x03F, 0x0000D4BC}, + {0x033, 0x00000027}, + {0x03E, 0x00000003}, + {0x03F, 0x00001FB0}, + {0x033, 0x00000026}, + {0x03E, 0x00000003}, + {0x03F, 0x00001FB0}, + {0x033, 0x00000025}, + {0x03E, 0x00000003}, + {0x03F, 0x00001DB0}, + {0x033, 0x00000024}, + {0x03E, 0x00000003}, + {0x03F, 0x00001CB0}, + {0x033, 0x00000023}, + {0x03E, 0x00000003}, + {0x03F, 0x00001BB0}, + {0x033, 0x00000022}, + {0x03E, 0x00000003}, + {0x03F, 0x00001AB0}, + {0x033, 0x00000021}, + {0x03E, 0x00000003}, + {0x03F, 0x0000D9BC}, + {0x033, 0x00000020}, + {0x03E, 0x00000003}, + {0x03F, 0x0000D4BC}, + {0x033, 0x0000000E}, + {0x03E, 0x00000003}, + {0x03F, 0x00001FB0}, + {0x033, 0x0000000D}, + {0x03E, 0x00000003}, + {0x03F, 0x00001DB0}, + {0x033, 0x0000000C}, + {0x03E, 0x00000003}, + {0x03F, 0x00001CB0}, + {0x033, 0x0000000B}, + {0x03E, 0x00000003}, + {0x03F, 0x00091BB0}, + {0x033, 0x0000000A}, + {0x03E, 0x00000003}, + {0x03F, 0x000A9AB0}, + {0x033, 0x00000009}, + {0x03E, 0x00000003}, + {0x03F, 0x000BD9BC}, + {0x033, 0x00000008}, + {0x03E, 0x00000003}, + {0x03F, 0x0009D4BC}, + {0x033, 0x0000001E}, + {0x03E, 0x00000003}, + {0x03F, 0x00001FB0}, + {0x033, 0x0000001D}, + {0x03E, 0x00000003}, + {0x03F, 0x00001DB0}, + {0x033, 0x0000001C}, + {0x03E, 0x00000003}, + {0x03F, 0x00001CB0}, + {0x033, 0x0000001B}, + {0x03E, 0x00000003}, + {0x03F, 0x00091BB0}, + {0x033, 0x0000001A}, + {0x03E, 0x00000003}, + {0x03F, 0x00099AB0}, + {0x033, 0x00000019}, + {0x03E, 0x00000003}, + {0x03F, 0x000AD9BC}, + {0x033, 0x00000018}, + {0x03E, 0x00000003}, + {0x03F, 0x0009D4BC}, + {0x033, 0x0000002E}, + {0x03E, 0x00000003}, + {0x03F, 0x00001FB0}, + {0x033, 0x0000002D}, + {0x03E, 0x00000003}, + {0x03F, 0x00001DB0}, + {0x033, 0x0000002C}, + {0x03E, 0x00000003}, + {0x03F, 0x00001CB0}, + {0x033, 0x0000002B}, + {0x03E, 0x00000003}, + {0x03F, 0x00001BB0}, + {0x033, 0x0000002A}, + {0x03E, 0x00000003}, + {0x03F, 0x00009AB0}, + {0x033, 0x00000029}, + {0x03E, 0x00000003}, + {0x03F, 0x0009D9BC}, + {0x033, 0x00000028}, + {0x03E, 0x00000003}, + {0x03F, 0x0000D4BC}, + {0x0EF, 0x00000000}, + {0x0EF, 0x00002000}, + {0x033, 0x00000000}, + {0x03F, 0x00000005}, + {0x033, 0x00000001}, + {0x03F, 0x00000004}, + {0x033, 0x00000002}, + {0x03F, 0x00000004}, + {0x033, 0x00000004}, + {0x03F, 0x0000000C}, + {0x033, 0x00000005}, + {0x03F, 0x00000003}, + {0x033, 0x00000006}, + {0x03F, 0x00000003}, + {0x0EF, 0x00000000}, + {0x06C, 0x00038085}, + {0x06D, 0x00000D6B}, + {0x06E, 0x0001FB89}, + {0x06F, 0x00097B99}, + {0x069, 0x00008040}, + {0x0EF, 0x00000200}, + {0x033, 0x00000004}, + {0x03F, 0x000008FF}, + {0x033, 0x00000005}, + {0x03F, 0x000004F2}, + {0x033, 0x00000006}, + {0x03F, 0x00000217}, + {0x033, 0x00000007}, + {0x03F, 0x00000131}, + {0x0EF, 0x00000000}, + {0x0EF, 0x00000400}, + {0x033, 0x00000004}, + {0x03F, 0x000004F7}, + {0x033, 0x00000005}, + {0x03F, 0x000004F7}, + {0x033, 0x00000006}, + {0x03F, 0x000004F2}, + {0x033, 0x00000007}, + {0x03F, 0x00000117}, + {0x0EF, 0x00000000}, + {0x043, 0x00005000}, + {0x036, 0x000147D0}, + {0x0B0, 0x0008677C}, + {0x0B1, 0x00012920}, + {0x0BB, 0x000EF000}, + {0x0CB, 0x000A9594}, + {0x0CC, 0x000C36D2}, + {0x0CD, 0x00024923}, + {0x0CE, 0x00020180}, + {0x0CF, 0x00000000}, + {0x0D5, 0x0006E27A}, + {0x0D8, 0x00000044}, + {0x0D9, 0x00000007}, + {0x0DD, 0x00000020}, + {0x0E3, 0x0000002C}, + {0x0B7, 0x0000000C}, + {0x0E1, 0x000080C0}, + {0x0E4, 0x00000380}, + {0x0ED, 0x00002000}, + {0x033, 0x00000001}, + {0x03D, 0x000A6094}, + {0x03E, 0x00003449}, + {0x03F, 0x00000001}, + {0x033, 0x00000003}, + {0x03D, 0x000AA094}, + {0x03E, 0x00003449}, + {0x03F, 0x00000001}, + {0x0ED, 0x00000000}, + {0x0ED, 0x00000100}, + {0x033, 0x00000000}, + {0x03F, 0x0000007F}, + {0x033, 0x00000001}, + {0x03F, 0x0000007F}, + {0x033, 0x00000002}, + {0x03F, 0x0000007F}, + {0x033, 0x00000003}, + {0x03F, 0x0000007F}, + {0x033, 0x00000004}, + {0x03F, 0x0000007F}, + {0x033, 0x00000005}, + {0x03F, 0x0000007F}, + {0x033, 0x00000007}, + {0x03F, 0x0000007F}, + {0x033, 0x00000008}, + {0x03F, 0x0000007F}, + {0x033, 0x00000009}, + {0x03F, 0x0000007F}, + {0x0ED, 0x00000000}, + {0x0ED, 0x00000080}, + {0x033, 0x00000000}, + {0x03E, 0x000007E1}, + {0x03F, 0x0001F87F}, + {0x033, 0x00000010}, + {0x03E, 0x000007E1}, + {0x03F, 0x0001F87F}, + {0x033, 0x00000030}, + {0x03E, 0x000007E1}, + {0x03F, 0x0001F87F}, + {0x033, 0x00000040}, + {0x03E, 0x000007E1}, + {0x03F, 0x0001F87F}, + {0x033, 0x00000050}, + {0x03E, 0x000007E1}, + {0x03F, 0x0001F87F}, + {0x033, 0x00000070}, + {0x03E, 0x000007E1}, + {0x03F, 0x0001F87F}, + {0x0ED, 0x00000000}, + {0x0ED, 0x00000004}, + {0x033, 0x00000000}, + {0x03F, 0x00008420}, + {0x0ED, 0x00000000}, + {0x018, 0x00011108}, + {0x0B9, 0x00000000}, + {0x0B9, 0x00000000}, + {0x0B9, 0x00000200}, + {0x0FF, 0x00000000}, + {0x0FF, 0x00000000}, + {0x0FF, 0x00000000}, + {0x0FF, 0x00000000}, + {0x0FF, 0x00000000}, + {0x0FF, 0x00000000}, + {0x0FF, 0x00000000}, + {0x0FF, 0x00000000}, + {0x0FF, 0x00000000}, + {0x0FF, 0x00000000}, + {0x0B9, 0x00000000}, + {0x018, 0x00013124}, + {0x05A, 0x0006808F}, + {0x0ED, 0x00000008}, + {0x033, 0x00000001}, + {0x03F, 0x0000000F}, + {0x0ED, 0x00000000}, + {0x000, 0x00020000}, + {0x018, 0x00010124}, + {0x0EE, 0x00000800}, + {0x033, 0x00000004}, + {0x03F, 0x00000002}, + {0x033, 0x00000005}, + {0x03F, 0x00000003}, + {0x033, 0x00000006}, + {0x03F, 0x00000006}, + {0x033, 0x00000007}, + {0x03F, 0x00000007}, + {0x0EE, 0x00000000}, + {0x0EE, 0x00001000}, + {0x033, 0x00000008}, + {0x03F, 0x00000000}, + {0x033, 0x00000009}, + {0x03F, 0x00000001}, + {0x033, 0x0000000A}, + {0x03F, 0x00000003}, + {0x033, 0x0000000B}, + {0x03F, 0x00000103}, + {0x033, 0x0000000C}, + {0x03F, 0x00000107}, + {0x033, 0x0000000D}, + {0x03F, 0x00000207}, + {0x033, 0x0000000E}, + {0x03F, 0x00000307}, + {0x033, 0x0000000F}, + {0x03F, 0x00000307}, + {0x0EE, 0x00000000}, + {0x0EE, 0x00000200}, + {0x033, 0x00000004}, + {0x03F, 0x00000000}, + {0x033, 0x00000005}, + {0x03F, 0x00000001}, + {0x033, 0x00000006}, + {0x03F, 0x00000002}, + {0x033, 0x00000007}, + {0x03F, 0x00000003}, + {0x0EE, 0x00000000}, + {0x011, 0x00014062}, + {0x0EF, 0x00000010}, + {0x033, 0x00000001}, + {0x03F, 0x00000DF3}, + {0x033, 0x00000002}, + {0x03F, 0x00000DF3}, + {0x033, 0x00000003}, + {0x03F, 0x00000A83}, + {0x033, 0x00000004}, + {0x03F, 0x00000A83}, + {0x033, 0x00000005}, + {0x03F, 0x00000643}, + {0x033, 0x00000006}, + {0x03F, 0x00000643}, + {0x0EF, 0x00000000}, + {0x0EF, 0x00000100}, + {0x033, 0x00000001}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000002}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000003}, + {0x03F, 0x0001B5A9}, + {0x033, 0x00000004}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000005}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000006}, + {0x03F, 0x0001B589}, + {0x033, 0x00000007}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000008}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000009}, + {0x03F, 0x0001B5A8}, + {0x033, 0x0000000A}, + {0x03F, 0x0001B5A8}, + {0x033, 0x0000000B}, + {0x03F, 0x0001B5A9}, + {0x033, 0x0000000C}, + {0x03F, 0x0001B5A8}, + {0x033, 0x0000000D}, + {0x03F, 0x0001B5A9}, + {0x033, 0x0000000E}, + {0x03F, 0x0001B5A8}, + {0x033, 0x0000000F}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000010}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000011}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000012}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000013}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000014}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000015}, + {0x03F, 0x0001B589}, + {0x033, 0x00000016}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000017}, + {0x03F, 0x0001B5A9}, + {0x033, 0x00000018}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000019}, + {0x03F, 0x0001B5A8}, + {0x033, 0x0000001A}, + {0x03F, 0x0001B5A8}, + {0x033, 0x0000001B}, + {0x03F, 0x0001B5A8}, + {0x033, 0x0000001C}, + {0x03F, 0x0001B5A8}, + {0x033, 0x0000001D}, + {0x03F, 0x0001B5A8}, + {0x033, 0x0000001E}, + {0x03F, 0x0001B5A8}, + {0x033, 0x0000001F}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000020}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000021}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000022}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000023}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000024}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000025}, + {0x03F, 0x0001B5A8}, + {0x033, 0x00000022}, + {0x03F, 0x0001B5A8}, + {0x0EF, 0x00000000}, + {0x0EF, 0x00000040}, + {0x033, 0x00000001}, + {0x03F, 0x000002C5}, + {0x033, 0x00000002}, + {0x03F, 0x000002C5}, + {0x033, 0x00000003}, + {0x03F, 0x000002C5}, + {0x033, 0x00000004}, + {0x03F, 0x000002C5}, + {0x033, 0x00000005}, + {0x03F, 0x000002C5}, + {0x033, 0x00000006}, + {0x03F, 0x000002C5}, + {0x033, 0x00000007}, + {0x03F, 0x000002C5}, + {0x033, 0x00000008}, + {0x03F, 0x000002C5}, + {0x033, 0x00000009}, + {0x03F, 0x000002C5}, + {0x033, 0x0000000A}, + {0x03F, 0x000002C5}, + {0x033, 0x0000000B}, + {0x03F, 0x000002C5}, + {0x0EF, 0x00000000}, + {0x059, 0x00050033}, + {0x061, 0x0005F48A}, + {0x062, 0x00077435}, + {0x063, 0x000F80A2}, + {0x065, 0x00018F22}, + {0x067, 0x00008060}, + {0x07E, 0x0009780B}, + {0x0EE, 0x00000004}, + {0x033, 0x0000000B}, + {0x03F, 0x0000000B}, + {0x033, 0x0000000C}, + {0x03F, 0x00000012}, + {0x033, 0x0000000D}, + {0x03F, 0x00000019}, + {0x033, 0x0000000F}, + {0x03F, 0x0000000B}, + {0x033, 0x00000010}, + {0x03F, 0x00000012}, + {0x033, 0x00000011}, + {0x03F, 0x00000019}, + {0x03F, 0x00000000}, + {0x0EE, 0x00000000}, + {0x0EE, 0x00000800}, + {0x033, 0x00000000}, + {0x03F, 0x00000001}, + {0x033, 0x00000001}, + {0x03F, 0x00000002}, + {0x033, 0x00000002}, + {0x03F, 0x00000003}, + {0x033, 0x00000003}, + {0x03F, 0x00000007}, + {0x0EE, 0x00000000}, + {0x0EE, 0x00001000}, + {0x033, 0x00000000}, + {0x03F, 0x00003000}, + {0x033, 0x00000001}, + {0x03F, 0x00000000}, + {0x033, 0x00000002}, + {0x03F, 0x00000001}, + {0x033, 0x00000003}, + {0x03F, 0x00000003}, + {0x033, 0x00000004}, + {0x03F, 0x00000007}, + {0x033, 0x00000005}, + {0x03F, 0x0000000F}, + {0x033, 0x00000006}, + {0x03F, 0x0000010F}, + {0x033, 0x00000007}, + {0x03F, 0x0000030F}, + {0x0EE, 0x00000000}, + {0x0EE, 0x00000200}, + {0x033, 0x00000000}, + {0x03F, 0x00000004}, + {0x033, 0x00000001}, + {0x03F, 0x00000005}, + {0x033, 0x00000002}, + {0x03F, 0x00000006}, + {0x033, 0x00000003}, + {0x03F, 0x00000007}, + {0x0EE, 0x00000000}, + {0x0EF, 0x00000080}, + {0x033, 0x00000004}, + {0x03E, 0x0000001D}, + {0x03F, 0x0001A241}, + {0x033, 0x00000005}, + {0x03E, 0x0000001D}, + {0x03F, 0x0001A241}, + {0x033, 0x00000006}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x00000007}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x00000008}, + {0x03E, 0x0000001D}, + {0x03F, 0x0001A241}, + {0x033, 0x00000009}, + {0x03E, 0x0001A241}, + {0x03F, 0x0001C3C1}, + {0x033, 0x0000000A}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x0000000B}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x0000000C}, + {0x03E, 0x0000001D}, + {0x03F, 0x0001A241}, + {0x033, 0x0000000D}, + {0x03E, 0x0000001D}, + {0x03F, 0x0001A241}, + {0x033, 0x0000000E}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x0000000F}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x00000010}, + {0x03E, 0x0000001C}, + {0x03F, 0x000199C1}, + {0x033, 0x00000011}, + {0x03E, 0x0000001C}, + {0x03F, 0x000199C1}, + {0x033, 0x00000012}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x00000013}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x00000014}, + {0x03E, 0x0000001C}, + {0x03F, 0x000199C1}, + {0x033, 0x00000015}, + {0x03E, 0x0000001C}, + {0x03F, 0x000199C1}, + {0x033, 0x00000016}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x00000017}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x00000018}, + {0x03E, 0x0000001C}, + {0x03F, 0x000199C1}, + {0x033, 0x00000019}, + {0x03E, 0x0000001C}, + {0x03F, 0x000199C1}, + {0x033, 0x0000001A}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x0000001B}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x0000001C}, + {0x03E, 0x0000001C}, + {0x03F, 0x000199C1}, + {0x033, 0x0000001D}, + {0x03E, 0x0000001C}, + {0x03F, 0x000199C1}, + {0x033, 0x0000001E}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x0000001F}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x00000020}, + {0x03E, 0x0000001C}, + {0x03F, 0x000199C1}, + {0x033, 0x00000021}, + {0x03E, 0x0000001C}, + {0x03F, 0x000199C1}, + {0x033, 0x00000022}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x00000023}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x00000024}, + {0x03E, 0x0000001C}, + {0x03F, 0x0001E141}, + {0x033, 0x00000025}, + {0x03E, 0x0000001C}, + {0x03F, 0x0001E141}, + {0x033, 0x00000026}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x00000027}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x00000028}, + {0x03E, 0x0000001C}, + {0x03F, 0x0001E141}, + {0x033, 0x00000029}, + {0x03E, 0x0000001C}, + {0x03F, 0x0001E141}, + {0x033, 0x0000002A}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x0000002B}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x0000002C}, + {0x03E, 0x0000001C}, + {0x03F, 0x0001E141}, + {0x033, 0x0000002D}, + {0x03E, 0x0000001C}, + {0x03F, 0x0001E141}, + {0x033, 0x0000002E}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x0000002F}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x00000030}, + {0x03E, 0x0000001C}, + {0x03F, 0x0001E141}, + {0x033, 0x00000031}, + {0x03E, 0x0000001C}, + {0x03F, 0x0001E141}, + {0x033, 0x00000032}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x00000033}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x00000034}, + {0x03E, 0x0000001C}, + {0x03F, 0x0001E141}, + {0x033, 0x00000035}, + {0x03E, 0x0000001C}, + {0x03F, 0x0001E141}, + {0x033, 0x00000036}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x00000037}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x00000038}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x00000039}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C1}, + {0x033, 0x0000003A}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C3}, + {0x033, 0x0000003B}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C3}, + {0x033, 0x0000003C}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C3}, + {0x033, 0x0000003D}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C3}, + {0x033, 0x0000003E}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C3}, + {0x033, 0x0000003F}, + {0x03E, 0x0000001B}, + {0x03F, 0x0001C3C3}, + {0x0EF, 0x00000000}, + {0x051, 0x0003D368}, + {0x052, 0x000A3338}, + {0x053, 0x000688AF}, + {0x054, 0x00012C04}, + {0x058, 0x00084221}, + {0x05B, 0x000EB000}, + {0x100EE, 0x00002000}, + {0x10030, 0x000000F9}, + {0x10030, 0x000004F6}, + {0x10030, 0x000008F3}, + {0x10030, 0x00000CF0}, + {0x10030, 0x000010ED}, + {0x10030, 0x000014EA}, + {0x10030, 0x000018E7}, + {0x10030, 0x00001CE4}, + {0x10030, 0x000020E1}, + {0x10030, 0x000024A4}, + {0x10030, 0x000028A1}, + {0x10030, 0x00002C9E}, + {0x10030, 0x0000309B}, + {0x10030, 0x0000341E}, + {0x10030, 0x0000381B}, + {0x10030, 0x00003C18}, + {0x10030, 0x00004015}, + {0x10030, 0x000200BC}, + {0x10030, 0x000204B9}, + {0x10030, 0x000208B6}, + {0x10030, 0x00020CB3}, + {0x10030, 0x000210B0}, + {0x10030, 0x000214AD}, + {0x10030, 0x0002186C}, + {0x10030, 0x00021C69}, + {0x10030, 0x00022066}, + {0x10030, 0x00022426}, + {0x10030, 0x00022823}, + {0x10030, 0x00022C20}, + {0x10030, 0x0002301D}, + {0x10030, 0x0002341A}, + {0x10030, 0x00023817}, + {0x10030, 0x00023C14}, + {0x10030, 0x00024011}, + {0x10030, 0x000280BC}, + {0x10030, 0x000284B9}, + {0x10030, 0x000288B6}, + {0x10030, 0x00028CB3}, + {0x10030, 0x000290B0}, + {0x10030, 0x000294AD}, + {0x10030, 0x0002986C}, + {0x10030, 0x00029C69}, + {0x10030, 0x0002A066}, + {0x10030, 0x0002A426}, + {0x10030, 0x0002A823}, + {0x10030, 0x0002AC20}, + {0x10030, 0x0002B01D}, + {0x10030, 0x0002B41A}, + {0x10030, 0x0002B817}, + {0x10030, 0x0002BC14}, + {0x10030, 0x0002C011}, + {0x10030, 0x000300BC}, + {0x10030, 0x000304B9}, + {0x10030, 0x000308B6}, + {0x10030, 0x00030CB3}, + {0x10030, 0x000310B0}, + {0x10030, 0x000314AD}, + {0x10030, 0x0003186C}, + {0x10030, 0x00031C69}, + {0x10030, 0x00032066}, + {0x10030, 0x00032426}, + {0x10030, 0x00032823}, + {0x10030, 0x00032C20}, + {0x10030, 0x0003301D}, + {0x10030, 0x0003341A}, + {0x10030, 0x00033817}, + {0x10030, 0x00033C14}, + {0x10030, 0x00034011}, + {0x100EE, 0x00000000}, + {0x100EE, 0x00004000}, + {0x10030, 0x000201EF}, + {0x10030, 0x000205E9}, + {0x10030, 0x000209E3}, + {0x10030, 0x00020DDD}, + {0x10030, 0x000211D7}, + {0x10030, 0x000215D1}, + {0x10030, 0x00021919}, + {0x10030, 0x00021D13}, + {0x10030, 0x000220D9}, + {0x10030, 0x000224D3}, + {0x10030, 0x00022899}, + {0x10030, 0x00022C93}, + {0x10030, 0x00023059}, + {0x10030, 0x00023453}, + {0x10030, 0x00023819}, + {0x10030, 0x00023C13}, + {0x10030, 0x0002400D}, + {0x10030, 0x00024407}, + {0x10030, 0x000281EF}, + {0x10030, 0x000285E9}, + {0x10030, 0x000289E3}, + {0x10030, 0x00028DDD}, + {0x10030, 0x000291D7}, + {0x10030, 0x000295D1}, + {0x10030, 0x00029919}, + {0x10030, 0x00029D13}, + {0x10030, 0x0002A0D9}, + {0x10030, 0x0002A4D3}, + {0x10030, 0x0002A899}, + {0x10030, 0x0002AC93}, + {0x10030, 0x0002B059}, + {0x10030, 0x0002B453}, + {0x10030, 0x0002B819}, + {0x10030, 0x0002BC13}, + {0x10030, 0x0002C00D}, + {0x10030, 0x0002C407}, + {0x10030, 0x000301EF}, + {0x10030, 0x000305E9}, + {0x10030, 0x000309E3}, + {0x10030, 0x00030DDD}, + {0x10030, 0x000311D7}, + {0x10030, 0x000315D1}, + {0x10030, 0x00031919}, + {0x10030, 0x00031D13}, + {0x10030, 0x000320D9}, + {0x10030, 0x000324D3}, + {0x10030, 0x00032899}, + {0x10030, 0x00032C93}, + {0x10030, 0x00033059}, + {0x10030, 0x00033453}, + {0x10030, 0x00033819}, + {0x10030, 0x00033C13}, + {0x10030, 0x0003400D}, + {0x10030, 0x00034407}, + {0x100EE, 0x00000000}, + {0x100EE, 0x00004000}, + {0x10030, 0x000001EF}, + {0x10030, 0x000005E9}, + {0x10030, 0x000009E3}, + {0x10030, 0x00000DDD}, + {0x10030, 0x000011A5}, + {0x10030, 0x0000159F}, + {0x10030, 0x00001965}, + {0x10030, 0x00001D5F}, + {0x10030, 0x00002125}, + {0x10030, 0x0000251F}, + {0x10030, 0x000028E5}, + {0x10030, 0x00002CDF}, + {0x10030, 0x000030A5}, + {0x10030, 0x0000349F}, + {0x10030, 0x00003865}, + {0x10030, 0x00003C5F}, + {0x10030, 0x00004025}, + {0x10030, 0x0000441F}, + {0x100EE, 0x00000000}, + {0x0EF, 0x00000008}, + {0x033, 0x00000000}, + {0x03F, 0x00000004}, + {0x0EF, 0x00000000}, + {0x005, 0x00000001}, + {0x10005, 0x00000001}, + {0x0FE, 0x00000022}, +}; + +static const struct rtw89_reg2_def rtw89_8851b_phy_nctl_regs[] = { + {0x8000, 0x00000008}, + {0x8008, 0x00000000}, + {0x8004, 0xe8862b66}, + {0x800c, 0x78000000}, + {0x8010, 0x88015000}, + {0x8014, 0x80010100}, + {0x8018, 0x10010100}, + {0x801c, 0xa210bc00}, + {0x8020, 0x000403e0}, + {0x8024, 0x00072160}, + {0x8028, 0x00180e00}, + {0x8030, 0x400000c0}, + {0x8034, 0x11000830}, + {0x8038, 0x40000000}, + {0x803c, 0x00000008}, + {0x8040, 0x00000046}, + {0x8044, 0x0010001f}, + {0x8048, 0x00000003}, + {0x804c, 0x420840e0}, + {0x8050, 0xce08cce0}, + {0x8054, 0x420840e0}, + {0x8058, 0xce08cce0}, + {0x805c, 0x150c0b02}, + {0x8060, 0x150c0b02}, + {0x8064, 0x2aa00047}, + {0x8074, 0x80000000}, + {0x807c, 0x000000ee}, + {0x8088, 0x80000000}, + {0x808c, 0x00000000}, + {0x80b0, 0x00000000}, + {0x80cc, 0x00000000}, + {0x80d0, 0x00000000}, + {0x80ec, 0x00000002}, + {0x8098, 0x0000ff00}, + {0x8070, 0x00e80000}, + {0x80b0, 0xffe00fff}, + {0x809c, 0x0000001f}, + {0x80b8, 0x00002000}, + {0x80bc, 0x00050033}, + {0xa400, 0x00000000}, + {0xa404, 0x00000180}, + {0xa408, 0x000001af}, + {0xa40c, 0x000001e3}, + {0xa410, 0x00000220}, + {0xa414, 0x00000262}, + {0xa418, 0x000002ac}, + {0xa41c, 0x0000035e}, + {0xa420, 0x000003c7}, + {0xa424, 0x0000043d}, + {0xa428, 0x000004c1}, + {0xa42c, 0x00000556}, + {0xa430, 0x000005fc}, + {0xa434, 0x000006b7}, + {0xa438, 0x00000789}, + {0xa43c, 0x00000875}, + {0xa440, 0x0000011f}, + {0x8104, 0x00000000}, + {0x810c, 0x00000000}, + {0x8110, 0x00000000}, + {0x8114, 0x00000000}, + {0x8120, 0x10010000}, + {0x8124, 0x00000000}, + {0x8128, 0x00000200}, + {0x812c, 0x0000c000}, + {0x8130, 0x40000000}, + {0x8138, 0x40000000}, + {0x813c, 0x40000000}, + {0x8140, 0x00000000}, + {0x8144, 0x0b040b03}, + {0x8148, 0x07020b04}, + {0x814c, 0x07020b04}, + {0x8150, 0xe4e40000}, + {0x8158, 0xffffffff}, + {0x815c, 0xffffffff}, + {0x8160, 0xffffffff}, + {0x8164, 0xffffffff}, + {0x8168, 0xffffffff}, + {0x816c, 0x1fffffff}, + {0x81cc, 0x00000000}, + {0x81dc, 0x00000002}, + {0x81e0, 0x00000000}, + {0x81e4, 0x00000001}, + {0x81a0, 0x00000000}, + {0x81ac, 0x3fc20400}, + {0x81b0, 0x3f914100}, + {0x81bc, 0x0000005b}, + {0x81c0, 0x0000005b}, + {0x81b4, 0x01e0f078}, + {0x81b8, 0x01e0f078}, + {0x81f0, 0x0000f078}, + {0x81d8, 0x00000001}, + {0x9500, 0x00000000}, + {0x9504, 0x00000000}, + {0x9508, 0x00000000}, + {0x950c, 0x00000000}, + {0x9510, 0x00000000}, + {0x9514, 0x00000000}, + {0x9518, 0x00000000}, + {0x951c, 0x00000000}, + {0x9520, 0x00000000}, + {0x9524, 0x00000000}, + {0x9528, 0x00000000}, + {0x952c, 0x00000000}, + {0x9530, 0x00000000}, + {0x9534, 0x00000000}, + {0x9538, 0x00000000}, + {0x953c, 0x00000000}, + {0x9540, 0x04000000}, + {0x9544, 0x00000000}, + {0x9548, 0x00000000}, + {0x954c, 0x00000000}, + {0x9550, 0x00000000}, + {0x9554, 0x00000000}, + {0x9558, 0x00000000}, + {0x955c, 0x00000000}, + {0x9560, 0x00000000}, + {0x9564, 0x00000000}, + {0x9568, 0x00000000}, + {0x956c, 0x00000000}, + {0x9570, 0x00000000}, + {0x9574, 0x00000000}, + {0x9578, 0x00000000}, + {0x957c, 0x00000000}, + {0x9580, 0x00000000}, + {0x9584, 0x04000000}, + {0x9588, 0x00000000}, + {0x958c, 0x00000000}, + {0x9590, 0x00000000}, + {0x9594, 0x00000000}, + {0x9598, 0x00000000}, + {0x959c, 0x00000000}, + {0x95a0, 0x00000000}, + {0x95a4, 0x00000000}, + {0x95a8, 0x00000000}, + {0x95ac, 0x00000000}, + {0x95b0, 0x00000000}, + {0x95b4, 0x00000000}, + {0x95b8, 0x00000000}, + {0x95bc, 0x00000000}, + {0x95c0, 0x00000000}, + {0x95c4, 0x00000000}, + {0x95c8, 0x04000000}, + {0x95cc, 0x00000000}, + {0x95d0, 0x00000000}, + {0x95d4, 0x00000000}, + {0x95d8, 0x00000000}, + {0x95dc, 0x00000000}, + {0x95e0, 0x00000000}, + {0x95e4, 0x00000000}, + {0x95e8, 0x00000000}, + {0x95ec, 0x00000000}, + {0x95f0, 0x00000000}, + {0x95f4, 0x00000000}, + {0x95f8, 0x00000000}, + {0x95fc, 0x00000000}, + {0x9600, 0x00000000}, + {0x9604, 0x00000000}, + {0x9608, 0x00000000}, + {0x960c, 0x04000000}, + {0x9610, 0x00000000}, + {0x9614, 0x00000000}, + {0x9618, 0x00000000}, + {0x961c, 0x00000000}, + {0x9620, 0x00000000}, + {0x9624, 0x00000000}, + {0x9628, 0x00000000}, + {0x962c, 0x00000000}, + {0x9630, 0x00000000}, + {0x9634, 0x00000000}, + {0x9638, 0x00000000}, + {0x963c, 0x00000000}, + {0x9640, 0x00000000}, + {0x9644, 0x00000000}, + {0x9648, 0x00000000}, + {0x964c, 0x00000000}, + {0x9650, 0x04000000}, + {0x9654, 0x00000000}, + {0x9658, 0x00000000}, + {0x965c, 0x00000000}, + {0x9660, 0x00000000}, + {0x9664, 0x00000000}, + {0x9668, 0x00000000}, + {0x966c, 0x00000000}, + {0x9670, 0x00000000}, + {0x9674, 0x00000000}, + {0x9678, 0x00000000}, + {0x967c, 0x00000000}, + {0x9680, 0x00000000}, + {0x9684, 0x00000000}, + {0x9688, 0x00000000}, + {0x968c, 0x00000000}, + {0x9690, 0x00000000}, + {0x9694, 0x04000000}, + {0x9698, 0x00000000}, + {0x969c, 0x00000000}, + {0x96a0, 0x00000000}, + {0x96a4, 0x00000000}, + {0x96a8, 0x00000000}, + {0x96ac, 0x00000000}, + {0x96b0, 0x00000000}, + {0x96b4, 0x00000000}, + {0x96b8, 0x00000000}, + {0x96bc, 0x00000000}, + {0x96c0, 0x00000000}, + {0x96c4, 0x00000000}, + {0x96c8, 0x00000000}, + {0x96cc, 0x00000000}, + {0x96d0, 0x00000000}, + {0x96d4, 0x00000000}, + {0x96d8, 0x04000000}, + {0x96dc, 0x00000000}, + {0x96e0, 0x00000000}, + {0x96e4, 0x00000000}, + {0x96e8, 0x00000000}, + {0x96ec, 0x00000000}, + {0x96f0, 0x00000000}, + {0x96f4, 0x00000000}, + {0x96f8, 0x00000000}, + {0x96fc, 0x00000000}, + {0x9700, 0x00000000}, + {0x9704, 0x00000000}, + {0x9708, 0x00000000}, + {0x970c, 0x00000000}, + {0x9710, 0x00000000}, + {0x9714, 0x00000000}, + {0x9718, 0x00000000}, + {0x971c, 0x04000000}, + {0x9720, 0x00000000}, + {0x9724, 0x00000000}, + {0x9728, 0x00000000}, + {0x972c, 0x00000000}, + {0x9730, 0x00000000}, + {0x9734, 0x00000000}, + {0x9738, 0x00000000}, + {0x973c, 0x00000000}, + {0x9740, 0x00000000}, + {0x9744, 0x00000000}, + {0x9748, 0x00000000}, + {0x974c, 0x00000000}, + {0x9750, 0x00000000}, + {0x9754, 0x00000000}, + {0x9758, 0x00000000}, + {0x975c, 0x00000000}, + {0x9760, 0x04000000}, + {0x9764, 0x00000000}, + {0x9768, 0x00000000}, + {0x976c, 0x00000000}, + {0x9770, 0x00000000}, + {0x9774, 0x00000000}, + {0x9778, 0x00000000}, + {0x977c, 0x00000000}, + {0x9780, 0x00000000}, + {0x9784, 0x00000000}, + {0x9788, 0x00000000}, + {0x978c, 0x00000000}, + {0x9790, 0x00000000}, + {0x9794, 0x00000000}, + {0x9798, 0x00000000}, + {0x979c, 0x00000000}, + {0x97a0, 0x00000000}, + {0x97a4, 0x04000000}, + {0x97a8, 0x00000000}, + {0x97ac, 0x00000000}, + {0x97b0, 0x00000000}, + {0x97b4, 0x00000000}, + {0x97b8, 0x00000000}, + {0x97bc, 0x00000000}, + {0x97c0, 0x00000000}, + {0x97c4, 0x00000000}, + {0x97c8, 0x00000000}, + {0x97cc, 0x00000000}, + {0x97d0, 0x00000000}, + {0x97d4, 0x00000000}, + {0x97d8, 0x00000000}, + {0x97dc, 0x00000000}, + {0x97e0, 0x00000000}, + {0x97e4, 0x00000000}, + {0x97e8, 0x04000000}, + {0x97ec, 0x00000000}, + {0x97f0, 0x00000000}, + {0x97f4, 0x00000000}, + {0x97f8, 0x00000000}, + {0x97fc, 0x00000000}, + {0x9800, 0x00000000}, + {0x9804, 0x00000000}, + {0x9808, 0x00000000}, + {0x980c, 0x00000000}, + {0x9810, 0x00000000}, + {0x9814, 0x00000000}, + {0x9818, 0x00000000}, + {0x981c, 0x00000000}, + {0x9820, 0x00000000}, + {0x9824, 0x00000000}, + {0x9828, 0x00000000}, + {0x982c, 0x04000000}, + {0x81d8, 0x00000000}, + {0xb104, 0x2b251f19}, + {0xb108, 0x433d3731}, + {0xb10c, 0x5b554f49}, + {0xb110, 0x736d6761}, + {0xb114, 0x7f7f7f79}, + {0xb118, 0x120f7f7f}, + {0xb11c, 0x1e1b1815}, + {0xb120, 0x2a272421}, + {0xb124, 0x3633302d}, + {0xb128, 0x3f3f3c39}, + {0xb12c, 0x3f3f3f3f}, + {0x8088, 0x00000110}, + {0x8000, 0x00000008}, + {0x8080, 0x00000005}, + {0x8500, 0x80000008}, + {0x8504, 0x43000004}, + {0x8508, 0x4b044a00}, + {0x850c, 0x40098604}, + {0x8510, 0x0004e01f}, + {0x8514, 0x74104b00}, + {0x8518, 0x000021e0}, + {0x851c, 0x74301658}, + {0x8520, 0x43800004}, + {0x8524, 0x4c000007}, + {0x8528, 0x43000004}, + {0x852c, 0x56030007}, + {0x8530, 0x57000004}, + {0x8534, 0x400042fe}, + {0x8538, 0x50554200}, + {0x853c, 0xb4183000}, + {0x8540, 0xe537a50f}, + {0x8544, 0xf12bf02b}, + {0x8548, 0xf32bf22b}, + {0x854c, 0xf62bf42b}, + {0x8550, 0xf82bf72b}, + {0x8554, 0xfa2bf92b}, + {0x8558, 0xfd2bfc2b}, + {0x855c, 0xe537fe2b}, + {0x8560, 0xf12af02a}, + {0x8564, 0xf32af22a}, + {0x8568, 0xf52af42a}, + {0x856c, 0x000bf62a}, + {0x8570, 0xf028a511}, + {0x8574, 0xf228f128}, + {0x8578, 0xf428f328}, + {0x857c, 0xf628f528}, + {0x8580, 0xf828f728}, + {0x8584, 0xfa28f928}, + {0x8588, 0xfc28fb28}, + {0x858c, 0xfe28fd28}, + {0x8590, 0xf028ff28}, + {0x8594, 0xf228f128}, + {0x8598, 0x30750001}, + {0x859c, 0x30753075}, + {0x85a0, 0x30b63097}, + {0x85a4, 0x30be30bb}, + {0x85a8, 0x30d930cc}, + {0x85ac, 0x316d30e6}, + {0x85b0, 0x3189317f}, + {0x85b4, 0x31d23193}, + {0x85b8, 0x31e43210}, + {0x85bc, 0x31e831dd}, + {0x85c0, 0x322831e1}, + {0x85c4, 0x323c3232}, + {0x85c8, 0x32503246}, + {0x85cc, 0x3264325a}, + {0x85d0, 0x3278326e}, + {0x85d4, 0x32983285}, + {0x85d8, 0x32aa32a6}, + {0x85dc, 0x330b32f3}, + {0x85e0, 0x333f330c}, + {0x85e4, 0x334c3341}, + {0x85e8, 0xe35e0001}, + {0x85ec, 0x20887410}, + {0x85f0, 0x140f0200}, + {0x85f4, 0x02002098}, + {0x85f8, 0x7430140f}, + {0x85fc, 0x5b10e39c}, + {0x8600, 0x20807410}, + {0x8604, 0x140f0000}, + {0x8608, 0x56015507}, + {0x860c, 0x7410e382}, + {0x8610, 0x02002088}, + {0x8614, 0x5517140f}, + {0x8618, 0xe34ee382}, + {0x861c, 0x468e7508}, + {0x8620, 0xe0ace38c}, + {0x8624, 0x5500f0e2}, + {0x8628, 0x5501e37e}, + {0x862c, 0x5b10f1de}, + {0x8630, 0x20907410}, + {0x8634, 0x140f0000}, + {0x8638, 0xe3825507}, + {0x863c, 0x20987410}, + {0x8640, 0x140f0200}, + {0x8644, 0xe3825517}, + {0x8648, 0x46967509}, + {0x864c, 0xe0ace38c}, + {0x8650, 0xe37e5500}, + {0x8654, 0x00015501}, + {0x8658, 0x4d000007}, + {0x865c, 0x74200004}, + {0x8660, 0x57005710}, + {0x8664, 0x9700140f}, + {0x8668, 0x00017430}, + {0x866c, 0xe39ce35e}, + {0x8670, 0xe52a0bbd}, + {0x8674, 0xe36a0001}, + {0x8678, 0x0001e3c4}, + {0x867c, 0x55005b30}, + {0x8680, 0x46500005}, + {0x8684, 0x74000004}, + {0x8688, 0x1658e37e}, + {0x868c, 0x74305501}, + {0x8690, 0x46100005}, + {0x8694, 0x00010004}, + {0x8698, 0x30f8e35e}, + {0x869c, 0xe52a0023}, + {0x86a0, 0x54ed0002}, + {0x86a4, 0x00230baa}, + {0x86a8, 0x0002e52a}, + {0x86ac, 0xe356e3e4}, + {0x86b0, 0xe35e0001}, + {0x86b4, 0x002230f3}, + {0x86b8, 0x0002e52a}, + {0x86bc, 0x0baa54ec}, + {0x86c0, 0xe52a0022}, + {0x86c4, 0xe3e40002}, + {0x86c8, 0x0001e356}, + {0x86cc, 0x0baae35e}, + {0x86d0, 0xe3e430ec}, + {0x86d4, 0x0001e356}, + {0x86d8, 0x6d0f6c67}, + {0x86dc, 0xe52ae39c}, + {0x86e0, 0xe39c6c8b}, + {0x86e4, 0x0bace52a}, + {0x86e8, 0x6d0f6cb3}, + {0x86ec, 0xe52ae39c}, + {0x86f0, 0x6cdb0bad}, + {0x86f4, 0xe39c6d0f}, + {0x86f8, 0x6cf5e52a}, + {0x86fc, 0xe39c6d0f}, + {0x8700, 0x6c0be52a}, + {0x8704, 0xe39c6d00}, + {0x8708, 0x6c25e52a}, + {0x870c, 0xe52ae39c}, + {0x8710, 0x6c4df8c6}, + {0x8714, 0xe52ae39c}, + {0x8718, 0x6c75f9cf}, + {0x871c, 0xe52ae39c}, + {0x8720, 0xe39c6c99}, + {0x8724, 0xfad6e52a}, + {0x8728, 0x21e87410}, + {0x872c, 0x6e670009}, + {0x8730, 0xe3c46f0f}, + {0x8734, 0x7410e52f}, + {0x8738, 0x000b21e8}, + {0x873c, 0xe3c46e8b}, + {0x8740, 0x7410e52f}, + {0x8744, 0x000d21e8}, + {0x8748, 0x6f0f6eb3}, + {0x874c, 0xe52fe3c4}, + {0x8750, 0xfe07ff08}, + {0x8754, 0x21e87410}, + {0x8758, 0x6ec7000e}, + {0x875c, 0xe52fe3c4}, + {0x8760, 0x21e87410}, + {0x8764, 0x6edb000f}, + {0x8768, 0xe3c46f0f}, + {0x876c, 0x7410e52f}, + {0x8770, 0x001021e8}, + {0x8774, 0xe3c46eef}, + {0x8778, 0xff03e52f}, + {0x877c, 0xe52ffe02}, + {0x8780, 0x21e87410}, + {0x8784, 0x6e110013}, + {0x8788, 0xe3c46f00}, + {0x878c, 0xff03e52f}, + {0x8790, 0xe52ffe02}, + {0x8794, 0x21e87410}, + {0x8798, 0x6e250014}, + {0x879c, 0xe52fe3c4}, + {0x87a0, 0xff08fc24}, + {0x87a4, 0x7410fe07}, + {0x87a8, 0x001521e8}, + {0x87ac, 0xe3c46e39}, + {0x87b0, 0x7410e52f}, + {0x87b4, 0x001621e8}, + {0x87b8, 0xe3c46e4d}, + {0x87bc, 0xfd27e52f}, + {0x87c0, 0x21e87410}, + {0x87c4, 0x6e750018}, + {0x87c8, 0xe52fe3c4}, + {0x87cc, 0x21e87410}, + {0x87d0, 0x6e99001a}, + {0x87d4, 0xe52fe3c4}, + {0x87d8, 0xe36afe24}, + {0x87dc, 0x63404380}, + {0x87e0, 0x43006880}, + {0x87e4, 0x31300bac}, + {0x87e8, 0xe52f0022}, + {0x87ec, 0x54ec0002}, + {0x87f0, 0x00220baa}, + {0x87f4, 0x0002e52f}, + {0x87f8, 0xe362e3e4}, + {0x87fc, 0xe36a0001}, + {0x8800, 0x63404380}, + {0x8804, 0x43006881}, + {0x8808, 0x31210baa}, + {0x880c, 0xe362e3e4}, + {0x8810, 0xe36a0001}, + {0x8814, 0x63414380}, + {0x8818, 0x43006882}, + {0x881c, 0x31140baa}, + {0x8820, 0xe362e3e4}, + {0x8824, 0x00040001}, + {0x8828, 0x000742fc}, + {0x882c, 0x00046001}, + {0x8830, 0x00074200}, + {0x8834, 0x62006220}, + {0x8838, 0x55010004}, + {0x883c, 0x66055b40}, + {0x8840, 0x62000007}, + {0x8844, 0xe40e6300}, + {0x8848, 0x09000004}, + {0x884c, 0x0b400a01}, + {0x8850, 0x0e010d00}, + {0x8854, 0x00040032}, + {0x8858, 0x42fb950b}, + {0x885c, 0x4d040007}, + {0x8860, 0x42000004}, + {0x8864, 0x00074380}, + {0x8868, 0x00044d01}, + {0x886c, 0x00074300}, + {0x8870, 0x05a30562}, + {0x8874, 0xe40e961f}, + {0x8878, 0xe37e0004}, + {0x887c, 0x06a20007}, + {0x8880, 0xe40e07a3}, + {0x8884, 0xe37e0004}, + {0x8888, 0x0002e3fe}, + {0x888c, 0x4380e406}, + {0x8890, 0x4d000007}, + {0x8894, 0x43000004}, + {0x8898, 0x000742fe}, + {0x889c, 0x00044d00}, + {0x88a0, 0x00014200}, + {0x88a4, 0x42fc0004}, + {0x88a8, 0x60030007}, + {0x88ac, 0x42000004}, + {0x88b0, 0x00073199}, + {0x88b4, 0x07a306a2}, + {0x88b8, 0xe1eb31c5}, + {0x88bc, 0xe1fee1f9}, + {0x88c0, 0xe1eb0001}, + {0x88c4, 0x0001e1fe}, + {0x88c8, 0xe1f9e1f2}, + {0x88cc, 0x0001e1fe}, + {0x88d0, 0xe1fee1f2}, + {0x88d4, 0x00040001}, + {0x88d8, 0x000742fc}, + {0x88dc, 0x00046003}, + {0x88e0, 0x00014200}, + {0x88e4, 0x42fc0004}, + {0x88e8, 0x60010007}, + {0x88ec, 0x42000004}, + {0x88f0, 0x00070001}, + {0x88f4, 0x62006220}, + {0x88f8, 0x0001e406}, + {0x88fc, 0x63000007}, + {0x8900, 0x09000004}, + {0x8904, 0x0e010a00}, + {0x8908, 0x00070032}, + {0x890c, 0xe40e06a2}, + {0x8910, 0x0002e41a}, + {0x8914, 0x000742fe}, + {0x8918, 0x00044d00}, + {0x891c, 0x00014200}, + {0x8920, 0x77000005}, + {0x8924, 0x52000007}, + {0x8928, 0x42fe0004}, + {0x892c, 0x60000007}, + {0x8930, 0x42000004}, + {0x8934, 0x60004380}, + {0x8938, 0x62016100}, + {0x893c, 0x68046310}, + {0x8940, 0x41000005}, + {0x8944, 0x00075500}, + {0x8948, 0x00045c02}, + {0x894c, 0x00014300}, + {0x8950, 0x6c060005}, + {0x8954, 0xe2aae298}, + {0x8958, 0xe42ae285}, + {0x895c, 0xe432e2f3}, + {0x8960, 0x0001e30c}, + {0x8964, 0x0005e285}, + {0x8968, 0xe2986c06}, + {0x896c, 0xe42ae4a9}, + {0x8970, 0xe432e2f3}, + {0x8974, 0x0001e30c}, + {0x8978, 0x6c000005}, + {0x897c, 0xe2aae298}, + {0x8980, 0xe445e285}, + {0x8984, 0xe44de2f3}, + {0x8988, 0x0001e30c}, + {0x898c, 0x0005e285}, + {0x8990, 0xe2986c00}, + {0x8994, 0xe445e4a9}, + {0x8998, 0xe44de2f3}, + {0x899c, 0x0001e30c}, + {0x89a0, 0x6c040005}, + {0x89a4, 0xe2aae298}, + {0x89a8, 0xe460e285}, + {0x89ac, 0xe468e2f3}, + {0x89b0, 0x0001e30c}, + {0x89b4, 0x0005e285}, + {0x89b8, 0xe2986c04}, + {0x89bc, 0xe460e4a9}, + {0x89c0, 0xe468e2f3}, + {0x89c4, 0x0001e30c}, + {0x89c8, 0x6c020005}, + {0x89cc, 0xe2aae298}, + {0x89d0, 0xe47be285}, + {0x89d4, 0xe483e2f3}, + {0x89d8, 0x0001e30c}, + {0x89dc, 0x0005e285}, + {0x89e0, 0xe2986c02}, + {0x89e4, 0xe47be4a9}, + {0x89e8, 0xe483e2f3}, + {0x89ec, 0x0001e30c}, + {0x89f0, 0x43800004}, + {0x89f4, 0x610a6008}, + {0x89f8, 0x63ce6200}, + {0x89fc, 0x60800006}, + {0x8a00, 0x00047f00}, + {0x8a04, 0xe4e04300}, + {0x8a08, 0x00070001}, + {0x8a0c, 0x4d015500}, + {0x8a10, 0x74200004}, + {0x8a14, 0x57107711}, + {0x8a18, 0x140f5700}, + {0x8a1c, 0x00077430}, + {0x8a20, 0x00044d00}, + {0x8a24, 0x00074380}, + {0x8a28, 0x00047200}, + {0x8a2c, 0x00014300}, + {0x8a30, 0x74200004}, + {0x8a34, 0x77000005}, + {0x8a38, 0x73887e07}, + {0x8a3c, 0x8f007380}, + {0x8a40, 0x0004140f}, + {0x8a44, 0x00057430}, + {0x8a48, 0x00017300}, + {0x8a4c, 0x0005e496}, + {0x8a50, 0x00017300}, + {0x8a54, 0x43800004}, + {0x8a58, 0x0006b103}, + {0x8a5c, 0x91037cdb}, + {0x8a60, 0x40db0007}, + {0x8a64, 0x43000004}, + {0x8a68, 0x0005e496}, + {0x8a6c, 0x00067380}, + {0x8a70, 0x60025d01}, + {0x8a74, 0xe4ba6200}, + {0x8a78, 0x73000005}, + {0x8a7c, 0x76080007}, + {0x8a80, 0x00047578}, + {0x8a84, 0x00074380}, + {0x8a88, 0x5e005e01}, + {0x8a8c, 0x0006140a}, + {0x8a90, 0x7f006380}, + {0x8a94, 0x00076080}, + {0x8a98, 0x4e204c3f}, + {0x8a9c, 0x73047280}, + {0x8aa0, 0x140a7300}, + {0x8aa4, 0x00044d20}, + {0x8aa8, 0x00064300}, + {0x8aac, 0x00077402}, + {0x8ab0, 0x40004001}, + {0x8ab4, 0x0006ab00}, + {0x8ab8, 0x00077404}, + {0x8abc, 0x40004001}, + {0x8ac0, 0x140aab00}, + {0x8ac4, 0x43800004}, + {0x8ac8, 0x52800007}, + {0x8acc, 0x140a5200}, + {0x8ad0, 0x4d004c00}, + {0x8ad4, 0x00064e00}, + {0x8ad8, 0x63006080}, + {0x8adc, 0x43000004}, + {0x8ae0, 0x76000007}, + {0x8ae4, 0x00040001}, + {0x8ae8, 0xb1034380}, + {0x8aec, 0x7cdb0006}, + {0x8af0, 0x00079103}, + {0x8af4, 0x000440db}, + {0x8af8, 0xe4964300}, + {0x8afc, 0xe4ba7e03}, + {0x8b00, 0x43800004}, + {0x8b04, 0x0006b103}, + {0x8b08, 0x91037c5b}, + {0x8b0c, 0x405b0007}, + {0x8b10, 0x43000004}, + {0x8b14, 0x00010001}, + {0x8b18, 0x43800004}, + {0x8b1c, 0x4e200007}, + {0x8b20, 0x63800006}, + {0x8b24, 0x5f807cdb}, + {0x8b28, 0x43000004}, + {0x8b2c, 0x76080007}, + {0x8b30, 0x00057560}, + {0x8b34, 0x00047380}, + {0x8b38, 0x0005420e}, + {0x8b3c, 0x14c86c01}, + {0x8b40, 0x6c001432}, + {0x8b44, 0x42000004}, + {0x8b48, 0x43800004}, + {0x8b4c, 0x5f000006}, + {0x8b50, 0x73010007}, + {0x8b54, 0x00047300}, + {0x8b58, 0x0007420f}, + {0x8b5c, 0x52005280}, + {0x8b60, 0x0004140a}, + {0x8b64, 0x00064200}, + {0x8b68, 0x7c5b6300}, + {0x8b6c, 0x4e000007}, + {0x8b70, 0x43000004}, + {0x8b74, 0x73000005}, + {0x8b78, 0x76000007}, + {0x8b7c, 0xe4c30001}, + {0x8b80, 0x00040001}, + {0x8b84, 0x60004380}, + {0x8b88, 0x62016100}, + {0x8b8c, 0x00066310}, + {0x8b90, 0x00046000}, + {0x8b94, 0x00014300}, + {0x8b98, 0x0001e4e0}, + {0x8b9c, 0x4e004f02}, + {0x8ba0, 0x52015302}, + {0x8ba4, 0x140f0001}, + {0x8ba8, 0x00019700}, + {0x8bac, 0x65014380}, + {0x8bb0, 0x79007800}, + {0x8bb4, 0x7b407a00}, + {0x8bb8, 0x00014300}, + {0x8bbc, 0x65004380}, + {0x8bc0, 0x00014300}, + {0x8bc4, 0x64014380}, + {0x8bc8, 0x7d007c00}, + {0x8bcc, 0x7f407e00}, + {0x8bd0, 0x00014300}, + {0x8bd4, 0x64004380}, + {0x8bd8, 0x00014300}, + {0x8bdc, 0x7b004380}, + {0x8be0, 0x79007a04}, + {0x8be4, 0x43007802}, + {0x8be8, 0x33825509}, + {0x8bec, 0x43800001}, + {0x8bf0, 0x7a007b40}, + {0x8bf4, 0x55194300}, + {0x8bf8, 0x00013382}, + {0x8bfc, 0x74007401}, + {0x8c00, 0x00018e00}, + {0x8c04, 0x52300007}, + {0x8c08, 0x74310004}, + {0x8c0c, 0x8e007430}, + {0x8c10, 0x52200007}, + {0x8c14, 0x00010004}, + {0x8c18, 0x57005702}, + {0x8c1c, 0x00018e00}, + {0x8c20, 0x57425740}, + {0x8c24, 0x8e005740}, + {0x8c28, 0x00015700}, + {0x8c2c, 0x561042ef}, + {0x8c30, 0x42005600}, + {0x8c34, 0x00018c00}, + {0x8c38, 0xe3a75b20}, + {0x8c3c, 0x54005480}, + {0x8c40, 0x54005481}, + {0x8c44, 0x54005482}, + {0x8c48, 0xbf1ae3ac}, + {0x8c4c, 0xe36e300b}, + {0x8c50, 0xe390e377}, + {0x8c54, 0x0001e523}, + {0x8c58, 0x54c054bf}, + {0x8c5c, 0x54c154a3}, + {0x8c60, 0x4c1854a4}, + {0x8c64, 0xbf091402}, + {0x8c68, 0x54a454c2}, + {0x8c6c, 0xbf051402}, + {0x8c70, 0x54a354c1}, + {0x8c74, 0xbf011402}, + {0x8c78, 0x54dfe534}, + {0x8c7c, 0x54bf0001}, + {0x8c80, 0x050a54e5}, + {0x8c84, 0x000154df}, + {0x8c88, 0x00071657}, + {0x8c8c, 0x00044c80}, + {0x8c90, 0x43807430}, + {0x8c94, 0x7e007f40}, + {0x8c98, 0x7c027d00}, + {0x8c9c, 0x5b404300}, + {0x8ca0, 0x5c015501}, + {0x8ca4, 0x5480e396}, + {0x8ca8, 0x54815400}, + {0x8cac, 0x54825400}, + {0x8cb0, 0x00075400}, + {0x8cb4, 0x00044c00}, + {0x8cb8, 0xe3ac7410}, + {0x8cbc, 0x300bbfe1}, + {0x8cc0, 0x56005610}, + {0x8cc4, 0x00018c00}, + {0x8cc8, 0x57005704}, + {0x8ccc, 0xa7038e00}, + {0x8cd0, 0x33f0aff7}, + {0x8cd4, 0xaf034019}, + {0x8cd8, 0x33f0402b}, + {0x8cdc, 0x33df402b}, + {0x8ce0, 0x57005708}, + {0x8ce4, 0x57818e00}, + {0x8ce8, 0x8e005780}, + {0x8cec, 0x00074380}, + {0x8cf0, 0x5c005c01}, + {0x8cf4, 0x00041403}, + {0x8cf8, 0x00014300}, + {0x8cfc, 0x0007427f}, + {0x8d00, 0x62006280}, + {0x8d04, 0x00049200}, + {0x8d08, 0x00014200}, + {0x8d0c, 0x0007427f}, + {0x8d10, 0x63146394}, + {0x8d14, 0x00049200}, + {0x8d18, 0x00014200}, + {0x8d1c, 0x42fe0004}, + {0x8d20, 0x4d010007}, + {0x8d24, 0x42000004}, + {0x8d28, 0x140f7420}, + {0x8d2c, 0x57005710}, + {0x8d30, 0x0001141f}, + {0x8d34, 0x42fe0004}, + {0x8d38, 0x4d010007}, + {0x8d3c, 0x42000004}, + {0x8d40, 0x140f7420}, + {0x8d44, 0x000742bf}, + {0x8d48, 0x62006240}, + {0x8d4c, 0x0004141f}, + {0x8d50, 0x00014200}, + {0x8d54, 0x5d060006}, + {0x8d58, 0x61046003}, + {0x8d5c, 0x00056201}, + {0x8d60, 0x00017310}, + {0x8d64, 0x43800004}, + {0x8d68, 0x5e010007}, + {0x8d6c, 0x140a5e00}, + {0x8d70, 0x0006b103}, + {0x8d74, 0x91037f07}, + {0x8d78, 0x43070007}, + {0x8d7c, 0x5c000006}, + {0x8d80, 0x5e035d02}, + {0x8d84, 0x43000004}, + {0x8d88, 0x00060001}, + {0x8d8c, 0x60005d04}, + {0x8d90, 0x62016104}, + {0x8d94, 0x73100005}, + {0x8d98, 0x00040001}, + {0x8d9c, 0x00074380}, + {0x8da0, 0x5e005e01}, + {0x8da4, 0xb103140a}, + {0x8da8, 0x7fc60006}, + {0x8dac, 0x00079103}, + {0x8db0, 0x000643c6}, + {0x8db4, 0x5d025c00}, + {0x8db8, 0x00045e03}, + {0x8dbc, 0x00014300}, + {0x8dc0, 0x5d040006}, + {0x8dc4, 0x61046000}, + {0x8dc8, 0x00056201}, + {0x8dcc, 0x00017310}, + {0x8dd0, 0x43800004}, + {0x8dd4, 0x5e010007}, + {0x8dd8, 0x140a5e00}, + {0x8ddc, 0x0006b103}, + {0x8de0, 0x91037fc6}, + {0x8de4, 0x43c60007}, + {0x8de8, 0x5c000006}, + {0x8dec, 0x5e035d02}, + {0x8df0, 0x43000004}, + {0x8df4, 0x00060001}, + {0x8df8, 0x60025d00}, + {0x8dfc, 0x62016100}, + {0x8e00, 0x73000005}, + {0x8e04, 0x00040001}, + {0x8e08, 0x00074380}, + {0x8e0c, 0x5e005e01}, + {0x8e10, 0xb103140a}, + {0x8e14, 0x7fc00006}, + {0x8e18, 0x00079103}, + {0x8e1c, 0x000643c0}, + {0x8e20, 0x5d025c00}, + {0x8e24, 0x00045e03}, + {0x8e28, 0x00014300}, + {0x8e2c, 0x7e020005}, + {0x8e30, 0x42f70004}, + {0x8e34, 0x6c080005}, + {0x8e38, 0x42700004}, + {0x8e3c, 0x73810005}, + {0x8e40, 0x93007380}, + {0x8e44, 0x42f70004}, + {0x8e48, 0x6c000005}, + {0x8e4c, 0x42000004}, + {0x8e50, 0x00040001}, + {0x8e54, 0x00074380}, + {0x8e58, 0x73007304}, + {0x8e5c, 0x72401405}, + {0x8e60, 0x43000004}, + {0x8e64, 0x74040006}, + {0x8e68, 0x40010007}, + {0x8e6c, 0xab004000}, + {0x8e70, 0x0001140f}, + {0x8e74, 0x140ae517}, + {0x8e78, 0x140ae4c3}, + {0x8e7c, 0x0001e51e}, + {0x8e80, 0xe4c3e517}, + {0x8e84, 0x00040001}, + {0x8e88, 0x00047410}, + {0x8e8c, 0x42f04380}, + {0x8e90, 0x62080007}, + {0x8e94, 0x24206301}, + {0x8e98, 0x14c80000}, + {0x8e9c, 0x00002428}, + {0x8ea0, 0x1a4215f4}, + {0x8ea4, 0x6300000b}, + {0x8ea8, 0x42000004}, + {0x8eac, 0x74304300}, + {0x8eb0, 0x4380140f}, + {0x8eb4, 0x73080007}, + {0x8eb8, 0x00047300}, + {0x8ebc, 0x00014300}, + {0x8ec0, 0x4bf00007}, + {0x8ec4, 0x490b4a8f}, + {0x8ec8, 0x4a8e48f1}, + {0x8ecc, 0x48a5490a}, + {0x8ed0, 0x49094a8d}, + {0x8ed4, 0x4a8c487d}, + {0x8ed8, 0x48754908}, + {0x8edc, 0x49074a8b}, + {0x8ee0, 0x4a8a4889}, + {0x8ee4, 0x48b74906}, + {0x8ee8, 0x49054a89}, + {0x8eec, 0x4a8848fc}, + {0x8ef0, 0x48564905}, + {0x8ef4, 0x49044a87}, + {0x8ef8, 0x4a8648c1}, + {0x8efc, 0x483d4904}, + {0x8f00, 0x49034a85}, + {0x8f04, 0x4a8448c7}, + {0x8f08, 0x485e4903}, + {0x8f0c, 0x49024a83}, + {0x8f10, 0x4a8248ac}, + {0x8f14, 0x48624902}, + {0x8f18, 0x49024a81}, + {0x8f1c, 0x4a804820}, + {0x8f20, 0x48004900}, + {0x8f24, 0x49014a90}, + {0x8f28, 0x4a10481f}, + {0x8f2c, 0x00060001}, + {0x8f30, 0x5f005f80}, + {0x8f34, 0x00059900}, + {0x8f38, 0x00017300}, + {0x8f3c, 0x63800006}, + {0x8f40, 0x98006300}, + {0x8f44, 0x549f0001}, + {0x8f48, 0x5c015400}, + {0x8f4c, 0x540054df}, + {0x8f50, 0x00015c02}, + {0x8f54, 0x07145c01}, + {0x8f58, 0x5c025400}, + {0x8f5c, 0x5c020001}, + {0x8f60, 0x54000714}, + {0x8f64, 0x00015c01}, + {0x8f68, 0x4c184c98}, + {0x8f6c, 0x00080001}, + {0x8f70, 0x5c020004}, + {0x8f74, 0x09017430}, + {0x8f78, 0x0ba60c01}, + {0x8f7c, 0x77800005}, + {0x8f80, 0x52200007}, + {0x8f84, 0x43800004}, + {0x8f88, 0x610a6008}, + {0x8f8c, 0x63c26200}, + {0x8f90, 0x5c000007}, + {0x8f94, 0x43000004}, + {0x8f98, 0x00000001}, + {0x8080, 0x00000004}, + {0x8080, 0x00000000}, + {0x8088, 0x00000000}, +}; + +static const struct rtw89_txpwr_byrate_cfg rtw89_8851b_txpwr_byrate[] = { + { 0, 0, 0, 0, 4, 0x50505050, }, + { 0, 0, 1, 0, 4, 0x54585858, }, + { 0, 0, 1, 4, 4, 0x44484c50, }, + { 0, 0, 2, 0, 4, 0x50545858, }, + { 0, 0, 2, 4, 4, 0x4044484c, }, + { 0, 0, 2, 8, 4, 0x3034383c, }, + { 0, 0, 3, 0, 4, 0x50505050, }, + { 0, 1, 2, 0, 4, 0x50545858, }, + { 0, 1, 2, 4, 4, 0x4044484c, }, + { 0, 1, 2, 8, 4, 0x3034383c, }, + { 0, 1, 3, 0, 4, 0x50505050, }, + { 0, 0, 4, 1, 4, 0x00000000, }, + { 0, 0, 4, 0, 1, 0x00000000, }, + { 1, 0, 1, 0, 4, 0x58585858, }, + { 1, 0, 1, 4, 4, 0x484c5054, }, + { 1, 0, 2, 0, 4, 0x54585858, }, + { 1, 0, 2, 4, 4, 0x44484c50, }, + { 1, 0, 2, 8, 4, 0x34383c40, }, + { 1, 0, 3, 0, 4, 0x40404040, }, + { 1, 1, 2, 0, 4, 0x54585858, }, + { 1, 1, 2, 4, 4, 0x44484c50, }, + { 1, 1, 2, 8, 4, 0x34383c40, }, + { 1, 1, 3, 0, 4, 0x48484848, }, + { 1, 0, 4, 0, 4, 0x00000000, }, + { 2, 0, 1, 0, 4, 0x40404040, }, + { 2, 0, 1, 4, 4, 0x383c4040, }, + { 2, 0, 2, 0, 4, 0x40404040, }, + { 2, 0, 2, 4, 4, 0x34383c40, }, + { 2, 0, 2, 8, 4, 0x24282c30, }, + { 2, 0, 3, 0, 4, 0x40404040, }, + { 2, 1, 2, 0, 4, 0x40404040, }, + { 2, 1, 2, 4, 4, 0x34383c40, }, + { 2, 1, 2, 8, 4, 0x24282c30, }, + { 2, 1, 3, 0, 4, 0x40404040, }, + { 2, 0, 4, 0, 4, 0x00000000, }, +}; + +static const s8 _txpwr_track_delta_swingidx_5ga_n[][DELTA_SWINGIDX_SIZE] = { + {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, + {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1}, + {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, + 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4}, +}; + +static const s8 _txpwr_track_delta_swingidx_5ga_p[][DELTA_SWINGIDX_SIZE] = { + {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, + {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, + {0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, + 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2}, +}; + +static const s8 _txpwr_track_delta_swingidx_2ga_n[] = { + 0, 0, 0, 0, -1, -1, -1, -2, -2, -2, -2, -3, -3, -3, -3, -3, + -4, -4, -4, -4, -4, -5, -5, -5, -5, -5, -5, -6, -6, -6 +}; + +static const s8 _txpwr_track_delta_swingidx_2ga_p[] = { + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 1, 1, 1, 1, 2, 2, 2, 3, 3, 3, 3, 4, 4 +}; + +static const s8 _txpwr_track_delta_swingidx_2g_cck_a_n[] = { + 0, 0, 0, 0, -1, -1, -1, -2, -2, -2, -2, -3, -3, -3, -3, -3, + -4, -4, -4, -4, -4, -5, -5, -5, -5, -5, -5, -6, -6, -6 +}; + +static const s8 _txpwr_track_delta_swingidx_2g_cck_a_p[] = { + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 1, 1, 1, 1, 2, 2, 2, 3, 3, 3, 3, 4, 4 +}; + +const u8 rtw89_8851b_tx_shape[RTW89_BAND_MAX][RTW89_RS_TX_SHAPE_NUM] + [RTW89_REGD_NUM] = { + [0][0][RTW89_ACMA] = 0, + [0][0][RTW89_CN] = 0, + [0][0][RTW89_ETSI] = 0, + [0][0][RTW89_FCC] = 1, + [0][0][RTW89_IC] = 1, + [0][0][RTW89_KCC] = 0, + [0][0][RTW89_MKK] = 0, + [0][0][RTW89_UK] = 0, + [0][1][RTW89_ACMA] = 0, + [0][1][RTW89_CN] = 0, + [0][1][RTW89_ETSI] = 0, + [0][1][RTW89_FCC] = 3, + [0][1][RTW89_IC] = 3, + [0][1][RTW89_KCC] = 0, + [0][1][RTW89_MKK] = 0, + [0][1][RTW89_UK] = 0, + [1][1][RTW89_ACMA] = 0, + [1][1][RTW89_CN] = 0, + [1][1][RTW89_ETSI] = 0, + [1][1][RTW89_FCC] = 3, + [1][1][RTW89_IC] = 3, + [1][1][RTW89_KCC] = 0, + [1][1][RTW89_MKK] = 0, + [1][1][RTW89_UK] = 0, +}; + +static +const s8 rtw89_8851b_txpwr_lmt_2g[RTW89_2G_BW_NUM][RTW89_NTX_NUM] + [RTW89_RS_LMT_NUM][RTW89_BF_NUM] + [RTW89_REGD_NUM][RTW89_2G_CH_NUM] = { + [0][0][0][0][RTW89_WW][0] = 58, + [0][0][0][0][RTW89_WW][1] = 58, + [0][0][0][0][RTW89_WW][2] = 58, + [0][0][0][0][RTW89_WW][3] = 58, + [0][0][0][0][RTW89_WW][4] = 58, + [0][0][0][0][RTW89_WW][5] = 58, + [0][0][0][0][RTW89_WW][6] = 58, + [0][0][0][0][RTW89_WW][7] = 58, + [0][0][0][0][RTW89_WW][8] = 58, + [0][0][0][0][RTW89_WW][9] = 58, + [0][0][0][0][RTW89_WW][10] = 58, + [0][0][0][0][RTW89_WW][11] = 58, + [0][0][0][0][RTW89_WW][12] = 52, + [0][0][0][0][RTW89_WW][13] = 76, + [0][1][0][0][RTW89_WW][0] = 0, + [0][1][0][0][RTW89_WW][1] = 0, + [0][1][0][0][RTW89_WW][2] = 0, + [0][1][0][0][RTW89_WW][3] = 0, + [0][1][0][0][RTW89_WW][4] = 0, + [0][1][0][0][RTW89_WW][5] = 0, + [0][1][0][0][RTW89_WW][6] = 0, + [0][1][0][0][RTW89_WW][7] = 0, + [0][1][0][0][RTW89_WW][8] = 0, + [0][1][0][0][RTW89_WW][9] = 0, + [0][1][0][0][RTW89_WW][10] = 0, + [0][1][0][0][RTW89_WW][11] = 0, + [0][1][0][0][RTW89_WW][12] = 0, + [0][1][0][0][RTW89_WW][13] = 0, + [1][0][0][0][RTW89_WW][0] = 0, + [1][0][0][0][RTW89_WW][1] = 0, + [1][0][0][0][RTW89_WW][2] = 58, + [1][0][0][0][RTW89_WW][3] = 58, + [1][0][0][0][RTW89_WW][4] = 58, + [1][0][0][0][RTW89_WW][5] = 58, + [1][0][0][0][RTW89_WW][6] = 58, + [1][0][0][0][RTW89_WW][7] = 58, + [1][0][0][0][RTW89_WW][8] = 58, + [1][0][0][0][RTW89_WW][9] = 58, + [1][0][0][0][RTW89_WW][10] = 58, + [1][0][0][0][RTW89_WW][11] = 0, + [1][0][0][0][RTW89_WW][12] = 0, + [1][0][0][0][RTW89_WW][13] = 0, + [1][1][0][0][RTW89_WW][0] = 0, + [1][1][0][0][RTW89_WW][1] = 0, + [1][1][0][0][RTW89_WW][2] = 0, + [1][1][0][0][RTW89_WW][3] = 0, + [1][1][0][0][RTW89_WW][4] = 0, + [1][1][0][0][RTW89_WW][5] = 0, + [1][1][0][0][RTW89_WW][6] = 0, + [1][1][0][0][RTW89_WW][7] = 0, + [1][1][0][0][RTW89_WW][8] = 0, + [1][1][0][0][RTW89_WW][9] = 0, + [1][1][0][0][RTW89_WW][10] = 0, + [1][1][0][0][RTW89_WW][11] = 0, + [1][1][0][0][RTW89_WW][12] = 0, + [1][1][0][0][RTW89_WW][13] = 0, + [0][0][1][0][RTW89_WW][0] = 58, + [0][0][1][0][RTW89_WW][1] = 60, + [0][0][1][0][RTW89_WW][2] = 60, + [0][0][1][0][RTW89_WW][3] = 60, + [0][0][1][0][RTW89_WW][4] = 60, + [0][0][1][0][RTW89_WW][5] = 60, + [0][0][1][0][RTW89_WW][6] = 60, + [0][0][1][0][RTW89_WW][7] = 60, + [0][0][1][0][RTW89_WW][8] = 60, + [0][0][1][0][RTW89_WW][9] = 60, + [0][0][1][0][RTW89_WW][10] = 60, + [0][0][1][0][RTW89_WW][11] = 60, + [0][0][1][0][RTW89_WW][12] = 58, + [0][0][1][0][RTW89_WW][13] = 0, + [0][1][1][0][RTW89_WW][0] = 0, + [0][1][1][0][RTW89_WW][1] = 0, + [0][1][1][0][RTW89_WW][2] = 0, + [0][1][1][0][RTW89_WW][3] = 0, + [0][1][1][0][RTW89_WW][4] = 0, + [0][1][1][0][RTW89_WW][5] = 0, + [0][1][1][0][RTW89_WW][6] = 0, + [0][1][1][0][RTW89_WW][7] = 0, + [0][1][1][0][RTW89_WW][8] = 0, + [0][1][1][0][RTW89_WW][9] = 0, + [0][1][1][0][RTW89_WW][10] = 0, + [0][1][1][0][RTW89_WW][11] = 0, + [0][1][1][0][RTW89_WW][12] = 0, + [0][1][1][0][RTW89_WW][13] = 0, + [0][0][2][0][RTW89_WW][0] = 60, + [0][0][2][0][RTW89_WW][1] = 60, + [0][0][2][0][RTW89_WW][2] = 60, + [0][0][2][0][RTW89_WW][3] = 60, + [0][0][2][0][RTW89_WW][4] = 60, + [0][0][2][0][RTW89_WW][5] = 60, + [0][0][2][0][RTW89_WW][6] = 60, + [0][0][2][0][RTW89_WW][7] = 60, + [0][0][2][0][RTW89_WW][8] = 60, + [0][0][2][0][RTW89_WW][9] = 60, + [0][0][2][0][RTW89_WW][10] = 60, + [0][0][2][0][RTW89_WW][11] = 60, + [0][0][2][0][RTW89_WW][12] = 60, + [0][0][2][0][RTW89_WW][13] = 0, + [0][1][2][0][RTW89_WW][0] = 0, + [0][1][2][0][RTW89_WW][1] = 0, + [0][1][2][0][RTW89_WW][2] = 0, + [0][1][2][0][RTW89_WW][3] = 0, + [0][1][2][0][RTW89_WW][4] = 0, + [0][1][2][0][RTW89_WW][5] = 0, + [0][1][2][0][RTW89_WW][6] = 0, + [0][1][2][0][RTW89_WW][7] = 0, + [0][1][2][0][RTW89_WW][8] = 0, + [0][1][2][0][RTW89_WW][9] = 0, + [0][1][2][0][RTW89_WW][10] = 0, + [0][1][2][0][RTW89_WW][11] = 0, + [0][1][2][0][RTW89_WW][12] = 0, + [0][1][2][0][RTW89_WW][13] = 0, + [0][1][2][1][RTW89_WW][0] = 0, + [0][1][2][1][RTW89_WW][1] = 0, + [0][1][2][1][RTW89_WW][2] = 0, + [0][1][2][1][RTW89_WW][3] = 0, + [0][1][2][1][RTW89_WW][4] = 0, + [0][1][2][1][RTW89_WW][5] = 0, + [0][1][2][1][RTW89_WW][6] = 0, + [0][1][2][1][RTW89_WW][7] = 0, + [0][1][2][1][RTW89_WW][8] = 0, + [0][1][2][1][RTW89_WW][9] = 0, + [0][1][2][1][RTW89_WW][10] = 0, + [0][1][2][1][RTW89_WW][11] = 0, + [0][1][2][1][RTW89_WW][12] = 0, + [0][1][2][1][RTW89_WW][13] = 0, + [1][0][2][0][RTW89_WW][0] = 0, + [1][0][2][0][RTW89_WW][1] = 0, + [1][0][2][0][RTW89_WW][2] = 58, + [1][0][2][0][RTW89_WW][3] = 58, + [1][0][2][0][RTW89_WW][4] = 58, + [1][0][2][0][RTW89_WW][5] = 58, + [1][0][2][0][RTW89_WW][6] = 58, + [1][0][2][0][RTW89_WW][7] = 58, + [1][0][2][0][RTW89_WW][8] = 58, + [1][0][2][0][RTW89_WW][9] = 58, + [1][0][2][0][RTW89_WW][10] = 58, + [1][0][2][0][RTW89_WW][11] = 0, + [1][0][2][0][RTW89_WW][12] = 0, + [1][0][2][0][RTW89_WW][13] = 0, + [1][1][2][0][RTW89_WW][0] = 0, + [1][1][2][0][RTW89_WW][1] = 0, + [1][1][2][0][RTW89_WW][2] = 0, + [1][1][2][0][RTW89_WW][3] = 0, + [1][1][2][0][RTW89_WW][4] = 0, + [1][1][2][0][RTW89_WW][5] = 0, + [1][1][2][0][RTW89_WW][6] = 0, + [1][1][2][0][RTW89_WW][7] = 0, + [1][1][2][0][RTW89_WW][8] = 0, + [1][1][2][0][RTW89_WW][9] = 0, + [1][1][2][0][RTW89_WW][10] = 0, + [1][1][2][0][RTW89_WW][11] = 0, + [1][1][2][0][RTW89_WW][12] = 0, + [1][1][2][0][RTW89_WW][13] = 0, + [1][1][2][1][RTW89_WW][0] = 0, + [1][1][2][1][RTW89_WW][1] = 0, + [1][1][2][1][RTW89_WW][2] = 0, + [1][1][2][1][RTW89_WW][3] = 0, + [1][1][2][1][RTW89_WW][4] = 0, + [1][1][2][1][RTW89_WW][5] = 0, + [1][1][2][1][RTW89_WW][6] = 0, + [1][1][2][1][RTW89_WW][7] = 0, + [1][1][2][1][RTW89_WW][8] = 0, + [1][1][2][1][RTW89_WW][9] = 0, + [1][1][2][1][RTW89_WW][10] = 0, + [1][1][2][1][RTW89_WW][11] = 0, + [1][1][2][1][RTW89_WW][12] = 0, + [1][1][2][1][RTW89_WW][13] = 0, + [0][0][0][0][RTW89_FCC][0] = 84, + [0][0][0][0][RTW89_ETSI][0] = 58, + [0][0][0][0][RTW89_MKK][0] = 68, + [0][0][0][0][RTW89_IC][0] = 84, + [0][0][0][0][RTW89_KCC][0] = 68, + [0][0][0][0][RTW89_ACMA][0] = 58, + [0][0][0][0][RTW89_CN][0] = 60, + [0][0][0][0][RTW89_UK][0] = 58, + [0][0][0][0][RTW89_FCC][1] = 84, + [0][0][0][0][RTW89_ETSI][1] = 58, + [0][0][0][0][RTW89_MKK][1] = 68, + [0][0][0][0][RTW89_IC][1] = 84, + [0][0][0][0][RTW89_KCC][1] = 68, + [0][0][0][0][RTW89_ACMA][1] = 58, + [0][0][0][0][RTW89_CN][1] = 60, + [0][0][0][0][RTW89_UK][1] = 58, + [0][0][0][0][RTW89_FCC][2] = 84, + [0][0][0][0][RTW89_ETSI][2] = 58, + [0][0][0][0][RTW89_MKK][2] = 68, + [0][0][0][0][RTW89_IC][2] = 84, + [0][0][0][0][RTW89_KCC][2] = 68, + [0][0][0][0][RTW89_ACMA][2] = 58, + [0][0][0][0][RTW89_CN][2] = 60, + [0][0][0][0][RTW89_UK][2] = 58, + [0][0][0][0][RTW89_FCC][3] = 84, + [0][0][0][0][RTW89_ETSI][3] = 58, + [0][0][0][0][RTW89_MKK][3] = 68, + [0][0][0][0][RTW89_IC][3] = 84, + [0][0][0][0][RTW89_KCC][3] = 68, + [0][0][0][0][RTW89_ACMA][3] = 58, + [0][0][0][0][RTW89_CN][3] = 60, + [0][0][0][0][RTW89_UK][3] = 58, + [0][0][0][0][RTW89_FCC][4] = 84, + [0][0][0][0][RTW89_ETSI][4] = 58, + [0][0][0][0][RTW89_MKK][4] = 68, + [0][0][0][0][RTW89_IC][4] = 84, + [0][0][0][0][RTW89_KCC][4] = 68, + [0][0][0][0][RTW89_ACMA][4] = 58, + [0][0][0][0][RTW89_CN][4] = 60, + [0][0][0][0][RTW89_UK][4] = 58, + [0][0][0][0][RTW89_FCC][5] = 84, + [0][0][0][0][RTW89_ETSI][5] = 58, + [0][0][0][0][RTW89_MKK][5] = 68, + [0][0][0][0][RTW89_IC][5] = 84, + [0][0][0][0][RTW89_KCC][5] = 68, + [0][0][0][0][RTW89_ACMA][5] = 58, + [0][0][0][0][RTW89_CN][5] = 60, + [0][0][0][0][RTW89_UK][5] = 58, + [0][0][0][0][RTW89_FCC][6] = 84, + [0][0][0][0][RTW89_ETSI][6] = 58, + [0][0][0][0][RTW89_MKK][6] = 68, + [0][0][0][0][RTW89_IC][6] = 84, + [0][0][0][0][RTW89_KCC][6] = 68, + [0][0][0][0][RTW89_ACMA][6] = 58, + [0][0][0][0][RTW89_CN][6] = 60, + [0][0][0][0][RTW89_UK][6] = 58, + [0][0][0][0][RTW89_FCC][7] = 84, + [0][0][0][0][RTW89_ETSI][7] = 58, + [0][0][0][0][RTW89_MKK][7] = 68, + [0][0][0][0][RTW89_IC][7] = 84, + [0][0][0][0][RTW89_KCC][7] = 68, + [0][0][0][0][RTW89_ACMA][7] = 58, + [0][0][0][0][RTW89_CN][7] = 60, + [0][0][0][0][RTW89_UK][7] = 58, + [0][0][0][0][RTW89_FCC][8] = 84, + [0][0][0][0][RTW89_ETSI][8] = 58, + [0][0][0][0][RTW89_MKK][8] = 68, + [0][0][0][0][RTW89_IC][8] = 84, + [0][0][0][0][RTW89_KCC][8] = 68, + [0][0][0][0][RTW89_ACMA][8] = 58, + [0][0][0][0][RTW89_CN][8] = 60, + [0][0][0][0][RTW89_UK][8] = 58, + [0][0][0][0][RTW89_FCC][9] = 84, + [0][0][0][0][RTW89_ETSI][9] = 58, + [0][0][0][0][RTW89_MKK][9] = 68, + [0][0][0][0][RTW89_IC][9] = 84, + [0][0][0][0][RTW89_KCC][9] = 68, + [0][0][0][0][RTW89_ACMA][9] = 58, + [0][0][0][0][RTW89_CN][9] = 60, + [0][0][0][0][RTW89_UK][9] = 58, + [0][0][0][0][RTW89_FCC][10] = 82, + [0][0][0][0][RTW89_ETSI][10] = 58, + [0][0][0][0][RTW89_MKK][10] = 68, + [0][0][0][0][RTW89_IC][10] = 82, + [0][0][0][0][RTW89_KCC][10] = 68, + [0][0][0][0][RTW89_ACMA][10] = 58, + [0][0][0][0][RTW89_CN][10] = 60, + [0][0][0][0][RTW89_UK][10] = 58, + [0][0][0][0][RTW89_FCC][11] = 62, + [0][0][0][0][RTW89_ETSI][11] = 58, + [0][0][0][0][RTW89_MKK][11] = 68, + [0][0][0][0][RTW89_IC][11] = 62, + [0][0][0][0][RTW89_KCC][11] = 68, + [0][0][0][0][RTW89_ACMA][11] = 58, + [0][0][0][0][RTW89_CN][11] = 60, + [0][0][0][0][RTW89_UK][11] = 58, + [0][0][0][0][RTW89_FCC][12] = 52, + [0][0][0][0][RTW89_ETSI][12] = 58, + [0][0][0][0][RTW89_MKK][12] = 68, + [0][0][0][0][RTW89_IC][12] = 52, + [0][0][0][0][RTW89_KCC][12] = 68, + [0][0][0][0][RTW89_ACMA][12] = 58, + [0][0][0][0][RTW89_CN][12] = 60, + [0][0][0][0][RTW89_UK][12] = 58, + [0][0][0][0][RTW89_FCC][13] = 127, + [0][0][0][0][RTW89_ETSI][13] = 127, + [0][0][0][0][RTW89_MKK][13] = 76, + [0][0][0][0][RTW89_IC][13] = 127, + [0][0][0][0][RTW89_KCC][13] = 127, + [0][0][0][0][RTW89_ACMA][13] = 127, + [0][0][0][0][RTW89_CN][13] = 127, + [0][0][0][0][RTW89_UK][13] = 127, + [0][1][0][0][RTW89_FCC][0] = 127, + [0][1][0][0][RTW89_ETSI][0] = 127, + [0][1][0][0][RTW89_MKK][0] = 127, + [0][1][0][0][RTW89_IC][0] = 127, + [0][1][0][0][RTW89_KCC][0] = 127, + [0][1][0][0][RTW89_ACMA][0] = 127, + [0][1][0][0][RTW89_CN][0] = 127, + [0][1][0][0][RTW89_UK][0] = 127, + [0][1][0][0][RTW89_FCC][1] = 127, + [0][1][0][0][RTW89_ETSI][1] = 127, + [0][1][0][0][RTW89_MKK][1] = 127, + [0][1][0][0][RTW89_IC][1] = 127, + [0][1][0][0][RTW89_KCC][1] = 127, + [0][1][0][0][RTW89_ACMA][1] = 127, + [0][1][0][0][RTW89_CN][1] = 127, + [0][1][0][0][RTW89_UK][1] = 127, + [0][1][0][0][RTW89_FCC][2] = 127, + [0][1][0][0][RTW89_ETSI][2] = 127, + [0][1][0][0][RTW89_MKK][2] = 127, + [0][1][0][0][RTW89_IC][2] = 127, + [0][1][0][0][RTW89_KCC][2] = 127, + [0][1][0][0][RTW89_ACMA][2] = 127, + [0][1][0][0][RTW89_CN][2] = 127, + [0][1][0][0][RTW89_UK][2] = 127, + [0][1][0][0][RTW89_FCC][3] = 127, + [0][1][0][0][RTW89_ETSI][3] = 127, + [0][1][0][0][RTW89_MKK][3] = 127, + [0][1][0][0][RTW89_IC][3] = 127, + [0][1][0][0][RTW89_KCC][3] = 127, + [0][1][0][0][RTW89_ACMA][3] = 127, + [0][1][0][0][RTW89_CN][3] = 127, + [0][1][0][0][RTW89_UK][3] = 127, + [0][1][0][0][RTW89_FCC][4] = 127, + [0][1][0][0][RTW89_ETSI][4] = 127, + [0][1][0][0][RTW89_MKK][4] = 127, + [0][1][0][0][RTW89_IC][4] = 127, + [0][1][0][0][RTW89_KCC][4] = 127, + [0][1][0][0][RTW89_ACMA][4] = 127, + [0][1][0][0][RTW89_CN][4] = 127, + [0][1][0][0][RTW89_UK][4] = 127, + [0][1][0][0][RTW89_FCC][5] = 127, + [0][1][0][0][RTW89_ETSI][5] = 127, + [0][1][0][0][RTW89_MKK][5] = 127, + [0][1][0][0][RTW89_IC][5] = 127, + [0][1][0][0][RTW89_KCC][5] = 127, + [0][1][0][0][RTW89_ACMA][5] = 127, + [0][1][0][0][RTW89_CN][5] = 127, + [0][1][0][0][RTW89_UK][5] = 127, + [0][1][0][0][RTW89_FCC][6] = 127, + [0][1][0][0][RTW89_ETSI][6] = 127, + [0][1][0][0][RTW89_MKK][6] = 127, + [0][1][0][0][RTW89_IC][6] = 127, + [0][1][0][0][RTW89_KCC][6] = 127, + [0][1][0][0][RTW89_ACMA][6] = 127, + [0][1][0][0][RTW89_CN][6] = 127, + [0][1][0][0][RTW89_UK][6] = 127, + [0][1][0][0][RTW89_FCC][7] = 127, + [0][1][0][0][RTW89_ETSI][7] = 127, + [0][1][0][0][RTW89_MKK][7] = 127, + [0][1][0][0][RTW89_IC][7] = 127, + [0][1][0][0][RTW89_KCC][7] = 127, + [0][1][0][0][RTW89_ACMA][7] = 127, + [0][1][0][0][RTW89_CN][7] = 127, + [0][1][0][0][RTW89_UK][7] = 127, + [0][1][0][0][RTW89_FCC][8] = 127, + [0][1][0][0][RTW89_ETSI][8] = 127, + [0][1][0][0][RTW89_MKK][8] = 127, + [0][1][0][0][RTW89_IC][8] = 127, + [0][1][0][0][RTW89_KCC][8] = 127, + [0][1][0][0][RTW89_ACMA][8] = 127, + [0][1][0][0][RTW89_CN][8] = 127, + [0][1][0][0][RTW89_UK][8] = 127, + [0][1][0][0][RTW89_FCC][9] = 127, + [0][1][0][0][RTW89_ETSI][9] = 127, + [0][1][0][0][RTW89_MKK][9] = 127, + [0][1][0][0][RTW89_IC][9] = 127, + [0][1][0][0][RTW89_KCC][9] = 127, + [0][1][0][0][RTW89_ACMA][9] = 127, + [0][1][0][0][RTW89_CN][9] = 127, + [0][1][0][0][RTW89_UK][9] = 127, + [0][1][0][0][RTW89_FCC][10] = 127, + [0][1][0][0][RTW89_ETSI][10] = 127, + [0][1][0][0][RTW89_MKK][10] = 127, + [0][1][0][0][RTW89_IC][10] = 127, + [0][1][0][0][RTW89_KCC][10] = 127, + [0][1][0][0][RTW89_ACMA][10] = 127, + [0][1][0][0][RTW89_CN][10] = 127, + [0][1][0][0][RTW89_UK][10] = 127, + [0][1][0][0][RTW89_FCC][11] = 127, + [0][1][0][0][RTW89_ETSI][11] = 127, + [0][1][0][0][RTW89_MKK][11] = 127, + [0][1][0][0][RTW89_IC][11] = 127, + [0][1][0][0][RTW89_KCC][11] = 127, + [0][1][0][0][RTW89_ACMA][11] = 127, + [0][1][0][0][RTW89_CN][11] = 127, + [0][1][0][0][RTW89_UK][11] = 127, + [0][1][0][0][RTW89_FCC][12] = 127, + [0][1][0][0][RTW89_ETSI][12] = 127, + [0][1][0][0][RTW89_MKK][12] = 127, + [0][1][0][0][RTW89_IC][12] = 127, + [0][1][0][0][RTW89_KCC][12] = 127, + [0][1][0][0][RTW89_ACMA][12] = 127, + [0][1][0][0][RTW89_CN][12] = 127, + [0][1][0][0][RTW89_UK][12] = 127, + [0][1][0][0][RTW89_FCC][13] = 127, + [0][1][0][0][RTW89_ETSI][13] = 127, + [0][1][0][0][RTW89_MKK][13] = 127, + [0][1][0][0][RTW89_IC][13] = 127, + [0][1][0][0][RTW89_KCC][13] = 127, + [0][1][0][0][RTW89_ACMA][13] = 127, + [0][1][0][0][RTW89_CN][13] = 127, + [0][1][0][0][RTW89_UK][13] = 127, + [1][0][0][0][RTW89_FCC][0] = 127, + [1][0][0][0][RTW89_ETSI][0] = 127, + [1][0][0][0][RTW89_MKK][0] = 127, + [1][0][0][0][RTW89_IC][0] = 127, + [1][0][0][0][RTW89_KCC][0] = 127, + [1][0][0][0][RTW89_ACMA][0] = 127, + [1][0][0][0][RTW89_CN][0] = 127, + [1][0][0][0][RTW89_UK][0] = 127, + [1][0][0][0][RTW89_FCC][1] = 127, + [1][0][0][0][RTW89_ETSI][1] = 127, + [1][0][0][0][RTW89_MKK][1] = 127, + [1][0][0][0][RTW89_IC][1] = 127, + [1][0][0][0][RTW89_KCC][1] = 127, + [1][0][0][0][RTW89_ACMA][1] = 127, + [1][0][0][0][RTW89_CN][1] = 127, + [1][0][0][0][RTW89_UK][1] = 127, + [1][0][0][0][RTW89_FCC][2] = 127, + [1][0][0][0][RTW89_ETSI][2] = 58, + [1][0][0][0][RTW89_MKK][2] = 70, + [1][0][0][0][RTW89_IC][2] = 127, + [1][0][0][0][RTW89_KCC][2] = 68, + [1][0][0][0][RTW89_ACMA][2] = 58, + [1][0][0][0][RTW89_CN][2] = 60, + [1][0][0][0][RTW89_UK][2] = 58, + [1][0][0][0][RTW89_FCC][3] = 127, + [1][0][0][0][RTW89_ETSI][3] = 58, + [1][0][0][0][RTW89_MKK][3] = 76, + [1][0][0][0][RTW89_IC][3] = 127, + [1][0][0][0][RTW89_KCC][3] = 68, + [1][0][0][0][RTW89_ACMA][3] = 58, + [1][0][0][0][RTW89_CN][3] = 60, + [1][0][0][0][RTW89_UK][3] = 58, + [1][0][0][0][RTW89_FCC][4] = 127, + [1][0][0][0][RTW89_ETSI][4] = 58, + [1][0][0][0][RTW89_MKK][4] = 76, + [1][0][0][0][RTW89_IC][4] = 127, + [1][0][0][0][RTW89_KCC][4] = 68, + [1][0][0][0][RTW89_ACMA][4] = 58, + [1][0][0][0][RTW89_CN][4] = 60, + [1][0][0][0][RTW89_UK][4] = 58, + [1][0][0][0][RTW89_FCC][5] = 127, + [1][0][0][0][RTW89_ETSI][5] = 58, + [1][0][0][0][RTW89_MKK][5] = 76, + [1][0][0][0][RTW89_IC][5] = 127, + [1][0][0][0][RTW89_KCC][5] = 68, + [1][0][0][0][RTW89_ACMA][5] = 58, + [1][0][0][0][RTW89_CN][5] = 60, + [1][0][0][0][RTW89_UK][5] = 58, + [1][0][0][0][RTW89_FCC][6] = 127, + [1][0][0][0][RTW89_ETSI][6] = 58, + [1][0][0][0][RTW89_MKK][6] = 76, + [1][0][0][0][RTW89_IC][6] = 127, + [1][0][0][0][RTW89_KCC][6] = 68, + [1][0][0][0][RTW89_ACMA][6] = 58, + [1][0][0][0][RTW89_CN][6] = 60, + [1][0][0][0][RTW89_UK][6] = 58, + [1][0][0][0][RTW89_FCC][7] = 127, + [1][0][0][0][RTW89_ETSI][7] = 58, + [1][0][0][0][RTW89_MKK][7] = 76, + [1][0][0][0][RTW89_IC][7] = 127, + [1][0][0][0][RTW89_KCC][7] = 68, + [1][0][0][0][RTW89_ACMA][7] = 58, + [1][0][0][0][RTW89_CN][7] = 60, + [1][0][0][0][RTW89_UK][7] = 58, + [1][0][0][0][RTW89_FCC][8] = 127, + [1][0][0][0][RTW89_ETSI][8] = 58, + [1][0][0][0][RTW89_MKK][8] = 76, + [1][0][0][0][RTW89_IC][8] = 127, + [1][0][0][0][RTW89_KCC][8] = 68, + [1][0][0][0][RTW89_ACMA][8] = 58, + [1][0][0][0][RTW89_CN][8] = 60, + [1][0][0][0][RTW89_UK][8] = 58, + [1][0][0][0][RTW89_FCC][9] = 127, + [1][0][0][0][RTW89_ETSI][9] = 58, + [1][0][0][0][RTW89_MKK][9] = 76, + [1][0][0][0][RTW89_IC][9] = 127, + [1][0][0][0][RTW89_KCC][9] = 68, + [1][0][0][0][RTW89_ACMA][9] = 58, + [1][0][0][0][RTW89_CN][9] = 60, + [1][0][0][0][RTW89_UK][9] = 58, + [1][0][0][0][RTW89_FCC][10] = 127, + [1][0][0][0][RTW89_ETSI][10] = 58, + [1][0][0][0][RTW89_MKK][10] = 66, + [1][0][0][0][RTW89_IC][10] = 127, + [1][0][0][0][RTW89_KCC][10] = 68, + [1][0][0][0][RTW89_ACMA][10] = 58, + [1][0][0][0][RTW89_CN][10] = 60, + [1][0][0][0][RTW89_UK][10] = 58, + [1][0][0][0][RTW89_FCC][11] = 127, + [1][0][0][0][RTW89_ETSI][11] = 127, + [1][0][0][0][RTW89_MKK][11] = 127, + [1][0][0][0][RTW89_IC][11] = 127, + [1][0][0][0][RTW89_KCC][11] = 127, + [1][0][0][0][RTW89_ACMA][11] = 127, + [1][0][0][0][RTW89_CN][11] = 127, + [1][0][0][0][RTW89_UK][11] = 127, + [1][0][0][0][RTW89_FCC][12] = 127, + [1][0][0][0][RTW89_ETSI][12] = 127, + [1][0][0][0][RTW89_MKK][12] = 127, + [1][0][0][0][RTW89_IC][12] = 127, + [1][0][0][0][RTW89_KCC][12] = 127, + [1][0][0][0][RTW89_ACMA][12] = 127, + [1][0][0][0][RTW89_CN][12] = 127, + [1][0][0][0][RTW89_UK][12] = 127, + [1][0][0][0][RTW89_FCC][13] = 127, + [1][0][0][0][RTW89_ETSI][13] = 127, + [1][0][0][0][RTW89_MKK][13] = 127, + [1][0][0][0][RTW89_IC][13] = 127, + [1][0][0][0][RTW89_KCC][13] = 127, + [1][0][0][0][RTW89_ACMA][13] = 127, + [1][0][0][0][RTW89_CN][13] = 127, + [1][0][0][0][RTW89_UK][13] = 127, + [1][1][0][0][RTW89_FCC][0] = 127, + [1][1][0][0][RTW89_ETSI][0] = 127, + [1][1][0][0][RTW89_MKK][0] = 127, + [1][1][0][0][RTW89_IC][0] = 127, + [1][1][0][0][RTW89_KCC][0] = 127, + [1][1][0][0][RTW89_ACMA][0] = 127, + [1][1][0][0][RTW89_CN][0] = 127, + [1][1][0][0][RTW89_UK][0] = 127, + [1][1][0][0][RTW89_FCC][1] = 127, + [1][1][0][0][RTW89_ETSI][1] = 127, + [1][1][0][0][RTW89_MKK][1] = 127, + [1][1][0][0][RTW89_IC][1] = 127, + [1][1][0][0][RTW89_KCC][1] = 127, + [1][1][0][0][RTW89_ACMA][1] = 127, + [1][1][0][0][RTW89_CN][1] = 127, + [1][1][0][0][RTW89_UK][1] = 127, + [1][1][0][0][RTW89_FCC][2] = 127, + [1][1][0][0][RTW89_ETSI][2] = 127, + [1][1][0][0][RTW89_MKK][2] = 127, + [1][1][0][0][RTW89_IC][2] = 127, + [1][1][0][0][RTW89_KCC][2] = 127, + [1][1][0][0][RTW89_ACMA][2] = 127, + [1][1][0][0][RTW89_CN][2] = 127, + [1][1][0][0][RTW89_UK][2] = 127, + [1][1][0][0][RTW89_FCC][3] = 127, + [1][1][0][0][RTW89_ETSI][3] = 127, + [1][1][0][0][RTW89_MKK][3] = 127, + [1][1][0][0][RTW89_IC][3] = 127, + [1][1][0][0][RTW89_KCC][3] = 127, + [1][1][0][0][RTW89_ACMA][3] = 127, + [1][1][0][0][RTW89_CN][3] = 127, + [1][1][0][0][RTW89_UK][3] = 127, + [1][1][0][0][RTW89_FCC][4] = 127, + [1][1][0][0][RTW89_ETSI][4] = 127, + [1][1][0][0][RTW89_MKK][4] = 127, + [1][1][0][0][RTW89_IC][4] = 127, + [1][1][0][0][RTW89_KCC][4] = 127, + [1][1][0][0][RTW89_ACMA][4] = 127, + [1][1][0][0][RTW89_CN][4] = 127, + [1][1][0][0][RTW89_UK][4] = 127, + [1][1][0][0][RTW89_FCC][5] = 127, + [1][1][0][0][RTW89_ETSI][5] = 127, + [1][1][0][0][RTW89_MKK][5] = 127, + [1][1][0][0][RTW89_IC][5] = 127, + [1][1][0][0][RTW89_KCC][5] = 127, + [1][1][0][0][RTW89_ACMA][5] = 127, + [1][1][0][0][RTW89_CN][5] = 127, + [1][1][0][0][RTW89_UK][5] = 127, + [1][1][0][0][RTW89_FCC][6] = 127, + [1][1][0][0][RTW89_ETSI][6] = 127, + [1][1][0][0][RTW89_MKK][6] = 127, + [1][1][0][0][RTW89_IC][6] = 127, + [1][1][0][0][RTW89_KCC][6] = 127, + [1][1][0][0][RTW89_ACMA][6] = 127, + [1][1][0][0][RTW89_CN][6] = 127, + [1][1][0][0][RTW89_UK][6] = 127, + [1][1][0][0][RTW89_FCC][7] = 127, + [1][1][0][0][RTW89_ETSI][7] = 127, + [1][1][0][0][RTW89_MKK][7] = 127, + [1][1][0][0][RTW89_IC][7] = 127, + [1][1][0][0][RTW89_KCC][7] = 127, + [1][1][0][0][RTW89_ACMA][7] = 127, + [1][1][0][0][RTW89_CN][7] = 127, + [1][1][0][0][RTW89_UK][7] = 127, + [1][1][0][0][RTW89_FCC][8] = 127, + [1][1][0][0][RTW89_ETSI][8] = 127, + [1][1][0][0][RTW89_MKK][8] = 127, + [1][1][0][0][RTW89_IC][8] = 127, + [1][1][0][0][RTW89_KCC][8] = 127, + [1][1][0][0][RTW89_ACMA][8] = 127, + [1][1][0][0][RTW89_CN][8] = 127, + [1][1][0][0][RTW89_UK][8] = 127, + [1][1][0][0][RTW89_FCC][9] = 127, + [1][1][0][0][RTW89_ETSI][9] = 127, + [1][1][0][0][RTW89_MKK][9] = 127, + [1][1][0][0][RTW89_IC][9] = 127, + [1][1][0][0][RTW89_KCC][9] = 127, + [1][1][0][0][RTW89_ACMA][9] = 127, + [1][1][0][0][RTW89_CN][9] = 127, + [1][1][0][0][RTW89_UK][9] = 127, + [1][1][0][0][RTW89_FCC][10] = 127, + [1][1][0][0][RTW89_ETSI][10] = 127, + [1][1][0][0][RTW89_MKK][10] = 127, + [1][1][0][0][RTW89_IC][10] = 127, + [1][1][0][0][RTW89_KCC][10] = 127, + [1][1][0][0][RTW89_ACMA][10] = 127, + [1][1][0][0][RTW89_CN][10] = 127, + [1][1][0][0][RTW89_UK][10] = 127, + [1][1][0][0][RTW89_FCC][11] = 127, + [1][1][0][0][RTW89_ETSI][11] = 127, + [1][1][0][0][RTW89_MKK][11] = 127, + [1][1][0][0][RTW89_IC][11] = 127, + [1][1][0][0][RTW89_KCC][11] = 127, + [1][1][0][0][RTW89_ACMA][11] = 127, + [1][1][0][0][RTW89_CN][11] = 127, + [1][1][0][0][RTW89_UK][11] = 127, + [1][1][0][0][RTW89_FCC][12] = 127, + [1][1][0][0][RTW89_ETSI][12] = 127, + [1][1][0][0][RTW89_MKK][12] = 127, + [1][1][0][0][RTW89_IC][12] = 127, + [1][1][0][0][RTW89_KCC][12] = 127, + [1][1][0][0][RTW89_ACMA][12] = 127, + [1][1][0][0][RTW89_CN][12] = 127, + [1][1][0][0][RTW89_UK][12] = 127, + [1][1][0][0][RTW89_FCC][13] = 127, + [1][1][0][0][RTW89_ETSI][13] = 127, + [1][1][0][0][RTW89_MKK][13] = 127, + [1][1][0][0][RTW89_IC][13] = 127, + [1][1][0][0][RTW89_KCC][13] = 127, + [1][1][0][0][RTW89_ACMA][13] = 127, + [1][1][0][0][RTW89_CN][13] = 127, + [1][1][0][0][RTW89_UK][13] = 127, + [0][0][1][0][RTW89_FCC][0] = 80, + [0][0][1][0][RTW89_ETSI][0] = 58, + [0][0][1][0][RTW89_MKK][0] = 72, + [0][0][1][0][RTW89_IC][0] = 80, + [0][0][1][0][RTW89_KCC][0] = 78, + [0][0][1][0][RTW89_ACMA][0] = 58, + [0][0][1][0][RTW89_CN][0] = 60, + [0][0][1][0][RTW89_UK][0] = 58, + [0][0][1][0][RTW89_FCC][1] = 80, + [0][0][1][0][RTW89_ETSI][1] = 60, + [0][0][1][0][RTW89_MKK][1] = 74, + [0][0][1][0][RTW89_IC][1] = 80, + [0][0][1][0][RTW89_KCC][1] = 78, + [0][0][1][0][RTW89_ACMA][1] = 60, + [0][0][1][0][RTW89_CN][1] = 60, + [0][0][1][0][RTW89_UK][1] = 60, + [0][0][1][0][RTW89_FCC][2] = 84, + [0][0][1][0][RTW89_ETSI][2] = 60, + [0][0][1][0][RTW89_MKK][2] = 74, + [0][0][1][0][RTW89_IC][2] = 84, + [0][0][1][0][RTW89_KCC][2] = 78, + [0][0][1][0][RTW89_ACMA][2] = 60, + [0][0][1][0][RTW89_CN][2] = 60, + [0][0][1][0][RTW89_UK][2] = 60, + [0][0][1][0][RTW89_FCC][3] = 84, + [0][0][1][0][RTW89_ETSI][3] = 60, + [0][0][1][0][RTW89_MKK][3] = 74, + [0][0][1][0][RTW89_IC][3] = 84, + [0][0][1][0][RTW89_KCC][3] = 78, + [0][0][1][0][RTW89_ACMA][3] = 60, + [0][0][1][0][RTW89_CN][3] = 60, + [0][0][1][0][RTW89_UK][3] = 60, + [0][0][1][0][RTW89_FCC][4] = 84, + [0][0][1][0][RTW89_ETSI][4] = 60, + [0][0][1][0][RTW89_MKK][4] = 74, + [0][0][1][0][RTW89_IC][4] = 84, + [0][0][1][0][RTW89_KCC][4] = 76, + [0][0][1][0][RTW89_ACMA][4] = 60, + [0][0][1][0][RTW89_CN][4] = 60, + [0][0][1][0][RTW89_UK][4] = 60, + [0][0][1][0][RTW89_FCC][5] = 84, + [0][0][1][0][RTW89_ETSI][5] = 60, + [0][0][1][0][RTW89_MKK][5] = 74, + [0][0][1][0][RTW89_IC][5] = 84, + [0][0][1][0][RTW89_KCC][5] = 76, + [0][0][1][0][RTW89_ACMA][5] = 60, + [0][0][1][0][RTW89_CN][5] = 60, + [0][0][1][0][RTW89_UK][5] = 60, + [0][0][1][0][RTW89_FCC][6] = 84, + [0][0][1][0][RTW89_ETSI][6] = 60, + [0][0][1][0][RTW89_MKK][6] = 74, + [0][0][1][0][RTW89_IC][6] = 84, + [0][0][1][0][RTW89_KCC][6] = 76, + [0][0][1][0][RTW89_ACMA][6] = 60, + [0][0][1][0][RTW89_CN][6] = 60, + [0][0][1][0][RTW89_UK][6] = 60, + [0][0][1][0][RTW89_FCC][7] = 84, + [0][0][1][0][RTW89_ETSI][7] = 60, + [0][0][1][0][RTW89_MKK][7] = 74, + [0][0][1][0][RTW89_IC][7] = 84, + [0][0][1][0][RTW89_KCC][7] = 76, + [0][0][1][0][RTW89_ACMA][7] = 60, + [0][0][1][0][RTW89_CN][7] = 60, + [0][0][1][0][RTW89_UK][7] = 60, + [0][0][1][0][RTW89_FCC][8] = 80, + [0][0][1][0][RTW89_ETSI][8] = 60, + [0][0][1][0][RTW89_MKK][8] = 74, + [0][0][1][0][RTW89_IC][8] = 80, + [0][0][1][0][RTW89_KCC][8] = 76, + [0][0][1][0][RTW89_ACMA][8] = 60, + [0][0][1][0][RTW89_CN][8] = 60, + [0][0][1][0][RTW89_UK][8] = 60, + [0][0][1][0][RTW89_FCC][9] = 76, + [0][0][1][0][RTW89_ETSI][9] = 60, + [0][0][1][0][RTW89_MKK][9] = 74, + [0][0][1][0][RTW89_IC][9] = 76, + [0][0][1][0][RTW89_KCC][9] = 74, + [0][0][1][0][RTW89_ACMA][9] = 60, + [0][0][1][0][RTW89_CN][9] = 60, + [0][0][1][0][RTW89_UK][9] = 60, + [0][0][1][0][RTW89_FCC][10] = 76, + [0][0][1][0][RTW89_ETSI][10] = 60, + [0][0][1][0][RTW89_MKK][10] = 74, + [0][0][1][0][RTW89_IC][10] = 76, + [0][0][1][0][RTW89_KCC][10] = 74, + [0][0][1][0][RTW89_ACMA][10] = 60, + [0][0][1][0][RTW89_CN][10] = 60, + [0][0][1][0][RTW89_UK][10] = 60, + [0][0][1][0][RTW89_FCC][11] = 68, + [0][0][1][0][RTW89_ETSI][11] = 60, + [0][0][1][0][RTW89_MKK][11] = 74, + [0][0][1][0][RTW89_IC][11] = 68, + [0][0][1][0][RTW89_KCC][11] = 74, + [0][0][1][0][RTW89_ACMA][11] = 60, + [0][0][1][0][RTW89_CN][11] = 60, + [0][0][1][0][RTW89_UK][11] = 60, + [0][0][1][0][RTW89_FCC][12] = 64, + [0][0][1][0][RTW89_ETSI][12] = 58, + [0][0][1][0][RTW89_MKK][12] = 70, + [0][0][1][0][RTW89_IC][12] = 64, + [0][0][1][0][RTW89_KCC][12] = 74, + [0][0][1][0][RTW89_ACMA][12] = 58, + [0][0][1][0][RTW89_CN][12] = 60, + [0][0][1][0][RTW89_UK][12] = 58, + [0][0][1][0][RTW89_FCC][13] = 127, + [0][0][1][0][RTW89_ETSI][13] = 127, + [0][0][1][0][RTW89_MKK][13] = 127, + [0][0][1][0][RTW89_IC][13] = 127, + [0][0][1][0][RTW89_KCC][13] = 127, + [0][0][1][0][RTW89_ACMA][13] = 127, + [0][0][1][0][RTW89_CN][13] = 127, + [0][0][1][0][RTW89_UK][13] = 127, + [0][1][1][0][RTW89_FCC][0] = 127, + [0][1][1][0][RTW89_ETSI][0] = 127, + [0][1][1][0][RTW89_MKK][0] = 127, + [0][1][1][0][RTW89_IC][0] = 127, + [0][1][1][0][RTW89_KCC][0] = 127, + [0][1][1][0][RTW89_ACMA][0] = 127, + [0][1][1][0][RTW89_CN][0] = 127, + [0][1][1][0][RTW89_UK][0] = 127, + [0][1][1][0][RTW89_FCC][1] = 127, + [0][1][1][0][RTW89_ETSI][1] = 127, + [0][1][1][0][RTW89_MKK][1] = 127, + [0][1][1][0][RTW89_IC][1] = 127, + [0][1][1][0][RTW89_KCC][1] = 127, + [0][1][1][0][RTW89_ACMA][1] = 127, + [0][1][1][0][RTW89_CN][1] = 127, + [0][1][1][0][RTW89_UK][1] = 127, + [0][1][1][0][RTW89_FCC][2] = 127, + [0][1][1][0][RTW89_ETSI][2] = 127, + [0][1][1][0][RTW89_MKK][2] = 127, + [0][1][1][0][RTW89_IC][2] = 127, + [0][1][1][0][RTW89_KCC][2] = 127, + [0][1][1][0][RTW89_ACMA][2] = 127, + [0][1][1][0][RTW89_CN][2] = 127, + [0][1][1][0][RTW89_UK][2] = 127, + [0][1][1][0][RTW89_FCC][3] = 127, + [0][1][1][0][RTW89_ETSI][3] = 127, + [0][1][1][0][RTW89_MKK][3] = 127, + [0][1][1][0][RTW89_IC][3] = 127, + [0][1][1][0][RTW89_KCC][3] = 127, + [0][1][1][0][RTW89_ACMA][3] = 127, + [0][1][1][0][RTW89_CN][3] = 127, + [0][1][1][0][RTW89_UK][3] = 127, + [0][1][1][0][RTW89_FCC][4] = 127, + [0][1][1][0][RTW89_ETSI][4] = 127, + [0][1][1][0][RTW89_MKK][4] = 127, + [0][1][1][0][RTW89_IC][4] = 127, + [0][1][1][0][RTW89_KCC][4] = 127, + [0][1][1][0][RTW89_ACMA][4] = 127, + [0][1][1][0][RTW89_CN][4] = 127, + [0][1][1][0][RTW89_UK][4] = 127, + [0][1][1][0][RTW89_FCC][5] = 127, + [0][1][1][0][RTW89_ETSI][5] = 127, + [0][1][1][0][RTW89_MKK][5] = 127, + [0][1][1][0][RTW89_IC][5] = 127, + [0][1][1][0][RTW89_KCC][5] = 127, + [0][1][1][0][RTW89_ACMA][5] = 127, + [0][1][1][0][RTW89_CN][5] = 127, + [0][1][1][0][RTW89_UK][5] = 127, + [0][1][1][0][RTW89_FCC][6] = 127, + [0][1][1][0][RTW89_ETSI][6] = 127, + [0][1][1][0][RTW89_MKK][6] = 127, + [0][1][1][0][RTW89_IC][6] = 127, + [0][1][1][0][RTW89_KCC][6] = 127, + [0][1][1][0][RTW89_ACMA][6] = 127, + [0][1][1][0][RTW89_CN][6] = 127, + [0][1][1][0][RTW89_UK][6] = 127, + [0][1][1][0][RTW89_FCC][7] = 127, + [0][1][1][0][RTW89_ETSI][7] = 127, + [0][1][1][0][RTW89_MKK][7] = 127, + [0][1][1][0][RTW89_IC][7] = 127, + [0][1][1][0][RTW89_KCC][7] = 127, + [0][1][1][0][RTW89_ACMA][7] = 127, + [0][1][1][0][RTW89_CN][7] = 127, + [0][1][1][0][RTW89_UK][7] = 127, + [0][1][1][0][RTW89_FCC][8] = 127, + [0][1][1][0][RTW89_ETSI][8] = 127, + [0][1][1][0][RTW89_MKK][8] = 127, + [0][1][1][0][RTW89_IC][8] = 127, + [0][1][1][0][RTW89_KCC][8] = 127, + [0][1][1][0][RTW89_ACMA][8] = 127, + [0][1][1][0][RTW89_CN][8] = 127, + [0][1][1][0][RTW89_UK][8] = 127, + [0][1][1][0][RTW89_FCC][9] = 127, + [0][1][1][0][RTW89_ETSI][9] = 127, + [0][1][1][0][RTW89_MKK][9] = 127, + [0][1][1][0][RTW89_IC][9] = 127, + [0][1][1][0][RTW89_KCC][9] = 127, + [0][1][1][0][RTW89_ACMA][9] = 127, + [0][1][1][0][RTW89_CN][9] = 127, + [0][1][1][0][RTW89_UK][9] = 127, + [0][1][1][0][RTW89_FCC][10] = 127, + [0][1][1][0][RTW89_ETSI][10] = 127, + [0][1][1][0][RTW89_MKK][10] = 127, + [0][1][1][0][RTW89_IC][10] = 127, + [0][1][1][0][RTW89_KCC][10] = 127, + [0][1][1][0][RTW89_ACMA][10] = 127, + [0][1][1][0][RTW89_CN][10] = 127, + [0][1][1][0][RTW89_UK][10] = 127, + [0][1][1][0][RTW89_FCC][11] = 127, + [0][1][1][0][RTW89_ETSI][11] = 127, + [0][1][1][0][RTW89_MKK][11] = 127, + [0][1][1][0][RTW89_IC][11] = 127, + [0][1][1][0][RTW89_KCC][11] = 127, + [0][1][1][0][RTW89_ACMA][11] = 127, + [0][1][1][0][RTW89_CN][11] = 127, + [0][1][1][0][RTW89_UK][11] = 127, + [0][1][1][0][RTW89_FCC][12] = 127, + [0][1][1][0][RTW89_ETSI][12] = 127, + [0][1][1][0][RTW89_MKK][12] = 127, + [0][1][1][0][RTW89_IC][12] = 127, + [0][1][1][0][RTW89_KCC][12] = 127, + [0][1][1][0][RTW89_ACMA][12] = 127, + [0][1][1][0][RTW89_CN][12] = 127, + [0][1][1][0][RTW89_UK][12] = 127, + [0][1][1][0][RTW89_FCC][13] = 127, + [0][1][1][0][RTW89_ETSI][13] = 127, + [0][1][1][0][RTW89_MKK][13] = 127, + [0][1][1][0][RTW89_IC][13] = 127, + [0][1][1][0][RTW89_KCC][13] = 127, + [0][1][1][0][RTW89_ACMA][13] = 127, + [0][1][1][0][RTW89_CN][13] = 127, + [0][1][1][0][RTW89_UK][13] = 127, + [0][0][2][0][RTW89_FCC][0] = 78, + [0][0][2][0][RTW89_ETSI][0] = 60, + [0][0][2][0][RTW89_MKK][0] = 72, + [0][0][2][0][RTW89_IC][0] = 78, + [0][0][2][0][RTW89_KCC][0] = 78, + [0][0][2][0][RTW89_ACMA][0] = 60, + [0][0][2][0][RTW89_CN][0] = 60, + [0][0][2][0][RTW89_UK][0] = 60, + [0][0][2][0][RTW89_FCC][1] = 78, + [0][0][2][0][RTW89_ETSI][1] = 60, + [0][0][2][0][RTW89_MKK][1] = 78, + [0][0][2][0][RTW89_IC][1] = 78, + [0][0][2][0][RTW89_KCC][1] = 78, + [0][0][2][0][RTW89_ACMA][1] = 60, + [0][0][2][0][RTW89_CN][1] = 60, + [0][0][2][0][RTW89_UK][1] = 60, + [0][0][2][0][RTW89_FCC][2] = 82, + [0][0][2][0][RTW89_ETSI][2] = 60, + [0][0][2][0][RTW89_MKK][2] = 78, + [0][0][2][0][RTW89_IC][2] = 82, + [0][0][2][0][RTW89_KCC][2] = 78, + [0][0][2][0][RTW89_ACMA][2] = 60, + [0][0][2][0][RTW89_CN][2] = 60, + [0][0][2][0][RTW89_UK][2] = 60, + [0][0][2][0][RTW89_FCC][3] = 82, + [0][0][2][0][RTW89_ETSI][3] = 60, + [0][0][2][0][RTW89_MKK][3] = 78, + [0][0][2][0][RTW89_IC][3] = 82, + [0][0][2][0][RTW89_KCC][3] = 78, + [0][0][2][0][RTW89_ACMA][3] = 60, + [0][0][2][0][RTW89_CN][3] = 60, + [0][0][2][0][RTW89_UK][3] = 60, + [0][0][2][0][RTW89_FCC][4] = 82, + [0][0][2][0][RTW89_ETSI][4] = 60, + [0][0][2][0][RTW89_MKK][4] = 78, + [0][0][2][0][RTW89_IC][4] = 82, + [0][0][2][0][RTW89_KCC][4] = 78, + [0][0][2][0][RTW89_ACMA][4] = 60, + [0][0][2][0][RTW89_CN][4] = 60, + [0][0][2][0][RTW89_UK][4] = 60, + [0][0][2][0][RTW89_FCC][5] = 82, + [0][0][2][0][RTW89_ETSI][5] = 60, + [0][0][2][0][RTW89_MKK][5] = 78, + [0][0][2][0][RTW89_IC][5] = 82, + [0][0][2][0][RTW89_KCC][5] = 78, + [0][0][2][0][RTW89_ACMA][5] = 60, + [0][0][2][0][RTW89_CN][5] = 60, + [0][0][2][0][RTW89_UK][5] = 60, + [0][0][2][0][RTW89_FCC][6] = 82, + [0][0][2][0][RTW89_ETSI][6] = 60, + [0][0][2][0][RTW89_MKK][6] = 78, + [0][0][2][0][RTW89_IC][6] = 82, + [0][0][2][0][RTW89_KCC][6] = 78, + [0][0][2][0][RTW89_ACMA][6] = 60, + [0][0][2][0][RTW89_CN][6] = 60, + [0][0][2][0][RTW89_UK][6] = 60, + [0][0][2][0][RTW89_FCC][7] = 82, + [0][0][2][0][RTW89_ETSI][7] = 60, + [0][0][2][0][RTW89_MKK][7] = 78, + [0][0][2][0][RTW89_IC][7] = 82, + [0][0][2][0][RTW89_KCC][7] = 78, + [0][0][2][0][RTW89_ACMA][7] = 60, + [0][0][2][0][RTW89_CN][7] = 60, + [0][0][2][0][RTW89_UK][7] = 60, + [0][0][2][0][RTW89_FCC][8] = 80, + [0][0][2][0][RTW89_ETSI][8] = 60, + [0][0][2][0][RTW89_MKK][8] = 78, + [0][0][2][0][RTW89_IC][8] = 80, + [0][0][2][0][RTW89_KCC][8] = 78, + [0][0][2][0][RTW89_ACMA][8] = 60, + [0][0][2][0][RTW89_CN][8] = 60, + [0][0][2][0][RTW89_UK][8] = 60, + [0][0][2][0][RTW89_FCC][9] = 76, + [0][0][2][0][RTW89_ETSI][9] = 60, + [0][0][2][0][RTW89_MKK][9] = 78, + [0][0][2][0][RTW89_IC][9] = 76, + [0][0][2][0][RTW89_KCC][9] = 78, + [0][0][2][0][RTW89_ACMA][9] = 60, + [0][0][2][0][RTW89_CN][9] = 60, + [0][0][2][0][RTW89_UK][9] = 60, + [0][0][2][0][RTW89_FCC][10] = 76, + [0][0][2][0][RTW89_ETSI][10] = 60, + [0][0][2][0][RTW89_MKK][10] = 78, + [0][0][2][0][RTW89_IC][10] = 76, + [0][0][2][0][RTW89_KCC][10] = 78, + [0][0][2][0][RTW89_ACMA][10] = 60, + [0][0][2][0][RTW89_CN][10] = 60, + [0][0][2][0][RTW89_UK][10] = 60, + [0][0][2][0][RTW89_FCC][11] = 70, + [0][0][2][0][RTW89_ETSI][11] = 60, + [0][0][2][0][RTW89_MKK][11] = 78, + [0][0][2][0][RTW89_IC][11] = 70, + [0][0][2][0][RTW89_KCC][11] = 78, + [0][0][2][0][RTW89_ACMA][11] = 60, + [0][0][2][0][RTW89_CN][11] = 60, + [0][0][2][0][RTW89_UK][11] = 60, + [0][0][2][0][RTW89_FCC][12] = 70, + [0][0][2][0][RTW89_ETSI][12] = 60, + [0][0][2][0][RTW89_MKK][12] = 70, + [0][0][2][0][RTW89_IC][12] = 70, + [0][0][2][0][RTW89_KCC][12] = 78, + [0][0][2][0][RTW89_ACMA][12] = 60, + [0][0][2][0][RTW89_CN][12] = 60, + [0][0][2][0][RTW89_UK][12] = 60, + [0][0][2][0][RTW89_FCC][13] = 127, + [0][0][2][0][RTW89_ETSI][13] = 127, + [0][0][2][0][RTW89_MKK][13] = 127, + [0][0][2][0][RTW89_IC][13] = 127, + [0][0][2][0][RTW89_KCC][13] = 127, + [0][0][2][0][RTW89_ACMA][13] = 127, + [0][0][2][0][RTW89_CN][13] = 127, + [0][0][2][0][RTW89_UK][13] = 127, + [0][1][2][0][RTW89_FCC][0] = 127, + [0][1][2][0][RTW89_ETSI][0] = 127, + [0][1][2][0][RTW89_MKK][0] = 127, + [0][1][2][0][RTW89_IC][0] = 127, + [0][1][2][0][RTW89_KCC][0] = 127, + [0][1][2][0][RTW89_ACMA][0] = 127, + [0][1][2][0][RTW89_CN][0] = 127, + [0][1][2][0][RTW89_UK][0] = 127, + [0][1][2][0][RTW89_FCC][1] = 127, + [0][1][2][0][RTW89_ETSI][1] = 127, + [0][1][2][0][RTW89_MKK][1] = 127, + [0][1][2][0][RTW89_IC][1] = 127, + [0][1][2][0][RTW89_KCC][1] = 127, + [0][1][2][0][RTW89_ACMA][1] = 127, + [0][1][2][0][RTW89_CN][1] = 127, + [0][1][2][0][RTW89_UK][1] = 127, + [0][1][2][0][RTW89_FCC][2] = 127, + [0][1][2][0][RTW89_ETSI][2] = 127, + [0][1][2][0][RTW89_MKK][2] = 127, + [0][1][2][0][RTW89_IC][2] = 127, + [0][1][2][0][RTW89_KCC][2] = 127, + [0][1][2][0][RTW89_ACMA][2] = 127, + [0][1][2][0][RTW89_CN][2] = 127, + [0][1][2][0][RTW89_UK][2] = 127, + [0][1][2][0][RTW89_FCC][3] = 127, + [0][1][2][0][RTW89_ETSI][3] = 127, + [0][1][2][0][RTW89_MKK][3] = 127, + [0][1][2][0][RTW89_IC][3] = 127, + [0][1][2][0][RTW89_KCC][3] = 127, + [0][1][2][0][RTW89_ACMA][3] = 127, + [0][1][2][0][RTW89_CN][3] = 127, + [0][1][2][0][RTW89_UK][3] = 127, + [0][1][2][0][RTW89_FCC][4] = 127, + [0][1][2][0][RTW89_ETSI][4] = 127, + [0][1][2][0][RTW89_MKK][4] = 127, + [0][1][2][0][RTW89_IC][4] = 127, + [0][1][2][0][RTW89_KCC][4] = 127, + [0][1][2][0][RTW89_ACMA][4] = 127, + [0][1][2][0][RTW89_CN][4] = 127, + [0][1][2][0][RTW89_UK][4] = 127, + [0][1][2][0][RTW89_FCC][5] = 127, + [0][1][2][0][RTW89_ETSI][5] = 127, + [0][1][2][0][RTW89_MKK][5] = 127, + [0][1][2][0][RTW89_IC][5] = 127, + [0][1][2][0][RTW89_KCC][5] = 127, + [0][1][2][0][RTW89_ACMA][5] = 127, + [0][1][2][0][RTW89_CN][5] = 127, + [0][1][2][0][RTW89_UK][5] = 127, + [0][1][2][0][RTW89_FCC][6] = 127, + [0][1][2][0][RTW89_ETSI][6] = 127, + [0][1][2][0][RTW89_MKK][6] = 127, + [0][1][2][0][RTW89_IC][6] = 127, + [0][1][2][0][RTW89_KCC][6] = 127, + [0][1][2][0][RTW89_ACMA][6] = 127, + [0][1][2][0][RTW89_CN][6] = 127, + [0][1][2][0][RTW89_UK][6] = 127, + [0][1][2][0][RTW89_FCC][7] = 127, + [0][1][2][0][RTW89_ETSI][7] = 127, + [0][1][2][0][RTW89_MKK][7] = 127, + [0][1][2][0][RTW89_IC][7] = 127, + [0][1][2][0][RTW89_KCC][7] = 127, + [0][1][2][0][RTW89_ACMA][7] = 127, + [0][1][2][0][RTW89_CN][7] = 127, + [0][1][2][0][RTW89_UK][7] = 127, + [0][1][2][0][RTW89_FCC][8] = 127, + [0][1][2][0][RTW89_ETSI][8] = 127, + [0][1][2][0][RTW89_MKK][8] = 127, + [0][1][2][0][RTW89_IC][8] = 127, + [0][1][2][0][RTW89_KCC][8] = 127, + [0][1][2][0][RTW89_ACMA][8] = 127, + [0][1][2][0][RTW89_CN][8] = 127, + [0][1][2][0][RTW89_UK][8] = 127, + [0][1][2][0][RTW89_FCC][9] = 127, + [0][1][2][0][RTW89_ETSI][9] = 127, + [0][1][2][0][RTW89_MKK][9] = 127, + [0][1][2][0][RTW89_IC][9] = 127, + [0][1][2][0][RTW89_KCC][9] = 127, + [0][1][2][0][RTW89_ACMA][9] = 127, + [0][1][2][0][RTW89_CN][9] = 127, + [0][1][2][0][RTW89_UK][9] = 127, + [0][1][2][0][RTW89_FCC][10] = 127, + [0][1][2][0][RTW89_ETSI][10] = 127, + [0][1][2][0][RTW89_MKK][10] = 127, + [0][1][2][0][RTW89_IC][10] = 127, + [0][1][2][0][RTW89_KCC][10] = 127, + [0][1][2][0][RTW89_ACMA][10] = 127, + [0][1][2][0][RTW89_CN][10] = 127, + [0][1][2][0][RTW89_UK][10] = 127, + [0][1][2][0][RTW89_FCC][11] = 127, + [0][1][2][0][RTW89_ETSI][11] = 127, + [0][1][2][0][RTW89_MKK][11] = 127, + [0][1][2][0][RTW89_IC][11] = 127, + [0][1][2][0][RTW89_KCC][11] = 127, + [0][1][2][0][RTW89_ACMA][11] = 127, + [0][1][2][0][RTW89_CN][11] = 127, + [0][1][2][0][RTW89_UK][11] = 127, + [0][1][2][0][RTW89_FCC][12] = 127, + [0][1][2][0][RTW89_ETSI][12] = 127, + [0][1][2][0][RTW89_MKK][12] = 127, + [0][1][2][0][RTW89_IC][12] = 127, + [0][1][2][0][RTW89_KCC][12] = 127, + [0][1][2][0][RTW89_ACMA][12] = 127, + [0][1][2][0][RTW89_CN][12] = 127, + [0][1][2][0][RTW89_UK][12] = 127, + [0][1][2][0][RTW89_FCC][13] = 127, + [0][1][2][0][RTW89_ETSI][13] = 127, + [0][1][2][0][RTW89_MKK][13] = 127, + [0][1][2][0][RTW89_IC][13] = 127, + [0][1][2][0][RTW89_KCC][13] = 127, + [0][1][2][0][RTW89_ACMA][13] = 127, + [0][1][2][0][RTW89_CN][13] = 127, + [0][1][2][0][RTW89_UK][13] = 127, + [0][1][2][1][RTW89_FCC][0] = 127, + [0][1][2][1][RTW89_ETSI][0] = 127, + [0][1][2][1][RTW89_MKK][0] = 127, + [0][1][2][1][RTW89_IC][0] = 127, + [0][1][2][1][RTW89_KCC][0] = 127, + [0][1][2][1][RTW89_ACMA][0] = 127, + [0][1][2][1][RTW89_CN][0] = 127, + [0][1][2][1][RTW89_UK][0] = 127, + [0][1][2][1][RTW89_FCC][1] = 127, + [0][1][2][1][RTW89_ETSI][1] = 127, + [0][1][2][1][RTW89_MKK][1] = 127, + [0][1][2][1][RTW89_IC][1] = 127, + [0][1][2][1][RTW89_KCC][1] = 127, + [0][1][2][1][RTW89_ACMA][1] = 127, + [0][1][2][1][RTW89_CN][1] = 127, + [0][1][2][1][RTW89_UK][1] = 127, + [0][1][2][1][RTW89_FCC][2] = 127, + [0][1][2][1][RTW89_ETSI][2] = 127, + [0][1][2][1][RTW89_MKK][2] = 127, + [0][1][2][1][RTW89_IC][2] = 127, + [0][1][2][1][RTW89_KCC][2] = 127, + [0][1][2][1][RTW89_ACMA][2] = 127, + [0][1][2][1][RTW89_CN][2] = 127, + [0][1][2][1][RTW89_UK][2] = 127, + [0][1][2][1][RTW89_FCC][3] = 127, + [0][1][2][1][RTW89_ETSI][3] = 127, + [0][1][2][1][RTW89_MKK][3] = 127, + [0][1][2][1][RTW89_IC][3] = 127, + [0][1][2][1][RTW89_KCC][3] = 127, + [0][1][2][1][RTW89_ACMA][3] = 127, + [0][1][2][1][RTW89_CN][3] = 127, + [0][1][2][1][RTW89_UK][3] = 127, + [0][1][2][1][RTW89_FCC][4] = 127, + [0][1][2][1][RTW89_ETSI][4] = 127, + [0][1][2][1][RTW89_MKK][4] = 127, + [0][1][2][1][RTW89_IC][4] = 127, + [0][1][2][1][RTW89_KCC][4] = 127, + [0][1][2][1][RTW89_ACMA][4] = 127, + [0][1][2][1][RTW89_CN][4] = 127, + [0][1][2][1][RTW89_UK][4] = 127, + [0][1][2][1][RTW89_FCC][5] = 127, + [0][1][2][1][RTW89_ETSI][5] = 127, + [0][1][2][1][RTW89_MKK][5] = 127, + [0][1][2][1][RTW89_IC][5] = 127, + [0][1][2][1][RTW89_KCC][5] = 127, + [0][1][2][1][RTW89_ACMA][5] = 127, + [0][1][2][1][RTW89_CN][5] = 127, + [0][1][2][1][RTW89_UK][5] = 127, + [0][1][2][1][RTW89_FCC][6] = 127, + [0][1][2][1][RTW89_ETSI][6] = 127, + [0][1][2][1][RTW89_MKK][6] = 127, + [0][1][2][1][RTW89_IC][6] = 127, + [0][1][2][1][RTW89_KCC][6] = 127, + [0][1][2][1][RTW89_ACMA][6] = 127, + [0][1][2][1][RTW89_CN][6] = 127, + [0][1][2][1][RTW89_UK][6] = 127, + [0][1][2][1][RTW89_FCC][7] = 127, + [0][1][2][1][RTW89_ETSI][7] = 127, + [0][1][2][1][RTW89_MKK][7] = 127, + [0][1][2][1][RTW89_IC][7] = 127, + [0][1][2][1][RTW89_KCC][7] = 127, + [0][1][2][1][RTW89_ACMA][7] = 127, + [0][1][2][1][RTW89_CN][7] = 127, + [0][1][2][1][RTW89_UK][7] = 127, + [0][1][2][1][RTW89_FCC][8] = 127, + [0][1][2][1][RTW89_ETSI][8] = 127, + [0][1][2][1][RTW89_MKK][8] = 127, + [0][1][2][1][RTW89_IC][8] = 127, + [0][1][2][1][RTW89_KCC][8] = 127, + [0][1][2][1][RTW89_ACMA][8] = 127, + [0][1][2][1][RTW89_CN][8] = 127, + [0][1][2][1][RTW89_UK][8] = 127, + [0][1][2][1][RTW89_FCC][9] = 127, + [0][1][2][1][RTW89_ETSI][9] = 127, + [0][1][2][1][RTW89_MKK][9] = 127, + [0][1][2][1][RTW89_IC][9] = 127, + [0][1][2][1][RTW89_KCC][9] = 127, + [0][1][2][1][RTW89_ACMA][9] = 127, + [0][1][2][1][RTW89_CN][9] = 127, + [0][1][2][1][RTW89_UK][9] = 127, + [0][1][2][1][RTW89_FCC][10] = 127, + [0][1][2][1][RTW89_ETSI][10] = 127, + [0][1][2][1][RTW89_MKK][10] = 127, + [0][1][2][1][RTW89_IC][10] = 127, + [0][1][2][1][RTW89_KCC][10] = 127, + [0][1][2][1][RTW89_ACMA][10] = 127, + [0][1][2][1][RTW89_CN][10] = 127, + [0][1][2][1][RTW89_UK][10] = 127, + [0][1][2][1][RTW89_FCC][11] = 127, + [0][1][2][1][RTW89_ETSI][11] = 127, + [0][1][2][1][RTW89_MKK][11] = 127, + [0][1][2][1][RTW89_IC][11] = 127, + [0][1][2][1][RTW89_KCC][11] = 127, + [0][1][2][1][RTW89_ACMA][11] = 127, + [0][1][2][1][RTW89_CN][11] = 127, + [0][1][2][1][RTW89_UK][11] = 127, + [0][1][2][1][RTW89_FCC][12] = 127, + [0][1][2][1][RTW89_ETSI][12] = 127, + [0][1][2][1][RTW89_MKK][12] = 127, + [0][1][2][1][RTW89_IC][12] = 127, + [0][1][2][1][RTW89_KCC][12] = 127, + [0][1][2][1][RTW89_ACMA][12] = 127, + [0][1][2][1][RTW89_CN][12] = 127, + [0][1][2][1][RTW89_UK][12] = 127, + [0][1][2][1][RTW89_FCC][13] = 127, + [0][1][2][1][RTW89_ETSI][13] = 127, + [0][1][2][1][RTW89_MKK][13] = 127, + [0][1][2][1][RTW89_IC][13] = 127, + [0][1][2][1][RTW89_KCC][13] = 127, + [0][1][2][1][RTW89_ACMA][13] = 127, + [0][1][2][1][RTW89_CN][13] = 127, + [0][1][2][1][RTW89_UK][13] = 127, + [1][0][2][0][RTW89_FCC][0] = 127, + [1][0][2][0][RTW89_ETSI][0] = 127, + [1][0][2][0][RTW89_MKK][0] = 127, + [1][0][2][0][RTW89_IC][0] = 127, + [1][0][2][0][RTW89_KCC][0] = 127, + [1][0][2][0][RTW89_ACMA][0] = 127, + [1][0][2][0][RTW89_CN][0] = 127, + [1][0][2][0][RTW89_UK][0] = 127, + [1][0][2][0][RTW89_FCC][1] = 127, + [1][0][2][0][RTW89_ETSI][1] = 127, + [1][0][2][0][RTW89_MKK][1] = 127, + [1][0][2][0][RTW89_IC][1] = 127, + [1][0][2][0][RTW89_KCC][1] = 127, + [1][0][2][0][RTW89_ACMA][1] = 127, + [1][0][2][0][RTW89_CN][1] = 127, + [1][0][2][0][RTW89_UK][1] = 127, + [1][0][2][0][RTW89_FCC][2] = 72, + [1][0][2][0][RTW89_ETSI][2] = 58, + [1][0][2][0][RTW89_MKK][2] = 80, + [1][0][2][0][RTW89_IC][2] = 72, + [1][0][2][0][RTW89_KCC][2] = 80, + [1][0][2][0][RTW89_ACMA][2] = 58, + [1][0][2][0][RTW89_CN][2] = 60, + [1][0][2][0][RTW89_UK][2] = 58, + [1][0][2][0][RTW89_FCC][3] = 72, + [1][0][2][0][RTW89_ETSI][3] = 58, + [1][0][2][0][RTW89_MKK][3] = 80, + [1][0][2][0][RTW89_IC][3] = 72, + [1][0][2][0][RTW89_KCC][3] = 80, + [1][0][2][0][RTW89_ACMA][3] = 58, + [1][0][2][0][RTW89_CN][3] = 60, + [1][0][2][0][RTW89_UK][3] = 58, + [1][0][2][0][RTW89_FCC][4] = 76, + [1][0][2][0][RTW89_ETSI][4] = 58, + [1][0][2][0][RTW89_MKK][4] = 80, + [1][0][2][0][RTW89_IC][4] = 76, + [1][0][2][0][RTW89_KCC][4] = 80, + [1][0][2][0][RTW89_ACMA][4] = 58, + [1][0][2][0][RTW89_CN][4] = 60, + [1][0][2][0][RTW89_UK][4] = 58, + [1][0][2][0][RTW89_FCC][5] = 78, + [1][0][2][0][RTW89_ETSI][5] = 58, + [1][0][2][0][RTW89_MKK][5] = 80, + [1][0][2][0][RTW89_IC][5] = 78, + [1][0][2][0][RTW89_KCC][5] = 80, + [1][0][2][0][RTW89_ACMA][5] = 58, + [1][0][2][0][RTW89_CN][5] = 60, + [1][0][2][0][RTW89_UK][5] = 58, + [1][0][2][0][RTW89_FCC][6] = 78, + [1][0][2][0][RTW89_ETSI][6] = 58, + [1][0][2][0][RTW89_MKK][6] = 78, + [1][0][2][0][RTW89_IC][6] = 78, + [1][0][2][0][RTW89_KCC][6] = 80, + [1][0][2][0][RTW89_ACMA][6] = 58, + [1][0][2][0][RTW89_CN][6] = 60, + [1][0][2][0][RTW89_UK][6] = 58, + [1][0][2][0][RTW89_FCC][7] = 78, + [1][0][2][0][RTW89_ETSI][7] = 58, + [1][0][2][0][RTW89_MKK][7] = 80, + [1][0][2][0][RTW89_IC][7] = 78, + [1][0][2][0][RTW89_KCC][7] = 80, + [1][0][2][0][RTW89_ACMA][7] = 58, + [1][0][2][0][RTW89_CN][7] = 60, + [1][0][2][0][RTW89_UK][7] = 58, + [1][0][2][0][RTW89_FCC][8] = 78, + [1][0][2][0][RTW89_ETSI][8] = 58, + [1][0][2][0][RTW89_MKK][8] = 80, + [1][0][2][0][RTW89_IC][8] = 78, + [1][0][2][0][RTW89_KCC][8] = 78, + [1][0][2][0][RTW89_ACMA][8] = 58, + [1][0][2][0][RTW89_CN][8] = 60, + [1][0][2][0][RTW89_UK][8] = 58, + [1][0][2][0][RTW89_FCC][9] = 76, + [1][0][2][0][RTW89_ETSI][9] = 58, + [1][0][2][0][RTW89_MKK][9] = 80, + [1][0][2][0][RTW89_IC][9] = 76, + [1][0][2][0][RTW89_KCC][9] = 78, + [1][0][2][0][RTW89_ACMA][9] = 58, + [1][0][2][0][RTW89_CN][9] = 60, + [1][0][2][0][RTW89_UK][9] = 58, + [1][0][2][0][RTW89_FCC][10] = 70, + [1][0][2][0][RTW89_ETSI][10] = 58, + [1][0][2][0][RTW89_MKK][10] = 78, + [1][0][2][0][RTW89_IC][10] = 70, + [1][0][2][0][RTW89_KCC][10] = 78, + [1][0][2][0][RTW89_ACMA][10] = 58, + [1][0][2][0][RTW89_CN][10] = 60, + [1][0][2][0][RTW89_UK][10] = 58, + [1][0][2][0][RTW89_FCC][11] = 127, + [1][0][2][0][RTW89_ETSI][11] = 127, + [1][0][2][0][RTW89_MKK][11] = 127, + [1][0][2][0][RTW89_IC][11] = 127, + [1][0][2][0][RTW89_KCC][11] = 127, + [1][0][2][0][RTW89_ACMA][11] = 127, + [1][0][2][0][RTW89_CN][11] = 127, + [1][0][2][0][RTW89_UK][11] = 127, + [1][0][2][0][RTW89_FCC][12] = 127, + [1][0][2][0][RTW89_ETSI][12] = 127, + [1][0][2][0][RTW89_MKK][12] = 127, + [1][0][2][0][RTW89_IC][12] = 127, + [1][0][2][0][RTW89_KCC][12] = 127, + [1][0][2][0][RTW89_ACMA][12] = 127, + [1][0][2][0][RTW89_CN][12] = 127, + [1][0][2][0][RTW89_UK][12] = 127, + [1][0][2][0][RTW89_FCC][13] = 127, + [1][0][2][0][RTW89_ETSI][13] = 127, + [1][0][2][0][RTW89_MKK][13] = 127, + [1][0][2][0][RTW89_IC][13] = 127, + [1][0][2][0][RTW89_KCC][13] = 127, + [1][0][2][0][RTW89_ACMA][13] = 127, + [1][0][2][0][RTW89_CN][13] = 127, + [1][0][2][0][RTW89_UK][13] = 127, + [1][1][2][0][RTW89_FCC][0] = 127, + [1][1][2][0][RTW89_ETSI][0] = 127, + [1][1][2][0][RTW89_MKK][0] = 127, + [1][1][2][0][RTW89_IC][0] = 127, + [1][1][2][0][RTW89_KCC][0] = 127, + [1][1][2][0][RTW89_ACMA][0] = 127, + [1][1][2][0][RTW89_CN][0] = 127, + [1][1][2][0][RTW89_UK][0] = 127, + [1][1][2][0][RTW89_FCC][1] = 127, + [1][1][2][0][RTW89_ETSI][1] = 127, + [1][1][2][0][RTW89_MKK][1] = 127, + [1][1][2][0][RTW89_IC][1] = 127, + [1][1][2][0][RTW89_KCC][1] = 127, + [1][1][2][0][RTW89_ACMA][1] = 127, + [1][1][2][0][RTW89_CN][1] = 127, + [1][1][2][0][RTW89_UK][1] = 127, + [1][1][2][0][RTW89_FCC][2] = 127, + [1][1][2][0][RTW89_ETSI][2] = 127, + [1][1][2][0][RTW89_MKK][2] = 127, + [1][1][2][0][RTW89_IC][2] = 127, + [1][1][2][0][RTW89_KCC][2] = 127, + [1][1][2][0][RTW89_ACMA][2] = 127, + [1][1][2][0][RTW89_CN][2] = 127, + [1][1][2][0][RTW89_UK][2] = 127, + [1][1][2][0][RTW89_FCC][3] = 127, + [1][1][2][0][RTW89_ETSI][3] = 127, + [1][1][2][0][RTW89_MKK][3] = 127, + [1][1][2][0][RTW89_IC][3] = 127, + [1][1][2][0][RTW89_KCC][3] = 127, + [1][1][2][0][RTW89_ACMA][3] = 127, + [1][1][2][0][RTW89_CN][3] = 127, + [1][1][2][0][RTW89_UK][3] = 127, + [1][1][2][0][RTW89_FCC][4] = 127, + [1][1][2][0][RTW89_ETSI][4] = 127, + [1][1][2][0][RTW89_MKK][4] = 127, + [1][1][2][0][RTW89_IC][4] = 127, + [1][1][2][0][RTW89_KCC][4] = 127, + [1][1][2][0][RTW89_ACMA][4] = 127, + [1][1][2][0][RTW89_CN][4] = 127, + [1][1][2][0][RTW89_UK][4] = 127, + [1][1][2][0][RTW89_FCC][5] = 127, + [1][1][2][0][RTW89_ETSI][5] = 127, + [1][1][2][0][RTW89_MKK][5] = 127, + [1][1][2][0][RTW89_IC][5] = 127, + [1][1][2][0][RTW89_KCC][5] = 127, + [1][1][2][0][RTW89_ACMA][5] = 127, + [1][1][2][0][RTW89_CN][5] = 127, + [1][1][2][0][RTW89_UK][5] = 127, + [1][1][2][0][RTW89_FCC][6] = 127, + [1][1][2][0][RTW89_ETSI][6] = 127, + [1][1][2][0][RTW89_MKK][6] = 127, + [1][1][2][0][RTW89_IC][6] = 127, + [1][1][2][0][RTW89_KCC][6] = 127, + [1][1][2][0][RTW89_ACMA][6] = 127, + [1][1][2][0][RTW89_CN][6] = 127, + [1][1][2][0][RTW89_UK][6] = 127, + [1][1][2][0][RTW89_FCC][7] = 127, + [1][1][2][0][RTW89_ETSI][7] = 127, + [1][1][2][0][RTW89_MKK][7] = 127, + [1][1][2][0][RTW89_IC][7] = 127, + [1][1][2][0][RTW89_KCC][7] = 127, + [1][1][2][0][RTW89_ACMA][7] = 127, + [1][1][2][0][RTW89_CN][7] = 127, + [1][1][2][0][RTW89_UK][7] = 127, + [1][1][2][0][RTW89_FCC][8] = 127, + [1][1][2][0][RTW89_ETSI][8] = 127, + [1][1][2][0][RTW89_MKK][8] = 127, + [1][1][2][0][RTW89_IC][8] = 127, + [1][1][2][0][RTW89_KCC][8] = 127, + [1][1][2][0][RTW89_ACMA][8] = 127, + [1][1][2][0][RTW89_CN][8] = 127, + [1][1][2][0][RTW89_UK][8] = 127, + [1][1][2][0][RTW89_FCC][9] = 127, + [1][1][2][0][RTW89_ETSI][9] = 127, + [1][1][2][0][RTW89_MKK][9] = 127, + [1][1][2][0][RTW89_IC][9] = 127, + [1][1][2][0][RTW89_KCC][9] = 127, + [1][1][2][0][RTW89_ACMA][9] = 127, + [1][1][2][0][RTW89_CN][9] = 127, + [1][1][2][0][RTW89_UK][9] = 127, + [1][1][2][0][RTW89_FCC][10] = 127, + [1][1][2][0][RTW89_ETSI][10] = 127, + [1][1][2][0][RTW89_MKK][10] = 127, + [1][1][2][0][RTW89_IC][10] = 127, + [1][1][2][0][RTW89_KCC][10] = 127, + [1][1][2][0][RTW89_ACMA][10] = 127, + [1][1][2][0][RTW89_CN][10] = 127, + [1][1][2][0][RTW89_UK][10] = 127, + [1][1][2][0][RTW89_FCC][11] = 127, + [1][1][2][0][RTW89_ETSI][11] = 127, + [1][1][2][0][RTW89_MKK][11] = 127, + [1][1][2][0][RTW89_IC][11] = 127, + [1][1][2][0][RTW89_KCC][11] = 127, + [1][1][2][0][RTW89_ACMA][11] = 127, + [1][1][2][0][RTW89_CN][11] = 127, + [1][1][2][0][RTW89_UK][11] = 127, + [1][1][2][0][RTW89_FCC][12] = 127, + [1][1][2][0][RTW89_ETSI][12] = 127, + [1][1][2][0][RTW89_MKK][12] = 127, + [1][1][2][0][RTW89_IC][12] = 127, + [1][1][2][0][RTW89_KCC][12] = 127, + [1][1][2][0][RTW89_ACMA][12] = 127, + [1][1][2][0][RTW89_CN][12] = 127, + [1][1][2][0][RTW89_UK][12] = 127, + [1][1][2][0][RTW89_FCC][13] = 127, + [1][1][2][0][RTW89_ETSI][13] = 127, + [1][1][2][0][RTW89_MKK][13] = 127, + [1][1][2][0][RTW89_IC][13] = 127, + [1][1][2][0][RTW89_KCC][13] = 127, + [1][1][2][0][RTW89_ACMA][13] = 127, + [1][1][2][0][RTW89_CN][13] = 127, + [1][1][2][0][RTW89_UK][13] = 127, + [1][1][2][1][RTW89_FCC][0] = 127, + [1][1][2][1][RTW89_ETSI][0] = 127, + [1][1][2][1][RTW89_MKK][0] = 127, + [1][1][2][1][RTW89_IC][0] = 127, + [1][1][2][1][RTW89_KCC][0] = 127, + [1][1][2][1][RTW89_ACMA][0] = 127, + [1][1][2][1][RTW89_CN][0] = 127, + [1][1][2][1][RTW89_UK][0] = 127, + [1][1][2][1][RTW89_FCC][1] = 127, + [1][1][2][1][RTW89_ETSI][1] = 127, + [1][1][2][1][RTW89_MKK][1] = 127, + [1][1][2][1][RTW89_IC][1] = 127, + [1][1][2][1][RTW89_KCC][1] = 127, + [1][1][2][1][RTW89_ACMA][1] = 127, + [1][1][2][1][RTW89_CN][1] = 127, + [1][1][2][1][RTW89_UK][1] = 127, + [1][1][2][1][RTW89_FCC][2] = 127, + [1][1][2][1][RTW89_ETSI][2] = 127, + [1][1][2][1][RTW89_MKK][2] = 127, + [1][1][2][1][RTW89_IC][2] = 127, + [1][1][2][1][RTW89_KCC][2] = 127, + [1][1][2][1][RTW89_ACMA][2] = 127, + [1][1][2][1][RTW89_CN][2] = 127, + [1][1][2][1][RTW89_UK][2] = 127, + [1][1][2][1][RTW89_FCC][3] = 127, + [1][1][2][1][RTW89_ETSI][3] = 127, + [1][1][2][1][RTW89_MKK][3] = 127, + [1][1][2][1][RTW89_IC][3] = 127, + [1][1][2][1][RTW89_KCC][3] = 127, + [1][1][2][1][RTW89_ACMA][3] = 127, + [1][1][2][1][RTW89_CN][3] = 127, + [1][1][2][1][RTW89_UK][3] = 127, + [1][1][2][1][RTW89_FCC][4] = 127, + [1][1][2][1][RTW89_ETSI][4] = 127, + [1][1][2][1][RTW89_MKK][4] = 127, + [1][1][2][1][RTW89_IC][4] = 127, + [1][1][2][1][RTW89_KCC][4] = 127, + [1][1][2][1][RTW89_ACMA][4] = 127, + [1][1][2][1][RTW89_CN][4] = 127, + [1][1][2][1][RTW89_UK][4] = 127, + [1][1][2][1][RTW89_FCC][5] = 127, + [1][1][2][1][RTW89_ETSI][5] = 127, + [1][1][2][1][RTW89_MKK][5] = 127, + [1][1][2][1][RTW89_IC][5] = 127, + [1][1][2][1][RTW89_KCC][5] = 127, + [1][1][2][1][RTW89_ACMA][5] = 127, + [1][1][2][1][RTW89_CN][5] = 127, + [1][1][2][1][RTW89_UK][5] = 127, + [1][1][2][1][RTW89_FCC][6] = 127, + [1][1][2][1][RTW89_ETSI][6] = 127, + [1][1][2][1][RTW89_MKK][6] = 127, + [1][1][2][1][RTW89_IC][6] = 127, + [1][1][2][1][RTW89_KCC][6] = 127, + [1][1][2][1][RTW89_ACMA][6] = 127, + [1][1][2][1][RTW89_CN][6] = 127, + [1][1][2][1][RTW89_UK][6] = 127, + [1][1][2][1][RTW89_FCC][7] = 127, + [1][1][2][1][RTW89_ETSI][7] = 127, + [1][1][2][1][RTW89_MKK][7] = 127, + [1][1][2][1][RTW89_IC][7] = 127, + [1][1][2][1][RTW89_KCC][7] = 127, + [1][1][2][1][RTW89_ACMA][7] = 127, + [1][1][2][1][RTW89_CN][7] = 127, + [1][1][2][1][RTW89_UK][7] = 127, + [1][1][2][1][RTW89_FCC][8] = 127, + [1][1][2][1][RTW89_ETSI][8] = 127, + [1][1][2][1][RTW89_MKK][8] = 127, + [1][1][2][1][RTW89_IC][8] = 127, + [1][1][2][1][RTW89_KCC][8] = 127, + [1][1][2][1][RTW89_ACMA][8] = 127, + [1][1][2][1][RTW89_CN][8] = 127, + [1][1][2][1][RTW89_UK][8] = 127, + [1][1][2][1][RTW89_FCC][9] = 127, + [1][1][2][1][RTW89_ETSI][9] = 127, + [1][1][2][1][RTW89_MKK][9] = 127, + [1][1][2][1][RTW89_IC][9] = 127, + [1][1][2][1][RTW89_KCC][9] = 127, + [1][1][2][1][RTW89_ACMA][9] = 127, + [1][1][2][1][RTW89_CN][9] = 127, + [1][1][2][1][RTW89_UK][9] = 127, + [1][1][2][1][RTW89_FCC][10] = 127, + [1][1][2][1][RTW89_ETSI][10] = 127, + [1][1][2][1][RTW89_MKK][10] = 127, + [1][1][2][1][RTW89_IC][10] = 127, + [1][1][2][1][RTW89_KCC][10] = 127, + [1][1][2][1][RTW89_ACMA][10] = 127, + [1][1][2][1][RTW89_CN][10] = 127, + [1][1][2][1][RTW89_UK][10] = 127, + [1][1][2][1][RTW89_FCC][11] = 127, + [1][1][2][1][RTW89_ETSI][11] = 127, + [1][1][2][1][RTW89_MKK][11] = 127, + [1][1][2][1][RTW89_IC][11] = 127, + [1][1][2][1][RTW89_KCC][11] = 127, + [1][1][2][1][RTW89_ACMA][11] = 127, + [1][1][2][1][RTW89_CN][11] = 127, + [1][1][2][1][RTW89_UK][11] = 127, + [1][1][2][1][RTW89_FCC][12] = 127, + [1][1][2][1][RTW89_ETSI][12] = 127, + [1][1][2][1][RTW89_MKK][12] = 127, + [1][1][2][1][RTW89_IC][12] = 127, + [1][1][2][1][RTW89_KCC][12] = 127, + [1][1][2][1][RTW89_ACMA][12] = 127, + [1][1][2][1][RTW89_CN][12] = 127, + [1][1][2][1][RTW89_UK][12] = 127, + [1][1][2][1][RTW89_FCC][13] = 127, + [1][1][2][1][RTW89_ETSI][13] = 127, + [1][1][2][1][RTW89_MKK][13] = 127, + [1][1][2][1][RTW89_IC][13] = 127, + [1][1][2][1][RTW89_KCC][13] = 127, + [1][1][2][1][RTW89_ACMA][13] = 127, + [1][1][2][1][RTW89_CN][13] = 127, + [1][1][2][1][RTW89_UK][13] = 127, +}; + +static +const s8 rtw89_8851b_txpwr_lmt_5g[RTW89_5G_BW_NUM][RTW89_NTX_NUM] + [RTW89_RS_LMT_NUM][RTW89_BF_NUM] + [RTW89_REGD_NUM][RTW89_5G_CH_NUM] = { + [0][0][1][0][RTW89_WW][0] = 58, + [0][0][1][0][RTW89_WW][2] = 58, + [0][0][1][0][RTW89_WW][4] = 58, + [0][0][1][0][RTW89_WW][6] = 50, + [0][0][1][0][RTW89_WW][8] = 58, + [0][0][1][0][RTW89_WW][10] = 58, + [0][0][1][0][RTW89_WW][12] = 58, + [0][0][1][0][RTW89_WW][14] = 58, + [0][0][1][0][RTW89_WW][15] = 58, + [0][0][1][0][RTW89_WW][17] = 60, + [0][0][1][0][RTW89_WW][19] = 60, + [0][0][1][0][RTW89_WW][21] = 60, + [0][0][1][0][RTW89_WW][23] = 60, + [0][0][1][0][RTW89_WW][25] = 60, + [0][0][1][0][RTW89_WW][27] = 60, + [0][0][1][0][RTW89_WW][29] = 60, + [0][0][1][0][RTW89_WW][31] = 60, + [0][0][1][0][RTW89_WW][33] = 60, + [0][0][1][0][RTW89_WW][35] = 60, + [0][0][1][0][RTW89_WW][37] = 74, + [0][0][1][0][RTW89_WW][38] = 30, + [0][0][1][0][RTW89_WW][40] = 30, + [0][0][1][0][RTW89_WW][42] = 30, + [0][0][1][0][RTW89_WW][44] = 30, + [0][0][1][0][RTW89_WW][46] = 30, + [0][0][1][0][RTW89_WW][48] = 72, + [0][0][1][0][RTW89_WW][50] = 72, + [0][0][1][0][RTW89_WW][52] = 72, + [0][1][1][0][RTW89_WW][0] = 0, + [0][1][1][0][RTW89_WW][2] = 0, + [0][1][1][0][RTW89_WW][4] = 0, + [0][1][1][0][RTW89_WW][6] = 0, + [0][1][1][0][RTW89_WW][8] = 0, + [0][1][1][0][RTW89_WW][10] = 0, + [0][1][1][0][RTW89_WW][12] = 0, + [0][1][1][0][RTW89_WW][14] = 0, + [0][1][1][0][RTW89_WW][15] = 0, + [0][1][1][0][RTW89_WW][17] = 0, + [0][1][1][0][RTW89_WW][19] = 0, + [0][1][1][0][RTW89_WW][21] = 0, + [0][1][1][0][RTW89_WW][23] = 0, + [0][1][1][0][RTW89_WW][25] = 0, + [0][1][1][0][RTW89_WW][27] = 0, + [0][1][1][0][RTW89_WW][29] = 0, + [0][1][1][0][RTW89_WW][31] = 0, + [0][1][1][0][RTW89_WW][33] = 0, + [0][1][1][0][RTW89_WW][35] = 0, + [0][1][1][0][RTW89_WW][37] = 0, + [0][1][1][0][RTW89_WW][38] = 0, + [0][1][1][0][RTW89_WW][40] = 0, + [0][1][1][0][RTW89_WW][42] = 0, + [0][1][1][0][RTW89_WW][44] = 0, + [0][1][1][0][RTW89_WW][46] = 0, + [0][1][1][0][RTW89_WW][48] = 0, + [0][1][1][0][RTW89_WW][50] = 0, + [0][1][1][0][RTW89_WW][52] = 0, + [0][0][2][0][RTW89_WW][0] = 62, + [0][0][2][0][RTW89_WW][2] = 62, + [0][0][2][0][RTW89_WW][4] = 62, + [0][0][2][0][RTW89_WW][6] = 54, + [0][0][2][0][RTW89_WW][8] = 62, + [0][0][2][0][RTW89_WW][10] = 62, + [0][0][2][0][RTW89_WW][12] = 62, + [0][0][2][0][RTW89_WW][14] = 62, + [0][0][2][0][RTW89_WW][15] = 60, + [0][0][2][0][RTW89_WW][17] = 62, + [0][0][2][0][RTW89_WW][19] = 62, + [0][0][2][0][RTW89_WW][21] = 62, + [0][0][2][0][RTW89_WW][23] = 62, + [0][0][2][0][RTW89_WW][25] = 62, + [0][0][2][0][RTW89_WW][27] = 62, + [0][0][2][0][RTW89_WW][29] = 62, + [0][0][2][0][RTW89_WW][31] = 62, + [0][0][2][0][RTW89_WW][33] = 62, + [0][0][2][0][RTW89_WW][35] = 62, + [0][0][2][0][RTW89_WW][37] = 76, + [0][0][2][0][RTW89_WW][38] = 30, + [0][0][2][0][RTW89_WW][40] = 30, + [0][0][2][0][RTW89_WW][42] = 30, + [0][0][2][0][RTW89_WW][44] = 30, + [0][0][2][0][RTW89_WW][46] = 30, + [0][0][2][0][RTW89_WW][48] = 74, + [0][0][2][0][RTW89_WW][50] = 76, + [0][0][2][0][RTW89_WW][52] = 76, + [0][1][2][0][RTW89_WW][0] = 0, + [0][1][2][0][RTW89_WW][2] = 0, + [0][1][2][0][RTW89_WW][4] = 0, + [0][1][2][0][RTW89_WW][6] = 0, + [0][1][2][0][RTW89_WW][8] = 0, + [0][1][2][0][RTW89_WW][10] = 0, + [0][1][2][0][RTW89_WW][12] = 0, + [0][1][2][0][RTW89_WW][14] = 0, + [0][1][2][0][RTW89_WW][15] = 0, + [0][1][2][0][RTW89_WW][17] = 0, + [0][1][2][0][RTW89_WW][19] = 0, + [0][1][2][0][RTW89_WW][21] = 0, + [0][1][2][0][RTW89_WW][23] = 0, + [0][1][2][0][RTW89_WW][25] = 0, + [0][1][2][0][RTW89_WW][27] = 0, + [0][1][2][0][RTW89_WW][29] = 0, + [0][1][2][0][RTW89_WW][31] = 0, + [0][1][2][0][RTW89_WW][33] = 0, + [0][1][2][0][RTW89_WW][35] = 0, + [0][1][2][0][RTW89_WW][37] = 0, + [0][1][2][0][RTW89_WW][38] = 0, + [0][1][2][0][RTW89_WW][40] = 0, + [0][1][2][0][RTW89_WW][42] = 0, + [0][1][2][0][RTW89_WW][44] = 0, + [0][1][2][0][RTW89_WW][46] = 0, + [0][1][2][0][RTW89_WW][48] = 0, + [0][1][2][0][RTW89_WW][50] = 0, + [0][1][2][0][RTW89_WW][52] = 0, + [0][1][2][1][RTW89_WW][0] = 0, + [0][1][2][1][RTW89_WW][2] = 0, + [0][1][2][1][RTW89_WW][4] = 0, + [0][1][2][1][RTW89_WW][6] = 0, + [0][1][2][1][RTW89_WW][8] = 0, + [0][1][2][1][RTW89_WW][10] = 0, + [0][1][2][1][RTW89_WW][12] = 0, + [0][1][2][1][RTW89_WW][14] = 0, + [0][1][2][1][RTW89_WW][15] = 0, + [0][1][2][1][RTW89_WW][17] = 0, + [0][1][2][1][RTW89_WW][19] = 0, + [0][1][2][1][RTW89_WW][21] = 0, + [0][1][2][1][RTW89_WW][23] = 0, + [0][1][2][1][RTW89_WW][25] = 0, + [0][1][2][1][RTW89_WW][27] = 0, + [0][1][2][1][RTW89_WW][29] = 0, + [0][1][2][1][RTW89_WW][31] = 0, + [0][1][2][1][RTW89_WW][33] = 0, + [0][1][2][1][RTW89_WW][35] = 0, + [0][1][2][1][RTW89_WW][37] = 0, + [0][1][2][1][RTW89_WW][38] = 0, + [0][1][2][1][RTW89_WW][40] = 0, + [0][1][2][1][RTW89_WW][42] = 0, + [0][1][2][1][RTW89_WW][44] = 0, + [0][1][2][1][RTW89_WW][46] = 0, + [0][1][2][1][RTW89_WW][48] = 0, + [0][1][2][1][RTW89_WW][50] = 0, + [0][1][2][1][RTW89_WW][52] = 0, + [1][0][2][0][RTW89_WW][1] = 64, + [1][0][2][0][RTW89_WW][5] = 62, + [1][0][2][0][RTW89_WW][9] = 64, + [1][0][2][0][RTW89_WW][13] = 64, + [1][0][2][0][RTW89_WW][16] = 66, + [1][0][2][0][RTW89_WW][20] = 66, + [1][0][2][0][RTW89_WW][24] = 66, + [1][0][2][0][RTW89_WW][28] = 66, + [1][0][2][0][RTW89_WW][32] = 66, + [1][0][2][0][RTW89_WW][36] = 76, + [1][0][2][0][RTW89_WW][39] = 30, + [1][0][2][0][RTW89_WW][43] = 30, + [1][0][2][0][RTW89_WW][47] = 84, + [1][0][2][0][RTW89_WW][51] = 84, + [1][1][2][0][RTW89_WW][1] = 0, + [1][1][2][0][RTW89_WW][5] = 0, + [1][1][2][0][RTW89_WW][9] = 0, + [1][1][2][0][RTW89_WW][13] = 0, + [1][1][2][0][RTW89_WW][16] = 0, + [1][1][2][0][RTW89_WW][20] = 0, + [1][1][2][0][RTW89_WW][24] = 0, + [1][1][2][0][RTW89_WW][28] = 0, + [1][1][2][0][RTW89_WW][32] = 0, + [1][1][2][0][RTW89_WW][36] = 0, + [1][1][2][0][RTW89_WW][39] = 0, + [1][1][2][0][RTW89_WW][43] = 0, + [1][1][2][0][RTW89_WW][47] = 0, + [1][1][2][0][RTW89_WW][51] = 0, + [1][1][2][1][RTW89_WW][1] = 0, + [1][1][2][1][RTW89_WW][5] = 0, + [1][1][2][1][RTW89_WW][9] = 0, + [1][1][2][1][RTW89_WW][13] = 0, + [1][1][2][1][RTW89_WW][16] = 0, + [1][1][2][1][RTW89_WW][20] = 0, + [1][1][2][1][RTW89_WW][24] = 0, + [1][1][2][1][RTW89_WW][28] = 0, + [1][1][2][1][RTW89_WW][32] = 0, + [1][1][2][1][RTW89_WW][36] = 0, + [1][1][2][1][RTW89_WW][39] = 0, + [1][1][2][1][RTW89_WW][43] = 0, + [1][1][2][1][RTW89_WW][47] = 0, + [1][1][2][1][RTW89_WW][51] = 0, + [2][0][2][0][RTW89_WW][3] = 62, + [2][0][2][0][RTW89_WW][11] = 62, + [2][0][2][0][RTW89_WW][18] = 64, + [2][0][2][0][RTW89_WW][26] = 64, + [2][0][2][0][RTW89_WW][34] = 72, + [2][0][2][0][RTW89_WW][41] = 30, + [2][0][2][0][RTW89_WW][49] = 74, + [2][1][2][0][RTW89_WW][3] = 0, + [2][1][2][0][RTW89_WW][11] = 0, + [2][1][2][0][RTW89_WW][18] = 0, + [2][1][2][0][RTW89_WW][26] = 0, + [2][1][2][0][RTW89_WW][34] = 0, + [2][1][2][0][RTW89_WW][41] = 0, + [2][1][2][0][RTW89_WW][49] = 0, + [2][1][2][1][RTW89_WW][3] = 0, + [2][1][2][1][RTW89_WW][11] = 0, + [2][1][2][1][RTW89_WW][18] = 0, + [2][1][2][1][RTW89_WW][26] = 0, + [2][1][2][1][RTW89_WW][34] = 0, + [2][1][2][1][RTW89_WW][41] = 0, + [2][1][2][1][RTW89_WW][49] = 0, + [3][0][2][0][RTW89_WW][7] = 58, + [3][0][2][0][RTW89_WW][22] = 58, + [3][0][2][0][RTW89_WW][45] = 0, + [3][1][2][0][RTW89_WW][7] = 0, + [3][1][2][0][RTW89_WW][22] = 0, + [3][1][2][0][RTW89_WW][45] = 0, + [3][1][2][1][RTW89_WW][7] = 0, + [3][1][2][1][RTW89_WW][22] = 0, + [3][1][2][1][RTW89_WW][45] = 0, + [0][0][1][0][RTW89_FCC][0] = 80, + [0][0][1][0][RTW89_ETSI][0] = 58, + [0][0][1][0][RTW89_MKK][0] = 60, + [0][0][1][0][RTW89_IC][0] = 62, + [0][0][1][0][RTW89_KCC][0] = 74, + [0][0][1][0][RTW89_ACMA][0] = 58, + [0][0][1][0][RTW89_CN][0] = 60, + [0][0][1][0][RTW89_UK][0] = 58, + [0][0][1][0][RTW89_FCC][2] = 82, + [0][0][1][0][RTW89_ETSI][2] = 58, + [0][0][1][0][RTW89_MKK][2] = 60, + [0][0][1][0][RTW89_IC][2] = 62, + [0][0][1][0][RTW89_KCC][2] = 74, + [0][0][1][0][RTW89_ACMA][2] = 58, + [0][0][1][0][RTW89_CN][2] = 60, + [0][0][1][0][RTW89_UK][2] = 58, + [0][0][1][0][RTW89_FCC][4] = 82, + [0][0][1][0][RTW89_ETSI][4] = 58, + [0][0][1][0][RTW89_MKK][4] = 60, + [0][0][1][0][RTW89_IC][4] = 62, + [0][0][1][0][RTW89_KCC][4] = 74, + [0][0][1][0][RTW89_ACMA][4] = 58, + [0][0][1][0][RTW89_CN][4] = 60, + [0][0][1][0][RTW89_UK][4] = 58, + [0][0][1][0][RTW89_FCC][6] = 82, + [0][0][1][0][RTW89_ETSI][6] = 58, + [0][0][1][0][RTW89_MKK][6] = 60, + [0][0][1][0][RTW89_IC][6] = 62, + [0][0][1][0][RTW89_KCC][6] = 50, + [0][0][1][0][RTW89_ACMA][6] = 58, + [0][0][1][0][RTW89_CN][6] = 60, + [0][0][1][0][RTW89_UK][6] = 58, + [0][0][1][0][RTW89_FCC][8] = 82, + [0][0][1][0][RTW89_ETSI][8] = 58, + [0][0][1][0][RTW89_MKK][8] = 60, + [0][0][1][0][RTW89_IC][8] = 64, + [0][0][1][0][RTW89_KCC][8] = 74, + [0][0][1][0][RTW89_ACMA][8] = 58, + [0][0][1][0][RTW89_CN][8] = 60, + [0][0][1][0][RTW89_UK][8] = 58, + [0][0][1][0][RTW89_FCC][10] = 82, + [0][0][1][0][RTW89_ETSI][10] = 58, + [0][0][1][0][RTW89_MKK][10] = 60, + [0][0][1][0][RTW89_IC][10] = 64, + [0][0][1][0][RTW89_KCC][10] = 74, + [0][0][1][0][RTW89_ACMA][10] = 58, + [0][0][1][0][RTW89_CN][10] = 60, + [0][0][1][0][RTW89_UK][10] = 58, + [0][0][1][0][RTW89_FCC][12] = 82, + [0][0][1][0][RTW89_ETSI][12] = 58, + [0][0][1][0][RTW89_MKK][12] = 60, + [0][0][1][0][RTW89_IC][12] = 64, + [0][0][1][0][RTW89_KCC][12] = 76, + [0][0][1][0][RTW89_ACMA][12] = 58, + [0][0][1][0][RTW89_CN][12] = 60, + [0][0][1][0][RTW89_UK][12] = 58, + [0][0][1][0][RTW89_FCC][14] = 78, + [0][0][1][0][RTW89_ETSI][14] = 58, + [0][0][1][0][RTW89_MKK][14] = 60, + [0][0][1][0][RTW89_IC][14] = 64, + [0][0][1][0][RTW89_KCC][14] = 76, + [0][0][1][0][RTW89_ACMA][14] = 58, + [0][0][1][0][RTW89_CN][14] = 60, + [0][0][1][0][RTW89_UK][14] = 58, + [0][0][1][0][RTW89_FCC][15] = 78, + [0][0][1][0][RTW89_ETSI][15] = 58, + [0][0][1][0][RTW89_MKK][15] = 78, + [0][0][1][0][RTW89_IC][15] = 78, + [0][0][1][0][RTW89_KCC][15] = 78, + [0][0][1][0][RTW89_ACMA][15] = 58, + [0][0][1][0][RTW89_CN][15] = 127, + [0][0][1][0][RTW89_UK][15] = 58, + [0][0][1][0][RTW89_FCC][17] = 82, + [0][0][1][0][RTW89_ETSI][17] = 60, + [0][0][1][0][RTW89_MKK][17] = 78, + [0][0][1][0][RTW89_IC][17] = 82, + [0][0][1][0][RTW89_KCC][17] = 78, + [0][0][1][0][RTW89_ACMA][17] = 60, + [0][0][1][0][RTW89_CN][17] = 127, + [0][0][1][0][RTW89_UK][17] = 60, + [0][0][1][0][RTW89_FCC][19] = 82, + [0][0][1][0][RTW89_ETSI][19] = 60, + [0][0][1][0][RTW89_MKK][19] = 78, + [0][0][1][0][RTW89_IC][19] = 82, + [0][0][1][0][RTW89_KCC][19] = 78, + [0][0][1][0][RTW89_ACMA][19] = 60, + [0][0][1][0][RTW89_CN][19] = 127, + [0][0][1][0][RTW89_UK][19] = 60, + [0][0][1][0][RTW89_FCC][21] = 82, + [0][0][1][0][RTW89_ETSI][21] = 60, + [0][0][1][0][RTW89_MKK][21] = 78, + [0][0][1][0][RTW89_IC][21] = 82, + [0][0][1][0][RTW89_KCC][21] = 78, + [0][0][1][0][RTW89_ACMA][21] = 60, + [0][0][1][0][RTW89_CN][21] = 127, + [0][0][1][0][RTW89_UK][21] = 60, + [0][0][1][0][RTW89_FCC][23] = 82, + [0][0][1][0][RTW89_ETSI][23] = 60, + [0][0][1][0][RTW89_MKK][23] = 78, + [0][0][1][0][RTW89_IC][23] = 82, + [0][0][1][0][RTW89_KCC][23] = 78, + [0][0][1][0][RTW89_ACMA][23] = 60, + [0][0][1][0][RTW89_CN][23] = 127, + [0][0][1][0][RTW89_UK][23] = 60, + [0][0][1][0][RTW89_FCC][25] = 82, + [0][0][1][0][RTW89_ETSI][25] = 60, + [0][0][1][0][RTW89_MKK][25] = 78, + [0][0][1][0][RTW89_IC][25] = 127, + [0][0][1][0][RTW89_KCC][25] = 78, + [0][0][1][0][RTW89_ACMA][25] = 127, + [0][0][1][0][RTW89_CN][25] = 127, + [0][0][1][0][RTW89_UK][25] = 60, + [0][0][1][0][RTW89_FCC][27] = 82, + [0][0][1][0][RTW89_ETSI][27] = 60, + [0][0][1][0][RTW89_MKK][27] = 78, + [0][0][1][0][RTW89_IC][27] = 127, + [0][0][1][0][RTW89_KCC][27] = 78, + [0][0][1][0][RTW89_ACMA][27] = 127, + [0][0][1][0][RTW89_CN][27] = 127, + [0][0][1][0][RTW89_UK][27] = 60, + [0][0][1][0][RTW89_FCC][29] = 82, + [0][0][1][0][RTW89_ETSI][29] = 60, + [0][0][1][0][RTW89_MKK][29] = 78, + [0][0][1][0][RTW89_IC][29] = 127, + [0][0][1][0][RTW89_KCC][29] = 78, + [0][0][1][0][RTW89_ACMA][29] = 127, + [0][0][1][0][RTW89_CN][29] = 127, + [0][0][1][0][RTW89_UK][29] = 60, + [0][0][1][0][RTW89_FCC][31] = 82, + [0][0][1][0][RTW89_ETSI][31] = 60, + [0][0][1][0][RTW89_MKK][31] = 78, + [0][0][1][0][RTW89_IC][31] = 82, + [0][0][1][0][RTW89_KCC][31] = 74, + [0][0][1][0][RTW89_ACMA][31] = 60, + [0][0][1][0][RTW89_CN][31] = 127, + [0][0][1][0][RTW89_UK][31] = 60, + [0][0][1][0][RTW89_FCC][33] = 82, + [0][0][1][0][RTW89_ETSI][33] = 60, + [0][0][1][0][RTW89_MKK][33] = 78, + [0][0][1][0][RTW89_IC][33] = 82, + [0][0][1][0][RTW89_KCC][33] = 74, + [0][0][1][0][RTW89_ACMA][33] = 60, + [0][0][1][0][RTW89_CN][33] = 127, + [0][0][1][0][RTW89_UK][33] = 60, + [0][0][1][0][RTW89_FCC][35] = 72, + [0][0][1][0][RTW89_ETSI][35] = 60, + [0][0][1][0][RTW89_MKK][35] = 78, + [0][0][1][0][RTW89_IC][35] = 72, + [0][0][1][0][RTW89_KCC][35] = 74, + [0][0][1][0][RTW89_ACMA][35] = 60, + [0][0][1][0][RTW89_CN][35] = 127, + [0][0][1][0][RTW89_UK][35] = 60, + [0][0][1][0][RTW89_FCC][37] = 82, + [0][0][1][0][RTW89_ETSI][37] = 127, + [0][0][1][0][RTW89_MKK][37] = 78, + [0][0][1][0][RTW89_IC][37] = 82, + [0][0][1][0][RTW89_KCC][37] = 74, + [0][0][1][0][RTW89_ACMA][37] = 78, + [0][0][1][0][RTW89_CN][37] = 127, + [0][0][1][0][RTW89_UK][37] = 78, + [0][0][1][0][RTW89_FCC][38] = 82, + [0][0][1][0][RTW89_ETSI][38] = 30, + [0][0][1][0][RTW89_MKK][38] = 127, + [0][0][1][0][RTW89_IC][38] = 82, + [0][0][1][0][RTW89_KCC][38] = 70, + [0][0][1][0][RTW89_ACMA][38] = 78, + [0][0][1][0][RTW89_CN][38] = 78, + [0][0][1][0][RTW89_UK][38] = 58, + [0][0][1][0][RTW89_FCC][40] = 82, + [0][0][1][0][RTW89_ETSI][40] = 30, + [0][0][1][0][RTW89_MKK][40] = 127, + [0][0][1][0][RTW89_IC][40] = 82, + [0][0][1][0][RTW89_KCC][40] = 76, + [0][0][1][0][RTW89_ACMA][40] = 78, + [0][0][1][0][RTW89_CN][40] = 78, + [0][0][1][0][RTW89_UK][40] = 58, + [0][0][1][0][RTW89_FCC][42] = 82, + [0][0][1][0][RTW89_ETSI][42] = 30, + [0][0][1][0][RTW89_MKK][42] = 127, + [0][0][1][0][RTW89_IC][42] = 82, + [0][0][1][0][RTW89_KCC][42] = 76, + [0][0][1][0][RTW89_ACMA][42] = 78, + [0][0][1][0][RTW89_CN][42] = 78, + [0][0][1][0][RTW89_UK][42] = 58, + [0][0][1][0][RTW89_FCC][44] = 82, + [0][0][1][0][RTW89_ETSI][44] = 30, + [0][0][1][0][RTW89_MKK][44] = 127, + [0][0][1][0][RTW89_IC][44] = 82, + [0][0][1][0][RTW89_KCC][44] = 76, + [0][0][1][0][RTW89_ACMA][44] = 78, + [0][0][1][0][RTW89_CN][44] = 78, + [0][0][1][0][RTW89_UK][44] = 58, + [0][0][1][0][RTW89_FCC][46] = 82, + [0][0][1][0][RTW89_ETSI][46] = 30, + [0][0][1][0][RTW89_MKK][46] = 127, + [0][0][1][0][RTW89_IC][46] = 82, + [0][0][1][0][RTW89_KCC][46] = 76, + [0][0][1][0][RTW89_ACMA][46] = 78, + [0][0][1][0][RTW89_CN][46] = 78, + [0][0][1][0][RTW89_UK][46] = 58, + [0][0][1][0][RTW89_FCC][48] = 72, + [0][0][1][0][RTW89_ETSI][48] = 127, + [0][0][1][0][RTW89_MKK][48] = 127, + [0][0][1][0][RTW89_IC][48] = 127, + [0][0][1][0][RTW89_KCC][48] = 127, + [0][0][1][0][RTW89_ACMA][48] = 127, + [0][0][1][0][RTW89_CN][48] = 127, + [0][0][1][0][RTW89_UK][48] = 127, + [0][0][1][0][RTW89_FCC][50] = 72, + [0][0][1][0][RTW89_ETSI][50] = 127, + [0][0][1][0][RTW89_MKK][50] = 127, + [0][0][1][0][RTW89_IC][50] = 127, + [0][0][1][0][RTW89_KCC][50] = 127, + [0][0][1][0][RTW89_ACMA][50] = 127, + [0][0][1][0][RTW89_CN][50] = 127, + [0][0][1][0][RTW89_UK][50] = 127, + [0][0][1][0][RTW89_FCC][52] = 72, + [0][0][1][0][RTW89_ETSI][52] = 127, + [0][0][1][0][RTW89_MKK][52] = 127, + [0][0][1][0][RTW89_IC][52] = 127, + [0][0][1][0][RTW89_KCC][52] = 127, + [0][0][1][0][RTW89_ACMA][52] = 127, + [0][0][1][0][RTW89_CN][52] = 127, + [0][0][1][0][RTW89_UK][52] = 127, + [0][1][1][0][RTW89_FCC][0] = 127, + [0][1][1][0][RTW89_ETSI][0] = 127, + [0][1][1][0][RTW89_MKK][0] = 127, + [0][1][1][0][RTW89_IC][0] = 127, + [0][1][1][0][RTW89_KCC][0] = 127, + [0][1][1][0][RTW89_ACMA][0] = 127, + [0][1][1][0][RTW89_CN][0] = 127, + [0][1][1][0][RTW89_UK][0] = 127, + [0][1][1][0][RTW89_FCC][2] = 127, + [0][1][1][0][RTW89_ETSI][2] = 127, + [0][1][1][0][RTW89_MKK][2] = 127, + [0][1][1][0][RTW89_IC][2] = 127, + [0][1][1][0][RTW89_KCC][2] = 127, + [0][1][1][0][RTW89_ACMA][2] = 127, + [0][1][1][0][RTW89_CN][2] = 127, + [0][1][1][0][RTW89_UK][2] = 127, + [0][1][1][0][RTW89_FCC][4] = 127, + [0][1][1][0][RTW89_ETSI][4] = 127, + [0][1][1][0][RTW89_MKK][4] = 127, + [0][1][1][0][RTW89_IC][4] = 127, + [0][1][1][0][RTW89_KCC][4] = 127, + [0][1][1][0][RTW89_ACMA][4] = 127, + [0][1][1][0][RTW89_CN][4] = 127, + [0][1][1][0][RTW89_UK][4] = 127, + [0][1][1][0][RTW89_FCC][6] = 127, + [0][1][1][0][RTW89_ETSI][6] = 127, + [0][1][1][0][RTW89_MKK][6] = 127, + [0][1][1][0][RTW89_IC][6] = 127, + [0][1][1][0][RTW89_KCC][6] = 127, + [0][1][1][0][RTW89_ACMA][6] = 127, + [0][1][1][0][RTW89_CN][6] = 127, + [0][1][1][0][RTW89_UK][6] = 127, + [0][1][1][0][RTW89_FCC][8] = 127, + [0][1][1][0][RTW89_ETSI][8] = 127, + [0][1][1][0][RTW89_MKK][8] = 127, + [0][1][1][0][RTW89_IC][8] = 127, + [0][1][1][0][RTW89_KCC][8] = 127, + [0][1][1][0][RTW89_ACMA][8] = 127, + [0][1][1][0][RTW89_CN][8] = 127, + [0][1][1][0][RTW89_UK][8] = 127, + [0][1][1][0][RTW89_FCC][10] = 127, + [0][1][1][0][RTW89_ETSI][10] = 127, + [0][1][1][0][RTW89_MKK][10] = 127, + [0][1][1][0][RTW89_IC][10] = 127, + [0][1][1][0][RTW89_KCC][10] = 127, + [0][1][1][0][RTW89_ACMA][10] = 127, + [0][1][1][0][RTW89_CN][10] = 127, + [0][1][1][0][RTW89_UK][10] = 127, + [0][1][1][0][RTW89_FCC][12] = 127, + [0][1][1][0][RTW89_ETSI][12] = 127, + [0][1][1][0][RTW89_MKK][12] = 127, + [0][1][1][0][RTW89_IC][12] = 127, + [0][1][1][0][RTW89_KCC][12] = 127, + [0][1][1][0][RTW89_ACMA][12] = 127, + [0][1][1][0][RTW89_CN][12] = 127, + [0][1][1][0][RTW89_UK][12] = 127, + [0][1][1][0][RTW89_FCC][14] = 127, + [0][1][1][0][RTW89_ETSI][14] = 127, + [0][1][1][0][RTW89_MKK][14] = 127, + [0][1][1][0][RTW89_IC][14] = 127, + [0][1][1][0][RTW89_KCC][14] = 127, + [0][1][1][0][RTW89_ACMA][14] = 127, + [0][1][1][0][RTW89_CN][14] = 127, + [0][1][1][0][RTW89_UK][14] = 127, + [0][1][1][0][RTW89_FCC][15] = 127, + [0][1][1][0][RTW89_ETSI][15] = 127, + [0][1][1][0][RTW89_MKK][15] = 127, + [0][1][1][0][RTW89_IC][15] = 127, + [0][1][1][0][RTW89_KCC][15] = 127, + [0][1][1][0][RTW89_ACMA][15] = 127, + [0][1][1][0][RTW89_CN][15] = 127, + [0][1][1][0][RTW89_UK][15] = 127, + [0][1][1][0][RTW89_FCC][17] = 127, + [0][1][1][0][RTW89_ETSI][17] = 127, + [0][1][1][0][RTW89_MKK][17] = 127, + [0][1][1][0][RTW89_IC][17] = 127, + [0][1][1][0][RTW89_KCC][17] = 127, + [0][1][1][0][RTW89_ACMA][17] = 127, + [0][1][1][0][RTW89_CN][17] = 127, + [0][1][1][0][RTW89_UK][17] = 127, + [0][1][1][0][RTW89_FCC][19] = 127, + [0][1][1][0][RTW89_ETSI][19] = 127, + [0][1][1][0][RTW89_MKK][19] = 127, + [0][1][1][0][RTW89_IC][19] = 127, + [0][1][1][0][RTW89_KCC][19] = 127, + [0][1][1][0][RTW89_ACMA][19] = 127, + [0][1][1][0][RTW89_CN][19] = 127, + [0][1][1][0][RTW89_UK][19] = 127, + [0][1][1][0][RTW89_FCC][21] = 127, + [0][1][1][0][RTW89_ETSI][21] = 127, + [0][1][1][0][RTW89_MKK][21] = 127, + [0][1][1][0][RTW89_IC][21] = 127, + [0][1][1][0][RTW89_KCC][21] = 127, + [0][1][1][0][RTW89_ACMA][21] = 127, + [0][1][1][0][RTW89_CN][21] = 127, + [0][1][1][0][RTW89_UK][21] = 127, + [0][1][1][0][RTW89_FCC][23] = 127, + [0][1][1][0][RTW89_ETSI][23] = 127, + [0][1][1][0][RTW89_MKK][23] = 127, + [0][1][1][0][RTW89_IC][23] = 127, + [0][1][1][0][RTW89_KCC][23] = 127, + [0][1][1][0][RTW89_ACMA][23] = 127, + [0][1][1][0][RTW89_CN][23] = 127, + [0][1][1][0][RTW89_UK][23] = 127, + [0][1][1][0][RTW89_FCC][25] = 127, + [0][1][1][0][RTW89_ETSI][25] = 127, + [0][1][1][0][RTW89_MKK][25] = 127, + [0][1][1][0][RTW89_IC][25] = 127, + [0][1][1][0][RTW89_KCC][25] = 127, + [0][1][1][0][RTW89_ACMA][25] = 127, + [0][1][1][0][RTW89_CN][25] = 127, + [0][1][1][0][RTW89_UK][25] = 127, + [0][1][1][0][RTW89_FCC][27] = 127, + [0][1][1][0][RTW89_ETSI][27] = 127, + [0][1][1][0][RTW89_MKK][27] = 127, + [0][1][1][0][RTW89_IC][27] = 127, + [0][1][1][0][RTW89_KCC][27] = 127, + [0][1][1][0][RTW89_ACMA][27] = 127, + [0][1][1][0][RTW89_CN][27] = 127, + [0][1][1][0][RTW89_UK][27] = 127, + [0][1][1][0][RTW89_FCC][29] = 127, + [0][1][1][0][RTW89_ETSI][29] = 127, + [0][1][1][0][RTW89_MKK][29] = 127, + [0][1][1][0][RTW89_IC][29] = 127, + [0][1][1][0][RTW89_KCC][29] = 127, + [0][1][1][0][RTW89_ACMA][29] = 127, + [0][1][1][0][RTW89_CN][29] = 127, + [0][1][1][0][RTW89_UK][29] = 127, + [0][1][1][0][RTW89_FCC][31] = 127, + [0][1][1][0][RTW89_ETSI][31] = 127, + [0][1][1][0][RTW89_MKK][31] = 127, + [0][1][1][0][RTW89_IC][31] = 127, + [0][1][1][0][RTW89_KCC][31] = 127, + [0][1][1][0][RTW89_ACMA][31] = 127, + [0][1][1][0][RTW89_CN][31] = 127, + [0][1][1][0][RTW89_UK][31] = 127, + [0][1][1][0][RTW89_FCC][33] = 127, + [0][1][1][0][RTW89_ETSI][33] = 127, + [0][1][1][0][RTW89_MKK][33] = 127, + [0][1][1][0][RTW89_IC][33] = 127, + [0][1][1][0][RTW89_KCC][33] = 127, + [0][1][1][0][RTW89_ACMA][33] = 127, + [0][1][1][0][RTW89_CN][33] = 127, + [0][1][1][0][RTW89_UK][33] = 127, + [0][1][1][0][RTW89_FCC][35] = 127, + [0][1][1][0][RTW89_ETSI][35] = 127, + [0][1][1][0][RTW89_MKK][35] = 127, + [0][1][1][0][RTW89_IC][35] = 127, + [0][1][1][0][RTW89_KCC][35] = 127, + [0][1][1][0][RTW89_ACMA][35] = 127, + [0][1][1][0][RTW89_CN][35] = 127, + [0][1][1][0][RTW89_UK][35] = 127, + [0][1][1][0][RTW89_FCC][37] = 127, + [0][1][1][0][RTW89_ETSI][37] = 127, + [0][1][1][0][RTW89_MKK][37] = 127, + [0][1][1][0][RTW89_IC][37] = 127, + [0][1][1][0][RTW89_KCC][37] = 127, + [0][1][1][0][RTW89_ACMA][37] = 127, + [0][1][1][0][RTW89_CN][37] = 127, + [0][1][1][0][RTW89_UK][37] = 127, + [0][1][1][0][RTW89_FCC][38] = 127, + [0][1][1][0][RTW89_ETSI][38] = 127, + [0][1][1][0][RTW89_MKK][38] = 127, + [0][1][1][0][RTW89_IC][38] = 127, + [0][1][1][0][RTW89_KCC][38] = 127, + [0][1][1][0][RTW89_ACMA][38] = 127, + [0][1][1][0][RTW89_CN][38] = 127, + [0][1][1][0][RTW89_UK][38] = 127, + [0][1][1][0][RTW89_FCC][40] = 127, + [0][1][1][0][RTW89_ETSI][40] = 127, + [0][1][1][0][RTW89_MKK][40] = 127, + [0][1][1][0][RTW89_IC][40] = 127, + [0][1][1][0][RTW89_KCC][40] = 127, + [0][1][1][0][RTW89_ACMA][40] = 127, + [0][1][1][0][RTW89_CN][40] = 127, + [0][1][1][0][RTW89_UK][40] = 127, + [0][1][1][0][RTW89_FCC][42] = 127, + [0][1][1][0][RTW89_ETSI][42] = 127, + [0][1][1][0][RTW89_MKK][42] = 127, + [0][1][1][0][RTW89_IC][42] = 127, + [0][1][1][0][RTW89_KCC][42] = 127, + [0][1][1][0][RTW89_ACMA][42] = 127, + [0][1][1][0][RTW89_CN][42] = 127, + [0][1][1][0][RTW89_UK][42] = 127, + [0][1][1][0][RTW89_FCC][44] = 127, + [0][1][1][0][RTW89_ETSI][44] = 127, + [0][1][1][0][RTW89_MKK][44] = 127, + [0][1][1][0][RTW89_IC][44] = 127, + [0][1][1][0][RTW89_KCC][44] = 127, + [0][1][1][0][RTW89_ACMA][44] = 127, + [0][1][1][0][RTW89_CN][44] = 127, + [0][1][1][0][RTW89_UK][44] = 127, + [0][1][1][0][RTW89_FCC][46] = 127, + [0][1][1][0][RTW89_ETSI][46] = 127, + [0][1][1][0][RTW89_MKK][46] = 127, + [0][1][1][0][RTW89_IC][46] = 127, + [0][1][1][0][RTW89_KCC][46] = 127, + [0][1][1][0][RTW89_ACMA][46] = 127, + [0][1][1][0][RTW89_CN][46] = 127, + [0][1][1][0][RTW89_UK][46] = 127, + [0][1][1][0][RTW89_FCC][48] = 127, + [0][1][1][0][RTW89_ETSI][48] = 127, + [0][1][1][0][RTW89_MKK][48] = 127, + [0][1][1][0][RTW89_IC][48] = 127, + [0][1][1][0][RTW89_KCC][48] = 127, + [0][1][1][0][RTW89_ACMA][48] = 127, + [0][1][1][0][RTW89_CN][48] = 127, + [0][1][1][0][RTW89_UK][48] = 127, + [0][1][1][0][RTW89_FCC][50] = 127, + [0][1][1][0][RTW89_ETSI][50] = 127, + [0][1][1][0][RTW89_MKK][50] = 127, + [0][1][1][0][RTW89_IC][50] = 127, + [0][1][1][0][RTW89_KCC][50] = 127, + [0][1][1][0][RTW89_ACMA][50] = 127, + [0][1][1][0][RTW89_CN][50] = 127, + [0][1][1][0][RTW89_UK][50] = 127, + [0][1][1][0][RTW89_FCC][52] = 127, + [0][1][1][0][RTW89_ETSI][52] = 127, + [0][1][1][0][RTW89_MKK][52] = 127, + [0][1][1][0][RTW89_IC][52] = 127, + [0][1][1][0][RTW89_KCC][52] = 127, + [0][1][1][0][RTW89_ACMA][52] = 127, + [0][1][1][0][RTW89_CN][52] = 127, + [0][1][1][0][RTW89_UK][52] = 127, + [0][0][2][0][RTW89_FCC][0] = 78, + [0][0][2][0][RTW89_ETSI][0] = 62, + [0][0][2][0][RTW89_MKK][0] = 62, + [0][0][2][0][RTW89_IC][0] = 64, + [0][0][2][0][RTW89_KCC][0] = 76, + [0][0][2][0][RTW89_ACMA][0] = 62, + [0][0][2][0][RTW89_CN][0] = 62, + [0][0][2][0][RTW89_UK][0] = 62, + [0][0][2][0][RTW89_FCC][2] = 82, + [0][0][2][0][RTW89_ETSI][2] = 62, + [0][0][2][0][RTW89_MKK][2] = 62, + [0][0][2][0][RTW89_IC][2] = 64, + [0][0][2][0][RTW89_KCC][2] = 76, + [0][0][2][0][RTW89_ACMA][2] = 62, + [0][0][2][0][RTW89_CN][2] = 62, + [0][0][2][0][RTW89_UK][2] = 62, + [0][0][2][0][RTW89_FCC][4] = 82, + [0][0][2][0][RTW89_ETSI][4] = 62, + [0][0][2][0][RTW89_MKK][4] = 62, + [0][0][2][0][RTW89_IC][4] = 64, + [0][0][2][0][RTW89_KCC][4] = 76, + [0][0][2][0][RTW89_ACMA][4] = 62, + [0][0][2][0][RTW89_CN][4] = 62, + [0][0][2][0][RTW89_UK][4] = 62, + [0][0][2][0][RTW89_FCC][6] = 82, + [0][0][2][0][RTW89_ETSI][6] = 62, + [0][0][2][0][RTW89_MKK][6] = 62, + [0][0][2][0][RTW89_IC][6] = 64, + [0][0][2][0][RTW89_KCC][6] = 54, + [0][0][2][0][RTW89_ACMA][6] = 62, + [0][0][2][0][RTW89_CN][6] = 62, + [0][0][2][0][RTW89_UK][6] = 62, + [0][0][2][0][RTW89_FCC][8] = 82, + [0][0][2][0][RTW89_ETSI][8] = 62, + [0][0][2][0][RTW89_MKK][8] = 62, + [0][0][2][0][RTW89_IC][8] = 64, + [0][0][2][0][RTW89_KCC][8] = 76, + [0][0][2][0][RTW89_ACMA][8] = 62, + [0][0][2][0][RTW89_CN][8] = 62, + [0][0][2][0][RTW89_UK][8] = 62, + [0][0][2][0][RTW89_FCC][10] = 82, + [0][0][2][0][RTW89_ETSI][10] = 62, + [0][0][2][0][RTW89_MKK][10] = 62, + [0][0][2][0][RTW89_IC][10] = 64, + [0][0][2][0][RTW89_KCC][10] = 76, + [0][0][2][0][RTW89_ACMA][10] = 62, + [0][0][2][0][RTW89_CN][10] = 62, + [0][0][2][0][RTW89_UK][10] = 62, + [0][0][2][0][RTW89_FCC][12] = 82, + [0][0][2][0][RTW89_ETSI][12] = 62, + [0][0][2][0][RTW89_MKK][12] = 62, + [0][0][2][0][RTW89_IC][12] = 64, + [0][0][2][0][RTW89_KCC][12] = 78, + [0][0][2][0][RTW89_ACMA][12] = 62, + [0][0][2][0][RTW89_CN][12] = 62, + [0][0][2][0][RTW89_UK][12] = 62, + [0][0][2][0][RTW89_FCC][14] = 76, + [0][0][2][0][RTW89_ETSI][14] = 62, + [0][0][2][0][RTW89_MKK][14] = 62, + [0][0][2][0][RTW89_IC][14] = 64, + [0][0][2][0][RTW89_KCC][14] = 78, + [0][0][2][0][RTW89_ACMA][14] = 62, + [0][0][2][0][RTW89_CN][14] = 62, + [0][0][2][0][RTW89_UK][14] = 62, + [0][0][2][0][RTW89_FCC][15] = 76, + [0][0][2][0][RTW89_ETSI][15] = 60, + [0][0][2][0][RTW89_MKK][15] = 78, + [0][0][2][0][RTW89_IC][15] = 76, + [0][0][2][0][RTW89_KCC][15] = 78, + [0][0][2][0][RTW89_ACMA][15] = 60, + [0][0][2][0][RTW89_CN][15] = 127, + [0][0][2][0][RTW89_UK][15] = 60, + [0][0][2][0][RTW89_FCC][17] = 82, + [0][0][2][0][RTW89_ETSI][17] = 62, + [0][0][2][0][RTW89_MKK][17] = 78, + [0][0][2][0][RTW89_IC][17] = 82, + [0][0][2][0][RTW89_KCC][17] = 78, + [0][0][2][0][RTW89_ACMA][17] = 62, + [0][0][2][0][RTW89_CN][17] = 127, + [0][0][2][0][RTW89_UK][17] = 62, + [0][0][2][0][RTW89_FCC][19] = 82, + [0][0][2][0][RTW89_ETSI][19] = 62, + [0][0][2][0][RTW89_MKK][19] = 78, + [0][0][2][0][RTW89_IC][19] = 82, + [0][0][2][0][RTW89_KCC][19] = 78, + [0][0][2][0][RTW89_ACMA][19] = 62, + [0][0][2][0][RTW89_CN][19] = 127, + [0][0][2][0][RTW89_UK][19] = 62, + [0][0][2][0][RTW89_FCC][21] = 82, + [0][0][2][0][RTW89_ETSI][21] = 62, + [0][0][2][0][RTW89_MKK][21] = 78, + [0][0][2][0][RTW89_IC][21] = 82, + [0][0][2][0][RTW89_KCC][21] = 78, + [0][0][2][0][RTW89_ACMA][21] = 62, + [0][0][2][0][RTW89_CN][21] = 127, + [0][0][2][0][RTW89_UK][21] = 62, + [0][0][2][0][RTW89_FCC][23] = 82, + [0][0][2][0][RTW89_ETSI][23] = 62, + [0][0][2][0][RTW89_MKK][23] = 78, + [0][0][2][0][RTW89_IC][23] = 82, + [0][0][2][0][RTW89_KCC][23] = 78, + [0][0][2][0][RTW89_ACMA][23] = 62, + [0][0][2][0][RTW89_CN][23] = 127, + [0][0][2][0][RTW89_UK][23] = 62, + [0][0][2][0][RTW89_FCC][25] = 82, + [0][0][2][0][RTW89_ETSI][25] = 62, + [0][0][2][0][RTW89_MKK][25] = 78, + [0][0][2][0][RTW89_IC][25] = 127, + [0][0][2][0][RTW89_KCC][25] = 78, + [0][0][2][0][RTW89_ACMA][25] = 127, + [0][0][2][0][RTW89_CN][25] = 127, + [0][0][2][0][RTW89_UK][25] = 62, + [0][0][2][0][RTW89_FCC][27] = 82, + [0][0][2][0][RTW89_ETSI][27] = 62, + [0][0][2][0][RTW89_MKK][27] = 78, + [0][0][2][0][RTW89_IC][27] = 127, + [0][0][2][0][RTW89_KCC][27] = 78, + [0][0][2][0][RTW89_ACMA][27] = 127, + [0][0][2][0][RTW89_CN][27] = 127, + [0][0][2][0][RTW89_UK][27] = 62, + [0][0][2][0][RTW89_FCC][29] = 82, + [0][0][2][0][RTW89_ETSI][29] = 62, + [0][0][2][0][RTW89_MKK][29] = 78, + [0][0][2][0][RTW89_IC][29] = 127, + [0][0][2][0][RTW89_KCC][29] = 78, + [0][0][2][0][RTW89_ACMA][29] = 127, + [0][0][2][0][RTW89_CN][29] = 127, + [0][0][2][0][RTW89_UK][29] = 62, + [0][0][2][0][RTW89_FCC][31] = 82, + [0][0][2][0][RTW89_ETSI][31] = 62, + [0][0][2][0][RTW89_MKK][31] = 78, + [0][0][2][0][RTW89_IC][31] = 82, + [0][0][2][0][RTW89_KCC][31] = 74, + [0][0][2][0][RTW89_ACMA][31] = 62, + [0][0][2][0][RTW89_CN][31] = 127, + [0][0][2][0][RTW89_UK][31] = 62, + [0][0][2][0][RTW89_FCC][33] = 82, + [0][0][2][0][RTW89_ETSI][33] = 62, + [0][0][2][0][RTW89_MKK][33] = 78, + [0][0][2][0][RTW89_IC][33] = 82, + [0][0][2][0][RTW89_KCC][33] = 74, + [0][0][2][0][RTW89_ACMA][33] = 62, + [0][0][2][0][RTW89_CN][33] = 127, + [0][0][2][0][RTW89_UK][33] = 62, + [0][0][2][0][RTW89_FCC][35] = 72, + [0][0][2][0][RTW89_ETSI][35] = 62, + [0][0][2][0][RTW89_MKK][35] = 78, + [0][0][2][0][RTW89_IC][35] = 72, + [0][0][2][0][RTW89_KCC][35] = 74, + [0][0][2][0][RTW89_ACMA][35] = 62, + [0][0][2][0][RTW89_CN][35] = 127, + [0][0][2][0][RTW89_UK][35] = 62, + [0][0][2][0][RTW89_FCC][37] = 82, + [0][0][2][0][RTW89_ETSI][37] = 127, + [0][0][2][0][RTW89_MKK][37] = 78, + [0][0][2][0][RTW89_IC][37] = 82, + [0][0][2][0][RTW89_KCC][37] = 76, + [0][0][2][0][RTW89_ACMA][37] = 78, + [0][0][2][0][RTW89_CN][37] = 127, + [0][0][2][0][RTW89_UK][37] = 78, + [0][0][2][0][RTW89_FCC][38] = 82, + [0][0][2][0][RTW89_ETSI][38] = 30, + [0][0][2][0][RTW89_MKK][38] = 127, + [0][0][2][0][RTW89_IC][38] = 82, + [0][0][2][0][RTW89_KCC][38] = 66, + [0][0][2][0][RTW89_ACMA][38] = 78, + [0][0][2][0][RTW89_CN][38] = 78, + [0][0][2][0][RTW89_UK][38] = 60, + [0][0][2][0][RTW89_FCC][40] = 82, + [0][0][2][0][RTW89_ETSI][40] = 30, + [0][0][2][0][RTW89_MKK][40] = 127, + [0][0][2][0][RTW89_IC][40] = 82, + [0][0][2][0][RTW89_KCC][40] = 74, + [0][0][2][0][RTW89_ACMA][40] = 78, + [0][0][2][0][RTW89_CN][40] = 78, + [0][0][2][0][RTW89_UK][40] = 60, + [0][0][2][0][RTW89_FCC][42] = 82, + [0][0][2][0][RTW89_ETSI][42] = 30, + [0][0][2][0][RTW89_MKK][42] = 127, + [0][0][2][0][RTW89_IC][42] = 82, + [0][0][2][0][RTW89_KCC][42] = 74, + [0][0][2][0][RTW89_ACMA][42] = 78, + [0][0][2][0][RTW89_CN][42] = 78, + [0][0][2][0][RTW89_UK][42] = 60, + [0][0][2][0][RTW89_FCC][44] = 82, + [0][0][2][0][RTW89_ETSI][44] = 30, + [0][0][2][0][RTW89_MKK][44] = 127, + [0][0][2][0][RTW89_IC][44] = 82, + [0][0][2][0][RTW89_KCC][44] = 74, + [0][0][2][0][RTW89_ACMA][44] = 78, + [0][0][2][0][RTW89_CN][44] = 78, + [0][0][2][0][RTW89_UK][44] = 60, + [0][0][2][0][RTW89_FCC][46] = 82, + [0][0][2][0][RTW89_ETSI][46] = 30, + [0][0][2][0][RTW89_MKK][46] = 127, + [0][0][2][0][RTW89_IC][46] = 82, + [0][0][2][0][RTW89_KCC][46] = 74, + [0][0][2][0][RTW89_ACMA][46] = 78, + [0][0][2][0][RTW89_CN][46] = 78, + [0][0][2][0][RTW89_UK][46] = 60, + [0][0][2][0][RTW89_FCC][48] = 74, + [0][0][2][0][RTW89_ETSI][48] = 127, + [0][0][2][0][RTW89_MKK][48] = 127, + [0][0][2][0][RTW89_IC][48] = 127, + [0][0][2][0][RTW89_KCC][48] = 127, + [0][0][2][0][RTW89_ACMA][48] = 127, + [0][0][2][0][RTW89_CN][48] = 127, + [0][0][2][0][RTW89_UK][48] = 127, + [0][0][2][0][RTW89_FCC][50] = 76, + [0][0][2][0][RTW89_ETSI][50] = 127, + [0][0][2][0][RTW89_MKK][50] = 127, + [0][0][2][0][RTW89_IC][50] = 127, + [0][0][2][0][RTW89_KCC][50] = 127, + [0][0][2][0][RTW89_ACMA][50] = 127, + [0][0][2][0][RTW89_CN][50] = 127, + [0][0][2][0][RTW89_UK][50] = 127, + [0][0][2][0][RTW89_FCC][52] = 76, + [0][0][2][0][RTW89_ETSI][52] = 127, + [0][0][2][0][RTW89_MKK][52] = 127, + [0][0][2][0][RTW89_IC][52] = 127, + [0][0][2][0][RTW89_KCC][52] = 127, + [0][0][2][0][RTW89_ACMA][52] = 127, + [0][0][2][0][RTW89_CN][52] = 127, + [0][0][2][0][RTW89_UK][52] = 127, + [0][1][2][0][RTW89_FCC][0] = 127, + [0][1][2][0][RTW89_ETSI][0] = 127, + [0][1][2][0][RTW89_MKK][0] = 127, + [0][1][2][0][RTW89_IC][0] = 127, + [0][1][2][0][RTW89_KCC][0] = 127, + [0][1][2][0][RTW89_ACMA][0] = 127, + [0][1][2][0][RTW89_CN][0] = 127, + [0][1][2][0][RTW89_UK][0] = 127, + [0][1][2][0][RTW89_FCC][2] = 127, + [0][1][2][0][RTW89_ETSI][2] = 127, + [0][1][2][0][RTW89_MKK][2] = 127, + [0][1][2][0][RTW89_IC][2] = 127, + [0][1][2][0][RTW89_KCC][2] = 127, + [0][1][2][0][RTW89_ACMA][2] = 127, + [0][1][2][0][RTW89_CN][2] = 127, + [0][1][2][0][RTW89_UK][2] = 127, + [0][1][2][0][RTW89_FCC][4] = 127, + [0][1][2][0][RTW89_ETSI][4] = 127, + [0][1][2][0][RTW89_MKK][4] = 127, + [0][1][2][0][RTW89_IC][4] = 127, + [0][1][2][0][RTW89_KCC][4] = 127, + [0][1][2][0][RTW89_ACMA][4] = 127, + [0][1][2][0][RTW89_CN][4] = 127, + [0][1][2][0][RTW89_UK][4] = 127, + [0][1][2][0][RTW89_FCC][6] = 127, + [0][1][2][0][RTW89_ETSI][6] = 127, + [0][1][2][0][RTW89_MKK][6] = 127, + [0][1][2][0][RTW89_IC][6] = 127, + [0][1][2][0][RTW89_KCC][6] = 127, + [0][1][2][0][RTW89_ACMA][6] = 127, + [0][1][2][0][RTW89_CN][6] = 127, + [0][1][2][0][RTW89_UK][6] = 127, + [0][1][2][0][RTW89_FCC][8] = 127, + [0][1][2][0][RTW89_ETSI][8] = 127, + [0][1][2][0][RTW89_MKK][8] = 127, + [0][1][2][0][RTW89_IC][8] = 127, + [0][1][2][0][RTW89_KCC][8] = 127, + [0][1][2][0][RTW89_ACMA][8] = 127, + [0][1][2][0][RTW89_CN][8] = 127, + [0][1][2][0][RTW89_UK][8] = 127, + [0][1][2][0][RTW89_FCC][10] = 127, + [0][1][2][0][RTW89_ETSI][10] = 127, + [0][1][2][0][RTW89_MKK][10] = 127, + [0][1][2][0][RTW89_IC][10] = 127, + [0][1][2][0][RTW89_KCC][10] = 127, + [0][1][2][0][RTW89_ACMA][10] = 127, + [0][1][2][0][RTW89_CN][10] = 127, + [0][1][2][0][RTW89_UK][10] = 127, + [0][1][2][0][RTW89_FCC][12] = 127, + [0][1][2][0][RTW89_ETSI][12] = 127, + [0][1][2][0][RTW89_MKK][12] = 127, + [0][1][2][0][RTW89_IC][12] = 127, + [0][1][2][0][RTW89_KCC][12] = 127, + [0][1][2][0][RTW89_ACMA][12] = 127, + [0][1][2][0][RTW89_CN][12] = 127, + [0][1][2][0][RTW89_UK][12] = 127, + [0][1][2][0][RTW89_FCC][14] = 127, + [0][1][2][0][RTW89_ETSI][14] = 127, + [0][1][2][0][RTW89_MKK][14] = 127, + [0][1][2][0][RTW89_IC][14] = 127, + [0][1][2][0][RTW89_KCC][14] = 127, + [0][1][2][0][RTW89_ACMA][14] = 127, + [0][1][2][0][RTW89_CN][14] = 127, + [0][1][2][0][RTW89_UK][14] = 127, + [0][1][2][0][RTW89_FCC][15] = 127, + [0][1][2][0][RTW89_ETSI][15] = 127, + [0][1][2][0][RTW89_MKK][15] = 127, + [0][1][2][0][RTW89_IC][15] = 127, + [0][1][2][0][RTW89_KCC][15] = 127, + [0][1][2][0][RTW89_ACMA][15] = 127, + [0][1][2][0][RTW89_CN][15] = 127, + [0][1][2][0][RTW89_UK][15] = 127, + [0][1][2][0][RTW89_FCC][17] = 127, + [0][1][2][0][RTW89_ETSI][17] = 127, + [0][1][2][0][RTW89_MKK][17] = 127, + [0][1][2][0][RTW89_IC][17] = 127, + [0][1][2][0][RTW89_KCC][17] = 127, + [0][1][2][0][RTW89_ACMA][17] = 127, + [0][1][2][0][RTW89_CN][17] = 127, + [0][1][2][0][RTW89_UK][17] = 127, + [0][1][2][0][RTW89_FCC][19] = 127, + [0][1][2][0][RTW89_ETSI][19] = 127, + [0][1][2][0][RTW89_MKK][19] = 127, + [0][1][2][0][RTW89_IC][19] = 127, + [0][1][2][0][RTW89_KCC][19] = 127, + [0][1][2][0][RTW89_ACMA][19] = 127, + [0][1][2][0][RTW89_CN][19] = 127, + [0][1][2][0][RTW89_UK][19] = 127, + [0][1][2][0][RTW89_FCC][21] = 127, + [0][1][2][0][RTW89_ETSI][21] = 127, + [0][1][2][0][RTW89_MKK][21] = 127, + [0][1][2][0][RTW89_IC][21] = 127, + [0][1][2][0][RTW89_KCC][21] = 127, + [0][1][2][0][RTW89_ACMA][21] = 127, + [0][1][2][0][RTW89_CN][21] = 127, + [0][1][2][0][RTW89_UK][21] = 127, + [0][1][2][0][RTW89_FCC][23] = 127, + [0][1][2][0][RTW89_ETSI][23] = 127, + [0][1][2][0][RTW89_MKK][23] = 127, + [0][1][2][0][RTW89_IC][23] = 127, + [0][1][2][0][RTW89_KCC][23] = 127, + [0][1][2][0][RTW89_ACMA][23] = 127, + [0][1][2][0][RTW89_CN][23] = 127, + [0][1][2][0][RTW89_UK][23] = 127, + [0][1][2][0][RTW89_FCC][25] = 127, + [0][1][2][0][RTW89_ETSI][25] = 127, + [0][1][2][0][RTW89_MKK][25] = 127, + [0][1][2][0][RTW89_IC][25] = 127, + [0][1][2][0][RTW89_KCC][25] = 127, + [0][1][2][0][RTW89_ACMA][25] = 127, + [0][1][2][0][RTW89_CN][25] = 127, + [0][1][2][0][RTW89_UK][25] = 127, + [0][1][2][0][RTW89_FCC][27] = 127, + [0][1][2][0][RTW89_ETSI][27] = 127, + [0][1][2][0][RTW89_MKK][27] = 127, + [0][1][2][0][RTW89_IC][27] = 127, + [0][1][2][0][RTW89_KCC][27] = 127, + [0][1][2][0][RTW89_ACMA][27] = 127, + [0][1][2][0][RTW89_CN][27] = 127, + [0][1][2][0][RTW89_UK][27] = 127, + [0][1][2][0][RTW89_FCC][29] = 127, + [0][1][2][0][RTW89_ETSI][29] = 127, + [0][1][2][0][RTW89_MKK][29] = 127, + [0][1][2][0][RTW89_IC][29] = 127, + [0][1][2][0][RTW89_KCC][29] = 127, + [0][1][2][0][RTW89_ACMA][29] = 127, + [0][1][2][0][RTW89_CN][29] = 127, + [0][1][2][0][RTW89_UK][29] = 127, + [0][1][2][0][RTW89_FCC][31] = 127, + [0][1][2][0][RTW89_ETSI][31] = 127, + [0][1][2][0][RTW89_MKK][31] = 127, + [0][1][2][0][RTW89_IC][31] = 127, + [0][1][2][0][RTW89_KCC][31] = 127, + [0][1][2][0][RTW89_ACMA][31] = 127, + [0][1][2][0][RTW89_CN][31] = 127, + [0][1][2][0][RTW89_UK][31] = 127, + [0][1][2][0][RTW89_FCC][33] = 127, + [0][1][2][0][RTW89_ETSI][33] = 127, + [0][1][2][0][RTW89_MKK][33] = 127, + [0][1][2][0][RTW89_IC][33] = 127, + [0][1][2][0][RTW89_KCC][33] = 127, + [0][1][2][0][RTW89_ACMA][33] = 127, + [0][1][2][0][RTW89_CN][33] = 127, + [0][1][2][0][RTW89_UK][33] = 127, + [0][1][2][0][RTW89_FCC][35] = 127, + [0][1][2][0][RTW89_ETSI][35] = 127, + [0][1][2][0][RTW89_MKK][35] = 127, + [0][1][2][0][RTW89_IC][35] = 127, + [0][1][2][0][RTW89_KCC][35] = 127, + [0][1][2][0][RTW89_ACMA][35] = 127, + [0][1][2][0][RTW89_CN][35] = 127, + [0][1][2][0][RTW89_UK][35] = 127, + [0][1][2][0][RTW89_FCC][37] = 127, + [0][1][2][0][RTW89_ETSI][37] = 127, + [0][1][2][0][RTW89_MKK][37] = 127, + [0][1][2][0][RTW89_IC][37] = 127, + [0][1][2][0][RTW89_KCC][37] = 127, + [0][1][2][0][RTW89_ACMA][37] = 127, + [0][1][2][0][RTW89_CN][37] = 127, + [0][1][2][0][RTW89_UK][37] = 127, + [0][1][2][0][RTW89_FCC][38] = 127, + [0][1][2][0][RTW89_ETSI][38] = 127, + [0][1][2][0][RTW89_MKK][38] = 127, + [0][1][2][0][RTW89_IC][38] = 127, + [0][1][2][0][RTW89_KCC][38] = 127, + [0][1][2][0][RTW89_ACMA][38] = 127, + [0][1][2][0][RTW89_CN][38] = 127, + [0][1][2][0][RTW89_UK][38] = 127, + [0][1][2][0][RTW89_FCC][40] = 127, + [0][1][2][0][RTW89_ETSI][40] = 127, + [0][1][2][0][RTW89_MKK][40] = 127, + [0][1][2][0][RTW89_IC][40] = 127, + [0][1][2][0][RTW89_KCC][40] = 127, + [0][1][2][0][RTW89_ACMA][40] = 127, + [0][1][2][0][RTW89_CN][40] = 127, + [0][1][2][0][RTW89_UK][40] = 127, + [0][1][2][0][RTW89_FCC][42] = 127, + [0][1][2][0][RTW89_ETSI][42] = 127, + [0][1][2][0][RTW89_MKK][42] = 127, + [0][1][2][0][RTW89_IC][42] = 127, + [0][1][2][0][RTW89_KCC][42] = 127, + [0][1][2][0][RTW89_ACMA][42] = 127, + [0][1][2][0][RTW89_CN][42] = 127, + [0][1][2][0][RTW89_UK][42] = 127, + [0][1][2][0][RTW89_FCC][44] = 127, + [0][1][2][0][RTW89_ETSI][44] = 127, + [0][1][2][0][RTW89_MKK][44] = 127, + [0][1][2][0][RTW89_IC][44] = 127, + [0][1][2][0][RTW89_KCC][44] = 127, + [0][1][2][0][RTW89_ACMA][44] = 127, + [0][1][2][0][RTW89_CN][44] = 127, + [0][1][2][0][RTW89_UK][44] = 127, + [0][1][2][0][RTW89_FCC][46] = 127, + [0][1][2][0][RTW89_ETSI][46] = 127, + [0][1][2][0][RTW89_MKK][46] = 127, + [0][1][2][0][RTW89_IC][46] = 127, + [0][1][2][0][RTW89_KCC][46] = 127, + [0][1][2][0][RTW89_ACMA][46] = 127, + [0][1][2][0][RTW89_CN][46] = 127, + [0][1][2][0][RTW89_UK][46] = 127, + [0][1][2][0][RTW89_FCC][48] = 127, + [0][1][2][0][RTW89_ETSI][48] = 127, + [0][1][2][0][RTW89_MKK][48] = 127, + [0][1][2][0][RTW89_IC][48] = 127, + [0][1][2][0][RTW89_KCC][48] = 127, + [0][1][2][0][RTW89_ACMA][48] = 127, + [0][1][2][0][RTW89_CN][48] = 127, + [0][1][2][0][RTW89_UK][48] = 127, + [0][1][2][0][RTW89_FCC][50] = 127, + [0][1][2][0][RTW89_ETSI][50] = 127, + [0][1][2][0][RTW89_MKK][50] = 127, + [0][1][2][0][RTW89_IC][50] = 127, + [0][1][2][0][RTW89_KCC][50] = 127, + [0][1][2][0][RTW89_ACMA][50] = 127, + [0][1][2][0][RTW89_CN][50] = 127, + [0][1][2][0][RTW89_UK][50] = 127, + [0][1][2][0][RTW89_FCC][52] = 127, + [0][1][2][0][RTW89_ETSI][52] = 127, + [0][1][2][0][RTW89_MKK][52] = 127, + [0][1][2][0][RTW89_IC][52] = 127, + [0][1][2][0][RTW89_KCC][52] = 127, + [0][1][2][0][RTW89_ACMA][52] = 127, + [0][1][2][0][RTW89_CN][52] = 127, + [0][1][2][0][RTW89_UK][52] = 127, + [0][1][2][1][RTW89_FCC][0] = 127, + [0][1][2][1][RTW89_ETSI][0] = 127, + [0][1][2][1][RTW89_MKK][0] = 127, + [0][1][2][1][RTW89_IC][0] = 127, + [0][1][2][1][RTW89_KCC][0] = 127, + [0][1][2][1][RTW89_ACMA][0] = 127, + [0][1][2][1][RTW89_CN][0] = 127, + [0][1][2][1][RTW89_UK][0] = 127, + [0][1][2][1][RTW89_FCC][2] = 127, + [0][1][2][1][RTW89_ETSI][2] = 127, + [0][1][2][1][RTW89_MKK][2] = 127, + [0][1][2][1][RTW89_IC][2] = 127, + [0][1][2][1][RTW89_KCC][2] = 127, + [0][1][2][1][RTW89_ACMA][2] = 127, + [0][1][2][1][RTW89_CN][2] = 127, + [0][1][2][1][RTW89_UK][2] = 127, + [0][1][2][1][RTW89_FCC][4] = 127, + [0][1][2][1][RTW89_ETSI][4] = 127, + [0][1][2][1][RTW89_MKK][4] = 127, + [0][1][2][1][RTW89_IC][4] = 127, + [0][1][2][1][RTW89_KCC][4] = 127, + [0][1][2][1][RTW89_ACMA][4] = 127, + [0][1][2][1][RTW89_CN][4] = 127, + [0][1][2][1][RTW89_UK][4] = 127, + [0][1][2][1][RTW89_FCC][6] = 127, + [0][1][2][1][RTW89_ETSI][6] = 127, + [0][1][2][1][RTW89_MKK][6] = 127, + [0][1][2][1][RTW89_IC][6] = 127, + [0][1][2][1][RTW89_KCC][6] = 127, + [0][1][2][1][RTW89_ACMA][6] = 127, + [0][1][2][1][RTW89_CN][6] = 127, + [0][1][2][1][RTW89_UK][6] = 127, + [0][1][2][1][RTW89_FCC][8] = 127, + [0][1][2][1][RTW89_ETSI][8] = 127, + [0][1][2][1][RTW89_MKK][8] = 127, + [0][1][2][1][RTW89_IC][8] = 127, + [0][1][2][1][RTW89_KCC][8] = 127, + [0][1][2][1][RTW89_ACMA][8] = 127, + [0][1][2][1][RTW89_CN][8] = 127, + [0][1][2][1][RTW89_UK][8] = 127, + [0][1][2][1][RTW89_FCC][10] = 127, + [0][1][2][1][RTW89_ETSI][10] = 127, + [0][1][2][1][RTW89_MKK][10] = 127, + [0][1][2][1][RTW89_IC][10] = 127, + [0][1][2][1][RTW89_KCC][10] = 127, + [0][1][2][1][RTW89_ACMA][10] = 127, + [0][1][2][1][RTW89_CN][10] = 127, + [0][1][2][1][RTW89_UK][10] = 127, + [0][1][2][1][RTW89_FCC][12] = 127, + [0][1][2][1][RTW89_ETSI][12] = 127, + [0][1][2][1][RTW89_MKK][12] = 127, + [0][1][2][1][RTW89_IC][12] = 127, + [0][1][2][1][RTW89_KCC][12] = 127, + [0][1][2][1][RTW89_ACMA][12] = 127, + [0][1][2][1][RTW89_CN][12] = 127, + [0][1][2][1][RTW89_UK][12] = 127, + [0][1][2][1][RTW89_FCC][14] = 127, + [0][1][2][1][RTW89_ETSI][14] = 127, + [0][1][2][1][RTW89_MKK][14] = 127, + [0][1][2][1][RTW89_IC][14] = 127, + [0][1][2][1][RTW89_KCC][14] = 127, + [0][1][2][1][RTW89_ACMA][14] = 127, + [0][1][2][1][RTW89_CN][14] = 127, + [0][1][2][1][RTW89_UK][14] = 127, + [0][1][2][1][RTW89_FCC][15] = 127, + [0][1][2][1][RTW89_ETSI][15] = 127, + [0][1][2][1][RTW89_MKK][15] = 127, + [0][1][2][1][RTW89_IC][15] = 127, + [0][1][2][1][RTW89_KCC][15] = 127, + [0][1][2][1][RTW89_ACMA][15] = 127, + [0][1][2][1][RTW89_CN][15] = 127, + [0][1][2][1][RTW89_UK][15] = 127, + [0][1][2][1][RTW89_FCC][17] = 127, + [0][1][2][1][RTW89_ETSI][17] = 127, + [0][1][2][1][RTW89_MKK][17] = 127, + [0][1][2][1][RTW89_IC][17] = 127, + [0][1][2][1][RTW89_KCC][17] = 127, + [0][1][2][1][RTW89_ACMA][17] = 127, + [0][1][2][1][RTW89_CN][17] = 127, + [0][1][2][1][RTW89_UK][17] = 127, + [0][1][2][1][RTW89_FCC][19] = 127, + [0][1][2][1][RTW89_ETSI][19] = 127, + [0][1][2][1][RTW89_MKK][19] = 127, + [0][1][2][1][RTW89_IC][19] = 127, + [0][1][2][1][RTW89_KCC][19] = 127, + [0][1][2][1][RTW89_ACMA][19] = 127, + [0][1][2][1][RTW89_CN][19] = 127, + [0][1][2][1][RTW89_UK][19] = 127, + [0][1][2][1][RTW89_FCC][21] = 127, + [0][1][2][1][RTW89_ETSI][21] = 127, + [0][1][2][1][RTW89_MKK][21] = 127, + [0][1][2][1][RTW89_IC][21] = 127, + [0][1][2][1][RTW89_KCC][21] = 127, + [0][1][2][1][RTW89_ACMA][21] = 127, + [0][1][2][1][RTW89_CN][21] = 127, + [0][1][2][1][RTW89_UK][21] = 127, + [0][1][2][1][RTW89_FCC][23] = 127, + [0][1][2][1][RTW89_ETSI][23] = 127, + [0][1][2][1][RTW89_MKK][23] = 127, + [0][1][2][1][RTW89_IC][23] = 127, + [0][1][2][1][RTW89_KCC][23] = 127, + [0][1][2][1][RTW89_ACMA][23] = 127, + [0][1][2][1][RTW89_CN][23] = 127, + [0][1][2][1][RTW89_UK][23] = 127, + [0][1][2][1][RTW89_FCC][25] = 127, + [0][1][2][1][RTW89_ETSI][25] = 127, + [0][1][2][1][RTW89_MKK][25] = 127, + [0][1][2][1][RTW89_IC][25] = 127, + [0][1][2][1][RTW89_KCC][25] = 127, + [0][1][2][1][RTW89_ACMA][25] = 127, + [0][1][2][1][RTW89_CN][25] = 127, + [0][1][2][1][RTW89_UK][25] = 127, + [0][1][2][1][RTW89_FCC][27] = 127, + [0][1][2][1][RTW89_ETSI][27] = 127, + [0][1][2][1][RTW89_MKK][27] = 127, + [0][1][2][1][RTW89_IC][27] = 127, + [0][1][2][1][RTW89_KCC][27] = 127, + [0][1][2][1][RTW89_ACMA][27] = 127, + [0][1][2][1][RTW89_CN][27] = 127, + [0][1][2][1][RTW89_UK][27] = 127, + [0][1][2][1][RTW89_FCC][29] = 127, + [0][1][2][1][RTW89_ETSI][29] = 127, + [0][1][2][1][RTW89_MKK][29] = 127, + [0][1][2][1][RTW89_IC][29] = 127, + [0][1][2][1][RTW89_KCC][29] = 127, + [0][1][2][1][RTW89_ACMA][29] = 127, + [0][1][2][1][RTW89_CN][29] = 127, + [0][1][2][1][RTW89_UK][29] = 127, + [0][1][2][1][RTW89_FCC][31] = 127, + [0][1][2][1][RTW89_ETSI][31] = 127, + [0][1][2][1][RTW89_MKK][31] = 127, + [0][1][2][1][RTW89_IC][31] = 127, + [0][1][2][1][RTW89_KCC][31] = 127, + [0][1][2][1][RTW89_ACMA][31] = 127, + [0][1][2][1][RTW89_CN][31] = 127, + [0][1][2][1][RTW89_UK][31] = 127, + [0][1][2][1][RTW89_FCC][33] = 127, + [0][1][2][1][RTW89_ETSI][33] = 127, + [0][1][2][1][RTW89_MKK][33] = 127, + [0][1][2][1][RTW89_IC][33] = 127, + [0][1][2][1][RTW89_KCC][33] = 127, + [0][1][2][1][RTW89_ACMA][33] = 127, + [0][1][2][1][RTW89_CN][33] = 127, + [0][1][2][1][RTW89_UK][33] = 127, + [0][1][2][1][RTW89_FCC][35] = 127, + [0][1][2][1][RTW89_ETSI][35] = 127, + [0][1][2][1][RTW89_MKK][35] = 127, + [0][1][2][1][RTW89_IC][35] = 127, + [0][1][2][1][RTW89_KCC][35] = 127, + [0][1][2][1][RTW89_ACMA][35] = 127, + [0][1][2][1][RTW89_CN][35] = 127, + [0][1][2][1][RTW89_UK][35] = 127, + [0][1][2][1][RTW89_FCC][37] = 127, + [0][1][2][1][RTW89_ETSI][37] = 127, + [0][1][2][1][RTW89_MKK][37] = 127, + [0][1][2][1][RTW89_IC][37] = 127, + [0][1][2][1][RTW89_KCC][37] = 127, + [0][1][2][1][RTW89_ACMA][37] = 127, + [0][1][2][1][RTW89_CN][37] = 127, + [0][1][2][1][RTW89_UK][37] = 127, + [0][1][2][1][RTW89_FCC][38] = 127, + [0][1][2][1][RTW89_ETSI][38] = 127, + [0][1][2][1][RTW89_MKK][38] = 127, + [0][1][2][1][RTW89_IC][38] = 127, + [0][1][2][1][RTW89_KCC][38] = 127, + [0][1][2][1][RTW89_ACMA][38] = 127, + [0][1][2][1][RTW89_CN][38] = 127, + [0][1][2][1][RTW89_UK][38] = 127, + [0][1][2][1][RTW89_FCC][40] = 127, + [0][1][2][1][RTW89_ETSI][40] = 127, + [0][1][2][1][RTW89_MKK][40] = 127, + [0][1][2][1][RTW89_IC][40] = 127, + [0][1][2][1][RTW89_KCC][40] = 127, + [0][1][2][1][RTW89_ACMA][40] = 127, + [0][1][2][1][RTW89_CN][40] = 127, + [0][1][2][1][RTW89_UK][40] = 127, + [0][1][2][1][RTW89_FCC][42] = 127, + [0][1][2][1][RTW89_ETSI][42] = 127, + [0][1][2][1][RTW89_MKK][42] = 127, + [0][1][2][1][RTW89_IC][42] = 127, + [0][1][2][1][RTW89_KCC][42] = 127, + [0][1][2][1][RTW89_ACMA][42] = 127, + [0][1][2][1][RTW89_CN][42] = 127, + [0][1][2][1][RTW89_UK][42] = 127, + [0][1][2][1][RTW89_FCC][44] = 127, + [0][1][2][1][RTW89_ETSI][44] = 127, + [0][1][2][1][RTW89_MKK][44] = 127, + [0][1][2][1][RTW89_IC][44] = 127, + [0][1][2][1][RTW89_KCC][44] = 127, + [0][1][2][1][RTW89_ACMA][44] = 127, + [0][1][2][1][RTW89_CN][44] = 127, + [0][1][2][1][RTW89_UK][44] = 127, + [0][1][2][1][RTW89_FCC][46] = 127, + [0][1][2][1][RTW89_ETSI][46] = 127, + [0][1][2][1][RTW89_MKK][46] = 127, + [0][1][2][1][RTW89_IC][46] = 127, + [0][1][2][1][RTW89_KCC][46] = 127, + [0][1][2][1][RTW89_ACMA][46] = 127, + [0][1][2][1][RTW89_CN][46] = 127, + [0][1][2][1][RTW89_UK][46] = 127, + [0][1][2][1][RTW89_FCC][48] = 127, + [0][1][2][1][RTW89_ETSI][48] = 127, + [0][1][2][1][RTW89_MKK][48] = 127, + [0][1][2][1][RTW89_IC][48] = 127, + [0][1][2][1][RTW89_KCC][48] = 127, + [0][1][2][1][RTW89_ACMA][48] = 127, + [0][1][2][1][RTW89_CN][48] = 127, + [0][1][2][1][RTW89_UK][48] = 127, + [0][1][2][1][RTW89_FCC][50] = 127, + [0][1][2][1][RTW89_ETSI][50] = 127, + [0][1][2][1][RTW89_MKK][50] = 127, + [0][1][2][1][RTW89_IC][50] = 127, + [0][1][2][1][RTW89_KCC][50] = 127, + [0][1][2][1][RTW89_ACMA][50] = 127, + [0][1][2][1][RTW89_CN][50] = 127, + [0][1][2][1][RTW89_UK][50] = 127, + [0][1][2][1][RTW89_FCC][52] = 127, + [0][1][2][1][RTW89_ETSI][52] = 127, + [0][1][2][1][RTW89_MKK][52] = 127, + [0][1][2][1][RTW89_IC][52] = 127, + [0][1][2][1][RTW89_KCC][52] = 127, + [0][1][2][1][RTW89_ACMA][52] = 127, + [0][1][2][1][RTW89_CN][52] = 127, + [0][1][2][1][RTW89_UK][52] = 127, + [1][0][2][0][RTW89_FCC][1] = 68, + [1][0][2][0][RTW89_ETSI][1] = 64, + [1][0][2][0][RTW89_MKK][1] = 64, + [1][0][2][0][RTW89_IC][1] = 64, + [1][0][2][0][RTW89_KCC][1] = 74, + [1][0][2][0][RTW89_ACMA][1] = 64, + [1][0][2][0][RTW89_CN][1] = 64, + [1][0][2][0][RTW89_UK][1] = 64, + [1][0][2][0][RTW89_FCC][5] = 82, + [1][0][2][0][RTW89_ETSI][5] = 64, + [1][0][2][0][RTW89_MKK][5] = 62, + [1][0][2][0][RTW89_IC][5] = 64, + [1][0][2][0][RTW89_KCC][5] = 66, + [1][0][2][0][RTW89_ACMA][5] = 64, + [1][0][2][0][RTW89_CN][5] = 64, + [1][0][2][0][RTW89_UK][5] = 64, + [1][0][2][0][RTW89_FCC][9] = 82, + [1][0][2][0][RTW89_ETSI][9] = 64, + [1][0][2][0][RTW89_MKK][9] = 64, + [1][0][2][0][RTW89_IC][9] = 64, + [1][0][2][0][RTW89_KCC][9] = 78, + [1][0][2][0][RTW89_ACMA][9] = 64, + [1][0][2][0][RTW89_CN][9] = 64, + [1][0][2][0][RTW89_UK][9] = 64, + [1][0][2][0][RTW89_FCC][13] = 66, + [1][0][2][0][RTW89_ETSI][13] = 64, + [1][0][2][0][RTW89_MKK][13] = 64, + [1][0][2][0][RTW89_IC][13] = 64, + [1][0][2][0][RTW89_KCC][13] = 72, + [1][0][2][0][RTW89_ACMA][13] = 64, + [1][0][2][0][RTW89_CN][13] = 64, + [1][0][2][0][RTW89_UK][13] = 64, + [1][0][2][0][RTW89_FCC][16] = 66, + [1][0][2][0][RTW89_ETSI][16] = 66, + [1][0][2][0][RTW89_MKK][16] = 80, + [1][0][2][0][RTW89_IC][16] = 66, + [1][0][2][0][RTW89_KCC][16] = 74, + [1][0][2][0][RTW89_ACMA][16] = 66, + [1][0][2][0][RTW89_CN][16] = 127, + [1][0][2][0][RTW89_UK][16] = 66, + [1][0][2][0][RTW89_FCC][20] = 80, + [1][0][2][0][RTW89_ETSI][20] = 66, + [1][0][2][0][RTW89_MKK][20] = 80, + [1][0][2][0][RTW89_IC][20] = 80, + [1][0][2][0][RTW89_KCC][20] = 74, + [1][0][2][0][RTW89_ACMA][20] = 66, + [1][0][2][0][RTW89_CN][20] = 127, + [1][0][2][0][RTW89_UK][20] = 66, + [1][0][2][0][RTW89_FCC][24] = 80, + [1][0][2][0][RTW89_ETSI][24] = 66, + [1][0][2][0][RTW89_MKK][24] = 80, + [1][0][2][0][RTW89_IC][24] = 127, + [1][0][2][0][RTW89_KCC][24] = 74, + [1][0][2][0][RTW89_ACMA][24] = 127, + [1][0][2][0][RTW89_CN][24] = 127, + [1][0][2][0][RTW89_UK][24] = 66, + [1][0][2][0][RTW89_FCC][28] = 80, + [1][0][2][0][RTW89_ETSI][28] = 66, + [1][0][2][0][RTW89_MKK][28] = 80, + [1][0][2][0][RTW89_IC][28] = 127, + [1][0][2][0][RTW89_KCC][28] = 74, + [1][0][2][0][RTW89_ACMA][28] = 127, + [1][0][2][0][RTW89_CN][28] = 127, + [1][0][2][0][RTW89_UK][28] = 66, + [1][0][2][0][RTW89_FCC][32] = 76, + [1][0][2][0][RTW89_ETSI][32] = 66, + [1][0][2][0][RTW89_MKK][32] = 80, + [1][0][2][0][RTW89_IC][32] = 76, + [1][0][2][0][RTW89_KCC][32] = 78, + [1][0][2][0][RTW89_ACMA][32] = 66, + [1][0][2][0][RTW89_CN][32] = 127, + [1][0][2][0][RTW89_UK][32] = 66, + [1][0][2][0][RTW89_FCC][36] = 80, + [1][0][2][0][RTW89_ETSI][36] = 127, + [1][0][2][0][RTW89_MKK][36] = 80, + [1][0][2][0][RTW89_IC][36] = 80, + [1][0][2][0][RTW89_KCC][36] = 76, + [1][0][2][0][RTW89_ACMA][36] = 78, + [1][0][2][0][RTW89_CN][36] = 127, + [1][0][2][0][RTW89_UK][36] = 80, + [1][0][2][0][RTW89_FCC][39] = 84, + [1][0][2][0][RTW89_ETSI][39] = 30, + [1][0][2][0][RTW89_MKK][39] = 127, + [1][0][2][0][RTW89_IC][39] = 84, + [1][0][2][0][RTW89_KCC][39] = 68, + [1][0][2][0][RTW89_ACMA][39] = 80, + [1][0][2][0][RTW89_CN][39] = 70, + [1][0][2][0][RTW89_UK][39] = 64, + [1][0][2][0][RTW89_FCC][43] = 84, + [1][0][2][0][RTW89_ETSI][43] = 30, + [1][0][2][0][RTW89_MKK][43] = 127, + [1][0][2][0][RTW89_IC][43] = 84, + [1][0][2][0][RTW89_KCC][43] = 78, + [1][0][2][0][RTW89_ACMA][43] = 80, + [1][0][2][0][RTW89_CN][43] = 80, + [1][0][2][0][RTW89_UK][43] = 64, + [1][0][2][0][RTW89_FCC][47] = 84, + [1][0][2][0][RTW89_ETSI][47] = 127, + [1][0][2][0][RTW89_MKK][47] = 127, + [1][0][2][0][RTW89_IC][47] = 127, + [1][0][2][0][RTW89_KCC][47] = 127, + [1][0][2][0][RTW89_ACMA][47] = 127, + [1][0][2][0][RTW89_CN][47] = 127, + [1][0][2][0][RTW89_UK][47] = 127, + [1][0][2][0][RTW89_FCC][51] = 84, + [1][0][2][0][RTW89_ETSI][51] = 127, + [1][0][2][0][RTW89_MKK][51] = 127, + [1][0][2][0][RTW89_IC][51] = 127, + [1][0][2][0][RTW89_KCC][51] = 127, + [1][0][2][0][RTW89_ACMA][51] = 127, + [1][0][2][0][RTW89_CN][51] = 127, + [1][0][2][0][RTW89_UK][51] = 127, + [1][1][2][0][RTW89_FCC][1] = 127, + [1][1][2][0][RTW89_ETSI][1] = 127, + [1][1][2][0][RTW89_MKK][1] = 127, + [1][1][2][0][RTW89_IC][1] = 127, + [1][1][2][0][RTW89_KCC][1] = 127, + [1][1][2][0][RTW89_ACMA][1] = 127, + [1][1][2][0][RTW89_CN][1] = 127, + [1][1][2][0][RTW89_UK][1] = 127, + [1][1][2][0][RTW89_FCC][5] = 127, + [1][1][2][0][RTW89_ETSI][5] = 127, + [1][1][2][0][RTW89_MKK][5] = 127, + [1][1][2][0][RTW89_IC][5] = 127, + [1][1][2][0][RTW89_KCC][5] = 127, + [1][1][2][0][RTW89_ACMA][5] = 127, + [1][1][2][0][RTW89_CN][5] = 127, + [1][1][2][0][RTW89_UK][5] = 127, + [1][1][2][0][RTW89_FCC][9] = 127, + [1][1][2][0][RTW89_ETSI][9] = 127, + [1][1][2][0][RTW89_MKK][9] = 127, + [1][1][2][0][RTW89_IC][9] = 127, + [1][1][2][0][RTW89_KCC][9] = 127, + [1][1][2][0][RTW89_ACMA][9] = 127, + [1][1][2][0][RTW89_CN][9] = 127, + [1][1][2][0][RTW89_UK][9] = 127, + [1][1][2][0][RTW89_FCC][13] = 127, + [1][1][2][0][RTW89_ETSI][13] = 127, + [1][1][2][0][RTW89_MKK][13] = 127, + [1][1][2][0][RTW89_IC][13] = 127, + [1][1][2][0][RTW89_KCC][13] = 127, + [1][1][2][0][RTW89_ACMA][13] = 127, + [1][1][2][0][RTW89_CN][13] = 127, + [1][1][2][0][RTW89_UK][13] = 127, + [1][1][2][0][RTW89_FCC][16] = 127, + [1][1][2][0][RTW89_ETSI][16] = 127, + [1][1][2][0][RTW89_MKK][16] = 127, + [1][1][2][0][RTW89_IC][16] = 127, + [1][1][2][0][RTW89_KCC][16] = 127, + [1][1][2][0][RTW89_ACMA][16] = 127, + [1][1][2][0][RTW89_CN][16] = 127, + [1][1][2][0][RTW89_UK][16] = 127, + [1][1][2][0][RTW89_FCC][20] = 127, + [1][1][2][0][RTW89_ETSI][20] = 127, + [1][1][2][0][RTW89_MKK][20] = 127, + [1][1][2][0][RTW89_IC][20] = 127, + [1][1][2][0][RTW89_KCC][20] = 127, + [1][1][2][0][RTW89_ACMA][20] = 127, + [1][1][2][0][RTW89_CN][20] = 127, + [1][1][2][0][RTW89_UK][20] = 127, + [1][1][2][0][RTW89_FCC][24] = 127, + [1][1][2][0][RTW89_ETSI][24] = 127, + [1][1][2][0][RTW89_MKK][24] = 127, + [1][1][2][0][RTW89_IC][24] = 127, + [1][1][2][0][RTW89_KCC][24] = 127, + [1][1][2][0][RTW89_ACMA][24] = 127, + [1][1][2][0][RTW89_CN][24] = 127, + [1][1][2][0][RTW89_UK][24] = 127, + [1][1][2][0][RTW89_FCC][28] = 127, + [1][1][2][0][RTW89_ETSI][28] = 127, + [1][1][2][0][RTW89_MKK][28] = 127, + [1][1][2][0][RTW89_IC][28] = 127, + [1][1][2][0][RTW89_KCC][28] = 127, + [1][1][2][0][RTW89_ACMA][28] = 127, + [1][1][2][0][RTW89_CN][28] = 127, + [1][1][2][0][RTW89_UK][28] = 127, + [1][1][2][0][RTW89_FCC][32] = 127, + [1][1][2][0][RTW89_ETSI][32] = 127, + [1][1][2][0][RTW89_MKK][32] = 127, + [1][1][2][0][RTW89_IC][32] = 127, + [1][1][2][0][RTW89_KCC][32] = 127, + [1][1][2][0][RTW89_ACMA][32] = 127, + [1][1][2][0][RTW89_CN][32] = 127, + [1][1][2][0][RTW89_UK][32] = 127, + [1][1][2][0][RTW89_FCC][36] = 127, + [1][1][2][0][RTW89_ETSI][36] = 127, + [1][1][2][0][RTW89_MKK][36] = 127, + [1][1][2][0][RTW89_IC][36] = 127, + [1][1][2][0][RTW89_KCC][36] = 127, + [1][1][2][0][RTW89_ACMA][36] = 127, + [1][1][2][0][RTW89_CN][36] = 127, + [1][1][2][0][RTW89_UK][36] = 127, + [1][1][2][0][RTW89_FCC][39] = 127, + [1][1][2][0][RTW89_ETSI][39] = 127, + [1][1][2][0][RTW89_MKK][39] = 127, + [1][1][2][0][RTW89_IC][39] = 127, + [1][1][2][0][RTW89_KCC][39] = 127, + [1][1][2][0][RTW89_ACMA][39] = 127, + [1][1][2][0][RTW89_CN][39] = 127, + [1][1][2][0][RTW89_UK][39] = 127, + [1][1][2][0][RTW89_FCC][43] = 127, + [1][1][2][0][RTW89_ETSI][43] = 127, + [1][1][2][0][RTW89_MKK][43] = 127, + [1][1][2][0][RTW89_IC][43] = 127, + [1][1][2][0][RTW89_KCC][43] = 127, + [1][1][2][0][RTW89_ACMA][43] = 127, + [1][1][2][0][RTW89_CN][43] = 127, + [1][1][2][0][RTW89_UK][43] = 127, + [1][1][2][0][RTW89_FCC][47] = 127, + [1][1][2][0][RTW89_ETSI][47] = 127, + [1][1][2][0][RTW89_MKK][47] = 127, + [1][1][2][0][RTW89_IC][47] = 127, + [1][1][2][0][RTW89_KCC][47] = 127, + [1][1][2][0][RTW89_ACMA][47] = 127, + [1][1][2][0][RTW89_CN][47] = 127, + [1][1][2][0][RTW89_UK][47] = 127, + [1][1][2][0][RTW89_FCC][51] = 127, + [1][1][2][0][RTW89_ETSI][51] = 127, + [1][1][2][0][RTW89_MKK][51] = 127, + [1][1][2][0][RTW89_IC][51] = 127, + [1][1][2][0][RTW89_KCC][51] = 127, + [1][1][2][0][RTW89_ACMA][51] = 127, + [1][1][2][0][RTW89_CN][51] = 127, + [1][1][2][0][RTW89_UK][51] = 127, + [1][1][2][1][RTW89_FCC][1] = 127, + [1][1][2][1][RTW89_ETSI][1] = 127, + [1][1][2][1][RTW89_MKK][1] = 127, + [1][1][2][1][RTW89_IC][1] = 127, + [1][1][2][1][RTW89_KCC][1] = 127, + [1][1][2][1][RTW89_ACMA][1] = 127, + [1][1][2][1][RTW89_CN][1] = 127, + [1][1][2][1][RTW89_UK][1] = 127, + [1][1][2][1][RTW89_FCC][5] = 127, + [1][1][2][1][RTW89_ETSI][5] = 127, + [1][1][2][1][RTW89_MKK][5] = 127, + [1][1][2][1][RTW89_IC][5] = 127, + [1][1][2][1][RTW89_KCC][5] = 127, + [1][1][2][1][RTW89_ACMA][5] = 127, + [1][1][2][1][RTW89_CN][5] = 127, + [1][1][2][1][RTW89_UK][5] = 127, + [1][1][2][1][RTW89_FCC][9] = 127, + [1][1][2][1][RTW89_ETSI][9] = 127, + [1][1][2][1][RTW89_MKK][9] = 127, + [1][1][2][1][RTW89_IC][9] = 127, + [1][1][2][1][RTW89_KCC][9] = 127, + [1][1][2][1][RTW89_ACMA][9] = 127, + [1][1][2][1][RTW89_CN][9] = 127, + [1][1][2][1][RTW89_UK][9] = 127, + [1][1][2][1][RTW89_FCC][13] = 127, + [1][1][2][1][RTW89_ETSI][13] = 127, + [1][1][2][1][RTW89_MKK][13] = 127, + [1][1][2][1][RTW89_IC][13] = 127, + [1][1][2][1][RTW89_KCC][13] = 127, + [1][1][2][1][RTW89_ACMA][13] = 127, + [1][1][2][1][RTW89_CN][13] = 127, + [1][1][2][1][RTW89_UK][13] = 127, + [1][1][2][1][RTW89_FCC][16] = 127, + [1][1][2][1][RTW89_ETSI][16] = 127, + [1][1][2][1][RTW89_MKK][16] = 127, + [1][1][2][1][RTW89_IC][16] = 127, + [1][1][2][1][RTW89_KCC][16] = 127, + [1][1][2][1][RTW89_ACMA][16] = 127, + [1][1][2][1][RTW89_CN][16] = 127, + [1][1][2][1][RTW89_UK][16] = 127, + [1][1][2][1][RTW89_FCC][20] = 127, + [1][1][2][1][RTW89_ETSI][20] = 127, + [1][1][2][1][RTW89_MKK][20] = 127, + [1][1][2][1][RTW89_IC][20] = 127, + [1][1][2][1][RTW89_KCC][20] = 127, + [1][1][2][1][RTW89_ACMA][20] = 127, + [1][1][2][1][RTW89_CN][20] = 127, + [1][1][2][1][RTW89_UK][20] = 127, + [1][1][2][1][RTW89_FCC][24] = 127, + [1][1][2][1][RTW89_ETSI][24] = 127, + [1][1][2][1][RTW89_MKK][24] = 127, + [1][1][2][1][RTW89_IC][24] = 127, + [1][1][2][1][RTW89_KCC][24] = 127, + [1][1][2][1][RTW89_ACMA][24] = 127, + [1][1][2][1][RTW89_CN][24] = 127, + [1][1][2][1][RTW89_UK][24] = 127, + [1][1][2][1][RTW89_FCC][28] = 127, + [1][1][2][1][RTW89_ETSI][28] = 127, + [1][1][2][1][RTW89_MKK][28] = 127, + [1][1][2][1][RTW89_IC][28] = 127, + [1][1][2][1][RTW89_KCC][28] = 127, + [1][1][2][1][RTW89_ACMA][28] = 127, + [1][1][2][1][RTW89_CN][28] = 127, + [1][1][2][1][RTW89_UK][28] = 127, + [1][1][2][1][RTW89_FCC][32] = 127, + [1][1][2][1][RTW89_ETSI][32] = 127, + [1][1][2][1][RTW89_MKK][32] = 127, + [1][1][2][1][RTW89_IC][32] = 127, + [1][1][2][1][RTW89_KCC][32] = 127, + [1][1][2][1][RTW89_ACMA][32] = 127, + [1][1][2][1][RTW89_CN][32] = 127, + [1][1][2][1][RTW89_UK][32] = 127, + [1][1][2][1][RTW89_FCC][36] = 127, + [1][1][2][1][RTW89_ETSI][36] = 127, + [1][1][2][1][RTW89_MKK][36] = 127, + [1][1][2][1][RTW89_IC][36] = 127, + [1][1][2][1][RTW89_KCC][36] = 127, + [1][1][2][1][RTW89_ACMA][36] = 127, + [1][1][2][1][RTW89_CN][36] = 127, + [1][1][2][1][RTW89_UK][36] = 127, + [1][1][2][1][RTW89_FCC][39] = 127, + [1][1][2][1][RTW89_ETSI][39] = 127, + [1][1][2][1][RTW89_MKK][39] = 127, + [1][1][2][1][RTW89_IC][39] = 127, + [1][1][2][1][RTW89_KCC][39] = 127, + [1][1][2][1][RTW89_ACMA][39] = 127, + [1][1][2][1][RTW89_CN][39] = 127, + [1][1][2][1][RTW89_UK][39] = 127, + [1][1][2][1][RTW89_FCC][43] = 127, + [1][1][2][1][RTW89_ETSI][43] = 127, + [1][1][2][1][RTW89_MKK][43] = 127, + [1][1][2][1][RTW89_IC][43] = 127, + [1][1][2][1][RTW89_KCC][43] = 127, + [1][1][2][1][RTW89_ACMA][43] = 127, + [1][1][2][1][RTW89_CN][43] = 127, + [1][1][2][1][RTW89_UK][43] = 127, + [1][1][2][1][RTW89_FCC][47] = 127, + [1][1][2][1][RTW89_ETSI][47] = 127, + [1][1][2][1][RTW89_MKK][47] = 127, + [1][1][2][1][RTW89_IC][47] = 127, + [1][1][2][1][RTW89_KCC][47] = 127, + [1][1][2][1][RTW89_ACMA][47] = 127, + [1][1][2][1][RTW89_CN][47] = 127, + [1][1][2][1][RTW89_UK][47] = 127, + [1][1][2][1][RTW89_FCC][51] = 127, + [1][1][2][1][RTW89_ETSI][51] = 127, + [1][1][2][1][RTW89_MKK][51] = 127, + [1][1][2][1][RTW89_IC][51] = 127, + [1][1][2][1][RTW89_KCC][51] = 127, + [1][1][2][1][RTW89_ACMA][51] = 127, + [1][1][2][1][RTW89_CN][51] = 127, + [1][1][2][1][RTW89_UK][51] = 127, + [2][0][2][0][RTW89_FCC][3] = 76, + [2][0][2][0][RTW89_ETSI][3] = 64, + [2][0][2][0][RTW89_MKK][3] = 62, + [2][0][2][0][RTW89_IC][3] = 64, + [2][0][2][0][RTW89_KCC][3] = 72, + [2][0][2][0][RTW89_ACMA][3] = 64, + [2][0][2][0][RTW89_CN][3] = 64, + [2][0][2][0][RTW89_UK][3] = 64, + [2][0][2][0][RTW89_FCC][11] = 64, + [2][0][2][0][RTW89_ETSI][11] = 64, + [2][0][2][0][RTW89_MKK][11] = 64, + [2][0][2][0][RTW89_IC][11] = 62, + [2][0][2][0][RTW89_KCC][11] = 72, + [2][0][2][0][RTW89_ACMA][11] = 64, + [2][0][2][0][RTW89_CN][11] = 64, + [2][0][2][0][RTW89_UK][11] = 64, + [2][0][2][0][RTW89_FCC][18] = 66, + [2][0][2][0][RTW89_ETSI][18] = 64, + [2][0][2][0][RTW89_MKK][18] = 72, + [2][0][2][0][RTW89_IC][18] = 66, + [2][0][2][0][RTW89_KCC][18] = 72, + [2][0][2][0][RTW89_ACMA][18] = 64, + [2][0][2][0][RTW89_CN][18] = 127, + [2][0][2][0][RTW89_UK][18] = 64, + [2][0][2][0][RTW89_FCC][26] = 76, + [2][0][2][0][RTW89_ETSI][26] = 64, + [2][0][2][0][RTW89_MKK][26] = 72, + [2][0][2][0][RTW89_IC][26] = 127, + [2][0][2][0][RTW89_KCC][26] = 72, + [2][0][2][0][RTW89_ACMA][26] = 127, + [2][0][2][0][RTW89_CN][26] = 127, + [2][0][2][0][RTW89_UK][26] = 64, + [2][0][2][0][RTW89_FCC][34] = 76, + [2][0][2][0][RTW89_ETSI][34] = 127, + [2][0][2][0][RTW89_MKK][34] = 72, + [2][0][2][0][RTW89_IC][34] = 76, + [2][0][2][0][RTW89_KCC][34] = 72, + [2][0][2][0][RTW89_ACMA][34] = 72, + [2][0][2][0][RTW89_CN][34] = 127, + [2][0][2][0][RTW89_UK][34] = 72, + [2][0][2][0][RTW89_FCC][41] = 76, + [2][0][2][0][RTW89_ETSI][41] = 30, + [2][0][2][0][RTW89_MKK][41] = 127, + [2][0][2][0][RTW89_IC][41] = 76, + [2][0][2][0][RTW89_KCC][41] = 64, + [2][0][2][0][RTW89_ACMA][41] = 72, + [2][0][2][0][RTW89_CN][41] = 72, + [2][0][2][0][RTW89_UK][41] = 64, + [2][0][2][0][RTW89_FCC][49] = 74, + [2][0][2][0][RTW89_ETSI][49] = 127, + [2][0][2][0][RTW89_MKK][49] = 127, + [2][0][2][0][RTW89_IC][49] = 127, + [2][0][2][0][RTW89_KCC][49] = 127, + [2][0][2][0][RTW89_ACMA][49] = 127, + [2][0][2][0][RTW89_CN][49] = 127, + [2][0][2][0][RTW89_UK][49] = 127, + [2][1][2][0][RTW89_FCC][3] = 127, + [2][1][2][0][RTW89_ETSI][3] = 127, + [2][1][2][0][RTW89_MKK][3] = 127, + [2][1][2][0][RTW89_IC][3] = 127, + [2][1][2][0][RTW89_KCC][3] = 127, + [2][1][2][0][RTW89_ACMA][3] = 127, + [2][1][2][0][RTW89_CN][3] = 127, + [2][1][2][0][RTW89_UK][3] = 127, + [2][1][2][0][RTW89_FCC][11] = 127, + [2][1][2][0][RTW89_ETSI][11] = 127, + [2][1][2][0][RTW89_MKK][11] = 127, + [2][1][2][0][RTW89_IC][11] = 127, + [2][1][2][0][RTW89_KCC][11] = 127, + [2][1][2][0][RTW89_ACMA][11] = 127, + [2][1][2][0][RTW89_CN][11] = 127, + [2][1][2][0][RTW89_UK][11] = 127, + [2][1][2][0][RTW89_FCC][18] = 127, + [2][1][2][0][RTW89_ETSI][18] = 127, + [2][1][2][0][RTW89_MKK][18] = 127, + [2][1][2][0][RTW89_IC][18] = 127, + [2][1][2][0][RTW89_KCC][18] = 127, + [2][1][2][0][RTW89_ACMA][18] = 127, + [2][1][2][0][RTW89_CN][18] = 127, + [2][1][2][0][RTW89_UK][18] = 127, + [2][1][2][0][RTW89_FCC][26] = 127, + [2][1][2][0][RTW89_ETSI][26] = 127, + [2][1][2][0][RTW89_MKK][26] = 127, + [2][1][2][0][RTW89_IC][26] = 127, + [2][1][2][0][RTW89_KCC][26] = 127, + [2][1][2][0][RTW89_ACMA][26] = 127, + [2][1][2][0][RTW89_CN][26] = 127, + [2][1][2][0][RTW89_UK][26] = 127, + [2][1][2][0][RTW89_FCC][34] = 127, + [2][1][2][0][RTW89_ETSI][34] = 127, + [2][1][2][0][RTW89_MKK][34] = 127, + [2][1][2][0][RTW89_IC][34] = 127, + [2][1][2][0][RTW89_KCC][34] = 127, + [2][1][2][0][RTW89_ACMA][34] = 127, + [2][1][2][0][RTW89_CN][34] = 127, + [2][1][2][0][RTW89_UK][34] = 127, + [2][1][2][0][RTW89_FCC][41] = 127, + [2][1][2][0][RTW89_ETSI][41] = 127, + [2][1][2][0][RTW89_MKK][41] = 127, + [2][1][2][0][RTW89_IC][41] = 127, + [2][1][2][0][RTW89_KCC][41] = 127, + [2][1][2][0][RTW89_ACMA][41] = 127, + [2][1][2][0][RTW89_CN][41] = 127, + [2][1][2][0][RTW89_UK][41] = 127, + [2][1][2][0][RTW89_FCC][49] = 127, + [2][1][2][0][RTW89_ETSI][49] = 127, + [2][1][2][0][RTW89_MKK][49] = 127, + [2][1][2][0][RTW89_IC][49] = 127, + [2][1][2][0][RTW89_KCC][49] = 127, + [2][1][2][0][RTW89_ACMA][49] = 127, + [2][1][2][0][RTW89_CN][49] = 127, + [2][1][2][0][RTW89_UK][49] = 127, + [2][1][2][1][RTW89_FCC][3] = 127, + [2][1][2][1][RTW89_ETSI][3] = 127, + [2][1][2][1][RTW89_MKK][3] = 127, + [2][1][2][1][RTW89_IC][3] = 127, + [2][1][2][1][RTW89_KCC][3] = 127, + [2][1][2][1][RTW89_ACMA][3] = 127, + [2][1][2][1][RTW89_CN][3] = 127, + [2][1][2][1][RTW89_UK][3] = 127, + [2][1][2][1][RTW89_FCC][11] = 127, + [2][1][2][1][RTW89_ETSI][11] = 127, + [2][1][2][1][RTW89_MKK][11] = 127, + [2][1][2][1][RTW89_IC][11] = 127, + [2][1][2][1][RTW89_KCC][11] = 127, + [2][1][2][1][RTW89_ACMA][11] = 127, + [2][1][2][1][RTW89_CN][11] = 127, + [2][1][2][1][RTW89_UK][11] = 127, + [2][1][2][1][RTW89_FCC][18] = 127, + [2][1][2][1][RTW89_ETSI][18] = 127, + [2][1][2][1][RTW89_MKK][18] = 127, + [2][1][2][1][RTW89_IC][18] = 127, + [2][1][2][1][RTW89_KCC][18] = 127, + [2][1][2][1][RTW89_ACMA][18] = 127, + [2][1][2][1][RTW89_CN][18] = 127, + [2][1][2][1][RTW89_UK][18] = 127, + [2][1][2][1][RTW89_FCC][26] = 127, + [2][1][2][1][RTW89_ETSI][26] = 127, + [2][1][2][1][RTW89_MKK][26] = 127, + [2][1][2][1][RTW89_IC][26] = 127, + [2][1][2][1][RTW89_KCC][26] = 127, + [2][1][2][1][RTW89_ACMA][26] = 127, + [2][1][2][1][RTW89_CN][26] = 127, + [2][1][2][1][RTW89_UK][26] = 127, + [2][1][2][1][RTW89_FCC][34] = 127, + [2][1][2][1][RTW89_ETSI][34] = 127, + [2][1][2][1][RTW89_MKK][34] = 127, + [2][1][2][1][RTW89_IC][34] = 127, + [2][1][2][1][RTW89_KCC][34] = 127, + [2][1][2][1][RTW89_ACMA][34] = 127, + [2][1][2][1][RTW89_CN][34] = 127, + [2][1][2][1][RTW89_UK][34] = 127, + [2][1][2][1][RTW89_FCC][41] = 127, + [2][1][2][1][RTW89_ETSI][41] = 127, + [2][1][2][1][RTW89_MKK][41] = 127, + [2][1][2][1][RTW89_IC][41] = 127, + [2][1][2][1][RTW89_KCC][41] = 127, + [2][1][2][1][RTW89_ACMA][41] = 127, + [2][1][2][1][RTW89_CN][41] = 127, + [2][1][2][1][RTW89_UK][41] = 127, + [2][1][2][1][RTW89_FCC][49] = 127, + [2][1][2][1][RTW89_ETSI][49] = 127, + [2][1][2][1][RTW89_MKK][49] = 127, + [2][1][2][1][RTW89_IC][49] = 127, + [2][1][2][1][RTW89_KCC][49] = 127, + [2][1][2][1][RTW89_ACMA][49] = 127, + [2][1][2][1][RTW89_CN][49] = 127, + [2][1][2][1][RTW89_UK][49] = 127, + [3][0][2][0][RTW89_FCC][7] = 127, + [3][0][2][0][RTW89_ETSI][7] = 127, + [3][0][2][0][RTW89_MKK][7] = 127, + [3][0][2][0][RTW89_IC][7] = 127, + [3][0][2][0][RTW89_KCC][7] = 127, + [3][0][2][0][RTW89_ACMA][7] = 127, + [3][0][2][0][RTW89_CN][7] = 58, + [3][0][2][0][RTW89_UK][7] = 127, + [3][0][2][0][RTW89_FCC][22] = 127, + [3][0][2][0][RTW89_ETSI][22] = 127, + [3][0][2][0][RTW89_MKK][22] = 127, + [3][0][2][0][RTW89_IC][22] = 127, + [3][0][2][0][RTW89_KCC][22] = 127, + [3][0][2][0][RTW89_ACMA][22] = 127, + [3][0][2][0][RTW89_CN][22] = 58, + [3][0][2][0][RTW89_UK][22] = 127, + [3][0][2][0][RTW89_FCC][45] = 127, + [3][0][2][0][RTW89_ETSI][45] = 127, + [3][0][2][0][RTW89_MKK][45] = 127, + [3][0][2][0][RTW89_IC][45] = 127, + [3][0][2][0][RTW89_KCC][45] = 127, + [3][0][2][0][RTW89_ACMA][45] = 127, + [3][0][2][0][RTW89_CN][45] = 127, + [3][0][2][0][RTW89_UK][45] = 127, + [3][1][2][0][RTW89_FCC][7] = 127, + [3][1][2][0][RTW89_ETSI][7] = 127, + [3][1][2][0][RTW89_MKK][7] = 127, + [3][1][2][0][RTW89_IC][7] = 127, + [3][1][2][0][RTW89_KCC][7] = 127, + [3][1][2][0][RTW89_ACMA][7] = 127, + [3][1][2][0][RTW89_CN][7] = 127, + [3][1][2][0][RTW89_UK][7] = 127, + [3][1][2][0][RTW89_FCC][22] = 127, + [3][1][2][0][RTW89_ETSI][22] = 127, + [3][1][2][0][RTW89_MKK][22] = 127, + [3][1][2][0][RTW89_IC][22] = 127, + [3][1][2][0][RTW89_KCC][22] = 127, + [3][1][2][0][RTW89_ACMA][22] = 127, + [3][1][2][0][RTW89_CN][22] = 127, + [3][1][2][0][RTW89_UK][22] = 127, + [3][1][2][0][RTW89_FCC][45] = 127, + [3][1][2][0][RTW89_ETSI][45] = 127, + [3][1][2][0][RTW89_MKK][45] = 127, + [3][1][2][0][RTW89_IC][45] = 127, + [3][1][2][0][RTW89_KCC][45] = 127, + [3][1][2][0][RTW89_ACMA][45] = 127, + [3][1][2][0][RTW89_CN][45] = 127, + [3][1][2][0][RTW89_UK][45] = 127, + [3][1][2][1][RTW89_FCC][7] = 127, + [3][1][2][1][RTW89_ETSI][7] = 127, + [3][1][2][1][RTW89_MKK][7] = 127, + [3][1][2][1][RTW89_IC][7] = 127, + [3][1][2][1][RTW89_KCC][7] = 127, + [3][1][2][1][RTW89_ACMA][7] = 127, + [3][1][2][1][RTW89_CN][7] = 127, + [3][1][2][1][RTW89_UK][7] = 127, + [3][1][2][1][RTW89_FCC][22] = 127, + [3][1][2][1][RTW89_ETSI][22] = 127, + [3][1][2][1][RTW89_MKK][22] = 127, + [3][1][2][1][RTW89_IC][22] = 127, + [3][1][2][1][RTW89_KCC][22] = 127, + [3][1][2][1][RTW89_ACMA][22] = 127, + [3][1][2][1][RTW89_CN][22] = 127, + [3][1][2][1][RTW89_UK][22] = 127, + [3][1][2][1][RTW89_FCC][45] = 127, + [3][1][2][1][RTW89_ETSI][45] = 127, + [3][1][2][1][RTW89_MKK][45] = 127, + [3][1][2][1][RTW89_IC][45] = 127, + [3][1][2][1][RTW89_KCC][45] = 127, + [3][1][2][1][RTW89_ACMA][45] = 127, + [3][1][2][1][RTW89_CN][45] = 127, + [3][1][2][1][RTW89_UK][45] = 127, +}; + +static +const s8 rtw89_8851b_txpwr_lmt_ru_2g[RTW89_RU_NUM][RTW89_NTX_NUM] + [RTW89_REGD_NUM][RTW89_2G_CH_NUM] = { + [0][0][RTW89_WW][0] = 30, + [0][0][RTW89_WW][1] = 30, + [0][0][RTW89_WW][2] = 30, + [0][0][RTW89_WW][3] = 30, + [0][0][RTW89_WW][4] = 30, + [0][0][RTW89_WW][5] = 30, + [0][0][RTW89_WW][6] = 30, + [0][0][RTW89_WW][7] = 30, + [0][0][RTW89_WW][8] = 30, + [0][0][RTW89_WW][9] = 30, + [0][0][RTW89_WW][10] = 30, + [0][0][RTW89_WW][11] = 30, + [0][0][RTW89_WW][12] = 30, + [0][0][RTW89_WW][13] = 0, + [0][1][RTW89_WW][0] = 20, + [0][1][RTW89_WW][1] = 22, + [0][1][RTW89_WW][2] = 22, + [0][1][RTW89_WW][3] = 22, + [0][1][RTW89_WW][4] = 22, + [0][1][RTW89_WW][5] = 22, + [0][1][RTW89_WW][6] = 22, + [0][1][RTW89_WW][7] = 22, + [0][1][RTW89_WW][8] = 22, + [0][1][RTW89_WW][9] = 22, + [0][1][RTW89_WW][10] = 22, + [0][1][RTW89_WW][11] = 22, + [0][1][RTW89_WW][12] = 20, + [0][1][RTW89_WW][13] = 0, + [1][0][RTW89_WW][0] = 42, + [1][0][RTW89_WW][1] = 42, + [1][0][RTW89_WW][2] = 42, + [1][0][RTW89_WW][3] = 42, + [1][0][RTW89_WW][4] = 42, + [1][0][RTW89_WW][5] = 42, + [1][0][RTW89_WW][6] = 42, + [1][0][RTW89_WW][7] = 42, + [1][0][RTW89_WW][8] = 42, + [1][0][RTW89_WW][9] = 42, + [1][0][RTW89_WW][10] = 42, + [1][0][RTW89_WW][11] = 42, + [1][0][RTW89_WW][12] = 34, + [1][0][RTW89_WW][13] = 0, + [1][1][RTW89_WW][0] = 32, + [1][1][RTW89_WW][1] = 32, + [1][1][RTW89_WW][2] = 32, + [1][1][RTW89_WW][3] = 32, + [1][1][RTW89_WW][4] = 32, + [1][1][RTW89_WW][5] = 32, + [1][1][RTW89_WW][6] = 32, + [1][1][RTW89_WW][7] = 32, + [1][1][RTW89_WW][8] = 32, + [1][1][RTW89_WW][9] = 32, + [1][1][RTW89_WW][10] = 32, + [1][1][RTW89_WW][11] = 32, + [1][1][RTW89_WW][12] = 32, + [1][1][RTW89_WW][13] = 0, + [2][0][RTW89_WW][0] = 54, + [2][0][RTW89_WW][1] = 54, + [2][0][RTW89_WW][2] = 54, + [2][0][RTW89_WW][3] = 54, + [2][0][RTW89_WW][4] = 54, + [2][0][RTW89_WW][5] = 54, + [2][0][RTW89_WW][6] = 54, + [2][0][RTW89_WW][7] = 54, + [2][0][RTW89_WW][8] = 54, + [2][0][RTW89_WW][9] = 54, + [2][0][RTW89_WW][10] = 54, + [2][0][RTW89_WW][11] = 54, + [2][0][RTW89_WW][12] = 34, + [2][0][RTW89_WW][13] = 0, + [2][1][RTW89_WW][0] = 44, + [2][1][RTW89_WW][1] = 44, + [2][1][RTW89_WW][2] = 44, + [2][1][RTW89_WW][3] = 44, + [2][1][RTW89_WW][4] = 44, + [2][1][RTW89_WW][5] = 44, + [2][1][RTW89_WW][6] = 44, + [2][1][RTW89_WW][7] = 44, + [2][1][RTW89_WW][8] = 44, + [2][1][RTW89_WW][9] = 44, + [2][1][RTW89_WW][10] = 44, + [2][1][RTW89_WW][11] = 44, + [2][1][RTW89_WW][12] = 42, + [2][1][RTW89_WW][13] = 0, + [0][0][RTW89_FCC][0] = 62, + [0][0][RTW89_ETSI][0] = 30, + [0][0][RTW89_MKK][0] = 40, + [0][0][RTW89_IC][0] = 62, + [0][0][RTW89_KCC][0] = 46, + [0][0][RTW89_ACMA][0] = 30, + [0][0][RTW89_CN][0] = 32, + [0][0][RTW89_UK][0] = 30, + [0][0][RTW89_FCC][1] = 62, + [0][0][RTW89_ETSI][1] = 30, + [0][0][RTW89_MKK][1] = 44, + [0][0][RTW89_IC][1] = 62, + [0][0][RTW89_KCC][1] = 46, + [0][0][RTW89_ACMA][1] = 30, + [0][0][RTW89_CN][1] = 32, + [0][0][RTW89_UK][1] = 30, + [0][0][RTW89_FCC][2] = 66, + [0][0][RTW89_ETSI][2] = 30, + [0][0][RTW89_MKK][2] = 44, + [0][0][RTW89_IC][2] = 66, + [0][0][RTW89_KCC][2] = 46, + [0][0][RTW89_ACMA][2] = 30, + [0][0][RTW89_CN][2] = 32, + [0][0][RTW89_UK][2] = 30, + [0][0][RTW89_FCC][3] = 70, + [0][0][RTW89_ETSI][3] = 30, + [0][0][RTW89_MKK][3] = 44, + [0][0][RTW89_IC][3] = 70, + [0][0][RTW89_KCC][3] = 46, + [0][0][RTW89_ACMA][3] = 30, + [0][0][RTW89_CN][3] = 32, + [0][0][RTW89_UK][3] = 30, + [0][0][RTW89_FCC][4] = 70, + [0][0][RTW89_ETSI][4] = 30, + [0][0][RTW89_MKK][4] = 44, + [0][0][RTW89_IC][4] = 70, + [0][0][RTW89_KCC][4] = 48, + [0][0][RTW89_ACMA][4] = 30, + [0][0][RTW89_CN][4] = 32, + [0][0][RTW89_UK][4] = 30, + [0][0][RTW89_FCC][5] = 84, + [0][0][RTW89_ETSI][5] = 30, + [0][0][RTW89_MKK][5] = 44, + [0][0][RTW89_IC][5] = 84, + [0][0][RTW89_KCC][5] = 48, + [0][0][RTW89_ACMA][5] = 30, + [0][0][RTW89_CN][5] = 32, + [0][0][RTW89_UK][5] = 30, + [0][0][RTW89_FCC][6] = 66, + [0][0][RTW89_ETSI][6] = 30, + [0][0][RTW89_MKK][6] = 44, + [0][0][RTW89_IC][6] = 66, + [0][0][RTW89_KCC][6] = 48, + [0][0][RTW89_ACMA][6] = 30, + [0][0][RTW89_CN][6] = 32, + [0][0][RTW89_UK][6] = 30, + [0][0][RTW89_FCC][7] = 66, + [0][0][RTW89_ETSI][7] = 30, + [0][0][RTW89_MKK][7] = 44, + [0][0][RTW89_IC][7] = 66, + [0][0][RTW89_KCC][7] = 48, + [0][0][RTW89_ACMA][7] = 30, + [0][0][RTW89_CN][7] = 32, + [0][0][RTW89_UK][7] = 30, + [0][0][RTW89_FCC][8] = 62, + [0][0][RTW89_ETSI][8] = 30, + [0][0][RTW89_MKK][8] = 44, + [0][0][RTW89_IC][8] = 62, + [0][0][RTW89_KCC][8] = 48, + [0][0][RTW89_ACMA][8] = 30, + [0][0][RTW89_CN][8] = 32, + [0][0][RTW89_UK][8] = 30, + [0][0][RTW89_FCC][9] = 58, + [0][0][RTW89_ETSI][9] = 30, + [0][0][RTW89_MKK][9] = 44, + [0][0][RTW89_IC][9] = 58, + [0][0][RTW89_KCC][9] = 44, + [0][0][RTW89_ACMA][9] = 30, + [0][0][RTW89_CN][9] = 32, + [0][0][RTW89_UK][9] = 30, + [0][0][RTW89_FCC][10] = 58, + [0][0][RTW89_ETSI][10] = 30, + [0][0][RTW89_MKK][10] = 44, + [0][0][RTW89_IC][10] = 58, + [0][0][RTW89_KCC][10] = 44, + [0][0][RTW89_ACMA][10] = 30, + [0][0][RTW89_CN][10] = 32, + [0][0][RTW89_UK][10] = 30, + [0][0][RTW89_FCC][11] = 54, + [0][0][RTW89_ETSI][11] = 30, + [0][0][RTW89_MKK][11] = 44, + [0][0][RTW89_IC][11] = 54, + [0][0][RTW89_KCC][11] = 44, + [0][0][RTW89_ACMA][11] = 30, + [0][0][RTW89_CN][11] = 32, + [0][0][RTW89_UK][11] = 30, + [0][0][RTW89_FCC][12] = 36, + [0][0][RTW89_ETSI][12] = 30, + [0][0][RTW89_MKK][12] = 40, + [0][0][RTW89_IC][12] = 36, + [0][0][RTW89_KCC][12] = 44, + [0][0][RTW89_ACMA][12] = 30, + [0][0][RTW89_CN][12] = 32, + [0][0][RTW89_UK][12] = 30, + [0][0][RTW89_FCC][13] = 127, + [0][0][RTW89_ETSI][13] = 127, + [0][0][RTW89_MKK][13] = 127, + [0][0][RTW89_IC][13] = 127, + [0][0][RTW89_KCC][13] = 127, + [0][0][RTW89_ACMA][13] = 127, + [0][0][RTW89_CN][13] = 127, + [0][0][RTW89_UK][13] = 127, + [0][1][RTW89_FCC][0] = 127, + [0][1][RTW89_ETSI][0] = 127, + [0][1][RTW89_MKK][0] = 127, + [0][1][RTW89_IC][0] = 127, + [0][1][RTW89_KCC][0] = 127, + [0][1][RTW89_ACMA][0] = 127, + [0][1][RTW89_CN][0] = 20, + [0][1][RTW89_UK][0] = 127, + [0][1][RTW89_FCC][1] = 127, + [0][1][RTW89_ETSI][1] = 127, + [0][1][RTW89_MKK][1] = 127, + [0][1][RTW89_IC][1] = 127, + [0][1][RTW89_KCC][1] = 127, + [0][1][RTW89_ACMA][1] = 127, + [0][1][RTW89_CN][1] = 22, + [0][1][RTW89_UK][1] = 127, + [0][1][RTW89_FCC][2] = 127, + [0][1][RTW89_ETSI][2] = 127, + [0][1][RTW89_MKK][2] = 127, + [0][1][RTW89_IC][2] = 127, + [0][1][RTW89_KCC][2] = 127, + [0][1][RTW89_ACMA][2] = 127, + [0][1][RTW89_CN][2] = 22, + [0][1][RTW89_UK][2] = 127, + [0][1][RTW89_FCC][3] = 127, + [0][1][RTW89_ETSI][3] = 127, + [0][1][RTW89_MKK][3] = 127, + [0][1][RTW89_IC][3] = 127, + [0][1][RTW89_KCC][3] = 127, + [0][1][RTW89_ACMA][3] = 127, + [0][1][RTW89_CN][3] = 22, + [0][1][RTW89_UK][3] = 127, + [0][1][RTW89_FCC][4] = 127, + [0][1][RTW89_ETSI][4] = 127, + [0][1][RTW89_MKK][4] = 127, + [0][1][RTW89_IC][4] = 127, + [0][1][RTW89_KCC][4] = 127, + [0][1][RTW89_ACMA][4] = 127, + [0][1][RTW89_CN][4] = 22, + [0][1][RTW89_UK][4] = 127, + [0][1][RTW89_FCC][5] = 127, + [0][1][RTW89_ETSI][5] = 127, + [0][1][RTW89_MKK][5] = 127, + [0][1][RTW89_IC][5] = 127, + [0][1][RTW89_KCC][5] = 127, + [0][1][RTW89_ACMA][5] = 127, + [0][1][RTW89_CN][5] = 22, + [0][1][RTW89_UK][5] = 127, + [0][1][RTW89_FCC][6] = 127, + [0][1][RTW89_ETSI][6] = 127, + [0][1][RTW89_MKK][6] = 127, + [0][1][RTW89_IC][6] = 127, + [0][1][RTW89_KCC][6] = 127, + [0][1][RTW89_ACMA][6] = 127, + [0][1][RTW89_CN][6] = 22, + [0][1][RTW89_UK][6] = 127, + [0][1][RTW89_FCC][7] = 127, + [0][1][RTW89_ETSI][7] = 127, + [0][1][RTW89_MKK][7] = 127, + [0][1][RTW89_IC][7] = 127, + [0][1][RTW89_KCC][7] = 127, + [0][1][RTW89_ACMA][7] = 127, + [0][1][RTW89_CN][7] = 22, + [0][1][RTW89_UK][7] = 127, + [0][1][RTW89_FCC][8] = 127, + [0][1][RTW89_ETSI][8] = 127, + [0][1][RTW89_MKK][8] = 127, + [0][1][RTW89_IC][8] = 127, + [0][1][RTW89_KCC][8] = 127, + [0][1][RTW89_ACMA][8] = 127, + [0][1][RTW89_CN][8] = 22, + [0][1][RTW89_UK][8] = 127, + [0][1][RTW89_FCC][9] = 127, + [0][1][RTW89_ETSI][9] = 127, + [0][1][RTW89_MKK][9] = 127, + [0][1][RTW89_IC][9] = 127, + [0][1][RTW89_KCC][9] = 127, + [0][1][RTW89_ACMA][9] = 127, + [0][1][RTW89_CN][9] = 22, + [0][1][RTW89_UK][9] = 127, + [0][1][RTW89_FCC][10] = 127, + [0][1][RTW89_ETSI][10] = 127, + [0][1][RTW89_MKK][10] = 127, + [0][1][RTW89_IC][10] = 127, + [0][1][RTW89_KCC][10] = 127, + [0][1][RTW89_ACMA][10] = 127, + [0][1][RTW89_CN][10] = 22, + [0][1][RTW89_UK][10] = 127, + [0][1][RTW89_FCC][11] = 127, + [0][1][RTW89_ETSI][11] = 127, + [0][1][RTW89_MKK][11] = 127, + [0][1][RTW89_IC][11] = 127, + [0][1][RTW89_KCC][11] = 127, + [0][1][RTW89_ACMA][11] = 127, + [0][1][RTW89_CN][11] = 22, + [0][1][RTW89_UK][11] = 127, + [0][1][RTW89_FCC][12] = 127, + [0][1][RTW89_ETSI][12] = 127, + [0][1][RTW89_MKK][12] = 127, + [0][1][RTW89_IC][12] = 127, + [0][1][RTW89_KCC][12] = 127, + [0][1][RTW89_ACMA][12] = 127, + [0][1][RTW89_CN][12] = 20, + [0][1][RTW89_UK][12] = 127, + [0][1][RTW89_FCC][13] = 127, + [0][1][RTW89_ETSI][13] = 127, + [0][1][RTW89_MKK][13] = 127, + [0][1][RTW89_IC][13] = 127, + [0][1][RTW89_KCC][13] = 127, + [0][1][RTW89_ACMA][13] = 127, + [0][1][RTW89_CN][13] = 127, + [0][1][RTW89_UK][13] = 127, + [1][0][RTW89_FCC][0] = 70, + [1][0][RTW89_ETSI][0] = 42, + [1][0][RTW89_MKK][0] = 52, + [1][0][RTW89_IC][0] = 70, + [1][0][RTW89_KCC][0] = 56, + [1][0][RTW89_ACMA][0] = 42, + [1][0][RTW89_CN][0] = 42, + [1][0][RTW89_UK][0] = 42, + [1][0][RTW89_FCC][1] = 70, + [1][0][RTW89_ETSI][1] = 42, + [1][0][RTW89_MKK][1] = 52, + [1][0][RTW89_IC][1] = 70, + [1][0][RTW89_KCC][1] = 56, + [1][0][RTW89_ACMA][1] = 42, + [1][0][RTW89_CN][1] = 44, + [1][0][RTW89_UK][1] = 42, + [1][0][RTW89_FCC][2] = 74, + [1][0][RTW89_ETSI][2] = 42, + [1][0][RTW89_MKK][2] = 52, + [1][0][RTW89_IC][2] = 74, + [1][0][RTW89_KCC][2] = 56, + [1][0][RTW89_ACMA][2] = 42, + [1][0][RTW89_CN][2] = 44, + [1][0][RTW89_UK][2] = 42, + [1][0][RTW89_FCC][3] = 76, + [1][0][RTW89_ETSI][3] = 42, + [1][0][RTW89_MKK][3] = 52, + [1][0][RTW89_IC][3] = 76, + [1][0][RTW89_KCC][3] = 56, + [1][0][RTW89_ACMA][3] = 42, + [1][0][RTW89_CN][3] = 44, + [1][0][RTW89_UK][3] = 42, + [1][0][RTW89_FCC][4] = 76, + [1][0][RTW89_ETSI][4] = 42, + [1][0][RTW89_MKK][4] = 52, + [1][0][RTW89_IC][4] = 76, + [1][0][RTW89_KCC][4] = 56, + [1][0][RTW89_ACMA][4] = 42, + [1][0][RTW89_CN][4] = 44, + [1][0][RTW89_UK][4] = 42, + [1][0][RTW89_FCC][5] = 82, + [1][0][RTW89_ETSI][5] = 42, + [1][0][RTW89_MKK][5] = 52, + [1][0][RTW89_IC][5] = 82, + [1][0][RTW89_KCC][5] = 56, + [1][0][RTW89_ACMA][5] = 42, + [1][0][RTW89_CN][5] = 44, + [1][0][RTW89_UK][5] = 42, + [1][0][RTW89_FCC][6] = 74, + [1][0][RTW89_ETSI][6] = 42, + [1][0][RTW89_MKK][6] = 52, + [1][0][RTW89_IC][6] = 74, + [1][0][RTW89_KCC][6] = 56, + [1][0][RTW89_ACMA][6] = 42, + [1][0][RTW89_CN][6] = 44, + [1][0][RTW89_UK][6] = 42, + [1][0][RTW89_FCC][7] = 74, + [1][0][RTW89_ETSI][7] = 42, + [1][0][RTW89_MKK][7] = 52, + [1][0][RTW89_IC][7] = 74, + [1][0][RTW89_KCC][7] = 56, + [1][0][RTW89_ACMA][7] = 42, + [1][0][RTW89_CN][7] = 44, + [1][0][RTW89_UK][7] = 42, + [1][0][RTW89_FCC][8] = 74, + [1][0][RTW89_ETSI][8] = 42, + [1][0][RTW89_MKK][8] = 52, + [1][0][RTW89_IC][8] = 74, + [1][0][RTW89_KCC][8] = 56, + [1][0][RTW89_ACMA][8] = 42, + [1][0][RTW89_CN][8] = 44, + [1][0][RTW89_UK][8] = 42, + [1][0][RTW89_FCC][9] = 70, + [1][0][RTW89_ETSI][9] = 42, + [1][0][RTW89_MKK][9] = 52, + [1][0][RTW89_IC][9] = 70, + [1][0][RTW89_KCC][9] = 58, + [1][0][RTW89_ACMA][9] = 42, + [1][0][RTW89_CN][9] = 44, + [1][0][RTW89_UK][9] = 42, + [1][0][RTW89_FCC][10] = 70, + [1][0][RTW89_ETSI][10] = 42, + [1][0][RTW89_MKK][10] = 52, + [1][0][RTW89_IC][10] = 70, + [1][0][RTW89_KCC][10] = 58, + [1][0][RTW89_ACMA][10] = 42, + [1][0][RTW89_CN][10] = 44, + [1][0][RTW89_UK][10] = 42, + [1][0][RTW89_FCC][11] = 66, + [1][0][RTW89_ETSI][11] = 42, + [1][0][RTW89_MKK][11] = 52, + [1][0][RTW89_IC][11] = 66, + [1][0][RTW89_KCC][11] = 58, + [1][0][RTW89_ACMA][11] = 42, + [1][0][RTW89_CN][11] = 44, + [1][0][RTW89_UK][11] = 42, + [1][0][RTW89_FCC][12] = 34, + [1][0][RTW89_ETSI][12] = 42, + [1][0][RTW89_MKK][12] = 52, + [1][0][RTW89_IC][12] = 34, + [1][0][RTW89_KCC][12] = 58, + [1][0][RTW89_ACMA][12] = 42, + [1][0][RTW89_CN][12] = 42, + [1][0][RTW89_UK][12] = 42, + [1][0][RTW89_FCC][13] = 127, + [1][0][RTW89_ETSI][13] = 127, + [1][0][RTW89_MKK][13] = 127, + [1][0][RTW89_IC][13] = 127, + [1][0][RTW89_KCC][13] = 127, + [1][0][RTW89_ACMA][13] = 127, + [1][0][RTW89_CN][13] = 127, + [1][0][RTW89_UK][13] = 127, + [1][1][RTW89_FCC][0] = 127, + [1][1][RTW89_ETSI][0] = 127, + [1][1][RTW89_MKK][0] = 127, + [1][1][RTW89_IC][0] = 127, + [1][1][RTW89_KCC][0] = 127, + [1][1][RTW89_ACMA][0] = 127, + [1][1][RTW89_CN][0] = 32, + [1][1][RTW89_UK][0] = 127, + [1][1][RTW89_FCC][1] = 127, + [1][1][RTW89_ETSI][1] = 127, + [1][1][RTW89_MKK][1] = 127, + [1][1][RTW89_IC][1] = 127, + [1][1][RTW89_KCC][1] = 127, + [1][1][RTW89_ACMA][1] = 127, + [1][1][RTW89_CN][1] = 32, + [1][1][RTW89_UK][1] = 127, + [1][1][RTW89_FCC][2] = 127, + [1][1][RTW89_ETSI][2] = 127, + [1][1][RTW89_MKK][2] = 127, + [1][1][RTW89_IC][2] = 127, + [1][1][RTW89_KCC][2] = 127, + [1][1][RTW89_ACMA][2] = 127, + [1][1][RTW89_CN][2] = 32, + [1][1][RTW89_UK][2] = 127, + [1][1][RTW89_FCC][3] = 127, + [1][1][RTW89_ETSI][3] = 127, + [1][1][RTW89_MKK][3] = 127, + [1][1][RTW89_IC][3] = 127, + [1][1][RTW89_KCC][3] = 127, + [1][1][RTW89_ACMA][3] = 127, + [1][1][RTW89_CN][3] = 32, + [1][1][RTW89_UK][3] = 127, + [1][1][RTW89_FCC][4] = 127, + [1][1][RTW89_ETSI][4] = 127, + [1][1][RTW89_MKK][4] = 127, + [1][1][RTW89_IC][4] = 127, + [1][1][RTW89_KCC][4] = 127, + [1][1][RTW89_ACMA][4] = 127, + [1][1][RTW89_CN][4] = 32, + [1][1][RTW89_UK][4] = 127, + [1][1][RTW89_FCC][5] = 127, + [1][1][RTW89_ETSI][5] = 127, + [1][1][RTW89_MKK][5] = 127, + [1][1][RTW89_IC][5] = 127, + [1][1][RTW89_KCC][5] = 127, + [1][1][RTW89_ACMA][5] = 127, + [1][1][RTW89_CN][5] = 32, + [1][1][RTW89_UK][5] = 127, + [1][1][RTW89_FCC][6] = 127, + [1][1][RTW89_ETSI][6] = 127, + [1][1][RTW89_MKK][6] = 127, + [1][1][RTW89_IC][6] = 127, + [1][1][RTW89_KCC][6] = 127, + [1][1][RTW89_ACMA][6] = 127, + [1][1][RTW89_CN][6] = 32, + [1][1][RTW89_UK][6] = 127, + [1][1][RTW89_FCC][7] = 127, + [1][1][RTW89_ETSI][7] = 127, + [1][1][RTW89_MKK][7] = 127, + [1][1][RTW89_IC][7] = 127, + [1][1][RTW89_KCC][7] = 127, + [1][1][RTW89_ACMA][7] = 127, + [1][1][RTW89_CN][7] = 32, + [1][1][RTW89_UK][7] = 127, + [1][1][RTW89_FCC][8] = 127, + [1][1][RTW89_ETSI][8] = 127, + [1][1][RTW89_MKK][8] = 127, + [1][1][RTW89_IC][8] = 127, + [1][1][RTW89_KCC][8] = 127, + [1][1][RTW89_ACMA][8] = 127, + [1][1][RTW89_CN][8] = 32, + [1][1][RTW89_UK][8] = 127, + [1][1][RTW89_FCC][9] = 127, + [1][1][RTW89_ETSI][9] = 127, + [1][1][RTW89_MKK][9] = 127, + [1][1][RTW89_IC][9] = 127, + [1][1][RTW89_KCC][9] = 127, + [1][1][RTW89_ACMA][9] = 127, + [1][1][RTW89_CN][9] = 32, + [1][1][RTW89_UK][9] = 127, + [1][1][RTW89_FCC][10] = 127, + [1][1][RTW89_ETSI][10] = 127, + [1][1][RTW89_MKK][10] = 127, + [1][1][RTW89_IC][10] = 127, + [1][1][RTW89_KCC][10] = 127, + [1][1][RTW89_ACMA][10] = 127, + [1][1][RTW89_CN][10] = 32, + [1][1][RTW89_UK][10] = 127, + [1][1][RTW89_FCC][11] = 127, + [1][1][RTW89_ETSI][11] = 127, + [1][1][RTW89_MKK][11] = 127, + [1][1][RTW89_IC][11] = 127, + [1][1][RTW89_KCC][11] = 127, + [1][1][RTW89_ACMA][11] = 127, + [1][1][RTW89_CN][11] = 32, + [1][1][RTW89_UK][11] = 127, + [1][1][RTW89_FCC][12] = 127, + [1][1][RTW89_ETSI][12] = 127, + [1][1][RTW89_MKK][12] = 127, + [1][1][RTW89_IC][12] = 127, + [1][1][RTW89_KCC][12] = 127, + [1][1][RTW89_ACMA][12] = 127, + [1][1][RTW89_CN][12] = 32, + [1][1][RTW89_UK][12] = 127, + [1][1][RTW89_FCC][13] = 127, + [1][1][RTW89_ETSI][13] = 127, + [1][1][RTW89_MKK][13] = 127, + [1][1][RTW89_IC][13] = 127, + [1][1][RTW89_KCC][13] = 127, + [1][1][RTW89_ACMA][13] = 127, + [1][1][RTW89_CN][13] = 127, + [1][1][RTW89_UK][13] = 127, + [2][0][RTW89_FCC][0] = 76, + [2][0][RTW89_ETSI][0] = 54, + [2][0][RTW89_MKK][0] = 64, + [2][0][RTW89_IC][0] = 76, + [2][0][RTW89_KCC][0] = 68, + [2][0][RTW89_ACMA][0] = 54, + [2][0][RTW89_CN][0] = 56, + [2][0][RTW89_UK][0] = 54, + [2][0][RTW89_FCC][1] = 76, + [2][0][RTW89_ETSI][1] = 54, + [2][0][RTW89_MKK][1] = 64, + [2][0][RTW89_IC][1] = 76, + [2][0][RTW89_KCC][1] = 68, + [2][0][RTW89_ACMA][1] = 54, + [2][0][RTW89_CN][1] = 56, + [2][0][RTW89_UK][1] = 54, + [2][0][RTW89_FCC][2] = 78, + [2][0][RTW89_ETSI][2] = 54, + [2][0][RTW89_MKK][2] = 64, + [2][0][RTW89_IC][2] = 78, + [2][0][RTW89_KCC][2] = 68, + [2][0][RTW89_ACMA][2] = 54, + [2][0][RTW89_CN][2] = 56, + [2][0][RTW89_UK][2] = 54, + [2][0][RTW89_FCC][3] = 78, + [2][0][RTW89_ETSI][3] = 54, + [2][0][RTW89_MKK][3] = 64, + [2][0][RTW89_IC][3] = 78, + [2][0][RTW89_KCC][3] = 68, + [2][0][RTW89_ACMA][3] = 54, + [2][0][RTW89_CN][3] = 56, + [2][0][RTW89_UK][3] = 54, + [2][0][RTW89_FCC][4] = 78, + [2][0][RTW89_ETSI][4] = 54, + [2][0][RTW89_MKK][4] = 64, + [2][0][RTW89_IC][4] = 78, + [2][0][RTW89_KCC][4] = 68, + [2][0][RTW89_ACMA][4] = 54, + [2][0][RTW89_CN][4] = 56, + [2][0][RTW89_UK][4] = 54, + [2][0][RTW89_FCC][5] = 82, + [2][0][RTW89_ETSI][5] = 54, + [2][0][RTW89_MKK][5] = 64, + [2][0][RTW89_IC][5] = 82, + [2][0][RTW89_KCC][5] = 68, + [2][0][RTW89_ACMA][5] = 54, + [2][0][RTW89_CN][5] = 56, + [2][0][RTW89_UK][5] = 54, + [2][0][RTW89_FCC][6] = 74, + [2][0][RTW89_ETSI][6] = 54, + [2][0][RTW89_MKK][6] = 64, + [2][0][RTW89_IC][6] = 74, + [2][0][RTW89_KCC][6] = 68, + [2][0][RTW89_ACMA][6] = 54, + [2][0][RTW89_CN][6] = 56, + [2][0][RTW89_UK][6] = 54, + [2][0][RTW89_FCC][7] = 74, + [2][0][RTW89_ETSI][7] = 54, + [2][0][RTW89_MKK][7] = 64, + [2][0][RTW89_IC][7] = 74, + [2][0][RTW89_KCC][7] = 68, + [2][0][RTW89_ACMA][7] = 54, + [2][0][RTW89_CN][7] = 56, + [2][0][RTW89_UK][7] = 54, + [2][0][RTW89_FCC][8] = 74, + [2][0][RTW89_ETSI][8] = 54, + [2][0][RTW89_MKK][8] = 64, + [2][0][RTW89_IC][8] = 74, + [2][0][RTW89_KCC][8] = 68, + [2][0][RTW89_ACMA][8] = 54, + [2][0][RTW89_CN][8] = 56, + [2][0][RTW89_UK][8] = 54, + [2][0][RTW89_FCC][9] = 72, + [2][0][RTW89_ETSI][9] = 54, + [2][0][RTW89_MKK][9] = 64, + [2][0][RTW89_IC][9] = 72, + [2][0][RTW89_KCC][9] = 68, + [2][0][RTW89_ACMA][9] = 54, + [2][0][RTW89_CN][9] = 56, + [2][0][RTW89_UK][9] = 54, + [2][0][RTW89_FCC][10] = 72, + [2][0][RTW89_ETSI][10] = 54, + [2][0][RTW89_MKK][10] = 64, + [2][0][RTW89_IC][10] = 72, + [2][0][RTW89_KCC][10] = 68, + [2][0][RTW89_ACMA][10] = 54, + [2][0][RTW89_CN][10] = 56, + [2][0][RTW89_UK][10] = 54, + [2][0][RTW89_FCC][11] = 64, + [2][0][RTW89_ETSI][11] = 54, + [2][0][RTW89_MKK][11] = 64, + [2][0][RTW89_IC][11] = 64, + [2][0][RTW89_KCC][11] = 68, + [2][0][RTW89_ACMA][11] = 54, + [2][0][RTW89_CN][11] = 56, + [2][0][RTW89_UK][11] = 54, + [2][0][RTW89_FCC][12] = 34, + [2][0][RTW89_ETSI][12] = 54, + [2][0][RTW89_MKK][12] = 64, + [2][0][RTW89_IC][12] = 34, + [2][0][RTW89_KCC][12] = 68, + [2][0][RTW89_ACMA][12] = 54, + [2][0][RTW89_CN][12] = 56, + [2][0][RTW89_UK][12] = 54, + [2][0][RTW89_FCC][13] = 127, + [2][0][RTW89_ETSI][13] = 127, + [2][0][RTW89_MKK][13] = 127, + [2][0][RTW89_IC][13] = 127, + [2][0][RTW89_KCC][13] = 127, + [2][0][RTW89_ACMA][13] = 127, + [2][0][RTW89_CN][13] = 127, + [2][0][RTW89_UK][13] = 127, + [2][1][RTW89_FCC][0] = 127, + [2][1][RTW89_ETSI][0] = 127, + [2][1][RTW89_MKK][0] = 127, + [2][1][RTW89_IC][0] = 127, + [2][1][RTW89_KCC][0] = 127, + [2][1][RTW89_ACMA][0] = 127, + [2][1][RTW89_CN][0] = 44, + [2][1][RTW89_UK][0] = 127, + [2][1][RTW89_FCC][1] = 127, + [2][1][RTW89_ETSI][1] = 127, + [2][1][RTW89_MKK][1] = 127, + [2][1][RTW89_IC][1] = 127, + [2][1][RTW89_KCC][1] = 127, + [2][1][RTW89_ACMA][1] = 127, + [2][1][RTW89_CN][1] = 44, + [2][1][RTW89_UK][1] = 127, + [2][1][RTW89_FCC][2] = 127, + [2][1][RTW89_ETSI][2] = 127, + [2][1][RTW89_MKK][2] = 127, + [2][1][RTW89_IC][2] = 127, + [2][1][RTW89_KCC][2] = 127, + [2][1][RTW89_ACMA][2] = 127, + [2][1][RTW89_CN][2] = 44, + [2][1][RTW89_UK][2] = 127, + [2][1][RTW89_FCC][3] = 127, + [2][1][RTW89_ETSI][3] = 127, + [2][1][RTW89_MKK][3] = 127, + [2][1][RTW89_IC][3] = 127, + [2][1][RTW89_KCC][3] = 127, + [2][1][RTW89_ACMA][3] = 127, + [2][1][RTW89_CN][3] = 44, + [2][1][RTW89_UK][3] = 127, + [2][1][RTW89_FCC][4] = 127, + [2][1][RTW89_ETSI][4] = 127, + [2][1][RTW89_MKK][4] = 127, + [2][1][RTW89_IC][4] = 127, + [2][1][RTW89_KCC][4] = 127, + [2][1][RTW89_ACMA][4] = 127, + [2][1][RTW89_CN][4] = 44, + [2][1][RTW89_UK][4] = 127, + [2][1][RTW89_FCC][5] = 127, + [2][1][RTW89_ETSI][5] = 127, + [2][1][RTW89_MKK][5] = 127, + [2][1][RTW89_IC][5] = 127, + [2][1][RTW89_KCC][5] = 127, + [2][1][RTW89_ACMA][5] = 127, + [2][1][RTW89_CN][5] = 44, + [2][1][RTW89_UK][5] = 127, + [2][1][RTW89_FCC][6] = 127, + [2][1][RTW89_ETSI][6] = 127, + [2][1][RTW89_MKK][6] = 127, + [2][1][RTW89_IC][6] = 127, + [2][1][RTW89_KCC][6] = 127, + [2][1][RTW89_ACMA][6] = 127, + [2][1][RTW89_CN][6] = 44, + [2][1][RTW89_UK][6] = 127, + [2][1][RTW89_FCC][7] = 127, + [2][1][RTW89_ETSI][7] = 127, + [2][1][RTW89_MKK][7] = 127, + [2][1][RTW89_IC][7] = 127, + [2][1][RTW89_KCC][7] = 127, + [2][1][RTW89_ACMA][7] = 127, + [2][1][RTW89_CN][7] = 44, + [2][1][RTW89_UK][7] = 127, + [2][1][RTW89_FCC][8] = 127, + [2][1][RTW89_ETSI][8] = 127, + [2][1][RTW89_MKK][8] = 127, + [2][1][RTW89_IC][8] = 127, + [2][1][RTW89_KCC][8] = 127, + [2][1][RTW89_ACMA][8] = 127, + [2][1][RTW89_CN][8] = 44, + [2][1][RTW89_UK][8] = 127, + [2][1][RTW89_FCC][9] = 127, + [2][1][RTW89_ETSI][9] = 127, + [2][1][RTW89_MKK][9] = 127, + [2][1][RTW89_IC][9] = 127, + [2][1][RTW89_KCC][9] = 127, + [2][1][RTW89_ACMA][9] = 127, + [2][1][RTW89_CN][9] = 44, + [2][1][RTW89_UK][9] = 127, + [2][1][RTW89_FCC][10] = 127, + [2][1][RTW89_ETSI][10] = 127, + [2][1][RTW89_MKK][10] = 127, + [2][1][RTW89_IC][10] = 127, + [2][1][RTW89_KCC][10] = 127, + [2][1][RTW89_ACMA][10] = 127, + [2][1][RTW89_CN][10] = 44, + [2][1][RTW89_UK][10] = 127, + [2][1][RTW89_FCC][11] = 127, + [2][1][RTW89_ETSI][11] = 127, + [2][1][RTW89_MKK][11] = 127, + [2][1][RTW89_IC][11] = 127, + [2][1][RTW89_KCC][11] = 127, + [2][1][RTW89_ACMA][11] = 127, + [2][1][RTW89_CN][11] = 44, + [2][1][RTW89_UK][11] = 127, + [2][1][RTW89_FCC][12] = 127, + [2][1][RTW89_ETSI][12] = 127, + [2][1][RTW89_MKK][12] = 127, + [2][1][RTW89_IC][12] = 127, + [2][1][RTW89_KCC][12] = 127, + [2][1][RTW89_ACMA][12] = 127, + [2][1][RTW89_CN][12] = 42, + [2][1][RTW89_UK][12] = 127, + [2][1][RTW89_FCC][13] = 127, + [2][1][RTW89_ETSI][13] = 127, + [2][1][RTW89_MKK][13] = 127, + [2][1][RTW89_IC][13] = 127, + [2][1][RTW89_KCC][13] = 127, + [2][1][RTW89_ACMA][13] = 127, + [2][1][RTW89_CN][13] = 127, + [2][1][RTW89_UK][13] = 127, +}; + +static +const s8 rtw89_8851b_txpwr_lmt_ru_5g[RTW89_RU_NUM][RTW89_NTX_NUM] + [RTW89_REGD_NUM][RTW89_5G_CH_NUM] = { + [0][0][RTW89_WW][0] = 16, + [0][0][RTW89_WW][2] = 16, + [0][0][RTW89_WW][4] = 16, + [0][0][RTW89_WW][6] = 16, + [0][0][RTW89_WW][8] = 16, + [0][0][RTW89_WW][10] = 16, + [0][0][RTW89_WW][12] = 16, + [0][0][RTW89_WW][14] = 16, + [0][0][RTW89_WW][15] = 24, + [0][0][RTW89_WW][17] = 24, + [0][0][RTW89_WW][19] = 24, + [0][0][RTW89_WW][21] = 24, + [0][0][RTW89_WW][23] = 24, + [0][0][RTW89_WW][25] = 24, + [0][0][RTW89_WW][27] = 24, + [0][0][RTW89_WW][29] = 24, + [0][0][RTW89_WW][31] = 24, + [0][0][RTW89_WW][33] = 24, + [0][0][RTW89_WW][35] = 24, + [0][0][RTW89_WW][37] = 44, + [0][0][RTW89_WW][38] = 24, + [0][0][RTW89_WW][40] = 24, + [0][0][RTW89_WW][42] = 24, + [0][0][RTW89_WW][44] = 24, + [0][0][RTW89_WW][46] = 24, + [0][0][RTW89_WW][48] = 42, + [0][0][RTW89_WW][50] = 42, + [0][0][RTW89_WW][52] = 40, + [0][1][RTW89_WW][0] = 4, + [0][1][RTW89_WW][2] = 4, + [0][1][RTW89_WW][4] = 4, + [0][1][RTW89_WW][6] = 4, + [0][1][RTW89_WW][8] = 4, + [0][1][RTW89_WW][10] = 4, + [0][1][RTW89_WW][12] = 4, + [0][1][RTW89_WW][14] = 4, + [0][1][RTW89_WW][15] = 0, + [0][1][RTW89_WW][17] = 0, + [0][1][RTW89_WW][19] = 0, + [0][1][RTW89_WW][21] = 0, + [0][1][RTW89_WW][23] = 0, + [0][1][RTW89_WW][25] = 0, + [0][1][RTW89_WW][27] = 0, + [0][1][RTW89_WW][29] = 0, + [0][1][RTW89_WW][31] = 0, + [0][1][RTW89_WW][33] = 0, + [0][1][RTW89_WW][35] = 0, + [0][1][RTW89_WW][37] = 0, + [0][1][RTW89_WW][38] = 42, + [0][1][RTW89_WW][40] = 42, + [0][1][RTW89_WW][42] = 42, + [0][1][RTW89_WW][44] = 42, + [0][1][RTW89_WW][46] = 42, + [0][1][RTW89_WW][48] = 0, + [0][1][RTW89_WW][50] = 0, + [0][1][RTW89_WW][52] = 0, + [1][0][RTW89_WW][0] = 26, + [1][0][RTW89_WW][2] = 26, + [1][0][RTW89_WW][4] = 26, + [1][0][RTW89_WW][6] = 26, + [1][0][RTW89_WW][8] = 26, + [1][0][RTW89_WW][10] = 26, + [1][0][RTW89_WW][12] = 26, + [1][0][RTW89_WW][14] = 26, + [1][0][RTW89_WW][15] = 34, + [1][0][RTW89_WW][17] = 34, + [1][0][RTW89_WW][19] = 34, + [1][0][RTW89_WW][21] = 34, + [1][0][RTW89_WW][23] = 34, + [1][0][RTW89_WW][25] = 34, + [1][0][RTW89_WW][27] = 34, + [1][0][RTW89_WW][29] = 34, + [1][0][RTW89_WW][31] = 34, + [1][0][RTW89_WW][33] = 34, + [1][0][RTW89_WW][35] = 34, + [1][0][RTW89_WW][37] = 54, + [1][0][RTW89_WW][38] = 28, + [1][0][RTW89_WW][40] = 28, + [1][0][RTW89_WW][42] = 28, + [1][0][RTW89_WW][44] = 28, + [1][0][RTW89_WW][46] = 28, + [1][0][RTW89_WW][48] = 52, + [1][0][RTW89_WW][50] = 52, + [1][0][RTW89_WW][52] = 52, + [1][1][RTW89_WW][0] = 14, + [1][1][RTW89_WW][2] = 14, + [1][1][RTW89_WW][4] = 14, + [1][1][RTW89_WW][6] = 14, + [1][1][RTW89_WW][8] = 14, + [1][1][RTW89_WW][10] = 14, + [1][1][RTW89_WW][12] = 14, + [1][1][RTW89_WW][14] = 14, + [1][1][RTW89_WW][15] = 0, + [1][1][RTW89_WW][17] = 0, + [1][1][RTW89_WW][19] = 0, + [1][1][RTW89_WW][21] = 0, + [1][1][RTW89_WW][23] = 0, + [1][1][RTW89_WW][25] = 0, + [1][1][RTW89_WW][27] = 0, + [1][1][RTW89_WW][29] = 0, + [1][1][RTW89_WW][31] = 0, + [1][1][RTW89_WW][33] = 0, + [1][1][RTW89_WW][35] = 0, + [1][1][RTW89_WW][37] = 0, + [1][1][RTW89_WW][38] = 54, + [1][1][RTW89_WW][40] = 54, + [1][1][RTW89_WW][42] = 54, + [1][1][RTW89_WW][44] = 54, + [1][1][RTW89_WW][46] = 54, + [1][1][RTW89_WW][48] = 0, + [1][1][RTW89_WW][50] = 0, + [1][1][RTW89_WW][52] = 0, + [2][0][RTW89_WW][0] = 40, + [2][0][RTW89_WW][2] = 40, + [2][0][RTW89_WW][4] = 40, + [2][0][RTW89_WW][6] = 40, + [2][0][RTW89_WW][8] = 40, + [2][0][RTW89_WW][10] = 40, + [2][0][RTW89_WW][12] = 40, + [2][0][RTW89_WW][14] = 40, + [2][0][RTW89_WW][15] = 46, + [2][0][RTW89_WW][17] = 46, + [2][0][RTW89_WW][19] = 46, + [2][0][RTW89_WW][21] = 46, + [2][0][RTW89_WW][23] = 46, + [2][0][RTW89_WW][25] = 46, + [2][0][RTW89_WW][27] = 46, + [2][0][RTW89_WW][29] = 46, + [2][0][RTW89_WW][31] = 46, + [2][0][RTW89_WW][33] = 46, + [2][0][RTW89_WW][35] = 46, + [2][0][RTW89_WW][37] = 66, + [2][0][RTW89_WW][38] = 28, + [2][0][RTW89_WW][40] = 28, + [2][0][RTW89_WW][42] = 28, + [2][0][RTW89_WW][44] = 28, + [2][0][RTW89_WW][46] = 28, + [2][0][RTW89_WW][48] = 64, + [2][0][RTW89_WW][50] = 64, + [2][0][RTW89_WW][52] = 60, + [2][1][RTW89_WW][0] = 28, + [2][1][RTW89_WW][2] = 28, + [2][1][RTW89_WW][4] = 28, + [2][1][RTW89_WW][6] = 28, + [2][1][RTW89_WW][8] = 28, + [2][1][RTW89_WW][10] = 28, + [2][1][RTW89_WW][12] = 28, + [2][1][RTW89_WW][14] = 28, + [2][1][RTW89_WW][15] = 0, + [2][1][RTW89_WW][17] = 0, + [2][1][RTW89_WW][19] = 0, + [2][1][RTW89_WW][21] = 0, + [2][1][RTW89_WW][23] = 0, + [2][1][RTW89_WW][25] = 0, + [2][1][RTW89_WW][27] = 0, + [2][1][RTW89_WW][29] = 0, + [2][1][RTW89_WW][31] = 0, + [2][1][RTW89_WW][33] = 0, + [2][1][RTW89_WW][35] = 0, + [2][1][RTW89_WW][37] = 0, + [2][1][RTW89_WW][38] = 56, + [2][1][RTW89_WW][40] = 56, + [2][1][RTW89_WW][42] = 56, + [2][1][RTW89_WW][44] = 56, + [2][1][RTW89_WW][46] = 56, + [2][1][RTW89_WW][48] = 0, + [2][1][RTW89_WW][50] = 0, + [2][1][RTW89_WW][52] = 0, + [0][0][RTW89_FCC][0] = 52, + [0][0][RTW89_ETSI][0] = 24, + [0][0][RTW89_MKK][0] = 26, + [0][0][RTW89_IC][0] = 28, + [0][0][RTW89_KCC][0] = 42, + [0][0][RTW89_ACMA][0] = 24, + [0][0][RTW89_CN][0] = 16, + [0][0][RTW89_UK][0] = 24, + [0][0][RTW89_FCC][2] = 54, + [0][0][RTW89_ETSI][2] = 24, + [0][0][RTW89_MKK][2] = 26, + [0][0][RTW89_IC][2] = 28, + [0][0][RTW89_KCC][2] = 42, + [0][0][RTW89_ACMA][2] = 24, + [0][0][RTW89_CN][2] = 16, + [0][0][RTW89_UK][2] = 24, + [0][0][RTW89_FCC][4] = 52, + [0][0][RTW89_ETSI][4] = 24, + [0][0][RTW89_MKK][4] = 26, + [0][0][RTW89_IC][4] = 28, + [0][0][RTW89_KCC][4] = 42, + [0][0][RTW89_ACMA][4] = 24, + [0][0][RTW89_CN][4] = 16, + [0][0][RTW89_UK][4] = 24, + [0][0][RTW89_FCC][6] = 52, + [0][0][RTW89_ETSI][6] = 24, + [0][0][RTW89_MKK][6] = 26, + [0][0][RTW89_IC][6] = 28, + [0][0][RTW89_KCC][6] = 18, + [0][0][RTW89_ACMA][6] = 24, + [0][0][RTW89_CN][6] = 16, + [0][0][RTW89_UK][6] = 24, + [0][0][RTW89_FCC][8] = 52, + [0][0][RTW89_ETSI][8] = 24, + [0][0][RTW89_MKK][8] = 26, + [0][0][RTW89_IC][8] = 52, + [0][0][RTW89_KCC][8] = 42, + [0][0][RTW89_ACMA][8] = 24, + [0][0][RTW89_CN][8] = 16, + [0][0][RTW89_UK][8] = 24, + [0][0][RTW89_FCC][10] = 52, + [0][0][RTW89_ETSI][10] = 24, + [0][0][RTW89_MKK][10] = 26, + [0][0][RTW89_IC][10] = 52, + [0][0][RTW89_KCC][10] = 42, + [0][0][RTW89_ACMA][10] = 24, + [0][0][RTW89_CN][10] = 16, + [0][0][RTW89_UK][10] = 24, + [0][0][RTW89_FCC][12] = 56, + [0][0][RTW89_ETSI][12] = 24, + [0][0][RTW89_MKK][12] = 26, + [0][0][RTW89_IC][12] = 56, + [0][0][RTW89_KCC][12] = 44, + [0][0][RTW89_ACMA][12] = 24, + [0][0][RTW89_CN][12] = 16, + [0][0][RTW89_UK][12] = 24, + [0][0][RTW89_FCC][14] = 56, + [0][0][RTW89_ETSI][14] = 24, + [0][0][RTW89_MKK][14] = 26, + [0][0][RTW89_IC][14] = 56, + [0][0][RTW89_KCC][14] = 44, + [0][0][RTW89_ACMA][14] = 24, + [0][0][RTW89_CN][14] = 16, + [0][0][RTW89_UK][14] = 24, + [0][0][RTW89_FCC][15] = 54, + [0][0][RTW89_ETSI][15] = 24, + [0][0][RTW89_MKK][15] = 46, + [0][0][RTW89_IC][15] = 54, + [0][0][RTW89_KCC][15] = 44, + [0][0][RTW89_ACMA][15] = 24, + [0][0][RTW89_CN][15] = 127, + [0][0][RTW89_UK][15] = 24, + [0][0][RTW89_FCC][17] = 54, + [0][0][RTW89_ETSI][17] = 24, + [0][0][RTW89_MKK][17] = 50, + [0][0][RTW89_IC][17] = 54, + [0][0][RTW89_KCC][17] = 44, + [0][0][RTW89_ACMA][17] = 24, + [0][0][RTW89_CN][17] = 127, + [0][0][RTW89_UK][17] = 24, + [0][0][RTW89_FCC][19] = 54, + [0][0][RTW89_ETSI][19] = 24, + [0][0][RTW89_MKK][19] = 50, + [0][0][RTW89_IC][19] = 54, + [0][0][RTW89_KCC][19] = 44, + [0][0][RTW89_ACMA][19] = 24, + [0][0][RTW89_CN][19] = 127, + [0][0][RTW89_UK][19] = 24, + [0][0][RTW89_FCC][21] = 54, + [0][0][RTW89_ETSI][21] = 24, + [0][0][RTW89_MKK][21] = 50, + [0][0][RTW89_IC][21] = 54, + [0][0][RTW89_KCC][21] = 44, + [0][0][RTW89_ACMA][21] = 24, + [0][0][RTW89_CN][21] = 127, + [0][0][RTW89_UK][21] = 24, + [0][0][RTW89_FCC][23] = 54, + [0][0][RTW89_ETSI][23] = 24, + [0][0][RTW89_MKK][23] = 50, + [0][0][RTW89_IC][23] = 54, + [0][0][RTW89_KCC][23] = 44, + [0][0][RTW89_ACMA][23] = 24, + [0][0][RTW89_CN][23] = 127, + [0][0][RTW89_UK][23] = 24, + [0][0][RTW89_FCC][25] = 54, + [0][0][RTW89_ETSI][25] = 24, + [0][0][RTW89_MKK][25] = 50, + [0][0][RTW89_IC][25] = 127, + [0][0][RTW89_KCC][25] = 44, + [0][0][RTW89_ACMA][25] = 127, + [0][0][RTW89_CN][25] = 127, + [0][0][RTW89_UK][25] = 24, + [0][0][RTW89_FCC][27] = 54, + [0][0][RTW89_ETSI][27] = 24, + [0][0][RTW89_MKK][27] = 50, + [0][0][RTW89_IC][27] = 127, + [0][0][RTW89_KCC][27] = 42, + [0][0][RTW89_ACMA][27] = 127, + [0][0][RTW89_CN][27] = 127, + [0][0][RTW89_UK][27] = 24, + [0][0][RTW89_FCC][29] = 54, + [0][0][RTW89_ETSI][29] = 24, + [0][0][RTW89_MKK][29] = 50, + [0][0][RTW89_IC][29] = 127, + [0][0][RTW89_KCC][29] = 42, + [0][0][RTW89_ACMA][29] = 127, + [0][0][RTW89_CN][29] = 127, + [0][0][RTW89_UK][29] = 24, + [0][0][RTW89_FCC][31] = 54, + [0][0][RTW89_ETSI][31] = 24, + [0][0][RTW89_MKK][31] = 50, + [0][0][RTW89_IC][31] = 56, + [0][0][RTW89_KCC][31] = 42, + [0][0][RTW89_ACMA][31] = 24, + [0][0][RTW89_CN][31] = 127, + [0][0][RTW89_UK][31] = 24, + [0][0][RTW89_FCC][33] = 56, + [0][0][RTW89_ETSI][33] = 24, + [0][0][RTW89_MKK][33] = 50, + [0][0][RTW89_IC][33] = 56, + [0][0][RTW89_KCC][33] = 42, + [0][0][RTW89_ACMA][33] = 24, + [0][0][RTW89_CN][33] = 127, + [0][0][RTW89_UK][33] = 24, + [0][0][RTW89_FCC][35] = 56, + [0][0][RTW89_ETSI][35] = 24, + [0][0][RTW89_MKK][35] = 50, + [0][0][RTW89_IC][35] = 56, + [0][0][RTW89_KCC][35] = 42, + [0][0][RTW89_ACMA][35] = 24, + [0][0][RTW89_CN][35] = 127, + [0][0][RTW89_UK][35] = 24, + [0][0][RTW89_FCC][37] = 86, + [0][0][RTW89_ETSI][37] = 127, + [0][0][RTW89_MKK][37] = 46, + [0][0][RTW89_IC][37] = 86, + [0][0][RTW89_KCC][37] = 44, + [0][0][RTW89_ACMA][37] = 50, + [0][0][RTW89_CN][37] = 127, + [0][0][RTW89_UK][37] = 52, + [0][0][RTW89_FCC][38] = 68, + [0][0][RTW89_ETSI][38] = 28, + [0][0][RTW89_MKK][38] = 127, + [0][0][RTW89_IC][38] = 68, + [0][0][RTW89_KCC][38] = 44, + [0][0][RTW89_ACMA][38] = 84, + [0][0][RTW89_CN][38] = 54, + [0][0][RTW89_UK][38] = 24, + [0][0][RTW89_FCC][40] = 68, + [0][0][RTW89_ETSI][40] = 28, + [0][0][RTW89_MKK][40] = 127, + [0][0][RTW89_IC][40] = 68, + [0][0][RTW89_KCC][40] = 44, + [0][0][RTW89_ACMA][40] = 84, + [0][0][RTW89_CN][40] = 54, + [0][0][RTW89_UK][40] = 24, + [0][0][RTW89_FCC][42] = 70, + [0][0][RTW89_ETSI][42] = 28, + [0][0][RTW89_MKK][42] = 127, + [0][0][RTW89_IC][42] = 70, + [0][0][RTW89_KCC][42] = 44, + [0][0][RTW89_ACMA][42] = 84, + [0][0][RTW89_CN][42] = 54, + [0][0][RTW89_UK][42] = 24, + [0][0][RTW89_FCC][44] = 62, + [0][0][RTW89_ETSI][44] = 28, + [0][0][RTW89_MKK][44] = 127, + [0][0][RTW89_IC][44] = 62, + [0][0][RTW89_KCC][44] = 44, + [0][0][RTW89_ACMA][44] = 84, + [0][0][RTW89_CN][44] = 54, + [0][0][RTW89_UK][44] = 24, + [0][0][RTW89_FCC][46] = 62, + [0][0][RTW89_ETSI][46] = 28, + [0][0][RTW89_MKK][46] = 127, + [0][0][RTW89_IC][46] = 62, + [0][0][RTW89_KCC][46] = 44, + [0][0][RTW89_ACMA][46] = 84, + [0][0][RTW89_CN][46] = 54, + [0][0][RTW89_UK][46] = 24, + [0][0][RTW89_FCC][48] = 42, + [0][0][RTW89_ETSI][48] = 127, + [0][0][RTW89_MKK][48] = 127, + [0][0][RTW89_IC][48] = 127, + [0][0][RTW89_KCC][48] = 127, + [0][0][RTW89_ACMA][48] = 127, + [0][0][RTW89_CN][48] = 127, + [0][0][RTW89_UK][48] = 127, + [0][0][RTW89_FCC][50] = 42, + [0][0][RTW89_ETSI][50] = 127, + [0][0][RTW89_MKK][50] = 127, + [0][0][RTW89_IC][50] = 127, + [0][0][RTW89_KCC][50] = 127, + [0][0][RTW89_ACMA][50] = 127, + [0][0][RTW89_CN][50] = 127, + [0][0][RTW89_UK][50] = 127, + [0][0][RTW89_FCC][52] = 40, + [0][0][RTW89_ETSI][52] = 127, + [0][0][RTW89_MKK][52] = 127, + [0][0][RTW89_IC][52] = 127, + [0][0][RTW89_KCC][52] = 127, + [0][0][RTW89_ACMA][52] = 127, + [0][0][RTW89_CN][52] = 127, + [0][0][RTW89_UK][52] = 127, + [0][1][RTW89_FCC][0] = 127, + [0][1][RTW89_ETSI][0] = 127, + [0][1][RTW89_MKK][0] = 127, + [0][1][RTW89_IC][0] = 127, + [0][1][RTW89_KCC][0] = 127, + [0][1][RTW89_ACMA][0] = 127, + [0][1][RTW89_CN][0] = 4, + [0][1][RTW89_UK][0] = 127, + [0][1][RTW89_FCC][2] = 127, + [0][1][RTW89_ETSI][2] = 127, + [0][1][RTW89_MKK][2] = 127, + [0][1][RTW89_IC][2] = 127, + [0][1][RTW89_KCC][2] = 127, + [0][1][RTW89_ACMA][2] = 127, + [0][1][RTW89_CN][2] = 4, + [0][1][RTW89_UK][2] = 127, + [0][1][RTW89_FCC][4] = 127, + [0][1][RTW89_ETSI][4] = 127, + [0][1][RTW89_MKK][4] = 127, + [0][1][RTW89_IC][4] = 127, + [0][1][RTW89_KCC][4] = 127, + [0][1][RTW89_ACMA][4] = 127, + [0][1][RTW89_CN][4] = 4, + [0][1][RTW89_UK][4] = 127, + [0][1][RTW89_FCC][6] = 127, + [0][1][RTW89_ETSI][6] = 127, + [0][1][RTW89_MKK][6] = 127, + [0][1][RTW89_IC][6] = 127, + [0][1][RTW89_KCC][6] = 127, + [0][1][RTW89_ACMA][6] = 127, + [0][1][RTW89_CN][6] = 4, + [0][1][RTW89_UK][6] = 127, + [0][1][RTW89_FCC][8] = 127, + [0][1][RTW89_ETSI][8] = 127, + [0][1][RTW89_MKK][8] = 127, + [0][1][RTW89_IC][8] = 127, + [0][1][RTW89_KCC][8] = 127, + [0][1][RTW89_ACMA][8] = 127, + [0][1][RTW89_CN][8] = 4, + [0][1][RTW89_UK][8] = 127, + [0][1][RTW89_FCC][10] = 127, + [0][1][RTW89_ETSI][10] = 127, + [0][1][RTW89_MKK][10] = 127, + [0][1][RTW89_IC][10] = 127, + [0][1][RTW89_KCC][10] = 127, + [0][1][RTW89_ACMA][10] = 127, + [0][1][RTW89_CN][10] = 4, + [0][1][RTW89_UK][10] = 127, + [0][1][RTW89_FCC][12] = 127, + [0][1][RTW89_ETSI][12] = 127, + [0][1][RTW89_MKK][12] = 127, + [0][1][RTW89_IC][12] = 127, + [0][1][RTW89_KCC][12] = 127, + [0][1][RTW89_ACMA][12] = 127, + [0][1][RTW89_CN][12] = 4, + [0][1][RTW89_UK][12] = 127, + [0][1][RTW89_FCC][14] = 127, + [0][1][RTW89_ETSI][14] = 127, + [0][1][RTW89_MKK][14] = 127, + [0][1][RTW89_IC][14] = 127, + [0][1][RTW89_KCC][14] = 127, + [0][1][RTW89_ACMA][14] = 127, + [0][1][RTW89_CN][14] = 4, + [0][1][RTW89_UK][14] = 127, + [0][1][RTW89_FCC][15] = 127, + [0][1][RTW89_ETSI][15] = 127, + [0][1][RTW89_MKK][15] = 127, + [0][1][RTW89_IC][15] = 127, + [0][1][RTW89_KCC][15] = 127, + [0][1][RTW89_ACMA][15] = 127, + [0][1][RTW89_CN][15] = 127, + [0][1][RTW89_UK][15] = 127, + [0][1][RTW89_FCC][17] = 127, + [0][1][RTW89_ETSI][17] = 127, + [0][1][RTW89_MKK][17] = 127, + [0][1][RTW89_IC][17] = 127, + [0][1][RTW89_KCC][17] = 127, + [0][1][RTW89_ACMA][17] = 127, + [0][1][RTW89_CN][17] = 127, + [0][1][RTW89_UK][17] = 127, + [0][1][RTW89_FCC][19] = 127, + [0][1][RTW89_ETSI][19] = 127, + [0][1][RTW89_MKK][19] = 127, + [0][1][RTW89_IC][19] = 127, + [0][1][RTW89_KCC][19] = 127, + [0][1][RTW89_ACMA][19] = 127, + [0][1][RTW89_CN][19] = 127, + [0][1][RTW89_UK][19] = 127, + [0][1][RTW89_FCC][21] = 127, + [0][1][RTW89_ETSI][21] = 127, + [0][1][RTW89_MKK][21] = 127, + [0][1][RTW89_IC][21] = 127, + [0][1][RTW89_KCC][21] = 127, + [0][1][RTW89_ACMA][21] = 127, + [0][1][RTW89_CN][21] = 127, + [0][1][RTW89_UK][21] = 127, + [0][1][RTW89_FCC][23] = 127, + [0][1][RTW89_ETSI][23] = 127, + [0][1][RTW89_MKK][23] = 127, + [0][1][RTW89_IC][23] = 127, + [0][1][RTW89_KCC][23] = 127, + [0][1][RTW89_ACMA][23] = 127, + [0][1][RTW89_CN][23] = 127, + [0][1][RTW89_UK][23] = 127, + [0][1][RTW89_FCC][25] = 127, + [0][1][RTW89_ETSI][25] = 127, + [0][1][RTW89_MKK][25] = 127, + [0][1][RTW89_IC][25] = 127, + [0][1][RTW89_KCC][25] = 127, + [0][1][RTW89_ACMA][25] = 127, + [0][1][RTW89_CN][25] = 127, + [0][1][RTW89_UK][25] = 127, + [0][1][RTW89_FCC][27] = 127, + [0][1][RTW89_ETSI][27] = 127, + [0][1][RTW89_MKK][27] = 127, + [0][1][RTW89_IC][27] = 127, + [0][1][RTW89_KCC][27] = 127, + [0][1][RTW89_ACMA][27] = 127, + [0][1][RTW89_CN][27] = 127, + [0][1][RTW89_UK][27] = 127, + [0][1][RTW89_FCC][29] = 127, + [0][1][RTW89_ETSI][29] = 127, + [0][1][RTW89_MKK][29] = 127, + [0][1][RTW89_IC][29] = 127, + [0][1][RTW89_KCC][29] = 127, + [0][1][RTW89_ACMA][29] = 127, + [0][1][RTW89_CN][29] = 127, + [0][1][RTW89_UK][29] = 127, + [0][1][RTW89_FCC][31] = 127, + [0][1][RTW89_ETSI][31] = 127, + [0][1][RTW89_MKK][31] = 127, + [0][1][RTW89_IC][31] = 127, + [0][1][RTW89_KCC][31] = 127, + [0][1][RTW89_ACMA][31] = 127, + [0][1][RTW89_CN][31] = 127, + [0][1][RTW89_UK][31] = 127, + [0][1][RTW89_FCC][33] = 127, + [0][1][RTW89_ETSI][33] = 127, + [0][1][RTW89_MKK][33] = 127, + [0][1][RTW89_IC][33] = 127, + [0][1][RTW89_KCC][33] = 127, + [0][1][RTW89_ACMA][33] = 127, + [0][1][RTW89_CN][33] = 127, + [0][1][RTW89_UK][33] = 127, + [0][1][RTW89_FCC][35] = 127, + [0][1][RTW89_ETSI][35] = 127, + [0][1][RTW89_MKK][35] = 127, + [0][1][RTW89_IC][35] = 127, + [0][1][RTW89_KCC][35] = 127, + [0][1][RTW89_ACMA][35] = 127, + [0][1][RTW89_CN][35] = 127, + [0][1][RTW89_UK][35] = 127, + [0][1][RTW89_FCC][37] = 127, + [0][1][RTW89_ETSI][37] = 127, + [0][1][RTW89_MKK][37] = 127, + [0][1][RTW89_IC][37] = 127, + [0][1][RTW89_KCC][37] = 127, + [0][1][RTW89_ACMA][37] = 127, + [0][1][RTW89_CN][37] = 127, + [0][1][RTW89_UK][37] = 127, + [0][1][RTW89_FCC][38] = 127, + [0][1][RTW89_ETSI][38] = 127, + [0][1][RTW89_MKK][38] = 127, + [0][1][RTW89_IC][38] = 127, + [0][1][RTW89_KCC][38] = 127, + [0][1][RTW89_ACMA][38] = 127, + [0][1][RTW89_CN][38] = 42, + [0][1][RTW89_UK][38] = 127, + [0][1][RTW89_FCC][40] = 127, + [0][1][RTW89_ETSI][40] = 127, + [0][1][RTW89_MKK][40] = 127, + [0][1][RTW89_IC][40] = 127, + [0][1][RTW89_KCC][40] = 127, + [0][1][RTW89_ACMA][40] = 127, + [0][1][RTW89_CN][40] = 42, + [0][1][RTW89_UK][40] = 127, + [0][1][RTW89_FCC][42] = 127, + [0][1][RTW89_ETSI][42] = 127, + [0][1][RTW89_MKK][42] = 127, + [0][1][RTW89_IC][42] = 127, + [0][1][RTW89_KCC][42] = 127, + [0][1][RTW89_ACMA][42] = 127, + [0][1][RTW89_CN][42] = 42, + [0][1][RTW89_UK][42] = 127, + [0][1][RTW89_FCC][44] = 127, + [0][1][RTW89_ETSI][44] = 127, + [0][1][RTW89_MKK][44] = 127, + [0][1][RTW89_IC][44] = 127, + [0][1][RTW89_KCC][44] = 127, + [0][1][RTW89_ACMA][44] = 127, + [0][1][RTW89_CN][44] = 42, + [0][1][RTW89_UK][44] = 127, + [0][1][RTW89_FCC][46] = 127, + [0][1][RTW89_ETSI][46] = 127, + [0][1][RTW89_MKK][46] = 127, + [0][1][RTW89_IC][46] = 127, + [0][1][RTW89_KCC][46] = 127, + [0][1][RTW89_ACMA][46] = 127, + [0][1][RTW89_CN][46] = 42, + [0][1][RTW89_UK][46] = 127, + [0][1][RTW89_FCC][48] = 127, + [0][1][RTW89_ETSI][48] = 127, + [0][1][RTW89_MKK][48] = 127, + [0][1][RTW89_IC][48] = 127, + [0][1][RTW89_KCC][48] = 127, + [0][1][RTW89_ACMA][48] = 127, + [0][1][RTW89_CN][48] = 127, + [0][1][RTW89_UK][48] = 127, + [0][1][RTW89_FCC][50] = 127, + [0][1][RTW89_ETSI][50] = 127, + [0][1][RTW89_MKK][50] = 127, + [0][1][RTW89_IC][50] = 127, + [0][1][RTW89_KCC][50] = 127, + [0][1][RTW89_ACMA][50] = 127, + [0][1][RTW89_CN][50] = 127, + [0][1][RTW89_UK][50] = 127, + [0][1][RTW89_FCC][52] = 127, + [0][1][RTW89_ETSI][52] = 127, + [0][1][RTW89_MKK][52] = 127, + [0][1][RTW89_IC][52] = 127, + [0][1][RTW89_KCC][52] = 127, + [0][1][RTW89_ACMA][52] = 127, + [0][1][RTW89_CN][52] = 127, + [0][1][RTW89_UK][52] = 127, + [1][0][RTW89_FCC][0] = 64, + [1][0][RTW89_ETSI][0] = 34, + [1][0][RTW89_MKK][0] = 38, + [1][0][RTW89_IC][0] = 38, + [1][0][RTW89_KCC][0] = 52, + [1][0][RTW89_ACMA][0] = 34, + [1][0][RTW89_CN][0] = 26, + [1][0][RTW89_UK][0] = 34, + [1][0][RTW89_FCC][2] = 66, + [1][0][RTW89_ETSI][2] = 34, + [1][0][RTW89_MKK][2] = 38, + [1][0][RTW89_IC][2] = 38, + [1][0][RTW89_KCC][2] = 52, + [1][0][RTW89_ACMA][2] = 34, + [1][0][RTW89_CN][2] = 26, + [1][0][RTW89_UK][2] = 34, + [1][0][RTW89_FCC][4] = 62, + [1][0][RTW89_ETSI][4] = 34, + [1][0][RTW89_MKK][4] = 36, + [1][0][RTW89_IC][4] = 38, + [1][0][RTW89_KCC][4] = 52, + [1][0][RTW89_ACMA][4] = 34, + [1][0][RTW89_CN][4] = 26, + [1][0][RTW89_UK][4] = 34, + [1][0][RTW89_FCC][6] = 62, + [1][0][RTW89_ETSI][6] = 34, + [1][0][RTW89_MKK][6] = 36, + [1][0][RTW89_IC][6] = 38, + [1][0][RTW89_KCC][6] = 32, + [1][0][RTW89_ACMA][6] = 34, + [1][0][RTW89_CN][6] = 26, + [1][0][RTW89_UK][6] = 34, + [1][0][RTW89_FCC][8] = 62, + [1][0][RTW89_ETSI][8] = 34, + [1][0][RTW89_MKK][8] = 38, + [1][0][RTW89_IC][8] = 62, + [1][0][RTW89_KCC][8] = 52, + [1][0][RTW89_ACMA][8] = 34, + [1][0][RTW89_CN][8] = 26, + [1][0][RTW89_UK][8] = 34, + [1][0][RTW89_FCC][10] = 62, + [1][0][RTW89_ETSI][10] = 34, + [1][0][RTW89_MKK][10] = 38, + [1][0][RTW89_IC][10] = 62, + [1][0][RTW89_KCC][10] = 52, + [1][0][RTW89_ACMA][10] = 34, + [1][0][RTW89_CN][10] = 26, + [1][0][RTW89_UK][10] = 34, + [1][0][RTW89_FCC][12] = 62, + [1][0][RTW89_ETSI][12] = 34, + [1][0][RTW89_MKK][12] = 38, + [1][0][RTW89_IC][12] = 62, + [1][0][RTW89_KCC][12] = 54, + [1][0][RTW89_ACMA][12] = 34, + [1][0][RTW89_CN][12] = 26, + [1][0][RTW89_UK][12] = 34, + [1][0][RTW89_FCC][14] = 64, + [1][0][RTW89_ETSI][14] = 34, + [1][0][RTW89_MKK][14] = 38, + [1][0][RTW89_IC][14] = 64, + [1][0][RTW89_KCC][14] = 54, + [1][0][RTW89_ACMA][14] = 34, + [1][0][RTW89_CN][14] = 26, + [1][0][RTW89_UK][14] = 34, + [1][0][RTW89_FCC][15] = 62, + [1][0][RTW89_ETSI][15] = 34, + [1][0][RTW89_MKK][15] = 58, + [1][0][RTW89_IC][15] = 62, + [1][0][RTW89_KCC][15] = 54, + [1][0][RTW89_ACMA][15] = 34, + [1][0][RTW89_CN][15] = 127, + [1][0][RTW89_UK][15] = 34, + [1][0][RTW89_FCC][17] = 62, + [1][0][RTW89_ETSI][17] = 34, + [1][0][RTW89_MKK][17] = 58, + [1][0][RTW89_IC][17] = 62, + [1][0][RTW89_KCC][17] = 54, + [1][0][RTW89_ACMA][17] = 34, + [1][0][RTW89_CN][17] = 127, + [1][0][RTW89_UK][17] = 34, + [1][0][RTW89_FCC][19] = 64, + [1][0][RTW89_ETSI][19] = 34, + [1][0][RTW89_MKK][19] = 58, + [1][0][RTW89_IC][19] = 64, + [1][0][RTW89_KCC][19] = 54, + [1][0][RTW89_ACMA][19] = 34, + [1][0][RTW89_CN][19] = 127, + [1][0][RTW89_UK][19] = 34, + [1][0][RTW89_FCC][21] = 64, + [1][0][RTW89_ETSI][21] = 34, + [1][0][RTW89_MKK][21] = 58, + [1][0][RTW89_IC][21] = 64, + [1][0][RTW89_KCC][21] = 54, + [1][0][RTW89_ACMA][21] = 34, + [1][0][RTW89_CN][21] = 127, + [1][0][RTW89_UK][21] = 34, + [1][0][RTW89_FCC][23] = 64, + [1][0][RTW89_ETSI][23] = 34, + [1][0][RTW89_MKK][23] = 58, + [1][0][RTW89_IC][23] = 64, + [1][0][RTW89_KCC][23] = 54, + [1][0][RTW89_ACMA][23] = 34, + [1][0][RTW89_CN][23] = 127, + [1][0][RTW89_UK][23] = 34, + [1][0][RTW89_FCC][25] = 64, + [1][0][RTW89_ETSI][25] = 34, + [1][0][RTW89_MKK][25] = 58, + [1][0][RTW89_IC][25] = 127, + [1][0][RTW89_KCC][25] = 54, + [1][0][RTW89_ACMA][25] = 127, + [1][0][RTW89_CN][25] = 127, + [1][0][RTW89_UK][25] = 34, + [1][0][RTW89_FCC][27] = 64, + [1][0][RTW89_ETSI][27] = 34, + [1][0][RTW89_MKK][27] = 58, + [1][0][RTW89_IC][27] = 127, + [1][0][RTW89_KCC][27] = 54, + [1][0][RTW89_ACMA][27] = 127, + [1][0][RTW89_CN][27] = 127, + [1][0][RTW89_UK][27] = 34, + [1][0][RTW89_FCC][29] = 64, + [1][0][RTW89_ETSI][29] = 34, + [1][0][RTW89_MKK][29] = 58, + [1][0][RTW89_IC][29] = 127, + [1][0][RTW89_KCC][29] = 54, + [1][0][RTW89_ACMA][29] = 127, + [1][0][RTW89_CN][29] = 127, + [1][0][RTW89_UK][29] = 34, + [1][0][RTW89_FCC][31] = 64, + [1][0][RTW89_ETSI][31] = 34, + [1][0][RTW89_MKK][31] = 58, + [1][0][RTW89_IC][31] = 64, + [1][0][RTW89_KCC][31] = 54, + [1][0][RTW89_ACMA][31] = 34, + [1][0][RTW89_CN][31] = 127, + [1][0][RTW89_UK][31] = 34, + [1][0][RTW89_FCC][33] = 64, + [1][0][RTW89_ETSI][33] = 34, + [1][0][RTW89_MKK][33] = 58, + [1][0][RTW89_IC][33] = 64, + [1][0][RTW89_KCC][33] = 54, + [1][0][RTW89_ACMA][33] = 34, + [1][0][RTW89_CN][33] = 127, + [1][0][RTW89_UK][33] = 34, + [1][0][RTW89_FCC][35] = 64, + [1][0][RTW89_ETSI][35] = 34, + [1][0][RTW89_MKK][35] = 58, + [1][0][RTW89_IC][35] = 64, + [1][0][RTW89_KCC][35] = 54, + [1][0][RTW89_ACMA][35] = 34, + [1][0][RTW89_CN][35] = 127, + [1][0][RTW89_UK][35] = 34, + [1][0][RTW89_FCC][37] = 78, + [1][0][RTW89_ETSI][37] = 127, + [1][0][RTW89_MKK][37] = 56, + [1][0][RTW89_IC][37] = 78, + [1][0][RTW89_KCC][37] = 54, + [1][0][RTW89_ACMA][37] = 62, + [1][0][RTW89_CN][37] = 127, + [1][0][RTW89_UK][37] = 62, + [1][0][RTW89_FCC][38] = 82, + [1][0][RTW89_ETSI][38] = 28, + [1][0][RTW89_MKK][38] = 127, + [1][0][RTW89_IC][38] = 82, + [1][0][RTW89_KCC][38] = 54, + [1][0][RTW89_ACMA][38] = 84, + [1][0][RTW89_CN][38] = 66, + [1][0][RTW89_UK][38] = 34, + [1][0][RTW89_FCC][40] = 82, + [1][0][RTW89_ETSI][40] = 28, + [1][0][RTW89_MKK][40] = 127, + [1][0][RTW89_IC][40] = 82, + [1][0][RTW89_KCC][40] = 54, + [1][0][RTW89_ACMA][40] = 84, + [1][0][RTW89_CN][40] = 66, + [1][0][RTW89_UK][40] = 34, + [1][0][RTW89_FCC][42] = 78, + [1][0][RTW89_ETSI][42] = 28, + [1][0][RTW89_MKK][42] = 127, + [1][0][RTW89_IC][42] = 78, + [1][0][RTW89_KCC][42] = 54, + [1][0][RTW89_ACMA][42] = 84, + [1][0][RTW89_CN][42] = 66, + [1][0][RTW89_UK][42] = 34, + [1][0][RTW89_FCC][44] = 82, + [1][0][RTW89_ETSI][44] = 28, + [1][0][RTW89_MKK][44] = 127, + [1][0][RTW89_IC][44] = 82, + [1][0][RTW89_KCC][44] = 54, + [1][0][RTW89_ACMA][44] = 84, + [1][0][RTW89_CN][44] = 66, + [1][0][RTW89_UK][44] = 34, + [1][0][RTW89_FCC][46] = 82, + [1][0][RTW89_ETSI][46] = 28, + [1][0][RTW89_MKK][46] = 127, + [1][0][RTW89_IC][46] = 82, + [1][0][RTW89_KCC][46] = 54, + [1][0][RTW89_ACMA][46] = 84, + [1][0][RTW89_CN][46] = 66, + [1][0][RTW89_UK][46] = 34, + [1][0][RTW89_FCC][48] = 52, + [1][0][RTW89_ETSI][48] = 127, + [1][0][RTW89_MKK][48] = 127, + [1][0][RTW89_IC][48] = 127, + [1][0][RTW89_KCC][48] = 127, + [1][0][RTW89_ACMA][48] = 127, + [1][0][RTW89_CN][48] = 127, + [1][0][RTW89_UK][48] = 127, + [1][0][RTW89_FCC][50] = 52, + [1][0][RTW89_ETSI][50] = 127, + [1][0][RTW89_MKK][50] = 127, + [1][0][RTW89_IC][50] = 127, + [1][0][RTW89_KCC][50] = 127, + [1][0][RTW89_ACMA][50] = 127, + [1][0][RTW89_CN][50] = 127, + [1][0][RTW89_UK][50] = 127, + [1][0][RTW89_FCC][52] = 52, + [1][0][RTW89_ETSI][52] = 127, + [1][0][RTW89_MKK][52] = 127, + [1][0][RTW89_IC][52] = 127, + [1][0][RTW89_KCC][52] = 127, + [1][0][RTW89_ACMA][52] = 127, + [1][0][RTW89_CN][52] = 127, + [1][0][RTW89_UK][52] = 127, + [1][1][RTW89_FCC][0] = 127, + [1][1][RTW89_ETSI][0] = 127, + [1][1][RTW89_MKK][0] = 127, + [1][1][RTW89_IC][0] = 127, + [1][1][RTW89_KCC][0] = 127, + [1][1][RTW89_ACMA][0] = 127, + [1][1][RTW89_CN][0] = 14, + [1][1][RTW89_UK][0] = 127, + [1][1][RTW89_FCC][2] = 127, + [1][1][RTW89_ETSI][2] = 127, + [1][1][RTW89_MKK][2] = 127, + [1][1][RTW89_IC][2] = 127, + [1][1][RTW89_KCC][2] = 127, + [1][1][RTW89_ACMA][2] = 127, + [1][1][RTW89_CN][2] = 14, + [1][1][RTW89_UK][2] = 127, + [1][1][RTW89_FCC][4] = 127, + [1][1][RTW89_ETSI][4] = 127, + [1][1][RTW89_MKK][4] = 127, + [1][1][RTW89_IC][4] = 127, + [1][1][RTW89_KCC][4] = 127, + [1][1][RTW89_ACMA][4] = 127, + [1][1][RTW89_CN][4] = 14, + [1][1][RTW89_UK][4] = 127, + [1][1][RTW89_FCC][6] = 127, + [1][1][RTW89_ETSI][6] = 127, + [1][1][RTW89_MKK][6] = 127, + [1][1][RTW89_IC][6] = 127, + [1][1][RTW89_KCC][6] = 127, + [1][1][RTW89_ACMA][6] = 127, + [1][1][RTW89_CN][6] = 14, + [1][1][RTW89_UK][6] = 127, + [1][1][RTW89_FCC][8] = 127, + [1][1][RTW89_ETSI][8] = 127, + [1][1][RTW89_MKK][8] = 127, + [1][1][RTW89_IC][8] = 127, + [1][1][RTW89_KCC][8] = 127, + [1][1][RTW89_ACMA][8] = 127, + [1][1][RTW89_CN][8] = 14, + [1][1][RTW89_UK][8] = 127, + [1][1][RTW89_FCC][10] = 127, + [1][1][RTW89_ETSI][10] = 127, + [1][1][RTW89_MKK][10] = 127, + [1][1][RTW89_IC][10] = 127, + [1][1][RTW89_KCC][10] = 127, + [1][1][RTW89_ACMA][10] = 127, + [1][1][RTW89_CN][10] = 14, + [1][1][RTW89_UK][10] = 127, + [1][1][RTW89_FCC][12] = 127, + [1][1][RTW89_ETSI][12] = 127, + [1][1][RTW89_MKK][12] = 127, + [1][1][RTW89_IC][12] = 127, + [1][1][RTW89_KCC][12] = 127, + [1][1][RTW89_ACMA][12] = 127, + [1][1][RTW89_CN][12] = 14, + [1][1][RTW89_UK][12] = 127, + [1][1][RTW89_FCC][14] = 127, + [1][1][RTW89_ETSI][14] = 127, + [1][1][RTW89_MKK][14] = 127, + [1][1][RTW89_IC][14] = 127, + [1][1][RTW89_KCC][14] = 127, + [1][1][RTW89_ACMA][14] = 127, + [1][1][RTW89_CN][14] = 14, + [1][1][RTW89_UK][14] = 127, + [1][1][RTW89_FCC][15] = 127, + [1][1][RTW89_ETSI][15] = 127, + [1][1][RTW89_MKK][15] = 127, + [1][1][RTW89_IC][15] = 127, + [1][1][RTW89_KCC][15] = 127, + [1][1][RTW89_ACMA][15] = 127, + [1][1][RTW89_CN][15] = 127, + [1][1][RTW89_UK][15] = 127, + [1][1][RTW89_FCC][17] = 127, + [1][1][RTW89_ETSI][17] = 127, + [1][1][RTW89_MKK][17] = 127, + [1][1][RTW89_IC][17] = 127, + [1][1][RTW89_KCC][17] = 127, + [1][1][RTW89_ACMA][17] = 127, + [1][1][RTW89_CN][17] = 127, + [1][1][RTW89_UK][17] = 127, + [1][1][RTW89_FCC][19] = 127, + [1][1][RTW89_ETSI][19] = 127, + [1][1][RTW89_MKK][19] = 127, + [1][1][RTW89_IC][19] = 127, + [1][1][RTW89_KCC][19] = 127, + [1][1][RTW89_ACMA][19] = 127, + [1][1][RTW89_CN][19] = 127, + [1][1][RTW89_UK][19] = 127, + [1][1][RTW89_FCC][21] = 127, + [1][1][RTW89_ETSI][21] = 127, + [1][1][RTW89_MKK][21] = 127, + [1][1][RTW89_IC][21] = 127, + [1][1][RTW89_KCC][21] = 127, + [1][1][RTW89_ACMA][21] = 127, + [1][1][RTW89_CN][21] = 127, + [1][1][RTW89_UK][21] = 127, + [1][1][RTW89_FCC][23] = 127, + [1][1][RTW89_ETSI][23] = 127, + [1][1][RTW89_MKK][23] = 127, + [1][1][RTW89_IC][23] = 127, + [1][1][RTW89_KCC][23] = 127, + [1][1][RTW89_ACMA][23] = 127, + [1][1][RTW89_CN][23] = 127, + [1][1][RTW89_UK][23] = 127, + [1][1][RTW89_FCC][25] = 127, + [1][1][RTW89_ETSI][25] = 127, + [1][1][RTW89_MKK][25] = 127, + [1][1][RTW89_IC][25] = 127, + [1][1][RTW89_KCC][25] = 127, + [1][1][RTW89_ACMA][25] = 127, + [1][1][RTW89_CN][25] = 127, + [1][1][RTW89_UK][25] = 127, + [1][1][RTW89_FCC][27] = 127, + [1][1][RTW89_ETSI][27] = 127, + [1][1][RTW89_MKK][27] = 127, + [1][1][RTW89_IC][27] = 127, + [1][1][RTW89_KCC][27] = 127, + [1][1][RTW89_ACMA][27] = 127, + [1][1][RTW89_CN][27] = 127, + [1][1][RTW89_UK][27] = 127, + [1][1][RTW89_FCC][29] = 127, + [1][1][RTW89_ETSI][29] = 127, + [1][1][RTW89_MKK][29] = 127, + [1][1][RTW89_IC][29] = 127, + [1][1][RTW89_KCC][29] = 127, + [1][1][RTW89_ACMA][29] = 127, + [1][1][RTW89_CN][29] = 127, + [1][1][RTW89_UK][29] = 127, + [1][1][RTW89_FCC][31] = 127, + [1][1][RTW89_ETSI][31] = 127, + [1][1][RTW89_MKK][31] = 127, + [1][1][RTW89_IC][31] = 127, + [1][1][RTW89_KCC][31] = 127, + [1][1][RTW89_ACMA][31] = 127, + [1][1][RTW89_CN][31] = 127, + [1][1][RTW89_UK][31] = 127, + [1][1][RTW89_FCC][33] = 127, + [1][1][RTW89_ETSI][33] = 127, + [1][1][RTW89_MKK][33] = 127, + [1][1][RTW89_IC][33] = 127, + [1][1][RTW89_KCC][33] = 127, + [1][1][RTW89_ACMA][33] = 127, + [1][1][RTW89_CN][33] = 127, + [1][1][RTW89_UK][33] = 127, + [1][1][RTW89_FCC][35] = 127, + [1][1][RTW89_ETSI][35] = 127, + [1][1][RTW89_MKK][35] = 127, + [1][1][RTW89_IC][35] = 127, + [1][1][RTW89_KCC][35] = 127, + [1][1][RTW89_ACMA][35] = 127, + [1][1][RTW89_CN][35] = 127, + [1][1][RTW89_UK][35] = 127, + [1][1][RTW89_FCC][37] = 127, + [1][1][RTW89_ETSI][37] = 127, + [1][1][RTW89_MKK][37] = 127, + [1][1][RTW89_IC][37] = 127, + [1][1][RTW89_KCC][37] = 127, + [1][1][RTW89_ACMA][37] = 127, + [1][1][RTW89_CN][37] = 127, + [1][1][RTW89_UK][37] = 127, + [1][1][RTW89_FCC][38] = 127, + [1][1][RTW89_ETSI][38] = 127, + [1][1][RTW89_MKK][38] = 127, + [1][1][RTW89_IC][38] = 127, + [1][1][RTW89_KCC][38] = 127, + [1][1][RTW89_ACMA][38] = 127, + [1][1][RTW89_CN][38] = 54, + [1][1][RTW89_UK][38] = 127, + [1][1][RTW89_FCC][40] = 127, + [1][1][RTW89_ETSI][40] = 127, + [1][1][RTW89_MKK][40] = 127, + [1][1][RTW89_IC][40] = 127, + [1][1][RTW89_KCC][40] = 127, + [1][1][RTW89_ACMA][40] = 127, + [1][1][RTW89_CN][40] = 54, + [1][1][RTW89_UK][40] = 127, + [1][1][RTW89_FCC][42] = 127, + [1][1][RTW89_ETSI][42] = 127, + [1][1][RTW89_MKK][42] = 127, + [1][1][RTW89_IC][42] = 127, + [1][1][RTW89_KCC][42] = 127, + [1][1][RTW89_ACMA][42] = 127, + [1][1][RTW89_CN][42] = 54, + [1][1][RTW89_UK][42] = 127, + [1][1][RTW89_FCC][44] = 127, + [1][1][RTW89_ETSI][44] = 127, + [1][1][RTW89_MKK][44] = 127, + [1][1][RTW89_IC][44] = 127, + [1][1][RTW89_KCC][44] = 127, + [1][1][RTW89_ACMA][44] = 127, + [1][1][RTW89_CN][44] = 54, + [1][1][RTW89_UK][44] = 127, + [1][1][RTW89_FCC][46] = 127, + [1][1][RTW89_ETSI][46] = 127, + [1][1][RTW89_MKK][46] = 127, + [1][1][RTW89_IC][46] = 127, + [1][1][RTW89_KCC][46] = 127, + [1][1][RTW89_ACMA][46] = 127, + [1][1][RTW89_CN][46] = 54, + [1][1][RTW89_UK][46] = 127, + [1][1][RTW89_FCC][48] = 127, + [1][1][RTW89_ETSI][48] = 127, + [1][1][RTW89_MKK][48] = 127, + [1][1][RTW89_IC][48] = 127, + [1][1][RTW89_KCC][48] = 127, + [1][1][RTW89_ACMA][48] = 127, + [1][1][RTW89_CN][48] = 127, + [1][1][RTW89_UK][48] = 127, + [1][1][RTW89_FCC][50] = 127, + [1][1][RTW89_ETSI][50] = 127, + [1][1][RTW89_MKK][50] = 127, + [1][1][RTW89_IC][50] = 127, + [1][1][RTW89_KCC][50] = 127, + [1][1][RTW89_ACMA][50] = 127, + [1][1][RTW89_CN][50] = 127, + [1][1][RTW89_UK][50] = 127, + [1][1][RTW89_FCC][52] = 127, + [1][1][RTW89_ETSI][52] = 127, + [1][1][RTW89_MKK][52] = 127, + [1][1][RTW89_IC][52] = 127, + [1][1][RTW89_KCC][52] = 127, + [1][1][RTW89_ACMA][52] = 127, + [1][1][RTW89_CN][52] = 127, + [1][1][RTW89_UK][52] = 127, + [2][0][RTW89_FCC][0] = 78, + [2][0][RTW89_ETSI][0] = 46, + [2][0][RTW89_MKK][0] = 48, + [2][0][RTW89_IC][0] = 50, + [2][0][RTW89_KCC][0] = 64, + [2][0][RTW89_ACMA][0] = 46, + [2][0][RTW89_CN][0] = 40, + [2][0][RTW89_UK][0] = 46, + [2][0][RTW89_FCC][2] = 74, + [2][0][RTW89_ETSI][2] = 46, + [2][0][RTW89_MKK][2] = 48, + [2][0][RTW89_IC][2] = 48, + [2][0][RTW89_KCC][2] = 64, + [2][0][RTW89_ACMA][2] = 46, + [2][0][RTW89_CN][2] = 40, + [2][0][RTW89_UK][2] = 46, + [2][0][RTW89_FCC][4] = 74, + [2][0][RTW89_ETSI][4] = 46, + [2][0][RTW89_MKK][4] = 48, + [2][0][RTW89_IC][4] = 48, + [2][0][RTW89_KCC][4] = 64, + [2][0][RTW89_ACMA][4] = 46, + [2][0][RTW89_CN][4] = 40, + [2][0][RTW89_UK][4] = 46, + [2][0][RTW89_FCC][6] = 74, + [2][0][RTW89_ETSI][6] = 46, + [2][0][RTW89_MKK][6] = 48, + [2][0][RTW89_IC][6] = 48, + [2][0][RTW89_KCC][6] = 40, + [2][0][RTW89_ACMA][6] = 46, + [2][0][RTW89_CN][6] = 40, + [2][0][RTW89_UK][6] = 46, + [2][0][RTW89_FCC][8] = 74, + [2][0][RTW89_ETSI][8] = 46, + [2][0][RTW89_MKK][8] = 48, + [2][0][RTW89_IC][8] = 64, + [2][0][RTW89_KCC][8] = 66, + [2][0][RTW89_ACMA][8] = 46, + [2][0][RTW89_CN][8] = 40, + [2][0][RTW89_UK][8] = 46, + [2][0][RTW89_FCC][10] = 74, + [2][0][RTW89_ETSI][10] = 46, + [2][0][RTW89_MKK][10] = 48, + [2][0][RTW89_IC][10] = 64, + [2][0][RTW89_KCC][10] = 66, + [2][0][RTW89_ACMA][10] = 46, + [2][0][RTW89_CN][10] = 40, + [2][0][RTW89_UK][10] = 46, + [2][0][RTW89_FCC][12] = 74, + [2][0][RTW89_ETSI][12] = 46, + [2][0][RTW89_MKK][12] = 48, + [2][0][RTW89_IC][12] = 64, + [2][0][RTW89_KCC][12] = 64, + [2][0][RTW89_ACMA][12] = 46, + [2][0][RTW89_CN][12] = 40, + [2][0][RTW89_UK][12] = 46, + [2][0][RTW89_FCC][14] = 80, + [2][0][RTW89_ETSI][14] = 46, + [2][0][RTW89_MKK][14] = 48, + [2][0][RTW89_IC][14] = 64, + [2][0][RTW89_KCC][14] = 64, + [2][0][RTW89_ACMA][14] = 46, + [2][0][RTW89_CN][14] = 40, + [2][0][RTW89_UK][14] = 46, + [2][0][RTW89_FCC][15] = 72, + [2][0][RTW89_ETSI][15] = 46, + [2][0][RTW89_MKK][15] = 70, + [2][0][RTW89_IC][15] = 72, + [2][0][RTW89_KCC][15] = 66, + [2][0][RTW89_ACMA][15] = 46, + [2][0][RTW89_CN][15] = 127, + [2][0][RTW89_UK][15] = 46, + [2][0][RTW89_FCC][17] = 72, + [2][0][RTW89_ETSI][17] = 46, + [2][0][RTW89_MKK][17] = 70, + [2][0][RTW89_IC][17] = 72, + [2][0][RTW89_KCC][17] = 66, + [2][0][RTW89_ACMA][17] = 46, + [2][0][RTW89_CN][17] = 127, + [2][0][RTW89_UK][17] = 46, + [2][0][RTW89_FCC][19] = 70, + [2][0][RTW89_ETSI][19] = 46, + [2][0][RTW89_MKK][19] = 70, + [2][0][RTW89_IC][19] = 70, + [2][0][RTW89_KCC][19] = 66, + [2][0][RTW89_ACMA][19] = 46, + [2][0][RTW89_CN][19] = 127, + [2][0][RTW89_UK][19] = 46, + [2][0][RTW89_FCC][21] = 70, + [2][0][RTW89_ETSI][21] = 46, + [2][0][RTW89_MKK][21] = 70, + [2][0][RTW89_IC][21] = 70, + [2][0][RTW89_KCC][21] = 66, + [2][0][RTW89_ACMA][21] = 46, + [2][0][RTW89_CN][21] = 127, + [2][0][RTW89_UK][21] = 46, + [2][0][RTW89_FCC][23] = 70, + [2][0][RTW89_ETSI][23] = 46, + [2][0][RTW89_MKK][23] = 70, + [2][0][RTW89_IC][23] = 70, + [2][0][RTW89_KCC][23] = 66, + [2][0][RTW89_ACMA][23] = 46, + [2][0][RTW89_CN][23] = 127, + [2][0][RTW89_UK][23] = 46, + [2][0][RTW89_FCC][25] = 70, + [2][0][RTW89_ETSI][25] = 46, + [2][0][RTW89_MKK][25] = 70, + [2][0][RTW89_IC][25] = 127, + [2][0][RTW89_KCC][25] = 66, + [2][0][RTW89_ACMA][25] = 127, + [2][0][RTW89_CN][25] = 127, + [2][0][RTW89_UK][25] = 46, + [2][0][RTW89_FCC][27] = 70, + [2][0][RTW89_ETSI][27] = 46, + [2][0][RTW89_MKK][27] = 70, + [2][0][RTW89_IC][27] = 127, + [2][0][RTW89_KCC][27] = 64, + [2][0][RTW89_ACMA][27] = 127, + [2][0][RTW89_CN][27] = 127, + [2][0][RTW89_UK][27] = 46, + [2][0][RTW89_FCC][29] = 70, + [2][0][RTW89_ETSI][29] = 46, + [2][0][RTW89_MKK][29] = 70, + [2][0][RTW89_IC][29] = 127, + [2][0][RTW89_KCC][29] = 64, + [2][0][RTW89_ACMA][29] = 127, + [2][0][RTW89_CN][29] = 127, + [2][0][RTW89_UK][29] = 46, + [2][0][RTW89_FCC][31] = 70, + [2][0][RTW89_ETSI][31] = 46, + [2][0][RTW89_MKK][31] = 70, + [2][0][RTW89_IC][31] = 70, + [2][0][RTW89_KCC][31] = 64, + [2][0][RTW89_ACMA][31] = 46, + [2][0][RTW89_CN][31] = 127, + [2][0][RTW89_UK][31] = 46, + [2][0][RTW89_FCC][33] = 70, + [2][0][RTW89_ETSI][33] = 46, + [2][0][RTW89_MKK][33] = 70, + [2][0][RTW89_IC][33] = 70, + [2][0][RTW89_KCC][33] = 64, + [2][0][RTW89_ACMA][33] = 46, + [2][0][RTW89_CN][33] = 127, + [2][0][RTW89_UK][33] = 46, + [2][0][RTW89_FCC][35] = 70, + [2][0][RTW89_ETSI][35] = 46, + [2][0][RTW89_MKK][35] = 70, + [2][0][RTW89_IC][35] = 70, + [2][0][RTW89_KCC][35] = 64, + [2][0][RTW89_ACMA][35] = 46, + [2][0][RTW89_CN][35] = 127, + [2][0][RTW89_UK][35] = 46, + [2][0][RTW89_FCC][37] = 84, + [2][0][RTW89_ETSI][37] = 127, + [2][0][RTW89_MKK][37] = 68, + [2][0][RTW89_IC][37] = 84, + [2][0][RTW89_KCC][37] = 66, + [2][0][RTW89_ACMA][37] = 74, + [2][0][RTW89_CN][37] = 127, + [2][0][RTW89_UK][37] = 74, + [2][0][RTW89_FCC][38] = 84, + [2][0][RTW89_ETSI][38] = 28, + [2][0][RTW89_MKK][38] = 127, + [2][0][RTW89_IC][38] = 84, + [2][0][RTW89_KCC][38] = 64, + [2][0][RTW89_ACMA][38] = 84, + [2][0][RTW89_CN][38] = 68, + [2][0][RTW89_UK][38] = 46, + [2][0][RTW89_FCC][40] = 84, + [2][0][RTW89_ETSI][40] = 28, + [2][0][RTW89_MKK][40] = 127, + [2][0][RTW89_IC][40] = 84, + [2][0][RTW89_KCC][40] = 64, + [2][0][RTW89_ACMA][40] = 84, + [2][0][RTW89_CN][40] = 68, + [2][0][RTW89_UK][40] = 46, + [2][0][RTW89_FCC][42] = 80, + [2][0][RTW89_ETSI][42] = 28, + [2][0][RTW89_MKK][42] = 127, + [2][0][RTW89_IC][42] = 80, + [2][0][RTW89_KCC][42] = 66, + [2][0][RTW89_ACMA][42] = 84, + [2][0][RTW89_CN][42] = 68, + [2][0][RTW89_UK][42] = 46, + [2][0][RTW89_FCC][44] = 82, + [2][0][RTW89_ETSI][44] = 28, + [2][0][RTW89_MKK][44] = 127, + [2][0][RTW89_IC][44] = 82, + [2][0][RTW89_KCC][44] = 66, + [2][0][RTW89_ACMA][44] = 84, + [2][0][RTW89_CN][44] = 68, + [2][0][RTW89_UK][44] = 46, + [2][0][RTW89_FCC][46] = 82, + [2][0][RTW89_ETSI][46] = 28, + [2][0][RTW89_MKK][46] = 127, + [2][0][RTW89_IC][46] = 82, + [2][0][RTW89_KCC][46] = 66, + [2][0][RTW89_ACMA][46] = 84, + [2][0][RTW89_CN][46] = 68, + [2][0][RTW89_UK][46] = 46, + [2][0][RTW89_FCC][48] = 64, + [2][0][RTW89_ETSI][48] = 127, + [2][0][RTW89_MKK][48] = 127, + [2][0][RTW89_IC][48] = 127, + [2][0][RTW89_KCC][48] = 127, + [2][0][RTW89_ACMA][48] = 127, + [2][0][RTW89_CN][48] = 127, + [2][0][RTW89_UK][48] = 127, + [2][0][RTW89_FCC][50] = 64, + [2][0][RTW89_ETSI][50] = 127, + [2][0][RTW89_MKK][50] = 127, + [2][0][RTW89_IC][50] = 127, + [2][0][RTW89_KCC][50] = 127, + [2][0][RTW89_ACMA][50] = 127, + [2][0][RTW89_CN][50] = 127, + [2][0][RTW89_UK][50] = 127, + [2][0][RTW89_FCC][52] = 60, + [2][0][RTW89_ETSI][52] = 127, + [2][0][RTW89_MKK][52] = 127, + [2][0][RTW89_IC][52] = 127, + [2][0][RTW89_KCC][52] = 127, + [2][0][RTW89_ACMA][52] = 127, + [2][0][RTW89_CN][52] = 127, + [2][0][RTW89_UK][52] = 127, + [2][1][RTW89_FCC][0] = 127, + [2][1][RTW89_ETSI][0] = 127, + [2][1][RTW89_MKK][0] = 127, + [2][1][RTW89_IC][0] = 127, + [2][1][RTW89_KCC][0] = 127, + [2][1][RTW89_ACMA][0] = 127, + [2][1][RTW89_CN][0] = 28, + [2][1][RTW89_UK][0] = 127, + [2][1][RTW89_FCC][2] = 127, + [2][1][RTW89_ETSI][2] = 127, + [2][1][RTW89_MKK][2] = 127, + [2][1][RTW89_IC][2] = 127, + [2][1][RTW89_KCC][2] = 127, + [2][1][RTW89_ACMA][2] = 127, + [2][1][RTW89_CN][2] = 28, + [2][1][RTW89_UK][2] = 127, + [2][1][RTW89_FCC][4] = 127, + [2][1][RTW89_ETSI][4] = 127, + [2][1][RTW89_MKK][4] = 127, + [2][1][RTW89_IC][4] = 127, + [2][1][RTW89_KCC][4] = 127, + [2][1][RTW89_ACMA][4] = 127, + [2][1][RTW89_CN][4] = 28, + [2][1][RTW89_UK][4] = 127, + [2][1][RTW89_FCC][6] = 127, + [2][1][RTW89_ETSI][6] = 127, + [2][1][RTW89_MKK][6] = 127, + [2][1][RTW89_IC][6] = 127, + [2][1][RTW89_KCC][6] = 127, + [2][1][RTW89_ACMA][6] = 127, + [2][1][RTW89_CN][6] = 28, + [2][1][RTW89_UK][6] = 127, + [2][1][RTW89_FCC][8] = 127, + [2][1][RTW89_ETSI][8] = 127, + [2][1][RTW89_MKK][8] = 127, + [2][1][RTW89_IC][8] = 127, + [2][1][RTW89_KCC][8] = 127, + [2][1][RTW89_ACMA][8] = 127, + [2][1][RTW89_CN][8] = 28, + [2][1][RTW89_UK][8] = 127, + [2][1][RTW89_FCC][10] = 127, + [2][1][RTW89_ETSI][10] = 127, + [2][1][RTW89_MKK][10] = 127, + [2][1][RTW89_IC][10] = 127, + [2][1][RTW89_KCC][10] = 127, + [2][1][RTW89_ACMA][10] = 127, + [2][1][RTW89_CN][10] = 28, + [2][1][RTW89_UK][10] = 127, + [2][1][RTW89_FCC][12] = 127, + [2][1][RTW89_ETSI][12] = 127, + [2][1][RTW89_MKK][12] = 127, + [2][1][RTW89_IC][12] = 127, + [2][1][RTW89_KCC][12] = 127, + [2][1][RTW89_ACMA][12] = 127, + [2][1][RTW89_CN][12] = 28, + [2][1][RTW89_UK][12] = 127, + [2][1][RTW89_FCC][14] = 127, + [2][1][RTW89_ETSI][14] = 127, + [2][1][RTW89_MKK][14] = 127, + [2][1][RTW89_IC][14] = 127, + [2][1][RTW89_KCC][14] = 127, + [2][1][RTW89_ACMA][14] = 127, + [2][1][RTW89_CN][14] = 28, + [2][1][RTW89_UK][14] = 127, + [2][1][RTW89_FCC][15] = 127, + [2][1][RTW89_ETSI][15] = 127, + [2][1][RTW89_MKK][15] = 127, + [2][1][RTW89_IC][15] = 127, + [2][1][RTW89_KCC][15] = 127, + [2][1][RTW89_ACMA][15] = 127, + [2][1][RTW89_CN][15] = 127, + [2][1][RTW89_UK][15] = 127, + [2][1][RTW89_FCC][17] = 127, + [2][1][RTW89_ETSI][17] = 127, + [2][1][RTW89_MKK][17] = 127, + [2][1][RTW89_IC][17] = 127, + [2][1][RTW89_KCC][17] = 127, + [2][1][RTW89_ACMA][17] = 127, + [2][1][RTW89_CN][17] = 127, + [2][1][RTW89_UK][17] = 127, + [2][1][RTW89_FCC][19] = 127, + [2][1][RTW89_ETSI][19] = 127, + [2][1][RTW89_MKK][19] = 127, + [2][1][RTW89_IC][19] = 127, + [2][1][RTW89_KCC][19] = 127, + [2][1][RTW89_ACMA][19] = 127, + [2][1][RTW89_CN][19] = 127, + [2][1][RTW89_UK][19] = 127, + [2][1][RTW89_FCC][21] = 127, + [2][1][RTW89_ETSI][21] = 127, + [2][1][RTW89_MKK][21] = 127, + [2][1][RTW89_IC][21] = 127, + [2][1][RTW89_KCC][21] = 127, + [2][1][RTW89_ACMA][21] = 127, + [2][1][RTW89_CN][21] = 127, + [2][1][RTW89_UK][21] = 127, + [2][1][RTW89_FCC][23] = 127, + [2][1][RTW89_ETSI][23] = 127, + [2][1][RTW89_MKK][23] = 127, + [2][1][RTW89_IC][23] = 127, + [2][1][RTW89_KCC][23] = 127, + [2][1][RTW89_ACMA][23] = 127, + [2][1][RTW89_CN][23] = 127, + [2][1][RTW89_UK][23] = 127, + [2][1][RTW89_FCC][25] = 127, + [2][1][RTW89_ETSI][25] = 127, + [2][1][RTW89_MKK][25] = 127, + [2][1][RTW89_IC][25] = 127, + [2][1][RTW89_KCC][25] = 127, + [2][1][RTW89_ACMA][25] = 127, + [2][1][RTW89_CN][25] = 127, + [2][1][RTW89_UK][25] = 127, + [2][1][RTW89_FCC][27] = 127, + [2][1][RTW89_ETSI][27] = 127, + [2][1][RTW89_MKK][27] = 127, + [2][1][RTW89_IC][27] = 127, + [2][1][RTW89_KCC][27] = 127, + [2][1][RTW89_ACMA][27] = 127, + [2][1][RTW89_CN][27] = 127, + [2][1][RTW89_UK][27] = 127, + [2][1][RTW89_FCC][29] = 127, + [2][1][RTW89_ETSI][29] = 127, + [2][1][RTW89_MKK][29] = 127, + [2][1][RTW89_IC][29] = 127, + [2][1][RTW89_KCC][29] = 127, + [2][1][RTW89_ACMA][29] = 127, + [2][1][RTW89_CN][29] = 127, + [2][1][RTW89_UK][29] = 127, + [2][1][RTW89_FCC][31] = 127, + [2][1][RTW89_ETSI][31] = 127, + [2][1][RTW89_MKK][31] = 127, + [2][1][RTW89_IC][31] = 127, + [2][1][RTW89_KCC][31] = 127, + [2][1][RTW89_ACMA][31] = 127, + [2][1][RTW89_CN][31] = 127, + [2][1][RTW89_UK][31] = 127, + [2][1][RTW89_FCC][33] = 127, + [2][1][RTW89_ETSI][33] = 127, + [2][1][RTW89_MKK][33] = 127, + [2][1][RTW89_IC][33] = 127, + [2][1][RTW89_KCC][33] = 127, + [2][1][RTW89_ACMA][33] = 127, + [2][1][RTW89_CN][33] = 127, + [2][1][RTW89_UK][33] = 127, + [2][1][RTW89_FCC][35] = 127, + [2][1][RTW89_ETSI][35] = 127, + [2][1][RTW89_MKK][35] = 127, + [2][1][RTW89_IC][35] = 127, + [2][1][RTW89_KCC][35] = 127, + [2][1][RTW89_ACMA][35] = 127, + [2][1][RTW89_CN][35] = 127, + [2][1][RTW89_UK][35] = 127, + [2][1][RTW89_FCC][37] = 127, + [2][1][RTW89_ETSI][37] = 127, + [2][1][RTW89_MKK][37] = 127, + [2][1][RTW89_IC][37] = 127, + [2][1][RTW89_KCC][37] = 127, + [2][1][RTW89_ACMA][37] = 127, + [2][1][RTW89_CN][37] = 127, + [2][1][RTW89_UK][37] = 127, + [2][1][RTW89_FCC][38] = 127, + [2][1][RTW89_ETSI][38] = 127, + [2][1][RTW89_MKK][38] = 127, + [2][1][RTW89_IC][38] = 127, + [2][1][RTW89_KCC][38] = 127, + [2][1][RTW89_ACMA][38] = 127, + [2][1][RTW89_CN][38] = 56, + [2][1][RTW89_UK][38] = 127, + [2][1][RTW89_FCC][40] = 127, + [2][1][RTW89_ETSI][40] = 127, + [2][1][RTW89_MKK][40] = 127, + [2][1][RTW89_IC][40] = 127, + [2][1][RTW89_KCC][40] = 127, + [2][1][RTW89_ACMA][40] = 127, + [2][1][RTW89_CN][40] = 56, + [2][1][RTW89_UK][40] = 127, + [2][1][RTW89_FCC][42] = 127, + [2][1][RTW89_ETSI][42] = 127, + [2][1][RTW89_MKK][42] = 127, + [2][1][RTW89_IC][42] = 127, + [2][1][RTW89_KCC][42] = 127, + [2][1][RTW89_ACMA][42] = 127, + [2][1][RTW89_CN][42] = 56, + [2][1][RTW89_UK][42] = 127, + [2][1][RTW89_FCC][44] = 127, + [2][1][RTW89_ETSI][44] = 127, + [2][1][RTW89_MKK][44] = 127, + [2][1][RTW89_IC][44] = 127, + [2][1][RTW89_KCC][44] = 127, + [2][1][RTW89_ACMA][44] = 127, + [2][1][RTW89_CN][44] = 56, + [2][1][RTW89_UK][44] = 127, + [2][1][RTW89_FCC][46] = 127, + [2][1][RTW89_ETSI][46] = 127, + [2][1][RTW89_MKK][46] = 127, + [2][1][RTW89_IC][46] = 127, + [2][1][RTW89_KCC][46] = 127, + [2][1][RTW89_ACMA][46] = 127, + [2][1][RTW89_CN][46] = 56, + [2][1][RTW89_UK][46] = 127, + [2][1][RTW89_FCC][48] = 127, + [2][1][RTW89_ETSI][48] = 127, + [2][1][RTW89_MKK][48] = 127, + [2][1][RTW89_IC][48] = 127, + [2][1][RTW89_KCC][48] = 127, + [2][1][RTW89_ACMA][48] = 127, + [2][1][RTW89_CN][48] = 127, + [2][1][RTW89_UK][48] = 127, + [2][1][RTW89_FCC][50] = 127, + [2][1][RTW89_ETSI][50] = 127, + [2][1][RTW89_MKK][50] = 127, + [2][1][RTW89_IC][50] = 127, + [2][1][RTW89_KCC][50] = 127, + [2][1][RTW89_ACMA][50] = 127, + [2][1][RTW89_CN][50] = 127, + [2][1][RTW89_UK][50] = 127, + [2][1][RTW89_FCC][52] = 127, + [2][1][RTW89_ETSI][52] = 127, + [2][1][RTW89_MKK][52] = 127, + [2][1][RTW89_IC][52] = 127, + [2][1][RTW89_KCC][52] = 127, + [2][1][RTW89_ACMA][52] = 127, + [2][1][RTW89_CN][52] = 127, + [2][1][RTW89_UK][52] = 127, +}; + +static +const s8 rtw89_8851b_txpwr_lmt_2g_type2[RTW89_2G_BW_NUM][RTW89_NTX_NUM] + [RTW89_RS_LMT_NUM][RTW89_BF_NUM] + [RTW89_REGD_NUM][RTW89_2G_CH_NUM] = { + [0][0][0][0][RTW89_WW][0] = 58, + [0][0][0][0][RTW89_WW][1] = 58, + [0][0][0][0][RTW89_WW][2] = 58, + [0][0][0][0][RTW89_WW][3] = 58, + [0][0][0][0][RTW89_WW][4] = 58, + [0][0][0][0][RTW89_WW][5] = 58, + [0][0][0][0][RTW89_WW][6] = 58, + [0][0][0][0][RTW89_WW][7] = 58, + [0][0][0][0][RTW89_WW][8] = 58, + [0][0][0][0][RTW89_WW][9] = 58, + [0][0][0][0][RTW89_WW][10] = 58, + [0][0][0][0][RTW89_WW][11] = 58, + [0][0][0][0][RTW89_WW][12] = 52, + [0][0][0][0][RTW89_WW][13] = 76, + [0][1][0][0][RTW89_WW][0] = 0, + [0][1][0][0][RTW89_WW][1] = 0, + [0][1][0][0][RTW89_WW][2] = 0, + [0][1][0][0][RTW89_WW][3] = 0, + [0][1][0][0][RTW89_WW][4] = 0, + [0][1][0][0][RTW89_WW][5] = 0, + [0][1][0][0][RTW89_WW][6] = 0, + [0][1][0][0][RTW89_WW][7] = 0, + [0][1][0][0][RTW89_WW][8] = 0, + [0][1][0][0][RTW89_WW][9] = 0, + [0][1][0][0][RTW89_WW][10] = 0, + [0][1][0][0][RTW89_WW][11] = 0, + [0][1][0][0][RTW89_WW][12] = 0, + [0][1][0][0][RTW89_WW][13] = 0, + [1][0][0][0][RTW89_WW][0] = 0, + [1][0][0][0][RTW89_WW][1] = 0, + [1][0][0][0][RTW89_WW][2] = 58, + [1][0][0][0][RTW89_WW][3] = 58, + [1][0][0][0][RTW89_WW][4] = 58, + [1][0][0][0][RTW89_WW][5] = 58, + [1][0][0][0][RTW89_WW][6] = 58, + [1][0][0][0][RTW89_WW][7] = 58, + [1][0][0][0][RTW89_WW][8] = 58, + [1][0][0][0][RTW89_WW][9] = 58, + [1][0][0][0][RTW89_WW][10] = 58, + [1][0][0][0][RTW89_WW][11] = 0, + [1][0][0][0][RTW89_WW][12] = 0, + [1][0][0][0][RTW89_WW][13] = 0, + [1][1][0][0][RTW89_WW][0] = 0, + [1][1][0][0][RTW89_WW][1] = 0, + [1][1][0][0][RTW89_WW][2] = 0, + [1][1][0][0][RTW89_WW][3] = 0, + [1][1][0][0][RTW89_WW][4] = 0, + [1][1][0][0][RTW89_WW][5] = 0, + [1][1][0][0][RTW89_WW][6] = 0, + [1][1][0][0][RTW89_WW][7] = 0, + [1][1][0][0][RTW89_WW][8] = 0, + [1][1][0][0][RTW89_WW][9] = 0, + [1][1][0][0][RTW89_WW][10] = 0, + [1][1][0][0][RTW89_WW][11] = 0, + [1][1][0][0][RTW89_WW][12] = 0, + [1][1][0][0][RTW89_WW][13] = 0, + [0][0][1][0][RTW89_WW][0] = 58, + [0][0][1][0][RTW89_WW][1] = 60, + [0][0][1][0][RTW89_WW][2] = 60, + [0][0][1][0][RTW89_WW][3] = 60, + [0][0][1][0][RTW89_WW][4] = 60, + [0][0][1][0][RTW89_WW][5] = 60, + [0][0][1][0][RTW89_WW][6] = 60, + [0][0][1][0][RTW89_WW][7] = 60, + [0][0][1][0][RTW89_WW][8] = 60, + [0][0][1][0][RTW89_WW][9] = 60, + [0][0][1][0][RTW89_WW][10] = 60, + [0][0][1][0][RTW89_WW][11] = 60, + [0][0][1][0][RTW89_WW][12] = 58, + [0][0][1][0][RTW89_WW][13] = 0, + [0][1][1][0][RTW89_WW][0] = 0, + [0][1][1][0][RTW89_WW][1] = 0, + [0][1][1][0][RTW89_WW][2] = 0, + [0][1][1][0][RTW89_WW][3] = 0, + [0][1][1][0][RTW89_WW][4] = 0, + [0][1][1][0][RTW89_WW][5] = 0, + [0][1][1][0][RTW89_WW][6] = 0, + [0][1][1][0][RTW89_WW][7] = 0, + [0][1][1][0][RTW89_WW][8] = 0, + [0][1][1][0][RTW89_WW][9] = 0, + [0][1][1][0][RTW89_WW][10] = 0, + [0][1][1][0][RTW89_WW][11] = 0, + [0][1][1][0][RTW89_WW][12] = 0, + [0][1][1][0][RTW89_WW][13] = 0, + [0][0][2][0][RTW89_WW][0] = 60, + [0][0][2][0][RTW89_WW][1] = 60, + [0][0][2][0][RTW89_WW][2] = 60, + [0][0][2][0][RTW89_WW][3] = 60, + [0][0][2][0][RTW89_WW][4] = 60, + [0][0][2][0][RTW89_WW][5] = 60, + [0][0][2][0][RTW89_WW][6] = 60, + [0][0][2][0][RTW89_WW][7] = 60, + [0][0][2][0][RTW89_WW][8] = 60, + [0][0][2][0][RTW89_WW][9] = 60, + [0][0][2][0][RTW89_WW][10] = 60, + [0][0][2][0][RTW89_WW][11] = 60, + [0][0][2][0][RTW89_WW][12] = 60, + [0][0][2][0][RTW89_WW][13] = 0, + [0][1][2][0][RTW89_WW][0] = 0, + [0][1][2][0][RTW89_WW][1] = 0, + [0][1][2][0][RTW89_WW][2] = 0, + [0][1][2][0][RTW89_WW][3] = 0, + [0][1][2][0][RTW89_WW][4] = 0, + [0][1][2][0][RTW89_WW][5] = 0, + [0][1][2][0][RTW89_WW][6] = 0, + [0][1][2][0][RTW89_WW][7] = 0, + [0][1][2][0][RTW89_WW][8] = 0, + [0][1][2][0][RTW89_WW][9] = 0, + [0][1][2][0][RTW89_WW][10] = 0, + [0][1][2][0][RTW89_WW][11] = 0, + [0][1][2][0][RTW89_WW][12] = 0, + [0][1][2][0][RTW89_WW][13] = 0, + [0][1][2][1][RTW89_WW][0] = 0, + [0][1][2][1][RTW89_WW][1] = 0, + [0][1][2][1][RTW89_WW][2] = 0, + [0][1][2][1][RTW89_WW][3] = 0, + [0][1][2][1][RTW89_WW][4] = 0, + [0][1][2][1][RTW89_WW][5] = 0, + [0][1][2][1][RTW89_WW][6] = 0, + [0][1][2][1][RTW89_WW][7] = 0, + [0][1][2][1][RTW89_WW][8] = 0, + [0][1][2][1][RTW89_WW][9] = 0, + [0][1][2][1][RTW89_WW][10] = 0, + [0][1][2][1][RTW89_WW][11] = 0, + [0][1][2][1][RTW89_WW][12] = 0, + [0][1][2][1][RTW89_WW][13] = 0, + [1][0][2][0][RTW89_WW][0] = 0, + [1][0][2][0][RTW89_WW][1] = 0, + [1][0][2][0][RTW89_WW][2] = 58, + [1][0][2][0][RTW89_WW][3] = 58, + [1][0][2][0][RTW89_WW][4] = 58, + [1][0][2][0][RTW89_WW][5] = 58, + [1][0][2][0][RTW89_WW][6] = 58, + [1][0][2][0][RTW89_WW][7] = 58, + [1][0][2][0][RTW89_WW][8] = 58, + [1][0][2][0][RTW89_WW][9] = 58, + [1][0][2][0][RTW89_WW][10] = 58, + [1][0][2][0][RTW89_WW][11] = 0, + [1][0][2][0][RTW89_WW][12] = 0, + [1][0][2][0][RTW89_WW][13] = 0, + [1][1][2][0][RTW89_WW][0] = 0, + [1][1][2][0][RTW89_WW][1] = 0, + [1][1][2][0][RTW89_WW][2] = 0, + [1][1][2][0][RTW89_WW][3] = 0, + [1][1][2][0][RTW89_WW][4] = 0, + [1][1][2][0][RTW89_WW][5] = 0, + [1][1][2][0][RTW89_WW][6] = 0, + [1][1][2][0][RTW89_WW][7] = 0, + [1][1][2][0][RTW89_WW][8] = 0, + [1][1][2][0][RTW89_WW][9] = 0, + [1][1][2][0][RTW89_WW][10] = 0, + [1][1][2][0][RTW89_WW][11] = 0, + [1][1][2][0][RTW89_WW][12] = 0, + [1][1][2][0][RTW89_WW][13] = 0, + [1][1][2][1][RTW89_WW][0] = 0, + [1][1][2][1][RTW89_WW][1] = 0, + [1][1][2][1][RTW89_WW][2] = 0, + [1][1][2][1][RTW89_WW][3] = 0, + [1][1][2][1][RTW89_WW][4] = 0, + [1][1][2][1][RTW89_WW][5] = 0, + [1][1][2][1][RTW89_WW][6] = 0, + [1][1][2][1][RTW89_WW][7] = 0, + [1][1][2][1][RTW89_WW][8] = 0, + [1][1][2][1][RTW89_WW][9] = 0, + [1][1][2][1][RTW89_WW][10] = 0, + [1][1][2][1][RTW89_WW][11] = 0, + [1][1][2][1][RTW89_WW][12] = 0, + [1][1][2][1][RTW89_WW][13] = 0, + [0][0][0][0][RTW89_FCC][0] = 82, + [0][0][0][0][RTW89_ETSI][0] = 58, + [0][0][0][0][RTW89_MKK][0] = 68, + [0][0][0][0][RTW89_IC][0] = 82, + [0][0][0][0][RTW89_KCC][0] = 68, + [0][0][0][0][RTW89_ACMA][0] = 58, + [0][0][0][0][RTW89_CN][0] = 60, + [0][0][0][0][RTW89_UK][0] = 58, + [0][0][0][0][RTW89_FCC][1] = 82, + [0][0][0][0][RTW89_ETSI][1] = 58, + [0][0][0][0][RTW89_MKK][1] = 68, + [0][0][0][0][RTW89_IC][1] = 82, + [0][0][0][0][RTW89_KCC][1] = 68, + [0][0][0][0][RTW89_ACMA][1] = 58, + [0][0][0][0][RTW89_CN][1] = 60, + [0][0][0][0][RTW89_UK][1] = 58, + [0][0][0][0][RTW89_FCC][2] = 82, + [0][0][0][0][RTW89_ETSI][2] = 58, + [0][0][0][0][RTW89_MKK][2] = 68, + [0][0][0][0][RTW89_IC][2] = 82, + [0][0][0][0][RTW89_KCC][2] = 68, + [0][0][0][0][RTW89_ACMA][2] = 58, + [0][0][0][0][RTW89_CN][2] = 60, + [0][0][0][0][RTW89_UK][2] = 58, + [0][0][0][0][RTW89_FCC][3] = 82, + [0][0][0][0][RTW89_ETSI][3] = 58, + [0][0][0][0][RTW89_MKK][3] = 68, + [0][0][0][0][RTW89_IC][3] = 82, + [0][0][0][0][RTW89_KCC][3] = 68, + [0][0][0][0][RTW89_ACMA][3] = 58, + [0][0][0][0][RTW89_CN][3] = 60, + [0][0][0][0][RTW89_UK][3] = 58, + [0][0][0][0][RTW89_FCC][4] = 82, + [0][0][0][0][RTW89_ETSI][4] = 58, + [0][0][0][0][RTW89_MKK][4] = 68, + [0][0][0][0][RTW89_IC][4] = 82, + [0][0][0][0][RTW89_KCC][4] = 68, + [0][0][0][0][RTW89_ACMA][4] = 58, + [0][0][0][0][RTW89_CN][4] = 60, + [0][0][0][0][RTW89_UK][4] = 58, + [0][0][0][0][RTW89_FCC][5] = 82, + [0][0][0][0][RTW89_ETSI][5] = 58, + [0][0][0][0][RTW89_MKK][5] = 68, + [0][0][0][0][RTW89_IC][5] = 82, + [0][0][0][0][RTW89_KCC][5] = 68, + [0][0][0][0][RTW89_ACMA][5] = 58, + [0][0][0][0][RTW89_CN][5] = 60, + [0][0][0][0][RTW89_UK][5] = 58, + [0][0][0][0][RTW89_FCC][6] = 82, + [0][0][0][0][RTW89_ETSI][6] = 58, + [0][0][0][0][RTW89_MKK][6] = 68, + [0][0][0][0][RTW89_IC][6] = 82, + [0][0][0][0][RTW89_KCC][6] = 68, + [0][0][0][0][RTW89_ACMA][6] = 58, + [0][0][0][0][RTW89_CN][6] = 60, + [0][0][0][0][RTW89_UK][6] = 58, + [0][0][0][0][RTW89_FCC][7] = 82, + [0][0][0][0][RTW89_ETSI][7] = 58, + [0][0][0][0][RTW89_MKK][7] = 68, + [0][0][0][0][RTW89_IC][7] = 82, + [0][0][0][0][RTW89_KCC][7] = 68, + [0][0][0][0][RTW89_ACMA][7] = 58, + [0][0][0][0][RTW89_CN][7] = 60, + [0][0][0][0][RTW89_UK][7] = 58, + [0][0][0][0][RTW89_FCC][8] = 82, + [0][0][0][0][RTW89_ETSI][8] = 58, + [0][0][0][0][RTW89_MKK][8] = 68, + [0][0][0][0][RTW89_IC][8] = 82, + [0][0][0][0][RTW89_KCC][8] = 68, + [0][0][0][0][RTW89_ACMA][8] = 58, + [0][0][0][0][RTW89_CN][8] = 60, + [0][0][0][0][RTW89_UK][8] = 58, + [0][0][0][0][RTW89_FCC][9] = 82, + [0][0][0][0][RTW89_ETSI][9] = 58, + [0][0][0][0][RTW89_MKK][9] = 68, + [0][0][0][0][RTW89_IC][9] = 82, + [0][0][0][0][RTW89_KCC][9] = 68, + [0][0][0][0][RTW89_ACMA][9] = 58, + [0][0][0][0][RTW89_CN][9] = 60, + [0][0][0][0][RTW89_UK][9] = 58, + [0][0][0][0][RTW89_FCC][10] = 80, + [0][0][0][0][RTW89_ETSI][10] = 58, + [0][0][0][0][RTW89_MKK][10] = 68, + [0][0][0][0][RTW89_IC][10] = 80, + [0][0][0][0][RTW89_KCC][10] = 68, + [0][0][0][0][RTW89_ACMA][10] = 58, + [0][0][0][0][RTW89_CN][10] = 60, + [0][0][0][0][RTW89_UK][10] = 58, + [0][0][0][0][RTW89_FCC][11] = 60, + [0][0][0][0][RTW89_ETSI][11] = 58, + [0][0][0][0][RTW89_MKK][11] = 68, + [0][0][0][0][RTW89_IC][11] = 60, + [0][0][0][0][RTW89_KCC][11] = 68, + [0][0][0][0][RTW89_ACMA][11] = 58, + [0][0][0][0][RTW89_CN][11] = 60, + [0][0][0][0][RTW89_UK][11] = 58, + [0][0][0][0][RTW89_FCC][12] = 52, + [0][0][0][0][RTW89_ETSI][12] = 58, + [0][0][0][0][RTW89_MKK][12] = 68, + [0][0][0][0][RTW89_IC][12] = 52, + [0][0][0][0][RTW89_KCC][12] = 68, + [0][0][0][0][RTW89_ACMA][12] = 58, + [0][0][0][0][RTW89_CN][12] = 60, + [0][0][0][0][RTW89_UK][12] = 58, + [0][0][0][0][RTW89_FCC][13] = 127, + [0][0][0][0][RTW89_ETSI][13] = 127, + [0][0][0][0][RTW89_MKK][13] = 76, + [0][0][0][0][RTW89_IC][13] = 127, + [0][0][0][0][RTW89_KCC][13] = 127, + [0][0][0][0][RTW89_ACMA][13] = 127, + [0][0][0][0][RTW89_CN][13] = 127, + [0][0][0][0][RTW89_UK][13] = 127, + [0][1][0][0][RTW89_FCC][0] = 127, + [0][1][0][0][RTW89_ETSI][0] = 127, + [0][1][0][0][RTW89_MKK][0] = 127, + [0][1][0][0][RTW89_IC][0] = 127, + [0][1][0][0][RTW89_KCC][0] = 127, + [0][1][0][0][RTW89_ACMA][0] = 127, + [0][1][0][0][RTW89_CN][0] = 127, + [0][1][0][0][RTW89_UK][0] = 127, + [0][1][0][0][RTW89_FCC][1] = 127, + [0][1][0][0][RTW89_ETSI][1] = 127, + [0][1][0][0][RTW89_MKK][1] = 127, + [0][1][0][0][RTW89_IC][1] = 127, + [0][1][0][0][RTW89_KCC][1] = 127, + [0][1][0][0][RTW89_ACMA][1] = 127, + [0][1][0][0][RTW89_CN][1] = 127, + [0][1][0][0][RTW89_UK][1] = 127, + [0][1][0][0][RTW89_FCC][2] = 127, + [0][1][0][0][RTW89_ETSI][2] = 127, + [0][1][0][0][RTW89_MKK][2] = 127, + [0][1][0][0][RTW89_IC][2] = 127, + [0][1][0][0][RTW89_KCC][2] = 127, + [0][1][0][0][RTW89_ACMA][2] = 127, + [0][1][0][0][RTW89_CN][2] = 127, + [0][1][0][0][RTW89_UK][2] = 127, + [0][1][0][0][RTW89_FCC][3] = 127, + [0][1][0][0][RTW89_ETSI][3] = 127, + [0][1][0][0][RTW89_MKK][3] = 127, + [0][1][0][0][RTW89_IC][3] = 127, + [0][1][0][0][RTW89_KCC][3] = 127, + [0][1][0][0][RTW89_ACMA][3] = 127, + [0][1][0][0][RTW89_CN][3] = 127, + [0][1][0][0][RTW89_UK][3] = 127, + [0][1][0][0][RTW89_FCC][4] = 127, + [0][1][0][0][RTW89_ETSI][4] = 127, + [0][1][0][0][RTW89_MKK][4] = 127, + [0][1][0][0][RTW89_IC][4] = 127, + [0][1][0][0][RTW89_KCC][4] = 127, + [0][1][0][0][RTW89_ACMA][4] = 127, + [0][1][0][0][RTW89_CN][4] = 127, + [0][1][0][0][RTW89_UK][4] = 127, + [0][1][0][0][RTW89_FCC][5] = 127, + [0][1][0][0][RTW89_ETSI][5] = 127, + [0][1][0][0][RTW89_MKK][5] = 127, + [0][1][0][0][RTW89_IC][5] = 127, + [0][1][0][0][RTW89_KCC][5] = 127, + [0][1][0][0][RTW89_ACMA][5] = 127, + [0][1][0][0][RTW89_CN][5] = 127, + [0][1][0][0][RTW89_UK][5] = 127, + [0][1][0][0][RTW89_FCC][6] = 127, + [0][1][0][0][RTW89_ETSI][6] = 127, + [0][1][0][0][RTW89_MKK][6] = 127, + [0][1][0][0][RTW89_IC][6] = 127, + [0][1][0][0][RTW89_KCC][6] = 127, + [0][1][0][0][RTW89_ACMA][6] = 127, + [0][1][0][0][RTW89_CN][6] = 127, + [0][1][0][0][RTW89_UK][6] = 127, + [0][1][0][0][RTW89_FCC][7] = 127, + [0][1][0][0][RTW89_ETSI][7] = 127, + [0][1][0][0][RTW89_MKK][7] = 127, + [0][1][0][0][RTW89_IC][7] = 127, + [0][1][0][0][RTW89_KCC][7] = 127, + [0][1][0][0][RTW89_ACMA][7] = 127, + [0][1][0][0][RTW89_CN][7] = 127, + [0][1][0][0][RTW89_UK][7] = 127, + [0][1][0][0][RTW89_FCC][8] = 127, + [0][1][0][0][RTW89_ETSI][8] = 127, + [0][1][0][0][RTW89_MKK][8] = 127, + [0][1][0][0][RTW89_IC][8] = 127, + [0][1][0][0][RTW89_KCC][8] = 127, + [0][1][0][0][RTW89_ACMA][8] = 127, + [0][1][0][0][RTW89_CN][8] = 127, + [0][1][0][0][RTW89_UK][8] = 127, + [0][1][0][0][RTW89_FCC][9] = 127, + [0][1][0][0][RTW89_ETSI][9] = 127, + [0][1][0][0][RTW89_MKK][9] = 127, + [0][1][0][0][RTW89_IC][9] = 127, + [0][1][0][0][RTW89_KCC][9] = 127, + [0][1][0][0][RTW89_ACMA][9] = 127, + [0][1][0][0][RTW89_CN][9] = 127, + [0][1][0][0][RTW89_UK][9] = 127, + [0][1][0][0][RTW89_FCC][10] = 127, + [0][1][0][0][RTW89_ETSI][10] = 127, + [0][1][0][0][RTW89_MKK][10] = 127, + [0][1][0][0][RTW89_IC][10] = 127, + [0][1][0][0][RTW89_KCC][10] = 127, + [0][1][0][0][RTW89_ACMA][10] = 127, + [0][1][0][0][RTW89_CN][10] = 127, + [0][1][0][0][RTW89_UK][10] = 127, + [0][1][0][0][RTW89_FCC][11] = 127, + [0][1][0][0][RTW89_ETSI][11] = 127, + [0][1][0][0][RTW89_MKK][11] = 127, + [0][1][0][0][RTW89_IC][11] = 127, + [0][1][0][0][RTW89_KCC][11] = 127, + [0][1][0][0][RTW89_ACMA][11] = 127, + [0][1][0][0][RTW89_CN][11] = 127, + [0][1][0][0][RTW89_UK][11] = 127, + [0][1][0][0][RTW89_FCC][12] = 127, + [0][1][0][0][RTW89_ETSI][12] = 127, + [0][1][0][0][RTW89_MKK][12] = 127, + [0][1][0][0][RTW89_IC][12] = 127, + [0][1][0][0][RTW89_KCC][12] = 127, + [0][1][0][0][RTW89_ACMA][12] = 127, + [0][1][0][0][RTW89_CN][12] = 127, + [0][1][0][0][RTW89_UK][12] = 127, + [0][1][0][0][RTW89_FCC][13] = 127, + [0][1][0][0][RTW89_ETSI][13] = 127, + [0][1][0][0][RTW89_MKK][13] = 127, + [0][1][0][0][RTW89_IC][13] = 127, + [0][1][0][0][RTW89_KCC][13] = 127, + [0][1][0][0][RTW89_ACMA][13] = 127, + [0][1][0][0][RTW89_CN][13] = 127, + [0][1][0][0][RTW89_UK][13] = 127, + [1][0][0][0][RTW89_FCC][0] = 127, + [1][0][0][0][RTW89_ETSI][0] = 127, + [1][0][0][0][RTW89_MKK][0] = 127, + [1][0][0][0][RTW89_IC][0] = 127, + [1][0][0][0][RTW89_KCC][0] = 127, + [1][0][0][0][RTW89_ACMA][0] = 127, + [1][0][0][0][RTW89_CN][0] = 127, + [1][0][0][0][RTW89_UK][0] = 127, + [1][0][0][0][RTW89_FCC][1] = 127, + [1][0][0][0][RTW89_ETSI][1] = 127, + [1][0][0][0][RTW89_MKK][1] = 127, + [1][0][0][0][RTW89_IC][1] = 127, + [1][0][0][0][RTW89_KCC][1] = 127, + [1][0][0][0][RTW89_ACMA][1] = 127, + [1][0][0][0][RTW89_CN][1] = 127, + [1][0][0][0][RTW89_UK][1] = 127, + [1][0][0][0][RTW89_FCC][2] = 127, + [1][0][0][0][RTW89_ETSI][2] = 58, + [1][0][0][0][RTW89_MKK][2] = 70, + [1][0][0][0][RTW89_IC][2] = 127, + [1][0][0][0][RTW89_KCC][2] = 68, + [1][0][0][0][RTW89_ACMA][2] = 58, + [1][0][0][0][RTW89_CN][2] = 60, + [1][0][0][0][RTW89_UK][2] = 58, + [1][0][0][0][RTW89_FCC][3] = 127, + [1][0][0][0][RTW89_ETSI][3] = 58, + [1][0][0][0][RTW89_MKK][3] = 76, + [1][0][0][0][RTW89_IC][3] = 127, + [1][0][0][0][RTW89_KCC][3] = 68, + [1][0][0][0][RTW89_ACMA][3] = 58, + [1][0][0][0][RTW89_CN][3] = 60, + [1][0][0][0][RTW89_UK][3] = 58, + [1][0][0][0][RTW89_FCC][4] = 127, + [1][0][0][0][RTW89_ETSI][4] = 58, + [1][0][0][0][RTW89_MKK][4] = 76, + [1][0][0][0][RTW89_IC][4] = 127, + [1][0][0][0][RTW89_KCC][4] = 68, + [1][0][0][0][RTW89_ACMA][4] = 58, + [1][0][0][0][RTW89_CN][4] = 60, + [1][0][0][0][RTW89_UK][4] = 58, + [1][0][0][0][RTW89_FCC][5] = 127, + [1][0][0][0][RTW89_ETSI][5] = 58, + [1][0][0][0][RTW89_MKK][5] = 76, + [1][0][0][0][RTW89_IC][5] = 127, + [1][0][0][0][RTW89_KCC][5] = 68, + [1][0][0][0][RTW89_ACMA][5] = 58, + [1][0][0][0][RTW89_CN][5] = 60, + [1][0][0][0][RTW89_UK][5] = 58, + [1][0][0][0][RTW89_FCC][6] = 127, + [1][0][0][0][RTW89_ETSI][6] = 58, + [1][0][0][0][RTW89_MKK][6] = 76, + [1][0][0][0][RTW89_IC][6] = 127, + [1][0][0][0][RTW89_KCC][6] = 68, + [1][0][0][0][RTW89_ACMA][6] = 58, + [1][0][0][0][RTW89_CN][6] = 60, + [1][0][0][0][RTW89_UK][6] = 58, + [1][0][0][0][RTW89_FCC][7] = 127, + [1][0][0][0][RTW89_ETSI][7] = 58, + [1][0][0][0][RTW89_MKK][7] = 76, + [1][0][0][0][RTW89_IC][7] = 127, + [1][0][0][0][RTW89_KCC][7] = 68, + [1][0][0][0][RTW89_ACMA][7] = 58, + [1][0][0][0][RTW89_CN][7] = 60, + [1][0][0][0][RTW89_UK][7] = 58, + [1][0][0][0][RTW89_FCC][8] = 127, + [1][0][0][0][RTW89_ETSI][8] = 58, + [1][0][0][0][RTW89_MKK][8] = 76, + [1][0][0][0][RTW89_IC][8] = 127, + [1][0][0][0][RTW89_KCC][8] = 68, + [1][0][0][0][RTW89_ACMA][8] = 58, + [1][0][0][0][RTW89_CN][8] = 60, + [1][0][0][0][RTW89_UK][8] = 58, + [1][0][0][0][RTW89_FCC][9] = 127, + [1][0][0][0][RTW89_ETSI][9] = 58, + [1][0][0][0][RTW89_MKK][9] = 76, + [1][0][0][0][RTW89_IC][9] = 127, + [1][0][0][0][RTW89_KCC][9] = 68, + [1][0][0][0][RTW89_ACMA][9] = 58, + [1][0][0][0][RTW89_CN][9] = 60, + [1][0][0][0][RTW89_UK][9] = 58, + [1][0][0][0][RTW89_FCC][10] = 127, + [1][0][0][0][RTW89_ETSI][10] = 58, + [1][0][0][0][RTW89_MKK][10] = 66, + [1][0][0][0][RTW89_IC][10] = 127, + [1][0][0][0][RTW89_KCC][10] = 68, + [1][0][0][0][RTW89_ACMA][10] = 58, + [1][0][0][0][RTW89_CN][10] = 60, + [1][0][0][0][RTW89_UK][10] = 58, + [1][0][0][0][RTW89_FCC][11] = 127, + [1][0][0][0][RTW89_ETSI][11] = 127, + [1][0][0][0][RTW89_MKK][11] = 127, + [1][0][0][0][RTW89_IC][11] = 127, + [1][0][0][0][RTW89_KCC][11] = 127, + [1][0][0][0][RTW89_ACMA][11] = 127, + [1][0][0][0][RTW89_CN][11] = 127, + [1][0][0][0][RTW89_UK][11] = 127, + [1][0][0][0][RTW89_FCC][12] = 127, + [1][0][0][0][RTW89_ETSI][12] = 127, + [1][0][0][0][RTW89_MKK][12] = 127, + [1][0][0][0][RTW89_IC][12] = 127, + [1][0][0][0][RTW89_KCC][12] = 127, + [1][0][0][0][RTW89_ACMA][12] = 127, + [1][0][0][0][RTW89_CN][12] = 127, + [1][0][0][0][RTW89_UK][12] = 127, + [1][0][0][0][RTW89_FCC][13] = 127, + [1][0][0][0][RTW89_ETSI][13] = 127, + [1][0][0][0][RTW89_MKK][13] = 127, + [1][0][0][0][RTW89_IC][13] = 127, + [1][0][0][0][RTW89_KCC][13] = 127, + [1][0][0][0][RTW89_ACMA][13] = 127, + [1][0][0][0][RTW89_CN][13] = 127, + [1][0][0][0][RTW89_UK][13] = 127, + [1][1][0][0][RTW89_FCC][0] = 127, + [1][1][0][0][RTW89_ETSI][0] = 127, + [1][1][0][0][RTW89_MKK][0] = 127, + [1][1][0][0][RTW89_IC][0] = 127, + [1][1][0][0][RTW89_KCC][0] = 127, + [1][1][0][0][RTW89_ACMA][0] = 127, + [1][1][0][0][RTW89_CN][0] = 127, + [1][1][0][0][RTW89_UK][0] = 127, + [1][1][0][0][RTW89_FCC][1] = 127, + [1][1][0][0][RTW89_ETSI][1] = 127, + [1][1][0][0][RTW89_MKK][1] = 127, + [1][1][0][0][RTW89_IC][1] = 127, + [1][1][0][0][RTW89_KCC][1] = 127, + [1][1][0][0][RTW89_ACMA][1] = 127, + [1][1][0][0][RTW89_CN][1] = 127, + [1][1][0][0][RTW89_UK][1] = 127, + [1][1][0][0][RTW89_FCC][2] = 127, + [1][1][0][0][RTW89_ETSI][2] = 127, + [1][1][0][0][RTW89_MKK][2] = 127, + [1][1][0][0][RTW89_IC][2] = 127, + [1][1][0][0][RTW89_KCC][2] = 127, + [1][1][0][0][RTW89_ACMA][2] = 127, + [1][1][0][0][RTW89_CN][2] = 127, + [1][1][0][0][RTW89_UK][2] = 127, + [1][1][0][0][RTW89_FCC][3] = 127, + [1][1][0][0][RTW89_ETSI][3] = 127, + [1][1][0][0][RTW89_MKK][3] = 127, + [1][1][0][0][RTW89_IC][3] = 127, + [1][1][0][0][RTW89_KCC][3] = 127, + [1][1][0][0][RTW89_ACMA][3] = 127, + [1][1][0][0][RTW89_CN][3] = 127, + [1][1][0][0][RTW89_UK][3] = 127, + [1][1][0][0][RTW89_FCC][4] = 127, + [1][1][0][0][RTW89_ETSI][4] = 127, + [1][1][0][0][RTW89_MKK][4] = 127, + [1][1][0][0][RTW89_IC][4] = 127, + [1][1][0][0][RTW89_KCC][4] = 127, + [1][1][0][0][RTW89_ACMA][4] = 127, + [1][1][0][0][RTW89_CN][4] = 127, + [1][1][0][0][RTW89_UK][4] = 127, + [1][1][0][0][RTW89_FCC][5] = 127, + [1][1][0][0][RTW89_ETSI][5] = 127, + [1][1][0][0][RTW89_MKK][5] = 127, + [1][1][0][0][RTW89_IC][5] = 127, + [1][1][0][0][RTW89_KCC][5] = 127, + [1][1][0][0][RTW89_ACMA][5] = 127, + [1][1][0][0][RTW89_CN][5] = 127, + [1][1][0][0][RTW89_UK][5] = 127, + [1][1][0][0][RTW89_FCC][6] = 127, + [1][1][0][0][RTW89_ETSI][6] = 127, + [1][1][0][0][RTW89_MKK][6] = 127, + [1][1][0][0][RTW89_IC][6] = 127, + [1][1][0][0][RTW89_KCC][6] = 127, + [1][1][0][0][RTW89_ACMA][6] = 127, + [1][1][0][0][RTW89_CN][6] = 127, + [1][1][0][0][RTW89_UK][6] = 127, + [1][1][0][0][RTW89_FCC][7] = 127, + [1][1][0][0][RTW89_ETSI][7] = 127, + [1][1][0][0][RTW89_MKK][7] = 127, + [1][1][0][0][RTW89_IC][7] = 127, + [1][1][0][0][RTW89_KCC][7] = 127, + [1][1][0][0][RTW89_ACMA][7] = 127, + [1][1][0][0][RTW89_CN][7] = 127, + [1][1][0][0][RTW89_UK][7] = 127, + [1][1][0][0][RTW89_FCC][8] = 127, + [1][1][0][0][RTW89_ETSI][8] = 127, + [1][1][0][0][RTW89_MKK][8] = 127, + [1][1][0][0][RTW89_IC][8] = 127, + [1][1][0][0][RTW89_KCC][8] = 127, + [1][1][0][0][RTW89_ACMA][8] = 127, + [1][1][0][0][RTW89_CN][8] = 127, + [1][1][0][0][RTW89_UK][8] = 127, + [1][1][0][0][RTW89_FCC][9] = 127, + [1][1][0][0][RTW89_ETSI][9] = 127, + [1][1][0][0][RTW89_MKK][9] = 127, + [1][1][0][0][RTW89_IC][9] = 127, + [1][1][0][0][RTW89_KCC][9] = 127, + [1][1][0][0][RTW89_ACMA][9] = 127, + [1][1][0][0][RTW89_CN][9] = 127, + [1][1][0][0][RTW89_UK][9] = 127, + [1][1][0][0][RTW89_FCC][10] = 127, + [1][1][0][0][RTW89_ETSI][10] = 127, + [1][1][0][0][RTW89_MKK][10] = 127, + [1][1][0][0][RTW89_IC][10] = 127, + [1][1][0][0][RTW89_KCC][10] = 127, + [1][1][0][0][RTW89_ACMA][10] = 127, + [1][1][0][0][RTW89_CN][10] = 127, + [1][1][0][0][RTW89_UK][10] = 127, + [1][1][0][0][RTW89_FCC][11] = 127, + [1][1][0][0][RTW89_ETSI][11] = 127, + [1][1][0][0][RTW89_MKK][11] = 127, + [1][1][0][0][RTW89_IC][11] = 127, + [1][1][0][0][RTW89_KCC][11] = 127, + [1][1][0][0][RTW89_ACMA][11] = 127, + [1][1][0][0][RTW89_CN][11] = 127, + [1][1][0][0][RTW89_UK][11] = 127, + [1][1][0][0][RTW89_FCC][12] = 127, + [1][1][0][0][RTW89_ETSI][12] = 127, + [1][1][0][0][RTW89_MKK][12] = 127, + [1][1][0][0][RTW89_IC][12] = 127, + [1][1][0][0][RTW89_KCC][12] = 127, + [1][1][0][0][RTW89_ACMA][12] = 127, + [1][1][0][0][RTW89_CN][12] = 127, + [1][1][0][0][RTW89_UK][12] = 127, + [1][1][0][0][RTW89_FCC][13] = 127, + [1][1][0][0][RTW89_ETSI][13] = 127, + [1][1][0][0][RTW89_MKK][13] = 127, + [1][1][0][0][RTW89_IC][13] = 127, + [1][1][0][0][RTW89_KCC][13] = 127, + [1][1][0][0][RTW89_ACMA][13] = 127, + [1][1][0][0][RTW89_CN][13] = 127, + [1][1][0][0][RTW89_UK][13] = 127, + [0][0][1][0][RTW89_FCC][0] = 78, + [0][0][1][0][RTW89_ETSI][0] = 58, + [0][0][1][0][RTW89_MKK][0] = 72, + [0][0][1][0][RTW89_IC][0] = 78, + [0][0][1][0][RTW89_KCC][0] = 76, + [0][0][1][0][RTW89_ACMA][0] = 58, + [0][0][1][0][RTW89_CN][0] = 60, + [0][0][1][0][RTW89_UK][0] = 58, + [0][0][1][0][RTW89_FCC][1] = 78, + [0][0][1][0][RTW89_ETSI][1] = 60, + [0][0][1][0][RTW89_MKK][1] = 74, + [0][0][1][0][RTW89_IC][1] = 78, + [0][0][1][0][RTW89_KCC][1] = 76, + [0][0][1][0][RTW89_ACMA][1] = 60, + [0][0][1][0][RTW89_CN][1] = 60, + [0][0][1][0][RTW89_UK][1] = 60, + [0][0][1][0][RTW89_FCC][2] = 80, + [0][0][1][0][RTW89_ETSI][2] = 60, + [0][0][1][0][RTW89_MKK][2] = 74, + [0][0][1][0][RTW89_IC][2] = 80, + [0][0][1][0][RTW89_KCC][2] = 76, + [0][0][1][0][RTW89_ACMA][2] = 60, + [0][0][1][0][RTW89_CN][2] = 60, + [0][0][1][0][RTW89_UK][2] = 60, + [0][0][1][0][RTW89_FCC][3] = 80, + [0][0][1][0][RTW89_ETSI][3] = 60, + [0][0][1][0][RTW89_MKK][3] = 74, + [0][0][1][0][RTW89_IC][3] = 80, + [0][0][1][0][RTW89_KCC][3] = 76, + [0][0][1][0][RTW89_ACMA][3] = 60, + [0][0][1][0][RTW89_CN][3] = 60, + [0][0][1][0][RTW89_UK][3] = 60, + [0][0][1][0][RTW89_FCC][4] = 80, + [0][0][1][0][RTW89_ETSI][4] = 60, + [0][0][1][0][RTW89_MKK][4] = 74, + [0][0][1][0][RTW89_IC][4] = 80, + [0][0][1][0][RTW89_KCC][4] = 76, + [0][0][1][0][RTW89_ACMA][4] = 60, + [0][0][1][0][RTW89_CN][4] = 60, + [0][0][1][0][RTW89_UK][4] = 60, + [0][0][1][0][RTW89_FCC][5] = 80, + [0][0][1][0][RTW89_ETSI][5] = 60, + [0][0][1][0][RTW89_MKK][5] = 74, + [0][0][1][0][RTW89_IC][5] = 80, + [0][0][1][0][RTW89_KCC][5] = 76, + [0][0][1][0][RTW89_ACMA][5] = 60, + [0][0][1][0][RTW89_CN][5] = 60, + [0][0][1][0][RTW89_UK][5] = 60, + [0][0][1][0][RTW89_FCC][6] = 80, + [0][0][1][0][RTW89_ETSI][6] = 60, + [0][0][1][0][RTW89_MKK][6] = 74, + [0][0][1][0][RTW89_IC][6] = 80, + [0][0][1][0][RTW89_KCC][6] = 76, + [0][0][1][0][RTW89_ACMA][6] = 60, + [0][0][1][0][RTW89_CN][6] = 60, + [0][0][1][0][RTW89_UK][6] = 60, + [0][0][1][0][RTW89_FCC][7] = 80, + [0][0][1][0][RTW89_ETSI][7] = 60, + [0][0][1][0][RTW89_MKK][7] = 74, + [0][0][1][0][RTW89_IC][7] = 80, + [0][0][1][0][RTW89_KCC][7] = 76, + [0][0][1][0][RTW89_ACMA][7] = 60, + [0][0][1][0][RTW89_CN][7] = 60, + [0][0][1][0][RTW89_UK][7] = 60, + [0][0][1][0][RTW89_FCC][8] = 80, + [0][0][1][0][RTW89_ETSI][8] = 60, + [0][0][1][0][RTW89_MKK][8] = 74, + [0][0][1][0][RTW89_IC][8] = 80, + [0][0][1][0][RTW89_KCC][8] = 76, + [0][0][1][0][RTW89_ACMA][8] = 60, + [0][0][1][0][RTW89_CN][8] = 60, + [0][0][1][0][RTW89_UK][8] = 60, + [0][0][1][0][RTW89_FCC][9] = 76, + [0][0][1][0][RTW89_ETSI][9] = 60, + [0][0][1][0][RTW89_MKK][9] = 74, + [0][0][1][0][RTW89_IC][9] = 76, + [0][0][1][0][RTW89_KCC][9] = 74, + [0][0][1][0][RTW89_ACMA][9] = 60, + [0][0][1][0][RTW89_CN][9] = 60, + [0][0][1][0][RTW89_UK][9] = 60, + [0][0][1][0][RTW89_FCC][10] = 76, + [0][0][1][0][RTW89_ETSI][10] = 60, + [0][0][1][0][RTW89_MKK][10] = 74, + [0][0][1][0][RTW89_IC][10] = 76, + [0][0][1][0][RTW89_KCC][10] = 74, + [0][0][1][0][RTW89_ACMA][10] = 60, + [0][0][1][0][RTW89_CN][10] = 60, + [0][0][1][0][RTW89_UK][10] = 60, + [0][0][1][0][RTW89_FCC][11] = 68, + [0][0][1][0][RTW89_ETSI][11] = 60, + [0][0][1][0][RTW89_MKK][11] = 74, + [0][0][1][0][RTW89_IC][11] = 68, + [0][0][1][0][RTW89_KCC][11] = 74, + [0][0][1][0][RTW89_ACMA][11] = 60, + [0][0][1][0][RTW89_CN][11] = 60, + [0][0][1][0][RTW89_UK][11] = 60, + [0][0][1][0][RTW89_FCC][12] = 64, + [0][0][1][0][RTW89_ETSI][12] = 58, + [0][0][1][0][RTW89_MKK][12] = 70, + [0][0][1][0][RTW89_IC][12] = 64, + [0][0][1][0][RTW89_KCC][12] = 74, + [0][0][1][0][RTW89_ACMA][12] = 58, + [0][0][1][0][RTW89_CN][12] = 60, + [0][0][1][0][RTW89_UK][12] = 58, + [0][0][1][0][RTW89_FCC][13] = 127, + [0][0][1][0][RTW89_ETSI][13] = 127, + [0][0][1][0][RTW89_MKK][13] = 127, + [0][0][1][0][RTW89_IC][13] = 127, + [0][0][1][0][RTW89_KCC][13] = 127, + [0][0][1][0][RTW89_ACMA][13] = 127, + [0][0][1][0][RTW89_CN][13] = 127, + [0][0][1][0][RTW89_UK][13] = 127, + [0][1][1][0][RTW89_FCC][0] = 127, + [0][1][1][0][RTW89_ETSI][0] = 127, + [0][1][1][0][RTW89_MKK][0] = 127, + [0][1][1][0][RTW89_IC][0] = 127, + [0][1][1][0][RTW89_KCC][0] = 127, + [0][1][1][0][RTW89_ACMA][0] = 127, + [0][1][1][0][RTW89_CN][0] = 127, + [0][1][1][0][RTW89_UK][0] = 127, + [0][1][1][0][RTW89_FCC][1] = 127, + [0][1][1][0][RTW89_ETSI][1] = 127, + [0][1][1][0][RTW89_MKK][1] = 127, + [0][1][1][0][RTW89_IC][1] = 127, + [0][1][1][0][RTW89_KCC][1] = 127, + [0][1][1][0][RTW89_ACMA][1] = 127, + [0][1][1][0][RTW89_CN][1] = 127, + [0][1][1][0][RTW89_UK][1] = 127, + [0][1][1][0][RTW89_FCC][2] = 127, + [0][1][1][0][RTW89_ETSI][2] = 127, + [0][1][1][0][RTW89_MKK][2] = 127, + [0][1][1][0][RTW89_IC][2] = 127, + [0][1][1][0][RTW89_KCC][2] = 127, + [0][1][1][0][RTW89_ACMA][2] = 127, + [0][1][1][0][RTW89_CN][2] = 127, + [0][1][1][0][RTW89_UK][2] = 127, + [0][1][1][0][RTW89_FCC][3] = 127, + [0][1][1][0][RTW89_ETSI][3] = 127, + [0][1][1][0][RTW89_MKK][3] = 127, + [0][1][1][0][RTW89_IC][3] = 127, + [0][1][1][0][RTW89_KCC][3] = 127, + [0][1][1][0][RTW89_ACMA][3] = 127, + [0][1][1][0][RTW89_CN][3] = 127, + [0][1][1][0][RTW89_UK][3] = 127, + [0][1][1][0][RTW89_FCC][4] = 127, + [0][1][1][0][RTW89_ETSI][4] = 127, + [0][1][1][0][RTW89_MKK][4] = 127, + [0][1][1][0][RTW89_IC][4] = 127, + [0][1][1][0][RTW89_KCC][4] = 127, + [0][1][1][0][RTW89_ACMA][4] = 127, + [0][1][1][0][RTW89_CN][4] = 127, + [0][1][1][0][RTW89_UK][4] = 127, + [0][1][1][0][RTW89_FCC][5] = 127, + [0][1][1][0][RTW89_ETSI][5] = 127, + [0][1][1][0][RTW89_MKK][5] = 127, + [0][1][1][0][RTW89_IC][5] = 127, + [0][1][1][0][RTW89_KCC][5] = 127, + [0][1][1][0][RTW89_ACMA][5] = 127, + [0][1][1][0][RTW89_CN][5] = 127, + [0][1][1][0][RTW89_UK][5] = 127, + [0][1][1][0][RTW89_FCC][6] = 127, + [0][1][1][0][RTW89_ETSI][6] = 127, + [0][1][1][0][RTW89_MKK][6] = 127, + [0][1][1][0][RTW89_IC][6] = 127, + [0][1][1][0][RTW89_KCC][6] = 127, + [0][1][1][0][RTW89_ACMA][6] = 127, + [0][1][1][0][RTW89_CN][6] = 127, + [0][1][1][0][RTW89_UK][6] = 127, + [0][1][1][0][RTW89_FCC][7] = 127, + [0][1][1][0][RTW89_ETSI][7] = 127, + [0][1][1][0][RTW89_MKK][7] = 127, + [0][1][1][0][RTW89_IC][7] = 127, + [0][1][1][0][RTW89_KCC][7] = 127, + [0][1][1][0][RTW89_ACMA][7] = 127, + [0][1][1][0][RTW89_CN][7] = 127, + [0][1][1][0][RTW89_UK][7] = 127, + [0][1][1][0][RTW89_FCC][8] = 127, + [0][1][1][0][RTW89_ETSI][8] = 127, + [0][1][1][0][RTW89_MKK][8] = 127, + [0][1][1][0][RTW89_IC][8] = 127, + [0][1][1][0][RTW89_KCC][8] = 127, + [0][1][1][0][RTW89_ACMA][8] = 127, + [0][1][1][0][RTW89_CN][8] = 127, + [0][1][1][0][RTW89_UK][8] = 127, + [0][1][1][0][RTW89_FCC][9] = 127, + [0][1][1][0][RTW89_ETSI][9] = 127, + [0][1][1][0][RTW89_MKK][9] = 127, + [0][1][1][0][RTW89_IC][9] = 127, + [0][1][1][0][RTW89_KCC][9] = 127, + [0][1][1][0][RTW89_ACMA][9] = 127, + [0][1][1][0][RTW89_CN][9] = 127, + [0][1][1][0][RTW89_UK][9] = 127, + [0][1][1][0][RTW89_FCC][10] = 127, + [0][1][1][0][RTW89_ETSI][10] = 127, + [0][1][1][0][RTW89_MKK][10] = 127, + [0][1][1][0][RTW89_IC][10] = 127, + [0][1][1][0][RTW89_KCC][10] = 127, + [0][1][1][0][RTW89_ACMA][10] = 127, + [0][1][1][0][RTW89_CN][10] = 127, + [0][1][1][0][RTW89_UK][10] = 127, + [0][1][1][0][RTW89_FCC][11] = 127, + [0][1][1][0][RTW89_ETSI][11] = 127, + [0][1][1][0][RTW89_MKK][11] = 127, + [0][1][1][0][RTW89_IC][11] = 127, + [0][1][1][0][RTW89_KCC][11] = 127, + [0][1][1][0][RTW89_ACMA][11] = 127, + [0][1][1][0][RTW89_CN][11] = 127, + [0][1][1][0][RTW89_UK][11] = 127, + [0][1][1][0][RTW89_FCC][12] = 127, + [0][1][1][0][RTW89_ETSI][12] = 127, + [0][1][1][0][RTW89_MKK][12] = 127, + [0][1][1][0][RTW89_IC][12] = 127, + [0][1][1][0][RTW89_KCC][12] = 127, + [0][1][1][0][RTW89_ACMA][12] = 127, + [0][1][1][0][RTW89_CN][12] = 127, + [0][1][1][0][RTW89_UK][12] = 127, + [0][1][1][0][RTW89_FCC][13] = 127, + [0][1][1][0][RTW89_ETSI][13] = 127, + [0][1][1][0][RTW89_MKK][13] = 127, + [0][1][1][0][RTW89_IC][13] = 127, + [0][1][1][0][RTW89_KCC][13] = 127, + [0][1][1][0][RTW89_ACMA][13] = 127, + [0][1][1][0][RTW89_CN][13] = 127, + [0][1][1][0][RTW89_UK][13] = 127, + [0][0][2][0][RTW89_FCC][0] = 78, + [0][0][2][0][RTW89_ETSI][0] = 60, + [0][0][2][0][RTW89_MKK][0] = 72, + [0][0][2][0][RTW89_IC][0] = 78, + [0][0][2][0][RTW89_KCC][0] = 76, + [0][0][2][0][RTW89_ACMA][0] = 60, + [0][0][2][0][RTW89_CN][0] = 60, + [0][0][2][0][RTW89_UK][0] = 60, + [0][0][2][0][RTW89_FCC][1] = 78, + [0][0][2][0][RTW89_ETSI][1] = 60, + [0][0][2][0][RTW89_MKK][1] = 76, + [0][0][2][0][RTW89_IC][1] = 78, + [0][0][2][0][RTW89_KCC][1] = 76, + [0][0][2][0][RTW89_ACMA][1] = 60, + [0][0][2][0][RTW89_CN][1] = 60, + [0][0][2][0][RTW89_UK][1] = 60, + [0][0][2][0][RTW89_FCC][2] = 80, + [0][0][2][0][RTW89_ETSI][2] = 60, + [0][0][2][0][RTW89_MKK][2] = 76, + [0][0][2][0][RTW89_IC][2] = 80, + [0][0][2][0][RTW89_KCC][2] = 76, + [0][0][2][0][RTW89_ACMA][2] = 60, + [0][0][2][0][RTW89_CN][2] = 60, + [0][0][2][0][RTW89_UK][2] = 60, + [0][0][2][0][RTW89_FCC][3] = 80, + [0][0][2][0][RTW89_ETSI][3] = 60, + [0][0][2][0][RTW89_MKK][3] = 76, + [0][0][2][0][RTW89_IC][3] = 80, + [0][0][2][0][RTW89_KCC][3] = 76, + [0][0][2][0][RTW89_ACMA][3] = 60, + [0][0][2][0][RTW89_CN][3] = 60, + [0][0][2][0][RTW89_UK][3] = 60, + [0][0][2][0][RTW89_FCC][4] = 80, + [0][0][2][0][RTW89_ETSI][4] = 60, + [0][0][2][0][RTW89_MKK][4] = 76, + [0][0][2][0][RTW89_IC][4] = 80, + [0][0][2][0][RTW89_KCC][4] = 76, + [0][0][2][0][RTW89_ACMA][4] = 60, + [0][0][2][0][RTW89_CN][4] = 60, + [0][0][2][0][RTW89_UK][4] = 60, + [0][0][2][0][RTW89_FCC][5] = 80, + [0][0][2][0][RTW89_ETSI][5] = 60, + [0][0][2][0][RTW89_MKK][5] = 76, + [0][0][2][0][RTW89_IC][5] = 80, + [0][0][2][0][RTW89_KCC][5] = 76, + [0][0][2][0][RTW89_ACMA][5] = 60, + [0][0][2][0][RTW89_CN][5] = 60, + [0][0][2][0][RTW89_UK][5] = 60, + [0][0][2][0][RTW89_FCC][6] = 80, + [0][0][2][0][RTW89_ETSI][6] = 60, + [0][0][2][0][RTW89_MKK][6] = 76, + [0][0][2][0][RTW89_IC][6] = 80, + [0][0][2][0][RTW89_KCC][6] = 76, + [0][0][2][0][RTW89_ACMA][6] = 60, + [0][0][2][0][RTW89_CN][6] = 60, + [0][0][2][0][RTW89_UK][6] = 60, + [0][0][2][0][RTW89_FCC][7] = 80, + [0][0][2][0][RTW89_ETSI][7] = 60, + [0][0][2][0][RTW89_MKK][7] = 76, + [0][0][2][0][RTW89_IC][7] = 80, + [0][0][2][0][RTW89_KCC][7] = 76, + [0][0][2][0][RTW89_ACMA][7] = 60, + [0][0][2][0][RTW89_CN][7] = 60, + [0][0][2][0][RTW89_UK][7] = 60, + [0][0][2][0][RTW89_FCC][8] = 78, + [0][0][2][0][RTW89_ETSI][8] = 60, + [0][0][2][0][RTW89_MKK][8] = 76, + [0][0][2][0][RTW89_IC][8] = 78, + [0][0][2][0][RTW89_KCC][8] = 76, + [0][0][2][0][RTW89_ACMA][8] = 60, + [0][0][2][0][RTW89_CN][8] = 60, + [0][0][2][0][RTW89_UK][8] = 60, + [0][0][2][0][RTW89_FCC][9] = 74, + [0][0][2][0][RTW89_ETSI][9] = 60, + [0][0][2][0][RTW89_MKK][9] = 76, + [0][0][2][0][RTW89_IC][9] = 74, + [0][0][2][0][RTW89_KCC][9] = 76, + [0][0][2][0][RTW89_ACMA][9] = 60, + [0][0][2][0][RTW89_CN][9] = 60, + [0][0][2][0][RTW89_UK][9] = 60, + [0][0][2][0][RTW89_FCC][10] = 74, + [0][0][2][0][RTW89_ETSI][10] = 60, + [0][0][2][0][RTW89_MKK][10] = 76, + [0][0][2][0][RTW89_IC][10] = 74, + [0][0][2][0][RTW89_KCC][10] = 76, + [0][0][2][0][RTW89_ACMA][10] = 60, + [0][0][2][0][RTW89_CN][10] = 60, + [0][0][2][0][RTW89_UK][10] = 60, + [0][0][2][0][RTW89_FCC][11] = 68, + [0][0][2][0][RTW89_ETSI][11] = 60, + [0][0][2][0][RTW89_MKK][11] = 76, + [0][0][2][0][RTW89_IC][11] = 68, + [0][0][2][0][RTW89_KCC][11] = 76, + [0][0][2][0][RTW89_ACMA][11] = 60, + [0][0][2][0][RTW89_CN][11] = 60, + [0][0][2][0][RTW89_UK][11] = 60, + [0][0][2][0][RTW89_FCC][12] = 68, + [0][0][2][0][RTW89_ETSI][12] = 60, + [0][0][2][0][RTW89_MKK][12] = 70, + [0][0][2][0][RTW89_IC][12] = 68, + [0][0][2][0][RTW89_KCC][12] = 76, + [0][0][2][0][RTW89_ACMA][12] = 60, + [0][0][2][0][RTW89_CN][12] = 60, + [0][0][2][0][RTW89_UK][12] = 60, + [0][0][2][0][RTW89_FCC][13] = 127, + [0][0][2][0][RTW89_ETSI][13] = 127, + [0][0][2][0][RTW89_MKK][13] = 127, + [0][0][2][0][RTW89_IC][13] = 127, + [0][0][2][0][RTW89_KCC][13] = 127, + [0][0][2][0][RTW89_ACMA][13] = 127, + [0][0][2][0][RTW89_CN][13] = 127, + [0][0][2][0][RTW89_UK][13] = 127, + [0][1][2][0][RTW89_FCC][0] = 127, + [0][1][2][0][RTW89_ETSI][0] = 127, + [0][1][2][0][RTW89_MKK][0] = 127, + [0][1][2][0][RTW89_IC][0] = 127, + [0][1][2][0][RTW89_KCC][0] = 127, + [0][1][2][0][RTW89_ACMA][0] = 127, + [0][1][2][0][RTW89_CN][0] = 127, + [0][1][2][0][RTW89_UK][0] = 127, + [0][1][2][0][RTW89_FCC][1] = 127, + [0][1][2][0][RTW89_ETSI][1] = 127, + [0][1][2][0][RTW89_MKK][1] = 127, + [0][1][2][0][RTW89_IC][1] = 127, + [0][1][2][0][RTW89_KCC][1] = 127, + [0][1][2][0][RTW89_ACMA][1] = 127, + [0][1][2][0][RTW89_CN][1] = 127, + [0][1][2][0][RTW89_UK][1] = 127, + [0][1][2][0][RTW89_FCC][2] = 127, + [0][1][2][0][RTW89_ETSI][2] = 127, + [0][1][2][0][RTW89_MKK][2] = 127, + [0][1][2][0][RTW89_IC][2] = 127, + [0][1][2][0][RTW89_KCC][2] = 127, + [0][1][2][0][RTW89_ACMA][2] = 127, + [0][1][2][0][RTW89_CN][2] = 127, + [0][1][2][0][RTW89_UK][2] = 127, + [0][1][2][0][RTW89_FCC][3] = 127, + [0][1][2][0][RTW89_ETSI][3] = 127, + [0][1][2][0][RTW89_MKK][3] = 127, + [0][1][2][0][RTW89_IC][3] = 127, + [0][1][2][0][RTW89_KCC][3] = 127, + [0][1][2][0][RTW89_ACMA][3] = 127, + [0][1][2][0][RTW89_CN][3] = 127, + [0][1][2][0][RTW89_UK][3] = 127, + [0][1][2][0][RTW89_FCC][4] = 127, + [0][1][2][0][RTW89_ETSI][4] = 127, + [0][1][2][0][RTW89_MKK][4] = 127, + [0][1][2][0][RTW89_IC][4] = 127, + [0][1][2][0][RTW89_KCC][4] = 127, + [0][1][2][0][RTW89_ACMA][4] = 127, + [0][1][2][0][RTW89_CN][4] = 127, + [0][1][2][0][RTW89_UK][4] = 127, + [0][1][2][0][RTW89_FCC][5] = 127, + [0][1][2][0][RTW89_ETSI][5] = 127, + [0][1][2][0][RTW89_MKK][5] = 127, + [0][1][2][0][RTW89_IC][5] = 127, + [0][1][2][0][RTW89_KCC][5] = 127, + [0][1][2][0][RTW89_ACMA][5] = 127, + [0][1][2][0][RTW89_CN][5] = 127, + [0][1][2][0][RTW89_UK][5] = 127, + [0][1][2][0][RTW89_FCC][6] = 127, + [0][1][2][0][RTW89_ETSI][6] = 127, + [0][1][2][0][RTW89_MKK][6] = 127, + [0][1][2][0][RTW89_IC][6] = 127, + [0][1][2][0][RTW89_KCC][6] = 127, + [0][1][2][0][RTW89_ACMA][6] = 127, + [0][1][2][0][RTW89_CN][6] = 127, + [0][1][2][0][RTW89_UK][6] = 127, + [0][1][2][0][RTW89_FCC][7] = 127, + [0][1][2][0][RTW89_ETSI][7] = 127, + [0][1][2][0][RTW89_MKK][7] = 127, + [0][1][2][0][RTW89_IC][7] = 127, + [0][1][2][0][RTW89_KCC][7] = 127, + [0][1][2][0][RTW89_ACMA][7] = 127, + [0][1][2][0][RTW89_CN][7] = 127, + [0][1][2][0][RTW89_UK][7] = 127, + [0][1][2][0][RTW89_FCC][8] = 127, + [0][1][2][0][RTW89_ETSI][8] = 127, + [0][1][2][0][RTW89_MKK][8] = 127, + [0][1][2][0][RTW89_IC][8] = 127, + [0][1][2][0][RTW89_KCC][8] = 127, + [0][1][2][0][RTW89_ACMA][8] = 127, + [0][1][2][0][RTW89_CN][8] = 127, + [0][1][2][0][RTW89_UK][8] = 127, + [0][1][2][0][RTW89_FCC][9] = 127, + [0][1][2][0][RTW89_ETSI][9] = 127, + [0][1][2][0][RTW89_MKK][9] = 127, + [0][1][2][0][RTW89_IC][9] = 127, + [0][1][2][0][RTW89_KCC][9] = 127, + [0][1][2][0][RTW89_ACMA][9] = 127, + [0][1][2][0][RTW89_CN][9] = 127, + [0][1][2][0][RTW89_UK][9] = 127, + [0][1][2][0][RTW89_FCC][10] = 127, + [0][1][2][0][RTW89_ETSI][10] = 127, + [0][1][2][0][RTW89_MKK][10] = 127, + [0][1][2][0][RTW89_IC][10] = 127, + [0][1][2][0][RTW89_KCC][10] = 127, + [0][1][2][0][RTW89_ACMA][10] = 127, + [0][1][2][0][RTW89_CN][10] = 127, + [0][1][2][0][RTW89_UK][10] = 127, + [0][1][2][0][RTW89_FCC][11] = 127, + [0][1][2][0][RTW89_ETSI][11] = 127, + [0][1][2][0][RTW89_MKK][11] = 127, + [0][1][2][0][RTW89_IC][11] = 127, + [0][1][2][0][RTW89_KCC][11] = 127, + [0][1][2][0][RTW89_ACMA][11] = 127, + [0][1][2][0][RTW89_CN][11] = 127, + [0][1][2][0][RTW89_UK][11] = 127, + [0][1][2][0][RTW89_FCC][12] = 127, + [0][1][2][0][RTW89_ETSI][12] = 127, + [0][1][2][0][RTW89_MKK][12] = 127, + [0][1][2][0][RTW89_IC][12] = 127, + [0][1][2][0][RTW89_KCC][12] = 127, + [0][1][2][0][RTW89_ACMA][12] = 127, + [0][1][2][0][RTW89_CN][12] = 127, + [0][1][2][0][RTW89_UK][12] = 127, + [0][1][2][0][RTW89_FCC][13] = 127, + [0][1][2][0][RTW89_ETSI][13] = 127, + [0][1][2][0][RTW89_MKK][13] = 127, + [0][1][2][0][RTW89_IC][13] = 127, + [0][1][2][0][RTW89_KCC][13] = 127, + [0][1][2][0][RTW89_ACMA][13] = 127, + [0][1][2][0][RTW89_CN][13] = 127, + [0][1][2][0][RTW89_UK][13] = 127, + [0][1][2][1][RTW89_FCC][0] = 127, + [0][1][2][1][RTW89_ETSI][0] = 127, + [0][1][2][1][RTW89_MKK][0] = 127, + [0][1][2][1][RTW89_IC][0] = 127, + [0][1][2][1][RTW89_KCC][0] = 127, + [0][1][2][1][RTW89_ACMA][0] = 127, + [0][1][2][1][RTW89_CN][0] = 127, + [0][1][2][1][RTW89_UK][0] = 127, + [0][1][2][1][RTW89_FCC][1] = 127, + [0][1][2][1][RTW89_ETSI][1] = 127, + [0][1][2][1][RTW89_MKK][1] = 127, + [0][1][2][1][RTW89_IC][1] = 127, + [0][1][2][1][RTW89_KCC][1] = 127, + [0][1][2][1][RTW89_ACMA][1] = 127, + [0][1][2][1][RTW89_CN][1] = 127, + [0][1][2][1][RTW89_UK][1] = 127, + [0][1][2][1][RTW89_FCC][2] = 127, + [0][1][2][1][RTW89_ETSI][2] = 127, + [0][1][2][1][RTW89_MKK][2] = 127, + [0][1][2][1][RTW89_IC][2] = 127, + [0][1][2][1][RTW89_KCC][2] = 127, + [0][1][2][1][RTW89_ACMA][2] = 127, + [0][1][2][1][RTW89_CN][2] = 127, + [0][1][2][1][RTW89_UK][2] = 127, + [0][1][2][1][RTW89_FCC][3] = 127, + [0][1][2][1][RTW89_ETSI][3] = 127, + [0][1][2][1][RTW89_MKK][3] = 127, + [0][1][2][1][RTW89_IC][3] = 127, + [0][1][2][1][RTW89_KCC][3] = 127, + [0][1][2][1][RTW89_ACMA][3] = 127, + [0][1][2][1][RTW89_CN][3] = 127, + [0][1][2][1][RTW89_UK][3] = 127, + [0][1][2][1][RTW89_FCC][4] = 127, + [0][1][2][1][RTW89_ETSI][4] = 127, + [0][1][2][1][RTW89_MKK][4] = 127, + [0][1][2][1][RTW89_IC][4] = 127, + [0][1][2][1][RTW89_KCC][4] = 127, + [0][1][2][1][RTW89_ACMA][4] = 127, + [0][1][2][1][RTW89_CN][4] = 127, + [0][1][2][1][RTW89_UK][4] = 127, + [0][1][2][1][RTW89_FCC][5] = 127, + [0][1][2][1][RTW89_ETSI][5] = 127, + [0][1][2][1][RTW89_MKK][5] = 127, + [0][1][2][1][RTW89_IC][5] = 127, + [0][1][2][1][RTW89_KCC][5] = 127, + [0][1][2][1][RTW89_ACMA][5] = 127, + [0][1][2][1][RTW89_CN][5] = 127, + [0][1][2][1][RTW89_UK][5] = 127, + [0][1][2][1][RTW89_FCC][6] = 127, + [0][1][2][1][RTW89_ETSI][6] = 127, + [0][1][2][1][RTW89_MKK][6] = 127, + [0][1][2][1][RTW89_IC][6] = 127, + [0][1][2][1][RTW89_KCC][6] = 127, + [0][1][2][1][RTW89_ACMA][6] = 127, + [0][1][2][1][RTW89_CN][6] = 127, + [0][1][2][1][RTW89_UK][6] = 127, + [0][1][2][1][RTW89_FCC][7] = 127, + [0][1][2][1][RTW89_ETSI][7] = 127, + [0][1][2][1][RTW89_MKK][7] = 127, + [0][1][2][1][RTW89_IC][7] = 127, + [0][1][2][1][RTW89_KCC][7] = 127, + [0][1][2][1][RTW89_ACMA][7] = 127, + [0][1][2][1][RTW89_CN][7] = 127, + [0][1][2][1][RTW89_UK][7] = 127, + [0][1][2][1][RTW89_FCC][8] = 127, + [0][1][2][1][RTW89_ETSI][8] = 127, + [0][1][2][1][RTW89_MKK][8] = 127, + [0][1][2][1][RTW89_IC][8] = 127, + [0][1][2][1][RTW89_KCC][8] = 127, + [0][1][2][1][RTW89_ACMA][8] = 127, + [0][1][2][1][RTW89_CN][8] = 127, + [0][1][2][1][RTW89_UK][8] = 127, + [0][1][2][1][RTW89_FCC][9] = 127, + [0][1][2][1][RTW89_ETSI][9] = 127, + [0][1][2][1][RTW89_MKK][9] = 127, + [0][1][2][1][RTW89_IC][9] = 127, + [0][1][2][1][RTW89_KCC][9] = 127, + [0][1][2][1][RTW89_ACMA][9] = 127, + [0][1][2][1][RTW89_CN][9] = 127, + [0][1][2][1][RTW89_UK][9] = 127, + [0][1][2][1][RTW89_FCC][10] = 127, + [0][1][2][1][RTW89_ETSI][10] = 127, + [0][1][2][1][RTW89_MKK][10] = 127, + [0][1][2][1][RTW89_IC][10] = 127, + [0][1][2][1][RTW89_KCC][10] = 127, + [0][1][2][1][RTW89_ACMA][10] = 127, + [0][1][2][1][RTW89_CN][10] = 127, + [0][1][2][1][RTW89_UK][10] = 127, + [0][1][2][1][RTW89_FCC][11] = 127, + [0][1][2][1][RTW89_ETSI][11] = 127, + [0][1][2][1][RTW89_MKK][11] = 127, + [0][1][2][1][RTW89_IC][11] = 127, + [0][1][2][1][RTW89_KCC][11] = 127, + [0][1][2][1][RTW89_ACMA][11] = 127, + [0][1][2][1][RTW89_CN][11] = 127, + [0][1][2][1][RTW89_UK][11] = 127, + [0][1][2][1][RTW89_FCC][12] = 127, + [0][1][2][1][RTW89_ETSI][12] = 127, + [0][1][2][1][RTW89_MKK][12] = 127, + [0][1][2][1][RTW89_IC][12] = 127, + [0][1][2][1][RTW89_KCC][12] = 127, + [0][1][2][1][RTW89_ACMA][12] = 127, + [0][1][2][1][RTW89_CN][12] = 127, + [0][1][2][1][RTW89_UK][12] = 127, + [0][1][2][1][RTW89_FCC][13] = 127, + [0][1][2][1][RTW89_ETSI][13] = 127, + [0][1][2][1][RTW89_MKK][13] = 127, + [0][1][2][1][RTW89_IC][13] = 127, + [0][1][2][1][RTW89_KCC][13] = 127, + [0][1][2][1][RTW89_ACMA][13] = 127, + [0][1][2][1][RTW89_CN][13] = 127, + [0][1][2][1][RTW89_UK][13] = 127, + [1][0][2][0][RTW89_FCC][0] = 127, + [1][0][2][0][RTW89_ETSI][0] = 127, + [1][0][2][0][RTW89_MKK][0] = 127, + [1][0][2][0][RTW89_IC][0] = 127, + [1][0][2][0][RTW89_KCC][0] = 127, + [1][0][2][0][RTW89_ACMA][0] = 127, + [1][0][2][0][RTW89_CN][0] = 127, + [1][0][2][0][RTW89_UK][0] = 127, + [1][0][2][0][RTW89_FCC][1] = 127, + [1][0][2][0][RTW89_ETSI][1] = 127, + [1][0][2][0][RTW89_MKK][1] = 127, + [1][0][2][0][RTW89_IC][1] = 127, + [1][0][2][0][RTW89_KCC][1] = 127, + [1][0][2][0][RTW89_ACMA][1] = 127, + [1][0][2][0][RTW89_CN][1] = 127, + [1][0][2][0][RTW89_UK][1] = 127, + [1][0][2][0][RTW89_FCC][2] = 70, + [1][0][2][0][RTW89_ETSI][2] = 58, + [1][0][2][0][RTW89_MKK][2] = 76, + [1][0][2][0][RTW89_IC][2] = 70, + [1][0][2][0][RTW89_KCC][2] = 76, + [1][0][2][0][RTW89_ACMA][2] = 58, + [1][0][2][0][RTW89_CN][2] = 60, + [1][0][2][0][RTW89_UK][2] = 58, + [1][0][2][0][RTW89_FCC][3] = 70, + [1][0][2][0][RTW89_ETSI][3] = 58, + [1][0][2][0][RTW89_MKK][3] = 76, + [1][0][2][0][RTW89_IC][3] = 70, + [1][0][2][0][RTW89_KCC][3] = 76, + [1][0][2][0][RTW89_ACMA][3] = 58, + [1][0][2][0][RTW89_CN][3] = 60, + [1][0][2][0][RTW89_UK][3] = 58, + [1][0][2][0][RTW89_FCC][4] = 74, + [1][0][2][0][RTW89_ETSI][4] = 58, + [1][0][2][0][RTW89_MKK][4] = 76, + [1][0][2][0][RTW89_IC][4] = 74, + [1][0][2][0][RTW89_KCC][4] = 76, + [1][0][2][0][RTW89_ACMA][4] = 58, + [1][0][2][0][RTW89_CN][4] = 60, + [1][0][2][0][RTW89_UK][4] = 58, + [1][0][2][0][RTW89_FCC][5] = 76, + [1][0][2][0][RTW89_ETSI][5] = 58, + [1][0][2][0][RTW89_MKK][5] = 76, + [1][0][2][0][RTW89_IC][5] = 76, + [1][0][2][0][RTW89_KCC][5] = 76, + [1][0][2][0][RTW89_ACMA][5] = 58, + [1][0][2][0][RTW89_CN][5] = 60, + [1][0][2][0][RTW89_UK][5] = 58, + [1][0][2][0][RTW89_FCC][6] = 76, + [1][0][2][0][RTW89_ETSI][6] = 58, + [1][0][2][0][RTW89_MKK][6] = 76, + [1][0][2][0][RTW89_IC][6] = 76, + [1][0][2][0][RTW89_KCC][6] = 76, + [1][0][2][0][RTW89_ACMA][6] = 58, + [1][0][2][0][RTW89_CN][6] = 60, + [1][0][2][0][RTW89_UK][6] = 58, + [1][0][2][0][RTW89_FCC][7] = 76, + [1][0][2][0][RTW89_ETSI][7] = 58, + [1][0][2][0][RTW89_MKK][7] = 76, + [1][0][2][0][RTW89_IC][7] = 76, + [1][0][2][0][RTW89_KCC][7] = 76, + [1][0][2][0][RTW89_ACMA][7] = 58, + [1][0][2][0][RTW89_CN][7] = 60, + [1][0][2][0][RTW89_UK][7] = 58, + [1][0][2][0][RTW89_FCC][8] = 78, + [1][0][2][0][RTW89_ETSI][8] = 58, + [1][0][2][0][RTW89_MKK][8] = 76, + [1][0][2][0][RTW89_IC][8] = 78, + [1][0][2][0][RTW89_KCC][8] = 76, + [1][0][2][0][RTW89_ACMA][8] = 58, + [1][0][2][0][RTW89_CN][8] = 60, + [1][0][2][0][RTW89_UK][8] = 58, + [1][0][2][0][RTW89_FCC][9] = 74, + [1][0][2][0][RTW89_ETSI][9] = 58, + [1][0][2][0][RTW89_MKK][9] = 76, + [1][0][2][0][RTW89_IC][9] = 74, + [1][0][2][0][RTW89_KCC][9] = 76, + [1][0][2][0][RTW89_ACMA][9] = 58, + [1][0][2][0][RTW89_CN][9] = 60, + [1][0][2][0][RTW89_UK][9] = 58, + [1][0][2][0][RTW89_FCC][10] = 68, + [1][0][2][0][RTW89_ETSI][10] = 58, + [1][0][2][0][RTW89_MKK][10] = 76, + [1][0][2][0][RTW89_IC][10] = 68, + [1][0][2][0][RTW89_KCC][10] = 76, + [1][0][2][0][RTW89_ACMA][10] = 58, + [1][0][2][0][RTW89_CN][10] = 60, + [1][0][2][0][RTW89_UK][10] = 58, + [1][0][2][0][RTW89_FCC][11] = 127, + [1][0][2][0][RTW89_ETSI][11] = 127, + [1][0][2][0][RTW89_MKK][11] = 127, + [1][0][2][0][RTW89_IC][11] = 127, + [1][0][2][0][RTW89_KCC][11] = 127, + [1][0][2][0][RTW89_ACMA][11] = 127, + [1][0][2][0][RTW89_CN][11] = 127, + [1][0][2][0][RTW89_UK][11] = 127, + [1][0][2][0][RTW89_FCC][12] = 127, + [1][0][2][0][RTW89_ETSI][12] = 127, + [1][0][2][0][RTW89_MKK][12] = 127, + [1][0][2][0][RTW89_IC][12] = 127, + [1][0][2][0][RTW89_KCC][12] = 127, + [1][0][2][0][RTW89_ACMA][12] = 127, + [1][0][2][0][RTW89_CN][12] = 127, + [1][0][2][0][RTW89_UK][12] = 127, + [1][0][2][0][RTW89_FCC][13] = 127, + [1][0][2][0][RTW89_ETSI][13] = 127, + [1][0][2][0][RTW89_MKK][13] = 127, + [1][0][2][0][RTW89_IC][13] = 127, + [1][0][2][0][RTW89_KCC][13] = 127, + [1][0][2][0][RTW89_ACMA][13] = 127, + [1][0][2][0][RTW89_CN][13] = 127, + [1][0][2][0][RTW89_UK][13] = 127, + [1][1][2][0][RTW89_FCC][0] = 127, + [1][1][2][0][RTW89_ETSI][0] = 127, + [1][1][2][0][RTW89_MKK][0] = 127, + [1][1][2][0][RTW89_IC][0] = 127, + [1][1][2][0][RTW89_KCC][0] = 127, + [1][1][2][0][RTW89_ACMA][0] = 127, + [1][1][2][0][RTW89_CN][0] = 127, + [1][1][2][0][RTW89_UK][0] = 127, + [1][1][2][0][RTW89_FCC][1] = 127, + [1][1][2][0][RTW89_ETSI][1] = 127, + [1][1][2][0][RTW89_MKK][1] = 127, + [1][1][2][0][RTW89_IC][1] = 127, + [1][1][2][0][RTW89_KCC][1] = 127, + [1][1][2][0][RTW89_ACMA][1] = 127, + [1][1][2][0][RTW89_CN][1] = 127, + [1][1][2][0][RTW89_UK][1] = 127, + [1][1][2][0][RTW89_FCC][2] = 127, + [1][1][2][0][RTW89_ETSI][2] = 127, + [1][1][2][0][RTW89_MKK][2] = 127, + [1][1][2][0][RTW89_IC][2] = 127, + [1][1][2][0][RTW89_KCC][2] = 127, + [1][1][2][0][RTW89_ACMA][2] = 127, + [1][1][2][0][RTW89_CN][2] = 127, + [1][1][2][0][RTW89_UK][2] = 127, + [1][1][2][0][RTW89_FCC][3] = 127, + [1][1][2][0][RTW89_ETSI][3] = 127, + [1][1][2][0][RTW89_MKK][3] = 127, + [1][1][2][0][RTW89_IC][3] = 127, + [1][1][2][0][RTW89_KCC][3] = 127, + [1][1][2][0][RTW89_ACMA][3] = 127, + [1][1][2][0][RTW89_CN][3] = 127, + [1][1][2][0][RTW89_UK][3] = 127, + [1][1][2][0][RTW89_FCC][4] = 127, + [1][1][2][0][RTW89_ETSI][4] = 127, + [1][1][2][0][RTW89_MKK][4] = 127, + [1][1][2][0][RTW89_IC][4] = 127, + [1][1][2][0][RTW89_KCC][4] = 127, + [1][1][2][0][RTW89_ACMA][4] = 127, + [1][1][2][0][RTW89_CN][4] = 127, + [1][1][2][0][RTW89_UK][4] = 127, + [1][1][2][0][RTW89_FCC][5] = 127, + [1][1][2][0][RTW89_ETSI][5] = 127, + [1][1][2][0][RTW89_MKK][5] = 127, + [1][1][2][0][RTW89_IC][5] = 127, + [1][1][2][0][RTW89_KCC][5] = 127, + [1][1][2][0][RTW89_ACMA][5] = 127, + [1][1][2][0][RTW89_CN][5] = 127, + [1][1][2][0][RTW89_UK][5] = 127, + [1][1][2][0][RTW89_FCC][6] = 127, + [1][1][2][0][RTW89_ETSI][6] = 127, + [1][1][2][0][RTW89_MKK][6] = 127, + [1][1][2][0][RTW89_IC][6] = 127, + [1][1][2][0][RTW89_KCC][6] = 127, + [1][1][2][0][RTW89_ACMA][6] = 127, + [1][1][2][0][RTW89_CN][6] = 127, + [1][1][2][0][RTW89_UK][6] = 127, + [1][1][2][0][RTW89_FCC][7] = 127, + [1][1][2][0][RTW89_ETSI][7] = 127, + [1][1][2][0][RTW89_MKK][7] = 127, + [1][1][2][0][RTW89_IC][7] = 127, + [1][1][2][0][RTW89_KCC][7] = 127, + [1][1][2][0][RTW89_ACMA][7] = 127, + [1][1][2][0][RTW89_CN][7] = 127, + [1][1][2][0][RTW89_UK][7] = 127, + [1][1][2][0][RTW89_FCC][8] = 127, + [1][1][2][0][RTW89_ETSI][8] = 127, + [1][1][2][0][RTW89_MKK][8] = 127, + [1][1][2][0][RTW89_IC][8] = 127, + [1][1][2][0][RTW89_KCC][8] = 127, + [1][1][2][0][RTW89_ACMA][8] = 127, + [1][1][2][0][RTW89_CN][8] = 127, + [1][1][2][0][RTW89_UK][8] = 127, + [1][1][2][0][RTW89_FCC][9] = 127, + [1][1][2][0][RTW89_ETSI][9] = 127, + [1][1][2][0][RTW89_MKK][9] = 127, + [1][1][2][0][RTW89_IC][9] = 127, + [1][1][2][0][RTW89_KCC][9] = 127, + [1][1][2][0][RTW89_ACMA][9] = 127, + [1][1][2][0][RTW89_CN][9] = 127, + [1][1][2][0][RTW89_UK][9] = 127, + [1][1][2][0][RTW89_FCC][10] = 127, + [1][1][2][0][RTW89_ETSI][10] = 127, + [1][1][2][0][RTW89_MKK][10] = 127, + [1][1][2][0][RTW89_IC][10] = 127, + [1][1][2][0][RTW89_KCC][10] = 127, + [1][1][2][0][RTW89_ACMA][10] = 127, + [1][1][2][0][RTW89_CN][10] = 127, + [1][1][2][0][RTW89_UK][10] = 127, + [1][1][2][0][RTW89_FCC][11] = 127, + [1][1][2][0][RTW89_ETSI][11] = 127, + [1][1][2][0][RTW89_MKK][11] = 127, + [1][1][2][0][RTW89_IC][11] = 127, + [1][1][2][0][RTW89_KCC][11] = 127, + [1][1][2][0][RTW89_ACMA][11] = 127, + [1][1][2][0][RTW89_CN][11] = 127, + [1][1][2][0][RTW89_UK][11] = 127, + [1][1][2][0][RTW89_FCC][12] = 127, + [1][1][2][0][RTW89_ETSI][12] = 127, + [1][1][2][0][RTW89_MKK][12] = 127, + [1][1][2][0][RTW89_IC][12] = 127, + [1][1][2][0][RTW89_KCC][12] = 127, + [1][1][2][0][RTW89_ACMA][12] = 127, + [1][1][2][0][RTW89_CN][12] = 127, + [1][1][2][0][RTW89_UK][12] = 127, + [1][1][2][0][RTW89_FCC][13] = 127, + [1][1][2][0][RTW89_ETSI][13] = 127, + [1][1][2][0][RTW89_MKK][13] = 127, + [1][1][2][0][RTW89_IC][13] = 127, + [1][1][2][0][RTW89_KCC][13] = 127, + [1][1][2][0][RTW89_ACMA][13] = 127, + [1][1][2][0][RTW89_CN][13] = 127, + [1][1][2][0][RTW89_UK][13] = 127, + [1][1][2][1][RTW89_FCC][0] = 127, + [1][1][2][1][RTW89_ETSI][0] = 127, + [1][1][2][1][RTW89_MKK][0] = 127, + [1][1][2][1][RTW89_IC][0] = 127, + [1][1][2][1][RTW89_KCC][0] = 127, + [1][1][2][1][RTW89_ACMA][0] = 127, + [1][1][2][1][RTW89_CN][0] = 127, + [1][1][2][1][RTW89_UK][0] = 127, + [1][1][2][1][RTW89_FCC][1] = 127, + [1][1][2][1][RTW89_ETSI][1] = 127, + [1][1][2][1][RTW89_MKK][1] = 127, + [1][1][2][1][RTW89_IC][1] = 127, + [1][1][2][1][RTW89_KCC][1] = 127, + [1][1][2][1][RTW89_ACMA][1] = 127, + [1][1][2][1][RTW89_CN][1] = 127, + [1][1][2][1][RTW89_UK][1] = 127, + [1][1][2][1][RTW89_FCC][2] = 127, + [1][1][2][1][RTW89_ETSI][2] = 127, + [1][1][2][1][RTW89_MKK][2] = 127, + [1][1][2][1][RTW89_IC][2] = 127, + [1][1][2][1][RTW89_KCC][2] = 127, + [1][1][2][1][RTW89_ACMA][2] = 127, + [1][1][2][1][RTW89_CN][2] = 127, + [1][1][2][1][RTW89_UK][2] = 127, + [1][1][2][1][RTW89_FCC][3] = 127, + [1][1][2][1][RTW89_ETSI][3] = 127, + [1][1][2][1][RTW89_MKK][3] = 127, + [1][1][2][1][RTW89_IC][3] = 127, + [1][1][2][1][RTW89_KCC][3] = 127, + [1][1][2][1][RTW89_ACMA][3] = 127, + [1][1][2][1][RTW89_CN][3] = 127, + [1][1][2][1][RTW89_UK][3] = 127, + [1][1][2][1][RTW89_FCC][4] = 127, + [1][1][2][1][RTW89_ETSI][4] = 127, + [1][1][2][1][RTW89_MKK][4] = 127, + [1][1][2][1][RTW89_IC][4] = 127, + [1][1][2][1][RTW89_KCC][4] = 127, + [1][1][2][1][RTW89_ACMA][4] = 127, + [1][1][2][1][RTW89_CN][4] = 127, + [1][1][2][1][RTW89_UK][4] = 127, + [1][1][2][1][RTW89_FCC][5] = 127, + [1][1][2][1][RTW89_ETSI][5] = 127, + [1][1][2][1][RTW89_MKK][5] = 127, + [1][1][2][1][RTW89_IC][5] = 127, + [1][1][2][1][RTW89_KCC][5] = 127, + [1][1][2][1][RTW89_ACMA][5] = 127, + [1][1][2][1][RTW89_CN][5] = 127, + [1][1][2][1][RTW89_UK][5] = 127, + [1][1][2][1][RTW89_FCC][6] = 127, + [1][1][2][1][RTW89_ETSI][6] = 127, + [1][1][2][1][RTW89_MKK][6] = 127, + [1][1][2][1][RTW89_IC][6] = 127, + [1][1][2][1][RTW89_KCC][6] = 127, + [1][1][2][1][RTW89_ACMA][6] = 127, + [1][1][2][1][RTW89_CN][6] = 127, + [1][1][2][1][RTW89_UK][6] = 127, + [1][1][2][1][RTW89_FCC][7] = 127, + [1][1][2][1][RTW89_ETSI][7] = 127, + [1][1][2][1][RTW89_MKK][7] = 127, + [1][1][2][1][RTW89_IC][7] = 127, + [1][1][2][1][RTW89_KCC][7] = 127, + [1][1][2][1][RTW89_ACMA][7] = 127, + [1][1][2][1][RTW89_CN][7] = 127, + [1][1][2][1][RTW89_UK][7] = 127, + [1][1][2][1][RTW89_FCC][8] = 127, + [1][1][2][1][RTW89_ETSI][8] = 127, + [1][1][2][1][RTW89_MKK][8] = 127, + [1][1][2][1][RTW89_IC][8] = 127, + [1][1][2][1][RTW89_KCC][8] = 127, + [1][1][2][1][RTW89_ACMA][8] = 127, + [1][1][2][1][RTW89_CN][8] = 127, + [1][1][2][1][RTW89_UK][8] = 127, + [1][1][2][1][RTW89_FCC][9] = 127, + [1][1][2][1][RTW89_ETSI][9] = 127, + [1][1][2][1][RTW89_MKK][9] = 127, + [1][1][2][1][RTW89_IC][9] = 127, + [1][1][2][1][RTW89_KCC][9] = 127, + [1][1][2][1][RTW89_ACMA][9] = 127, + [1][1][2][1][RTW89_CN][9] = 127, + [1][1][2][1][RTW89_UK][9] = 127, + [1][1][2][1][RTW89_FCC][10] = 127, + [1][1][2][1][RTW89_ETSI][10] = 127, + [1][1][2][1][RTW89_MKK][10] = 127, + [1][1][2][1][RTW89_IC][10] = 127, + [1][1][2][1][RTW89_KCC][10] = 127, + [1][1][2][1][RTW89_ACMA][10] = 127, + [1][1][2][1][RTW89_CN][10] = 127, + [1][1][2][1][RTW89_UK][10] = 127, + [1][1][2][1][RTW89_FCC][11] = 127, + [1][1][2][1][RTW89_ETSI][11] = 127, + [1][1][2][1][RTW89_MKK][11] = 127, + [1][1][2][1][RTW89_IC][11] = 127, + [1][1][2][1][RTW89_KCC][11] = 127, + [1][1][2][1][RTW89_ACMA][11] = 127, + [1][1][2][1][RTW89_CN][11] = 127, + [1][1][2][1][RTW89_UK][11] = 127, + [1][1][2][1][RTW89_FCC][12] = 127, + [1][1][2][1][RTW89_ETSI][12] = 127, + [1][1][2][1][RTW89_MKK][12] = 127, + [1][1][2][1][RTW89_IC][12] = 127, + [1][1][2][1][RTW89_KCC][12] = 127, + [1][1][2][1][RTW89_ACMA][12] = 127, + [1][1][2][1][RTW89_CN][12] = 127, + [1][1][2][1][RTW89_UK][12] = 127, + [1][1][2][1][RTW89_FCC][13] = 127, + [1][1][2][1][RTW89_ETSI][13] = 127, + [1][1][2][1][RTW89_MKK][13] = 127, + [1][1][2][1][RTW89_IC][13] = 127, + [1][1][2][1][RTW89_KCC][13] = 127, + [1][1][2][1][RTW89_ACMA][13] = 127, + [1][1][2][1][RTW89_CN][13] = 127, + [1][1][2][1][RTW89_UK][13] = 127, +}; + +static +const s8 rtw89_8851b_txpwr_lmt_5g_type2[RTW89_5G_BW_NUM][RTW89_NTX_NUM] + [RTW89_RS_LMT_NUM][RTW89_BF_NUM] + [RTW89_REGD_NUM][RTW89_5G_CH_NUM] = { + [0][0][1][0][RTW89_WW][0] = 58, + [0][0][1][0][RTW89_WW][2] = 58, + [0][0][1][0][RTW89_WW][4] = 58, + [0][0][1][0][RTW89_WW][6] = 50, + [0][0][1][0][RTW89_WW][8] = 58, + [0][0][1][0][RTW89_WW][10] = 58, + [0][0][1][0][RTW89_WW][12] = 58, + [0][0][1][0][RTW89_WW][14] = 58, + [0][0][1][0][RTW89_WW][15] = 58, + [0][0][1][0][RTW89_WW][17] = 60, + [0][0][1][0][RTW89_WW][19] = 60, + [0][0][1][0][RTW89_WW][21] = 60, + [0][0][1][0][RTW89_WW][23] = 60, + [0][0][1][0][RTW89_WW][25] = 60, + [0][0][1][0][RTW89_WW][27] = 60, + [0][0][1][0][RTW89_WW][29] = 60, + [0][0][1][0][RTW89_WW][31] = 60, + [0][0][1][0][RTW89_WW][33] = 60, + [0][0][1][0][RTW89_WW][35] = 60, + [0][0][1][0][RTW89_WW][37] = 74, + [0][0][1][0][RTW89_WW][38] = 30, + [0][0][1][0][RTW89_WW][40] = 30, + [0][0][1][0][RTW89_WW][42] = 30, + [0][0][1][0][RTW89_WW][44] = 30, + [0][0][1][0][RTW89_WW][46] = 30, + [0][0][1][0][RTW89_WW][48] = 72, + [0][0][1][0][RTW89_WW][50] = 72, + [0][0][1][0][RTW89_WW][52] = 72, + [0][1][1][0][RTW89_WW][0] = 0, + [0][1][1][0][RTW89_WW][2] = 0, + [0][1][1][0][RTW89_WW][4] = 0, + [0][1][1][0][RTW89_WW][6] = 0, + [0][1][1][0][RTW89_WW][8] = 0, + [0][1][1][0][RTW89_WW][10] = 0, + [0][1][1][0][RTW89_WW][12] = 0, + [0][1][1][0][RTW89_WW][14] = 0, + [0][1][1][0][RTW89_WW][15] = 0, + [0][1][1][0][RTW89_WW][17] = 0, + [0][1][1][0][RTW89_WW][19] = 0, + [0][1][1][0][RTW89_WW][21] = 0, + [0][1][1][0][RTW89_WW][23] = 0, + [0][1][1][0][RTW89_WW][25] = 0, + [0][1][1][0][RTW89_WW][27] = 0, + [0][1][1][0][RTW89_WW][29] = 0, + [0][1][1][0][RTW89_WW][31] = 0, + [0][1][1][0][RTW89_WW][33] = 0, + [0][1][1][0][RTW89_WW][35] = 0, + [0][1][1][0][RTW89_WW][37] = 0, + [0][1][1][0][RTW89_WW][38] = 0, + [0][1][1][0][RTW89_WW][40] = 0, + [0][1][1][0][RTW89_WW][42] = 0, + [0][1][1][0][RTW89_WW][44] = 0, + [0][1][1][0][RTW89_WW][46] = 0, + [0][1][1][0][RTW89_WW][48] = 0, + [0][1][1][0][RTW89_WW][50] = 0, + [0][1][1][0][RTW89_WW][52] = 0, + [0][0][2][0][RTW89_WW][0] = 62, + [0][0][2][0][RTW89_WW][2] = 62, + [0][0][2][0][RTW89_WW][4] = 62, + [0][0][2][0][RTW89_WW][6] = 54, + [0][0][2][0][RTW89_WW][8] = 62, + [0][0][2][0][RTW89_WW][10] = 62, + [0][0][2][0][RTW89_WW][12] = 62, + [0][0][2][0][RTW89_WW][14] = 62, + [0][0][2][0][RTW89_WW][15] = 60, + [0][0][2][0][RTW89_WW][17] = 62, + [0][0][2][0][RTW89_WW][19] = 62, + [0][0][2][0][RTW89_WW][21] = 62, + [0][0][2][0][RTW89_WW][23] = 62, + [0][0][2][0][RTW89_WW][25] = 62, + [0][0][2][0][RTW89_WW][27] = 62, + [0][0][2][0][RTW89_WW][29] = 62, + [0][0][2][0][RTW89_WW][31] = 62, + [0][0][2][0][RTW89_WW][33] = 62, + [0][0][2][0][RTW89_WW][35] = 62, + [0][0][2][0][RTW89_WW][37] = 74, + [0][0][2][0][RTW89_WW][38] = 30, + [0][0][2][0][RTW89_WW][40] = 30, + [0][0][2][0][RTW89_WW][42] = 30, + [0][0][2][0][RTW89_WW][44] = 30, + [0][0][2][0][RTW89_WW][46] = 30, + [0][0][2][0][RTW89_WW][48] = 74, + [0][0][2][0][RTW89_WW][50] = 74, + [0][0][2][0][RTW89_WW][52] = 74, + [0][1][2][0][RTW89_WW][0] = 0, + [0][1][2][0][RTW89_WW][2] = 0, + [0][1][2][0][RTW89_WW][4] = 0, + [0][1][2][0][RTW89_WW][6] = 0, + [0][1][2][0][RTW89_WW][8] = 0, + [0][1][2][0][RTW89_WW][10] = 0, + [0][1][2][0][RTW89_WW][12] = 0, + [0][1][2][0][RTW89_WW][14] = 0, + [0][1][2][0][RTW89_WW][15] = 0, + [0][1][2][0][RTW89_WW][17] = 0, + [0][1][2][0][RTW89_WW][19] = 0, + [0][1][2][0][RTW89_WW][21] = 0, + [0][1][2][0][RTW89_WW][23] = 0, + [0][1][2][0][RTW89_WW][25] = 0, + [0][1][2][0][RTW89_WW][27] = 0, + [0][1][2][0][RTW89_WW][29] = 0, + [0][1][2][0][RTW89_WW][31] = 0, + [0][1][2][0][RTW89_WW][33] = 0, + [0][1][2][0][RTW89_WW][35] = 0, + [0][1][2][0][RTW89_WW][37] = 0, + [0][1][2][0][RTW89_WW][38] = 0, + [0][1][2][0][RTW89_WW][40] = 0, + [0][1][2][0][RTW89_WW][42] = 0, + [0][1][2][0][RTW89_WW][44] = 0, + [0][1][2][0][RTW89_WW][46] = 0, + [0][1][2][0][RTW89_WW][48] = 0, + [0][1][2][0][RTW89_WW][50] = 0, + [0][1][2][0][RTW89_WW][52] = 0, + [0][1][2][1][RTW89_WW][0] = 0, + [0][1][2][1][RTW89_WW][2] = 0, + [0][1][2][1][RTW89_WW][4] = 0, + [0][1][2][1][RTW89_WW][6] = 0, + [0][1][2][1][RTW89_WW][8] = 0, + [0][1][2][1][RTW89_WW][10] = 0, + [0][1][2][1][RTW89_WW][12] = 0, + [0][1][2][1][RTW89_WW][14] = 0, + [0][1][2][1][RTW89_WW][15] = 0, + [0][1][2][1][RTW89_WW][17] = 0, + [0][1][2][1][RTW89_WW][19] = 0, + [0][1][2][1][RTW89_WW][21] = 0, + [0][1][2][1][RTW89_WW][23] = 0, + [0][1][2][1][RTW89_WW][25] = 0, + [0][1][2][1][RTW89_WW][27] = 0, + [0][1][2][1][RTW89_WW][29] = 0, + [0][1][2][1][RTW89_WW][31] = 0, + [0][1][2][1][RTW89_WW][33] = 0, + [0][1][2][1][RTW89_WW][35] = 0, + [0][1][2][1][RTW89_WW][37] = 0, + [0][1][2][1][RTW89_WW][38] = 0, + [0][1][2][1][RTW89_WW][40] = 0, + [0][1][2][1][RTW89_WW][42] = 0, + [0][1][2][1][RTW89_WW][44] = 0, + [0][1][2][1][RTW89_WW][46] = 0, + [0][1][2][1][RTW89_WW][48] = 0, + [0][1][2][1][RTW89_WW][50] = 0, + [0][1][2][1][RTW89_WW][52] = 0, + [1][0][2][0][RTW89_WW][1] = 64, + [1][0][2][0][RTW89_WW][5] = 62, + [1][0][2][0][RTW89_WW][9] = 64, + [1][0][2][0][RTW89_WW][13] = 64, + [1][0][2][0][RTW89_WW][16] = 66, + [1][0][2][0][RTW89_WW][20] = 66, + [1][0][2][0][RTW89_WW][24] = 66, + [1][0][2][0][RTW89_WW][28] = 66, + [1][0][2][0][RTW89_WW][32] = 66, + [1][0][2][0][RTW89_WW][36] = 76, + [1][0][2][0][RTW89_WW][39] = 30, + [1][0][2][0][RTW89_WW][43] = 30, + [1][0][2][0][RTW89_WW][47] = 80, + [1][0][2][0][RTW89_WW][51] = 80, + [1][1][2][0][RTW89_WW][1] = 0, + [1][1][2][0][RTW89_WW][5] = 0, + [1][1][2][0][RTW89_WW][9] = 0, + [1][1][2][0][RTW89_WW][13] = 0, + [1][1][2][0][RTW89_WW][16] = 0, + [1][1][2][0][RTW89_WW][20] = 0, + [1][1][2][0][RTW89_WW][24] = 0, + [1][1][2][0][RTW89_WW][28] = 0, + [1][1][2][0][RTW89_WW][32] = 0, + [1][1][2][0][RTW89_WW][36] = 0, + [1][1][2][0][RTW89_WW][39] = 0, + [1][1][2][0][RTW89_WW][43] = 0, + [1][1][2][0][RTW89_WW][47] = 0, + [1][1][2][0][RTW89_WW][51] = 0, + [1][1][2][1][RTW89_WW][1] = 0, + [1][1][2][1][RTW89_WW][5] = 0, + [1][1][2][1][RTW89_WW][9] = 0, + [1][1][2][1][RTW89_WW][13] = 0, + [1][1][2][1][RTW89_WW][16] = 0, + [1][1][2][1][RTW89_WW][20] = 0, + [1][1][2][1][RTW89_WW][24] = 0, + [1][1][2][1][RTW89_WW][28] = 0, + [1][1][2][1][RTW89_WW][32] = 0, + [1][1][2][1][RTW89_WW][36] = 0, + [1][1][2][1][RTW89_WW][39] = 0, + [1][1][2][1][RTW89_WW][43] = 0, + [1][1][2][1][RTW89_WW][47] = 0, + [1][1][2][1][RTW89_WW][51] = 0, + [2][0][2][0][RTW89_WW][3] = 62, + [2][0][2][0][RTW89_WW][11] = 62, + [2][0][2][0][RTW89_WW][18] = 64, + [2][0][2][0][RTW89_WW][26] = 64, + [2][0][2][0][RTW89_WW][34] = 68, + [2][0][2][0][RTW89_WW][41] = 30, + [2][0][2][0][RTW89_WW][49] = 72, + [2][1][2][0][RTW89_WW][3] = 0, + [2][1][2][0][RTW89_WW][11] = 0, + [2][1][2][0][RTW89_WW][18] = 0, + [2][1][2][0][RTW89_WW][26] = 0, + [2][1][2][0][RTW89_WW][34] = 0, + [2][1][2][0][RTW89_WW][41] = 0, + [2][1][2][0][RTW89_WW][49] = 0, + [2][1][2][1][RTW89_WW][3] = 0, + [2][1][2][1][RTW89_WW][11] = 0, + [2][1][2][1][RTW89_WW][18] = 0, + [2][1][2][1][RTW89_WW][26] = 0, + [2][1][2][1][RTW89_WW][34] = 0, + [2][1][2][1][RTW89_WW][41] = 0, + [2][1][2][1][RTW89_WW][49] = 0, + [3][0][2][0][RTW89_WW][7] = 58, + [3][0][2][0][RTW89_WW][22] = 58, + [3][0][2][0][RTW89_WW][45] = 0, + [3][1][2][0][RTW89_WW][7] = 0, + [3][1][2][0][RTW89_WW][22] = 0, + [3][1][2][0][RTW89_WW][45] = 0, + [3][1][2][1][RTW89_WW][7] = 0, + [3][1][2][1][RTW89_WW][22] = 0, + [3][1][2][1][RTW89_WW][45] = 0, + [0][0][1][0][RTW89_FCC][0] = 78, + [0][0][1][0][RTW89_ETSI][0] = 58, + [0][0][1][0][RTW89_MKK][0] = 60, + [0][0][1][0][RTW89_IC][0] = 62, + [0][0][1][0][RTW89_KCC][0] = 74, + [0][0][1][0][RTW89_ACMA][0] = 58, + [0][0][1][0][RTW89_CN][0] = 60, + [0][0][1][0][RTW89_UK][0] = 58, + [0][0][1][0][RTW89_FCC][2] = 78, + [0][0][1][0][RTW89_ETSI][2] = 58, + [0][0][1][0][RTW89_MKK][2] = 60, + [0][0][1][0][RTW89_IC][2] = 62, + [0][0][1][0][RTW89_KCC][2] = 74, + [0][0][1][0][RTW89_ACMA][2] = 58, + [0][0][1][0][RTW89_CN][2] = 60, + [0][0][1][0][RTW89_UK][2] = 58, + [0][0][1][0][RTW89_FCC][4] = 78, + [0][0][1][0][RTW89_ETSI][4] = 58, + [0][0][1][0][RTW89_MKK][4] = 60, + [0][0][1][0][RTW89_IC][4] = 62, + [0][0][1][0][RTW89_KCC][4] = 74, + [0][0][1][0][RTW89_ACMA][4] = 58, + [0][0][1][0][RTW89_CN][4] = 60, + [0][0][1][0][RTW89_UK][4] = 58, + [0][0][1][0][RTW89_FCC][6] = 78, + [0][0][1][0][RTW89_ETSI][6] = 58, + [0][0][1][0][RTW89_MKK][6] = 60, + [0][0][1][0][RTW89_IC][6] = 62, + [0][0][1][0][RTW89_KCC][6] = 50, + [0][0][1][0][RTW89_ACMA][6] = 58, + [0][0][1][0][RTW89_CN][6] = 60, + [0][0][1][0][RTW89_UK][6] = 58, + [0][0][1][0][RTW89_FCC][8] = 78, + [0][0][1][0][RTW89_ETSI][8] = 58, + [0][0][1][0][RTW89_MKK][8] = 60, + [0][0][1][0][RTW89_IC][8] = 62, + [0][0][1][0][RTW89_KCC][8] = 74, + [0][0][1][0][RTW89_ACMA][8] = 58, + [0][0][1][0][RTW89_CN][8] = 60, + [0][0][1][0][RTW89_UK][8] = 58, + [0][0][1][0][RTW89_FCC][10] = 78, + [0][0][1][0][RTW89_ETSI][10] = 58, + [0][0][1][0][RTW89_MKK][10] = 60, + [0][0][1][0][RTW89_IC][10] = 64, + [0][0][1][0][RTW89_KCC][10] = 74, + [0][0][1][0][RTW89_ACMA][10] = 58, + [0][0][1][0][RTW89_CN][10] = 60, + [0][0][1][0][RTW89_UK][10] = 58, + [0][0][1][0][RTW89_FCC][12] = 78, + [0][0][1][0][RTW89_ETSI][12] = 58, + [0][0][1][0][RTW89_MKK][12] = 60, + [0][0][1][0][RTW89_IC][12] = 64, + [0][0][1][0][RTW89_KCC][12] = 74, + [0][0][1][0][RTW89_ACMA][12] = 58, + [0][0][1][0][RTW89_CN][12] = 60, + [0][0][1][0][RTW89_UK][12] = 58, + [0][0][1][0][RTW89_FCC][14] = 76, + [0][0][1][0][RTW89_ETSI][14] = 58, + [0][0][1][0][RTW89_MKK][14] = 60, + [0][0][1][0][RTW89_IC][14] = 62, + [0][0][1][0][RTW89_KCC][14] = 74, + [0][0][1][0][RTW89_ACMA][14] = 58, + [0][0][1][0][RTW89_CN][14] = 60, + [0][0][1][0][RTW89_UK][14] = 58, + [0][0][1][0][RTW89_FCC][15] = 76, + [0][0][1][0][RTW89_ETSI][15] = 58, + [0][0][1][0][RTW89_MKK][15] = 74, + [0][0][1][0][RTW89_IC][15] = 76, + [0][0][1][0][RTW89_KCC][15] = 74, + [0][0][1][0][RTW89_ACMA][15] = 58, + [0][0][1][0][RTW89_CN][15] = 127, + [0][0][1][0][RTW89_UK][15] = 58, + [0][0][1][0][RTW89_FCC][17] = 78, + [0][0][1][0][RTW89_ETSI][17] = 60, + [0][0][1][0][RTW89_MKK][17] = 74, + [0][0][1][0][RTW89_IC][17] = 78, + [0][0][1][0][RTW89_KCC][17] = 74, + [0][0][1][0][RTW89_ACMA][17] = 60, + [0][0][1][0][RTW89_CN][17] = 127, + [0][0][1][0][RTW89_UK][17] = 60, + [0][0][1][0][RTW89_FCC][19] = 78, + [0][0][1][0][RTW89_ETSI][19] = 60, + [0][0][1][0][RTW89_MKK][19] = 74, + [0][0][1][0][RTW89_IC][19] = 78, + [0][0][1][0][RTW89_KCC][19] = 74, + [0][0][1][0][RTW89_ACMA][19] = 60, + [0][0][1][0][RTW89_CN][19] = 127, + [0][0][1][0][RTW89_UK][19] = 60, + [0][0][1][0][RTW89_FCC][21] = 78, + [0][0][1][0][RTW89_ETSI][21] = 60, + [0][0][1][0][RTW89_MKK][21] = 74, + [0][0][1][0][RTW89_IC][21] = 78, + [0][0][1][0][RTW89_KCC][21] = 74, + [0][0][1][0][RTW89_ACMA][21] = 60, + [0][0][1][0][RTW89_CN][21] = 127, + [0][0][1][0][RTW89_UK][21] = 60, + [0][0][1][0][RTW89_FCC][23] = 78, + [0][0][1][0][RTW89_ETSI][23] = 60, + [0][0][1][0][RTW89_MKK][23] = 74, + [0][0][1][0][RTW89_IC][23] = 78, + [0][0][1][0][RTW89_KCC][23] = 74, + [0][0][1][0][RTW89_ACMA][23] = 60, + [0][0][1][0][RTW89_CN][23] = 127, + [0][0][1][0][RTW89_UK][23] = 60, + [0][0][1][0][RTW89_FCC][25] = 78, + [0][0][1][0][RTW89_ETSI][25] = 60, + [0][0][1][0][RTW89_MKK][25] = 74, + [0][0][1][0][RTW89_IC][25] = 127, + [0][0][1][0][RTW89_KCC][25] = 74, + [0][0][1][0][RTW89_ACMA][25] = 127, + [0][0][1][0][RTW89_CN][25] = 127, + [0][0][1][0][RTW89_UK][25] = 60, + [0][0][1][0][RTW89_FCC][27] = 78, + [0][0][1][0][RTW89_ETSI][27] = 60, + [0][0][1][0][RTW89_MKK][27] = 74, + [0][0][1][0][RTW89_IC][27] = 127, + [0][0][1][0][RTW89_KCC][27] = 74, + [0][0][1][0][RTW89_ACMA][27] = 127, + [0][0][1][0][RTW89_CN][27] = 127, + [0][0][1][0][RTW89_UK][27] = 60, + [0][0][1][0][RTW89_FCC][29] = 78, + [0][0][1][0][RTW89_ETSI][29] = 60, + [0][0][1][0][RTW89_MKK][29] = 74, + [0][0][1][0][RTW89_IC][29] = 127, + [0][0][1][0][RTW89_KCC][29] = 74, + [0][0][1][0][RTW89_ACMA][29] = 127, + [0][0][1][0][RTW89_CN][29] = 127, + [0][0][1][0][RTW89_UK][29] = 60, + [0][0][1][0][RTW89_FCC][31] = 78, + [0][0][1][0][RTW89_ETSI][31] = 60, + [0][0][1][0][RTW89_MKK][31] = 74, + [0][0][1][0][RTW89_IC][31] = 78, + [0][0][1][0][RTW89_KCC][31] = 74, + [0][0][1][0][RTW89_ACMA][31] = 60, + [0][0][1][0][RTW89_CN][31] = 127, + [0][0][1][0][RTW89_UK][31] = 60, + [0][0][1][0][RTW89_FCC][33] = 78, + [0][0][1][0][RTW89_ETSI][33] = 60, + [0][0][1][0][RTW89_MKK][33] = 74, + [0][0][1][0][RTW89_IC][33] = 78, + [0][0][1][0][RTW89_KCC][33] = 74, + [0][0][1][0][RTW89_ACMA][33] = 60, + [0][0][1][0][RTW89_CN][33] = 127, + [0][0][1][0][RTW89_UK][33] = 60, + [0][0][1][0][RTW89_FCC][35] = 70, + [0][0][1][0][RTW89_ETSI][35] = 60, + [0][0][1][0][RTW89_MKK][35] = 74, + [0][0][1][0][RTW89_IC][35] = 70, + [0][0][1][0][RTW89_KCC][35] = 74, + [0][0][1][0][RTW89_ACMA][35] = 60, + [0][0][1][0][RTW89_CN][35] = 127, + [0][0][1][0][RTW89_UK][35] = 60, + [0][0][1][0][RTW89_FCC][37] = 78, + [0][0][1][0][RTW89_ETSI][37] = 127, + [0][0][1][0][RTW89_MKK][37] = 74, + [0][0][1][0][RTW89_IC][37] = 78, + [0][0][1][0][RTW89_KCC][37] = 74, + [0][0][1][0][RTW89_ACMA][37] = 74, + [0][0][1][0][RTW89_CN][37] = 127, + [0][0][1][0][RTW89_UK][37] = 74, + [0][0][1][0][RTW89_FCC][38] = 78, + [0][0][1][0][RTW89_ETSI][38] = 30, + [0][0][1][0][RTW89_MKK][38] = 127, + [0][0][1][0][RTW89_IC][38] = 78, + [0][0][1][0][RTW89_KCC][38] = 70, + [0][0][1][0][RTW89_ACMA][38] = 74, + [0][0][1][0][RTW89_CN][38] = 74, + [0][0][1][0][RTW89_UK][38] = 58, + [0][0][1][0][RTW89_FCC][40] = 78, + [0][0][1][0][RTW89_ETSI][40] = 30, + [0][0][1][0][RTW89_MKK][40] = 127, + [0][0][1][0][RTW89_IC][40] = 78, + [0][0][1][0][RTW89_KCC][40] = 74, + [0][0][1][0][RTW89_ACMA][40] = 74, + [0][0][1][0][RTW89_CN][40] = 74, + [0][0][1][0][RTW89_UK][40] = 58, + [0][0][1][0][RTW89_FCC][42] = 78, + [0][0][1][0][RTW89_ETSI][42] = 30, + [0][0][1][0][RTW89_MKK][42] = 127, + [0][0][1][0][RTW89_IC][42] = 78, + [0][0][1][0][RTW89_KCC][42] = 74, + [0][0][1][0][RTW89_ACMA][42] = 74, + [0][0][1][0][RTW89_CN][42] = 74, + [0][0][1][0][RTW89_UK][42] = 58, + [0][0][1][0][RTW89_FCC][44] = 78, + [0][0][1][0][RTW89_ETSI][44] = 30, + [0][0][1][0][RTW89_MKK][44] = 127, + [0][0][1][0][RTW89_IC][44] = 78, + [0][0][1][0][RTW89_KCC][44] = 74, + [0][0][1][0][RTW89_ACMA][44] = 74, + [0][0][1][0][RTW89_CN][44] = 74, + [0][0][1][0][RTW89_UK][44] = 58, + [0][0][1][0][RTW89_FCC][46] = 78, + [0][0][1][0][RTW89_ETSI][46] = 30, + [0][0][1][0][RTW89_MKK][46] = 127, + [0][0][1][0][RTW89_IC][46] = 78, + [0][0][1][0][RTW89_KCC][46] = 74, + [0][0][1][0][RTW89_ACMA][46] = 74, + [0][0][1][0][RTW89_CN][46] = 74, + [0][0][1][0][RTW89_UK][46] = 58, + [0][0][1][0][RTW89_FCC][48] = 72, + [0][0][1][0][RTW89_ETSI][48] = 127, + [0][0][1][0][RTW89_MKK][48] = 127, + [0][0][1][0][RTW89_IC][48] = 127, + [0][0][1][0][RTW89_KCC][48] = 127, + [0][0][1][0][RTW89_ACMA][48] = 127, + [0][0][1][0][RTW89_CN][48] = 127, + [0][0][1][0][RTW89_UK][48] = 127, + [0][0][1][0][RTW89_FCC][50] = 72, + [0][0][1][0][RTW89_ETSI][50] = 127, + [0][0][1][0][RTW89_MKK][50] = 127, + [0][0][1][0][RTW89_IC][50] = 127, + [0][0][1][0][RTW89_KCC][50] = 127, + [0][0][1][0][RTW89_ACMA][50] = 127, + [0][0][1][0][RTW89_CN][50] = 127, + [0][0][1][0][RTW89_UK][50] = 127, + [0][0][1][0][RTW89_FCC][52] = 72, + [0][0][1][0][RTW89_ETSI][52] = 127, + [0][0][1][0][RTW89_MKK][52] = 127, + [0][0][1][0][RTW89_IC][52] = 127, + [0][0][1][0][RTW89_KCC][52] = 127, + [0][0][1][0][RTW89_ACMA][52] = 127, + [0][0][1][0][RTW89_CN][52] = 127, + [0][0][1][0][RTW89_UK][52] = 127, + [0][1][1][0][RTW89_FCC][0] = 127, + [0][1][1][0][RTW89_ETSI][0] = 127, + [0][1][1][0][RTW89_MKK][0] = 127, + [0][1][1][0][RTW89_IC][0] = 127, + [0][1][1][0][RTW89_KCC][0] = 127, + [0][1][1][0][RTW89_ACMA][0] = 127, + [0][1][1][0][RTW89_CN][0] = 127, + [0][1][1][0][RTW89_UK][0] = 127, + [0][1][1][0][RTW89_FCC][2] = 127, + [0][1][1][0][RTW89_ETSI][2] = 127, + [0][1][1][0][RTW89_MKK][2] = 127, + [0][1][1][0][RTW89_IC][2] = 127, + [0][1][1][0][RTW89_KCC][2] = 127, + [0][1][1][0][RTW89_ACMA][2] = 127, + [0][1][1][0][RTW89_CN][2] = 127, + [0][1][1][0][RTW89_UK][2] = 127, + [0][1][1][0][RTW89_FCC][4] = 127, + [0][1][1][0][RTW89_ETSI][4] = 127, + [0][1][1][0][RTW89_MKK][4] = 127, + [0][1][1][0][RTW89_IC][4] = 127, + [0][1][1][0][RTW89_KCC][4] = 127, + [0][1][1][0][RTW89_ACMA][4] = 127, + [0][1][1][0][RTW89_CN][4] = 127, + [0][1][1][0][RTW89_UK][4] = 127, + [0][1][1][0][RTW89_FCC][6] = 127, + [0][1][1][0][RTW89_ETSI][6] = 127, + [0][1][1][0][RTW89_MKK][6] = 127, + [0][1][1][0][RTW89_IC][6] = 127, + [0][1][1][0][RTW89_KCC][6] = 127, + [0][1][1][0][RTW89_ACMA][6] = 127, + [0][1][1][0][RTW89_CN][6] = 127, + [0][1][1][0][RTW89_UK][6] = 127, + [0][1][1][0][RTW89_FCC][8] = 127, + [0][1][1][0][RTW89_ETSI][8] = 127, + [0][1][1][0][RTW89_MKK][8] = 127, + [0][1][1][0][RTW89_IC][8] = 127, + [0][1][1][0][RTW89_KCC][8] = 127, + [0][1][1][0][RTW89_ACMA][8] = 127, + [0][1][1][0][RTW89_CN][8] = 127, + [0][1][1][0][RTW89_UK][8] = 127, + [0][1][1][0][RTW89_FCC][10] = 127, + [0][1][1][0][RTW89_ETSI][10] = 127, + [0][1][1][0][RTW89_MKK][10] = 127, + [0][1][1][0][RTW89_IC][10] = 127, + [0][1][1][0][RTW89_KCC][10] = 127, + [0][1][1][0][RTW89_ACMA][10] = 127, + [0][1][1][0][RTW89_CN][10] = 127, + [0][1][1][0][RTW89_UK][10] = 127, + [0][1][1][0][RTW89_FCC][12] = 127, + [0][1][1][0][RTW89_ETSI][12] = 127, + [0][1][1][0][RTW89_MKK][12] = 127, + [0][1][1][0][RTW89_IC][12] = 127, + [0][1][1][0][RTW89_KCC][12] = 127, + [0][1][1][0][RTW89_ACMA][12] = 127, + [0][1][1][0][RTW89_CN][12] = 127, + [0][1][1][0][RTW89_UK][12] = 127, + [0][1][1][0][RTW89_FCC][14] = 127, + [0][1][1][0][RTW89_ETSI][14] = 127, + [0][1][1][0][RTW89_MKK][14] = 127, + [0][1][1][0][RTW89_IC][14] = 127, + [0][1][1][0][RTW89_KCC][14] = 127, + [0][1][1][0][RTW89_ACMA][14] = 127, + [0][1][1][0][RTW89_CN][14] = 127, + [0][1][1][0][RTW89_UK][14] = 127, + [0][1][1][0][RTW89_FCC][15] = 127, + [0][1][1][0][RTW89_ETSI][15] = 127, + [0][1][1][0][RTW89_MKK][15] = 127, + [0][1][1][0][RTW89_IC][15] = 127, + [0][1][1][0][RTW89_KCC][15] = 127, + [0][1][1][0][RTW89_ACMA][15] = 127, + [0][1][1][0][RTW89_CN][15] = 127, + [0][1][1][0][RTW89_UK][15] = 127, + [0][1][1][0][RTW89_FCC][17] = 127, + [0][1][1][0][RTW89_ETSI][17] = 127, + [0][1][1][0][RTW89_MKK][17] = 127, + [0][1][1][0][RTW89_IC][17] = 127, + [0][1][1][0][RTW89_KCC][17] = 127, + [0][1][1][0][RTW89_ACMA][17] = 127, + [0][1][1][0][RTW89_CN][17] = 127, + [0][1][1][0][RTW89_UK][17] = 127, + [0][1][1][0][RTW89_FCC][19] = 127, + [0][1][1][0][RTW89_ETSI][19] = 127, + [0][1][1][0][RTW89_MKK][19] = 127, + [0][1][1][0][RTW89_IC][19] = 127, + [0][1][1][0][RTW89_KCC][19] = 127, + [0][1][1][0][RTW89_ACMA][19] = 127, + [0][1][1][0][RTW89_CN][19] = 127, + [0][1][1][0][RTW89_UK][19] = 127, + [0][1][1][0][RTW89_FCC][21] = 127, + [0][1][1][0][RTW89_ETSI][21] = 127, + [0][1][1][0][RTW89_MKK][21] = 127, + [0][1][1][0][RTW89_IC][21] = 127, + [0][1][1][0][RTW89_KCC][21] = 127, + [0][1][1][0][RTW89_ACMA][21] = 127, + [0][1][1][0][RTW89_CN][21] = 127, + [0][1][1][0][RTW89_UK][21] = 127, + [0][1][1][0][RTW89_FCC][23] = 127, + [0][1][1][0][RTW89_ETSI][23] = 127, + [0][1][1][0][RTW89_MKK][23] = 127, + [0][1][1][0][RTW89_IC][23] = 127, + [0][1][1][0][RTW89_KCC][23] = 127, + [0][1][1][0][RTW89_ACMA][23] = 127, + [0][1][1][0][RTW89_CN][23] = 127, + [0][1][1][0][RTW89_UK][23] = 127, + [0][1][1][0][RTW89_FCC][25] = 127, + [0][1][1][0][RTW89_ETSI][25] = 127, + [0][1][1][0][RTW89_MKK][25] = 127, + [0][1][1][0][RTW89_IC][25] = 127, + [0][1][1][0][RTW89_KCC][25] = 127, + [0][1][1][0][RTW89_ACMA][25] = 127, + [0][1][1][0][RTW89_CN][25] = 127, + [0][1][1][0][RTW89_UK][25] = 127, + [0][1][1][0][RTW89_FCC][27] = 127, + [0][1][1][0][RTW89_ETSI][27] = 127, + [0][1][1][0][RTW89_MKK][27] = 127, + [0][1][1][0][RTW89_IC][27] = 127, + [0][1][1][0][RTW89_KCC][27] = 127, + [0][1][1][0][RTW89_ACMA][27] = 127, + [0][1][1][0][RTW89_CN][27] = 127, + [0][1][1][0][RTW89_UK][27] = 127, + [0][1][1][0][RTW89_FCC][29] = 127, + [0][1][1][0][RTW89_ETSI][29] = 127, + [0][1][1][0][RTW89_MKK][29] = 127, + [0][1][1][0][RTW89_IC][29] = 127, + [0][1][1][0][RTW89_KCC][29] = 127, + [0][1][1][0][RTW89_ACMA][29] = 127, + [0][1][1][0][RTW89_CN][29] = 127, + [0][1][1][0][RTW89_UK][29] = 127, + [0][1][1][0][RTW89_FCC][31] = 127, + [0][1][1][0][RTW89_ETSI][31] = 127, + [0][1][1][0][RTW89_MKK][31] = 127, + [0][1][1][0][RTW89_IC][31] = 127, + [0][1][1][0][RTW89_KCC][31] = 127, + [0][1][1][0][RTW89_ACMA][31] = 127, + [0][1][1][0][RTW89_CN][31] = 127, + [0][1][1][0][RTW89_UK][31] = 127, + [0][1][1][0][RTW89_FCC][33] = 127, + [0][1][1][0][RTW89_ETSI][33] = 127, + [0][1][1][0][RTW89_MKK][33] = 127, + [0][1][1][0][RTW89_IC][33] = 127, + [0][1][1][0][RTW89_KCC][33] = 127, + [0][1][1][0][RTW89_ACMA][33] = 127, + [0][1][1][0][RTW89_CN][33] = 127, + [0][1][1][0][RTW89_UK][33] = 127, + [0][1][1][0][RTW89_FCC][35] = 127, + [0][1][1][0][RTW89_ETSI][35] = 127, + [0][1][1][0][RTW89_MKK][35] = 127, + [0][1][1][0][RTW89_IC][35] = 127, + [0][1][1][0][RTW89_KCC][35] = 127, + [0][1][1][0][RTW89_ACMA][35] = 127, + [0][1][1][0][RTW89_CN][35] = 127, + [0][1][1][0][RTW89_UK][35] = 127, + [0][1][1][0][RTW89_FCC][37] = 127, + [0][1][1][0][RTW89_ETSI][37] = 127, + [0][1][1][0][RTW89_MKK][37] = 127, + [0][1][1][0][RTW89_IC][37] = 127, + [0][1][1][0][RTW89_KCC][37] = 127, + [0][1][1][0][RTW89_ACMA][37] = 127, + [0][1][1][0][RTW89_CN][37] = 127, + [0][1][1][0][RTW89_UK][37] = 127, + [0][1][1][0][RTW89_FCC][38] = 127, + [0][1][1][0][RTW89_ETSI][38] = 127, + [0][1][1][0][RTW89_MKK][38] = 127, + [0][1][1][0][RTW89_IC][38] = 127, + [0][1][1][0][RTW89_KCC][38] = 127, + [0][1][1][0][RTW89_ACMA][38] = 127, + [0][1][1][0][RTW89_CN][38] = 127, + [0][1][1][0][RTW89_UK][38] = 127, + [0][1][1][0][RTW89_FCC][40] = 127, + [0][1][1][0][RTW89_ETSI][40] = 127, + [0][1][1][0][RTW89_MKK][40] = 127, + [0][1][1][0][RTW89_IC][40] = 127, + [0][1][1][0][RTW89_KCC][40] = 127, + [0][1][1][0][RTW89_ACMA][40] = 127, + [0][1][1][0][RTW89_CN][40] = 127, + [0][1][1][0][RTW89_UK][40] = 127, + [0][1][1][0][RTW89_FCC][42] = 127, + [0][1][1][0][RTW89_ETSI][42] = 127, + [0][1][1][0][RTW89_MKK][42] = 127, + [0][1][1][0][RTW89_IC][42] = 127, + [0][1][1][0][RTW89_KCC][42] = 127, + [0][1][1][0][RTW89_ACMA][42] = 127, + [0][1][1][0][RTW89_CN][42] = 127, + [0][1][1][0][RTW89_UK][42] = 127, + [0][1][1][0][RTW89_FCC][44] = 127, + [0][1][1][0][RTW89_ETSI][44] = 127, + [0][1][1][0][RTW89_MKK][44] = 127, + [0][1][1][0][RTW89_IC][44] = 127, + [0][1][1][0][RTW89_KCC][44] = 127, + [0][1][1][0][RTW89_ACMA][44] = 127, + [0][1][1][0][RTW89_CN][44] = 127, + [0][1][1][0][RTW89_UK][44] = 127, + [0][1][1][0][RTW89_FCC][46] = 127, + [0][1][1][0][RTW89_ETSI][46] = 127, + [0][1][1][0][RTW89_MKK][46] = 127, + [0][1][1][0][RTW89_IC][46] = 127, + [0][1][1][0][RTW89_KCC][46] = 127, + [0][1][1][0][RTW89_ACMA][46] = 127, + [0][1][1][0][RTW89_CN][46] = 127, + [0][1][1][0][RTW89_UK][46] = 127, + [0][1][1][0][RTW89_FCC][48] = 127, + [0][1][1][0][RTW89_ETSI][48] = 127, + [0][1][1][0][RTW89_MKK][48] = 127, + [0][1][1][0][RTW89_IC][48] = 127, + [0][1][1][0][RTW89_KCC][48] = 127, + [0][1][1][0][RTW89_ACMA][48] = 127, + [0][1][1][0][RTW89_CN][48] = 127, + [0][1][1][0][RTW89_UK][48] = 127, + [0][1][1][0][RTW89_FCC][50] = 127, + [0][1][1][0][RTW89_ETSI][50] = 127, + [0][1][1][0][RTW89_MKK][50] = 127, + [0][1][1][0][RTW89_IC][50] = 127, + [0][1][1][0][RTW89_KCC][50] = 127, + [0][1][1][0][RTW89_ACMA][50] = 127, + [0][1][1][0][RTW89_CN][50] = 127, + [0][1][1][0][RTW89_UK][50] = 127, + [0][1][1][0][RTW89_FCC][52] = 127, + [0][1][1][0][RTW89_ETSI][52] = 127, + [0][1][1][0][RTW89_MKK][52] = 127, + [0][1][1][0][RTW89_IC][52] = 127, + [0][1][1][0][RTW89_KCC][52] = 127, + [0][1][1][0][RTW89_ACMA][52] = 127, + [0][1][1][0][RTW89_CN][52] = 127, + [0][1][1][0][RTW89_UK][52] = 127, + [0][0][2][0][RTW89_FCC][0] = 76, + [0][0][2][0][RTW89_ETSI][0] = 62, + [0][0][2][0][RTW89_MKK][0] = 62, + [0][0][2][0][RTW89_IC][0] = 64, + [0][0][2][0][RTW89_KCC][0] = 74, + [0][0][2][0][RTW89_ACMA][0] = 62, + [0][0][2][0][RTW89_CN][0] = 62, + [0][0][2][0][RTW89_UK][0] = 62, + [0][0][2][0][RTW89_FCC][2] = 78, + [0][0][2][0][RTW89_ETSI][2] = 62, + [0][0][2][0][RTW89_MKK][2] = 62, + [0][0][2][0][RTW89_IC][2] = 64, + [0][0][2][0][RTW89_KCC][2] = 74, + [0][0][2][0][RTW89_ACMA][2] = 62, + [0][0][2][0][RTW89_CN][2] = 62, + [0][0][2][0][RTW89_UK][2] = 62, + [0][0][2][0][RTW89_FCC][4] = 78, + [0][0][2][0][RTW89_ETSI][4] = 62, + [0][0][2][0][RTW89_MKK][4] = 62, + [0][0][2][0][RTW89_IC][4] = 64, + [0][0][2][0][RTW89_KCC][4] = 74, + [0][0][2][0][RTW89_ACMA][4] = 62, + [0][0][2][0][RTW89_CN][4] = 62, + [0][0][2][0][RTW89_UK][4] = 62, + [0][0][2][0][RTW89_FCC][6] = 78, + [0][0][2][0][RTW89_ETSI][6] = 62, + [0][0][2][0][RTW89_MKK][6] = 62, + [0][0][2][0][RTW89_IC][6] = 64, + [0][0][2][0][RTW89_KCC][6] = 54, + [0][0][2][0][RTW89_ACMA][6] = 62, + [0][0][2][0][RTW89_CN][6] = 62, + [0][0][2][0][RTW89_UK][6] = 62, + [0][0][2][0][RTW89_FCC][8] = 78, + [0][0][2][0][RTW89_ETSI][8] = 62, + [0][0][2][0][RTW89_MKK][8] = 62, + [0][0][2][0][RTW89_IC][8] = 64, + [0][0][2][0][RTW89_KCC][8] = 74, + [0][0][2][0][RTW89_ACMA][8] = 62, + [0][0][2][0][RTW89_CN][8] = 62, + [0][0][2][0][RTW89_UK][8] = 62, + [0][0][2][0][RTW89_FCC][10] = 78, + [0][0][2][0][RTW89_ETSI][10] = 62, + [0][0][2][0][RTW89_MKK][10] = 62, + [0][0][2][0][RTW89_IC][10] = 64, + [0][0][2][0][RTW89_KCC][10] = 74, + [0][0][2][0][RTW89_ACMA][10] = 62, + [0][0][2][0][RTW89_CN][10] = 62, + [0][0][2][0][RTW89_UK][10] = 62, + [0][0][2][0][RTW89_FCC][12] = 78, + [0][0][2][0][RTW89_ETSI][12] = 62, + [0][0][2][0][RTW89_MKK][12] = 62, + [0][0][2][0][RTW89_IC][12] = 64, + [0][0][2][0][RTW89_KCC][12] = 74, + [0][0][2][0][RTW89_ACMA][12] = 62, + [0][0][2][0][RTW89_CN][12] = 62, + [0][0][2][0][RTW89_UK][12] = 62, + [0][0][2][0][RTW89_FCC][14] = 74, + [0][0][2][0][RTW89_ETSI][14] = 62, + [0][0][2][0][RTW89_MKK][14] = 62, + [0][0][2][0][RTW89_IC][14] = 64, + [0][0][2][0][RTW89_KCC][14] = 74, + [0][0][2][0][RTW89_ACMA][14] = 62, + [0][0][2][0][RTW89_CN][14] = 62, + [0][0][2][0][RTW89_UK][14] = 62, + [0][0][2][0][RTW89_FCC][15] = 74, + [0][0][2][0][RTW89_ETSI][15] = 60, + [0][0][2][0][RTW89_MKK][15] = 74, + [0][0][2][0][RTW89_IC][15] = 74, + [0][0][2][0][RTW89_KCC][15] = 74, + [0][0][2][0][RTW89_ACMA][15] = 60, + [0][0][2][0][RTW89_CN][15] = 127, + [0][0][2][0][RTW89_UK][15] = 60, + [0][0][2][0][RTW89_FCC][17] = 78, + [0][0][2][0][RTW89_ETSI][17] = 62, + [0][0][2][0][RTW89_MKK][17] = 74, + [0][0][2][0][RTW89_IC][17] = 78, + [0][0][2][0][RTW89_KCC][17] = 74, + [0][0][2][0][RTW89_ACMA][17] = 62, + [0][0][2][0][RTW89_CN][17] = 127, + [0][0][2][0][RTW89_UK][17] = 62, + [0][0][2][0][RTW89_FCC][19] = 78, + [0][0][2][0][RTW89_ETSI][19] = 62, + [0][0][2][0][RTW89_MKK][19] = 74, + [0][0][2][0][RTW89_IC][19] = 78, + [0][0][2][0][RTW89_KCC][19] = 74, + [0][0][2][0][RTW89_ACMA][19] = 62, + [0][0][2][0][RTW89_CN][19] = 127, + [0][0][2][0][RTW89_UK][19] = 62, + [0][0][2][0][RTW89_FCC][21] = 78, + [0][0][2][0][RTW89_ETSI][21] = 62, + [0][0][2][0][RTW89_MKK][21] = 74, + [0][0][2][0][RTW89_IC][21] = 78, + [0][0][2][0][RTW89_KCC][21] = 74, + [0][0][2][0][RTW89_ACMA][21] = 62, + [0][0][2][0][RTW89_CN][21] = 127, + [0][0][2][0][RTW89_UK][21] = 62, + [0][0][2][0][RTW89_FCC][23] = 78, + [0][0][2][0][RTW89_ETSI][23] = 62, + [0][0][2][0][RTW89_MKK][23] = 74, + [0][0][2][0][RTW89_IC][23] = 78, + [0][0][2][0][RTW89_KCC][23] = 74, + [0][0][2][0][RTW89_ACMA][23] = 62, + [0][0][2][0][RTW89_CN][23] = 127, + [0][0][2][0][RTW89_UK][23] = 62, + [0][0][2][0][RTW89_FCC][25] = 78, + [0][0][2][0][RTW89_ETSI][25] = 62, + [0][0][2][0][RTW89_MKK][25] = 74, + [0][0][2][0][RTW89_IC][25] = 127, + [0][0][2][0][RTW89_KCC][25] = 74, + [0][0][2][0][RTW89_ACMA][25] = 127, + [0][0][2][0][RTW89_CN][25] = 127, + [0][0][2][0][RTW89_UK][25] = 62, + [0][0][2][0][RTW89_FCC][27] = 78, + [0][0][2][0][RTW89_ETSI][27] = 62, + [0][0][2][0][RTW89_MKK][27] = 74, + [0][0][2][0][RTW89_IC][27] = 127, + [0][0][2][0][RTW89_KCC][27] = 74, + [0][0][2][0][RTW89_ACMA][27] = 127, + [0][0][2][0][RTW89_CN][27] = 127, + [0][0][2][0][RTW89_UK][27] = 62, + [0][0][2][0][RTW89_FCC][29] = 78, + [0][0][2][0][RTW89_ETSI][29] = 62, + [0][0][2][0][RTW89_MKK][29] = 74, + [0][0][2][0][RTW89_IC][29] = 127, + [0][0][2][0][RTW89_KCC][29] = 74, + [0][0][2][0][RTW89_ACMA][29] = 127, + [0][0][2][0][RTW89_CN][29] = 127, + [0][0][2][0][RTW89_UK][29] = 62, + [0][0][2][0][RTW89_FCC][31] = 78, + [0][0][2][0][RTW89_ETSI][31] = 62, + [0][0][2][0][RTW89_MKK][31] = 74, + [0][0][2][0][RTW89_IC][31] = 78, + [0][0][2][0][RTW89_KCC][31] = 74, + [0][0][2][0][RTW89_ACMA][31] = 62, + [0][0][2][0][RTW89_CN][31] = 127, + [0][0][2][0][RTW89_UK][31] = 62, + [0][0][2][0][RTW89_FCC][33] = 78, + [0][0][2][0][RTW89_ETSI][33] = 62, + [0][0][2][0][RTW89_MKK][33] = 74, + [0][0][2][0][RTW89_IC][33] = 78, + [0][0][2][0][RTW89_KCC][33] = 74, + [0][0][2][0][RTW89_ACMA][33] = 62, + [0][0][2][0][RTW89_CN][33] = 127, + [0][0][2][0][RTW89_UK][33] = 62, + [0][0][2][0][RTW89_FCC][35] = 72, + [0][0][2][0][RTW89_ETSI][35] = 62, + [0][0][2][0][RTW89_MKK][35] = 74, + [0][0][2][0][RTW89_IC][35] = 72, + [0][0][2][0][RTW89_KCC][35] = 74, + [0][0][2][0][RTW89_ACMA][35] = 62, + [0][0][2][0][RTW89_CN][35] = 127, + [0][0][2][0][RTW89_UK][35] = 62, + [0][0][2][0][RTW89_FCC][37] = 78, + [0][0][2][0][RTW89_ETSI][37] = 127, + [0][0][2][0][RTW89_MKK][37] = 74, + [0][0][2][0][RTW89_IC][37] = 78, + [0][0][2][0][RTW89_KCC][37] = 74, + [0][0][2][0][RTW89_ACMA][37] = 74, + [0][0][2][0][RTW89_CN][37] = 127, + [0][0][2][0][RTW89_UK][37] = 74, + [0][0][2][0][RTW89_FCC][38] = 78, + [0][0][2][0][RTW89_ETSI][38] = 30, + [0][0][2][0][RTW89_MKK][38] = 127, + [0][0][2][0][RTW89_IC][38] = 78, + [0][0][2][0][RTW89_KCC][38] = 66, + [0][0][2][0][RTW89_ACMA][38] = 74, + [0][0][2][0][RTW89_CN][38] = 74, + [0][0][2][0][RTW89_UK][38] = 60, + [0][0][2][0][RTW89_FCC][40] = 78, + [0][0][2][0][RTW89_ETSI][40] = 30, + [0][0][2][0][RTW89_MKK][40] = 127, + [0][0][2][0][RTW89_IC][40] = 78, + [0][0][2][0][RTW89_KCC][40] = 74, + [0][0][2][0][RTW89_ACMA][40] = 74, + [0][0][2][0][RTW89_CN][40] = 74, + [0][0][2][0][RTW89_UK][40] = 60, + [0][0][2][0][RTW89_FCC][42] = 78, + [0][0][2][0][RTW89_ETSI][42] = 30, + [0][0][2][0][RTW89_MKK][42] = 127, + [0][0][2][0][RTW89_IC][42] = 78, + [0][0][2][0][RTW89_KCC][42] = 74, + [0][0][2][0][RTW89_ACMA][42] = 74, + [0][0][2][0][RTW89_CN][42] = 74, + [0][0][2][0][RTW89_UK][42] = 60, + [0][0][2][0][RTW89_FCC][44] = 78, + [0][0][2][0][RTW89_ETSI][44] = 30, + [0][0][2][0][RTW89_MKK][44] = 127, + [0][0][2][0][RTW89_IC][44] = 78, + [0][0][2][0][RTW89_KCC][44] = 74, + [0][0][2][0][RTW89_ACMA][44] = 74, + [0][0][2][0][RTW89_CN][44] = 74, + [0][0][2][0][RTW89_UK][44] = 60, + [0][0][2][0][RTW89_FCC][46] = 78, + [0][0][2][0][RTW89_ETSI][46] = 30, + [0][0][2][0][RTW89_MKK][46] = 127, + [0][0][2][0][RTW89_IC][46] = 78, + [0][0][2][0][RTW89_KCC][46] = 74, + [0][0][2][0][RTW89_ACMA][46] = 74, + [0][0][2][0][RTW89_CN][46] = 74, + [0][0][2][0][RTW89_UK][46] = 60, + [0][0][2][0][RTW89_FCC][48] = 74, + [0][0][2][0][RTW89_ETSI][48] = 127, + [0][0][2][0][RTW89_MKK][48] = 127, + [0][0][2][0][RTW89_IC][48] = 127, + [0][0][2][0][RTW89_KCC][48] = 127, + [0][0][2][0][RTW89_ACMA][48] = 127, + [0][0][2][0][RTW89_CN][48] = 127, + [0][0][2][0][RTW89_UK][48] = 127, + [0][0][2][0][RTW89_FCC][50] = 74, + [0][0][2][0][RTW89_ETSI][50] = 127, + [0][0][2][0][RTW89_MKK][50] = 127, + [0][0][2][0][RTW89_IC][50] = 127, + [0][0][2][0][RTW89_KCC][50] = 127, + [0][0][2][0][RTW89_ACMA][50] = 127, + [0][0][2][0][RTW89_CN][50] = 127, + [0][0][2][0][RTW89_UK][50] = 127, + [0][0][2][0][RTW89_FCC][52] = 74, + [0][0][2][0][RTW89_ETSI][52] = 127, + [0][0][2][0][RTW89_MKK][52] = 127, + [0][0][2][0][RTW89_IC][52] = 127, + [0][0][2][0][RTW89_KCC][52] = 127, + [0][0][2][0][RTW89_ACMA][52] = 127, + [0][0][2][0][RTW89_CN][52] = 127, + [0][0][2][0][RTW89_UK][52] = 127, + [0][1][2][0][RTW89_FCC][0] = 127, + [0][1][2][0][RTW89_ETSI][0] = 127, + [0][1][2][0][RTW89_MKK][0] = 127, + [0][1][2][0][RTW89_IC][0] = 127, + [0][1][2][0][RTW89_KCC][0] = 127, + [0][1][2][0][RTW89_ACMA][0] = 127, + [0][1][2][0][RTW89_CN][0] = 127, + [0][1][2][0][RTW89_UK][0] = 127, + [0][1][2][0][RTW89_FCC][2] = 127, + [0][1][2][0][RTW89_ETSI][2] = 127, + [0][1][2][0][RTW89_MKK][2] = 127, + [0][1][2][0][RTW89_IC][2] = 127, + [0][1][2][0][RTW89_KCC][2] = 127, + [0][1][2][0][RTW89_ACMA][2] = 127, + [0][1][2][0][RTW89_CN][2] = 127, + [0][1][2][0][RTW89_UK][2] = 127, + [0][1][2][0][RTW89_FCC][4] = 127, + [0][1][2][0][RTW89_ETSI][4] = 127, + [0][1][2][0][RTW89_MKK][4] = 127, + [0][1][2][0][RTW89_IC][4] = 127, + [0][1][2][0][RTW89_KCC][4] = 127, + [0][1][2][0][RTW89_ACMA][4] = 127, + [0][1][2][0][RTW89_CN][4] = 127, + [0][1][2][0][RTW89_UK][4] = 127, + [0][1][2][0][RTW89_FCC][6] = 127, + [0][1][2][0][RTW89_ETSI][6] = 127, + [0][1][2][0][RTW89_MKK][6] = 127, + [0][1][2][0][RTW89_IC][6] = 127, + [0][1][2][0][RTW89_KCC][6] = 127, + [0][1][2][0][RTW89_ACMA][6] = 127, + [0][1][2][0][RTW89_CN][6] = 127, + [0][1][2][0][RTW89_UK][6] = 127, + [0][1][2][0][RTW89_FCC][8] = 127, + [0][1][2][0][RTW89_ETSI][8] = 127, + [0][1][2][0][RTW89_MKK][8] = 127, + [0][1][2][0][RTW89_IC][8] = 127, + [0][1][2][0][RTW89_KCC][8] = 127, + [0][1][2][0][RTW89_ACMA][8] = 127, + [0][1][2][0][RTW89_CN][8] = 127, + [0][1][2][0][RTW89_UK][8] = 127, + [0][1][2][0][RTW89_FCC][10] = 127, + [0][1][2][0][RTW89_ETSI][10] = 127, + [0][1][2][0][RTW89_MKK][10] = 127, + [0][1][2][0][RTW89_IC][10] = 127, + [0][1][2][0][RTW89_KCC][10] = 127, + [0][1][2][0][RTW89_ACMA][10] = 127, + [0][1][2][0][RTW89_CN][10] = 127, + [0][1][2][0][RTW89_UK][10] = 127, + [0][1][2][0][RTW89_FCC][12] = 127, + [0][1][2][0][RTW89_ETSI][12] = 127, + [0][1][2][0][RTW89_MKK][12] = 127, + [0][1][2][0][RTW89_IC][12] = 127, + [0][1][2][0][RTW89_KCC][12] = 127, + [0][1][2][0][RTW89_ACMA][12] = 127, + [0][1][2][0][RTW89_CN][12] = 127, + [0][1][2][0][RTW89_UK][12] = 127, + [0][1][2][0][RTW89_FCC][14] = 127, + [0][1][2][0][RTW89_ETSI][14] = 127, + [0][1][2][0][RTW89_MKK][14] = 127, + [0][1][2][0][RTW89_IC][14] = 127, + [0][1][2][0][RTW89_KCC][14] = 127, + [0][1][2][0][RTW89_ACMA][14] = 127, + [0][1][2][0][RTW89_CN][14] = 127, + [0][1][2][0][RTW89_UK][14] = 127, + [0][1][2][0][RTW89_FCC][15] = 127, + [0][1][2][0][RTW89_ETSI][15] = 127, + [0][1][2][0][RTW89_MKK][15] = 127, + [0][1][2][0][RTW89_IC][15] = 127, + [0][1][2][0][RTW89_KCC][15] = 127, + [0][1][2][0][RTW89_ACMA][15] = 127, + [0][1][2][0][RTW89_CN][15] = 127, + [0][1][2][0][RTW89_UK][15] = 127, + [0][1][2][0][RTW89_FCC][17] = 127, + [0][1][2][0][RTW89_ETSI][17] = 127, + [0][1][2][0][RTW89_MKK][17] = 127, + [0][1][2][0][RTW89_IC][17] = 127, + [0][1][2][0][RTW89_KCC][17] = 127, + [0][1][2][0][RTW89_ACMA][17] = 127, + [0][1][2][0][RTW89_CN][17] = 127, + [0][1][2][0][RTW89_UK][17] = 127, + [0][1][2][0][RTW89_FCC][19] = 127, + [0][1][2][0][RTW89_ETSI][19] = 127, + [0][1][2][0][RTW89_MKK][19] = 127, + [0][1][2][0][RTW89_IC][19] = 127, + [0][1][2][0][RTW89_KCC][19] = 127, + [0][1][2][0][RTW89_ACMA][19] = 127, + [0][1][2][0][RTW89_CN][19] = 127, + [0][1][2][0][RTW89_UK][19] = 127, + [0][1][2][0][RTW89_FCC][21] = 127, + [0][1][2][0][RTW89_ETSI][21] = 127, + [0][1][2][0][RTW89_MKK][21] = 127, + [0][1][2][0][RTW89_IC][21] = 127, + [0][1][2][0][RTW89_KCC][21] = 127, + [0][1][2][0][RTW89_ACMA][21] = 127, + [0][1][2][0][RTW89_CN][21] = 127, + [0][1][2][0][RTW89_UK][21] = 127, + [0][1][2][0][RTW89_FCC][23] = 127, + [0][1][2][0][RTW89_ETSI][23] = 127, + [0][1][2][0][RTW89_MKK][23] = 127, + [0][1][2][0][RTW89_IC][23] = 127, + [0][1][2][0][RTW89_KCC][23] = 127, + [0][1][2][0][RTW89_ACMA][23] = 127, + [0][1][2][0][RTW89_CN][23] = 127, + [0][1][2][0][RTW89_UK][23] = 127, + [0][1][2][0][RTW89_FCC][25] = 127, + [0][1][2][0][RTW89_ETSI][25] = 127, + [0][1][2][0][RTW89_MKK][25] = 127, + [0][1][2][0][RTW89_IC][25] = 127, + [0][1][2][0][RTW89_KCC][25] = 127, + [0][1][2][0][RTW89_ACMA][25] = 127, + [0][1][2][0][RTW89_CN][25] = 127, + [0][1][2][0][RTW89_UK][25] = 127, + [0][1][2][0][RTW89_FCC][27] = 127, + [0][1][2][0][RTW89_ETSI][27] = 127, + [0][1][2][0][RTW89_MKK][27] = 127, + [0][1][2][0][RTW89_IC][27] = 127, + [0][1][2][0][RTW89_KCC][27] = 127, + [0][1][2][0][RTW89_ACMA][27] = 127, + [0][1][2][0][RTW89_CN][27] = 127, + [0][1][2][0][RTW89_UK][27] = 127, + [0][1][2][0][RTW89_FCC][29] = 127, + [0][1][2][0][RTW89_ETSI][29] = 127, + [0][1][2][0][RTW89_MKK][29] = 127, + [0][1][2][0][RTW89_IC][29] = 127, + [0][1][2][0][RTW89_KCC][29] = 127, + [0][1][2][0][RTW89_ACMA][29] = 127, + [0][1][2][0][RTW89_CN][29] = 127, + [0][1][2][0][RTW89_UK][29] = 127, + [0][1][2][0][RTW89_FCC][31] = 127, + [0][1][2][0][RTW89_ETSI][31] = 127, + [0][1][2][0][RTW89_MKK][31] = 127, + [0][1][2][0][RTW89_IC][31] = 127, + [0][1][2][0][RTW89_KCC][31] = 127, + [0][1][2][0][RTW89_ACMA][31] = 127, + [0][1][2][0][RTW89_CN][31] = 127, + [0][1][2][0][RTW89_UK][31] = 127, + [0][1][2][0][RTW89_FCC][33] = 127, + [0][1][2][0][RTW89_ETSI][33] = 127, + [0][1][2][0][RTW89_MKK][33] = 127, + [0][1][2][0][RTW89_IC][33] = 127, + [0][1][2][0][RTW89_KCC][33] = 127, + [0][1][2][0][RTW89_ACMA][33] = 127, + [0][1][2][0][RTW89_CN][33] = 127, + [0][1][2][0][RTW89_UK][33] = 127, + [0][1][2][0][RTW89_FCC][35] = 127, + [0][1][2][0][RTW89_ETSI][35] = 127, + [0][1][2][0][RTW89_MKK][35] = 127, + [0][1][2][0][RTW89_IC][35] = 127, + [0][1][2][0][RTW89_KCC][35] = 127, + [0][1][2][0][RTW89_ACMA][35] = 127, + [0][1][2][0][RTW89_CN][35] = 127, + [0][1][2][0][RTW89_UK][35] = 127, + [0][1][2][0][RTW89_FCC][37] = 127, + [0][1][2][0][RTW89_ETSI][37] = 127, + [0][1][2][0][RTW89_MKK][37] = 127, + [0][1][2][0][RTW89_IC][37] = 127, + [0][1][2][0][RTW89_KCC][37] = 127, + [0][1][2][0][RTW89_ACMA][37] = 127, + [0][1][2][0][RTW89_CN][37] = 127, + [0][1][2][0][RTW89_UK][37] = 127, + [0][1][2][0][RTW89_FCC][38] = 127, + [0][1][2][0][RTW89_ETSI][38] = 127, + [0][1][2][0][RTW89_MKK][38] = 127, + [0][1][2][0][RTW89_IC][38] = 127, + [0][1][2][0][RTW89_KCC][38] = 127, + [0][1][2][0][RTW89_ACMA][38] = 127, + [0][1][2][0][RTW89_CN][38] = 127, + [0][1][2][0][RTW89_UK][38] = 127, + [0][1][2][0][RTW89_FCC][40] = 127, + [0][1][2][0][RTW89_ETSI][40] = 127, + [0][1][2][0][RTW89_MKK][40] = 127, + [0][1][2][0][RTW89_IC][40] = 127, + [0][1][2][0][RTW89_KCC][40] = 127, + [0][1][2][0][RTW89_ACMA][40] = 127, + [0][1][2][0][RTW89_CN][40] = 127, + [0][1][2][0][RTW89_UK][40] = 127, + [0][1][2][0][RTW89_FCC][42] = 127, + [0][1][2][0][RTW89_ETSI][42] = 127, + [0][1][2][0][RTW89_MKK][42] = 127, + [0][1][2][0][RTW89_IC][42] = 127, + [0][1][2][0][RTW89_KCC][42] = 127, + [0][1][2][0][RTW89_ACMA][42] = 127, + [0][1][2][0][RTW89_CN][42] = 127, + [0][1][2][0][RTW89_UK][42] = 127, + [0][1][2][0][RTW89_FCC][44] = 127, + [0][1][2][0][RTW89_ETSI][44] = 127, + [0][1][2][0][RTW89_MKK][44] = 127, + [0][1][2][0][RTW89_IC][44] = 127, + [0][1][2][0][RTW89_KCC][44] = 127, + [0][1][2][0][RTW89_ACMA][44] = 127, + [0][1][2][0][RTW89_CN][44] = 127, + [0][1][2][0][RTW89_UK][44] = 127, + [0][1][2][0][RTW89_FCC][46] = 127, + [0][1][2][0][RTW89_ETSI][46] = 127, + [0][1][2][0][RTW89_MKK][46] = 127, + [0][1][2][0][RTW89_IC][46] = 127, + [0][1][2][0][RTW89_KCC][46] = 127, + [0][1][2][0][RTW89_ACMA][46] = 127, + [0][1][2][0][RTW89_CN][46] = 127, + [0][1][2][0][RTW89_UK][46] = 127, + [0][1][2][0][RTW89_FCC][48] = 127, + [0][1][2][0][RTW89_ETSI][48] = 127, + [0][1][2][0][RTW89_MKK][48] = 127, + [0][1][2][0][RTW89_IC][48] = 127, + [0][1][2][0][RTW89_KCC][48] = 127, + [0][1][2][0][RTW89_ACMA][48] = 127, + [0][1][2][0][RTW89_CN][48] = 127, + [0][1][2][0][RTW89_UK][48] = 127, + [0][1][2][0][RTW89_FCC][50] = 127, + [0][1][2][0][RTW89_ETSI][50] = 127, + [0][1][2][0][RTW89_MKK][50] = 127, + [0][1][2][0][RTW89_IC][50] = 127, + [0][1][2][0][RTW89_KCC][50] = 127, + [0][1][2][0][RTW89_ACMA][50] = 127, + [0][1][2][0][RTW89_CN][50] = 127, + [0][1][2][0][RTW89_UK][50] = 127, + [0][1][2][0][RTW89_FCC][52] = 127, + [0][1][2][0][RTW89_ETSI][52] = 127, + [0][1][2][0][RTW89_MKK][52] = 127, + [0][1][2][0][RTW89_IC][52] = 127, + [0][1][2][0][RTW89_KCC][52] = 127, + [0][1][2][0][RTW89_ACMA][52] = 127, + [0][1][2][0][RTW89_CN][52] = 127, + [0][1][2][0][RTW89_UK][52] = 127, + [0][1][2][1][RTW89_FCC][0] = 127, + [0][1][2][1][RTW89_ETSI][0] = 127, + [0][1][2][1][RTW89_MKK][0] = 127, + [0][1][2][1][RTW89_IC][0] = 127, + [0][1][2][1][RTW89_KCC][0] = 127, + [0][1][2][1][RTW89_ACMA][0] = 127, + [0][1][2][1][RTW89_CN][0] = 127, + [0][1][2][1][RTW89_UK][0] = 127, + [0][1][2][1][RTW89_FCC][2] = 127, + [0][1][2][1][RTW89_ETSI][2] = 127, + [0][1][2][1][RTW89_MKK][2] = 127, + [0][1][2][1][RTW89_IC][2] = 127, + [0][1][2][1][RTW89_KCC][2] = 127, + [0][1][2][1][RTW89_ACMA][2] = 127, + [0][1][2][1][RTW89_CN][2] = 127, + [0][1][2][1][RTW89_UK][2] = 127, + [0][1][2][1][RTW89_FCC][4] = 127, + [0][1][2][1][RTW89_ETSI][4] = 127, + [0][1][2][1][RTW89_MKK][4] = 127, + [0][1][2][1][RTW89_IC][4] = 127, + [0][1][2][1][RTW89_KCC][4] = 127, + [0][1][2][1][RTW89_ACMA][4] = 127, + [0][1][2][1][RTW89_CN][4] = 127, + [0][1][2][1][RTW89_UK][4] = 127, + [0][1][2][1][RTW89_FCC][6] = 127, + [0][1][2][1][RTW89_ETSI][6] = 127, + [0][1][2][1][RTW89_MKK][6] = 127, + [0][1][2][1][RTW89_IC][6] = 127, + [0][1][2][1][RTW89_KCC][6] = 127, + [0][1][2][1][RTW89_ACMA][6] = 127, + [0][1][2][1][RTW89_CN][6] = 127, + [0][1][2][1][RTW89_UK][6] = 127, + [0][1][2][1][RTW89_FCC][8] = 127, + [0][1][2][1][RTW89_ETSI][8] = 127, + [0][1][2][1][RTW89_MKK][8] = 127, + [0][1][2][1][RTW89_IC][8] = 127, + [0][1][2][1][RTW89_KCC][8] = 127, + [0][1][2][1][RTW89_ACMA][8] = 127, + [0][1][2][1][RTW89_CN][8] = 127, + [0][1][2][1][RTW89_UK][8] = 127, + [0][1][2][1][RTW89_FCC][10] = 127, + [0][1][2][1][RTW89_ETSI][10] = 127, + [0][1][2][1][RTW89_MKK][10] = 127, + [0][1][2][1][RTW89_IC][10] = 127, + [0][1][2][1][RTW89_KCC][10] = 127, + [0][1][2][1][RTW89_ACMA][10] = 127, + [0][1][2][1][RTW89_CN][10] = 127, + [0][1][2][1][RTW89_UK][10] = 127, + [0][1][2][1][RTW89_FCC][12] = 127, + [0][1][2][1][RTW89_ETSI][12] = 127, + [0][1][2][1][RTW89_MKK][12] = 127, + [0][1][2][1][RTW89_IC][12] = 127, + [0][1][2][1][RTW89_KCC][12] = 127, + [0][1][2][1][RTW89_ACMA][12] = 127, + [0][1][2][1][RTW89_CN][12] = 127, + [0][1][2][1][RTW89_UK][12] = 127, + [0][1][2][1][RTW89_FCC][14] = 127, + [0][1][2][1][RTW89_ETSI][14] = 127, + [0][1][2][1][RTW89_MKK][14] = 127, + [0][1][2][1][RTW89_IC][14] = 127, + [0][1][2][1][RTW89_KCC][14] = 127, + [0][1][2][1][RTW89_ACMA][14] = 127, + [0][1][2][1][RTW89_CN][14] = 127, + [0][1][2][1][RTW89_UK][14] = 127, + [0][1][2][1][RTW89_FCC][15] = 127, + [0][1][2][1][RTW89_ETSI][15] = 127, + [0][1][2][1][RTW89_MKK][15] = 127, + [0][1][2][1][RTW89_IC][15] = 127, + [0][1][2][1][RTW89_KCC][15] = 127, + [0][1][2][1][RTW89_ACMA][15] = 127, + [0][1][2][1][RTW89_CN][15] = 127, + [0][1][2][1][RTW89_UK][15] = 127, + [0][1][2][1][RTW89_FCC][17] = 127, + [0][1][2][1][RTW89_ETSI][17] = 127, + [0][1][2][1][RTW89_MKK][17] = 127, + [0][1][2][1][RTW89_IC][17] = 127, + [0][1][2][1][RTW89_KCC][17] = 127, + [0][1][2][1][RTW89_ACMA][17] = 127, + [0][1][2][1][RTW89_CN][17] = 127, + [0][1][2][1][RTW89_UK][17] = 127, + [0][1][2][1][RTW89_FCC][19] = 127, + [0][1][2][1][RTW89_ETSI][19] = 127, + [0][1][2][1][RTW89_MKK][19] = 127, + [0][1][2][1][RTW89_IC][19] = 127, + [0][1][2][1][RTW89_KCC][19] = 127, + [0][1][2][1][RTW89_ACMA][19] = 127, + [0][1][2][1][RTW89_CN][19] = 127, + [0][1][2][1][RTW89_UK][19] = 127, + [0][1][2][1][RTW89_FCC][21] = 127, + [0][1][2][1][RTW89_ETSI][21] = 127, + [0][1][2][1][RTW89_MKK][21] = 127, + [0][1][2][1][RTW89_IC][21] = 127, + [0][1][2][1][RTW89_KCC][21] = 127, + [0][1][2][1][RTW89_ACMA][21] = 127, + [0][1][2][1][RTW89_CN][21] = 127, + [0][1][2][1][RTW89_UK][21] = 127, + [0][1][2][1][RTW89_FCC][23] = 127, + [0][1][2][1][RTW89_ETSI][23] = 127, + [0][1][2][1][RTW89_MKK][23] = 127, + [0][1][2][1][RTW89_IC][23] = 127, + [0][1][2][1][RTW89_KCC][23] = 127, + [0][1][2][1][RTW89_ACMA][23] = 127, + [0][1][2][1][RTW89_CN][23] = 127, + [0][1][2][1][RTW89_UK][23] = 127, + [0][1][2][1][RTW89_FCC][25] = 127, + [0][1][2][1][RTW89_ETSI][25] = 127, + [0][1][2][1][RTW89_MKK][25] = 127, + [0][1][2][1][RTW89_IC][25] = 127, + [0][1][2][1][RTW89_KCC][25] = 127, + [0][1][2][1][RTW89_ACMA][25] = 127, + [0][1][2][1][RTW89_CN][25] = 127, + [0][1][2][1][RTW89_UK][25] = 127, + [0][1][2][1][RTW89_FCC][27] = 127, + [0][1][2][1][RTW89_ETSI][27] = 127, + [0][1][2][1][RTW89_MKK][27] = 127, + [0][1][2][1][RTW89_IC][27] = 127, + [0][1][2][1][RTW89_KCC][27] = 127, + [0][1][2][1][RTW89_ACMA][27] = 127, + [0][1][2][1][RTW89_CN][27] = 127, + [0][1][2][1][RTW89_UK][27] = 127, + [0][1][2][1][RTW89_FCC][29] = 127, + [0][1][2][1][RTW89_ETSI][29] = 127, + [0][1][2][1][RTW89_MKK][29] = 127, + [0][1][2][1][RTW89_IC][29] = 127, + [0][1][2][1][RTW89_KCC][29] = 127, + [0][1][2][1][RTW89_ACMA][29] = 127, + [0][1][2][1][RTW89_CN][29] = 127, + [0][1][2][1][RTW89_UK][29] = 127, + [0][1][2][1][RTW89_FCC][31] = 127, + [0][1][2][1][RTW89_ETSI][31] = 127, + [0][1][2][1][RTW89_MKK][31] = 127, + [0][1][2][1][RTW89_IC][31] = 127, + [0][1][2][1][RTW89_KCC][31] = 127, + [0][1][2][1][RTW89_ACMA][31] = 127, + [0][1][2][1][RTW89_CN][31] = 127, + [0][1][2][1][RTW89_UK][31] = 127, + [0][1][2][1][RTW89_FCC][33] = 127, + [0][1][2][1][RTW89_ETSI][33] = 127, + [0][1][2][1][RTW89_MKK][33] = 127, + [0][1][2][1][RTW89_IC][33] = 127, + [0][1][2][1][RTW89_KCC][33] = 127, + [0][1][2][1][RTW89_ACMA][33] = 127, + [0][1][2][1][RTW89_CN][33] = 127, + [0][1][2][1][RTW89_UK][33] = 127, + [0][1][2][1][RTW89_FCC][35] = 127, + [0][1][2][1][RTW89_ETSI][35] = 127, + [0][1][2][1][RTW89_MKK][35] = 127, + [0][1][2][1][RTW89_IC][35] = 127, + [0][1][2][1][RTW89_KCC][35] = 127, + [0][1][2][1][RTW89_ACMA][35] = 127, + [0][1][2][1][RTW89_CN][35] = 127, + [0][1][2][1][RTW89_UK][35] = 127, + [0][1][2][1][RTW89_FCC][37] = 127, + [0][1][2][1][RTW89_ETSI][37] = 127, + [0][1][2][1][RTW89_MKK][37] = 127, + [0][1][2][1][RTW89_IC][37] = 127, + [0][1][2][1][RTW89_KCC][37] = 127, + [0][1][2][1][RTW89_ACMA][37] = 127, + [0][1][2][1][RTW89_CN][37] = 127, + [0][1][2][1][RTW89_UK][37] = 127, + [0][1][2][1][RTW89_FCC][38] = 127, + [0][1][2][1][RTW89_ETSI][38] = 127, + [0][1][2][1][RTW89_MKK][38] = 127, + [0][1][2][1][RTW89_IC][38] = 127, + [0][1][2][1][RTW89_KCC][38] = 127, + [0][1][2][1][RTW89_ACMA][38] = 127, + [0][1][2][1][RTW89_CN][38] = 127, + [0][1][2][1][RTW89_UK][38] = 127, + [0][1][2][1][RTW89_FCC][40] = 127, + [0][1][2][1][RTW89_ETSI][40] = 127, + [0][1][2][1][RTW89_MKK][40] = 127, + [0][1][2][1][RTW89_IC][40] = 127, + [0][1][2][1][RTW89_KCC][40] = 127, + [0][1][2][1][RTW89_ACMA][40] = 127, + [0][1][2][1][RTW89_CN][40] = 127, + [0][1][2][1][RTW89_UK][40] = 127, + [0][1][2][1][RTW89_FCC][42] = 127, + [0][1][2][1][RTW89_ETSI][42] = 127, + [0][1][2][1][RTW89_MKK][42] = 127, + [0][1][2][1][RTW89_IC][42] = 127, + [0][1][2][1][RTW89_KCC][42] = 127, + [0][1][2][1][RTW89_ACMA][42] = 127, + [0][1][2][1][RTW89_CN][42] = 127, + [0][1][2][1][RTW89_UK][42] = 127, + [0][1][2][1][RTW89_FCC][44] = 127, + [0][1][2][1][RTW89_ETSI][44] = 127, + [0][1][2][1][RTW89_MKK][44] = 127, + [0][1][2][1][RTW89_IC][44] = 127, + [0][1][2][1][RTW89_KCC][44] = 127, + [0][1][2][1][RTW89_ACMA][44] = 127, + [0][1][2][1][RTW89_CN][44] = 127, + [0][1][2][1][RTW89_UK][44] = 127, + [0][1][2][1][RTW89_FCC][46] = 127, + [0][1][2][1][RTW89_ETSI][46] = 127, + [0][1][2][1][RTW89_MKK][46] = 127, + [0][1][2][1][RTW89_IC][46] = 127, + [0][1][2][1][RTW89_KCC][46] = 127, + [0][1][2][1][RTW89_ACMA][46] = 127, + [0][1][2][1][RTW89_CN][46] = 127, + [0][1][2][1][RTW89_UK][46] = 127, + [0][1][2][1][RTW89_FCC][48] = 127, + [0][1][2][1][RTW89_ETSI][48] = 127, + [0][1][2][1][RTW89_MKK][48] = 127, + [0][1][2][1][RTW89_IC][48] = 127, + [0][1][2][1][RTW89_KCC][48] = 127, + [0][1][2][1][RTW89_ACMA][48] = 127, + [0][1][2][1][RTW89_CN][48] = 127, + [0][1][2][1][RTW89_UK][48] = 127, + [0][1][2][1][RTW89_FCC][50] = 127, + [0][1][2][1][RTW89_ETSI][50] = 127, + [0][1][2][1][RTW89_MKK][50] = 127, + [0][1][2][1][RTW89_IC][50] = 127, + [0][1][2][1][RTW89_KCC][50] = 127, + [0][1][2][1][RTW89_ACMA][50] = 127, + [0][1][2][1][RTW89_CN][50] = 127, + [0][1][2][1][RTW89_UK][50] = 127, + [0][1][2][1][RTW89_FCC][52] = 127, + [0][1][2][1][RTW89_ETSI][52] = 127, + [0][1][2][1][RTW89_MKK][52] = 127, + [0][1][2][1][RTW89_IC][52] = 127, + [0][1][2][1][RTW89_KCC][52] = 127, + [0][1][2][1][RTW89_ACMA][52] = 127, + [0][1][2][1][RTW89_CN][52] = 127, + [0][1][2][1][RTW89_UK][52] = 127, + [1][0][2][0][RTW89_FCC][1] = 66, + [1][0][2][0][RTW89_ETSI][1] = 64, + [1][0][2][0][RTW89_MKK][1] = 64, + [1][0][2][0][RTW89_IC][1] = 64, + [1][0][2][0][RTW89_KCC][1] = 74, + [1][0][2][0][RTW89_ACMA][1] = 64, + [1][0][2][0][RTW89_CN][1] = 64, + [1][0][2][0][RTW89_UK][1] = 64, + [1][0][2][0][RTW89_FCC][5] = 80, + [1][0][2][0][RTW89_ETSI][5] = 64, + [1][0][2][0][RTW89_MKK][5] = 62, + [1][0][2][0][RTW89_IC][5] = 64, + [1][0][2][0][RTW89_KCC][5] = 66, + [1][0][2][0][RTW89_ACMA][5] = 64, + [1][0][2][0][RTW89_CN][5] = 64, + [1][0][2][0][RTW89_UK][5] = 64, + [1][0][2][0][RTW89_FCC][9] = 80, + [1][0][2][0][RTW89_ETSI][9] = 64, + [1][0][2][0][RTW89_MKK][9] = 64, + [1][0][2][0][RTW89_IC][9] = 64, + [1][0][2][0][RTW89_KCC][9] = 76, + [1][0][2][0][RTW89_ACMA][9] = 64, + [1][0][2][0][RTW89_CN][9] = 64, + [1][0][2][0][RTW89_UK][9] = 64, + [1][0][2][0][RTW89_FCC][13] = 64, + [1][0][2][0][RTW89_ETSI][13] = 64, + [1][0][2][0][RTW89_MKK][13] = 64, + [1][0][2][0][RTW89_IC][13] = 64, + [1][0][2][0][RTW89_KCC][13] = 72, + [1][0][2][0][RTW89_ACMA][13] = 64, + [1][0][2][0][RTW89_CN][13] = 64, + [1][0][2][0][RTW89_UK][13] = 64, + [1][0][2][0][RTW89_FCC][16] = 66, + [1][0][2][0][RTW89_ETSI][16] = 66, + [1][0][2][0][RTW89_MKK][16] = 76, + [1][0][2][0][RTW89_IC][16] = 66, + [1][0][2][0][RTW89_KCC][16] = 74, + [1][0][2][0][RTW89_ACMA][16] = 66, + [1][0][2][0][RTW89_CN][16] = 127, + [1][0][2][0][RTW89_UK][16] = 66, + [1][0][2][0][RTW89_FCC][20] = 80, + [1][0][2][0][RTW89_ETSI][20] = 66, + [1][0][2][0][RTW89_MKK][20] = 76, + [1][0][2][0][RTW89_IC][20] = 80, + [1][0][2][0][RTW89_KCC][20] = 74, + [1][0][2][0][RTW89_ACMA][20] = 66, + [1][0][2][0][RTW89_CN][20] = 127, + [1][0][2][0][RTW89_UK][20] = 66, + [1][0][2][0][RTW89_FCC][24] = 80, + [1][0][2][0][RTW89_ETSI][24] = 66, + [1][0][2][0][RTW89_MKK][24] = 76, + [1][0][2][0][RTW89_IC][24] = 127, + [1][0][2][0][RTW89_KCC][24] = 74, + [1][0][2][0][RTW89_ACMA][24] = 127, + [1][0][2][0][RTW89_CN][24] = 127, + [1][0][2][0][RTW89_UK][24] = 66, + [1][0][2][0][RTW89_FCC][28] = 80, + [1][0][2][0][RTW89_ETSI][28] = 66, + [1][0][2][0][RTW89_MKK][28] = 76, + [1][0][2][0][RTW89_IC][28] = 127, + [1][0][2][0][RTW89_KCC][28] = 74, + [1][0][2][0][RTW89_ACMA][28] = 127, + [1][0][2][0][RTW89_CN][28] = 127, + [1][0][2][0][RTW89_UK][28] = 66, + [1][0][2][0][RTW89_FCC][32] = 74, + [1][0][2][0][RTW89_ETSI][32] = 66, + [1][0][2][0][RTW89_MKK][32] = 76, + [1][0][2][0][RTW89_IC][32] = 74, + [1][0][2][0][RTW89_KCC][32] = 76, + [1][0][2][0][RTW89_ACMA][32] = 66, + [1][0][2][0][RTW89_CN][32] = 127, + [1][0][2][0][RTW89_UK][32] = 66, + [1][0][2][0][RTW89_FCC][36] = 78, + [1][0][2][0][RTW89_ETSI][36] = 127, + [1][0][2][0][RTW89_MKK][36] = 76, + [1][0][2][0][RTW89_IC][36] = 78, + [1][0][2][0][RTW89_KCC][36] = 76, + [1][0][2][0][RTW89_ACMA][36] = 76, + [1][0][2][0][RTW89_CN][36] = 127, + [1][0][2][0][RTW89_UK][36] = 76, + [1][0][2][0][RTW89_FCC][39] = 80, + [1][0][2][0][RTW89_ETSI][39] = 30, + [1][0][2][0][RTW89_MKK][39] = 127, + [1][0][2][0][RTW89_IC][39] = 80, + [1][0][2][0][RTW89_KCC][39] = 68, + [1][0][2][0][RTW89_ACMA][39] = 76, + [1][0][2][0][RTW89_CN][39] = 70, + [1][0][2][0][RTW89_UK][39] = 64, + [1][0][2][0][RTW89_FCC][43] = 80, + [1][0][2][0][RTW89_ETSI][43] = 30, + [1][0][2][0][RTW89_MKK][43] = 127, + [1][0][2][0][RTW89_IC][43] = 80, + [1][0][2][0][RTW89_KCC][43] = 76, + [1][0][2][0][RTW89_ACMA][43] = 76, + [1][0][2][0][RTW89_CN][43] = 76, + [1][0][2][0][RTW89_UK][43] = 64, + [1][0][2][0][RTW89_FCC][47] = 80, + [1][0][2][0][RTW89_ETSI][47] = 127, + [1][0][2][0][RTW89_MKK][47] = 127, + [1][0][2][0][RTW89_IC][47] = 127, + [1][0][2][0][RTW89_KCC][47] = 127, + [1][0][2][0][RTW89_ACMA][47] = 127, + [1][0][2][0][RTW89_CN][47] = 127, + [1][0][2][0][RTW89_UK][47] = 127, + [1][0][2][0][RTW89_FCC][51] = 80, + [1][0][2][0][RTW89_ETSI][51] = 127, + [1][0][2][0][RTW89_MKK][51] = 127, + [1][0][2][0][RTW89_IC][51] = 127, + [1][0][2][0][RTW89_KCC][51] = 127, + [1][0][2][0][RTW89_ACMA][51] = 127, + [1][0][2][0][RTW89_CN][51] = 127, + [1][0][2][0][RTW89_UK][51] = 127, + [1][1][2][0][RTW89_FCC][1] = 127, + [1][1][2][0][RTW89_ETSI][1] = 127, + [1][1][2][0][RTW89_MKK][1] = 127, + [1][1][2][0][RTW89_IC][1] = 127, + [1][1][2][0][RTW89_KCC][1] = 127, + [1][1][2][0][RTW89_ACMA][1] = 127, + [1][1][2][0][RTW89_CN][1] = 127, + [1][1][2][0][RTW89_UK][1] = 127, + [1][1][2][0][RTW89_FCC][5] = 127, + [1][1][2][0][RTW89_ETSI][5] = 127, + [1][1][2][0][RTW89_MKK][5] = 127, + [1][1][2][0][RTW89_IC][5] = 127, + [1][1][2][0][RTW89_KCC][5] = 127, + [1][1][2][0][RTW89_ACMA][5] = 127, + [1][1][2][0][RTW89_CN][5] = 127, + [1][1][2][0][RTW89_UK][5] = 127, + [1][1][2][0][RTW89_FCC][9] = 127, + [1][1][2][0][RTW89_ETSI][9] = 127, + [1][1][2][0][RTW89_MKK][9] = 127, + [1][1][2][0][RTW89_IC][9] = 127, + [1][1][2][0][RTW89_KCC][9] = 127, + [1][1][2][0][RTW89_ACMA][9] = 127, + [1][1][2][0][RTW89_CN][9] = 127, + [1][1][2][0][RTW89_UK][9] = 127, + [1][1][2][0][RTW89_FCC][13] = 127, + [1][1][2][0][RTW89_ETSI][13] = 127, + [1][1][2][0][RTW89_MKK][13] = 127, + [1][1][2][0][RTW89_IC][13] = 127, + [1][1][2][0][RTW89_KCC][13] = 127, + [1][1][2][0][RTW89_ACMA][13] = 127, + [1][1][2][0][RTW89_CN][13] = 127, + [1][1][2][0][RTW89_UK][13] = 127, + [1][1][2][0][RTW89_FCC][16] = 127, + [1][1][2][0][RTW89_ETSI][16] = 127, + [1][1][2][0][RTW89_MKK][16] = 127, + [1][1][2][0][RTW89_IC][16] = 127, + [1][1][2][0][RTW89_KCC][16] = 127, + [1][1][2][0][RTW89_ACMA][16] = 127, + [1][1][2][0][RTW89_CN][16] = 127, + [1][1][2][0][RTW89_UK][16] = 127, + [1][1][2][0][RTW89_FCC][20] = 127, + [1][1][2][0][RTW89_ETSI][20] = 127, + [1][1][2][0][RTW89_MKK][20] = 127, + [1][1][2][0][RTW89_IC][20] = 127, + [1][1][2][0][RTW89_KCC][20] = 127, + [1][1][2][0][RTW89_ACMA][20] = 127, + [1][1][2][0][RTW89_CN][20] = 127, + [1][1][2][0][RTW89_UK][20] = 127, + [1][1][2][0][RTW89_FCC][24] = 127, + [1][1][2][0][RTW89_ETSI][24] = 127, + [1][1][2][0][RTW89_MKK][24] = 127, + [1][1][2][0][RTW89_IC][24] = 127, + [1][1][2][0][RTW89_KCC][24] = 127, + [1][1][2][0][RTW89_ACMA][24] = 127, + [1][1][2][0][RTW89_CN][24] = 127, + [1][1][2][0][RTW89_UK][24] = 127, + [1][1][2][0][RTW89_FCC][28] = 127, + [1][1][2][0][RTW89_ETSI][28] = 127, + [1][1][2][0][RTW89_MKK][28] = 127, + [1][1][2][0][RTW89_IC][28] = 127, + [1][1][2][0][RTW89_KCC][28] = 127, + [1][1][2][0][RTW89_ACMA][28] = 127, + [1][1][2][0][RTW89_CN][28] = 127, + [1][1][2][0][RTW89_UK][28] = 127, + [1][1][2][0][RTW89_FCC][32] = 127, + [1][1][2][0][RTW89_ETSI][32] = 127, + [1][1][2][0][RTW89_MKK][32] = 127, + [1][1][2][0][RTW89_IC][32] = 127, + [1][1][2][0][RTW89_KCC][32] = 127, + [1][1][2][0][RTW89_ACMA][32] = 127, + [1][1][2][0][RTW89_CN][32] = 127, + [1][1][2][0][RTW89_UK][32] = 127, + [1][1][2][0][RTW89_FCC][36] = 127, + [1][1][2][0][RTW89_ETSI][36] = 127, + [1][1][2][0][RTW89_MKK][36] = 127, + [1][1][2][0][RTW89_IC][36] = 127, + [1][1][2][0][RTW89_KCC][36] = 127, + [1][1][2][0][RTW89_ACMA][36] = 127, + [1][1][2][0][RTW89_CN][36] = 127, + [1][1][2][0][RTW89_UK][36] = 127, + [1][1][2][0][RTW89_FCC][39] = 127, + [1][1][2][0][RTW89_ETSI][39] = 127, + [1][1][2][0][RTW89_MKK][39] = 127, + [1][1][2][0][RTW89_IC][39] = 127, + [1][1][2][0][RTW89_KCC][39] = 127, + [1][1][2][0][RTW89_ACMA][39] = 127, + [1][1][2][0][RTW89_CN][39] = 127, + [1][1][2][0][RTW89_UK][39] = 127, + [1][1][2][0][RTW89_FCC][43] = 127, + [1][1][2][0][RTW89_ETSI][43] = 127, + [1][1][2][0][RTW89_MKK][43] = 127, + [1][1][2][0][RTW89_IC][43] = 127, + [1][1][2][0][RTW89_KCC][43] = 127, + [1][1][2][0][RTW89_ACMA][43] = 127, + [1][1][2][0][RTW89_CN][43] = 127, + [1][1][2][0][RTW89_UK][43] = 127, + [1][1][2][0][RTW89_FCC][47] = 127, + [1][1][2][0][RTW89_ETSI][47] = 127, + [1][1][2][0][RTW89_MKK][47] = 127, + [1][1][2][0][RTW89_IC][47] = 127, + [1][1][2][0][RTW89_KCC][47] = 127, + [1][1][2][0][RTW89_ACMA][47] = 127, + [1][1][2][0][RTW89_CN][47] = 127, + [1][1][2][0][RTW89_UK][47] = 127, + [1][1][2][0][RTW89_FCC][51] = 127, + [1][1][2][0][RTW89_ETSI][51] = 127, + [1][1][2][0][RTW89_MKK][51] = 127, + [1][1][2][0][RTW89_IC][51] = 127, + [1][1][2][0][RTW89_KCC][51] = 127, + [1][1][2][0][RTW89_ACMA][51] = 127, + [1][1][2][0][RTW89_CN][51] = 127, + [1][1][2][0][RTW89_UK][51] = 127, + [1][1][2][1][RTW89_FCC][1] = 127, + [1][1][2][1][RTW89_ETSI][1] = 127, + [1][1][2][1][RTW89_MKK][1] = 127, + [1][1][2][1][RTW89_IC][1] = 127, + [1][1][2][1][RTW89_KCC][1] = 127, + [1][1][2][1][RTW89_ACMA][1] = 127, + [1][1][2][1][RTW89_CN][1] = 127, + [1][1][2][1][RTW89_UK][1] = 127, + [1][1][2][1][RTW89_FCC][5] = 127, + [1][1][2][1][RTW89_ETSI][5] = 127, + [1][1][2][1][RTW89_MKK][5] = 127, + [1][1][2][1][RTW89_IC][5] = 127, + [1][1][2][1][RTW89_KCC][5] = 127, + [1][1][2][1][RTW89_ACMA][5] = 127, + [1][1][2][1][RTW89_CN][5] = 127, + [1][1][2][1][RTW89_UK][5] = 127, + [1][1][2][1][RTW89_FCC][9] = 127, + [1][1][2][1][RTW89_ETSI][9] = 127, + [1][1][2][1][RTW89_MKK][9] = 127, + [1][1][2][1][RTW89_IC][9] = 127, + [1][1][2][1][RTW89_KCC][9] = 127, + [1][1][2][1][RTW89_ACMA][9] = 127, + [1][1][2][1][RTW89_CN][9] = 127, + [1][1][2][1][RTW89_UK][9] = 127, + [1][1][2][1][RTW89_FCC][13] = 127, + [1][1][2][1][RTW89_ETSI][13] = 127, + [1][1][2][1][RTW89_MKK][13] = 127, + [1][1][2][1][RTW89_IC][13] = 127, + [1][1][2][1][RTW89_KCC][13] = 127, + [1][1][2][1][RTW89_ACMA][13] = 127, + [1][1][2][1][RTW89_CN][13] = 127, + [1][1][2][1][RTW89_UK][13] = 127, + [1][1][2][1][RTW89_FCC][16] = 127, + [1][1][2][1][RTW89_ETSI][16] = 127, + [1][1][2][1][RTW89_MKK][16] = 127, + [1][1][2][1][RTW89_IC][16] = 127, + [1][1][2][1][RTW89_KCC][16] = 127, + [1][1][2][1][RTW89_ACMA][16] = 127, + [1][1][2][1][RTW89_CN][16] = 127, + [1][1][2][1][RTW89_UK][16] = 127, + [1][1][2][1][RTW89_FCC][20] = 127, + [1][1][2][1][RTW89_ETSI][20] = 127, + [1][1][2][1][RTW89_MKK][20] = 127, + [1][1][2][1][RTW89_IC][20] = 127, + [1][1][2][1][RTW89_KCC][20] = 127, + [1][1][2][1][RTW89_ACMA][20] = 127, + [1][1][2][1][RTW89_CN][20] = 127, + [1][1][2][1][RTW89_UK][20] = 127, + [1][1][2][1][RTW89_FCC][24] = 127, + [1][1][2][1][RTW89_ETSI][24] = 127, + [1][1][2][1][RTW89_MKK][24] = 127, + [1][1][2][1][RTW89_IC][24] = 127, + [1][1][2][1][RTW89_KCC][24] = 127, + [1][1][2][1][RTW89_ACMA][24] = 127, + [1][1][2][1][RTW89_CN][24] = 127, + [1][1][2][1][RTW89_UK][24] = 127, + [1][1][2][1][RTW89_FCC][28] = 127, + [1][1][2][1][RTW89_ETSI][28] = 127, + [1][1][2][1][RTW89_MKK][28] = 127, + [1][1][2][1][RTW89_IC][28] = 127, + [1][1][2][1][RTW89_KCC][28] = 127, + [1][1][2][1][RTW89_ACMA][28] = 127, + [1][1][2][1][RTW89_CN][28] = 127, + [1][1][2][1][RTW89_UK][28] = 127, + [1][1][2][1][RTW89_FCC][32] = 127, + [1][1][2][1][RTW89_ETSI][32] = 127, + [1][1][2][1][RTW89_MKK][32] = 127, + [1][1][2][1][RTW89_IC][32] = 127, + [1][1][2][1][RTW89_KCC][32] = 127, + [1][1][2][1][RTW89_ACMA][32] = 127, + [1][1][2][1][RTW89_CN][32] = 127, + [1][1][2][1][RTW89_UK][32] = 127, + [1][1][2][1][RTW89_FCC][36] = 127, + [1][1][2][1][RTW89_ETSI][36] = 127, + [1][1][2][1][RTW89_MKK][36] = 127, + [1][1][2][1][RTW89_IC][36] = 127, + [1][1][2][1][RTW89_KCC][36] = 127, + [1][1][2][1][RTW89_ACMA][36] = 127, + [1][1][2][1][RTW89_CN][36] = 127, + [1][1][2][1][RTW89_UK][36] = 127, + [1][1][2][1][RTW89_FCC][39] = 127, + [1][1][2][1][RTW89_ETSI][39] = 127, + [1][1][2][1][RTW89_MKK][39] = 127, + [1][1][2][1][RTW89_IC][39] = 127, + [1][1][2][1][RTW89_KCC][39] = 127, + [1][1][2][1][RTW89_ACMA][39] = 127, + [1][1][2][1][RTW89_CN][39] = 127, + [1][1][2][1][RTW89_UK][39] = 127, + [1][1][2][1][RTW89_FCC][43] = 127, + [1][1][2][1][RTW89_ETSI][43] = 127, + [1][1][2][1][RTW89_MKK][43] = 127, + [1][1][2][1][RTW89_IC][43] = 127, + [1][1][2][1][RTW89_KCC][43] = 127, + [1][1][2][1][RTW89_ACMA][43] = 127, + [1][1][2][1][RTW89_CN][43] = 127, + [1][1][2][1][RTW89_UK][43] = 127, + [1][1][2][1][RTW89_FCC][47] = 127, + [1][1][2][1][RTW89_ETSI][47] = 127, + [1][1][2][1][RTW89_MKK][47] = 127, + [1][1][2][1][RTW89_IC][47] = 127, + [1][1][2][1][RTW89_KCC][47] = 127, + [1][1][2][1][RTW89_ACMA][47] = 127, + [1][1][2][1][RTW89_CN][47] = 127, + [1][1][2][1][RTW89_UK][47] = 127, + [1][1][2][1][RTW89_FCC][51] = 127, + [1][1][2][1][RTW89_ETSI][51] = 127, + [1][1][2][1][RTW89_MKK][51] = 127, + [1][1][2][1][RTW89_IC][51] = 127, + [1][1][2][1][RTW89_KCC][51] = 127, + [1][1][2][1][RTW89_ACMA][51] = 127, + [1][1][2][1][RTW89_CN][51] = 127, + [1][1][2][1][RTW89_UK][51] = 127, + [2][0][2][0][RTW89_FCC][3] = 72, + [2][0][2][0][RTW89_ETSI][3] = 64, + [2][0][2][0][RTW89_MKK][3] = 62, + [2][0][2][0][RTW89_IC][3] = 64, + [2][0][2][0][RTW89_KCC][3] = 68, + [2][0][2][0][RTW89_ACMA][3] = 64, + [2][0][2][0][RTW89_CN][3] = 64, + [2][0][2][0][RTW89_UK][3] = 64, + [2][0][2][0][RTW89_FCC][11] = 62, + [2][0][2][0][RTW89_ETSI][11] = 64, + [2][0][2][0][RTW89_MKK][11] = 64, + [2][0][2][0][RTW89_IC][11] = 62, + [2][0][2][0][RTW89_KCC][11] = 68, + [2][0][2][0][RTW89_ACMA][11] = 64, + [2][0][2][0][RTW89_CN][11] = 64, + [2][0][2][0][RTW89_UK][11] = 64, + [2][0][2][0][RTW89_FCC][18] = 66, + [2][0][2][0][RTW89_ETSI][18] = 64, + [2][0][2][0][RTW89_MKK][18] = 68, + [2][0][2][0][RTW89_IC][18] = 66, + [2][0][2][0][RTW89_KCC][18] = 68, + [2][0][2][0][RTW89_ACMA][18] = 64, + [2][0][2][0][RTW89_CN][18] = 127, + [2][0][2][0][RTW89_UK][18] = 64, + [2][0][2][0][RTW89_FCC][26] = 72, + [2][0][2][0][RTW89_ETSI][26] = 64, + [2][0][2][0][RTW89_MKK][26] = 68, + [2][0][2][0][RTW89_IC][26] = 127, + [2][0][2][0][RTW89_KCC][26] = 68, + [2][0][2][0][RTW89_ACMA][26] = 127, + [2][0][2][0][RTW89_CN][26] = 127, + [2][0][2][0][RTW89_UK][26] = 64, + [2][0][2][0][RTW89_FCC][34] = 72, + [2][0][2][0][RTW89_ETSI][34] = 127, + [2][0][2][0][RTW89_MKK][34] = 68, + [2][0][2][0][RTW89_IC][34] = 72, + [2][0][2][0][RTW89_KCC][34] = 68, + [2][0][2][0][RTW89_ACMA][34] = 68, + [2][0][2][0][RTW89_CN][34] = 127, + [2][0][2][0][RTW89_UK][34] = 68, + [2][0][2][0][RTW89_FCC][41] = 72, + [2][0][2][0][RTW89_ETSI][41] = 30, + [2][0][2][0][RTW89_MKK][41] = 127, + [2][0][2][0][RTW89_IC][41] = 72, + [2][0][2][0][RTW89_KCC][41] = 64, + [2][0][2][0][RTW89_ACMA][41] = 68, + [2][0][2][0][RTW89_CN][41] = 68, + [2][0][2][0][RTW89_UK][41] = 64, + [2][0][2][0][RTW89_FCC][49] = 72, + [2][0][2][0][RTW89_ETSI][49] = 127, + [2][0][2][0][RTW89_MKK][49] = 127, + [2][0][2][0][RTW89_IC][49] = 127, + [2][0][2][0][RTW89_KCC][49] = 127, + [2][0][2][0][RTW89_ACMA][49] = 127, + [2][0][2][0][RTW89_CN][49] = 127, + [2][0][2][0][RTW89_UK][49] = 127, + [2][1][2][0][RTW89_FCC][3] = 127, + [2][1][2][0][RTW89_ETSI][3] = 127, + [2][1][2][0][RTW89_MKK][3] = 127, + [2][1][2][0][RTW89_IC][3] = 127, + [2][1][2][0][RTW89_KCC][3] = 127, + [2][1][2][0][RTW89_ACMA][3] = 127, + [2][1][2][0][RTW89_CN][3] = 127, + [2][1][2][0][RTW89_UK][3] = 127, + [2][1][2][0][RTW89_FCC][11] = 127, + [2][1][2][0][RTW89_ETSI][11] = 127, + [2][1][2][0][RTW89_MKK][11] = 127, + [2][1][2][0][RTW89_IC][11] = 127, + [2][1][2][0][RTW89_KCC][11] = 127, + [2][1][2][0][RTW89_ACMA][11] = 127, + [2][1][2][0][RTW89_CN][11] = 127, + [2][1][2][0][RTW89_UK][11] = 127, + [2][1][2][0][RTW89_FCC][18] = 127, + [2][1][2][0][RTW89_ETSI][18] = 127, + [2][1][2][0][RTW89_MKK][18] = 127, + [2][1][2][0][RTW89_IC][18] = 127, + [2][1][2][0][RTW89_KCC][18] = 127, + [2][1][2][0][RTW89_ACMA][18] = 127, + [2][1][2][0][RTW89_CN][18] = 127, + [2][1][2][0][RTW89_UK][18] = 127, + [2][1][2][0][RTW89_FCC][26] = 127, + [2][1][2][0][RTW89_ETSI][26] = 127, + [2][1][2][0][RTW89_MKK][26] = 127, + [2][1][2][0][RTW89_IC][26] = 127, + [2][1][2][0][RTW89_KCC][26] = 127, + [2][1][2][0][RTW89_ACMA][26] = 127, + [2][1][2][0][RTW89_CN][26] = 127, + [2][1][2][0][RTW89_UK][26] = 127, + [2][1][2][0][RTW89_FCC][34] = 127, + [2][1][2][0][RTW89_ETSI][34] = 127, + [2][1][2][0][RTW89_MKK][34] = 127, + [2][1][2][0][RTW89_IC][34] = 127, + [2][1][2][0][RTW89_KCC][34] = 127, + [2][1][2][0][RTW89_ACMA][34] = 127, + [2][1][2][0][RTW89_CN][34] = 127, + [2][1][2][0][RTW89_UK][34] = 127, + [2][1][2][0][RTW89_FCC][41] = 127, + [2][1][2][0][RTW89_ETSI][41] = 127, + [2][1][2][0][RTW89_MKK][41] = 127, + [2][1][2][0][RTW89_IC][41] = 127, + [2][1][2][0][RTW89_KCC][41] = 127, + [2][1][2][0][RTW89_ACMA][41] = 127, + [2][1][2][0][RTW89_CN][41] = 127, + [2][1][2][0][RTW89_UK][41] = 127, + [2][1][2][0][RTW89_FCC][49] = 127, + [2][1][2][0][RTW89_ETSI][49] = 127, + [2][1][2][0][RTW89_MKK][49] = 127, + [2][1][2][0][RTW89_IC][49] = 127, + [2][1][2][0][RTW89_KCC][49] = 127, + [2][1][2][0][RTW89_ACMA][49] = 127, + [2][1][2][0][RTW89_CN][49] = 127, + [2][1][2][0][RTW89_UK][49] = 127, + [2][1][2][1][RTW89_FCC][3] = 127, + [2][1][2][1][RTW89_ETSI][3] = 127, + [2][1][2][1][RTW89_MKK][3] = 127, + [2][1][2][1][RTW89_IC][3] = 127, + [2][1][2][1][RTW89_KCC][3] = 127, + [2][1][2][1][RTW89_ACMA][3] = 127, + [2][1][2][1][RTW89_CN][3] = 127, + [2][1][2][1][RTW89_UK][3] = 127, + [2][1][2][1][RTW89_FCC][11] = 127, + [2][1][2][1][RTW89_ETSI][11] = 127, + [2][1][2][1][RTW89_MKK][11] = 127, + [2][1][2][1][RTW89_IC][11] = 127, + [2][1][2][1][RTW89_KCC][11] = 127, + [2][1][2][1][RTW89_ACMA][11] = 127, + [2][1][2][1][RTW89_CN][11] = 127, + [2][1][2][1][RTW89_UK][11] = 127, + [2][1][2][1][RTW89_FCC][18] = 127, + [2][1][2][1][RTW89_ETSI][18] = 127, + [2][1][2][1][RTW89_MKK][18] = 127, + [2][1][2][1][RTW89_IC][18] = 127, + [2][1][2][1][RTW89_KCC][18] = 127, + [2][1][2][1][RTW89_ACMA][18] = 127, + [2][1][2][1][RTW89_CN][18] = 127, + [2][1][2][1][RTW89_UK][18] = 127, + [2][1][2][1][RTW89_FCC][26] = 127, + [2][1][2][1][RTW89_ETSI][26] = 127, + [2][1][2][1][RTW89_MKK][26] = 127, + [2][1][2][1][RTW89_IC][26] = 127, + [2][1][2][1][RTW89_KCC][26] = 127, + [2][1][2][1][RTW89_ACMA][26] = 127, + [2][1][2][1][RTW89_CN][26] = 127, + [2][1][2][1][RTW89_UK][26] = 127, + [2][1][2][1][RTW89_FCC][34] = 127, + [2][1][2][1][RTW89_ETSI][34] = 127, + [2][1][2][1][RTW89_MKK][34] = 127, + [2][1][2][1][RTW89_IC][34] = 127, + [2][1][2][1][RTW89_KCC][34] = 127, + [2][1][2][1][RTW89_ACMA][34] = 127, + [2][1][2][1][RTW89_CN][34] = 127, + [2][1][2][1][RTW89_UK][34] = 127, + [2][1][2][1][RTW89_FCC][41] = 127, + [2][1][2][1][RTW89_ETSI][41] = 127, + [2][1][2][1][RTW89_MKK][41] = 127, + [2][1][2][1][RTW89_IC][41] = 127, + [2][1][2][1][RTW89_KCC][41] = 127, + [2][1][2][1][RTW89_ACMA][41] = 127, + [2][1][2][1][RTW89_CN][41] = 127, + [2][1][2][1][RTW89_UK][41] = 127, + [2][1][2][1][RTW89_FCC][49] = 127, + [2][1][2][1][RTW89_ETSI][49] = 127, + [2][1][2][1][RTW89_MKK][49] = 127, + [2][1][2][1][RTW89_IC][49] = 127, + [2][1][2][1][RTW89_KCC][49] = 127, + [2][1][2][1][RTW89_ACMA][49] = 127, + [2][1][2][1][RTW89_CN][49] = 127, + [2][1][2][1][RTW89_UK][49] = 127, + [3][0][2][0][RTW89_FCC][7] = 127, + [3][0][2][0][RTW89_ETSI][7] = 127, + [3][0][2][0][RTW89_MKK][7] = 127, + [3][0][2][0][RTW89_IC][7] = 127, + [3][0][2][0][RTW89_KCC][7] = 127, + [3][0][2][0][RTW89_ACMA][7] = 127, + [3][0][2][0][RTW89_CN][7] = 58, + [3][0][2][0][RTW89_UK][7] = 127, + [3][0][2][0][RTW89_FCC][22] = 127, + [3][0][2][0][RTW89_ETSI][22] = 127, + [3][0][2][0][RTW89_MKK][22] = 127, + [3][0][2][0][RTW89_IC][22] = 127, + [3][0][2][0][RTW89_KCC][22] = 127, + [3][0][2][0][RTW89_ACMA][22] = 127, + [3][0][2][0][RTW89_CN][22] = 58, + [3][0][2][0][RTW89_UK][22] = 127, + [3][0][2][0][RTW89_FCC][45] = 127, + [3][0][2][0][RTW89_ETSI][45] = 127, + [3][0][2][0][RTW89_MKK][45] = 127, + [3][0][2][0][RTW89_IC][45] = 127, + [3][0][2][0][RTW89_KCC][45] = 127, + [3][0][2][0][RTW89_ACMA][45] = 127, + [3][0][2][0][RTW89_CN][45] = 127, + [3][0][2][0][RTW89_UK][45] = 127, + [3][1][2][0][RTW89_FCC][7] = 127, + [3][1][2][0][RTW89_ETSI][7] = 127, + [3][1][2][0][RTW89_MKK][7] = 127, + [3][1][2][0][RTW89_IC][7] = 127, + [3][1][2][0][RTW89_KCC][7] = 127, + [3][1][2][0][RTW89_ACMA][7] = 127, + [3][1][2][0][RTW89_CN][7] = 127, + [3][1][2][0][RTW89_UK][7] = 127, + [3][1][2][0][RTW89_FCC][22] = 127, + [3][1][2][0][RTW89_ETSI][22] = 127, + [3][1][2][0][RTW89_MKK][22] = 127, + [3][1][2][0][RTW89_IC][22] = 127, + [3][1][2][0][RTW89_KCC][22] = 127, + [3][1][2][0][RTW89_ACMA][22] = 127, + [3][1][2][0][RTW89_CN][22] = 127, + [3][1][2][0][RTW89_UK][22] = 127, + [3][1][2][0][RTW89_FCC][45] = 127, + [3][1][2][0][RTW89_ETSI][45] = 127, + [3][1][2][0][RTW89_MKK][45] = 127, + [3][1][2][0][RTW89_IC][45] = 127, + [3][1][2][0][RTW89_KCC][45] = 127, + [3][1][2][0][RTW89_ACMA][45] = 127, + [3][1][2][0][RTW89_CN][45] = 127, + [3][1][2][0][RTW89_UK][45] = 127, + [3][1][2][1][RTW89_FCC][7] = 127, + [3][1][2][1][RTW89_ETSI][7] = 127, + [3][1][2][1][RTW89_MKK][7] = 127, + [3][1][2][1][RTW89_IC][7] = 127, + [3][1][2][1][RTW89_KCC][7] = 127, + [3][1][2][1][RTW89_ACMA][7] = 127, + [3][1][2][1][RTW89_CN][7] = 127, + [3][1][2][1][RTW89_UK][7] = 127, + [3][1][2][1][RTW89_FCC][22] = 127, + [3][1][2][1][RTW89_ETSI][22] = 127, + [3][1][2][1][RTW89_MKK][22] = 127, + [3][1][2][1][RTW89_IC][22] = 127, + [3][1][2][1][RTW89_KCC][22] = 127, + [3][1][2][1][RTW89_ACMA][22] = 127, + [3][1][2][1][RTW89_CN][22] = 127, + [3][1][2][1][RTW89_UK][22] = 127, + [3][1][2][1][RTW89_FCC][45] = 127, + [3][1][2][1][RTW89_ETSI][45] = 127, + [3][1][2][1][RTW89_MKK][45] = 127, + [3][1][2][1][RTW89_IC][45] = 127, + [3][1][2][1][RTW89_KCC][45] = 127, + [3][1][2][1][RTW89_ACMA][45] = 127, + [3][1][2][1][RTW89_CN][45] = 127, + [3][1][2][1][RTW89_UK][45] = 127, +}; + +static +const s8 rtw89_8851b_txpwr_lmt_ru_2g_type2[RTW89_RU_NUM][RTW89_NTX_NUM] + [RTW89_REGD_NUM][RTW89_2G_CH_NUM] = { + [0][0][RTW89_WW][0] = 30, + [0][0][RTW89_WW][1] = 30, + [0][0][RTW89_WW][2] = 30, + [0][0][RTW89_WW][3] = 30, + [0][0][RTW89_WW][4] = 30, + [0][0][RTW89_WW][5] = 30, + [0][0][RTW89_WW][6] = 30, + [0][0][RTW89_WW][7] = 30, + [0][0][RTW89_WW][8] = 30, + [0][0][RTW89_WW][9] = 30, + [0][0][RTW89_WW][10] = 30, + [0][0][RTW89_WW][11] = 30, + [0][0][RTW89_WW][12] = 30, + [0][0][RTW89_WW][13] = 0, + [0][1][RTW89_WW][0] = 20, + [0][1][RTW89_WW][1] = 22, + [0][1][RTW89_WW][2] = 22, + [0][1][RTW89_WW][3] = 22, + [0][1][RTW89_WW][4] = 22, + [0][1][RTW89_WW][5] = 22, + [0][1][RTW89_WW][6] = 22, + [0][1][RTW89_WW][7] = 22, + [0][1][RTW89_WW][8] = 22, + [0][1][RTW89_WW][9] = 22, + [0][1][RTW89_WW][10] = 22, + [0][1][RTW89_WW][11] = 22, + [0][1][RTW89_WW][12] = 20, + [0][1][RTW89_WW][13] = 0, + [1][0][RTW89_WW][0] = 42, + [1][0][RTW89_WW][1] = 42, + [1][0][RTW89_WW][2] = 42, + [1][0][RTW89_WW][3] = 42, + [1][0][RTW89_WW][4] = 42, + [1][0][RTW89_WW][5] = 42, + [1][0][RTW89_WW][6] = 42, + [1][0][RTW89_WW][7] = 42, + [1][0][RTW89_WW][8] = 42, + [1][0][RTW89_WW][9] = 42, + [1][0][RTW89_WW][10] = 42, + [1][0][RTW89_WW][11] = 42, + [1][0][RTW89_WW][12] = 34, + [1][0][RTW89_WW][13] = 0, + [1][1][RTW89_WW][0] = 32, + [1][1][RTW89_WW][1] = 32, + [1][1][RTW89_WW][2] = 32, + [1][1][RTW89_WW][3] = 32, + [1][1][RTW89_WW][4] = 32, + [1][1][RTW89_WW][5] = 32, + [1][1][RTW89_WW][6] = 32, + [1][1][RTW89_WW][7] = 32, + [1][1][RTW89_WW][8] = 32, + [1][1][RTW89_WW][9] = 32, + [1][1][RTW89_WW][10] = 32, + [1][1][RTW89_WW][11] = 32, + [1][1][RTW89_WW][12] = 32, + [1][1][RTW89_WW][13] = 0, + [2][0][RTW89_WW][0] = 54, + [2][0][RTW89_WW][1] = 54, + [2][0][RTW89_WW][2] = 54, + [2][0][RTW89_WW][3] = 54, + [2][0][RTW89_WW][4] = 54, + [2][0][RTW89_WW][5] = 54, + [2][0][RTW89_WW][6] = 54, + [2][0][RTW89_WW][7] = 54, + [2][0][RTW89_WW][8] = 54, + [2][0][RTW89_WW][9] = 54, + [2][0][RTW89_WW][10] = 54, + [2][0][RTW89_WW][11] = 54, + [2][0][RTW89_WW][12] = 34, + [2][0][RTW89_WW][13] = 0, + [2][1][RTW89_WW][0] = 44, + [2][1][RTW89_WW][1] = 44, + [2][1][RTW89_WW][2] = 44, + [2][1][RTW89_WW][3] = 44, + [2][1][RTW89_WW][4] = 44, + [2][1][RTW89_WW][5] = 44, + [2][1][RTW89_WW][6] = 44, + [2][1][RTW89_WW][7] = 44, + [2][1][RTW89_WW][8] = 44, + [2][1][RTW89_WW][9] = 44, + [2][1][RTW89_WW][10] = 44, + [2][1][RTW89_WW][11] = 44, + [2][1][RTW89_WW][12] = 42, + [2][1][RTW89_WW][13] = 0, + [0][0][RTW89_FCC][0] = 60, + [0][0][RTW89_ETSI][0] = 30, + [0][0][RTW89_MKK][0] = 40, + [0][0][RTW89_IC][0] = 60, + [0][0][RTW89_KCC][0] = 46, + [0][0][RTW89_ACMA][0] = 30, + [0][0][RTW89_CN][0] = 32, + [0][0][RTW89_UK][0] = 30, + [0][0][RTW89_FCC][1] = 60, + [0][0][RTW89_ETSI][1] = 30, + [0][0][RTW89_MKK][1] = 44, + [0][0][RTW89_IC][1] = 60, + [0][0][RTW89_KCC][1] = 46, + [0][0][RTW89_ACMA][1] = 30, + [0][0][RTW89_CN][1] = 32, + [0][0][RTW89_UK][1] = 30, + [0][0][RTW89_FCC][2] = 64, + [0][0][RTW89_ETSI][2] = 30, + [0][0][RTW89_MKK][2] = 44, + [0][0][RTW89_IC][2] = 64, + [0][0][RTW89_KCC][2] = 46, + [0][0][RTW89_ACMA][2] = 30, + [0][0][RTW89_CN][2] = 32, + [0][0][RTW89_UK][2] = 30, + [0][0][RTW89_FCC][3] = 68, + [0][0][RTW89_ETSI][3] = 30, + [0][0][RTW89_MKK][3] = 44, + [0][0][RTW89_IC][3] = 68, + [0][0][RTW89_KCC][3] = 46, + [0][0][RTW89_ACMA][3] = 30, + [0][0][RTW89_CN][3] = 32, + [0][0][RTW89_UK][3] = 30, + [0][0][RTW89_FCC][4] = 68, + [0][0][RTW89_ETSI][4] = 30, + [0][0][RTW89_MKK][4] = 44, + [0][0][RTW89_IC][4] = 68, + [0][0][RTW89_KCC][4] = 48, + [0][0][RTW89_ACMA][4] = 30, + [0][0][RTW89_CN][4] = 32, + [0][0][RTW89_UK][4] = 30, + [0][0][RTW89_FCC][5] = 82, + [0][0][RTW89_ETSI][5] = 30, + [0][0][RTW89_MKK][5] = 44, + [0][0][RTW89_IC][5] = 82, + [0][0][RTW89_KCC][5] = 48, + [0][0][RTW89_ACMA][5] = 30, + [0][0][RTW89_CN][5] = 32, + [0][0][RTW89_UK][5] = 30, + [0][0][RTW89_FCC][6] = 64, + [0][0][RTW89_ETSI][6] = 30, + [0][0][RTW89_MKK][6] = 44, + [0][0][RTW89_IC][6] = 64, + [0][0][RTW89_KCC][6] = 48, + [0][0][RTW89_ACMA][6] = 30, + [0][0][RTW89_CN][6] = 32, + [0][0][RTW89_UK][6] = 30, + [0][0][RTW89_FCC][7] = 64, + [0][0][RTW89_ETSI][7] = 30, + [0][0][RTW89_MKK][7] = 44, + [0][0][RTW89_IC][7] = 64, + [0][0][RTW89_KCC][7] = 48, + [0][0][RTW89_ACMA][7] = 30, + [0][0][RTW89_CN][7] = 32, + [0][0][RTW89_UK][7] = 30, + [0][0][RTW89_FCC][8] = 60, + [0][0][RTW89_ETSI][8] = 30, + [0][0][RTW89_MKK][8] = 44, + [0][0][RTW89_IC][8] = 60, + [0][0][RTW89_KCC][8] = 48, + [0][0][RTW89_ACMA][8] = 30, + [0][0][RTW89_CN][8] = 32, + [0][0][RTW89_UK][8] = 30, + [0][0][RTW89_FCC][9] = 56, + [0][0][RTW89_ETSI][9] = 30, + [0][0][RTW89_MKK][9] = 44, + [0][0][RTW89_IC][9] = 56, + [0][0][RTW89_KCC][9] = 44, + [0][0][RTW89_ACMA][9] = 30, + [0][0][RTW89_CN][9] = 32, + [0][0][RTW89_UK][9] = 30, + [0][0][RTW89_FCC][10] = 56, + [0][0][RTW89_ETSI][10] = 30, + [0][0][RTW89_MKK][10] = 44, + [0][0][RTW89_IC][10] = 56, + [0][0][RTW89_KCC][10] = 44, + [0][0][RTW89_ACMA][10] = 30, + [0][0][RTW89_CN][10] = 32, + [0][0][RTW89_UK][10] = 30, + [0][0][RTW89_FCC][11] = 54, + [0][0][RTW89_ETSI][11] = 30, + [0][0][RTW89_MKK][11] = 44, + [0][0][RTW89_IC][11] = 54, + [0][0][RTW89_KCC][11] = 44, + [0][0][RTW89_ACMA][11] = 30, + [0][0][RTW89_CN][11] = 32, + [0][0][RTW89_UK][11] = 30, + [0][0][RTW89_FCC][12] = 34, + [0][0][RTW89_ETSI][12] = 30, + [0][0][RTW89_MKK][12] = 40, + [0][0][RTW89_IC][12] = 34, + [0][0][RTW89_KCC][12] = 44, + [0][0][RTW89_ACMA][12] = 30, + [0][0][RTW89_CN][12] = 32, + [0][0][RTW89_UK][12] = 30, + [0][0][RTW89_FCC][13] = 127, + [0][0][RTW89_ETSI][13] = 127, + [0][0][RTW89_MKK][13] = 127, + [0][0][RTW89_IC][13] = 127, + [0][0][RTW89_KCC][13] = 127, + [0][0][RTW89_ACMA][13] = 127, + [0][0][RTW89_CN][13] = 127, + [0][0][RTW89_UK][13] = 127, + [0][1][RTW89_FCC][0] = 127, + [0][1][RTW89_ETSI][0] = 127, + [0][1][RTW89_MKK][0] = 127, + [0][1][RTW89_IC][0] = 127, + [0][1][RTW89_KCC][0] = 127, + [0][1][RTW89_ACMA][0] = 127, + [0][1][RTW89_CN][0] = 20, + [0][1][RTW89_UK][0] = 127, + [0][1][RTW89_FCC][1] = 127, + [0][1][RTW89_ETSI][1] = 127, + [0][1][RTW89_MKK][1] = 127, + [0][1][RTW89_IC][1] = 127, + [0][1][RTW89_KCC][1] = 127, + [0][1][RTW89_ACMA][1] = 127, + [0][1][RTW89_CN][1] = 22, + [0][1][RTW89_UK][1] = 127, + [0][1][RTW89_FCC][2] = 127, + [0][1][RTW89_ETSI][2] = 127, + [0][1][RTW89_MKK][2] = 127, + [0][1][RTW89_IC][2] = 127, + [0][1][RTW89_KCC][2] = 127, + [0][1][RTW89_ACMA][2] = 127, + [0][1][RTW89_CN][2] = 22, + [0][1][RTW89_UK][2] = 127, + [0][1][RTW89_FCC][3] = 127, + [0][1][RTW89_ETSI][3] = 127, + [0][1][RTW89_MKK][3] = 127, + [0][1][RTW89_IC][3] = 127, + [0][1][RTW89_KCC][3] = 127, + [0][1][RTW89_ACMA][3] = 127, + [0][1][RTW89_CN][3] = 22, + [0][1][RTW89_UK][3] = 127, + [0][1][RTW89_FCC][4] = 127, + [0][1][RTW89_ETSI][4] = 127, + [0][1][RTW89_MKK][4] = 127, + [0][1][RTW89_IC][4] = 127, + [0][1][RTW89_KCC][4] = 127, + [0][1][RTW89_ACMA][4] = 127, + [0][1][RTW89_CN][4] = 22, + [0][1][RTW89_UK][4] = 127, + [0][1][RTW89_FCC][5] = 127, + [0][1][RTW89_ETSI][5] = 127, + [0][1][RTW89_MKK][5] = 127, + [0][1][RTW89_IC][5] = 127, + [0][1][RTW89_KCC][5] = 127, + [0][1][RTW89_ACMA][5] = 127, + [0][1][RTW89_CN][5] = 22, + [0][1][RTW89_UK][5] = 127, + [0][1][RTW89_FCC][6] = 127, + [0][1][RTW89_ETSI][6] = 127, + [0][1][RTW89_MKK][6] = 127, + [0][1][RTW89_IC][6] = 127, + [0][1][RTW89_KCC][6] = 127, + [0][1][RTW89_ACMA][6] = 127, + [0][1][RTW89_CN][6] = 22, + [0][1][RTW89_UK][6] = 127, + [0][1][RTW89_FCC][7] = 127, + [0][1][RTW89_ETSI][7] = 127, + [0][1][RTW89_MKK][7] = 127, + [0][1][RTW89_IC][7] = 127, + [0][1][RTW89_KCC][7] = 127, + [0][1][RTW89_ACMA][7] = 127, + [0][1][RTW89_CN][7] = 22, + [0][1][RTW89_UK][7] = 127, + [0][1][RTW89_FCC][8] = 127, + [0][1][RTW89_ETSI][8] = 127, + [0][1][RTW89_MKK][8] = 127, + [0][1][RTW89_IC][8] = 127, + [0][1][RTW89_KCC][8] = 127, + [0][1][RTW89_ACMA][8] = 127, + [0][1][RTW89_CN][8] = 22, + [0][1][RTW89_UK][8] = 127, + [0][1][RTW89_FCC][9] = 127, + [0][1][RTW89_ETSI][9] = 127, + [0][1][RTW89_MKK][9] = 127, + [0][1][RTW89_IC][9] = 127, + [0][1][RTW89_KCC][9] = 127, + [0][1][RTW89_ACMA][9] = 127, + [0][1][RTW89_CN][9] = 22, + [0][1][RTW89_UK][9] = 127, + [0][1][RTW89_FCC][10] = 127, + [0][1][RTW89_ETSI][10] = 127, + [0][1][RTW89_MKK][10] = 127, + [0][1][RTW89_IC][10] = 127, + [0][1][RTW89_KCC][10] = 127, + [0][1][RTW89_ACMA][10] = 127, + [0][1][RTW89_CN][10] = 22, + [0][1][RTW89_UK][10] = 127, + [0][1][RTW89_FCC][11] = 127, + [0][1][RTW89_ETSI][11] = 127, + [0][1][RTW89_MKK][11] = 127, + [0][1][RTW89_IC][11] = 127, + [0][1][RTW89_KCC][11] = 127, + [0][1][RTW89_ACMA][11] = 127, + [0][1][RTW89_CN][11] = 22, + [0][1][RTW89_UK][11] = 127, + [0][1][RTW89_FCC][12] = 127, + [0][1][RTW89_ETSI][12] = 127, + [0][1][RTW89_MKK][12] = 127, + [0][1][RTW89_IC][12] = 127, + [0][1][RTW89_KCC][12] = 127, + [0][1][RTW89_ACMA][12] = 127, + [0][1][RTW89_CN][12] = 20, + [0][1][RTW89_UK][12] = 127, + [0][1][RTW89_FCC][13] = 127, + [0][1][RTW89_ETSI][13] = 127, + [0][1][RTW89_MKK][13] = 127, + [0][1][RTW89_IC][13] = 127, + [0][1][RTW89_KCC][13] = 127, + [0][1][RTW89_ACMA][13] = 127, + [0][1][RTW89_CN][13] = 127, + [0][1][RTW89_UK][13] = 127, + [1][0][RTW89_FCC][0] = 70, + [1][0][RTW89_ETSI][0] = 42, + [1][0][RTW89_MKK][0] = 52, + [1][0][RTW89_IC][0] = 70, + [1][0][RTW89_KCC][0] = 56, + [1][0][RTW89_ACMA][0] = 42, + [1][0][RTW89_CN][0] = 42, + [1][0][RTW89_UK][0] = 42, + [1][0][RTW89_FCC][1] = 70, + [1][0][RTW89_ETSI][1] = 42, + [1][0][RTW89_MKK][1] = 52, + [1][0][RTW89_IC][1] = 70, + [1][0][RTW89_KCC][1] = 56, + [1][0][RTW89_ACMA][1] = 42, + [1][0][RTW89_CN][1] = 44, + [1][0][RTW89_UK][1] = 42, + [1][0][RTW89_FCC][2] = 74, + [1][0][RTW89_ETSI][2] = 42, + [1][0][RTW89_MKK][2] = 52, + [1][0][RTW89_IC][2] = 74, + [1][0][RTW89_KCC][2] = 56, + [1][0][RTW89_ACMA][2] = 42, + [1][0][RTW89_CN][2] = 44, + [1][0][RTW89_UK][2] = 42, + [1][0][RTW89_FCC][3] = 76, + [1][0][RTW89_ETSI][3] = 42, + [1][0][RTW89_MKK][3] = 52, + [1][0][RTW89_IC][3] = 76, + [1][0][RTW89_KCC][3] = 56, + [1][0][RTW89_ACMA][3] = 42, + [1][0][RTW89_CN][3] = 44, + [1][0][RTW89_UK][3] = 42, + [1][0][RTW89_FCC][4] = 76, + [1][0][RTW89_ETSI][4] = 42, + [1][0][RTW89_MKK][4] = 52, + [1][0][RTW89_IC][4] = 76, + [1][0][RTW89_KCC][4] = 56, + [1][0][RTW89_ACMA][4] = 42, + [1][0][RTW89_CN][4] = 44, + [1][0][RTW89_UK][4] = 42, + [1][0][RTW89_FCC][5] = 82, + [1][0][RTW89_ETSI][5] = 42, + [1][0][RTW89_MKK][5] = 52, + [1][0][RTW89_IC][5] = 82, + [1][0][RTW89_KCC][5] = 56, + [1][0][RTW89_ACMA][5] = 42, + [1][0][RTW89_CN][5] = 44, + [1][0][RTW89_UK][5] = 42, + [1][0][RTW89_FCC][6] = 72, + [1][0][RTW89_ETSI][6] = 42, + [1][0][RTW89_MKK][6] = 52, + [1][0][RTW89_IC][6] = 72, + [1][0][RTW89_KCC][6] = 56, + [1][0][RTW89_ACMA][6] = 42, + [1][0][RTW89_CN][6] = 44, + [1][0][RTW89_UK][6] = 42, + [1][0][RTW89_FCC][7] = 72, + [1][0][RTW89_ETSI][7] = 42, + [1][0][RTW89_MKK][7] = 52, + [1][0][RTW89_IC][7] = 72, + [1][0][RTW89_KCC][7] = 56, + [1][0][RTW89_ACMA][7] = 42, + [1][0][RTW89_CN][7] = 44, + [1][0][RTW89_UK][7] = 42, + [1][0][RTW89_FCC][8] = 72, + [1][0][RTW89_ETSI][8] = 42, + [1][0][RTW89_MKK][8] = 52, + [1][0][RTW89_IC][8] = 72, + [1][0][RTW89_KCC][8] = 56, + [1][0][RTW89_ACMA][8] = 42, + [1][0][RTW89_CN][8] = 44, + [1][0][RTW89_UK][8] = 42, + [1][0][RTW89_FCC][9] = 68, + [1][0][RTW89_ETSI][9] = 42, + [1][0][RTW89_MKK][9] = 52, + [1][0][RTW89_IC][9] = 68, + [1][0][RTW89_KCC][9] = 58, + [1][0][RTW89_ACMA][9] = 42, + [1][0][RTW89_CN][9] = 44, + [1][0][RTW89_UK][9] = 42, + [1][0][RTW89_FCC][10] = 68, + [1][0][RTW89_ETSI][10] = 42, + [1][0][RTW89_MKK][10] = 52, + [1][0][RTW89_IC][10] = 68, + [1][0][RTW89_KCC][10] = 58, + [1][0][RTW89_ACMA][10] = 42, + [1][0][RTW89_CN][10] = 44, + [1][0][RTW89_UK][10] = 42, + [1][0][RTW89_FCC][11] = 66, + [1][0][RTW89_ETSI][11] = 42, + [1][0][RTW89_MKK][11] = 52, + [1][0][RTW89_IC][11] = 66, + [1][0][RTW89_KCC][11] = 58, + [1][0][RTW89_ACMA][11] = 42, + [1][0][RTW89_CN][11] = 44, + [1][0][RTW89_UK][11] = 42, + [1][0][RTW89_FCC][12] = 34, + [1][0][RTW89_ETSI][12] = 42, + [1][0][RTW89_MKK][12] = 52, + [1][0][RTW89_IC][12] = 34, + [1][0][RTW89_KCC][12] = 58, + [1][0][RTW89_ACMA][12] = 42, + [1][0][RTW89_CN][12] = 42, + [1][0][RTW89_UK][12] = 42, + [1][0][RTW89_FCC][13] = 127, + [1][0][RTW89_ETSI][13] = 127, + [1][0][RTW89_MKK][13] = 127, + [1][0][RTW89_IC][13] = 127, + [1][0][RTW89_KCC][13] = 127, + [1][0][RTW89_ACMA][13] = 127, + [1][0][RTW89_CN][13] = 127, + [1][0][RTW89_UK][13] = 127, + [1][1][RTW89_FCC][0] = 127, + [1][1][RTW89_ETSI][0] = 127, + [1][1][RTW89_MKK][0] = 127, + [1][1][RTW89_IC][0] = 127, + [1][1][RTW89_KCC][0] = 127, + [1][1][RTW89_ACMA][0] = 127, + [1][1][RTW89_CN][0] = 32, + [1][1][RTW89_UK][0] = 127, + [1][1][RTW89_FCC][1] = 127, + [1][1][RTW89_ETSI][1] = 127, + [1][1][RTW89_MKK][1] = 127, + [1][1][RTW89_IC][1] = 127, + [1][1][RTW89_KCC][1] = 127, + [1][1][RTW89_ACMA][1] = 127, + [1][1][RTW89_CN][1] = 32, + [1][1][RTW89_UK][1] = 127, + [1][1][RTW89_FCC][2] = 127, + [1][1][RTW89_ETSI][2] = 127, + [1][1][RTW89_MKK][2] = 127, + [1][1][RTW89_IC][2] = 127, + [1][1][RTW89_KCC][2] = 127, + [1][1][RTW89_ACMA][2] = 127, + [1][1][RTW89_CN][2] = 32, + [1][1][RTW89_UK][2] = 127, + [1][1][RTW89_FCC][3] = 127, + [1][1][RTW89_ETSI][3] = 127, + [1][1][RTW89_MKK][3] = 127, + [1][1][RTW89_IC][3] = 127, + [1][1][RTW89_KCC][3] = 127, + [1][1][RTW89_ACMA][3] = 127, + [1][1][RTW89_CN][3] = 32, + [1][1][RTW89_UK][3] = 127, + [1][1][RTW89_FCC][4] = 127, + [1][1][RTW89_ETSI][4] = 127, + [1][1][RTW89_MKK][4] = 127, + [1][1][RTW89_IC][4] = 127, + [1][1][RTW89_KCC][4] = 127, + [1][1][RTW89_ACMA][4] = 127, + [1][1][RTW89_CN][4] = 32, + [1][1][RTW89_UK][4] = 127, + [1][1][RTW89_FCC][5] = 127, + [1][1][RTW89_ETSI][5] = 127, + [1][1][RTW89_MKK][5] = 127, + [1][1][RTW89_IC][5] = 127, + [1][1][RTW89_KCC][5] = 127, + [1][1][RTW89_ACMA][5] = 127, + [1][1][RTW89_CN][5] = 32, + [1][1][RTW89_UK][5] = 127, + [1][1][RTW89_FCC][6] = 127, + [1][1][RTW89_ETSI][6] = 127, + [1][1][RTW89_MKK][6] = 127, + [1][1][RTW89_IC][6] = 127, + [1][1][RTW89_KCC][6] = 127, + [1][1][RTW89_ACMA][6] = 127, + [1][1][RTW89_CN][6] = 32, + [1][1][RTW89_UK][6] = 127, + [1][1][RTW89_FCC][7] = 127, + [1][1][RTW89_ETSI][7] = 127, + [1][1][RTW89_MKK][7] = 127, + [1][1][RTW89_IC][7] = 127, + [1][1][RTW89_KCC][7] = 127, + [1][1][RTW89_ACMA][7] = 127, + [1][1][RTW89_CN][7] = 32, + [1][1][RTW89_UK][7] = 127, + [1][1][RTW89_FCC][8] = 127, + [1][1][RTW89_ETSI][8] = 127, + [1][1][RTW89_MKK][8] = 127, + [1][1][RTW89_IC][8] = 127, + [1][1][RTW89_KCC][8] = 127, + [1][1][RTW89_ACMA][8] = 127, + [1][1][RTW89_CN][8] = 32, + [1][1][RTW89_UK][8] = 127, + [1][1][RTW89_FCC][9] = 127, + [1][1][RTW89_ETSI][9] = 127, + [1][1][RTW89_MKK][9] = 127, + [1][1][RTW89_IC][9] = 127, + [1][1][RTW89_KCC][9] = 127, + [1][1][RTW89_ACMA][9] = 127, + [1][1][RTW89_CN][9] = 32, + [1][1][RTW89_UK][9] = 127, + [1][1][RTW89_FCC][10] = 127, + [1][1][RTW89_ETSI][10] = 127, + [1][1][RTW89_MKK][10] = 127, + [1][1][RTW89_IC][10] = 127, + [1][1][RTW89_KCC][10] = 127, + [1][1][RTW89_ACMA][10] = 127, + [1][1][RTW89_CN][10] = 32, + [1][1][RTW89_UK][10] = 127, + [1][1][RTW89_FCC][11] = 127, + [1][1][RTW89_ETSI][11] = 127, + [1][1][RTW89_MKK][11] = 127, + [1][1][RTW89_IC][11] = 127, + [1][1][RTW89_KCC][11] = 127, + [1][1][RTW89_ACMA][11] = 127, + [1][1][RTW89_CN][11] = 32, + [1][1][RTW89_UK][11] = 127, + [1][1][RTW89_FCC][12] = 127, + [1][1][RTW89_ETSI][12] = 127, + [1][1][RTW89_MKK][12] = 127, + [1][1][RTW89_IC][12] = 127, + [1][1][RTW89_KCC][12] = 127, + [1][1][RTW89_ACMA][12] = 127, + [1][1][RTW89_CN][12] = 32, + [1][1][RTW89_UK][12] = 127, + [1][1][RTW89_FCC][13] = 127, + [1][1][RTW89_ETSI][13] = 127, + [1][1][RTW89_MKK][13] = 127, + [1][1][RTW89_IC][13] = 127, + [1][1][RTW89_KCC][13] = 127, + [1][1][RTW89_ACMA][13] = 127, + [1][1][RTW89_CN][13] = 127, + [1][1][RTW89_UK][13] = 127, + [2][0][RTW89_FCC][0] = 74, + [2][0][RTW89_ETSI][0] = 54, + [2][0][RTW89_MKK][0] = 64, + [2][0][RTW89_IC][0] = 74, + [2][0][RTW89_KCC][0] = 68, + [2][0][RTW89_ACMA][0] = 54, + [2][0][RTW89_CN][0] = 56, + [2][0][RTW89_UK][0] = 54, + [2][0][RTW89_FCC][1] = 74, + [2][0][RTW89_ETSI][1] = 54, + [2][0][RTW89_MKK][1] = 64, + [2][0][RTW89_IC][1] = 74, + [2][0][RTW89_KCC][1] = 68, + [2][0][RTW89_ACMA][1] = 54, + [2][0][RTW89_CN][1] = 56, + [2][0][RTW89_UK][1] = 54, + [2][0][RTW89_FCC][2] = 76, + [2][0][RTW89_ETSI][2] = 54, + [2][0][RTW89_MKK][2] = 64, + [2][0][RTW89_IC][2] = 76, + [2][0][RTW89_KCC][2] = 68, + [2][0][RTW89_ACMA][2] = 54, + [2][0][RTW89_CN][2] = 56, + [2][0][RTW89_UK][2] = 54, + [2][0][RTW89_FCC][3] = 76, + [2][0][RTW89_ETSI][3] = 54, + [2][0][RTW89_MKK][3] = 64, + [2][0][RTW89_IC][3] = 76, + [2][0][RTW89_KCC][3] = 68, + [2][0][RTW89_ACMA][3] = 54, + [2][0][RTW89_CN][3] = 56, + [2][0][RTW89_UK][3] = 54, + [2][0][RTW89_FCC][4] = 76, + [2][0][RTW89_ETSI][4] = 54, + [2][0][RTW89_MKK][4] = 64, + [2][0][RTW89_IC][4] = 76, + [2][0][RTW89_KCC][4] = 68, + [2][0][RTW89_ACMA][4] = 54, + [2][0][RTW89_CN][4] = 56, + [2][0][RTW89_UK][4] = 54, + [2][0][RTW89_FCC][5] = 80, + [2][0][RTW89_ETSI][5] = 54, + [2][0][RTW89_MKK][5] = 64, + [2][0][RTW89_IC][5] = 80, + [2][0][RTW89_KCC][5] = 68, + [2][0][RTW89_ACMA][5] = 54, + [2][0][RTW89_CN][5] = 56, + [2][0][RTW89_UK][5] = 54, + [2][0][RTW89_FCC][6] = 72, + [2][0][RTW89_ETSI][6] = 54, + [2][0][RTW89_MKK][6] = 64, + [2][0][RTW89_IC][6] = 72, + [2][0][RTW89_KCC][6] = 68, + [2][0][RTW89_ACMA][6] = 54, + [2][0][RTW89_CN][6] = 56, + [2][0][RTW89_UK][6] = 54, + [2][0][RTW89_FCC][7] = 72, + [2][0][RTW89_ETSI][7] = 54, + [2][0][RTW89_MKK][7] = 64, + [2][0][RTW89_IC][7] = 72, + [2][0][RTW89_KCC][7] = 68, + [2][0][RTW89_ACMA][7] = 54, + [2][0][RTW89_CN][7] = 56, + [2][0][RTW89_UK][7] = 54, + [2][0][RTW89_FCC][8] = 72, + [2][0][RTW89_ETSI][8] = 54, + [2][0][RTW89_MKK][8] = 64, + [2][0][RTW89_IC][8] = 72, + [2][0][RTW89_KCC][8] = 68, + [2][0][RTW89_ACMA][8] = 54, + [2][0][RTW89_CN][8] = 56, + [2][0][RTW89_UK][8] = 54, + [2][0][RTW89_FCC][9] = 70, + [2][0][RTW89_ETSI][9] = 54, + [2][0][RTW89_MKK][9] = 64, + [2][0][RTW89_IC][9] = 70, + [2][0][RTW89_KCC][9] = 68, + [2][0][RTW89_ACMA][9] = 54, + [2][0][RTW89_CN][9] = 56, + [2][0][RTW89_UK][9] = 54, + [2][0][RTW89_FCC][10] = 70, + [2][0][RTW89_ETSI][10] = 54, + [2][0][RTW89_MKK][10] = 64, + [2][0][RTW89_IC][10] = 70, + [2][0][RTW89_KCC][10] = 68, + [2][0][RTW89_ACMA][10] = 54, + [2][0][RTW89_CN][10] = 56, + [2][0][RTW89_UK][10] = 54, + [2][0][RTW89_FCC][11] = 62, + [2][0][RTW89_ETSI][11] = 54, + [2][0][RTW89_MKK][11] = 64, + [2][0][RTW89_IC][11] = 62, + [2][0][RTW89_KCC][11] = 68, + [2][0][RTW89_ACMA][11] = 54, + [2][0][RTW89_CN][11] = 56, + [2][0][RTW89_UK][11] = 54, + [2][0][RTW89_FCC][12] = 34, + [2][0][RTW89_ETSI][12] = 54, + [2][0][RTW89_MKK][12] = 64, + [2][0][RTW89_IC][12] = 34, + [2][0][RTW89_KCC][12] = 68, + [2][0][RTW89_ACMA][12] = 54, + [2][0][RTW89_CN][12] = 56, + [2][0][RTW89_UK][12] = 54, + [2][0][RTW89_FCC][13] = 127, + [2][0][RTW89_ETSI][13] = 127, + [2][0][RTW89_MKK][13] = 127, + [2][0][RTW89_IC][13] = 127, + [2][0][RTW89_KCC][13] = 127, + [2][0][RTW89_ACMA][13] = 127, + [2][0][RTW89_CN][13] = 127, + [2][0][RTW89_UK][13] = 127, + [2][1][RTW89_FCC][0] = 127, + [2][1][RTW89_ETSI][0] = 127, + [2][1][RTW89_MKK][0] = 127, + [2][1][RTW89_IC][0] = 127, + [2][1][RTW89_KCC][0] = 127, + [2][1][RTW89_ACMA][0] = 127, + [2][1][RTW89_CN][0] = 44, + [2][1][RTW89_UK][0] = 127, + [2][1][RTW89_FCC][1] = 127, + [2][1][RTW89_ETSI][1] = 127, + [2][1][RTW89_MKK][1] = 127, + [2][1][RTW89_IC][1] = 127, + [2][1][RTW89_KCC][1] = 127, + [2][1][RTW89_ACMA][1] = 127, + [2][1][RTW89_CN][1] = 44, + [2][1][RTW89_UK][1] = 127, + [2][1][RTW89_FCC][2] = 127, + [2][1][RTW89_ETSI][2] = 127, + [2][1][RTW89_MKK][2] = 127, + [2][1][RTW89_IC][2] = 127, + [2][1][RTW89_KCC][2] = 127, + [2][1][RTW89_ACMA][2] = 127, + [2][1][RTW89_CN][2] = 44, + [2][1][RTW89_UK][2] = 127, + [2][1][RTW89_FCC][3] = 127, + [2][1][RTW89_ETSI][3] = 127, + [2][1][RTW89_MKK][3] = 127, + [2][1][RTW89_IC][3] = 127, + [2][1][RTW89_KCC][3] = 127, + [2][1][RTW89_ACMA][3] = 127, + [2][1][RTW89_CN][3] = 44, + [2][1][RTW89_UK][3] = 127, + [2][1][RTW89_FCC][4] = 127, + [2][1][RTW89_ETSI][4] = 127, + [2][1][RTW89_MKK][4] = 127, + [2][1][RTW89_IC][4] = 127, + [2][1][RTW89_KCC][4] = 127, + [2][1][RTW89_ACMA][4] = 127, + [2][1][RTW89_CN][4] = 44, + [2][1][RTW89_UK][4] = 127, + [2][1][RTW89_FCC][5] = 127, + [2][1][RTW89_ETSI][5] = 127, + [2][1][RTW89_MKK][5] = 127, + [2][1][RTW89_IC][5] = 127, + [2][1][RTW89_KCC][5] = 127, + [2][1][RTW89_ACMA][5] = 127, + [2][1][RTW89_CN][5] = 44, + [2][1][RTW89_UK][5] = 127, + [2][1][RTW89_FCC][6] = 127, + [2][1][RTW89_ETSI][6] = 127, + [2][1][RTW89_MKK][6] = 127, + [2][1][RTW89_IC][6] = 127, + [2][1][RTW89_KCC][6] = 127, + [2][1][RTW89_ACMA][6] = 127, + [2][1][RTW89_CN][6] = 44, + [2][1][RTW89_UK][6] = 127, + [2][1][RTW89_FCC][7] = 127, + [2][1][RTW89_ETSI][7] = 127, + [2][1][RTW89_MKK][7] = 127, + [2][1][RTW89_IC][7] = 127, + [2][1][RTW89_KCC][7] = 127, + [2][1][RTW89_ACMA][7] = 127, + [2][1][RTW89_CN][7] = 44, + [2][1][RTW89_UK][7] = 127, + [2][1][RTW89_FCC][8] = 127, + [2][1][RTW89_ETSI][8] = 127, + [2][1][RTW89_MKK][8] = 127, + [2][1][RTW89_IC][8] = 127, + [2][1][RTW89_KCC][8] = 127, + [2][1][RTW89_ACMA][8] = 127, + [2][1][RTW89_CN][8] = 44, + [2][1][RTW89_UK][8] = 127, + [2][1][RTW89_FCC][9] = 127, + [2][1][RTW89_ETSI][9] = 127, + [2][1][RTW89_MKK][9] = 127, + [2][1][RTW89_IC][9] = 127, + [2][1][RTW89_KCC][9] = 127, + [2][1][RTW89_ACMA][9] = 127, + [2][1][RTW89_CN][9] = 44, + [2][1][RTW89_UK][9] = 127, + [2][1][RTW89_FCC][10] = 127, + [2][1][RTW89_ETSI][10] = 127, + [2][1][RTW89_MKK][10] = 127, + [2][1][RTW89_IC][10] = 127, + [2][1][RTW89_KCC][10] = 127, + [2][1][RTW89_ACMA][10] = 127, + [2][1][RTW89_CN][10] = 44, + [2][1][RTW89_UK][10] = 127, + [2][1][RTW89_FCC][11] = 127, + [2][1][RTW89_ETSI][11] = 127, + [2][1][RTW89_MKK][11] = 127, + [2][1][RTW89_IC][11] = 127, + [2][1][RTW89_KCC][11] = 127, + [2][1][RTW89_ACMA][11] = 127, + [2][1][RTW89_CN][11] = 44, + [2][1][RTW89_UK][11] = 127, + [2][1][RTW89_FCC][12] = 127, + [2][1][RTW89_ETSI][12] = 127, + [2][1][RTW89_MKK][12] = 127, + [2][1][RTW89_IC][12] = 127, + [2][1][RTW89_KCC][12] = 127, + [2][1][RTW89_ACMA][12] = 127, + [2][1][RTW89_CN][12] = 42, + [2][1][RTW89_UK][12] = 127, + [2][1][RTW89_FCC][13] = 127, + [2][1][RTW89_ETSI][13] = 127, + [2][1][RTW89_MKK][13] = 127, + [2][1][RTW89_IC][13] = 127, + [2][1][RTW89_KCC][13] = 127, + [2][1][RTW89_ACMA][13] = 127, + [2][1][RTW89_CN][13] = 127, + [2][1][RTW89_UK][13] = 127, +}; + +static +const s8 rtw89_8851b_txpwr_lmt_ru_5g_type2[RTW89_RU_NUM][RTW89_NTX_NUM] + [RTW89_REGD_NUM][RTW89_5G_CH_NUM] = { + [0][0][RTW89_WW][0] = 16, + [0][0][RTW89_WW][2] = 16, + [0][0][RTW89_WW][4] = 16, + [0][0][RTW89_WW][6] = 16, + [0][0][RTW89_WW][8] = 16, + [0][0][RTW89_WW][10] = 16, + [0][0][RTW89_WW][12] = 16, + [0][0][RTW89_WW][14] = 16, + [0][0][RTW89_WW][15] = 24, + [0][0][RTW89_WW][17] = 24, + [0][0][RTW89_WW][19] = 24, + [0][0][RTW89_WW][21] = 24, + [0][0][RTW89_WW][23] = 24, + [0][0][RTW89_WW][25] = 24, + [0][0][RTW89_WW][27] = 24, + [0][0][RTW89_WW][29] = 24, + [0][0][RTW89_WW][31] = 24, + [0][0][RTW89_WW][33] = 24, + [0][0][RTW89_WW][35] = 24, + [0][0][RTW89_WW][37] = 44, + [0][0][RTW89_WW][38] = 24, + [0][0][RTW89_WW][40] = 24, + [0][0][RTW89_WW][42] = 24, + [0][0][RTW89_WW][44] = 24, + [0][0][RTW89_WW][46] = 24, + [0][0][RTW89_WW][48] = 40, + [0][0][RTW89_WW][50] = 42, + [0][0][RTW89_WW][52] = 38, + [0][1][RTW89_WW][0] = 4, + [0][1][RTW89_WW][2] = 4, + [0][1][RTW89_WW][4] = 4, + [0][1][RTW89_WW][6] = 4, + [0][1][RTW89_WW][8] = 4, + [0][1][RTW89_WW][10] = 4, + [0][1][RTW89_WW][12] = 4, + [0][1][RTW89_WW][14] = 4, + [0][1][RTW89_WW][15] = 0, + [0][1][RTW89_WW][17] = 0, + [0][1][RTW89_WW][19] = 0, + [0][1][RTW89_WW][21] = 0, + [0][1][RTW89_WW][23] = 0, + [0][1][RTW89_WW][25] = 0, + [0][1][RTW89_WW][27] = 0, + [0][1][RTW89_WW][29] = 0, + [0][1][RTW89_WW][31] = 0, + [0][1][RTW89_WW][33] = 0, + [0][1][RTW89_WW][35] = 0, + [0][1][RTW89_WW][37] = 0, + [0][1][RTW89_WW][38] = 42, + [0][1][RTW89_WW][40] = 42, + [0][1][RTW89_WW][42] = 42, + [0][1][RTW89_WW][44] = 42, + [0][1][RTW89_WW][46] = 42, + [0][1][RTW89_WW][48] = 0, + [0][1][RTW89_WW][50] = 0, + [0][1][RTW89_WW][52] = 0, + [1][0][RTW89_WW][0] = 26, + [1][0][RTW89_WW][2] = 26, + [1][0][RTW89_WW][4] = 26, + [1][0][RTW89_WW][6] = 26, + [1][0][RTW89_WW][8] = 26, + [1][0][RTW89_WW][10] = 26, + [1][0][RTW89_WW][12] = 26, + [1][0][RTW89_WW][14] = 26, + [1][0][RTW89_WW][15] = 34, + [1][0][RTW89_WW][17] = 34, + [1][0][RTW89_WW][19] = 34, + [1][0][RTW89_WW][21] = 34, + [1][0][RTW89_WW][23] = 34, + [1][0][RTW89_WW][25] = 34, + [1][0][RTW89_WW][27] = 34, + [1][0][RTW89_WW][29] = 34, + [1][0][RTW89_WW][31] = 34, + [1][0][RTW89_WW][33] = 34, + [1][0][RTW89_WW][35] = 34, + [1][0][RTW89_WW][37] = 54, + [1][0][RTW89_WW][38] = 28, + [1][0][RTW89_WW][40] = 28, + [1][0][RTW89_WW][42] = 28, + [1][0][RTW89_WW][44] = 28, + [1][0][RTW89_WW][46] = 28, + [1][0][RTW89_WW][48] = 52, + [1][0][RTW89_WW][50] = 52, + [1][0][RTW89_WW][52] = 50, + [1][1][RTW89_WW][0] = 14, + [1][1][RTW89_WW][2] = 14, + [1][1][RTW89_WW][4] = 14, + [1][1][RTW89_WW][6] = 14, + [1][1][RTW89_WW][8] = 14, + [1][1][RTW89_WW][10] = 14, + [1][1][RTW89_WW][12] = 14, + [1][1][RTW89_WW][14] = 14, + [1][1][RTW89_WW][15] = 0, + [1][1][RTW89_WW][17] = 0, + [1][1][RTW89_WW][19] = 0, + [1][1][RTW89_WW][21] = 0, + [1][1][RTW89_WW][23] = 0, + [1][1][RTW89_WW][25] = 0, + [1][1][RTW89_WW][27] = 0, + [1][1][RTW89_WW][29] = 0, + [1][1][RTW89_WW][31] = 0, + [1][1][RTW89_WW][33] = 0, + [1][1][RTW89_WW][35] = 0, + [1][1][RTW89_WW][37] = 0, + [1][1][RTW89_WW][38] = 54, + [1][1][RTW89_WW][40] = 54, + [1][1][RTW89_WW][42] = 54, + [1][1][RTW89_WW][44] = 54, + [1][1][RTW89_WW][46] = 54, + [1][1][RTW89_WW][48] = 0, + [1][1][RTW89_WW][50] = 0, + [1][1][RTW89_WW][52] = 0, + [2][0][RTW89_WW][0] = 40, + [2][0][RTW89_WW][2] = 40, + [2][0][RTW89_WW][4] = 40, + [2][0][RTW89_WW][6] = 40, + [2][0][RTW89_WW][8] = 40, + [2][0][RTW89_WW][10] = 40, + [2][0][RTW89_WW][12] = 40, + [2][0][RTW89_WW][14] = 40, + [2][0][RTW89_WW][15] = 46, + [2][0][RTW89_WW][17] = 46, + [2][0][RTW89_WW][19] = 46, + [2][0][RTW89_WW][21] = 46, + [2][0][RTW89_WW][23] = 46, + [2][0][RTW89_WW][25] = 46, + [2][0][RTW89_WW][27] = 46, + [2][0][RTW89_WW][29] = 46, + [2][0][RTW89_WW][31] = 46, + [2][0][RTW89_WW][33] = 46, + [2][0][RTW89_WW][35] = 46, + [2][0][RTW89_WW][37] = 66, + [2][0][RTW89_WW][38] = 28, + [2][0][RTW89_WW][40] = 28, + [2][0][RTW89_WW][42] = 28, + [2][0][RTW89_WW][44] = 28, + [2][0][RTW89_WW][46] = 28, + [2][0][RTW89_WW][48] = 62, + [2][0][RTW89_WW][50] = 62, + [2][0][RTW89_WW][52] = 60, + [2][1][RTW89_WW][0] = 28, + [2][1][RTW89_WW][2] = 28, + [2][1][RTW89_WW][4] = 28, + [2][1][RTW89_WW][6] = 28, + [2][1][RTW89_WW][8] = 28, + [2][1][RTW89_WW][10] = 28, + [2][1][RTW89_WW][12] = 28, + [2][1][RTW89_WW][14] = 28, + [2][1][RTW89_WW][15] = 0, + [2][1][RTW89_WW][17] = 0, + [2][1][RTW89_WW][19] = 0, + [2][1][RTW89_WW][21] = 0, + [2][1][RTW89_WW][23] = 0, + [2][1][RTW89_WW][25] = 0, + [2][1][RTW89_WW][27] = 0, + [2][1][RTW89_WW][29] = 0, + [2][1][RTW89_WW][31] = 0, + [2][1][RTW89_WW][33] = 0, + [2][1][RTW89_WW][35] = 0, + [2][1][RTW89_WW][37] = 0, + [2][1][RTW89_WW][38] = 56, + [2][1][RTW89_WW][40] = 56, + [2][1][RTW89_WW][42] = 56, + [2][1][RTW89_WW][44] = 56, + [2][1][RTW89_WW][46] = 56, + [2][1][RTW89_WW][48] = 0, + [2][1][RTW89_WW][50] = 0, + [2][1][RTW89_WW][52] = 0, + [0][0][RTW89_FCC][0] = 50, + [0][0][RTW89_ETSI][0] = 24, + [0][0][RTW89_MKK][0] = 26, + [0][0][RTW89_IC][0] = 28, + [0][0][RTW89_KCC][0] = 42, + [0][0][RTW89_ACMA][0] = 24, + [0][0][RTW89_CN][0] = 16, + [0][0][RTW89_UK][0] = 24, + [0][0][RTW89_FCC][2] = 54, + [0][0][RTW89_ETSI][2] = 24, + [0][0][RTW89_MKK][2] = 26, + [0][0][RTW89_IC][2] = 28, + [0][0][RTW89_KCC][2] = 42, + [0][0][RTW89_ACMA][2] = 24, + [0][0][RTW89_CN][2] = 16, + [0][0][RTW89_UK][2] = 24, + [0][0][RTW89_FCC][4] = 50, + [0][0][RTW89_ETSI][4] = 24, + [0][0][RTW89_MKK][4] = 26, + [0][0][RTW89_IC][4] = 28, + [0][0][RTW89_KCC][4] = 42, + [0][0][RTW89_ACMA][4] = 24, + [0][0][RTW89_CN][4] = 16, + [0][0][RTW89_UK][4] = 24, + [0][0][RTW89_FCC][6] = 50, + [0][0][RTW89_ETSI][6] = 24, + [0][0][RTW89_MKK][6] = 26, + [0][0][RTW89_IC][6] = 28, + [0][0][RTW89_KCC][6] = 18, + [0][0][RTW89_ACMA][6] = 24, + [0][0][RTW89_CN][6] = 16, + [0][0][RTW89_UK][6] = 24, + [0][0][RTW89_FCC][8] = 52, + [0][0][RTW89_ETSI][8] = 24, + [0][0][RTW89_MKK][8] = 26, + [0][0][RTW89_IC][8] = 52, + [0][0][RTW89_KCC][8] = 42, + [0][0][RTW89_ACMA][8] = 24, + [0][0][RTW89_CN][8] = 16, + [0][0][RTW89_UK][8] = 24, + [0][0][RTW89_FCC][10] = 52, + [0][0][RTW89_ETSI][10] = 24, + [0][0][RTW89_MKK][10] = 26, + [0][0][RTW89_IC][10] = 52, + [0][0][RTW89_KCC][10] = 42, + [0][0][RTW89_ACMA][10] = 24, + [0][0][RTW89_CN][10] = 16, + [0][0][RTW89_UK][10] = 24, + [0][0][RTW89_FCC][12] = 56, + [0][0][RTW89_ETSI][12] = 24, + [0][0][RTW89_MKK][12] = 26, + [0][0][RTW89_IC][12] = 56, + [0][0][RTW89_KCC][12] = 44, + [0][0][RTW89_ACMA][12] = 24, + [0][0][RTW89_CN][12] = 16, + [0][0][RTW89_UK][12] = 24, + [0][0][RTW89_FCC][14] = 56, + [0][0][RTW89_ETSI][14] = 24, + [0][0][RTW89_MKK][14] = 26, + [0][0][RTW89_IC][14] = 56, + [0][0][RTW89_KCC][14] = 44, + [0][0][RTW89_ACMA][14] = 24, + [0][0][RTW89_CN][14] = 16, + [0][0][RTW89_UK][14] = 24, + [0][0][RTW89_FCC][15] = 52, + [0][0][RTW89_ETSI][15] = 24, + [0][0][RTW89_MKK][15] = 46, + [0][0][RTW89_IC][15] = 52, + [0][0][RTW89_KCC][15] = 44, + [0][0][RTW89_ACMA][15] = 24, + [0][0][RTW89_CN][15] = 127, + [0][0][RTW89_UK][15] = 24, + [0][0][RTW89_FCC][17] = 52, + [0][0][RTW89_ETSI][17] = 24, + [0][0][RTW89_MKK][17] = 50, + [0][0][RTW89_IC][17] = 52, + [0][0][RTW89_KCC][17] = 44, + [0][0][RTW89_ACMA][17] = 24, + [0][0][RTW89_CN][17] = 127, + [0][0][RTW89_UK][17] = 24, + [0][0][RTW89_FCC][19] = 52, + [0][0][RTW89_ETSI][19] = 24, + [0][0][RTW89_MKK][19] = 50, + [0][0][RTW89_IC][19] = 52, + [0][0][RTW89_KCC][19] = 44, + [0][0][RTW89_ACMA][19] = 24, + [0][0][RTW89_CN][19] = 127, + [0][0][RTW89_UK][19] = 24, + [0][0][RTW89_FCC][21] = 52, + [0][0][RTW89_ETSI][21] = 24, + [0][0][RTW89_MKK][21] = 50, + [0][0][RTW89_IC][21] = 52, + [0][0][RTW89_KCC][21] = 44, + [0][0][RTW89_ACMA][21] = 24, + [0][0][RTW89_CN][21] = 127, + [0][0][RTW89_UK][21] = 24, + [0][0][RTW89_FCC][23] = 52, + [0][0][RTW89_ETSI][23] = 24, + [0][0][RTW89_MKK][23] = 50, + [0][0][RTW89_IC][23] = 52, + [0][0][RTW89_KCC][23] = 44, + [0][0][RTW89_ACMA][23] = 24, + [0][0][RTW89_CN][23] = 127, + [0][0][RTW89_UK][23] = 24, + [0][0][RTW89_FCC][25] = 52, + [0][0][RTW89_ETSI][25] = 24, + [0][0][RTW89_MKK][25] = 50, + [0][0][RTW89_IC][25] = 127, + [0][0][RTW89_KCC][25] = 44, + [0][0][RTW89_ACMA][25] = 127, + [0][0][RTW89_CN][25] = 127, + [0][0][RTW89_UK][25] = 24, + [0][0][RTW89_FCC][27] = 52, + [0][0][RTW89_ETSI][27] = 24, + [0][0][RTW89_MKK][27] = 50, + [0][0][RTW89_IC][27] = 127, + [0][0][RTW89_KCC][27] = 42, + [0][0][RTW89_ACMA][27] = 127, + [0][0][RTW89_CN][27] = 127, + [0][0][RTW89_UK][27] = 24, + [0][0][RTW89_FCC][29] = 52, + [0][0][RTW89_ETSI][29] = 24, + [0][0][RTW89_MKK][29] = 50, + [0][0][RTW89_IC][29] = 127, + [0][0][RTW89_KCC][29] = 42, + [0][0][RTW89_ACMA][29] = 127, + [0][0][RTW89_CN][29] = 127, + [0][0][RTW89_UK][29] = 24, + [0][0][RTW89_FCC][31] = 52, + [0][0][RTW89_ETSI][31] = 24, + [0][0][RTW89_MKK][31] = 50, + [0][0][RTW89_IC][31] = 56, + [0][0][RTW89_KCC][31] = 42, + [0][0][RTW89_ACMA][31] = 24, + [0][0][RTW89_CN][31] = 127, + [0][0][RTW89_UK][31] = 24, + [0][0][RTW89_FCC][33] = 56, + [0][0][RTW89_ETSI][33] = 24, + [0][0][RTW89_MKK][33] = 50, + [0][0][RTW89_IC][33] = 56, + [0][0][RTW89_KCC][33] = 42, + [0][0][RTW89_ACMA][33] = 24, + [0][0][RTW89_CN][33] = 127, + [0][0][RTW89_UK][33] = 24, + [0][0][RTW89_FCC][35] = 56, + [0][0][RTW89_ETSI][35] = 24, + [0][0][RTW89_MKK][35] = 50, + [0][0][RTW89_IC][35] = 56, + [0][0][RTW89_KCC][35] = 42, + [0][0][RTW89_ACMA][35] = 24, + [0][0][RTW89_CN][35] = 127, + [0][0][RTW89_UK][35] = 24, + [0][0][RTW89_FCC][37] = 84, + [0][0][RTW89_ETSI][37] = 127, + [0][0][RTW89_MKK][37] = 46, + [0][0][RTW89_IC][37] = 84, + [0][0][RTW89_KCC][37] = 44, + [0][0][RTW89_ACMA][37] = 50, + [0][0][RTW89_CN][37] = 127, + [0][0][RTW89_UK][37] = 52, + [0][0][RTW89_FCC][38] = 68, + [0][0][RTW89_ETSI][38] = 28, + [0][0][RTW89_MKK][38] = 127, + [0][0][RTW89_IC][38] = 68, + [0][0][RTW89_KCC][38] = 44, + [0][0][RTW89_ACMA][38] = 84, + [0][0][RTW89_CN][38] = 54, + [0][0][RTW89_UK][38] = 24, + [0][0][RTW89_FCC][40] = 68, + [0][0][RTW89_ETSI][40] = 28, + [0][0][RTW89_MKK][40] = 127, + [0][0][RTW89_IC][40] = 68, + [0][0][RTW89_KCC][40] = 44, + [0][0][RTW89_ACMA][40] = 84, + [0][0][RTW89_CN][40] = 54, + [0][0][RTW89_UK][40] = 24, + [0][0][RTW89_FCC][42] = 70, + [0][0][RTW89_ETSI][42] = 28, + [0][0][RTW89_MKK][42] = 127, + [0][0][RTW89_IC][42] = 70, + [0][0][RTW89_KCC][42] = 44, + [0][0][RTW89_ACMA][42] = 84, + [0][0][RTW89_CN][42] = 54, + [0][0][RTW89_UK][42] = 24, + [0][0][RTW89_FCC][44] = 62, + [0][0][RTW89_ETSI][44] = 28, + [0][0][RTW89_MKK][44] = 127, + [0][0][RTW89_IC][44] = 62, + [0][0][RTW89_KCC][44] = 44, + [0][0][RTW89_ACMA][44] = 84, + [0][0][RTW89_CN][44] = 54, + [0][0][RTW89_UK][44] = 24, + [0][0][RTW89_FCC][46] = 62, + [0][0][RTW89_ETSI][46] = 28, + [0][0][RTW89_MKK][46] = 127, + [0][0][RTW89_IC][46] = 62, + [0][0][RTW89_KCC][46] = 44, + [0][0][RTW89_ACMA][46] = 84, + [0][0][RTW89_CN][46] = 54, + [0][0][RTW89_UK][46] = 24, + [0][0][RTW89_FCC][48] = 40, + [0][0][RTW89_ETSI][48] = 127, + [0][0][RTW89_MKK][48] = 127, + [0][0][RTW89_IC][48] = 127, + [0][0][RTW89_KCC][48] = 127, + [0][0][RTW89_ACMA][48] = 127, + [0][0][RTW89_CN][48] = 127, + [0][0][RTW89_UK][48] = 127, + [0][0][RTW89_FCC][50] = 42, + [0][0][RTW89_ETSI][50] = 127, + [0][0][RTW89_MKK][50] = 127, + [0][0][RTW89_IC][50] = 127, + [0][0][RTW89_KCC][50] = 127, + [0][0][RTW89_ACMA][50] = 127, + [0][0][RTW89_CN][50] = 127, + [0][0][RTW89_UK][50] = 127, + [0][0][RTW89_FCC][52] = 38, + [0][0][RTW89_ETSI][52] = 127, + [0][0][RTW89_MKK][52] = 127, + [0][0][RTW89_IC][52] = 127, + [0][0][RTW89_KCC][52] = 127, + [0][0][RTW89_ACMA][52] = 127, + [0][0][RTW89_CN][52] = 127, + [0][0][RTW89_UK][52] = 127, + [0][1][RTW89_FCC][0] = 127, + [0][1][RTW89_ETSI][0] = 127, + [0][1][RTW89_MKK][0] = 127, + [0][1][RTW89_IC][0] = 127, + [0][1][RTW89_KCC][0] = 127, + [0][1][RTW89_ACMA][0] = 127, + [0][1][RTW89_CN][0] = 4, + [0][1][RTW89_UK][0] = 127, + [0][1][RTW89_FCC][2] = 127, + [0][1][RTW89_ETSI][2] = 127, + [0][1][RTW89_MKK][2] = 127, + [0][1][RTW89_IC][2] = 127, + [0][1][RTW89_KCC][2] = 127, + [0][1][RTW89_ACMA][2] = 127, + [0][1][RTW89_CN][2] = 4, + [0][1][RTW89_UK][2] = 127, + [0][1][RTW89_FCC][4] = 127, + [0][1][RTW89_ETSI][4] = 127, + [0][1][RTW89_MKK][4] = 127, + [0][1][RTW89_IC][4] = 127, + [0][1][RTW89_KCC][4] = 127, + [0][1][RTW89_ACMA][4] = 127, + [0][1][RTW89_CN][4] = 4, + [0][1][RTW89_UK][4] = 127, + [0][1][RTW89_FCC][6] = 127, + [0][1][RTW89_ETSI][6] = 127, + [0][1][RTW89_MKK][6] = 127, + [0][1][RTW89_IC][6] = 127, + [0][1][RTW89_KCC][6] = 127, + [0][1][RTW89_ACMA][6] = 127, + [0][1][RTW89_CN][6] = 4, + [0][1][RTW89_UK][6] = 127, + [0][1][RTW89_FCC][8] = 127, + [0][1][RTW89_ETSI][8] = 127, + [0][1][RTW89_MKK][8] = 127, + [0][1][RTW89_IC][8] = 127, + [0][1][RTW89_KCC][8] = 127, + [0][1][RTW89_ACMA][8] = 127, + [0][1][RTW89_CN][8] = 4, + [0][1][RTW89_UK][8] = 127, + [0][1][RTW89_FCC][10] = 127, + [0][1][RTW89_ETSI][10] = 127, + [0][1][RTW89_MKK][10] = 127, + [0][1][RTW89_IC][10] = 127, + [0][1][RTW89_KCC][10] = 127, + [0][1][RTW89_ACMA][10] = 127, + [0][1][RTW89_CN][10] = 4, + [0][1][RTW89_UK][10] = 127, + [0][1][RTW89_FCC][12] = 127, + [0][1][RTW89_ETSI][12] = 127, + [0][1][RTW89_MKK][12] = 127, + [0][1][RTW89_IC][12] = 127, + [0][1][RTW89_KCC][12] = 127, + [0][1][RTW89_ACMA][12] = 127, + [0][1][RTW89_CN][12] = 4, + [0][1][RTW89_UK][12] = 127, + [0][1][RTW89_FCC][14] = 127, + [0][1][RTW89_ETSI][14] = 127, + [0][1][RTW89_MKK][14] = 127, + [0][1][RTW89_IC][14] = 127, + [0][1][RTW89_KCC][14] = 127, + [0][1][RTW89_ACMA][14] = 127, + [0][1][RTW89_CN][14] = 4, + [0][1][RTW89_UK][14] = 127, + [0][1][RTW89_FCC][15] = 127, + [0][1][RTW89_ETSI][15] = 127, + [0][1][RTW89_MKK][15] = 127, + [0][1][RTW89_IC][15] = 127, + [0][1][RTW89_KCC][15] = 127, + [0][1][RTW89_ACMA][15] = 127, + [0][1][RTW89_CN][15] = 127, + [0][1][RTW89_UK][15] = 127, + [0][1][RTW89_FCC][17] = 127, + [0][1][RTW89_ETSI][17] = 127, + [0][1][RTW89_MKK][17] = 127, + [0][1][RTW89_IC][17] = 127, + [0][1][RTW89_KCC][17] = 127, + [0][1][RTW89_ACMA][17] = 127, + [0][1][RTW89_CN][17] = 127, + [0][1][RTW89_UK][17] = 127, + [0][1][RTW89_FCC][19] = 127, + [0][1][RTW89_ETSI][19] = 127, + [0][1][RTW89_MKK][19] = 127, + [0][1][RTW89_IC][19] = 127, + [0][1][RTW89_KCC][19] = 127, + [0][1][RTW89_ACMA][19] = 127, + [0][1][RTW89_CN][19] = 127, + [0][1][RTW89_UK][19] = 127, + [0][1][RTW89_FCC][21] = 127, + [0][1][RTW89_ETSI][21] = 127, + [0][1][RTW89_MKK][21] = 127, + [0][1][RTW89_IC][21] = 127, + [0][1][RTW89_KCC][21] = 127, + [0][1][RTW89_ACMA][21] = 127, + [0][1][RTW89_CN][21] = 127, + [0][1][RTW89_UK][21] = 127, + [0][1][RTW89_FCC][23] = 127, + [0][1][RTW89_ETSI][23] = 127, + [0][1][RTW89_MKK][23] = 127, + [0][1][RTW89_IC][23] = 127, + [0][1][RTW89_KCC][23] = 127, + [0][1][RTW89_ACMA][23] = 127, + [0][1][RTW89_CN][23] = 127, + [0][1][RTW89_UK][23] = 127, + [0][1][RTW89_FCC][25] = 127, + [0][1][RTW89_ETSI][25] = 127, + [0][1][RTW89_MKK][25] = 127, + [0][1][RTW89_IC][25] = 127, + [0][1][RTW89_KCC][25] = 127, + [0][1][RTW89_ACMA][25] = 127, + [0][1][RTW89_CN][25] = 127, + [0][1][RTW89_UK][25] = 127, + [0][1][RTW89_FCC][27] = 127, + [0][1][RTW89_ETSI][27] = 127, + [0][1][RTW89_MKK][27] = 127, + [0][1][RTW89_IC][27] = 127, + [0][1][RTW89_KCC][27] = 127, + [0][1][RTW89_ACMA][27] = 127, + [0][1][RTW89_CN][27] = 127, + [0][1][RTW89_UK][27] = 127, + [0][1][RTW89_FCC][29] = 127, + [0][1][RTW89_ETSI][29] = 127, + [0][1][RTW89_MKK][29] = 127, + [0][1][RTW89_IC][29] = 127, + [0][1][RTW89_KCC][29] = 127, + [0][1][RTW89_ACMA][29] = 127, + [0][1][RTW89_CN][29] = 127, + [0][1][RTW89_UK][29] = 127, + [0][1][RTW89_FCC][31] = 127, + [0][1][RTW89_ETSI][31] = 127, + [0][1][RTW89_MKK][31] = 127, + [0][1][RTW89_IC][31] = 127, + [0][1][RTW89_KCC][31] = 127, + [0][1][RTW89_ACMA][31] = 127, + [0][1][RTW89_CN][31] = 127, + [0][1][RTW89_UK][31] = 127, + [0][1][RTW89_FCC][33] = 127, + [0][1][RTW89_ETSI][33] = 127, + [0][1][RTW89_MKK][33] = 127, + [0][1][RTW89_IC][33] = 127, + [0][1][RTW89_KCC][33] = 127, + [0][1][RTW89_ACMA][33] = 127, + [0][1][RTW89_CN][33] = 127, + [0][1][RTW89_UK][33] = 127, + [0][1][RTW89_FCC][35] = 127, + [0][1][RTW89_ETSI][35] = 127, + [0][1][RTW89_MKK][35] = 127, + [0][1][RTW89_IC][35] = 127, + [0][1][RTW89_KCC][35] = 127, + [0][1][RTW89_ACMA][35] = 127, + [0][1][RTW89_CN][35] = 127, + [0][1][RTW89_UK][35] = 127, + [0][1][RTW89_FCC][37] = 127, + [0][1][RTW89_ETSI][37] = 127, + [0][1][RTW89_MKK][37] = 127, + [0][1][RTW89_IC][37] = 127, + [0][1][RTW89_KCC][37] = 127, + [0][1][RTW89_ACMA][37] = 127, + [0][1][RTW89_CN][37] = 127, + [0][1][RTW89_UK][37] = 127, + [0][1][RTW89_FCC][38] = 127, + [0][1][RTW89_ETSI][38] = 127, + [0][1][RTW89_MKK][38] = 127, + [0][1][RTW89_IC][38] = 127, + [0][1][RTW89_KCC][38] = 127, + [0][1][RTW89_ACMA][38] = 127, + [0][1][RTW89_CN][38] = 42, + [0][1][RTW89_UK][38] = 127, + [0][1][RTW89_FCC][40] = 127, + [0][1][RTW89_ETSI][40] = 127, + [0][1][RTW89_MKK][40] = 127, + [0][1][RTW89_IC][40] = 127, + [0][1][RTW89_KCC][40] = 127, + [0][1][RTW89_ACMA][40] = 127, + [0][1][RTW89_CN][40] = 42, + [0][1][RTW89_UK][40] = 127, + [0][1][RTW89_FCC][42] = 127, + [0][1][RTW89_ETSI][42] = 127, + [0][1][RTW89_MKK][42] = 127, + [0][1][RTW89_IC][42] = 127, + [0][1][RTW89_KCC][42] = 127, + [0][1][RTW89_ACMA][42] = 127, + [0][1][RTW89_CN][42] = 42, + [0][1][RTW89_UK][42] = 127, + [0][1][RTW89_FCC][44] = 127, + [0][1][RTW89_ETSI][44] = 127, + [0][1][RTW89_MKK][44] = 127, + [0][1][RTW89_IC][44] = 127, + [0][1][RTW89_KCC][44] = 127, + [0][1][RTW89_ACMA][44] = 127, + [0][1][RTW89_CN][44] = 42, + [0][1][RTW89_UK][44] = 127, + [0][1][RTW89_FCC][46] = 127, + [0][1][RTW89_ETSI][46] = 127, + [0][1][RTW89_MKK][46] = 127, + [0][1][RTW89_IC][46] = 127, + [0][1][RTW89_KCC][46] = 127, + [0][1][RTW89_ACMA][46] = 127, + [0][1][RTW89_CN][46] = 42, + [0][1][RTW89_UK][46] = 127, + [0][1][RTW89_FCC][48] = 127, + [0][1][RTW89_ETSI][48] = 127, + [0][1][RTW89_MKK][48] = 127, + [0][1][RTW89_IC][48] = 127, + [0][1][RTW89_KCC][48] = 127, + [0][1][RTW89_ACMA][48] = 127, + [0][1][RTW89_CN][48] = 127, + [0][1][RTW89_UK][48] = 127, + [0][1][RTW89_FCC][50] = 127, + [0][1][RTW89_ETSI][50] = 127, + [0][1][RTW89_MKK][50] = 127, + [0][1][RTW89_IC][50] = 127, + [0][1][RTW89_KCC][50] = 127, + [0][1][RTW89_ACMA][50] = 127, + [0][1][RTW89_CN][50] = 127, + [0][1][RTW89_UK][50] = 127, + [0][1][RTW89_FCC][52] = 127, + [0][1][RTW89_ETSI][52] = 127, + [0][1][RTW89_MKK][52] = 127, + [0][1][RTW89_IC][52] = 127, + [0][1][RTW89_KCC][52] = 127, + [0][1][RTW89_ACMA][52] = 127, + [0][1][RTW89_CN][52] = 127, + [0][1][RTW89_UK][52] = 127, + [1][0][RTW89_FCC][0] = 64, + [1][0][RTW89_ETSI][0] = 34, + [1][0][RTW89_MKK][0] = 38, + [1][0][RTW89_IC][0] = 38, + [1][0][RTW89_KCC][0] = 52, + [1][0][RTW89_ACMA][0] = 34, + [1][0][RTW89_CN][0] = 26, + [1][0][RTW89_UK][0] = 34, + [1][0][RTW89_FCC][2] = 66, + [1][0][RTW89_ETSI][2] = 34, + [1][0][RTW89_MKK][2] = 38, + [1][0][RTW89_IC][2] = 38, + [1][0][RTW89_KCC][2] = 52, + [1][0][RTW89_ACMA][2] = 34, + [1][0][RTW89_CN][2] = 26, + [1][0][RTW89_UK][2] = 34, + [1][0][RTW89_FCC][4] = 60, + [1][0][RTW89_ETSI][4] = 34, + [1][0][RTW89_MKK][4] = 36, + [1][0][RTW89_IC][4] = 38, + [1][0][RTW89_KCC][4] = 52, + [1][0][RTW89_ACMA][4] = 34, + [1][0][RTW89_CN][4] = 26, + [1][0][RTW89_UK][4] = 34, + [1][0][RTW89_FCC][6] = 60, + [1][0][RTW89_ETSI][6] = 34, + [1][0][RTW89_MKK][6] = 36, + [1][0][RTW89_IC][6] = 38, + [1][0][RTW89_KCC][6] = 32, + [1][0][RTW89_ACMA][6] = 34, + [1][0][RTW89_CN][6] = 26, + [1][0][RTW89_UK][6] = 34, + [1][0][RTW89_FCC][8] = 62, + [1][0][RTW89_ETSI][8] = 34, + [1][0][RTW89_MKK][8] = 38, + [1][0][RTW89_IC][8] = 62, + [1][0][RTW89_KCC][8] = 52, + [1][0][RTW89_ACMA][8] = 34, + [1][0][RTW89_CN][8] = 26, + [1][0][RTW89_UK][8] = 34, + [1][0][RTW89_FCC][10] = 62, + [1][0][RTW89_ETSI][10] = 34, + [1][0][RTW89_MKK][10] = 38, + [1][0][RTW89_IC][10] = 62, + [1][0][RTW89_KCC][10] = 52, + [1][0][RTW89_ACMA][10] = 34, + [1][0][RTW89_CN][10] = 26, + [1][0][RTW89_UK][10] = 34, + [1][0][RTW89_FCC][12] = 62, + [1][0][RTW89_ETSI][12] = 34, + [1][0][RTW89_MKK][12] = 38, + [1][0][RTW89_IC][12] = 62, + [1][0][RTW89_KCC][12] = 54, + [1][0][RTW89_ACMA][12] = 34, + [1][0][RTW89_CN][12] = 26, + [1][0][RTW89_UK][12] = 34, + [1][0][RTW89_FCC][14] = 62, + [1][0][RTW89_ETSI][14] = 34, + [1][0][RTW89_MKK][14] = 38, + [1][0][RTW89_IC][14] = 62, + [1][0][RTW89_KCC][14] = 54, + [1][0][RTW89_ACMA][14] = 34, + [1][0][RTW89_CN][14] = 26, + [1][0][RTW89_UK][14] = 34, + [1][0][RTW89_FCC][15] = 60, + [1][0][RTW89_ETSI][15] = 34, + [1][0][RTW89_MKK][15] = 58, + [1][0][RTW89_IC][15] = 60, + [1][0][RTW89_KCC][15] = 54, + [1][0][RTW89_ACMA][15] = 34, + [1][0][RTW89_CN][15] = 127, + [1][0][RTW89_UK][15] = 34, + [1][0][RTW89_FCC][17] = 60, + [1][0][RTW89_ETSI][17] = 34, + [1][0][RTW89_MKK][17] = 58, + [1][0][RTW89_IC][17] = 60, + [1][0][RTW89_KCC][17] = 54, + [1][0][RTW89_ACMA][17] = 34, + [1][0][RTW89_CN][17] = 127, + [1][0][RTW89_UK][17] = 34, + [1][0][RTW89_FCC][19] = 62, + [1][0][RTW89_ETSI][19] = 34, + [1][0][RTW89_MKK][19] = 58, + [1][0][RTW89_IC][19] = 62, + [1][0][RTW89_KCC][19] = 54, + [1][0][RTW89_ACMA][19] = 34, + [1][0][RTW89_CN][19] = 127, + [1][0][RTW89_UK][19] = 34, + [1][0][RTW89_FCC][21] = 62, + [1][0][RTW89_ETSI][21] = 34, + [1][0][RTW89_MKK][21] = 58, + [1][0][RTW89_IC][21] = 62, + [1][0][RTW89_KCC][21] = 54, + [1][0][RTW89_ACMA][21] = 34, + [1][0][RTW89_CN][21] = 127, + [1][0][RTW89_UK][21] = 34, + [1][0][RTW89_FCC][23] = 62, + [1][0][RTW89_ETSI][23] = 34, + [1][0][RTW89_MKK][23] = 58, + [1][0][RTW89_IC][23] = 62, + [1][0][RTW89_KCC][23] = 54, + [1][0][RTW89_ACMA][23] = 34, + [1][0][RTW89_CN][23] = 127, + [1][0][RTW89_UK][23] = 34, + [1][0][RTW89_FCC][25] = 62, + [1][0][RTW89_ETSI][25] = 34, + [1][0][RTW89_MKK][25] = 58, + [1][0][RTW89_IC][25] = 127, + [1][0][RTW89_KCC][25] = 54, + [1][0][RTW89_ACMA][25] = 127, + [1][0][RTW89_CN][25] = 127, + [1][0][RTW89_UK][25] = 34, + [1][0][RTW89_FCC][27] = 62, + [1][0][RTW89_ETSI][27] = 34, + [1][0][RTW89_MKK][27] = 58, + [1][0][RTW89_IC][27] = 127, + [1][0][RTW89_KCC][27] = 54, + [1][0][RTW89_ACMA][27] = 127, + [1][0][RTW89_CN][27] = 127, + [1][0][RTW89_UK][27] = 34, + [1][0][RTW89_FCC][29] = 62, + [1][0][RTW89_ETSI][29] = 34, + [1][0][RTW89_MKK][29] = 58, + [1][0][RTW89_IC][29] = 127, + [1][0][RTW89_KCC][29] = 54, + [1][0][RTW89_ACMA][29] = 127, + [1][0][RTW89_CN][29] = 127, + [1][0][RTW89_UK][29] = 34, + [1][0][RTW89_FCC][31] = 62, + [1][0][RTW89_ETSI][31] = 34, + [1][0][RTW89_MKK][31] = 58, + [1][0][RTW89_IC][31] = 64, + [1][0][RTW89_KCC][31] = 54, + [1][0][RTW89_ACMA][31] = 34, + [1][0][RTW89_CN][31] = 127, + [1][0][RTW89_UK][31] = 34, + [1][0][RTW89_FCC][33] = 64, + [1][0][RTW89_ETSI][33] = 34, + [1][0][RTW89_MKK][33] = 58, + [1][0][RTW89_IC][33] = 64, + [1][0][RTW89_KCC][33] = 54, + [1][0][RTW89_ACMA][33] = 34, + [1][0][RTW89_CN][33] = 127, + [1][0][RTW89_UK][33] = 34, + [1][0][RTW89_FCC][35] = 64, + [1][0][RTW89_ETSI][35] = 34, + [1][0][RTW89_MKK][35] = 58, + [1][0][RTW89_IC][35] = 64, + [1][0][RTW89_KCC][35] = 54, + [1][0][RTW89_ACMA][35] = 34, + [1][0][RTW89_CN][35] = 127, + [1][0][RTW89_UK][35] = 34, + [1][0][RTW89_FCC][37] = 76, + [1][0][RTW89_ETSI][37] = 127, + [1][0][RTW89_MKK][37] = 56, + [1][0][RTW89_IC][37] = 76, + [1][0][RTW89_KCC][37] = 54, + [1][0][RTW89_ACMA][37] = 62, + [1][0][RTW89_CN][37] = 127, + [1][0][RTW89_UK][37] = 62, + [1][0][RTW89_FCC][38] = 82, + [1][0][RTW89_ETSI][38] = 28, + [1][0][RTW89_MKK][38] = 127, + [1][0][RTW89_IC][38] = 82, + [1][0][RTW89_KCC][38] = 54, + [1][0][RTW89_ACMA][38] = 84, + [1][0][RTW89_CN][38] = 66, + [1][0][RTW89_UK][38] = 34, + [1][0][RTW89_FCC][40] = 82, + [1][0][RTW89_ETSI][40] = 28, + [1][0][RTW89_MKK][40] = 127, + [1][0][RTW89_IC][40] = 82, + [1][0][RTW89_KCC][40] = 54, + [1][0][RTW89_ACMA][40] = 84, + [1][0][RTW89_CN][40] = 66, + [1][0][RTW89_UK][40] = 34, + [1][0][RTW89_FCC][42] = 78, + [1][0][RTW89_ETSI][42] = 28, + [1][0][RTW89_MKK][42] = 127, + [1][0][RTW89_IC][42] = 78, + [1][0][RTW89_KCC][42] = 54, + [1][0][RTW89_ACMA][42] = 84, + [1][0][RTW89_CN][42] = 66, + [1][0][RTW89_UK][42] = 34, + [1][0][RTW89_FCC][44] = 82, + [1][0][RTW89_ETSI][44] = 28, + [1][0][RTW89_MKK][44] = 127, + [1][0][RTW89_IC][44] = 82, + [1][0][RTW89_KCC][44] = 54, + [1][0][RTW89_ACMA][44] = 84, + [1][0][RTW89_CN][44] = 66, + [1][0][RTW89_UK][44] = 34, + [1][0][RTW89_FCC][46] = 82, + [1][0][RTW89_ETSI][46] = 28, + [1][0][RTW89_MKK][46] = 127, + [1][0][RTW89_IC][46] = 82, + [1][0][RTW89_KCC][46] = 54, + [1][0][RTW89_ACMA][46] = 84, + [1][0][RTW89_CN][46] = 66, + [1][0][RTW89_UK][46] = 34, + [1][0][RTW89_FCC][48] = 52, + [1][0][RTW89_ETSI][48] = 127, + [1][0][RTW89_MKK][48] = 127, + [1][0][RTW89_IC][48] = 127, + [1][0][RTW89_KCC][48] = 127, + [1][0][RTW89_ACMA][48] = 127, + [1][0][RTW89_CN][48] = 127, + [1][0][RTW89_UK][48] = 127, + [1][0][RTW89_FCC][50] = 52, + [1][0][RTW89_ETSI][50] = 127, + [1][0][RTW89_MKK][50] = 127, + [1][0][RTW89_IC][50] = 127, + [1][0][RTW89_KCC][50] = 127, + [1][0][RTW89_ACMA][50] = 127, + [1][0][RTW89_CN][50] = 127, + [1][0][RTW89_UK][50] = 127, + [1][0][RTW89_FCC][52] = 50, + [1][0][RTW89_ETSI][52] = 127, + [1][0][RTW89_MKK][52] = 127, + [1][0][RTW89_IC][52] = 127, + [1][0][RTW89_KCC][52] = 127, + [1][0][RTW89_ACMA][52] = 127, + [1][0][RTW89_CN][52] = 127, + [1][0][RTW89_UK][52] = 127, + [1][1][RTW89_FCC][0] = 127, + [1][1][RTW89_ETSI][0] = 127, + [1][1][RTW89_MKK][0] = 127, + [1][1][RTW89_IC][0] = 127, + [1][1][RTW89_KCC][0] = 127, + [1][1][RTW89_ACMA][0] = 127, + [1][1][RTW89_CN][0] = 14, + [1][1][RTW89_UK][0] = 127, + [1][1][RTW89_FCC][2] = 127, + [1][1][RTW89_ETSI][2] = 127, + [1][1][RTW89_MKK][2] = 127, + [1][1][RTW89_IC][2] = 127, + [1][1][RTW89_KCC][2] = 127, + [1][1][RTW89_ACMA][2] = 127, + [1][1][RTW89_CN][2] = 14, + [1][1][RTW89_UK][2] = 127, + [1][1][RTW89_FCC][4] = 127, + [1][1][RTW89_ETSI][4] = 127, + [1][1][RTW89_MKK][4] = 127, + [1][1][RTW89_IC][4] = 127, + [1][1][RTW89_KCC][4] = 127, + [1][1][RTW89_ACMA][4] = 127, + [1][1][RTW89_CN][4] = 14, + [1][1][RTW89_UK][4] = 127, + [1][1][RTW89_FCC][6] = 127, + [1][1][RTW89_ETSI][6] = 127, + [1][1][RTW89_MKK][6] = 127, + [1][1][RTW89_IC][6] = 127, + [1][1][RTW89_KCC][6] = 127, + [1][1][RTW89_ACMA][6] = 127, + [1][1][RTW89_CN][6] = 14, + [1][1][RTW89_UK][6] = 127, + [1][1][RTW89_FCC][8] = 127, + [1][1][RTW89_ETSI][8] = 127, + [1][1][RTW89_MKK][8] = 127, + [1][1][RTW89_IC][8] = 127, + [1][1][RTW89_KCC][8] = 127, + [1][1][RTW89_ACMA][8] = 127, + [1][1][RTW89_CN][8] = 14, + [1][1][RTW89_UK][8] = 127, + [1][1][RTW89_FCC][10] = 127, + [1][1][RTW89_ETSI][10] = 127, + [1][1][RTW89_MKK][10] = 127, + [1][1][RTW89_IC][10] = 127, + [1][1][RTW89_KCC][10] = 127, + [1][1][RTW89_ACMA][10] = 127, + [1][1][RTW89_CN][10] = 14, + [1][1][RTW89_UK][10] = 127, + [1][1][RTW89_FCC][12] = 127, + [1][1][RTW89_ETSI][12] = 127, + [1][1][RTW89_MKK][12] = 127, + [1][1][RTW89_IC][12] = 127, + [1][1][RTW89_KCC][12] = 127, + [1][1][RTW89_ACMA][12] = 127, + [1][1][RTW89_CN][12] = 14, + [1][1][RTW89_UK][12] = 127, + [1][1][RTW89_FCC][14] = 127, + [1][1][RTW89_ETSI][14] = 127, + [1][1][RTW89_MKK][14] = 127, + [1][1][RTW89_IC][14] = 127, + [1][1][RTW89_KCC][14] = 127, + [1][1][RTW89_ACMA][14] = 127, + [1][1][RTW89_CN][14] = 14, + [1][1][RTW89_UK][14] = 127, + [1][1][RTW89_FCC][15] = 127, + [1][1][RTW89_ETSI][15] = 127, + [1][1][RTW89_MKK][15] = 127, + [1][1][RTW89_IC][15] = 127, + [1][1][RTW89_KCC][15] = 127, + [1][1][RTW89_ACMA][15] = 127, + [1][1][RTW89_CN][15] = 127, + [1][1][RTW89_UK][15] = 127, + [1][1][RTW89_FCC][17] = 127, + [1][1][RTW89_ETSI][17] = 127, + [1][1][RTW89_MKK][17] = 127, + [1][1][RTW89_IC][17] = 127, + [1][1][RTW89_KCC][17] = 127, + [1][1][RTW89_ACMA][17] = 127, + [1][1][RTW89_CN][17] = 127, + [1][1][RTW89_UK][17] = 127, + [1][1][RTW89_FCC][19] = 127, + [1][1][RTW89_ETSI][19] = 127, + [1][1][RTW89_MKK][19] = 127, + [1][1][RTW89_IC][19] = 127, + [1][1][RTW89_KCC][19] = 127, + [1][1][RTW89_ACMA][19] = 127, + [1][1][RTW89_CN][19] = 127, + [1][1][RTW89_UK][19] = 127, + [1][1][RTW89_FCC][21] = 127, + [1][1][RTW89_ETSI][21] = 127, + [1][1][RTW89_MKK][21] = 127, + [1][1][RTW89_IC][21] = 127, + [1][1][RTW89_KCC][21] = 127, + [1][1][RTW89_ACMA][21] = 127, + [1][1][RTW89_CN][21] = 127, + [1][1][RTW89_UK][21] = 127, + [1][1][RTW89_FCC][23] = 127, + [1][1][RTW89_ETSI][23] = 127, + [1][1][RTW89_MKK][23] = 127, + [1][1][RTW89_IC][23] = 127, + [1][1][RTW89_KCC][23] = 127, + [1][1][RTW89_ACMA][23] = 127, + [1][1][RTW89_CN][23] = 127, + [1][1][RTW89_UK][23] = 127, + [1][1][RTW89_FCC][25] = 127, + [1][1][RTW89_ETSI][25] = 127, + [1][1][RTW89_MKK][25] = 127, + [1][1][RTW89_IC][25] = 127, + [1][1][RTW89_KCC][25] = 127, + [1][1][RTW89_ACMA][25] = 127, + [1][1][RTW89_CN][25] = 127, + [1][1][RTW89_UK][25] = 127, + [1][1][RTW89_FCC][27] = 127, + [1][1][RTW89_ETSI][27] = 127, + [1][1][RTW89_MKK][27] = 127, + [1][1][RTW89_IC][27] = 127, + [1][1][RTW89_KCC][27] = 127, + [1][1][RTW89_ACMA][27] = 127, + [1][1][RTW89_CN][27] = 127, + [1][1][RTW89_UK][27] = 127, + [1][1][RTW89_FCC][29] = 127, + [1][1][RTW89_ETSI][29] = 127, + [1][1][RTW89_MKK][29] = 127, + [1][1][RTW89_IC][29] = 127, + [1][1][RTW89_KCC][29] = 127, + [1][1][RTW89_ACMA][29] = 127, + [1][1][RTW89_CN][29] = 127, + [1][1][RTW89_UK][29] = 127, + [1][1][RTW89_FCC][31] = 127, + [1][1][RTW89_ETSI][31] = 127, + [1][1][RTW89_MKK][31] = 127, + [1][1][RTW89_IC][31] = 127, + [1][1][RTW89_KCC][31] = 127, + [1][1][RTW89_ACMA][31] = 127, + [1][1][RTW89_CN][31] = 127, + [1][1][RTW89_UK][31] = 127, + [1][1][RTW89_FCC][33] = 127, + [1][1][RTW89_ETSI][33] = 127, + [1][1][RTW89_MKK][33] = 127, + [1][1][RTW89_IC][33] = 127, + [1][1][RTW89_KCC][33] = 127, + [1][1][RTW89_ACMA][33] = 127, + [1][1][RTW89_CN][33] = 127, + [1][1][RTW89_UK][33] = 127, + [1][1][RTW89_FCC][35] = 127, + [1][1][RTW89_ETSI][35] = 127, + [1][1][RTW89_MKK][35] = 127, + [1][1][RTW89_IC][35] = 127, + [1][1][RTW89_KCC][35] = 127, + [1][1][RTW89_ACMA][35] = 127, + [1][1][RTW89_CN][35] = 127, + [1][1][RTW89_UK][35] = 127, + [1][1][RTW89_FCC][37] = 127, + [1][1][RTW89_ETSI][37] = 127, + [1][1][RTW89_MKK][37] = 127, + [1][1][RTW89_IC][37] = 127, + [1][1][RTW89_KCC][37] = 127, + [1][1][RTW89_ACMA][37] = 127, + [1][1][RTW89_CN][37] = 127, + [1][1][RTW89_UK][37] = 127, + [1][1][RTW89_FCC][38] = 127, + [1][1][RTW89_ETSI][38] = 127, + [1][1][RTW89_MKK][38] = 127, + [1][1][RTW89_IC][38] = 127, + [1][1][RTW89_KCC][38] = 127, + [1][1][RTW89_ACMA][38] = 127, + [1][1][RTW89_CN][38] = 54, + [1][1][RTW89_UK][38] = 127, + [1][1][RTW89_FCC][40] = 127, + [1][1][RTW89_ETSI][40] = 127, + [1][1][RTW89_MKK][40] = 127, + [1][1][RTW89_IC][40] = 127, + [1][1][RTW89_KCC][40] = 127, + [1][1][RTW89_ACMA][40] = 127, + [1][1][RTW89_CN][40] = 54, + [1][1][RTW89_UK][40] = 127, + [1][1][RTW89_FCC][42] = 127, + [1][1][RTW89_ETSI][42] = 127, + [1][1][RTW89_MKK][42] = 127, + [1][1][RTW89_IC][42] = 127, + [1][1][RTW89_KCC][42] = 127, + [1][1][RTW89_ACMA][42] = 127, + [1][1][RTW89_CN][42] = 54, + [1][1][RTW89_UK][42] = 127, + [1][1][RTW89_FCC][44] = 127, + [1][1][RTW89_ETSI][44] = 127, + [1][1][RTW89_MKK][44] = 127, + [1][1][RTW89_IC][44] = 127, + [1][1][RTW89_KCC][44] = 127, + [1][1][RTW89_ACMA][44] = 127, + [1][1][RTW89_CN][44] = 54, + [1][1][RTW89_UK][44] = 127, + [1][1][RTW89_FCC][46] = 127, + [1][1][RTW89_ETSI][46] = 127, + [1][1][RTW89_MKK][46] = 127, + [1][1][RTW89_IC][46] = 127, + [1][1][RTW89_KCC][46] = 127, + [1][1][RTW89_ACMA][46] = 127, + [1][1][RTW89_CN][46] = 54, + [1][1][RTW89_UK][46] = 127, + [1][1][RTW89_FCC][48] = 127, + [1][1][RTW89_ETSI][48] = 127, + [1][1][RTW89_MKK][48] = 127, + [1][1][RTW89_IC][48] = 127, + [1][1][RTW89_KCC][48] = 127, + [1][1][RTW89_ACMA][48] = 127, + [1][1][RTW89_CN][48] = 127, + [1][1][RTW89_UK][48] = 127, + [1][1][RTW89_FCC][50] = 127, + [1][1][RTW89_ETSI][50] = 127, + [1][1][RTW89_MKK][50] = 127, + [1][1][RTW89_IC][50] = 127, + [1][1][RTW89_KCC][50] = 127, + [1][1][RTW89_ACMA][50] = 127, + [1][1][RTW89_CN][50] = 127, + [1][1][RTW89_UK][50] = 127, + [1][1][RTW89_FCC][52] = 127, + [1][1][RTW89_ETSI][52] = 127, + [1][1][RTW89_MKK][52] = 127, + [1][1][RTW89_IC][52] = 127, + [1][1][RTW89_KCC][52] = 127, + [1][1][RTW89_ACMA][52] = 127, + [1][1][RTW89_CN][52] = 127, + [1][1][RTW89_UK][52] = 127, + [2][0][RTW89_FCC][0] = 76, + [2][0][RTW89_ETSI][0] = 46, + [2][0][RTW89_MKK][0] = 48, + [2][0][RTW89_IC][0] = 50, + [2][0][RTW89_KCC][0] = 64, + [2][0][RTW89_ACMA][0] = 46, + [2][0][RTW89_CN][0] = 40, + [2][0][RTW89_UK][0] = 46, + [2][0][RTW89_FCC][2] = 72, + [2][0][RTW89_ETSI][2] = 46, + [2][0][RTW89_MKK][2] = 48, + [2][0][RTW89_IC][2] = 48, + [2][0][RTW89_KCC][2] = 64, + [2][0][RTW89_ACMA][2] = 46, + [2][0][RTW89_CN][2] = 40, + [2][0][RTW89_UK][2] = 46, + [2][0][RTW89_FCC][4] = 74, + [2][0][RTW89_ETSI][4] = 46, + [2][0][RTW89_MKK][4] = 48, + [2][0][RTW89_IC][4] = 48, + [2][0][RTW89_KCC][4] = 64, + [2][0][RTW89_ACMA][4] = 46, + [2][0][RTW89_CN][4] = 40, + [2][0][RTW89_UK][4] = 46, + [2][0][RTW89_FCC][6] = 74, + [2][0][RTW89_ETSI][6] = 46, + [2][0][RTW89_MKK][6] = 48, + [2][0][RTW89_IC][6] = 48, + [2][0][RTW89_KCC][6] = 40, + [2][0][RTW89_ACMA][6] = 46, + [2][0][RTW89_CN][6] = 40, + [2][0][RTW89_UK][6] = 46, + [2][0][RTW89_FCC][8] = 72, + [2][0][RTW89_ETSI][8] = 46, + [2][0][RTW89_MKK][8] = 48, + [2][0][RTW89_IC][8] = 64, + [2][0][RTW89_KCC][8] = 66, + [2][0][RTW89_ACMA][8] = 46, + [2][0][RTW89_CN][8] = 40, + [2][0][RTW89_UK][8] = 46, + [2][0][RTW89_FCC][10] = 72, + [2][0][RTW89_ETSI][10] = 46, + [2][0][RTW89_MKK][10] = 48, + [2][0][RTW89_IC][10] = 64, + [2][0][RTW89_KCC][10] = 66, + [2][0][RTW89_ACMA][10] = 46, + [2][0][RTW89_CN][10] = 40, + [2][0][RTW89_UK][10] = 46, + [2][0][RTW89_FCC][12] = 74, + [2][0][RTW89_ETSI][12] = 46, + [2][0][RTW89_MKK][12] = 48, + [2][0][RTW89_IC][12] = 64, + [2][0][RTW89_KCC][12] = 64, + [2][0][RTW89_ACMA][12] = 46, + [2][0][RTW89_CN][12] = 40, + [2][0][RTW89_UK][12] = 46, + [2][0][RTW89_FCC][14] = 80, + [2][0][RTW89_ETSI][14] = 46, + [2][0][RTW89_MKK][14] = 48, + [2][0][RTW89_IC][14] = 64, + [2][0][RTW89_KCC][14] = 64, + [2][0][RTW89_ACMA][14] = 46, + [2][0][RTW89_CN][14] = 40, + [2][0][RTW89_UK][14] = 46, + [2][0][RTW89_FCC][15] = 72, + [2][0][RTW89_ETSI][15] = 46, + [2][0][RTW89_MKK][15] = 70, + [2][0][RTW89_IC][15] = 72, + [2][0][RTW89_KCC][15] = 66, + [2][0][RTW89_ACMA][15] = 46, + [2][0][RTW89_CN][15] = 127, + [2][0][RTW89_UK][15] = 46, + [2][0][RTW89_FCC][17] = 72, + [2][0][RTW89_ETSI][17] = 46, + [2][0][RTW89_MKK][17] = 70, + [2][0][RTW89_IC][17] = 72, + [2][0][RTW89_KCC][17] = 66, + [2][0][RTW89_ACMA][17] = 46, + [2][0][RTW89_CN][17] = 127, + [2][0][RTW89_UK][17] = 46, + [2][0][RTW89_FCC][19] = 68, + [2][0][RTW89_ETSI][19] = 46, + [2][0][RTW89_MKK][19] = 70, + [2][0][RTW89_IC][19] = 68, + [2][0][RTW89_KCC][19] = 66, + [2][0][RTW89_ACMA][19] = 46, + [2][0][RTW89_CN][19] = 127, + [2][0][RTW89_UK][19] = 46, + [2][0][RTW89_FCC][21] = 68, + [2][0][RTW89_ETSI][21] = 46, + [2][0][RTW89_MKK][21] = 70, + [2][0][RTW89_IC][21] = 68, + [2][0][RTW89_KCC][21] = 66, + [2][0][RTW89_ACMA][21] = 46, + [2][0][RTW89_CN][21] = 127, + [2][0][RTW89_UK][21] = 46, + [2][0][RTW89_FCC][23] = 68, + [2][0][RTW89_ETSI][23] = 46, + [2][0][RTW89_MKK][23] = 70, + [2][0][RTW89_IC][23] = 68, + [2][0][RTW89_KCC][23] = 66, + [2][0][RTW89_ACMA][23] = 46, + [2][0][RTW89_CN][23] = 127, + [2][0][RTW89_UK][23] = 46, + [2][0][RTW89_FCC][25] = 68, + [2][0][RTW89_ETSI][25] = 46, + [2][0][RTW89_MKK][25] = 70, + [2][0][RTW89_IC][25] = 127, + [2][0][RTW89_KCC][25] = 66, + [2][0][RTW89_ACMA][25] = 127, + [2][0][RTW89_CN][25] = 127, + [2][0][RTW89_UK][25] = 46, + [2][0][RTW89_FCC][27] = 68, + [2][0][RTW89_ETSI][27] = 46, + [2][0][RTW89_MKK][27] = 70, + [2][0][RTW89_IC][27] = 127, + [2][0][RTW89_KCC][27] = 64, + [2][0][RTW89_ACMA][27] = 127, + [2][0][RTW89_CN][27] = 127, + [2][0][RTW89_UK][27] = 46, + [2][0][RTW89_FCC][29] = 68, + [2][0][RTW89_ETSI][29] = 46, + [2][0][RTW89_MKK][29] = 70, + [2][0][RTW89_IC][29] = 127, + [2][0][RTW89_KCC][29] = 64, + [2][0][RTW89_ACMA][29] = 127, + [2][0][RTW89_CN][29] = 127, + [2][0][RTW89_UK][29] = 46, + [2][0][RTW89_FCC][31] = 68, + [2][0][RTW89_ETSI][31] = 46, + [2][0][RTW89_MKK][31] = 70, + [2][0][RTW89_IC][31] = 70, + [2][0][RTW89_KCC][31] = 64, + [2][0][RTW89_ACMA][31] = 46, + [2][0][RTW89_CN][31] = 127, + [2][0][RTW89_UK][31] = 46, + [2][0][RTW89_FCC][33] = 70, + [2][0][RTW89_ETSI][33] = 46, + [2][0][RTW89_MKK][33] = 70, + [2][0][RTW89_IC][33] = 70, + [2][0][RTW89_KCC][33] = 64, + [2][0][RTW89_ACMA][33] = 46, + [2][0][RTW89_CN][33] = 127, + [2][0][RTW89_UK][33] = 46, + [2][0][RTW89_FCC][35] = 70, + [2][0][RTW89_ETSI][35] = 46, + [2][0][RTW89_MKK][35] = 70, + [2][0][RTW89_IC][35] = 70, + [2][0][RTW89_KCC][35] = 64, + [2][0][RTW89_ACMA][35] = 46, + [2][0][RTW89_CN][35] = 127, + [2][0][RTW89_UK][35] = 46, + [2][0][RTW89_FCC][37] = 84, + [2][0][RTW89_ETSI][37] = 127, + [2][0][RTW89_MKK][37] = 68, + [2][0][RTW89_IC][37] = 84, + [2][0][RTW89_KCC][37] = 66, + [2][0][RTW89_ACMA][37] = 74, + [2][0][RTW89_CN][37] = 127, + [2][0][RTW89_UK][37] = 74, + [2][0][RTW89_FCC][38] = 84, + [2][0][RTW89_ETSI][38] = 28, + [2][0][RTW89_MKK][38] = 127, + [2][0][RTW89_IC][38] = 84, + [2][0][RTW89_KCC][38] = 64, + [2][0][RTW89_ACMA][38] = 84, + [2][0][RTW89_CN][38] = 68, + [2][0][RTW89_UK][38] = 46, + [2][0][RTW89_FCC][40] = 84, + [2][0][RTW89_ETSI][40] = 28, + [2][0][RTW89_MKK][40] = 127, + [2][0][RTW89_IC][40] = 84, + [2][0][RTW89_KCC][40] = 64, + [2][0][RTW89_ACMA][40] = 84, + [2][0][RTW89_CN][40] = 68, + [2][0][RTW89_UK][40] = 46, + [2][0][RTW89_FCC][42] = 78, + [2][0][RTW89_ETSI][42] = 28, + [2][0][RTW89_MKK][42] = 127, + [2][0][RTW89_IC][42] = 78, + [2][0][RTW89_KCC][42] = 66, + [2][0][RTW89_ACMA][42] = 84, + [2][0][RTW89_CN][42] = 68, + [2][0][RTW89_UK][42] = 46, + [2][0][RTW89_FCC][44] = 80, + [2][0][RTW89_ETSI][44] = 28, + [2][0][RTW89_MKK][44] = 127, + [2][0][RTW89_IC][44] = 80, + [2][0][RTW89_KCC][44] = 66, + [2][0][RTW89_ACMA][44] = 84, + [2][0][RTW89_CN][44] = 68, + [2][0][RTW89_UK][44] = 46, + [2][0][RTW89_FCC][46] = 80, + [2][0][RTW89_ETSI][46] = 28, + [2][0][RTW89_MKK][46] = 127, + [2][0][RTW89_IC][46] = 80, + [2][0][RTW89_KCC][46] = 66, + [2][0][RTW89_ACMA][46] = 84, + [2][0][RTW89_CN][46] = 68, + [2][0][RTW89_UK][46] = 46, + [2][0][RTW89_FCC][48] = 62, + [2][0][RTW89_ETSI][48] = 127, + [2][0][RTW89_MKK][48] = 127, + [2][0][RTW89_IC][48] = 127, + [2][0][RTW89_KCC][48] = 127, + [2][0][RTW89_ACMA][48] = 127, + [2][0][RTW89_CN][48] = 127, + [2][0][RTW89_UK][48] = 127, + [2][0][RTW89_FCC][50] = 62, + [2][0][RTW89_ETSI][50] = 127, + [2][0][RTW89_MKK][50] = 127, + [2][0][RTW89_IC][50] = 127, + [2][0][RTW89_KCC][50] = 127, + [2][0][RTW89_ACMA][50] = 127, + [2][0][RTW89_CN][50] = 127, + [2][0][RTW89_UK][50] = 127, + [2][0][RTW89_FCC][52] = 60, + [2][0][RTW89_ETSI][52] = 127, + [2][0][RTW89_MKK][52] = 127, + [2][0][RTW89_IC][52] = 127, + [2][0][RTW89_KCC][52] = 127, + [2][0][RTW89_ACMA][52] = 127, + [2][0][RTW89_CN][52] = 127, + [2][0][RTW89_UK][52] = 127, + [2][1][RTW89_FCC][0] = 127, + [2][1][RTW89_ETSI][0] = 127, + [2][1][RTW89_MKK][0] = 127, + [2][1][RTW89_IC][0] = 127, + [2][1][RTW89_KCC][0] = 127, + [2][1][RTW89_ACMA][0] = 127, + [2][1][RTW89_CN][0] = 28, + [2][1][RTW89_UK][0] = 127, + [2][1][RTW89_FCC][2] = 127, + [2][1][RTW89_ETSI][2] = 127, + [2][1][RTW89_MKK][2] = 127, + [2][1][RTW89_IC][2] = 127, + [2][1][RTW89_KCC][2] = 127, + [2][1][RTW89_ACMA][2] = 127, + [2][1][RTW89_CN][2] = 28, + [2][1][RTW89_UK][2] = 127, + [2][1][RTW89_FCC][4] = 127, + [2][1][RTW89_ETSI][4] = 127, + [2][1][RTW89_MKK][4] = 127, + [2][1][RTW89_IC][4] = 127, + [2][1][RTW89_KCC][4] = 127, + [2][1][RTW89_ACMA][4] = 127, + [2][1][RTW89_CN][4] = 28, + [2][1][RTW89_UK][4] = 127, + [2][1][RTW89_FCC][6] = 127, + [2][1][RTW89_ETSI][6] = 127, + [2][1][RTW89_MKK][6] = 127, + [2][1][RTW89_IC][6] = 127, + [2][1][RTW89_KCC][6] = 127, + [2][1][RTW89_ACMA][6] = 127, + [2][1][RTW89_CN][6] = 28, + [2][1][RTW89_UK][6] = 127, + [2][1][RTW89_FCC][8] = 127, + [2][1][RTW89_ETSI][8] = 127, + [2][1][RTW89_MKK][8] = 127, + [2][1][RTW89_IC][8] = 127, + [2][1][RTW89_KCC][8] = 127, + [2][1][RTW89_ACMA][8] = 127, + [2][1][RTW89_CN][8] = 28, + [2][1][RTW89_UK][8] = 127, + [2][1][RTW89_FCC][10] = 127, + [2][1][RTW89_ETSI][10] = 127, + [2][1][RTW89_MKK][10] = 127, + [2][1][RTW89_IC][10] = 127, + [2][1][RTW89_KCC][10] = 127, + [2][1][RTW89_ACMA][10] = 127, + [2][1][RTW89_CN][10] = 28, + [2][1][RTW89_UK][10] = 127, + [2][1][RTW89_FCC][12] = 127, + [2][1][RTW89_ETSI][12] = 127, + [2][1][RTW89_MKK][12] = 127, + [2][1][RTW89_IC][12] = 127, + [2][1][RTW89_KCC][12] = 127, + [2][1][RTW89_ACMA][12] = 127, + [2][1][RTW89_CN][12] = 28, + [2][1][RTW89_UK][12] = 127, + [2][1][RTW89_FCC][14] = 127, + [2][1][RTW89_ETSI][14] = 127, + [2][1][RTW89_MKK][14] = 127, + [2][1][RTW89_IC][14] = 127, + [2][1][RTW89_KCC][14] = 127, + [2][1][RTW89_ACMA][14] = 127, + [2][1][RTW89_CN][14] = 28, + [2][1][RTW89_UK][14] = 127, + [2][1][RTW89_FCC][15] = 127, + [2][1][RTW89_ETSI][15] = 127, + [2][1][RTW89_MKK][15] = 127, + [2][1][RTW89_IC][15] = 127, + [2][1][RTW89_KCC][15] = 127, + [2][1][RTW89_ACMA][15] = 127, + [2][1][RTW89_CN][15] = 127, + [2][1][RTW89_UK][15] = 127, + [2][1][RTW89_FCC][17] = 127, + [2][1][RTW89_ETSI][17] = 127, + [2][1][RTW89_MKK][17] = 127, + [2][1][RTW89_IC][17] = 127, + [2][1][RTW89_KCC][17] = 127, + [2][1][RTW89_ACMA][17] = 127, + [2][1][RTW89_CN][17] = 127, + [2][1][RTW89_UK][17] = 127, + [2][1][RTW89_FCC][19] = 127, + [2][1][RTW89_ETSI][19] = 127, + [2][1][RTW89_MKK][19] = 127, + [2][1][RTW89_IC][19] = 127, + [2][1][RTW89_KCC][19] = 127, + [2][1][RTW89_ACMA][19] = 127, + [2][1][RTW89_CN][19] = 127, + [2][1][RTW89_UK][19] = 127, + [2][1][RTW89_FCC][21] = 127, + [2][1][RTW89_ETSI][21] = 127, + [2][1][RTW89_MKK][21] = 127, + [2][1][RTW89_IC][21] = 127, + [2][1][RTW89_KCC][21] = 127, + [2][1][RTW89_ACMA][21] = 127, + [2][1][RTW89_CN][21] = 127, + [2][1][RTW89_UK][21] = 127, + [2][1][RTW89_FCC][23] = 127, + [2][1][RTW89_ETSI][23] = 127, + [2][1][RTW89_MKK][23] = 127, + [2][1][RTW89_IC][23] = 127, + [2][1][RTW89_KCC][23] = 127, + [2][1][RTW89_ACMA][23] = 127, + [2][1][RTW89_CN][23] = 127, + [2][1][RTW89_UK][23] = 127, + [2][1][RTW89_FCC][25] = 127, + [2][1][RTW89_ETSI][25] = 127, + [2][1][RTW89_MKK][25] = 127, + [2][1][RTW89_IC][25] = 127, + [2][1][RTW89_KCC][25] = 127, + [2][1][RTW89_ACMA][25] = 127, + [2][1][RTW89_CN][25] = 127, + [2][1][RTW89_UK][25] = 127, + [2][1][RTW89_FCC][27] = 127, + [2][1][RTW89_ETSI][27] = 127, + [2][1][RTW89_MKK][27] = 127, + [2][1][RTW89_IC][27] = 127, + [2][1][RTW89_KCC][27] = 127, + [2][1][RTW89_ACMA][27] = 127, + [2][1][RTW89_CN][27] = 127, + [2][1][RTW89_UK][27] = 127, + [2][1][RTW89_FCC][29] = 127, + [2][1][RTW89_ETSI][29] = 127, + [2][1][RTW89_MKK][29] = 127, + [2][1][RTW89_IC][29] = 127, + [2][1][RTW89_KCC][29] = 127, + [2][1][RTW89_ACMA][29] = 127, + [2][1][RTW89_CN][29] = 127, + [2][1][RTW89_UK][29] = 127, + [2][1][RTW89_FCC][31] = 127, + [2][1][RTW89_ETSI][31] = 127, + [2][1][RTW89_MKK][31] = 127, + [2][1][RTW89_IC][31] = 127, + [2][1][RTW89_KCC][31] = 127, + [2][1][RTW89_ACMA][31] = 127, + [2][1][RTW89_CN][31] = 127, + [2][1][RTW89_UK][31] = 127, + [2][1][RTW89_FCC][33] = 127, + [2][1][RTW89_ETSI][33] = 127, + [2][1][RTW89_MKK][33] = 127, + [2][1][RTW89_IC][33] = 127, + [2][1][RTW89_KCC][33] = 127, + [2][1][RTW89_ACMA][33] = 127, + [2][1][RTW89_CN][33] = 127, + [2][1][RTW89_UK][33] = 127, + [2][1][RTW89_FCC][35] = 127, + [2][1][RTW89_ETSI][35] = 127, + [2][1][RTW89_MKK][35] = 127, + [2][1][RTW89_IC][35] = 127, + [2][1][RTW89_KCC][35] = 127, + [2][1][RTW89_ACMA][35] = 127, + [2][1][RTW89_CN][35] = 127, + [2][1][RTW89_UK][35] = 127, + [2][1][RTW89_FCC][37] = 127, + [2][1][RTW89_ETSI][37] = 127, + [2][1][RTW89_MKK][37] = 127, + [2][1][RTW89_IC][37] = 127, + [2][1][RTW89_KCC][37] = 127, + [2][1][RTW89_ACMA][37] = 127, + [2][1][RTW89_CN][37] = 127, + [2][1][RTW89_UK][37] = 127, + [2][1][RTW89_FCC][38] = 127, + [2][1][RTW89_ETSI][38] = 127, + [2][1][RTW89_MKK][38] = 127, + [2][1][RTW89_IC][38] = 127, + [2][1][RTW89_KCC][38] = 127, + [2][1][RTW89_ACMA][38] = 127, + [2][1][RTW89_CN][38] = 56, + [2][1][RTW89_UK][38] = 127, + [2][1][RTW89_FCC][40] = 127, + [2][1][RTW89_ETSI][40] = 127, + [2][1][RTW89_MKK][40] = 127, + [2][1][RTW89_IC][40] = 127, + [2][1][RTW89_KCC][40] = 127, + [2][1][RTW89_ACMA][40] = 127, + [2][1][RTW89_CN][40] = 56, + [2][1][RTW89_UK][40] = 127, + [2][1][RTW89_FCC][42] = 127, + [2][1][RTW89_ETSI][42] = 127, + [2][1][RTW89_MKK][42] = 127, + [2][1][RTW89_IC][42] = 127, + [2][1][RTW89_KCC][42] = 127, + [2][1][RTW89_ACMA][42] = 127, + [2][1][RTW89_CN][42] = 56, + [2][1][RTW89_UK][42] = 127, + [2][1][RTW89_FCC][44] = 127, + [2][1][RTW89_ETSI][44] = 127, + [2][1][RTW89_MKK][44] = 127, + [2][1][RTW89_IC][44] = 127, + [2][1][RTW89_KCC][44] = 127, + [2][1][RTW89_ACMA][44] = 127, + [2][1][RTW89_CN][44] = 56, + [2][1][RTW89_UK][44] = 127, + [2][1][RTW89_FCC][46] = 127, + [2][1][RTW89_ETSI][46] = 127, + [2][1][RTW89_MKK][46] = 127, + [2][1][RTW89_IC][46] = 127, + [2][1][RTW89_KCC][46] = 127, + [2][1][RTW89_ACMA][46] = 127, + [2][1][RTW89_CN][46] = 56, + [2][1][RTW89_UK][46] = 127, + [2][1][RTW89_FCC][48] = 127, + [2][1][RTW89_ETSI][48] = 127, + [2][1][RTW89_MKK][48] = 127, + [2][1][RTW89_IC][48] = 127, + [2][1][RTW89_KCC][48] = 127, + [2][1][RTW89_ACMA][48] = 127, + [2][1][RTW89_CN][48] = 127, + [2][1][RTW89_UK][48] = 127, + [2][1][RTW89_FCC][50] = 127, + [2][1][RTW89_ETSI][50] = 127, + [2][1][RTW89_MKK][50] = 127, + [2][1][RTW89_IC][50] = 127, + [2][1][RTW89_KCC][50] = 127, + [2][1][RTW89_ACMA][50] = 127, + [2][1][RTW89_CN][50] = 127, + [2][1][RTW89_UK][50] = 127, + [2][1][RTW89_FCC][52] = 127, + [2][1][RTW89_ETSI][52] = 127, + [2][1][RTW89_MKK][52] = 127, + [2][1][RTW89_IC][52] = 127, + [2][1][RTW89_KCC][52] = 127, + [2][1][RTW89_ACMA][52] = 127, + [2][1][RTW89_CN][52] = 127, + [2][1][RTW89_UK][52] = 127, +}; + +const struct rtw89_phy_table rtw89_8851b_phy_bb_table = { + .regs = rtw89_8851b_phy_bb_regs, + .n_regs = ARRAY_SIZE(rtw89_8851b_phy_bb_regs), + .rf_path = 0, /* don't care */ +}; + +const struct rtw89_phy_table rtw89_8851b_phy_bb_gain_table = { + .regs = rtw89_8851b_phy_bb_reg_gain, + .n_regs = ARRAY_SIZE(rtw89_8851b_phy_bb_reg_gain), + .rf_path = 0, /* don't care */ +}; + +const struct rtw89_phy_table rtw89_8851b_phy_radioa_table = { + .regs = rtw89_8851b_phy_radioa_regs, + .n_regs = ARRAY_SIZE(rtw89_8851b_phy_radioa_regs), + .rf_path = RF_PATH_A, + .config = rtw89_phy_config_rf_reg_v1, +}; + +const struct rtw89_phy_table rtw89_8851b_phy_nctl_table = { + .regs = rtw89_8851b_phy_nctl_regs, + .n_regs = ARRAY_SIZE(rtw89_8851b_phy_nctl_regs), + .rf_path = 0, /* don't care */ +}; + +const struct rtw89_txpwr_table rtw89_8851b_byr_table = { + .data = rtw89_8851b_txpwr_byrate, + .size = ARRAY_SIZE(rtw89_8851b_txpwr_byrate), + .load = rtw89_phy_load_txpwr_byrate, +}; + +const struct rtw89_txpwr_track_cfg rtw89_8851b_trk_cfg = { + .delta_swingidx_5ga_n = _txpwr_track_delta_swingidx_5ga_n, + .delta_swingidx_5ga_p = _txpwr_track_delta_swingidx_5ga_p, + .delta_swingidx_2ga_n = _txpwr_track_delta_swingidx_2ga_n, + .delta_swingidx_2ga_p = _txpwr_track_delta_swingidx_2ga_p, + .delta_swingidx_2g_cck_a_n = _txpwr_track_delta_swingidx_2g_cck_a_n, + .delta_swingidx_2g_cck_a_p = _txpwr_track_delta_swingidx_2g_cck_a_p, +}; + +const struct rtw89_rfe_parms rtw89_8851b_dflt_parms = { + .rule_2ghz = { + .lmt = &rtw89_8851b_txpwr_lmt_2g, + .lmt_ru = &rtw89_8851b_txpwr_lmt_ru_2g, + }, + .rule_5ghz = { + .lmt = &rtw89_8851b_txpwr_lmt_5g, + .lmt_ru = &rtw89_8851b_txpwr_lmt_ru_5g, + }, +}; + +static const struct rtw89_rfe_parms rtw89_8851b_rfe_parms_type2 = { + .rule_2ghz = { + .lmt = &rtw89_8851b_txpwr_lmt_2g_type2, + .lmt_ru = &rtw89_8851b_txpwr_lmt_ru_2g_type2, + }, + .rule_5ghz = { + .lmt = &rtw89_8851b_txpwr_lmt_5g_type2, + .lmt_ru = &rtw89_8851b_txpwr_lmt_ru_5g_type2, + }, +}; + +const struct rtw89_rfe_parms_conf rtw89_8851b_rfe_parms_conf[] = { + { + .rfe_parms = &rtw89_8851b_rfe_parms_type2, + .rfe_type = 2, + }, + {}, +}; diff --git a/drivers/net/wireless/realtek/rtw89/rtw8851b_table.h b/drivers/net/wireless/realtek/rtw89/rtw8851b_table.h new file mode 100644 index 000000000000..f2e673ba39c8 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw89/rtw8851b_table.h @@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2022-2023 Realtek Corporation + */ + +#ifndef __RTW89_8851B_TABLE_H__ +#define __RTW89_8851B_TABLE_H__ + +#include "core.h" + +extern const struct rtw89_phy_table rtw89_8851b_phy_bb_table; +extern const struct rtw89_phy_table rtw89_8851b_phy_bb_gain_table; +extern const struct rtw89_phy_table rtw89_8851b_phy_radioa_table; +extern const struct rtw89_phy_table rtw89_8851b_phy_nctl_table; +extern const struct rtw89_txpwr_table rtw89_8851b_byr_table; +extern const struct rtw89_txpwr_track_cfg rtw89_8851b_trk_cfg; +extern const u8 rtw89_8851b_tx_shape[RTW89_BAND_MAX][RTW89_RS_TX_SHAPE_NUM] + [RTW89_REGD_NUM]; +extern const struct rtw89_rfe_parms rtw89_8851b_dflt_parms; +extern const struct rtw89_rfe_parms_conf rtw89_8851b_rfe_parms_conf[]; + +#endif diff --git a/drivers/net/wireless/realtek/rtw89/rtw8852a.c b/drivers/net/wireless/realtek/rtw89/rtw8852a.c index 46907748778b..d7930efd89b7 100644 --- a/drivers/net/wireless/realtek/rtw89/rtw8852a.c +++ b/drivers/net/wireless/realtek/rtw89/rtw8852a.c @@ -12,6 +12,11 @@ #include "rtw8852a_table.h" #include "txrx.h" +#define RTW8852A_FW_FORMAT_MAX 0 +#define RTW8852A_FW_BASENAME "rtw89/rtw8852a_fw" +#define RTW8852A_MODULE_FIRMWARE \ + RTW8852A_FW_BASENAME ".bin" + static const struct rtw89_hfc_ch_cfg rtw8852a_hfc_chcfg_pcie[] = { {128, 1896, grp_0}, /* ACH 0 */ {128, 1896, grp_0}, /* ACH 1 */ @@ -1827,7 +1832,8 @@ rtw8852a_btc_set_wl_txpwr_ctrl(struct rtw89_dev *rtwdev, u32 txpwr_val) static s8 rtw8852a_btc_get_bt_rssi(struct rtw89_dev *rtwdev, s8 val) { - return clamp_t(s8, val, -100, 0) + 100; + /* +6 for compensate offset */ + return clamp_t(s8, val + 6, -100, 0) + 100; } static struct rtw89_btc_rf_trx_para rtw89_btc_8852a_rf_ul[] = { @@ -2059,7 +2065,8 @@ static const struct rtw89_chip_ops rtw8852a_chip_ops = { const struct rtw89_chip_info rtw8852a_chip_info = { .chip_id = RTL8852A, .ops = &rtw8852a_chip_ops, - .fw_name = "rtw89/rtw8852a_fw.bin", + .fw_basename = RTW8852A_FW_BASENAME, + .fw_format_max = RTW8852A_FW_FORMAT_MAX, .try_ce_fw = false, .fifo_size = 458752, .dle_scc_rsvd_size = 0, @@ -2079,10 +2086,8 @@ const struct rtw89_chip_info rtw8852a_chip_info = { &rtw89_8852a_phy_radiob_table,}, .nctl_table = &rtw89_8852a_phy_nctl_table, .byr_table = &rtw89_8852a_byr_table, - .txpwr_lmt_2g = &rtw89_8852a_txpwr_lmt_2g, - .txpwr_lmt_5g = &rtw89_8852a_txpwr_lmt_5g, - .txpwr_lmt_ru_2g = &rtw89_8852a_txpwr_lmt_ru_2g, - .txpwr_lmt_ru_5g = &rtw89_8852a_txpwr_lmt_ru_5g, + .dflt_parms = &rtw89_8852a_dflt_parms, + .rfe_parms_conf = NULL, .txpwr_factor_rf = 2, .txpwr_factor_mac = 1, .dig_table = &rtw89_8852a_phy_dig_table, @@ -2143,8 +2148,9 @@ const struct rtw89_chip_info rtw8852a_chip_info = { .c2h_counter_reg = {R_AX_UDM1 + 1, B_AX_UDM1_HALMAC_C2H_ENQ_CNT_MASK >> 8}, .page_regs = &rtw8852a_page_regs, .cfo_src_fd = false, + .cfo_hw_comp = false, .dcfo_comp = &rtw8852a_dcfo_comp, - .dcfo_comp_sft = 3, + .dcfo_comp_sft = 10, .imr_info = &rtw8852a_imr_info, .rrsr_cfgs = &rtw8852a_rrsr_cfgs, .bss_clr_map_reg = R_BSS_CLR_MAP, @@ -2156,7 +2162,7 @@ const struct rtw89_chip_info rtw8852a_chip_info = { }; EXPORT_SYMBOL(rtw8852a_chip_info); -MODULE_FIRMWARE("rtw89/rtw8852a_fw.bin"); +MODULE_FIRMWARE(RTW8852A_MODULE_FIRMWARE); MODULE_AUTHOR("Realtek Corporation"); MODULE_DESCRIPTION("Realtek 802.11ax wireless 8852A driver"); MODULE_LICENSE("Dual BSD/GPL"); diff --git a/drivers/net/wireless/realtek/rtw89/rtw8852a_table.c b/drivers/net/wireless/realtek/rtw89/rtw8852a_table.c index 320bcd4852c6..be54194558ff 100644 --- a/drivers/net/wireless/realtek/rtw89/rtw8852a_table.c +++ b/drivers/net/wireless/realtek/rtw89/rtw8852a_table.c @@ -43377,6 +43377,7 @@ static const s8 _txpwr_track_delta_swingidx_2g_cck_a_p[] = { 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5, 6, 6, 6, 7, 7, 7, 8, 8, 8, 9, 9, 9, 10, 10}; +static const s8 rtw89_8852a_txpwr_lmt_2g[RTW89_2G_BW_NUM][RTW89_NTX_NUM] [RTW89_RS_LMT_NUM][RTW89_BF_NUM] [RTW89_REGD_NUM][RTW89_2G_CH_NUM] = { @@ -45566,6 +45567,7 @@ const s8 rtw89_8852a_txpwr_lmt_2g[RTW89_2G_BW_NUM][RTW89_NTX_NUM] [1][1][2][1][RTW89_UK][13] = 127, }; +static const s8 rtw89_8852a_txpwr_lmt_5g[RTW89_5G_BW_NUM][RTW89_NTX_NUM] [RTW89_RS_LMT_NUM][RTW89_BF_NUM] [RTW89_REGD_NUM][RTW89_5G_CH_NUM] = { @@ -47898,6 +47900,7 @@ const s8 rtw89_8852a_txpwr_lmt_5g[RTW89_5G_BW_NUM][RTW89_NTX_NUM] [2][1][2][1][RTW89_UK][41] = 40, }; +static const s8 rtw89_8852a_txpwr_lmt_ru_2g[RTW89_RU_NUM][RTW89_NTX_NUM] [RTW89_REGD_NUM][RTW89_2G_CH_NUM] = { [0][0][RTW89_WW][0] = 32, @@ -48994,6 +48997,7 @@ const s8 rtw89_8852a_txpwr_lmt_ru_2g[RTW89_RU_NUM][RTW89_NTX_NUM] [2][1][RTW89_UK][13] = 127, }; +static const s8 rtw89_8852a_txpwr_lmt_ru_5g[RTW89_RU_NUM][RTW89_NTX_NUM] [RTW89_REGD_NUM][RTW89_5G_CH_NUM] = { [0][0][RTW89_WW][0] = 22, @@ -51043,3 +51047,14 @@ const struct rtw89_phy_dig_gain_table rtw89_8852a_phy_dig_table = { .cfg_lna_a = &rtw89_8852a_lna_gain_a_table, .cfg_tia_a = &rtw89_8852a_tia_gain_a_table }; + +const struct rtw89_rfe_parms rtw89_8852a_dflt_parms = { + .rule_2ghz = { + .lmt = &rtw89_8852a_txpwr_lmt_2g, + .lmt_ru = &rtw89_8852a_txpwr_lmt_ru_2g, + }, + .rule_5ghz = { + .lmt = &rtw89_8852a_txpwr_lmt_5g, + .lmt_ru = &rtw89_8852a_txpwr_lmt_ru_5g, + }, +}; diff --git a/drivers/net/wireless/realtek/rtw89/rtw8852a_table.h b/drivers/net/wireless/realtek/rtw89/rtw8852a_table.h index 913796506286..41c379b1044d 100644 --- a/drivers/net/wireless/realtek/rtw89/rtw8852a_table.h +++ b/drivers/net/wireless/realtek/rtw89/rtw8852a_table.h @@ -14,15 +14,6 @@ extern const struct rtw89_phy_table rtw89_8852a_phy_nctl_table; extern const struct rtw89_txpwr_table rtw89_8852a_byr_table; extern const struct rtw89_phy_dig_gain_table rtw89_8852a_phy_dig_table; extern const struct rtw89_txpwr_track_cfg rtw89_8852a_trk_cfg; -extern const s8 rtw89_8852a_txpwr_lmt_2g[RTW89_2G_BW_NUM][RTW89_NTX_NUM] - [RTW89_RS_LMT_NUM][RTW89_BF_NUM] - [RTW89_REGD_NUM][RTW89_2G_CH_NUM]; -extern const s8 rtw89_8852a_txpwr_lmt_5g[RTW89_5G_BW_NUM][RTW89_NTX_NUM] - [RTW89_RS_LMT_NUM][RTW89_BF_NUM] - [RTW89_REGD_NUM][RTW89_5G_CH_NUM]; -extern const s8 rtw89_8852a_txpwr_lmt_ru_2g[RTW89_RU_NUM][RTW89_NTX_NUM] - [RTW89_REGD_NUM][RTW89_2G_CH_NUM]; -extern const s8 rtw89_8852a_txpwr_lmt_ru_5g[RTW89_RU_NUM][RTW89_NTX_NUM] - [RTW89_REGD_NUM][RTW89_5G_CH_NUM]; +extern const struct rtw89_rfe_parms rtw89_8852a_dflt_parms; #endif diff --git a/drivers/net/wireless/realtek/rtw89/rtw8852b.c b/drivers/net/wireless/realtek/rtw89/rtw8852b.c index bae80984cfd5..eaa2ea0586bc 100644 --- a/drivers/net/wireless/realtek/rtw89/rtw8852b.c +++ b/drivers/net/wireless/realtek/rtw89/rtw8852b.c @@ -12,6 +12,11 @@ #include "rtw8852b_table.h" #include "txrx.h" +#define RTW8852B_FW_FORMAT_MAX 1 +#define RTW8852B_FW_BASENAME "rtw89/rtw8852b_fw" +#define RTW8852B_MODULE_FIRMWARE \ + RTW8852B_FW_BASENAME "-" __stringify(RTW8852B_FW_FORMAT_MAX) ".bin" + static const struct rtw89_hfc_ch_cfg rtw8852b_hfc_chcfg_pcie[] = { {5, 343, grp_0}, /* ACH 0 */ {5, 343, grp_0}, /* ACH 1 */ @@ -48,6 +53,10 @@ static const struct rtw89_dle_mem rtw8852b_dle_mem_pcie[] = { &rtw89_mac_size.ple_size6, &rtw89_mac_size.wde_qt6, &rtw89_mac_size.wde_qt6, &rtw89_mac_size.ple_qt18, &rtw89_mac_size.ple_qt58}, + [RTW89_QTA_WOW] = {RTW89_QTA_WOW, &rtw89_mac_size.wde_size6, + &rtw89_mac_size.ple_size6, &rtw89_mac_size.wde_qt6, + &rtw89_mac_size.wde_qt6, &rtw89_mac_size.ple_qt18, + &rtw89_mac_size.ple_qt_52b_wow}, [RTW89_QTA_DLFW] = {RTW89_QTA_DLFW, &rtw89_mac_size.wde_size9, &rtw89_mac_size.ple_size8, &rtw89_mac_size.wde_qt4, &rtw89_mac_size.wde_qt4, &rtw89_mac_size.ple_qt13, @@ -323,7 +332,7 @@ static const struct rtw89_btc_rf_trx_para rtw89_btc_8852b_rf_ul[] = { {255, 0, 0, 7}, /* 2 ->reserved for shared-antenna */ {255, 0, 0, 7}, /* 3- >reserved for shared-antenna */ {255, 0, 0, 7}, /* 4 ->reserved for shared-antenna */ - {255, 0, 0, 7}, /* the below id is for non-shared-antenna free-run */ + {255, 1, 0, 7}, /* the below id is for non-shared-antenna free-run */ {6, 1, 0, 7}, {13, 1, 0, 7}, {13, 1, 0, 7} @@ -335,7 +344,7 @@ static const struct rtw89_btc_rf_trx_para rtw89_btc_8852b_rf_dl[] = { {255, 0, 0, 7}, /* 2 ->reserved for shared-antenna */ {255, 0, 0, 7}, /* 3- >reserved for shared-antenna */ {255, 0, 0, 7}, /* 4 ->reserved for shared-antenna */ - {255, 0, 0, 7}, /* the below id is for non-shared-antenna free-run */ + {255, 1, 0, 7}, /* the below id is for non-shared-antenna free-run */ {255, 1, 0, 7}, {255, 1, 0, 7}, {255, 1, 0, 7} @@ -355,7 +364,9 @@ static const struct rtw89_btc_fbtc_mreg rtw89_btc_8852b_mon_reg[] = { RTW89_DEF_FBTC_MREG(REG_MAC, 4, 0xd200), RTW89_DEF_FBTC_MREG(REG_MAC, 4, 0xd220), RTW89_DEF_FBTC_MREG(REG_BB, 4, 0x980), - RTW89_DEF_FBTC_MREG(REG_BT_MODEM, 4, 0x178), + RTW89_DEF_FBTC_MREG(REG_BB, 4, 0x4738), + RTW89_DEF_FBTC_MREG(REG_BB, 4, 0x4688), + RTW89_DEF_FBTC_MREG(REG_BB, 4, 0x4694), }; static const u8 rtw89_btc_8852b_wl_rssi_thres[BTC_WL_RSSI_THMAX] = {70, 60, 50, 40}; @@ -1284,7 +1295,7 @@ static void rtw8852b_ctrl_cck_en(struct rtw89_dev *rtwdev, bool cck_en) static void rtw8852b_5m_mask(struct rtw89_dev *rtwdev, const struct rtw89_chan *chan, enum rtw89_phy_idx phy_idx) { - u8 pri_ch = chan->primary_channel; + u8 pri_ch = chan->pri_ch_idx; bool mask_5m_low; bool mask_5m_en; @@ -1292,12 +1303,13 @@ static void rtw8852b_5m_mask(struct rtw89_dev *rtwdev, const struct rtw89_chan * case RTW89_CHANNEL_WIDTH_40: /* Prich=1: Mask 5M High, Prich=2: Mask 5M Low */ mask_5m_en = true; - mask_5m_low = pri_ch == 2; + mask_5m_low = pri_ch == RTW89_SC_20_LOWER; break; case RTW89_CHANNEL_WIDTH_80: /* Prich=3: Mask 5M High, Prich=4: Mask 5M Low, Else: Disable */ - mask_5m_en = pri_ch == 3 || pri_ch == 4; - mask_5m_low = pri_ch == 4; + mask_5m_en = pri_ch == RTW89_SC_20_UPMOST || + pri_ch == RTW89_SC_20_LOWEST; + mask_5m_low = pri_ch == RTW89_SC_20_LOWEST; break; default: mask_5m_en = false; @@ -2267,7 +2279,8 @@ do { \ static s8 rtw8852b_btc_get_bt_rssi(struct rtw89_dev *rtwdev, s8 val) { - return clamp_t(s8, val, -100, 0) + 100; + /* +6 for compensate offset */ + return clamp_t(s8, val + 6, -100, 0) + 100; } static @@ -2477,10 +2490,20 @@ static const struct rtw89_chip_ops rtw8852b_chip_ops = { .btc_set_policy = rtw89_btc_set_policy_v1, }; +#ifdef CONFIG_PM +static const struct wiphy_wowlan_support rtw_wowlan_stub_8852b = { + .flags = WIPHY_WOWLAN_MAGIC_PKT | WIPHY_WOWLAN_DISCONNECT, + .n_patterns = RTW89_MAX_PATTERN_NUM, + .pattern_max_len = RTW89_MAX_PATTERN_SIZE, + .pattern_min_len = 1, +}; +#endif + const struct rtw89_chip_info rtw8852b_chip_info = { .chip_id = RTL8852B, .ops = &rtw8852b_chip_ops, - .fw_name = "rtw89/rtw8852b_fw.bin", + .fw_basename = RTW8852B_FW_BASENAME, + .fw_format_max = RTW8852B_FW_FORMAT_MAX, .try_ce_fw = true, .fifo_size = 196608, .dle_scc_rsvd_size = 98304, @@ -2500,10 +2523,8 @@ const struct rtw89_chip_info rtw8852b_chip_info = { &rtw89_8852b_phy_radiob_table,}, .nctl_table = &rtw89_8852b_phy_nctl_table, .byr_table = &rtw89_8852b_byr_table, - .txpwr_lmt_2g = &rtw89_8852b_txpwr_lmt_2g, - .txpwr_lmt_5g = &rtw89_8852b_txpwr_lmt_5g, - .txpwr_lmt_ru_2g = &rtw89_8852b_txpwr_lmt_ru_2g, - .txpwr_lmt_ru_5g = &rtw89_8852b_txpwr_lmt_ru_5g, + .dflt_parms = &rtw89_8852b_dflt_parms, + .rfe_parms_conf = NULL, .txpwr_factor_rf = 2, .txpwr_factor_mac = 1, .dig_table = NULL, @@ -2564,8 +2585,9 @@ const struct rtw89_chip_info rtw8852b_chip_info = { .c2h_regs = rtw8852b_c2h_regs, .page_regs = &rtw8852b_page_regs, .cfo_src_fd = true, + .cfo_hw_comp = true, .dcfo_comp = &rtw8852b_dcfo_comp, - .dcfo_comp_sft = 3, + .dcfo_comp_sft = 10, .imr_info = &rtw8852b_imr_info, .rrsr_cfgs = &rtw8852b_rrsr_cfgs, .bss_clr_map_reg = R_BSS_CLR_MAP_V1, @@ -2573,10 +2595,13 @@ const struct rtw89_chip_info rtw8852b_chip_info = { BIT(RTW89_DMA_ACH6) | BIT(RTW89_DMA_ACH7) | BIT(RTW89_DMA_B1MG) | BIT(RTW89_DMA_B1HI), .edcca_lvl_reg = R_SEG0R_EDCCA_LVL_V1, +#ifdef CONFIG_PM + .wowlan_stub = &rtw_wowlan_stub_8852b, +#endif }; EXPORT_SYMBOL(rtw8852b_chip_info); -MODULE_FIRMWARE("rtw89/rtw8852b_fw.bin"); +MODULE_FIRMWARE(RTW8852B_MODULE_FIRMWARE); MODULE_AUTHOR("Realtek Corporation"); MODULE_DESCRIPTION("Realtek 802.11ax wireless 8852B driver"); MODULE_LICENSE("Dual BSD/GPL"); diff --git a/drivers/net/wireless/realtek/rtw89/rtw8852b_table.c b/drivers/net/wireless/realtek/rtw89/rtw8852b_table.c index a6734965361f..904cdb9e56fa 100644 --- a/drivers/net/wireless/realtek/rtw89/rtw8852b_table.c +++ b/drivers/net/wireless/realtek/rtw89/rtw8852b_table.c @@ -14706,6 +14706,7 @@ const u8 rtw89_8852b_tx_shape[RTW89_BAND_MAX][RTW89_RS_TX_SHAPE_NUM] [1][1][RTW89_UKRAINE] = 0, }; +static const s8 rtw89_8852b_txpwr_lmt_2g[RTW89_2G_BW_NUM][RTW89_NTX_NUM] [RTW89_RS_LMT_NUM][RTW89_BF_NUM] [RTW89_REGD_NUM][RTW89_2G_CH_NUM] = { @@ -16895,6 +16896,7 @@ const s8 rtw89_8852b_txpwr_lmt_2g[RTW89_2G_BW_NUM][RTW89_NTX_NUM] [1][1][2][1][RTW89_UK][13] = 127, }; +static const s8 rtw89_8852b_txpwr_lmt_5g[RTW89_5G_BW_NUM][RTW89_NTX_NUM] [RTW89_RS_LMT_NUM][RTW89_BF_NUM] [RTW89_REGD_NUM][RTW89_5G_CH_NUM] = { @@ -19539,6 +19541,7 @@ const s8 rtw89_8852b_txpwr_lmt_5g[RTW89_5G_BW_NUM][RTW89_NTX_NUM] [2][1][2][1][RTW89_UK][49] = 127, }; +static const s8 rtw89_8852b_txpwr_lmt_ru_2g[RTW89_RU_NUM][RTW89_NTX_NUM] [RTW89_REGD_NUM][RTW89_2G_CH_NUM] = { [0][0][RTW89_WW][0] = 32, @@ -20635,6 +20638,7 @@ const s8 rtw89_8852b_txpwr_lmt_ru_2g[RTW89_RU_NUM][RTW89_NTX_NUM] [2][1][RTW89_UK][13] = 127, }; +static const s8 rtw89_8852b_txpwr_lmt_ru_5g[RTW89_RU_NUM][RTW89_NTX_NUM] [RTW89_REGD_NUM][RTW89_5G_CH_NUM] = { [0][0][RTW89_WW][0] = 24, @@ -22875,3 +22879,14 @@ const struct rtw89_txpwr_track_cfg rtw89_8852b_trk_cfg = { .delta_swingidx_2g_cck_a_n = _txpwr_track_delta_swingidx_2g_cck_a_n, .delta_swingidx_2g_cck_a_p = _txpwr_track_delta_swingidx_2g_cck_a_p, }; + +const struct rtw89_rfe_parms rtw89_8852b_dflt_parms = { + .rule_2ghz = { + .lmt = &rtw89_8852b_txpwr_lmt_2g, + .lmt_ru = &rtw89_8852b_txpwr_lmt_ru_2g, + }, + .rule_5ghz = { + .lmt = &rtw89_8852b_txpwr_lmt_5g, + .lmt_ru = &rtw89_8852b_txpwr_lmt_ru_5g, + }, +}; diff --git a/drivers/net/wireless/realtek/rtw89/rtw8852b_table.h b/drivers/net/wireless/realtek/rtw89/rtw8852b_table.h index 114337ac9fb0..5f4161496a58 100644 --- a/drivers/net/wireless/realtek/rtw89/rtw8852b_table.h +++ b/drivers/net/wireless/realtek/rtw89/rtw8852b_table.h @@ -16,15 +16,6 @@ extern const struct rtw89_txpwr_table rtw89_8852b_byr_table; extern const struct rtw89_txpwr_track_cfg rtw89_8852b_trk_cfg; extern const u8 rtw89_8852b_tx_shape[RTW89_BAND_MAX][RTW89_RS_TX_SHAPE_NUM] [RTW89_REGD_NUM]; -extern const s8 rtw89_8852b_txpwr_lmt_2g[RTW89_2G_BW_NUM][RTW89_NTX_NUM] - [RTW89_RS_LMT_NUM][RTW89_BF_NUM] - [RTW89_REGD_NUM][RTW89_2G_CH_NUM]; -extern const s8 rtw89_8852b_txpwr_lmt_5g[RTW89_5G_BW_NUM][RTW89_NTX_NUM] - [RTW89_RS_LMT_NUM][RTW89_BF_NUM] - [RTW89_REGD_NUM][RTW89_5G_CH_NUM]; -extern const s8 rtw89_8852b_txpwr_lmt_ru_2g[RTW89_RU_NUM][RTW89_NTX_NUM] - [RTW89_REGD_NUM][RTW89_2G_CH_NUM]; -extern const s8 rtw89_8852b_txpwr_lmt_ru_5g[RTW89_RU_NUM][RTW89_NTX_NUM] - [RTW89_REGD_NUM][RTW89_5G_CH_NUM]; +extern const struct rtw89_rfe_parms rtw89_8852b_dflt_parms; #endif diff --git a/drivers/net/wireless/realtek/rtw89/rtw8852c.c b/drivers/net/wireless/realtek/rtw89/rtw8852c.c index ba728fca9188..ceb819a62efc 100644 --- a/drivers/net/wireless/realtek/rtw89/rtw8852c.c +++ b/drivers/net/wireless/realtek/rtw89/rtw8852c.c @@ -13,6 +13,11 @@ #include "rtw8852c_table.h" #include "util.h" +#define RTW8852C_FW_FORMAT_MAX 0 +#define RTW8852C_FW_BASENAME "rtw89/rtw8852c_fw" +#define RTW8852C_MODULE_FIRMWARE \ + RTW8852C_FW_BASENAME ".bin" + static const struct rtw89_hfc_ch_cfg rtw8852c_hfc_chcfg_pcie[] = { {13, 1614, grp_0}, /* ACH 0 */ {13, 1614, grp_0}, /* ACH 1 */ @@ -1375,18 +1380,19 @@ static void rtw8852c_5m_mask(struct rtw89_dev *rtwdev, const struct rtw89_chan *chan, enum rtw89_phy_idx phy_idx) { - u8 pri_ch = chan->primary_channel; + u8 pri_ch = chan->pri_ch_idx; bool mask_5m_low; bool mask_5m_en; switch (chan->band_width) { case RTW89_CHANNEL_WIDTH_40: mask_5m_en = true; - mask_5m_low = pri_ch == 2; + mask_5m_low = pri_ch == RTW89_SC_20_LOWER; break; case RTW89_CHANNEL_WIDTH_80: - mask_5m_en = ((pri_ch == 3) || (pri_ch == 4)); - mask_5m_low = pri_ch == 4; + mask_5m_en = pri_ch == RTW89_SC_20_UPMOST || + pri_ch == RTW89_SC_20_LOWEST; + mask_5m_low = pri_ch == RTW89_SC_20_LOWEST; break; default: mask_5m_en = false; @@ -2527,7 +2533,8 @@ do { \ static s8 rtw8852c_btc_get_bt_rssi(struct rtw89_dev *rtwdev, s8 val) { - return clamp_t(s8, val, -100, 0) + 100; + /* +6 for compensate offset */ + return clamp_t(s8, val + 6, -100, 0) + 100; } static const struct rtw89_btc_rf_trx_para rtw89_btc_8852c_rf_ul[] = { @@ -2536,7 +2543,7 @@ static const struct rtw89_btc_rf_trx_para rtw89_btc_8852c_rf_ul[] = { {255, 0, 0, 7}, /* 2 ->reserved for shared-antenna */ {255, 0, 0, 7}, /* 3- >reserved for shared-antenna */ {255, 0, 0, 7}, /* 4 ->reserved for shared-antenna */ - {255, 0, 0, 7}, /* the below id is for non-shared-antenna free-run */ + {255, 1, 0, 7}, /* the below id is for non-shared-antenna free-run */ {6, 1, 0, 7}, {13, 1, 0, 7}, {13, 1, 0, 7} @@ -2548,7 +2555,7 @@ static const struct rtw89_btc_rf_trx_para rtw89_btc_8852c_rf_dl[] = { {255, 0, 0, 7}, /* 2 ->reserved for shared-antenna */ {255, 0, 0, 7}, /* 3- >reserved for shared-antenna */ {255, 0, 0, 7}, /* 4 ->reserved for shared-antenna */ - {255, 0, 0, 7}, /* the below id is for non-shared-antenna free-run */ + {255, 1, 0, 7}, /* the below id is for non-shared-antenna free-run */ {255, 1, 0, 7}, {255, 1, 0, 7}, {255, 1, 0, 7} @@ -2570,6 +2577,9 @@ static const struct rtw89_btc_fbtc_mreg rtw89_btc_8852c_mon_reg[] = { RTW89_DEF_FBTC_MREG(REG_MAC, 4, 0xd200), RTW89_DEF_FBTC_MREG(REG_MAC, 4, 0xd220), RTW89_DEF_FBTC_MREG(REG_BB, 4, 0x980), + RTW89_DEF_FBTC_MREG(REG_BB, 4, 0x4aa4), + RTW89_DEF_FBTC_MREG(REG_BB, 4, 0x4778), + RTW89_DEF_FBTC_MREG(REG_BB, 4, 0x476c), }; static @@ -2791,7 +2801,8 @@ static const struct rtw89_chip_ops rtw8852c_chip_ops = { const struct rtw89_chip_info rtw8852c_chip_info = { .chip_id = RTL8852C, .ops = &rtw8852c_chip_ops, - .fw_name = "rtw89/rtw8852c_fw.bin", + .fw_basename = RTW8852C_FW_BASENAME, + .fw_format_max = RTW8852C_FW_FORMAT_MAX, .try_ce_fw = false, .fifo_size = 458752, .dle_scc_rsvd_size = 0, @@ -2811,12 +2822,8 @@ const struct rtw89_chip_info rtw8852c_chip_info = { &rtw89_8852c_phy_radioa_table,}, .nctl_table = &rtw89_8852c_phy_nctl_table, .byr_table = &rtw89_8852c_byr_table, - .txpwr_lmt_2g = &rtw89_8852c_txpwr_lmt_2g, - .txpwr_lmt_5g = &rtw89_8852c_txpwr_lmt_5g, - .txpwr_lmt_6g = &rtw89_8852c_txpwr_lmt_6g, - .txpwr_lmt_ru_2g = &rtw89_8852c_txpwr_lmt_ru_2g, - .txpwr_lmt_ru_5g = &rtw89_8852c_txpwr_lmt_ru_5g, - .txpwr_lmt_ru_6g = &rtw89_8852c_txpwr_lmt_ru_6g, + .dflt_parms = &rtw89_8852c_dflt_parms, + .rfe_parms_conf = NULL, .txpwr_factor_rf = 2, .txpwr_factor_mac = 1, .dig_table = NULL, @@ -2879,8 +2886,9 @@ const struct rtw89_chip_info rtw8852c_chip_info = { .c2h_regs = rtw8852c_c2h_regs, .page_regs = &rtw8852c_page_regs, .cfo_src_fd = false, + .cfo_hw_comp = false, .dcfo_comp = &rtw8852c_dcfo_comp, - .dcfo_comp_sft = 5, + .dcfo_comp_sft = 12, .imr_info = &rtw8852c_imr_info, .rrsr_cfgs = &rtw8852c_rrsr_cfgs, .bss_clr_map_reg = R_BSS_CLR_MAP, @@ -2892,7 +2900,7 @@ const struct rtw89_chip_info rtw8852c_chip_info = { }; EXPORT_SYMBOL(rtw8852c_chip_info); -MODULE_FIRMWARE("rtw89/rtw8852c_fw.bin"); +MODULE_FIRMWARE(RTW8852C_MODULE_FIRMWARE); MODULE_AUTHOR("Realtek Corporation"); MODULE_DESCRIPTION("Realtek 802.11ax wireless 8852C driver"); MODULE_LICENSE("Dual BSD/GPL"); diff --git a/drivers/net/wireless/realtek/rtw89/rtw8852c_table.c b/drivers/net/wireless/realtek/rtw89/rtw8852c_table.c index 96c264a057ff..7011e5a6f8fd 100644 --- a/drivers/net/wireless/realtek/rtw89/rtw8852c_table.c +++ b/drivers/net/wireless/realtek/rtw89/rtw8852c_table.c @@ -28590,6 +28590,7 @@ const u8 rtw89_8852c_tx_shape[RTW89_BAND_MAX][RTW89_RS_TX_SHAPE_NUM] [2][1][RTW89_KCC] = 0, }; +static const s8 rtw89_8852c_txpwr_lmt_2g[RTW89_2G_BW_NUM][RTW89_NTX_NUM] [RTW89_RS_LMT_NUM][RTW89_BF_NUM] [RTW89_REGD_NUM][RTW89_2G_CH_NUM] = { @@ -30107,6 +30108,7 @@ const s8 rtw89_8852c_txpwr_lmt_2g[RTW89_2G_BW_NUM][RTW89_NTX_NUM] [1][1][2][1][RTW89_UK][13] = 127, }; +static const s8 rtw89_8852c_txpwr_lmt_5g[RTW89_5G_BW_NUM][RTW89_NTX_NUM] [RTW89_RS_LMT_NUM][RTW89_BF_NUM] [RTW89_REGD_NUM][RTW89_5G_CH_NUM] = { @@ -32020,6 +32022,7 @@ const s8 rtw89_8852c_txpwr_lmt_5g[RTW89_5G_BW_NUM][RTW89_NTX_NUM] [3][1][2][1][RTW89_UK][45] = 127, }; +static const s8 rtw89_8852c_txpwr_lmt_6g[RTW89_6G_BW_NUM][RTW89_NTX_NUM] [RTW89_RS_LMT_NUM][RTW89_BF_NUM] [RTW89_REGD_NUM][RTW89_6G_CH_NUM] = { @@ -33977,6 +33980,7 @@ const s8 rtw89_8852c_txpwr_lmt_6g[RTW89_6G_BW_NUM][RTW89_NTX_NUM] [3][1][2][1][RTW89_KCC][112] = 127, }; +static const s8 rtw89_8852c_txpwr_lmt_ru_2g[RTW89_RU_NUM][RTW89_NTX_NUM] [RTW89_REGD_NUM][RTW89_2G_CH_NUM] = { [0][0][RTW89_WW][0] = 32, @@ -34737,6 +34741,7 @@ const s8 rtw89_8852c_txpwr_lmt_ru_2g[RTW89_RU_NUM][RTW89_NTX_NUM] [2][1][RTW89_UK][13] = 127, }; +static const s8 rtw89_8852c_txpwr_lmt_ru_5g[RTW89_RU_NUM][RTW89_NTX_NUM] [RTW89_REGD_NUM][RTW89_5G_CH_NUM] = { [0][0][RTW89_WW][0] = 16, @@ -36253,6 +36258,7 @@ const s8 rtw89_8852c_txpwr_lmt_ru_5g[RTW89_RU_NUM][RTW89_NTX_NUM] [2][1][RTW89_UK][52] = 127, }; +static const s8 rtw89_8852c_txpwr_lmt_ru_6g[RTW89_RU_NUM][RTW89_NTX_NUM] [RTW89_REGD_NUM][RTW89_6G_CH_NUM] = { [0][0][RTW89_WW][0] = -16, @@ -37472,3 +37478,18 @@ const struct rtw89_phy_tssi_dbw_table rtw89_8852c_tssi_dbw_table = { .data[RTW89_TSSI_BANDEDGE_MID] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, .data[RTW89_TSSI_BANDEDGE_HIGH] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, }; + +const struct rtw89_rfe_parms rtw89_8852c_dflt_parms = { + .rule_2ghz = { + .lmt = &rtw89_8852c_txpwr_lmt_2g, + .lmt_ru = &rtw89_8852c_txpwr_lmt_ru_2g, + }, + .rule_5ghz = { + .lmt = &rtw89_8852c_txpwr_lmt_5g, + .lmt_ru = &rtw89_8852c_txpwr_lmt_ru_5g, + }, + .rule_6ghz = { + .lmt = &rtw89_8852c_txpwr_lmt_6g, + .lmt_ru = &rtw89_8852c_txpwr_lmt_ru_6g, + }, +}; diff --git a/drivers/net/wireless/realtek/rtw89/rtw8852c_table.h b/drivers/net/wireless/realtek/rtw89/rtw8852c_table.h index 7d71a92e2d27..6da1849fb1fa 100644 --- a/drivers/net/wireless/realtek/rtw89/rtw8852c_table.h +++ b/drivers/net/wireless/realtek/rtw89/rtw8852c_table.h @@ -17,20 +17,6 @@ extern const struct rtw89_phy_tssi_dbw_table rtw89_8852c_tssi_dbw_table; extern const struct rtw89_txpwr_track_cfg rtw89_8852c_trk_cfg; extern const u8 rtw89_8852c_tx_shape[RTW89_BAND_MAX][RTW89_RS_TX_SHAPE_NUM] [RTW89_REGD_NUM]; -extern const s8 rtw89_8852c_txpwr_lmt_2g[RTW89_2G_BW_NUM][RTW89_NTX_NUM] - [RTW89_RS_LMT_NUM][RTW89_BF_NUM] - [RTW89_REGD_NUM][RTW89_2G_CH_NUM]; -extern const s8 rtw89_8852c_txpwr_lmt_5g[RTW89_5G_BW_NUM][RTW89_NTX_NUM] - [RTW89_RS_LMT_NUM][RTW89_BF_NUM] - [RTW89_REGD_NUM][RTW89_5G_CH_NUM]; -extern const s8 rtw89_8852c_txpwr_lmt_6g[RTW89_6G_BW_NUM][RTW89_NTX_NUM] - [RTW89_RS_LMT_NUM][RTW89_BF_NUM] - [RTW89_REGD_NUM][RTW89_6G_CH_NUM]; -extern const s8 rtw89_8852c_txpwr_lmt_ru_2g[RTW89_RU_NUM][RTW89_NTX_NUM] - [RTW89_REGD_NUM][RTW89_2G_CH_NUM]; -extern const s8 rtw89_8852c_txpwr_lmt_ru_5g[RTW89_RU_NUM][RTW89_NTX_NUM] - [RTW89_REGD_NUM][RTW89_5G_CH_NUM]; -extern const s8 rtw89_8852c_txpwr_lmt_ru_6g[RTW89_RU_NUM][RTW89_NTX_NUM] - [RTW89_REGD_NUM][RTW89_6G_CH_NUM]; +extern const struct rtw89_rfe_parms rtw89_8852c_dflt_parms; #endif diff --git a/drivers/net/wireless/realtek/rtw89/wow.c b/drivers/net/wireless/realtek/rtw89/wow.c index 7cff9c1d8d37..2ca8abb70f11 100644 --- a/drivers/net/wireless/realtek/rtw89/wow.c +++ b/drivers/net/wireless/realtek/rtw89/wow.c @@ -30,7 +30,7 @@ static void rtw89_wow_enter_lps(struct rtw89_dev *rtwdev) struct ieee80211_vif *wow_vif = rtwdev->wow.wow_vif; struct rtw89_vif *rtwvif = (struct rtw89_vif *)wow_vif->drv_priv; - rtw89_enter_lps(rtwdev, rtwvif); + rtw89_enter_lps(rtwdev, rtwvif, false); } static void rtw89_wow_leave_lps(struct rtw89_dev *rtwdev) diff --git a/drivers/net/wireless/virtual/mac80211_hwsim.c b/drivers/net/wireless/virtual/mac80211_hwsim.c index f446d8f6e1f6..2211fa58fe41 100644 --- a/drivers/net/wireless/virtual/mac80211_hwsim.c +++ b/drivers/net/wireless/virtual/mac80211_hwsim.c @@ -3760,6 +3760,9 @@ static int hwsim_pmsr_report_nl(struct sk_buff *msg, struct genl_info *info) int err; int rem; + if (!info->attrs[HWSIM_ATTR_ADDR_TRANSMITTER]) + return -EINVAL; + src = nla_data(info->attrs[HWSIM_ATTR_ADDR_TRANSMITTER]); data = get_hwsim_data_ref_from_addr(src); if (!data) diff --git a/drivers/net/wwan/wwan_core.c b/drivers/net/wwan/wwan_core.c index 2e1c01cf00a9..aa54fa6d5f90 100644 --- a/drivers/net/wwan/wwan_core.c +++ b/drivers/net/wwan/wwan_core.c @@ -492,6 +492,7 @@ struct wwan_port *wwan_create_port(struct device *parent, if (err) goto error_put_device; + dev_info(&wwandev->dev, "port %s attached\n", dev_name(&port->dev)); return port; error_put_device: @@ -517,6 +518,8 @@ void wwan_remove_port(struct wwan_port *port) skb_queue_purge(&port->rxq); dev_set_drvdata(&port->dev, NULL); + + dev_info(&wwandev->dev, "port %s disconnected\n", dev_name(&port->dev)); device_unregister(&port->dev); /* Release related wwan device */ diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 282d808400c5..cd7873de3121 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -3443,6 +3443,8 @@ static const struct pci_device_id nvme_id_table[] = { { PCI_DEVICE(0x1d97, 0x2269), /* Lexar NM760 */ .driver_data = NVME_QUIRK_BOGUS_NID | NVME_QUIRK_IGNORE_DEV_SUBNQN, }, + { PCI_DEVICE(0x10ec, 0x5763), /* TEAMGROUP T-FORCE CARDEA ZERO Z330 SSD */ + .driver_data = NVME_QUIRK_BOGUS_NID, }, { PCI_DEVICE(PCI_VENDOR_ID_AMAZON, 0x0061), .driver_data = NVME_QUIRK_DMA_ADDRESS_BITS_48, }, { PCI_DEVICE(PCI_VENDOR_ID_AMAZON, 0x0065), diff --git a/drivers/perf/amlogic/meson_g12_ddr_pmu.c b/drivers/perf/amlogic/meson_g12_ddr_pmu.c index a78fdb15e26c..8b643888d503 100644 --- a/drivers/perf/amlogic/meson_g12_ddr_pmu.c +++ b/drivers/perf/amlogic/meson_g12_ddr_pmu.c @@ -21,23 +21,23 @@ #define DMC_QOS_IRQ BIT(30) /* DMC bandwidth monitor register address offset */ -#define DMC_MON_G12_CTRL0 (0x20 << 2) -#define DMC_MON_G12_CTRL1 (0x21 << 2) -#define DMC_MON_G12_CTRL2 (0x22 << 2) -#define DMC_MON_G12_CTRL3 (0x23 << 2) -#define DMC_MON_G12_CTRL4 (0x24 << 2) -#define DMC_MON_G12_CTRL5 (0x25 << 2) -#define DMC_MON_G12_CTRL6 (0x26 << 2) -#define DMC_MON_G12_CTRL7 (0x27 << 2) -#define DMC_MON_G12_CTRL8 (0x28 << 2) - -#define DMC_MON_G12_ALL_REQ_CNT (0x29 << 2) -#define DMC_MON_G12_ALL_GRANT_CNT (0x2a << 2) -#define DMC_MON_G12_ONE_GRANT_CNT (0x2b << 2) -#define DMC_MON_G12_SEC_GRANT_CNT (0x2c << 2) -#define DMC_MON_G12_THD_GRANT_CNT (0x2d << 2) -#define DMC_MON_G12_FOR_GRANT_CNT (0x2e << 2) -#define DMC_MON_G12_TIMER (0x2f << 2) +#define DMC_MON_G12_CTRL0 (0x0 << 2) +#define DMC_MON_G12_CTRL1 (0x1 << 2) +#define DMC_MON_G12_CTRL2 (0x2 << 2) +#define DMC_MON_G12_CTRL3 (0x3 << 2) +#define DMC_MON_G12_CTRL4 (0x4 << 2) +#define DMC_MON_G12_CTRL5 (0x5 << 2) +#define DMC_MON_G12_CTRL6 (0x6 << 2) +#define DMC_MON_G12_CTRL7 (0x7 << 2) +#define DMC_MON_G12_CTRL8 (0x8 << 2) + +#define DMC_MON_G12_ALL_REQ_CNT (0x9 << 2) +#define DMC_MON_G12_ALL_GRANT_CNT (0xa << 2) +#define DMC_MON_G12_ONE_GRANT_CNT (0xb << 2) +#define DMC_MON_G12_SEC_GRANT_CNT (0xc << 2) +#define DMC_MON_G12_THD_GRANT_CNT (0xd << 2) +#define DMC_MON_G12_FOR_GRANT_CNT (0xe << 2) +#define DMC_MON_G12_TIMER (0xf << 2) /* Each bit represent a axi line */ PMU_FORMAT_ATTR(event, "config:0-7"); diff --git a/drivers/regulator/fan53555.c b/drivers/regulator/fan53555.c index 529963a7e4f5..41537c45f036 100644 --- a/drivers/regulator/fan53555.c +++ b/drivers/regulator/fan53555.c @@ -8,18 +8,19 @@ // Copyright (c) 2012 Marvell Technology Ltd. // Yunfan Zhang <[email protected]> +#include <linux/bits.h> +#include <linux/err.h> +#include <linux/i2c.h> #include <linux/module.h> +#include <linux/of_device.h> #include <linux/param.h> -#include <linux/err.h> #include <linux/platform_device.h> +#include <linux/regmap.h> #include <linux/regulator/driver.h> +#include <linux/regulator/fan53555.h> #include <linux/regulator/machine.h> #include <linux/regulator/of_regulator.h> -#include <linux/of_device.h> -#include <linux/i2c.h> #include <linux/slab.h> -#include <linux/regmap.h> -#include <linux/regulator/fan53555.h> /* Voltage setting */ #define FAN53555_VSEL0 0x00 @@ -60,7 +61,7 @@ #define TCS_VSEL1_MODE (1 << 6) #define TCS_SLEW_SHIFT 3 -#define TCS_SLEW_MASK (0x3 < 3) +#define TCS_SLEW_MASK GENMASK(4, 3) enum fan53555_vendor { FAN53526_VENDOR_FAIRCHILD = 0, diff --git a/drivers/regulator/sm5703-regulator.c b/drivers/regulator/sm5703-regulator.c index 05ad28fc4da8..229df7170792 100644 --- a/drivers/regulator/sm5703-regulator.c +++ b/drivers/regulator/sm5703-regulator.c @@ -42,6 +42,7 @@ static const int sm5703_buck_voltagemap[] = { .type = REGULATOR_VOLTAGE, \ .id = SM5703_USBLDO ## _id, \ .ops = &sm5703_regulator_ops_fixed, \ + .n_voltages = 1, \ .fixed_uV = SM5703_USBLDO_MICROVOLT, \ .enable_reg = SM5703_REG_USBLDO12, \ .enable_mask = SM5703_REG_EN_USBLDO ##_id, \ @@ -56,6 +57,7 @@ static const int sm5703_buck_voltagemap[] = { .type = REGULATOR_VOLTAGE, \ .id = SM5703_VBUS, \ .ops = &sm5703_regulator_ops_fixed, \ + .n_voltages = 1, \ .fixed_uV = SM5703_VBUS_MICROVOLT, \ .enable_reg = SM5703_REG_CNTL, \ .enable_mask = SM5703_OPERATION_MODE_MASK, \ diff --git a/drivers/scsi/ses.c b/drivers/scsi/ses.c index b11a9162e73a..b54f2c6c08c3 100644 --- a/drivers/scsi/ses.c +++ b/drivers/scsi/ses.c @@ -509,9 +509,6 @@ static int ses_enclosure_find_by_addr(struct enclosure_device *edev, int i; struct ses_component *scomp; - if (!edev->component[0].scratch) - return 0; - for (i = 0; i < edev->components; i++) { scomp = edev->component[i].scratch; if (scomp->addr != efd->addr) @@ -602,8 +599,10 @@ static void ses_enclosure_data_process(struct enclosure_device *edev, components++, type_ptr[0], name); - else + else if (components < edev->components) ecomp = &edev->component[components++]; + else + ecomp = ERR_PTR(-EINVAL); if (!IS_ERR(ecomp)) { if (addl_desc_ptr) { @@ -734,11 +733,6 @@ static int ses_intf_add(struct device *cdev, components += type_ptr[1]; } - if (components == 0) { - sdev_printk(KERN_WARNING, sdev, "enclosure has no enumerated components\n"); - goto err_free; - } - ses_dev->page1 = buf; ses_dev->page1_len = len; buf = NULL; @@ -780,9 +774,11 @@ static int ses_intf_add(struct device *cdev, buf = NULL; } page2_not_supported: - scomp = kcalloc(components, sizeof(struct ses_component), GFP_KERNEL); - if (!scomp) - goto err_free; + if (components > 0) { + scomp = kcalloc(components, sizeof(struct ses_component), GFP_KERNEL); + if (!scomp) + goto err_free; + } edev = enclosure_register(cdev->parent, dev_name(&sdev->sdev_gendev), components, &ses_enclosure_callbacks); diff --git a/drivers/spi/spi-rockchip-sfc.c b/drivers/spi/spi-rockchip-sfc.c index bd87d3c92dd3..69347b6bf60c 100644 --- a/drivers/spi/spi-rockchip-sfc.c +++ b/drivers/spi/spi-rockchip-sfc.c @@ -632,7 +632,7 @@ static int rockchip_sfc_probe(struct platform_device *pdev) if (ret) { dev_err(dev, "Failed to request irq\n"); - return ret; + goto err_irq; } ret = rockchip_sfc_init(sfc); diff --git a/drivers/tee/optee/call.c b/drivers/tee/optee/call.c index 290b1bb0e9cd..df5fb5410b72 100644 --- a/drivers/tee/optee/call.c +++ b/drivers/tee/optee/call.c @@ -488,7 +488,7 @@ static bool is_normal_memory(pgprot_t p) #elif defined(CONFIG_ARM64) return (pgprot_val(p) & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL); #else -#error "Unuspported architecture" +#error "Unsupported architecture" #endif } diff --git a/drivers/tee/tee_shm.c b/drivers/tee/tee_shm.c index b1c6231defad..673cf0359494 100644 --- a/drivers/tee/tee_shm.c +++ b/drivers/tee/tee_shm.c @@ -32,7 +32,7 @@ static int shm_get_kernel_pages(unsigned long start, size_t page_count, is_kmap_addr((void *)start))) return -EINVAL; - page = virt_to_page(start); + page = virt_to_page((void *)start); for (n = 0; n < page_count; n++) { pages[n] = page + n; get_page(pages[n]); diff --git a/drivers/thermal/intel/therm_throt.c b/drivers/thermal/intel/therm_throt.c index 2e22bb82b738..e69868e868eb 100644 --- a/drivers/thermal/intel/therm_throt.c +++ b/drivers/thermal/intel/therm_throt.c @@ -193,8 +193,67 @@ static const struct attribute_group thermal_attr_group = { #define THERM_THROT_POLL_INTERVAL HZ #define THERM_STATUS_PROCHOT_LOG BIT(1) -#define THERM_STATUS_CLEAR_CORE_MASK (BIT(1) | BIT(3) | BIT(5) | BIT(7) | BIT(9) | BIT(11) | BIT(13) | BIT(15)) -#define THERM_STATUS_CLEAR_PKG_MASK (BIT(1) | BIT(3) | BIT(5) | BIT(7) | BIT(9) | BIT(11)) +static u64 therm_intr_core_clear_mask; +static u64 therm_intr_pkg_clear_mask; + +static void thermal_intr_init_core_clear_mask(void) +{ + if (therm_intr_core_clear_mask) + return; + + /* + * Reference: Intel SDM Volume 4 + * "Table 2-2. IA-32 Architectural MSRs", MSR 0x19C + * IA32_THERM_STATUS. + */ + + /* + * Bit 1, 3, 5: CPUID.01H:EDX[22] = 1. This driver will not + * enable interrupts, when 0 as it checks for X86_FEATURE_ACPI. + */ + therm_intr_core_clear_mask = (BIT(1) | BIT(3) | BIT(5)); + + /* + * Bit 7 and 9: Thermal Threshold #1 and #2 log + * If CPUID.01H:ECX[8] = 1 + */ + if (boot_cpu_has(X86_FEATURE_TM2)) + therm_intr_core_clear_mask |= (BIT(7) | BIT(9)); + + /* Bit 11: Power Limitation log (R/WC0) If CPUID.06H:EAX[4] = 1 */ + if (boot_cpu_has(X86_FEATURE_PLN)) + therm_intr_core_clear_mask |= BIT(11); + + /* + * Bit 13: Current Limit log (R/WC0) If CPUID.06H:EAX[7] = 1 + * Bit 15: Cross Domain Limit log (R/WC0) If CPUID.06H:EAX[7] = 1 + */ + if (boot_cpu_has(X86_FEATURE_HWP)) + therm_intr_core_clear_mask |= (BIT(13) | BIT(15)); +} + +static void thermal_intr_init_pkg_clear_mask(void) +{ + if (therm_intr_pkg_clear_mask) + return; + + /* + * Reference: Intel SDM Volume 4 + * "Table 2-2. IA-32 Architectural MSRs", MSR 0x1B1 + * IA32_PACKAGE_THERM_STATUS. + */ + + /* All bits except BIT 26 depend on CPUID.06H: EAX[6] = 1 */ + if (boot_cpu_has(X86_FEATURE_PTS)) + therm_intr_pkg_clear_mask = (BIT(1) | BIT(3) | BIT(5) | BIT(7) | BIT(9) | BIT(11)); + + /* + * Intel SDM Volume 2A: Thermal and Power Management Leaf + * Bit 26: CPUID.06H: EAX[19] = 1 + */ + if (boot_cpu_has(X86_FEATURE_HFI)) + therm_intr_pkg_clear_mask |= BIT(26); +} /* * Clear the bits in package thermal status register for bit = 1 @@ -207,13 +266,10 @@ void thermal_clear_package_intr_status(int level, u64 bit_mask) if (level == CORE_LEVEL) { msr = MSR_IA32_THERM_STATUS; - msr_val = THERM_STATUS_CLEAR_CORE_MASK; + msr_val = therm_intr_core_clear_mask; } else { msr = MSR_IA32_PACKAGE_THERM_STATUS; - msr_val = THERM_STATUS_CLEAR_PKG_MASK; - if (boot_cpu_has(X86_FEATURE_HFI)) - msr_val |= BIT(26); - + msr_val = therm_intr_pkg_clear_mask; } msr_val &= ~bit_mask; @@ -708,6 +764,9 @@ void intel_init_thermal(struct cpuinfo_x86 *c) h = THERMAL_APIC_VECTOR | APIC_DM_FIXED | APIC_LVT_MASKED; apic_write(APIC_LVTTHMR, h); + thermal_intr_init_core_clear_mask(); + thermal_intr_init_pkg_clear_mask(); + rdmsr(MSR_IA32_THERM_INTERRUPT, l, h); if (cpu_has(c, X86_FEATURE_PLN) && !int_pln_enable) wrmsr(MSR_IA32_THERM_INTERRUPT, diff --git a/fs/cifs/smb2pdu.c b/fs/cifs/smb2pdu.c index 2b92132097dc..4245249dbba8 100644 --- a/fs/cifs/smb2pdu.c +++ b/fs/cifs/smb2pdu.c @@ -587,11 +587,15 @@ assemble_neg_contexts(struct smb2_negotiate_req *req, } +/* If invalid preauth context warn but use what we requested, SHA-512 */ static void decode_preauth_context(struct smb2_preauth_neg_context *ctxt) { unsigned int len = le16_to_cpu(ctxt->DataLength); - /* If invalid preauth context warn but use what we requested, SHA-512 */ + /* + * Caller checked that DataLength remains within SMB boundary. We still + * need to confirm that one HashAlgorithms member is accounted for. + */ if (len < MIN_PREAUTH_CTXT_DATA_LEN) { pr_warn_once("server sent bad preauth context\n"); return; @@ -610,7 +614,11 @@ static void decode_compress_ctx(struct TCP_Server_Info *server, { unsigned int len = le16_to_cpu(ctxt->DataLength); - /* sizeof compress context is a one element compression capbility struct */ + /* + * Caller checked that DataLength remains within SMB boundary. We still + * need to confirm that one CompressionAlgorithms member is accounted + * for. + */ if (len < 10) { pr_warn_once("server sent bad compression cntxt\n"); return; @@ -632,6 +640,11 @@ static int decode_encrypt_ctx(struct TCP_Server_Info *server, unsigned int len = le16_to_cpu(ctxt->DataLength); cifs_dbg(FYI, "decode SMB3.11 encryption neg context of len %d\n", len); + /* + * Caller checked that DataLength remains within SMB boundary. We still + * need to confirm that one Cipher flexible array member is accounted + * for. + */ if (len < MIN_ENCRYPT_CTXT_DATA_LEN) { pr_warn_once("server sent bad crypto ctxt len\n"); return -EINVAL; @@ -678,6 +691,11 @@ static void decode_signing_ctx(struct TCP_Server_Info *server, { unsigned int len = le16_to_cpu(pctxt->DataLength); + /* + * Caller checked that DataLength remains within SMB boundary. We still + * need to confirm that one SigningAlgorithms flexible array member is + * accounted for. + */ if ((len < 4) || (len > 16)) { pr_warn_once("server sent bad signing negcontext\n"); return; @@ -719,14 +737,19 @@ static int smb311_decode_neg_context(struct smb2_negotiate_rsp *rsp, for (i = 0; i < ctxt_cnt; i++) { int clen; /* check that offset is not beyond end of SMB */ - if (len_of_ctxts == 0) - break; - if (len_of_ctxts < sizeof(struct smb2_neg_context)) break; pctx = (struct smb2_neg_context *)(offset + (char *)rsp); - clen = le16_to_cpu(pctx->DataLength); + clen = sizeof(struct smb2_neg_context) + + le16_to_cpu(pctx->DataLength); + /* + * 2.2.4 SMB2 NEGOTIATE Response + * Subsequent negotiate contexts MUST appear at the first 8-byte + * aligned offset following the previous negotiate context. + */ + if (i + 1 != ctxt_cnt) + clen = ALIGN(clen, 8); if (clen > len_of_ctxts) break; @@ -747,12 +770,10 @@ static int smb311_decode_neg_context(struct smb2_negotiate_rsp *rsp, else cifs_server_dbg(VFS, "unknown negcontext of type %d ignored\n", le16_to_cpu(pctx->ContextType)); - if (rc) break; - /* offsets must be 8 byte aligned */ - clen = ALIGN(clen, 8); - offset += clen + sizeof(struct smb2_neg_context); + + offset += clen; len_of_ctxts -= clen; } return rc; diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c index 195dc23e0d83..1db3e3c24b43 100644 --- a/fs/fs-writeback.c +++ b/fs/fs-writeback.c @@ -978,6 +978,16 @@ restart: continue; } + /* + * If wb_tryget fails, the wb has been shutdown, skip it. + * + * Pin @wb so that it stays on @bdi->wb_list. This allows + * continuing iteration from @wb after dropping and + * regrabbing rcu read lock. + */ + if (!wb_tryget(wb)) + continue; + /* alloc failed, execute synchronously using on-stack fallback */ work = &fallback_work; *work = *base_work; @@ -986,13 +996,6 @@ restart: work->done = &fallback_work_done; wb_queue_work(wb, work); - - /* - * Pin @wb so that it stays on @bdi->wb_list. This allows - * continuing iteration from @wb after dropping and - * regrabbing rcu read lock. - */ - wb_get(wb); last_wb = wb; rcu_read_unlock(); diff --git a/fs/ksmbd/smb2pdu.c b/fs/ksmbd/smb2pdu.c index 8af939a181be..67b7e766a06b 100644 --- a/fs/ksmbd/smb2pdu.c +++ b/fs/ksmbd/smb2pdu.c @@ -876,17 +876,21 @@ static void assemble_neg_contexts(struct ksmbd_conn *conn, } static __le32 decode_preauth_ctxt(struct ksmbd_conn *conn, - struct smb2_preauth_neg_context *pneg_ctxt) + struct smb2_preauth_neg_context *pneg_ctxt, + int len_of_ctxts) { - __le32 err = STATUS_NO_PREAUTH_INTEGRITY_HASH_OVERLAP; + /* + * sizeof(smb2_preauth_neg_context) assumes SMB311_SALT_SIZE Salt, + * which may not be present. Only check for used HashAlgorithms[1]. + */ + if (len_of_ctxts < MIN_PREAUTH_CTXT_DATA_LEN) + return STATUS_INVALID_PARAMETER; - if (pneg_ctxt->HashAlgorithms == SMB2_PREAUTH_INTEGRITY_SHA512) { - conn->preauth_info->Preauth_HashId = - SMB2_PREAUTH_INTEGRITY_SHA512; - err = STATUS_SUCCESS; - } + if (pneg_ctxt->HashAlgorithms != SMB2_PREAUTH_INTEGRITY_SHA512) + return STATUS_NO_PREAUTH_INTEGRITY_HASH_OVERLAP; - return err; + conn->preauth_info->Preauth_HashId = SMB2_PREAUTH_INTEGRITY_SHA512; + return STATUS_SUCCESS; } static void decode_encrypt_ctxt(struct ksmbd_conn *conn, @@ -1014,7 +1018,8 @@ static __le32 deassemble_neg_contexts(struct ksmbd_conn *conn, break; status = decode_preauth_ctxt(conn, - (struct smb2_preauth_neg_context *)pctx); + (struct smb2_preauth_neg_context *)pctx, + len_of_ctxts); if (status != STATUS_SUCCESS) break; } else if (pctx->ContextType == SMB2_ENCRYPTION_CAPABILITIES) { diff --git a/fs/nilfs2/segment.c b/fs/nilfs2/segment.c index 6ad41390fa74..228659612c0d 100644 --- a/fs/nilfs2/segment.c +++ b/fs/nilfs2/segment.c @@ -430,6 +430,23 @@ static int nilfs_segctor_reset_segment_buffer(struct nilfs_sc_info *sci) return 0; } +/** + * nilfs_segctor_zeropad_segsum - zero pad the rest of the segment summary area + * @sci: segment constructor object + * + * nilfs_segctor_zeropad_segsum() zero-fills unallocated space at the end of + * the current segment summary block. + */ +static void nilfs_segctor_zeropad_segsum(struct nilfs_sc_info *sci) +{ + struct nilfs_segsum_pointer *ssp; + + ssp = sci->sc_blk_cnt > 0 ? &sci->sc_binfo_ptr : &sci->sc_finfo_ptr; + if (ssp->offset < ssp->bh->b_size) + memset(ssp->bh->b_data + ssp->offset, 0, + ssp->bh->b_size - ssp->offset); +} + static int nilfs_segctor_feed_segment(struct nilfs_sc_info *sci) { sci->sc_nblk_this_inc += sci->sc_curseg->sb_sum.nblocks; @@ -438,6 +455,7 @@ static int nilfs_segctor_feed_segment(struct nilfs_sc_info *sci) * The current segment is filled up * (internal code) */ + nilfs_segctor_zeropad_segsum(sci); sci->sc_curseg = NILFS_NEXT_SEGBUF(sci->sc_curseg); return nilfs_segctor_reset_segment_buffer(sci); } @@ -542,6 +560,7 @@ static int nilfs_segctor_add_file_block(struct nilfs_sc_info *sci, goto retry; } if (unlikely(required)) { + nilfs_segctor_zeropad_segsum(sci); err = nilfs_segbuf_extend_segsum(segbuf); if (unlikely(err)) goto failed; @@ -1533,6 +1552,7 @@ static int nilfs_segctor_collect(struct nilfs_sc_info *sci, nadd = min_t(int, nadd << 1, SC_MAX_SEGDELTA); sci->sc_stage = prev_stage; } + nilfs_segctor_zeropad_segsum(sci); nilfs_segctor_truncate_segments(sci, sci->sc_curseg, nilfs->ns_sufile); return 0; diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 44d1ee429eb0..40f9e1a2ebdd 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -1955,8 +1955,10 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, ret = -EFAULT; if (copy_from_user(&uffdio_api, buf, sizeof(uffdio_api))) goto out; - /* Ignore unsupported features (userspace built against newer kernel) */ - features = uffdio_api.features & UFFD_API_FEATURES; + features = uffdio_api.features; + ret = -EINVAL; + if (uffdio_api.api != UFFD_API || (features & ~UFFD_API_FEATURES)) + goto err_out; ret = -EPERM; if ((features & UFFD_FEATURE_EVENT_FORK) && !capable(CAP_SYS_PTRACE)) goto err_out; diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 2c6095bd7d69..e53ceee1df37 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -187,6 +187,7 @@ enum btf_field_type { BPF_RB_NODE = (1 << 7), BPF_GRAPH_NODE_OR_ROOT = BPF_LIST_NODE | BPF_LIST_HEAD | BPF_RB_NODE | BPF_RB_ROOT, + BPF_REFCOUNT = (1 << 8), }; typedef void (*btf_dtor_kfunc_t)(void *); @@ -210,6 +211,7 @@ struct btf_field_graph_root { struct btf_field { u32 offset; + u32 size; enum btf_field_type type; union { struct btf_field_kptr kptr; @@ -222,15 +224,10 @@ struct btf_record { u32 field_mask; int spin_lock_off; int timer_off; + int refcount_off; struct btf_field fields[]; }; -struct btf_field_offs { - u32 cnt; - u32 field_off[BTF_FIELDS_MAX]; - u8 field_sz[BTF_FIELDS_MAX]; -}; - struct bpf_map { /* The first two cachelines with read-mostly members of which some * are also accessed in fast-path (e.g. ops, max_entries). @@ -257,7 +254,6 @@ struct bpf_map { struct obj_cgroup *objcg; #endif char name[BPF_OBJ_NAME_LEN]; - struct btf_field_offs *field_offs; /* The 3rd and 4th cacheline with misc members to avoid false sharing * particularly with refcounting. */ @@ -299,6 +295,8 @@ static inline const char *btf_field_type_name(enum btf_field_type type) return "bpf_rb_root"; case BPF_RB_NODE: return "bpf_rb_node"; + case BPF_REFCOUNT: + return "bpf_refcount"; default: WARN_ON_ONCE(1); return "unknown"; @@ -323,6 +321,8 @@ static inline u32 btf_field_type_size(enum btf_field_type type) return sizeof(struct bpf_rb_root); case BPF_RB_NODE: return sizeof(struct bpf_rb_node); + case BPF_REFCOUNT: + return sizeof(struct bpf_refcount); default: WARN_ON_ONCE(1); return 0; @@ -347,12 +347,42 @@ static inline u32 btf_field_type_align(enum btf_field_type type) return __alignof__(struct bpf_rb_root); case BPF_RB_NODE: return __alignof__(struct bpf_rb_node); + case BPF_REFCOUNT: + return __alignof__(struct bpf_refcount); default: WARN_ON_ONCE(1); return 0; } } +static inline void bpf_obj_init_field(const struct btf_field *field, void *addr) +{ + memset(addr, 0, field->size); + + switch (field->type) { + case BPF_REFCOUNT: + refcount_set((refcount_t *)addr, 1); + break; + case BPF_RB_NODE: + RB_CLEAR_NODE((struct rb_node *)addr); + break; + case BPF_LIST_HEAD: + case BPF_LIST_NODE: + INIT_LIST_HEAD((struct list_head *)addr); + break; + case BPF_RB_ROOT: + /* RB_ROOT_CACHED 0-inits, no need to do anything after memset */ + case BPF_SPIN_LOCK: + case BPF_TIMER: + case BPF_KPTR_UNREF: + case BPF_KPTR_REF: + break; + default: + WARN_ON_ONCE(1); + return; + } +} + static inline bool btf_record_has_field(const struct btf_record *rec, enum btf_field_type type) { if (IS_ERR_OR_NULL(rec)) @@ -360,14 +390,14 @@ static inline bool btf_record_has_field(const struct btf_record *rec, enum btf_f return rec->field_mask & type; } -static inline void bpf_obj_init(const struct btf_field_offs *foffs, void *obj) +static inline void bpf_obj_init(const struct btf_record *rec, void *obj) { int i; - if (!foffs) + if (IS_ERR_OR_NULL(rec)) return; - for (i = 0; i < foffs->cnt; i++) - memset(obj + foffs->field_off[i], 0, foffs->field_sz[i]); + for (i = 0; i < rec->cnt; i++) + bpf_obj_init_field(&rec->fields[i], obj + rec->fields[i].offset); } /* 'dst' must be a temporary buffer and should not point to memory that is being @@ -379,7 +409,7 @@ static inline void bpf_obj_init(const struct btf_field_offs *foffs, void *obj) */ static inline void check_and_init_map_value(struct bpf_map *map, void *dst) { - bpf_obj_init(map->field_offs, dst); + bpf_obj_init(map->record, dst); } /* memcpy that is used with 8-byte aligned pointers, power-of-8 size and @@ -399,14 +429,14 @@ static inline void bpf_long_memcpy(void *dst, const void *src, u32 size) } /* copy everything but bpf_spin_lock, bpf_timer, and kptrs. There could be one of each. */ -static inline void bpf_obj_memcpy(struct btf_field_offs *foffs, +static inline void bpf_obj_memcpy(struct btf_record *rec, void *dst, void *src, u32 size, bool long_memcpy) { u32 curr_off = 0; int i; - if (likely(!foffs)) { + if (IS_ERR_OR_NULL(rec)) { if (long_memcpy) bpf_long_memcpy(dst, src, round_up(size, 8)); else @@ -414,49 +444,49 @@ static inline void bpf_obj_memcpy(struct btf_field_offs *foffs, return; } - for (i = 0; i < foffs->cnt; i++) { - u32 next_off = foffs->field_off[i]; + for (i = 0; i < rec->cnt; i++) { + u32 next_off = rec->fields[i].offset; u32 sz = next_off - curr_off; memcpy(dst + curr_off, src + curr_off, sz); - curr_off += foffs->field_sz[i] + sz; + curr_off += rec->fields[i].size + sz; } memcpy(dst + curr_off, src + curr_off, size - curr_off); } static inline void copy_map_value(struct bpf_map *map, void *dst, void *src) { - bpf_obj_memcpy(map->field_offs, dst, src, map->value_size, false); + bpf_obj_memcpy(map->record, dst, src, map->value_size, false); } static inline void copy_map_value_long(struct bpf_map *map, void *dst, void *src) { - bpf_obj_memcpy(map->field_offs, dst, src, map->value_size, true); + bpf_obj_memcpy(map->record, dst, src, map->value_size, true); } -static inline void bpf_obj_memzero(struct btf_field_offs *foffs, void *dst, u32 size) +static inline void bpf_obj_memzero(struct btf_record *rec, void *dst, u32 size) { u32 curr_off = 0; int i; - if (likely(!foffs)) { + if (IS_ERR_OR_NULL(rec)) { memset(dst, 0, size); return; } - for (i = 0; i < foffs->cnt; i++) { - u32 next_off = foffs->field_off[i]; + for (i = 0; i < rec->cnt; i++) { + u32 next_off = rec->fields[i].offset; u32 sz = next_off - curr_off; memset(dst + curr_off, 0, sz); - curr_off += foffs->field_sz[i] + sz; + curr_off += rec->fields[i].size + sz; } memset(dst + curr_off, 0, size - curr_off); } static inline void zero_map_value(struct bpf_map *map, void *dst) { - bpf_obj_memzero(map->field_offs, dst, map->value_size); + bpf_obj_memzero(map->record, dst, map->value_size); } void copy_map_value_locked(struct bpf_map *map, void *dst, void *src, @@ -2234,6 +2264,9 @@ int bpf_prog_test_run_raw_tp(struct bpf_prog *prog, int bpf_prog_test_run_sk_lookup(struct bpf_prog *prog, const union bpf_attr *kattr, union bpf_attr __user *uattr); +int bpf_prog_test_run_nf(struct bpf_prog *prog, + const union bpf_attr *kattr, + union bpf_attr __user *uattr); bool btf_ctx_access(int off, int size, enum bpf_access_type type, const struct bpf_prog *prog, struct bpf_insn_access_aux *info); @@ -2295,6 +2328,9 @@ bool bpf_prog_has_kfunc_call(const struct bpf_prog *prog); const struct btf_func_model * bpf_jit_find_kfunc_model(const struct bpf_prog *prog, const struct bpf_insn *insn); +int bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id, + u16 btf_fd_idx, u8 **func_addr); + struct bpf_core_ctx { struct bpf_verifier_log *log; const struct btf *btf; @@ -2545,6 +2581,13 @@ bpf_jit_find_kfunc_model(const struct bpf_prog *prog, return NULL; } +static inline int +bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id, + u16 btf_fd_idx, u8 **func_addr) +{ + return -ENOTSUPP; +} + static inline bool unprivileged_ebpf_enabled(void) { return false; diff --git a/include/linux/bpf_types.h b/include/linux/bpf_types.h index d4ee3ccd3753..39a999abb0ce 100644 --- a/include/linux/bpf_types.h +++ b/include/linux/bpf_types.h @@ -79,6 +79,10 @@ BPF_PROG_TYPE(BPF_PROG_TYPE_LSM, lsm, #endif BPF_PROG_TYPE(BPF_PROG_TYPE_SYSCALL, bpf_syscall, void *, void *) +#ifdef CONFIG_NETFILTER +BPF_PROG_TYPE(BPF_PROG_TYPE_NETFILTER, netfilter, + struct bpf_nf_ctx, struct bpf_nf_ctx) +#endif BPF_MAP_TYPE(BPF_MAP_TYPE_ARRAY, array_map_ops) BPF_MAP_TYPE(BPF_MAP_TYPE_PERCPU_ARRAY, percpu_array_map_ops) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index f03852b89d28..3dd29a53b711 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -464,7 +464,12 @@ struct bpf_insn_aux_data { */ struct bpf_loop_inline_state loop_inline_state; }; - u64 obj_new_size; /* remember the size of type passed to bpf_obj_new to rewrite R1 */ + union { + /* remember the size of type passed to bpf_obj_new to rewrite R1 */ + u64 obj_new_size; + /* remember the offset of node field within type to rewrite */ + u64 insert_off; + }; struct btf_struct_meta *kptr_struct_meta; u64 map_key_state; /* constant (32 bit) key tracking for maps */ int ctx_field_size; /* the ctx field size for load insn, maybe 0 */ diff --git a/include/linux/btf.h b/include/linux/btf.h index 495250162422..508199e38415 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -18,7 +18,6 @@ #define KF_ACQUIRE (1 << 0) /* kfunc is an acquire function */ #define KF_RELEASE (1 << 1) /* kfunc is a release function */ #define KF_RET_NULL (1 << 2) /* kfunc returns a pointer that may be NULL */ -#define KF_KPTR_GET (1 << 3) /* kfunc returns reference to a kptr */ /* Trusted arguments are those which are guaranteed to be valid when passed to * the kfunc. It is used to enforce that pointers obtained from either acquire * kfuncs, or from the main kernel on a tracepoint or struct_ops callback @@ -113,7 +112,6 @@ struct btf_id_dtor_kfunc { struct btf_struct_meta { u32 btf_id; struct btf_record *record; - struct btf_field_offs *field_offs; }; struct btf_struct_metas { @@ -207,7 +205,6 @@ int btf_find_timer(const struct btf *btf, const struct btf_type *t); struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type *t, u32 field_mask, u32 value_size); int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec); -struct btf_field_offs *btf_parse_field_offs(struct btf_record *rec); bool btf_type_is_void(const struct btf_type *t); s32 btf_find_by_name_kind(const struct btf *btf, const char *name, u8 kind); const struct btf_type *btf_type_skip_modifiers(const struct btf *btf, diff --git a/include/linux/filter.h b/include/linux/filter.h index 5364b0c52c1d..bbce89937fde 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -920,6 +920,7 @@ void bpf_jit_compile(struct bpf_prog *prog); bool bpf_jit_needs_zext(void); bool bpf_jit_supports_subprog_tailcalls(void); bool bpf_jit_supports_kfunc_call(void); +bool bpf_jit_supports_far_kfunc_call(void); bool bpf_helper_changes_pkt_data(void *func); static inline bool bpf_dump_raw_ok(const struct cred *cred) diff --git a/include/linux/ieee80211.h b/include/linux/ieee80211.h index 2463bdd2a382..c4cf296e7eaf 100644 --- a/include/linux/ieee80211.h +++ b/include/linux/ieee80211.h @@ -9,7 +9,7 @@ * Copyright (c) 2006, Michael Wu <[email protected]> * Copyright (c) 2013 - 2014 Intel Mobile Communications GmbH * Copyright (c) 2016 - 2017 Intel Deutschland GmbH - * Copyright (c) 2018 - 2022 Intel Corporation + * Copyright (c) 2018 - 2023 Intel Corporation */ #ifndef LINUX_IEEE80211_H @@ -783,20 +783,6 @@ static inline bool ieee80211_is_any_nullfunc(__le16 fc) } /** - * ieee80211_is_bufferable_mmpdu - check if frame is bufferable MMPDU - * @fc: frame control field in little-endian byteorder - */ -static inline bool ieee80211_is_bufferable_mmpdu(__le16 fc) -{ - /* IEEE 802.11-2012, definition of "bufferable management frame"; - * note that this ignores the IBSS special case. */ - return ieee80211_is_mgmt(fc) && - (ieee80211_is_action(fc) || - ieee80211_is_disassoc(fc) || - ieee80211_is_deauth(fc)); -} - -/** * ieee80211_is_first_frag - check if IEEE80211_SCTL_FRAG is not set * @seq_ctrl: frame sequence control bytes in little-endian byteorder */ @@ -3557,11 +3543,6 @@ enum ieee80211_unprotected_wnm_actioncode { WLAN_UNPROTECTED_WNM_ACTION_TIMING_MEASUREMENT_RESPONSE = 1, }; -/* Public action codes */ -enum ieee80211_public_actioncode { - WLAN_PUBLIC_ACTION_FTM_RESPONSE = 33, -}; - /* Security key length */ enum ieee80211_key_len { WLAN_KEY_LEN_WEP40 = 5, @@ -3653,7 +3634,7 @@ enum ieee80211_pub_actioncode { WLAN_PUB_ACTION_NETWORK_CHANNEL_CONTROL = 30, WLAN_PUB_ACTION_WHITE_SPACE_MAP_ANN = 31, WLAN_PUB_ACTION_FTM_REQUEST = 32, - WLAN_PUB_ACTION_FTM = 33, + WLAN_PUB_ACTION_FTM_RESPONSE = 33, WLAN_PUB_ACTION_FILS_DISCOVERY = 34, }; @@ -4138,6 +4119,44 @@ static inline u8 *ieee80211_get_DA(struct ieee80211_hdr *hdr) } /** + * ieee80211_is_bufferable_mmpdu - check if frame is bufferable MMPDU + * @skb: the skb to check, starting with the 802.11 header + */ +static inline bool ieee80211_is_bufferable_mmpdu(struct sk_buff *skb) +{ + struct ieee80211_mgmt *mgmt = (void *)skb->data; + __le16 fc = mgmt->frame_control; + + /* + * IEEE 802.11 REVme D2.0 definition of bufferable MMPDU; + * note that this ignores the IBSS special case. + */ + if (!ieee80211_is_mgmt(fc)) + return false; + + if (ieee80211_is_disassoc(fc) || ieee80211_is_deauth(fc)) + return true; + + if (!ieee80211_is_action(fc)) + return false; + + if (skb->len < offsetofend(typeof(*mgmt), u.action.u.ftm.action_code)) + return true; + + /* action frame - additionally check for non-bufferable FTM */ + + if (mgmt->u.action.category != WLAN_CATEGORY_PUBLIC && + mgmt->u.action.category != WLAN_CATEGORY_PROTECTED_DUAL_OF_ACTION) + return true; + + if (mgmt->u.action.u.ftm.action_code == WLAN_PUB_ACTION_FTM_REQUEST || + mgmt->u.action.u.ftm.action_code == WLAN_PUB_ACTION_FTM_RESPONSE) + return false; + + return true; +} + +/** * _ieee80211_is_robust_mgmt_frame - check if frame is a robust management frame * @hdr: the frame (buffer must include at least the first octet of payload) */ @@ -4383,7 +4402,7 @@ static inline bool ieee80211_is_ftm(struct sk_buff *skb) return false; if (mgmt->u.action.u.ftm.action_code == - WLAN_PUBLIC_ACTION_FTM_RESPONSE && + WLAN_PUB_ACTION_FTM_RESPONSE && skb->len >= offsetofend(typeof(*mgmt), u.action.u.ftm)) return true; diff --git a/include/linux/if_bridge.h b/include/linux/if_bridge.h index 1668ac4d7adc..3ff96ae31bf6 100644 --- a/include/linux/if_bridge.h +++ b/include/linux/if_bridge.h @@ -60,6 +60,7 @@ struct br_ip_list { #define BR_TX_FWD_OFFLOAD BIT(20) #define BR_PORT_LOCKED BIT(21) #define BR_PORT_MAB BIT(22) +#define BR_NEIGH_VLAN_SUPPRESS BIT(23) #define BR_DEFAULT_AGEING_TIME (300 * HZ) diff --git a/include/linux/kmsan.h b/include/linux/kmsan.h index e38ae3c34618..30b17647ce3c 100644 --- a/include/linux/kmsan.h +++ b/include/linux/kmsan.h @@ -134,11 +134,12 @@ void kmsan_kfree_large(const void *ptr); * @page_shift: page_shift passed to vmap_range_noflush(). * * KMSAN maps shadow and origin pages of @pages into contiguous ranges in - * vmalloc metadata address range. + * vmalloc metadata address range. Returns 0 on success, callers must check + * for non-zero return value. */ -void kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end, - pgprot_t prot, struct page **pages, - unsigned int page_shift); +int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end, + pgprot_t prot, struct page **pages, + unsigned int page_shift); /** * kmsan_vunmap_kernel_range_noflush() - Notify KMSAN about a vunmap. @@ -159,11 +160,12 @@ void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end); * @page_shift: page_shift argument passed to vmap_range_noflush(). * * KMSAN creates new metadata pages for the physical pages mapped into the - * virtual memory. + * virtual memory. Returns 0 on success, callers must check for non-zero return + * value. */ -void kmsan_ioremap_page_range(unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int page_shift); +int kmsan_ioremap_page_range(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int page_shift); /** * kmsan_iounmap_page_range() - Notify KMSAN about a iounmap_page_range() call. @@ -281,12 +283,13 @@ static inline void kmsan_kfree_large(const void *ptr) { } -static inline void kmsan_vmap_pages_range_noflush(unsigned long start, - unsigned long end, - pgprot_t prot, - struct page **pages, - unsigned int page_shift) +static inline int kmsan_vmap_pages_range_noflush(unsigned long start, + unsigned long end, + pgprot_t prot, + struct page **pages, + unsigned int page_shift) { + return 0; } static inline void kmsan_vunmap_range_noflush(unsigned long start, @@ -294,12 +297,12 @@ static inline void kmsan_vunmap_range_noflush(unsigned long start, { } -static inline void kmsan_ioremap_page_range(unsigned long start, - unsigned long end, - phys_addr_t phys_addr, - pgprot_t prot, - unsigned int page_shift) +static inline int kmsan_ioremap_page_range(unsigned long start, + unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int page_shift) { + return 0; } static inline void kmsan_iounmap_page_range(unsigned long start, diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index 135a3c8d8237..a4c4f737f9c1 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -439,6 +439,7 @@ struct mlx5_core_health { struct work_struct report_work; struct devlink_health_reporter *fw_reporter; struct devlink_health_reporter *fw_fatal_reporter; + struct devlink_health_reporter *vnic_reporter; struct delayed_work update_fw_log_ts_work; }; @@ -1215,11 +1216,6 @@ static inline bool mlx5_core_is_vf(const struct mlx5_core_dev *dev) return dev->coredev_type == MLX5_COREDEV_VF; } -static inline bool mlx5_core_is_management_pf(const struct mlx5_core_dev *dev) -{ - return MLX5_CAP_GEN(dev, num_ports) == 1 && !MLX5_CAP_GEN(dev, native_port_num); -} - static inline bool mlx5_core_is_ecpf(const struct mlx5_core_dev *dev) { return dev->caps.embedded_cpu; diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index 20d00e09b168..b42696d74c9f 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -69,7 +69,7 @@ enum { MLX5_SET_HCA_CAP_OP_MOD_ATOMIC = 0x3, MLX5_SET_HCA_CAP_OP_MOD_ROCE = 0x4, MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE2 = 0x20, - MLX5_SET_HCA_CAP_OP_MODE_PORT_SELECTION = 0x25, + MLX5_SET_HCA_CAP_OP_MOD_PORT_SELECTION = 0x25, }; enum { diff --git a/include/linux/mmc/sdio_ids.h b/include/linux/mmc/sdio_ids.h index 0e4ef9c5127a..c653accdc7fd 100644 --- a/include/linux/mmc/sdio_ids.h +++ b/include/linux/mmc/sdio_ids.h @@ -74,10 +74,13 @@ #define SDIO_DEVICE_ID_BROADCOM_43362 0xa962 #define SDIO_DEVICE_ID_BROADCOM_43364 0xa9a4 #define SDIO_DEVICE_ID_BROADCOM_43430 0xa9a6 -#define SDIO_DEVICE_ID_BROADCOM_CYPRESS_43439 0xa9af +#define SDIO_DEVICE_ID_BROADCOM_43439 0xa9af #define SDIO_DEVICE_ID_BROADCOM_43455 0xa9bf #define SDIO_DEVICE_ID_BROADCOM_CYPRESS_43752 0xaae8 +#define SDIO_VENDOR_ID_CYPRESS 0x04b4 +#define SDIO_DEVICE_ID_BROADCOM_CYPRESS_43439 0xbd3d + #define SDIO_VENDOR_ID_MARVELL 0x02df #define SDIO_DEVICE_ID_MARVELL_LIBERTAS 0x9103 #define SDIO_DEVICE_ID_MARVELL_8688_WLAN 0x9104 @@ -112,6 +115,15 @@ #define SDIO_VENDOR_ID_MICROCHIP_WILC 0x0296 #define SDIO_DEVICE_ID_MICROCHIP_WILC1000 0x5347 +#define SDIO_VENDOR_ID_REALTEK 0x024c +#define SDIO_DEVICE_ID_REALTEK_RTW8723BS 0xb723 +#define SDIO_DEVICE_ID_REALTEK_RTW8821BS 0xb821 +#define SDIO_DEVICE_ID_REALTEK_RTW8822BS 0xb822 +#define SDIO_DEVICE_ID_REALTEK_RTW8821CS 0xc821 +#define SDIO_DEVICE_ID_REALTEK_RTW8822CS 0xc822 +#define SDIO_DEVICE_ID_REALTEK_RTW8723DS 0xd723 +#define SDIO_DEVICE_ID_REALTEK_RTW8821DS 0xd821 + #define SDIO_VENDOR_ID_SIANO 0x039a #define SDIO_DEVICE_ID_SIANO_NOVA_B0 0x0201 #define SDIO_DEVICE_ID_SIANO_NICE 0x0202 diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 203c0df2046c..a6a3e9457d6c 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -52,7 +52,7 @@ #include <linux/rbtree.h> #include <net/net_trackers.h> #include <net/net_debug.h> -#include <net/dropreason.h> +#include <net/dropreason-core.h> struct netpoll_info; struct device; diff --git a/include/linux/netfilter.h b/include/linux/netfilter.h index c8e03bcaecaa..0762444e3767 100644 --- a/include/linux/netfilter.h +++ b/include/linux/netfilter.h @@ -80,6 +80,7 @@ typedef unsigned int nf_hookfn(void *priv, enum nf_hook_ops_type { NF_HOOK_OP_UNDEFINED, NF_HOOK_OP_NF_TABLES, + NF_HOOK_OP_BPF, }; struct nf_hook_ops { diff --git a/include/linux/pds/pds_adminq.h b/include/linux/pds/pds_adminq.h new file mode 100644 index 000000000000..98a60ce87b92 --- /dev/null +++ b/include/linux/pds/pds_adminq.h @@ -0,0 +1,647 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#ifndef _PDS_CORE_ADMINQ_H_ +#define _PDS_CORE_ADMINQ_H_ + +#define PDSC_ADMINQ_MAX_POLL_INTERVAL 256 + +enum pds_core_adminq_flags { + PDS_AQ_FLAG_FASTPOLL = BIT(1), /* completion poll at 1ms */ +}; + +/* + * enum pds_core_adminq_opcode - AdminQ command opcodes + * These commands are only processed on AdminQ, not available in devcmd + */ +enum pds_core_adminq_opcode { + PDS_AQ_CMD_NOP = 0, + + /* Client control */ + PDS_AQ_CMD_CLIENT_REG = 6, + PDS_AQ_CMD_CLIENT_UNREG = 7, + PDS_AQ_CMD_CLIENT_CMD = 8, + + /* LIF commands */ + PDS_AQ_CMD_LIF_IDENTIFY = 20, + PDS_AQ_CMD_LIF_INIT = 21, + PDS_AQ_CMD_LIF_RESET = 22, + PDS_AQ_CMD_LIF_GETATTR = 23, + PDS_AQ_CMD_LIF_SETATTR = 24, + PDS_AQ_CMD_LIF_SETPHC = 25, + + PDS_AQ_CMD_RX_MODE_SET = 30, + PDS_AQ_CMD_RX_FILTER_ADD = 31, + PDS_AQ_CMD_RX_FILTER_DEL = 32, + + /* Queue commands */ + PDS_AQ_CMD_Q_IDENTIFY = 39, + PDS_AQ_CMD_Q_INIT = 40, + PDS_AQ_CMD_Q_CONTROL = 41, + + /* SR/IOV commands */ + PDS_AQ_CMD_VF_GETATTR = 60, + PDS_AQ_CMD_VF_SETATTR = 61, +}; + +/* + * enum pds_core_notifyq_opcode - NotifyQ event codes + */ +enum pds_core_notifyq_opcode { + PDS_EVENT_LINK_CHANGE = 1, + PDS_EVENT_RESET = 2, + PDS_EVENT_XCVR = 5, + PDS_EVENT_CLIENT = 6, +}; + +#define PDS_COMP_COLOR_MASK 0x80 + +/** + * struct pds_core_notifyq_event - Generic event reporting structure + * @eid: event number + * @ecode: event code + * + * This is the generic event report struct from which the other + * actual events will be formed. + */ +struct pds_core_notifyq_event { + __le64 eid; + __le16 ecode; +}; + +/** + * struct pds_core_link_change_event - Link change event notification + * @eid: event number + * @ecode: event code = PDS_EVENT_LINK_CHANGE + * @link_status: link up/down, with error bits + * @link_speed: speed of the network link + * + * Sent when the network link state changes between UP and DOWN + */ +struct pds_core_link_change_event { + __le64 eid; + __le16 ecode; + __le16 link_status; + __le32 link_speed; /* units of 1Mbps: e.g. 10000 = 10Gbps */ +}; + +/** + * struct pds_core_reset_event - Reset event notification + * @eid: event number + * @ecode: event code = PDS_EVENT_RESET + * @reset_code: reset type + * @state: 0=pending, 1=complete, 2=error + * + * Sent when the NIC or some subsystem is going to be or + * has been reset. + */ +struct pds_core_reset_event { + __le64 eid; + __le16 ecode; + u8 reset_code; + u8 state; +}; + +/** + * struct pds_core_client_event - Client event notification + * @eid: event number + * @ecode: event code = PDS_EVENT_CLIENT + * @client_id: client to sent event to + * @client_event: wrapped event struct for the client + * + * Sent when an event needs to be passed on to a client + */ +struct pds_core_client_event { + __le64 eid; + __le16 ecode; + __le16 client_id; + u8 client_event[54]; +}; + +/** + * struct pds_core_notifyq_cmd - Placeholder for building qcq + * @data: anonymous field for building the qcq + */ +struct pds_core_notifyq_cmd { + __le32 data; /* Not used but needed for qcq structure */ +}; + +/* + * union pds_core_notifyq_comp - Overlay of notifyq event structures + */ +union pds_core_notifyq_comp { + struct { + __le64 eid; + __le16 ecode; + }; + struct pds_core_notifyq_event event; + struct pds_core_link_change_event link_change; + struct pds_core_reset_event reset; + u8 data[64]; +}; + +#define PDS_DEVNAME_LEN 32 +/** + * struct pds_core_client_reg_cmd - Register a new client with DSC + * @opcode: opcode PDS_AQ_CMD_CLIENT_REG + * @rsvd: word boundary padding + * @devname: text name of client device + * @vif_type: what type of device (enum pds_core_vif_types) + * + * Tell the DSC of the new client, and receive a client_id from DSC. + */ +struct pds_core_client_reg_cmd { + u8 opcode; + u8 rsvd[3]; + char devname[PDS_DEVNAME_LEN]; + u8 vif_type; +}; + +/** + * struct pds_core_client_reg_comp - Client registration completion + * @status: Status of the command (enum pdc_core_status_code) + * @rsvd: Word boundary padding + * @comp_index: Index in the descriptor ring for which this is the completion + * @client_id: New id assigned by DSC + * @rsvd1: Word boundary padding + * @color: Color bit + */ +struct pds_core_client_reg_comp { + u8 status; + u8 rsvd; + __le16 comp_index; + __le16 client_id; + u8 rsvd1[9]; + u8 color; +}; + +/** + * struct pds_core_client_unreg_cmd - Unregister a client from DSC + * @opcode: opcode PDS_AQ_CMD_CLIENT_UNREG + * @rsvd: word boundary padding + * @client_id: id of client being removed + * + * Tell the DSC this client is going away and remove its context + * This uses the generic completion. + */ +struct pds_core_client_unreg_cmd { + u8 opcode; + u8 rsvd; + __le16 client_id; +}; + +/** + * struct pds_core_client_request_cmd - Pass along a wrapped client AdminQ cmd + * @opcode: opcode PDS_AQ_CMD_CLIENT_CMD + * @rsvd: word boundary padding + * @client_id: id of client being removed + * @client_cmd: the wrapped client command + * + * Proxy post an adminq command for the client. + * This uses the generic completion. + */ +struct pds_core_client_request_cmd { + u8 opcode; + u8 rsvd; + __le16 client_id; + u8 client_cmd[60]; +}; + +#define PDS_CORE_MAX_FRAGS 16 + +#define PDS_CORE_QCQ_F_INITED BIT(0) +#define PDS_CORE_QCQ_F_SG BIT(1) +#define PDS_CORE_QCQ_F_INTR BIT(2) +#define PDS_CORE_QCQ_F_TX_STATS BIT(3) +#define PDS_CORE_QCQ_F_RX_STATS BIT(4) +#define PDS_CORE_QCQ_F_NOTIFYQ BIT(5) +#define PDS_CORE_QCQ_F_CMB_RINGS BIT(6) +#define PDS_CORE_QCQ_F_CORE BIT(7) + +enum pds_core_lif_type { + PDS_CORE_LIF_TYPE_DEFAULT = 0, +}; + +/** + * union pds_core_lif_config - LIF configuration + * @state: LIF state (enum pds_core_lif_state) + * @rsvd: Word boundary padding + * @name: LIF name + * @rsvd2: Word boundary padding + * @features: LIF features active (enum pds_core_hw_features) + * @queue_count: Queue counts per queue-type + * @words: Full union buffer size + */ +union pds_core_lif_config { + struct { + u8 state; + u8 rsvd[3]; + char name[PDS_CORE_IFNAMSIZ]; + u8 rsvd2[12]; + __le64 features; + __le32 queue_count[PDS_CORE_QTYPE_MAX]; + } __packed; + __le32 words[64]; +}; + +/** + * struct pds_core_lif_status - LIF status register + * @eid: most recent NotifyQ event id + * @rsvd: full struct size + */ +struct pds_core_lif_status { + __le64 eid; + u8 rsvd[56]; +}; + +/** + * struct pds_core_lif_info - LIF info structure + * @config: LIF configuration structure + * @status: LIF status structure + */ +struct pds_core_lif_info { + union pds_core_lif_config config; + struct pds_core_lif_status status; +}; + +/** + * struct pds_core_lif_identity - LIF identity information (type-specific) + * @features: LIF features (see enum pds_core_hw_features) + * @version: Identify structure version + * @hw_index: LIF hardware index + * @rsvd: Word boundary padding + * @max_nb_sessions: Maximum number of sessions supported + * @rsvd2: buffer padding + * @config: LIF config struct with features, q counts + */ +struct pds_core_lif_identity { + __le64 features; + u8 version; + u8 hw_index; + u8 rsvd[2]; + __le32 max_nb_sessions; + u8 rsvd2[120]; + union pds_core_lif_config config; +}; + +/** + * struct pds_core_lif_identify_cmd - Get LIF identity info command + * @opcode: Opcode PDS_AQ_CMD_LIF_IDENTIFY + * @type: LIF type (enum pds_core_lif_type) + * @client_id: Client identifier + * @ver: Version of identify returned by device + * @rsvd: Word boundary padding + * @ident_pa: DMA address to receive identity info + * + * Firmware will copy LIF identity data (struct pds_core_lif_identity) + * into the buffer address given. + */ +struct pds_core_lif_identify_cmd { + u8 opcode; + u8 type; + __le16 client_id; + u8 ver; + u8 rsvd[3]; + __le64 ident_pa; +}; + +/** + * struct pds_core_lif_identify_comp - LIF identify command completion + * @status: Status of the command (enum pds_core_status_code) + * @ver: Version of identify returned by device + * @bytes: Bytes copied into the buffer + * @rsvd: Word boundary padding + * @color: Color bit + */ +struct pds_core_lif_identify_comp { + u8 status; + u8 ver; + __le16 bytes; + u8 rsvd[11]; + u8 color; +}; + +/** + * struct pds_core_lif_init_cmd - LIF init command + * @opcode: Opcode PDS_AQ_CMD_LIF_INIT + * @type: LIF type (enum pds_core_lif_type) + * @client_id: Client identifier + * @rsvd: Word boundary padding + * @info_pa: Destination address for LIF info (struct pds_core_lif_info) + */ +struct pds_core_lif_init_cmd { + u8 opcode; + u8 type; + __le16 client_id; + __le32 rsvd; + __le64 info_pa; +}; + +/** + * struct pds_core_lif_init_comp - LIF init command completion + * @status: Status of the command (enum pds_core_status_code) + * @rsvd: Word boundary padding + * @hw_index: Hardware index of the initialized LIF + * @rsvd1: Word boundary padding + * @color: Color bit + */ +struct pds_core_lif_init_comp { + u8 status; + u8 rsvd; + __le16 hw_index; + u8 rsvd1[11]; + u8 color; +}; + +/** + * struct pds_core_lif_reset_cmd - LIF reset command + * Will reset only the specified LIF. + * @opcode: Opcode PDS_AQ_CMD_LIF_RESET + * @rsvd: Word boundary padding + * @client_id: Client identifier + */ +struct pds_core_lif_reset_cmd { + u8 opcode; + u8 rsvd; + __le16 client_id; +}; + +/** + * enum pds_core_lif_attr - List of LIF attributes + * @PDS_CORE_LIF_ATTR_STATE: LIF state attribute + * @PDS_CORE_LIF_ATTR_NAME: LIF name attribute + * @PDS_CORE_LIF_ATTR_FEATURES: LIF features attribute + * @PDS_CORE_LIF_ATTR_STATS_CTRL: LIF statistics control attribute + */ +enum pds_core_lif_attr { + PDS_CORE_LIF_ATTR_STATE = 0, + PDS_CORE_LIF_ATTR_NAME = 1, + PDS_CORE_LIF_ATTR_FEATURES = 4, + PDS_CORE_LIF_ATTR_STATS_CTRL = 6, +}; + +/** + * struct pds_core_lif_setattr_cmd - Set LIF attributes on the NIC + * @opcode: Opcode PDS_AQ_CMD_LIF_SETATTR + * @attr: Attribute type (enum pds_core_lif_attr) + * @client_id: Client identifier + * @state: LIF state (enum pds_core_lif_state) + * @name: The name string, 0 terminated + * @features: Features (enum pds_core_hw_features) + * @stats_ctl: Stats control commands (enum pds_core_stats_ctl_cmd) + * @rsvd: Command Buffer padding + */ +struct pds_core_lif_setattr_cmd { + u8 opcode; + u8 attr; + __le16 client_id; + union { + u8 state; + char name[PDS_CORE_IFNAMSIZ]; + __le64 features; + u8 stats_ctl; + u8 rsvd[60]; + } __packed; +}; + +/** + * struct pds_core_lif_setattr_comp - LIF set attr command completion + * @status: Status of the command (enum pds_core_status_code) + * @rsvd: Word boundary padding + * @comp_index: Index in the descriptor ring for which this is the completion + * @features: Features (enum pds_core_hw_features) + * @rsvd2: Word boundary padding + * @color: Color bit + */ +struct pds_core_lif_setattr_comp { + u8 status; + u8 rsvd; + __le16 comp_index; + union { + __le64 features; + u8 rsvd2[11]; + } __packed; + u8 color; +}; + +/** + * struct pds_core_lif_getattr_cmd - Get LIF attributes from the NIC + * @opcode: Opcode PDS_AQ_CMD_LIF_GETATTR + * @attr: Attribute type (enum pds_core_lif_attr) + * @client_id: Client identifier + */ +struct pds_core_lif_getattr_cmd { + u8 opcode; + u8 attr; + __le16 client_id; +}; + +/** + * struct pds_core_lif_getattr_comp - LIF get attr command completion + * @status: Status of the command (enum pds_core_status_code) + * @rsvd: Word boundary padding + * @comp_index: Index in the descriptor ring for which this is the completion + * @state: LIF state (enum pds_core_lif_state) + * @name: LIF name string, 0 terminated + * @features: Features (enum pds_core_hw_features) + * @rsvd2: Word boundary padding + * @color: Color bit + */ +struct pds_core_lif_getattr_comp { + u8 status; + u8 rsvd; + __le16 comp_index; + union { + u8 state; + __le64 features; + u8 rsvd2[11]; + } __packed; + u8 color; +}; + +/** + * union pds_core_q_identity - Queue identity information + * @version: Queue type version that can be used with FW + * @supported: Bitfield of queue versions, first bit = ver 0 + * @rsvd: Word boundary padding + * @features: Queue features + * @desc_sz: Descriptor size + * @comp_sz: Completion descriptor size + * @rsvd2: Word boundary padding + */ +struct pds_core_q_identity { + u8 version; + u8 supported; + u8 rsvd[6]; +#define PDS_CORE_QIDENT_F_CQ 0x01 /* queue has completion ring */ + __le64 features; + __le16 desc_sz; + __le16 comp_sz; + u8 rsvd2[6]; +}; + +/** + * struct pds_core_q_identify_cmd - queue identify command + * @opcode: Opcode PDS_AQ_CMD_Q_IDENTIFY + * @type: Logical queue type (enum pds_core_logical_qtype) + * @client_id: Client identifier + * @ver: Highest queue type version that the driver supports + * @rsvd: Word boundary padding + * @ident_pa: DMA address to receive the data (struct pds_core_q_identity) + */ +struct pds_core_q_identify_cmd { + u8 opcode; + u8 type; + __le16 client_id; + u8 ver; + u8 rsvd[3]; + __le64 ident_pa; +}; + +/** + * struct pds_core_q_identify_comp - queue identify command completion + * @status: Status of the command (enum pds_core_status_code) + * @rsvd: Word boundary padding + * @comp_index: Index in the descriptor ring for which this is the completion + * @ver: Queue type version that can be used with FW + * @rsvd1: Word boundary padding + * @color: Color bit + */ +struct pds_core_q_identify_comp { + u8 status; + u8 rsvd; + __le16 comp_index; + u8 ver; + u8 rsvd1[10]; + u8 color; +}; + +/** + * struct pds_core_q_init_cmd - Queue init command + * @opcode: Opcode PDS_AQ_CMD_Q_INIT + * @type: Logical queue type + * @client_id: Client identifier + * @ver: Queue type version + * @rsvd: Word boundary padding + * @index: (LIF, qtype) relative admin queue index + * @intr_index: Interrupt control register index, or Event queue index + * @pid: Process ID + * @flags: + * IRQ: Interrupt requested on completion + * ENA: Enable the queue. If ENA=0 the queue is initialized + * but remains disabled, to be later enabled with the + * Queue Enable command. If ENA=1, then queue is + * initialized and then enabled. + * @cos: Class of service for this queue + * @ring_size: Queue ring size, encoded as a log2(size), in + * number of descriptors. The actual ring size is + * (1 << ring_size). For example, to select a ring size + * of 64 descriptors write ring_size = 6. The minimum + * ring_size value is 2 for a ring of 4 descriptors. + * The maximum ring_size value is 12 for a ring of 4k + * descriptors. Values of ring_size <2 and >12 are + * reserved. + * @ring_base: Queue ring base address + * @cq_ring_base: Completion queue ring base address + */ +struct pds_core_q_init_cmd { + u8 opcode; + u8 type; + __le16 client_id; + u8 ver; + u8 rsvd[3]; + __le32 index; + __le16 pid; + __le16 intr_index; + __le16 flags; +#define PDS_CORE_QINIT_F_IRQ 0x01 /* Request interrupt on completion */ +#define PDS_CORE_QINIT_F_ENA 0x02 /* Enable the queue */ + u8 cos; +#define PDS_CORE_QSIZE_MIN_LG2 2 +#define PDS_CORE_QSIZE_MAX_LG2 12 + u8 ring_size; + __le64 ring_base; + __le64 cq_ring_base; +} __packed; + +/** + * struct pds_core_q_init_comp - Queue init command completion + * @status: Status of the command (enum pds_core_status_code) + * @rsvd: Word boundary padding + * @comp_index: Index in the descriptor ring for which this is the completion + * @hw_index: Hardware Queue ID + * @hw_type: Hardware Queue type + * @rsvd2: Word boundary padding + * @color: Color + */ +struct pds_core_q_init_comp { + u8 status; + u8 rsvd; + __le16 comp_index; + __le32 hw_index; + u8 hw_type; + u8 rsvd2[6]; + u8 color; +}; + +union pds_core_adminq_cmd { + u8 opcode; + u8 bytes[64]; + + struct pds_core_client_reg_cmd client_reg; + struct pds_core_client_unreg_cmd client_unreg; + struct pds_core_client_request_cmd client_request; + + struct pds_core_lif_identify_cmd lif_ident; + struct pds_core_lif_init_cmd lif_init; + struct pds_core_lif_reset_cmd lif_reset; + struct pds_core_lif_setattr_cmd lif_setattr; + struct pds_core_lif_getattr_cmd lif_getattr; + + struct pds_core_q_identify_cmd q_ident; + struct pds_core_q_init_cmd q_init; +}; + +union pds_core_adminq_comp { + struct { + u8 status; + u8 rsvd; + __le16 comp_index; + u8 rsvd2[11]; + u8 color; + }; + u32 words[4]; + + struct pds_core_client_reg_comp client_reg; + + struct pds_core_lif_identify_comp lif_ident; + struct pds_core_lif_init_comp lif_init; + struct pds_core_lif_setattr_comp lif_setattr; + struct pds_core_lif_getattr_comp lif_getattr; + + struct pds_core_q_identify_comp q_ident; + struct pds_core_q_init_comp q_init; +}; + +#ifndef __CHECKER__ +static_assert(sizeof(union pds_core_adminq_cmd) == 64); +static_assert(sizeof(union pds_core_adminq_comp) == 16); +static_assert(sizeof(union pds_core_notifyq_comp) == 64); +#endif /* __CHECKER__ */ + +/* The color bit is a 'done' bit for the completion descriptors + * where the meaning alternates between '1' and '0' for alternating + * passes through the completion descriptor ring. + */ +static inline bool pdsc_color_match(u8 color, bool done_color) +{ + return (!!(color & PDS_COMP_COLOR_MASK)) == done_color; +} + +struct pdsc; +int pdsc_adminq_post(struct pdsc *pdsc, + union pds_core_adminq_cmd *cmd, + union pds_core_adminq_comp *comp, + bool fast_poll); + +#endif /* _PDS_CORE_ADMINQ_H_ */ diff --git a/include/linux/pds/pds_auxbus.h b/include/linux/pds/pds_auxbus.h new file mode 100644 index 000000000000..214ef12302d0 --- /dev/null +++ b/include/linux/pds/pds_auxbus.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright(c) 2023 Advanced Micro Devices, Inc */ + +#ifndef _PDSC_AUXBUS_H_ +#define _PDSC_AUXBUS_H_ + +#include <linux/auxiliary_bus.h> + +struct pds_auxiliary_dev { + struct auxiliary_device aux_dev; + struct pci_dev *vf_pdev; + u16 client_id; +}; + +int pds_client_adminq_cmd(struct pds_auxiliary_dev *padev, + union pds_core_adminq_cmd *req, + size_t req_len, + union pds_core_adminq_comp *resp, + u64 flags); +#endif /* _PDSC_AUXBUS_H_ */ diff --git a/include/linux/pds/pds_common.h b/include/linux/pds/pds_common.h new file mode 100644 index 000000000000..060331486d50 --- /dev/null +++ b/include/linux/pds/pds_common.h @@ -0,0 +1,68 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) OR BSD-2-Clause */ +/* Copyright(c) 2023 Advanced Micro Devices, Inc. */ + +#ifndef _PDS_COMMON_H_ +#define _PDS_COMMON_H_ + +#define PDS_CORE_DRV_NAME "pds_core" + +/* the device's internal addressing uses up to 52 bits */ +#define PDS_CORE_ADDR_LEN 52 +#define PDS_CORE_ADDR_MASK (BIT_ULL(PDS_ADDR_LEN) - 1) +#define PDS_PAGE_SIZE 4096 + +enum pds_core_driver_type { + PDS_DRIVER_LINUX = 1, + PDS_DRIVER_WIN = 2, + PDS_DRIVER_DPDK = 3, + PDS_DRIVER_FREEBSD = 4, + PDS_DRIVER_IPXE = 5, + PDS_DRIVER_ESXI = 6, +}; + +enum pds_core_vif_types { + PDS_DEV_TYPE_CORE = 0, + PDS_DEV_TYPE_VDPA = 1, + PDS_DEV_TYPE_VFIO = 2, + PDS_DEV_TYPE_ETH = 3, + PDS_DEV_TYPE_RDMA = 4, + PDS_DEV_TYPE_LM = 5, + + /* new ones added before this line */ + PDS_DEV_TYPE_MAX = 16 /* don't change - used in struct size */ +}; + +#define PDS_DEV_TYPE_CORE_STR "Core" +#define PDS_DEV_TYPE_VDPA_STR "vDPA" +#define PDS_DEV_TYPE_VFIO_STR "VFio" +#define PDS_DEV_TYPE_ETH_STR "Eth" +#define PDS_DEV_TYPE_RDMA_STR "RDMA" +#define PDS_DEV_TYPE_LM_STR "LM" + +#define PDS_CORE_IFNAMSIZ 16 + +/** + * enum pds_core_logical_qtype - Logical Queue Types + * @PDS_CORE_QTYPE_ADMINQ: Administrative Queue + * @PDS_CORE_QTYPE_NOTIFYQ: Notify Queue + * @PDS_CORE_QTYPE_RXQ: Receive Queue + * @PDS_CORE_QTYPE_TXQ: Transmit Queue + * @PDS_CORE_QTYPE_EQ: Event Queue + * @PDS_CORE_QTYPE_MAX: Max queue type supported + */ +enum pds_core_logical_qtype { + PDS_CORE_QTYPE_ADMINQ = 0, + PDS_CORE_QTYPE_NOTIFYQ = 1, + PDS_CORE_QTYPE_RXQ = 2, + PDS_CORE_QTYPE_TXQ = 3, + PDS_CORE_QTYPE_EQ = 4, + + PDS_CORE_QTYPE_MAX = 16 /* don't change - used in struct size */ +}; + +int pdsc_register_notify(struct notifier_block *nb); +void pdsc_unregister_notify(struct notifier_block *nb); +void *pdsc_get_pf_struct(struct pci_dev *vf_pdev); +int pds_client_register(struct pci_dev *pf_pdev, char *devname); +int pds_client_unregister(struct pci_dev *pf_pdev, u16 client_id); +#endif /* _PDS_COMMON_H_ */ diff --git a/include/linux/pds/pds_core_if.h b/include/linux/pds/pds_core_if.h new file mode 100644 index 000000000000..e838a2b90440 --- /dev/null +++ b/include/linux/pds/pds_core_if.h @@ -0,0 +1,571 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) OR BSD-2-Clause */ +/* Copyright(c) 2023 Advanced Micro Devices, Inc. */ + +#ifndef _PDS_CORE_IF_H_ +#define _PDS_CORE_IF_H_ + +#define PCI_VENDOR_ID_PENSANDO 0x1dd8 +#define PCI_DEVICE_ID_PENSANDO_CORE_PF 0x100c +#define PCI_DEVICE_ID_VIRTIO_NET_TRANS 0x1000 +#define PCI_DEVICE_ID_PENSANDO_IONIC_ETH_VF 0x1003 +#define PCI_DEVICE_ID_PENSANDO_VDPA_VF 0x100b +#define PDS_CORE_BARS_MAX 4 +#define PDS_CORE_PCI_BAR_DBELL 1 + +/* Bar0 */ +#define PDS_CORE_DEV_INFO_SIGNATURE 0x44455649 /* 'DEVI' */ +#define PDS_CORE_BAR0_SIZE 0x8000 +#define PDS_CORE_BAR0_DEV_INFO_REGS_OFFSET 0x0000 +#define PDS_CORE_BAR0_DEV_CMD_REGS_OFFSET 0x0800 +#define PDS_CORE_BAR0_DEV_CMD_DATA_REGS_OFFSET 0x0c00 +#define PDS_CORE_BAR0_INTR_STATUS_OFFSET 0x1000 +#define PDS_CORE_BAR0_INTR_CTRL_OFFSET 0x2000 +#define PDS_CORE_DEV_CMD_DONE 0x00000001 + +#define PDS_CORE_DEVCMD_TIMEOUT 5 + +#define PDS_CORE_CLIENT_ID 0 +#define PDS_CORE_ASIC_TYPE_CAPRI 0 + +/* + * enum pds_core_cmd_opcode - Device commands + */ +enum pds_core_cmd_opcode { + /* Core init */ + PDS_CORE_CMD_NOP = 0, + PDS_CORE_CMD_IDENTIFY = 1, + PDS_CORE_CMD_RESET = 2, + PDS_CORE_CMD_INIT = 3, + + PDS_CORE_CMD_FW_DOWNLOAD = 4, + PDS_CORE_CMD_FW_CONTROL = 5, + + /* SR/IOV commands */ + PDS_CORE_CMD_VF_GETATTR = 60, + PDS_CORE_CMD_VF_SETATTR = 61, + PDS_CORE_CMD_VF_CTRL = 62, + + /* Add commands before this line */ + PDS_CORE_CMD_MAX, + PDS_CORE_CMD_COUNT +}; + +/* + * enum pds_core_status_code - Device command return codes + */ +enum pds_core_status_code { + PDS_RC_SUCCESS = 0, /* Success */ + PDS_RC_EVERSION = 1, /* Incorrect version for request */ + PDS_RC_EOPCODE = 2, /* Invalid cmd opcode */ + PDS_RC_EIO = 3, /* I/O error */ + PDS_RC_EPERM = 4, /* Permission denied */ + PDS_RC_EQID = 5, /* Bad qid */ + PDS_RC_EQTYPE = 6, /* Bad qtype */ + PDS_RC_ENOENT = 7, /* No such element */ + PDS_RC_EINTR = 8, /* operation interrupted */ + PDS_RC_EAGAIN = 9, /* Try again */ + PDS_RC_ENOMEM = 10, /* Out of memory */ + PDS_RC_EFAULT = 11, /* Bad address */ + PDS_RC_EBUSY = 12, /* Device or resource busy */ + PDS_RC_EEXIST = 13, /* object already exists */ + PDS_RC_EINVAL = 14, /* Invalid argument */ + PDS_RC_ENOSPC = 15, /* No space left or alloc failure */ + PDS_RC_ERANGE = 16, /* Parameter out of range */ + PDS_RC_BAD_ADDR = 17, /* Descriptor contains a bad ptr */ + PDS_RC_DEV_CMD = 18, /* Device cmd attempted on AdminQ */ + PDS_RC_ENOSUPP = 19, /* Operation not supported */ + PDS_RC_ERROR = 29, /* Generic error */ + PDS_RC_ERDMA = 30, /* Generic RDMA error */ + PDS_RC_EVFID = 31, /* VF ID does not exist */ + PDS_RC_BAD_FW = 32, /* FW file is invalid or corrupted */ + PDS_RC_ECLIENT = 33, /* No such client id */ +}; + +/** + * struct pds_core_drv_identity - Driver identity information + * @drv_type: Driver type (enum pds_core_driver_type) + * @os_dist: OS distribution, numeric format + * @os_dist_str: OS distribution, string format + * @kernel_ver: Kernel version, numeric format + * @kernel_ver_str: Kernel version, string format + * @driver_ver_str: Driver version, string format + */ +struct pds_core_drv_identity { + __le32 drv_type; + __le32 os_dist; + char os_dist_str[128]; + __le32 kernel_ver; + char kernel_ver_str[32]; + char driver_ver_str[32]; +}; + +#define PDS_DEV_TYPE_MAX 16 +/** + * struct pds_core_dev_identity - Device identity information + * @version: Version of device identify + * @type: Identify type (0 for now) + * @state: Device state + * @rsvd: Word boundary padding + * @nlifs: Number of LIFs provisioned + * @nintrs: Number of interrupts provisioned + * @ndbpgs_per_lif: Number of doorbell pages per LIF + * @intr_coal_mult: Interrupt coalescing multiplication factor + * Scale user-supplied interrupt coalescing + * value in usecs to device units using: + * device units = usecs * mult / div + * @intr_coal_div: Interrupt coalescing division factor + * Scale user-supplied interrupt coalescing + * value in usecs to device units using: + * device units = usecs * mult / div + * @vif_types: How many of each VIF device type is supported + */ +struct pds_core_dev_identity { + u8 version; + u8 type; + u8 state; + u8 rsvd; + __le32 nlifs; + __le32 nintrs; + __le32 ndbpgs_per_lif; + __le32 intr_coal_mult; + __le32 intr_coal_div; + __le16 vif_types[PDS_DEV_TYPE_MAX]; +}; + +#define PDS_CORE_IDENTITY_VERSION_1 1 + +/** + * struct pds_core_dev_identify_cmd - Driver/device identify command + * @opcode: Opcode PDS_CORE_CMD_IDENTIFY + * @ver: Highest version of identify supported by driver + * + * Expects to find driver identification info (struct pds_core_drv_identity) + * in cmd_regs->data. Driver should keep the devcmd interface locked + * while preparing the driver info. + */ +struct pds_core_dev_identify_cmd { + u8 opcode; + u8 ver; +}; + +/** + * struct pds_core_dev_identify_comp - Device identify command completion + * @status: Status of the command (enum pds_core_status_code) + * @ver: Version of identify returned by device + * + * Device identification info (struct pds_core_dev_identity) can be found + * in cmd_regs->data. Driver should keep the devcmd interface locked + * while reading the results. + */ +struct pds_core_dev_identify_comp { + u8 status; + u8 ver; +}; + +/** + * struct pds_core_dev_reset_cmd - Device reset command + * @opcode: Opcode PDS_CORE_CMD_RESET + * + * Resets and clears all LIFs, VDevs, and VIFs on the device. + */ +struct pds_core_dev_reset_cmd { + u8 opcode; +}; + +/** + * struct pds_core_dev_reset_comp - Reset command completion + * @status: Status of the command (enum pds_core_status_code) + */ +struct pds_core_dev_reset_comp { + u8 status; +}; + +/* + * struct pds_core_dev_init_data - Pointers and info needed for the Core + * initialization PDS_CORE_CMD_INIT command. The in and out structs are + * overlays on the pds_core_dev_cmd_regs.data space for passing data down + * to the firmware on init, and then returning initialization results. + */ +struct pds_core_dev_init_data_in { + __le64 adminq_q_base; + __le64 adminq_cq_base; + __le64 notifyq_cq_base; + __le32 flags; + __le16 intr_index; + u8 adminq_ring_size; + u8 notifyq_ring_size; +}; + +struct pds_core_dev_init_data_out { + __le32 core_hw_index; + __le32 adminq_hw_index; + __le32 notifyq_hw_index; + u8 adminq_hw_type; + u8 notifyq_hw_type; +}; + +/** + * struct pds_core_dev_init_cmd - Core device initialize + * @opcode: opcode PDS_CORE_CMD_INIT + * + * Initializes the core device and sets up the AdminQ and NotifyQ. + * Expects to find initialization data (struct pds_core_dev_init_data_in) + * in cmd_regs->data. Driver should keep the devcmd interface locked + * while preparing the driver info. + */ +struct pds_core_dev_init_cmd { + u8 opcode; +}; + +/** + * struct pds_core_dev_init_comp - Core init completion + * @status: Status of the command (enum pds_core_status_code) + * + * Initialization result data (struct pds_core_dev_init_data_in) + * is found in cmd_regs->data. + */ +struct pds_core_dev_init_comp { + u8 status; +}; + +/** + * struct pds_core_fw_download_cmd - Firmware download command + * @opcode: opcode + * @rsvd: Word boundary padding + * @addr: DMA address of the firmware buffer + * @offset: offset of the firmware buffer within the full image + * @length: number of valid bytes in the firmware buffer + */ +struct pds_core_fw_download_cmd { + u8 opcode; + u8 rsvd[3]; + __le32 offset; + __le64 addr; + __le32 length; +}; + +/** + * struct pds_core_fw_download_comp - Firmware download completion + * @status: Status of the command (enum pds_core_status_code) + */ +struct pds_core_fw_download_comp { + u8 status; +}; + +/** + * enum pds_core_fw_control_oper - FW control operations + * @PDS_CORE_FW_INSTALL_ASYNC: Install firmware asynchronously + * @PDS_CORE_FW_INSTALL_STATUS: Firmware installation status + * @PDS_CORE_FW_ACTIVATE_ASYNC: Activate firmware asynchronously + * @PDS_CORE_FW_ACTIVATE_STATUS: Firmware activate status + * @PDS_CORE_FW_UPDATE_CLEANUP: Cleanup any firmware update leftovers + * @PDS_CORE_FW_GET_BOOT: Return current active firmware slot + * @PDS_CORE_FW_SET_BOOT: Set active firmware slot for next boot + * @PDS_CORE_FW_GET_LIST: Return list of installed firmware images + */ +enum pds_core_fw_control_oper { + PDS_CORE_FW_INSTALL_ASYNC = 0, + PDS_CORE_FW_INSTALL_STATUS = 1, + PDS_CORE_FW_ACTIVATE_ASYNC = 2, + PDS_CORE_FW_ACTIVATE_STATUS = 3, + PDS_CORE_FW_UPDATE_CLEANUP = 4, + PDS_CORE_FW_GET_BOOT = 5, + PDS_CORE_FW_SET_BOOT = 6, + PDS_CORE_FW_GET_LIST = 7, +}; + +enum pds_core_fw_slot { + PDS_CORE_FW_SLOT_INVALID = 0, + PDS_CORE_FW_SLOT_A = 1, + PDS_CORE_FW_SLOT_B = 2, + PDS_CORE_FW_SLOT_GOLD = 3, +}; + +/** + * struct pds_core_fw_control_cmd - Firmware control command + * @opcode: opcode + * @rsvd: Word boundary padding + * @oper: firmware control operation (enum pds_core_fw_control_oper) + * @slot: slot to operate on (enum pds_core_fw_slot) + */ +struct pds_core_fw_control_cmd { + u8 opcode; + u8 rsvd[3]; + u8 oper; + u8 slot; +}; + +/** + * struct pds_core_fw_control_comp - Firmware control copletion + * @status: Status of the command (enum pds_core_status_code) + * @rsvd: Word alignment space + * @slot: Slot number (enum pds_core_fw_slot) + * @rsvd1: Struct padding + * @color: Color bit + */ +struct pds_core_fw_control_comp { + u8 status; + u8 rsvd[3]; + u8 slot; + u8 rsvd1[10]; + u8 color; +}; + +struct pds_core_fw_name_info { +#define PDS_CORE_FWSLOT_BUFLEN 8 +#define PDS_CORE_FWVERS_BUFLEN 32 + char slotname[PDS_CORE_FWSLOT_BUFLEN]; + char fw_version[PDS_CORE_FWVERS_BUFLEN]; +}; + +struct pds_core_fw_list_info { +#define PDS_CORE_FWVERS_LIST_LEN 16 + u8 num_fw_slots; + struct pds_core_fw_name_info fw_names[PDS_CORE_FWVERS_LIST_LEN]; +} __packed; + +enum pds_core_vf_attr { + PDS_CORE_VF_ATTR_SPOOFCHK = 1, + PDS_CORE_VF_ATTR_TRUST = 2, + PDS_CORE_VF_ATTR_MAC = 3, + PDS_CORE_VF_ATTR_LINKSTATE = 4, + PDS_CORE_VF_ATTR_VLAN = 5, + PDS_CORE_VF_ATTR_RATE = 6, + PDS_CORE_VF_ATTR_STATSADDR = 7, +}; + +/** + * enum pds_core_vf_link_status - Virtual Function link status + * @PDS_CORE_VF_LINK_STATUS_AUTO: Use link state of the uplink + * @PDS_CORE_VF_LINK_STATUS_UP: Link always up + * @PDS_CORE_VF_LINK_STATUS_DOWN: Link always down + */ +enum pds_core_vf_link_status { + PDS_CORE_VF_LINK_STATUS_AUTO = 0, + PDS_CORE_VF_LINK_STATUS_UP = 1, + PDS_CORE_VF_LINK_STATUS_DOWN = 2, +}; + +/** + * struct pds_core_vf_setattr_cmd - Set VF attributes on the NIC + * @opcode: Opcode + * @attr: Attribute type (enum pds_core_vf_attr) + * @vf_index: VF index + * @macaddr: mac address + * @vlanid: vlan ID + * @maxrate: max Tx rate in Mbps + * @spoofchk: enable address spoof checking + * @trust: enable VF trust + * @linkstate: set link up or down + * @stats: stats addr struct + * @stats.pa: set DMA address for VF stats + * @stats.len: length of VF stats space + * @pad: force union to specific size + */ +struct pds_core_vf_setattr_cmd { + u8 opcode; + u8 attr; + __le16 vf_index; + union { + u8 macaddr[6]; + __le16 vlanid; + __le32 maxrate; + u8 spoofchk; + u8 trust; + u8 linkstate; + struct { + __le64 pa; + __le32 len; + } stats; + u8 pad[60]; + } __packed; +}; + +struct pds_core_vf_setattr_comp { + u8 status; + u8 attr; + __le16 vf_index; + __le16 comp_index; + u8 rsvd[9]; + u8 color; +}; + +/** + * struct pds_core_vf_getattr_cmd - Get VF attributes from the NIC + * @opcode: Opcode + * @attr: Attribute type (enum pds_core_vf_attr) + * @vf_index: VF index + */ +struct pds_core_vf_getattr_cmd { + u8 opcode; + u8 attr; + __le16 vf_index; +}; + +struct pds_core_vf_getattr_comp { + u8 status; + u8 attr; + __le16 vf_index; + union { + u8 macaddr[6]; + __le16 vlanid; + __le32 maxrate; + u8 spoofchk; + u8 trust; + u8 linkstate; + __le64 stats_pa; + u8 pad[11]; + } __packed; + u8 color; +}; + +enum pds_core_vf_ctrl_opcode { + PDS_CORE_VF_CTRL_START_ALL = 0, + PDS_CORE_VF_CTRL_START = 1, +}; + +/** + * struct pds_core_vf_ctrl_cmd - VF control command + * @opcode: Opcode for the command + * @ctrl_opcode: VF control operation type + * @vf_index: VF Index. It is unused if op START_ALL is used. + */ + +struct pds_core_vf_ctrl_cmd { + u8 opcode; + u8 ctrl_opcode; + __le16 vf_index; +}; + +/** + * struct pds_core_vf_ctrl_comp - VF_CTRL command completion. + * @status: Status of the command (enum pds_core_status_code) + */ +struct pds_core_vf_ctrl_comp { + u8 status; +}; + +/* + * union pds_core_dev_cmd - Overlay of core device command structures + */ +union pds_core_dev_cmd { + u8 opcode; + u32 words[16]; + + struct pds_core_dev_identify_cmd identify; + struct pds_core_dev_init_cmd init; + struct pds_core_dev_reset_cmd reset; + struct pds_core_fw_download_cmd fw_download; + struct pds_core_fw_control_cmd fw_control; + + struct pds_core_vf_setattr_cmd vf_setattr; + struct pds_core_vf_getattr_cmd vf_getattr; + struct pds_core_vf_ctrl_cmd vf_ctrl; +}; + +/* + * union pds_core_dev_comp - Overlay of core device completion structures + */ +union pds_core_dev_comp { + u8 status; + u8 bytes[16]; + + struct pds_core_dev_identify_comp identify; + struct pds_core_dev_reset_comp reset; + struct pds_core_dev_init_comp init; + struct pds_core_fw_download_comp fw_download; + struct pds_core_fw_control_comp fw_control; + + struct pds_core_vf_setattr_comp vf_setattr; + struct pds_core_vf_getattr_comp vf_getattr; + struct pds_core_vf_ctrl_comp vf_ctrl; +}; + +/** + * struct pds_core_dev_hwstamp_regs - Hardware current timestamp registers + * @tick_low: Low 32 bits of hardware timestamp + * @tick_high: High 32 bits of hardware timestamp + */ +struct pds_core_dev_hwstamp_regs { + u32 tick_low; + u32 tick_high; +}; + +/** + * struct pds_core_dev_info_regs - Device info register format (read-only) + * @signature: Signature value of 0x44455649 ('DEVI') + * @version: Current version of info + * @asic_type: Asic type + * @asic_rev: Asic revision + * @fw_status: Firmware status + * bit 0 - 1 = fw running + * bit 4-7 - 4 bit generation number, changes on fw restart + * @fw_heartbeat: Firmware heartbeat counter + * @serial_num: Serial number + * @fw_version: Firmware version + * @oprom_regs: oprom_regs to store oprom debug enable/disable and bmp + * @rsvd_pad1024: Struct padding + * @hwstamp: Hardware current timestamp registers + * @rsvd_pad2048: Struct padding + */ +struct pds_core_dev_info_regs { +#define PDS_CORE_DEVINFO_FWVERS_BUFLEN 32 +#define PDS_CORE_DEVINFO_SERIAL_BUFLEN 32 + u32 signature; + u8 version; + u8 asic_type; + u8 asic_rev; +#define PDS_CORE_FW_STS_F_STOPPED 0x00 +#define PDS_CORE_FW_STS_F_RUNNING 0x01 +#define PDS_CORE_FW_STS_F_GENERATION 0xF0 + u8 fw_status; + __le32 fw_heartbeat; + char fw_version[PDS_CORE_DEVINFO_FWVERS_BUFLEN]; + char serial_num[PDS_CORE_DEVINFO_SERIAL_BUFLEN]; + u8 oprom_regs[32]; /* reserved */ + u8 rsvd_pad1024[916]; + struct pds_core_dev_hwstamp_regs hwstamp; /* on 1k boundary */ + u8 rsvd_pad2048[1016]; +} __packed; + +/** + * struct pds_core_dev_cmd_regs - Device command register format (read-write) + * @doorbell: Device Cmd Doorbell, write-only + * Write a 1 to signal device to process cmd + * @done: Command completed indicator, poll for completion + * bit 0 == 1 when command is complete + * @cmd: Opcode-specific command bytes + * @comp: Opcode-specific response bytes + * @rsvd: Struct padding + * @data: Opcode-specific side-data + */ +struct pds_core_dev_cmd_regs { + u32 doorbell; + u32 done; + union pds_core_dev_cmd cmd; + union pds_core_dev_comp comp; + u8 rsvd[48]; + u32 data[478]; +} __packed; + +/** + * struct pds_core_dev_regs - Device register format for bar 0 page 0 + * @info: Device info registers + * @devcmd: Device command registers + */ +struct pds_core_dev_regs { + struct pds_core_dev_info_regs info; + struct pds_core_dev_cmd_regs devcmd; +} __packed; + +#ifndef __CHECKER__ +static_assert(sizeof(struct pds_core_drv_identity) <= 1912); +static_assert(sizeof(struct pds_core_dev_identity) <= 1912); +static_assert(sizeof(union pds_core_dev_cmd) == 64); +static_assert(sizeof(union pds_core_dev_comp) == 16); +static_assert(sizeof(struct pds_core_dev_info_regs) == 2048); +static_assert(sizeof(struct pds_core_dev_cmd_regs) == 2048); +static_assert(sizeof(struct pds_core_dev_regs) == 4096); +#endif /* __CHECKER__ */ + +#endif /* _PDS_CORE_IF_H_ */ diff --git a/include/linux/pds/pds_intr.h b/include/linux/pds/pds_intr.h new file mode 100644 index 000000000000..56277c37248c --- /dev/null +++ b/include/linux/pds/pds_intr.h @@ -0,0 +1,163 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) OR BSD-2-Clause */ +/* Copyright(c) 2023 Advanced Micro Devices, Inc. */ + +#ifndef _PDS_INTR_H_ +#define _PDS_INTR_H_ + +/* + * Interrupt control register + * @coal_init: Coalescing timer initial value, in + * device units. Use @identity->intr_coal_mult + * and @identity->intr_coal_div to convert from + * usecs to device units: + * + * coal_init = coal_usecs * coal_mutl / coal_div + * + * When an interrupt is sent the interrupt + * coalescing timer current value + * (@coalescing_curr) is initialized with this + * value and begins counting down. No more + * interrupts are sent until the coalescing + * timer reaches 0. When @coalescing_init=0 + * interrupt coalescing is effectively disabled + * and every interrupt assert results in an + * interrupt. Reset value: 0 + * @mask: Interrupt mask. When @mask=1 the interrupt + * resource will not send an interrupt. When + * @mask=0 the interrupt resource will send an + * interrupt if an interrupt event is pending + * or on the next interrupt assertion event. + * Reset value: 1 + * @credits: Interrupt credits. This register indicates + * how many interrupt events the hardware has + * sent. When written by software this + * register atomically decrements @int_credits + * by the value written. When @int_credits + * becomes 0 then the "pending interrupt" bit + * in the Interrupt Status register is cleared + * by the hardware and any pending but unsent + * interrupts are cleared. + * !!!IMPORTANT!!! This is a signed register. + * @flags: Interrupt control flags + * @unmask -- When this bit is written with a 1 + * the interrupt resource will set mask=0. + * @coal_timer_reset -- When this + * bit is written with a 1 the + * @coalescing_curr will be reloaded with + * @coalescing_init to reset the coalescing + * timer. + * @mask_on_assert: Automatically mask on assertion. When + * @mask_on_assert=1 the interrupt resource + * will set @mask=1 whenever an interrupt is + * sent. When using interrupts in Legacy + * Interrupt mode the driver must select + * @mask_on_assert=0 for proper interrupt + * operation. + * @coalescing_curr: Coalescing timer current value, in + * microseconds. When this value reaches 0 + * the interrupt resource is again eligible to + * send an interrupt. If an interrupt event + * is already pending when @coalescing_curr + * reaches 0 the pending interrupt will be + * sent, otherwise an interrupt will be sent + * on the next interrupt assertion event. + */ +struct pds_core_intr { + u32 coal_init; + u32 mask; + u16 credits; + u16 flags; +#define PDS_CORE_INTR_F_UNMASK 0x0001 +#define PDS_CORE_INTR_F_TIMER_RESET 0x0002 + u32 mask_on_assert; + u32 coalescing_curr; + u32 rsvd6[3]; +}; + +#ifndef __CHECKER__ +static_assert(sizeof(struct pds_core_intr) == 32); +#endif /* __CHECKER__ */ + +#define PDS_CORE_INTR_CTRL_REGS_MAX 2048 +#define PDS_CORE_INTR_CTRL_COAL_MAX 0x3F +#define PDS_CORE_INTR_INDEX_NOT_ASSIGNED -1 + +struct pds_core_intr_status { + u32 status[2]; +}; + +/** + * enum pds_core_intr_mask_vals - valid values for mask and mask_assert. + * @PDS_CORE_INTR_MASK_CLEAR: unmask interrupt. + * @PDS_CORE_INTR_MASK_SET: mask interrupt. + */ +enum pds_core_intr_mask_vals { + PDS_CORE_INTR_MASK_CLEAR = 0, + PDS_CORE_INTR_MASK_SET = 1, +}; + +/** + * enum pds_core_intr_credits_bits - Bitwise composition of credits values. + * @PDS_CORE_INTR_CRED_COUNT: bit mask of credit count, no shift needed. + * @PDS_CORE_INTR_CRED_COUNT_SIGNED: bit mask of credit count, including sign bit. + * @PDS_CORE_INTR_CRED_UNMASK: unmask the interrupt. + * @PDS_CORE_INTR_CRED_RESET_COALESCE: reset the coalesce timer. + * @PDS_CORE_INTR_CRED_REARM: unmask the and reset the timer. + */ +enum pds_core_intr_credits_bits { + PDS_CORE_INTR_CRED_COUNT = 0x7fffu, + PDS_CORE_INTR_CRED_COUNT_SIGNED = 0xffffu, + PDS_CORE_INTR_CRED_UNMASK = 0x10000u, + PDS_CORE_INTR_CRED_RESET_COALESCE = 0x20000u, + PDS_CORE_INTR_CRED_REARM = (PDS_CORE_INTR_CRED_UNMASK | + PDS_CORE_INTR_CRED_RESET_COALESCE), +}; + +static inline void +pds_core_intr_coal_init(struct pds_core_intr __iomem *intr_ctrl, u32 coal) +{ + iowrite32(coal, &intr_ctrl->coal_init); +} + +static inline void +pds_core_intr_mask(struct pds_core_intr __iomem *intr_ctrl, u32 mask) +{ + iowrite32(mask, &intr_ctrl->mask); +} + +static inline void +pds_core_intr_credits(struct pds_core_intr __iomem *intr_ctrl, + u32 cred, u32 flags) +{ + if (WARN_ON_ONCE(cred > PDS_CORE_INTR_CRED_COUNT)) { + cred = ioread32(&intr_ctrl->credits); + cred &= PDS_CORE_INTR_CRED_COUNT_SIGNED; + } + + iowrite32(cred | flags, &intr_ctrl->credits); +} + +static inline void +pds_core_intr_clean_flags(struct pds_core_intr __iomem *intr_ctrl, u32 flags) +{ + u32 cred; + + cred = ioread32(&intr_ctrl->credits); + cred &= PDS_CORE_INTR_CRED_COUNT_SIGNED; + cred |= flags; + iowrite32(cred, &intr_ctrl->credits); +} + +static inline void +pds_core_intr_clean(struct pds_core_intr __iomem *intr_ctrl) +{ + pds_core_intr_clean_flags(intr_ctrl, PDS_CORE_INTR_CRED_RESET_COALESCE); +} + +static inline void +pds_core_intr_mask_assert(struct pds_core_intr __iomem *intr_ctrl, u32 mask) +{ + iowrite32(mask, &intr_ctrl->mask_on_assert); +} + +#endif /* _PDS_INTR_H_ */ diff --git a/include/linux/sctp.h b/include/linux/sctp.h index 358dc08e0831..836a7e200f39 100644 --- a/include/linux/sctp.h +++ b/include/linux/sctp.h @@ -222,7 +222,7 @@ struct sctp_datahdr { __be16 stream; __be16 ssn; __u32 ppid; - __u8 payload[]; + /* __u8 payload[]; */ }; struct sctp_data_chunk { @@ -270,7 +270,7 @@ struct sctp_inithdr { __be16 num_outbound_streams; __be16 num_inbound_streams; __be32 initial_tsn; - __u8 params[]; + /* __u8 params[]; */ }; struct sctp_init_chunk { @@ -385,7 +385,7 @@ struct sctp_sackhdr { __be32 a_rwnd; __be16 num_gap_ack_blocks; __be16 num_dup_tsns; - union sctp_sack_variable variable[]; + /* union sctp_sack_variable variable[]; */ }; struct sctp_sack_chunk { @@ -443,7 +443,7 @@ struct sctp_shutdown_chunk { struct sctp_errhdr { __be16 cause; __be16 length; - __u8 variable[]; + /* __u8 variable[]; */ }; struct sctp_operr_chunk { @@ -603,7 +603,7 @@ struct sctp_fwdtsn_skip { struct sctp_fwdtsn_hdr { __be32 new_cum_tsn; - struct sctp_fwdtsn_skip skip[]; + /* struct sctp_fwdtsn_skip skip[]; */ }; struct sctp_fwdtsn_chunk { @@ -620,7 +620,7 @@ struct sctp_ifwdtsn_skip { struct sctp_ifwdtsn_hdr { __be32 new_cum_tsn; - struct sctp_ifwdtsn_skip skip[]; + /* struct sctp_ifwdtsn_skip skip[]; */ }; struct sctp_ifwdtsn_chunk { @@ -667,7 +667,7 @@ struct sctp_addip_param { struct sctp_addiphdr { __be32 serial; - __u8 params[]; + /* __u8 params[]; */ }; struct sctp_addip_chunk { @@ -727,7 +727,7 @@ struct sctp_addip_chunk { struct sctp_authhdr { __be16 shkey_id; __be16 hmac_id; - __u8 hmac[]; + /* __u8 hmac[]; */ }; struct sctp_auth_chunk { @@ -742,7 +742,7 @@ struct sctp_infox { struct sctp_reconf_chunk { struct sctp_chunkhdr chunk_hdr; - __u8 params[]; + /* __u8 params[]; */ }; struct sctp_strreset_outreq { diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 5f120bbab9da..738776ab8838 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -37,7 +37,7 @@ #include <linux/netfilter/nf_conntrack_common.h> #endif #include <net/net_debug.h> -#include <net/dropreason.h> +#include <net/dropreason-core.h> /** * DOC: skb checksums @@ -294,6 +294,7 @@ struct nf_bridge_info { u8 pkt_otherhost:1; u8 in_prerouting:1; u8 bridged_dnat:1; + u8 sabotage_in_done:1; __u16 frag_max_size; struct net_device *physindev; @@ -3252,7 +3253,7 @@ static inline struct sk_buff *napi_alloc_skb(struct napi_struct *napi, void napi_consume_skb(struct sk_buff *skb, int budget); void napi_skb_free_stolen_head(struct sk_buff *skb); -void __kfree_skb_defer(struct sk_buff *skb); +void __napi_kfree_skb(struct sk_buff *skb, enum skb_drop_reason reason); /** * __dev_alloc_pages - allocate page for network Rx @@ -4728,7 +4729,7 @@ static inline void nf_reset_ct(struct sk_buff *skb) static inline void nf_reset_trace(struct sk_buff *skb) { -#if IS_ENABLED(CONFIG_NETFILTER_XT_TARGET_TRACE) || defined(CONFIG_NF_TABLES) +#if IS_ENABLED(CONFIG_NETFILTER_XT_TARGET_TRACE) || IS_ENABLED(CONFIG_NF_TABLES) skb->nf_trace = 0; #endif } @@ -4748,7 +4749,7 @@ static inline void __nf_copy(struct sk_buff *dst, const struct sk_buff *src, dst->_nfct = src->_nfct; nf_conntrack_get(skb_nfct(src)); #endif -#if IS_ENABLED(CONFIG_NETFILTER_XT_TARGET_TRACE) || defined(CONFIG_NF_TABLES) +#if IS_ENABLED(CONFIG_NETFILTER_XT_TARGET_TRACE) || IS_ENABLED(CONFIG_NF_TABLES) if (copy) dst->nf_trace = src->nf_trace; #endif @@ -5065,6 +5066,15 @@ static inline void skb_reset_redirect(struct sk_buff *skb) skb->redirected = 0; } +static inline void skb_set_redirected_noclear(struct sk_buff *skb, + bool from_ingress) +{ + skb->redirected = 1; +#ifdef CONFIG_NET_REDIRECT + skb->from_ingress = from_ingress; +#endif +} + static inline bool skb_csum_is_sctp(struct sk_buff *skb) { #if IS_ENABLED(CONFIG_IP_SCTP) diff --git a/include/net/dropreason-core.h b/include/net/dropreason-core.h new file mode 100644 index 000000000000..a2b953b57689 --- /dev/null +++ b/include/net/dropreason-core.h @@ -0,0 +1,370 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +#ifndef _LINUX_DROPREASON_CORE_H +#define _LINUX_DROPREASON_CORE_H + +#define DEFINE_DROP_REASON(FN, FNe) \ + FN(NOT_SPECIFIED) \ + FN(NO_SOCKET) \ + FN(PKT_TOO_SMALL) \ + FN(TCP_CSUM) \ + FN(SOCKET_FILTER) \ + FN(UDP_CSUM) \ + FN(NETFILTER_DROP) \ + FN(OTHERHOST) \ + FN(IP_CSUM) \ + FN(IP_INHDR) \ + FN(IP_RPFILTER) \ + FN(UNICAST_IN_L2_MULTICAST) \ + FN(XFRM_POLICY) \ + FN(IP_NOPROTO) \ + FN(SOCKET_RCVBUFF) \ + FN(PROTO_MEM) \ + FN(TCP_MD5NOTFOUND) \ + FN(TCP_MD5UNEXPECTED) \ + FN(TCP_MD5FAILURE) \ + FN(SOCKET_BACKLOG) \ + FN(TCP_FLAGS) \ + FN(TCP_ZEROWINDOW) \ + FN(TCP_OLD_DATA) \ + FN(TCP_OVERWINDOW) \ + FN(TCP_OFOMERGE) \ + FN(TCP_RFC7323_PAWS) \ + FN(TCP_INVALID_SEQUENCE) \ + FN(TCP_RESET) \ + FN(TCP_INVALID_SYN) \ + FN(TCP_CLOSE) \ + FN(TCP_FASTOPEN) \ + FN(TCP_OLD_ACK) \ + FN(TCP_TOO_OLD_ACK) \ + FN(TCP_ACK_UNSENT_DATA) \ + FN(TCP_OFO_QUEUE_PRUNE) \ + FN(TCP_OFO_DROP) \ + FN(IP_OUTNOROUTES) \ + FN(BPF_CGROUP_EGRESS) \ + FN(IPV6DISABLED) \ + FN(NEIGH_CREATEFAIL) \ + FN(NEIGH_FAILED) \ + FN(NEIGH_QUEUEFULL) \ + FN(NEIGH_DEAD) \ + FN(TC_EGRESS) \ + FN(QDISC_DROP) \ + FN(CPU_BACKLOG) \ + FN(XDP) \ + FN(TC_INGRESS) \ + FN(UNHANDLED_PROTO) \ + FN(SKB_CSUM) \ + FN(SKB_GSO_SEG) \ + FN(SKB_UCOPY_FAULT) \ + FN(DEV_HDR) \ + FN(DEV_READY) \ + FN(FULL_RING) \ + FN(NOMEM) \ + FN(HDR_TRUNC) \ + FN(TAP_FILTER) \ + FN(TAP_TXFILTER) \ + FN(ICMP_CSUM) \ + FN(INVALID_PROTO) \ + FN(IP_INADDRERRORS) \ + FN(IP_INNOROUTES) \ + FN(PKT_TOO_BIG) \ + FN(DUP_FRAG) \ + FN(FRAG_REASM_TIMEOUT) \ + FN(FRAG_TOO_FAR) \ + FN(TCP_MINTTL) \ + FN(IPV6_BAD_EXTHDR) \ + FN(IPV6_NDISC_FRAG) \ + FN(IPV6_NDISC_HOP_LIMIT) \ + FN(IPV6_NDISC_BAD_CODE) \ + FN(IPV6_NDISC_BAD_OPTIONS) \ + FN(IPV6_NDISC_NS_OTHERHOST) \ + FNe(MAX) + +/** + * enum skb_drop_reason - the reasons of skb drops + * + * The reason of skb drop, which is used in kfree_skb_reason(). + */ +enum skb_drop_reason { + /** + * @SKB_NOT_DROPPED_YET: skb is not dropped yet (used for no-drop case) + */ + SKB_NOT_DROPPED_YET = 0, + /** @SKB_CONSUMED: packet has been consumed */ + SKB_CONSUMED, + /** @SKB_DROP_REASON_NOT_SPECIFIED: drop reason is not specified */ + SKB_DROP_REASON_NOT_SPECIFIED, + /** @SKB_DROP_REASON_NO_SOCKET: socket not found */ + SKB_DROP_REASON_NO_SOCKET, + /** @SKB_DROP_REASON_PKT_TOO_SMALL: packet size is too small */ + SKB_DROP_REASON_PKT_TOO_SMALL, + /** @SKB_DROP_REASON_TCP_CSUM: TCP checksum error */ + SKB_DROP_REASON_TCP_CSUM, + /** @SKB_DROP_REASON_SOCKET_FILTER: dropped by socket filter */ + SKB_DROP_REASON_SOCKET_FILTER, + /** @SKB_DROP_REASON_UDP_CSUM: UDP checksum error */ + SKB_DROP_REASON_UDP_CSUM, + /** @SKB_DROP_REASON_NETFILTER_DROP: dropped by netfilter */ + SKB_DROP_REASON_NETFILTER_DROP, + /** + * @SKB_DROP_REASON_OTHERHOST: packet don't belong to current host + * (interface is in promisc mode) + */ + SKB_DROP_REASON_OTHERHOST, + /** @SKB_DROP_REASON_IP_CSUM: IP checksum error */ + SKB_DROP_REASON_IP_CSUM, + /** + * @SKB_DROP_REASON_IP_INHDR: there is something wrong with IP header (see + * IPSTATS_MIB_INHDRERRORS) + */ + SKB_DROP_REASON_IP_INHDR, + /** + * @SKB_DROP_REASON_IP_RPFILTER: IP rpfilter validate failed. see the + * document for rp_filter in ip-sysctl.rst for more information + */ + SKB_DROP_REASON_IP_RPFILTER, + /** + * @SKB_DROP_REASON_UNICAST_IN_L2_MULTICAST: destination address of L2 is + * multicast, but L3 is unicast. + */ + SKB_DROP_REASON_UNICAST_IN_L2_MULTICAST, + /** @SKB_DROP_REASON_XFRM_POLICY: xfrm policy check failed */ + SKB_DROP_REASON_XFRM_POLICY, + /** @SKB_DROP_REASON_IP_NOPROTO: no support for IP protocol */ + SKB_DROP_REASON_IP_NOPROTO, + /** @SKB_DROP_REASON_SOCKET_RCVBUFF: socket receive buff is full */ + SKB_DROP_REASON_SOCKET_RCVBUFF, + /** + * @SKB_DROP_REASON_PROTO_MEM: proto memory limition, such as udp packet + * drop out of udp_memory_allocated. + */ + SKB_DROP_REASON_PROTO_MEM, + /** + * @SKB_DROP_REASON_TCP_MD5NOTFOUND: no MD5 hash and one expected, + * corresponding to LINUX_MIB_TCPMD5NOTFOUND + */ + SKB_DROP_REASON_TCP_MD5NOTFOUND, + /** + * @SKB_DROP_REASON_TCP_MD5UNEXPECTED: MD5 hash and we're not expecting + * one, corresponding to LINUX_MIB_TCPMD5UNEXPECTED + */ + SKB_DROP_REASON_TCP_MD5UNEXPECTED, + /** + * @SKB_DROP_REASON_TCP_MD5FAILURE: MD5 hash and its wrong, corresponding + * to LINUX_MIB_TCPMD5FAILURE + */ + SKB_DROP_REASON_TCP_MD5FAILURE, + /** + * @SKB_DROP_REASON_SOCKET_BACKLOG: failed to add skb to socket backlog ( + * see LINUX_MIB_TCPBACKLOGDROP) + */ + SKB_DROP_REASON_SOCKET_BACKLOG, + /** @SKB_DROP_REASON_TCP_FLAGS: TCP flags invalid */ + SKB_DROP_REASON_TCP_FLAGS, + /** + * @SKB_DROP_REASON_TCP_ZEROWINDOW: TCP receive window size is zero, + * see LINUX_MIB_TCPZEROWINDOWDROP + */ + SKB_DROP_REASON_TCP_ZEROWINDOW, + /** + * @SKB_DROP_REASON_TCP_OLD_DATA: the TCP data reveived is already + * received before (spurious retrans may happened), see + * LINUX_MIB_DELAYEDACKLOST + */ + SKB_DROP_REASON_TCP_OLD_DATA, + /** + * @SKB_DROP_REASON_TCP_OVERWINDOW: the TCP data is out of window, + * the seq of the first byte exceed the right edges of receive + * window + */ + SKB_DROP_REASON_TCP_OVERWINDOW, + /** + * @SKB_DROP_REASON_TCP_OFOMERGE: the data of skb is already in the ofo + * queue, corresponding to LINUX_MIB_TCPOFOMERGE + */ + SKB_DROP_REASON_TCP_OFOMERGE, + /** + * @SKB_DROP_REASON_TCP_RFC7323_PAWS: PAWS check, corresponding to + * LINUX_MIB_PAWSESTABREJECTED + */ + SKB_DROP_REASON_TCP_RFC7323_PAWS, + /** @SKB_DROP_REASON_TCP_INVALID_SEQUENCE: Not acceptable SEQ field */ + SKB_DROP_REASON_TCP_INVALID_SEQUENCE, + /** @SKB_DROP_REASON_TCP_RESET: Invalid RST packet */ + SKB_DROP_REASON_TCP_RESET, + /** + * @SKB_DROP_REASON_TCP_INVALID_SYN: Incoming packet has unexpected + * SYN flag + */ + SKB_DROP_REASON_TCP_INVALID_SYN, + /** @SKB_DROP_REASON_TCP_CLOSE: TCP socket in CLOSE state */ + SKB_DROP_REASON_TCP_CLOSE, + /** @SKB_DROP_REASON_TCP_FASTOPEN: dropped by FASTOPEN request socket */ + SKB_DROP_REASON_TCP_FASTOPEN, + /** @SKB_DROP_REASON_TCP_OLD_ACK: TCP ACK is old, but in window */ + SKB_DROP_REASON_TCP_OLD_ACK, + /** @SKB_DROP_REASON_TCP_TOO_OLD_ACK: TCP ACK is too old */ + SKB_DROP_REASON_TCP_TOO_OLD_ACK, + /** + * @SKB_DROP_REASON_TCP_ACK_UNSENT_DATA: TCP ACK for data we haven't + * sent yet + */ + SKB_DROP_REASON_TCP_ACK_UNSENT_DATA, + /** @SKB_DROP_REASON_TCP_OFO_QUEUE_PRUNE: pruned from TCP OFO queue */ + SKB_DROP_REASON_TCP_OFO_QUEUE_PRUNE, + /** @SKB_DROP_REASON_TCP_OFO_DROP: data already in receive queue */ + SKB_DROP_REASON_TCP_OFO_DROP, + /** @SKB_DROP_REASON_IP_OUTNOROUTES: route lookup failed */ + SKB_DROP_REASON_IP_OUTNOROUTES, + /** + * @SKB_DROP_REASON_BPF_CGROUP_EGRESS: dropped by BPF_PROG_TYPE_CGROUP_SKB + * eBPF program + */ + SKB_DROP_REASON_BPF_CGROUP_EGRESS, + /** @SKB_DROP_REASON_IPV6DISABLED: IPv6 is disabled on the device */ + SKB_DROP_REASON_IPV6DISABLED, + /** @SKB_DROP_REASON_NEIGH_CREATEFAIL: failed to create neigh entry */ + SKB_DROP_REASON_NEIGH_CREATEFAIL, + /** @SKB_DROP_REASON_NEIGH_FAILED: neigh entry in failed state */ + SKB_DROP_REASON_NEIGH_FAILED, + /** @SKB_DROP_REASON_NEIGH_QUEUEFULL: arp_queue for neigh entry is full */ + SKB_DROP_REASON_NEIGH_QUEUEFULL, + /** @SKB_DROP_REASON_NEIGH_DEAD: neigh entry is dead */ + SKB_DROP_REASON_NEIGH_DEAD, + /** @SKB_DROP_REASON_TC_EGRESS: dropped in TC egress HOOK */ + SKB_DROP_REASON_TC_EGRESS, + /** + * @SKB_DROP_REASON_QDISC_DROP: dropped by qdisc when packet outputting ( + * failed to enqueue to current qdisc) + */ + SKB_DROP_REASON_QDISC_DROP, + /** + * @SKB_DROP_REASON_CPU_BACKLOG: failed to enqueue the skb to the per CPU + * backlog queue. This can be caused by backlog queue full (see + * netdev_max_backlog in net.rst) or RPS flow limit + */ + SKB_DROP_REASON_CPU_BACKLOG, + /** @SKB_DROP_REASON_XDP: dropped by XDP in input path */ + SKB_DROP_REASON_XDP, + /** @SKB_DROP_REASON_TC_INGRESS: dropped in TC ingress HOOK */ + SKB_DROP_REASON_TC_INGRESS, + /** @SKB_DROP_REASON_UNHANDLED_PROTO: protocol not implemented or not supported */ + SKB_DROP_REASON_UNHANDLED_PROTO, + /** @SKB_DROP_REASON_SKB_CSUM: sk_buff checksum computation error */ + SKB_DROP_REASON_SKB_CSUM, + /** @SKB_DROP_REASON_SKB_GSO_SEG: gso segmentation error */ + SKB_DROP_REASON_SKB_GSO_SEG, + /** + * @SKB_DROP_REASON_SKB_UCOPY_FAULT: failed to copy data from user space, + * e.g., via zerocopy_sg_from_iter() or skb_orphan_frags_rx() + */ + SKB_DROP_REASON_SKB_UCOPY_FAULT, + /** @SKB_DROP_REASON_DEV_HDR: device driver specific header/metadata is invalid */ + SKB_DROP_REASON_DEV_HDR, + /** + * @SKB_DROP_REASON_DEV_READY: the device is not ready to xmit/recv due to + * any of its data structure that is not up/ready/initialized, + * e.g., the IFF_UP is not set, or driver specific tun->tfiles[txq] + * is not initialized + */ + SKB_DROP_REASON_DEV_READY, + /** @SKB_DROP_REASON_FULL_RING: ring buffer is full */ + SKB_DROP_REASON_FULL_RING, + /** @SKB_DROP_REASON_NOMEM: error due to OOM */ + SKB_DROP_REASON_NOMEM, + /** + * @SKB_DROP_REASON_HDR_TRUNC: failed to trunc/extract the header from + * networking data, e.g., failed to pull the protocol header from + * frags via pskb_may_pull() + */ + SKB_DROP_REASON_HDR_TRUNC, + /** + * @SKB_DROP_REASON_TAP_FILTER: dropped by (ebpf) filter directly attached + * to tun/tap, e.g., via TUNSETFILTEREBPF + */ + SKB_DROP_REASON_TAP_FILTER, + /** + * @SKB_DROP_REASON_TAP_TXFILTER: dropped by tx filter implemented at + * tun/tap, e.g., check_filter() + */ + SKB_DROP_REASON_TAP_TXFILTER, + /** @SKB_DROP_REASON_ICMP_CSUM: ICMP checksum error */ + SKB_DROP_REASON_ICMP_CSUM, + /** + * @SKB_DROP_REASON_INVALID_PROTO: the packet doesn't follow RFC 2211, + * such as a broadcasts ICMP_TIMESTAMP + */ + SKB_DROP_REASON_INVALID_PROTO, + /** + * @SKB_DROP_REASON_IP_INADDRERRORS: host unreachable, corresponding to + * IPSTATS_MIB_INADDRERRORS + */ + SKB_DROP_REASON_IP_INADDRERRORS, + /** + * @SKB_DROP_REASON_IP_INNOROUTES: network unreachable, corresponding to + * IPSTATS_MIB_INADDRERRORS + */ + SKB_DROP_REASON_IP_INNOROUTES, + /** + * @SKB_DROP_REASON_PKT_TOO_BIG: packet size is too big (maybe exceed the + * MTU) + */ + SKB_DROP_REASON_PKT_TOO_BIG, + /** @SKB_DROP_REASON_DUP_FRAG: duplicate fragment */ + SKB_DROP_REASON_DUP_FRAG, + /** @SKB_DROP_REASON_FRAG_REASM_TIMEOUT: fragment reassembly timeout */ + SKB_DROP_REASON_FRAG_REASM_TIMEOUT, + /** + * @SKB_DROP_REASON_FRAG_TOO_FAR: ipv4 fragment too far. + * (/proc/sys/net/ipv4/ipfrag_max_dist) + */ + SKB_DROP_REASON_FRAG_TOO_FAR, + /** + * @SKB_DROP_REASON_TCP_MINTTL: ipv4 ttl or ipv6 hoplimit below + * the threshold (IP_MINTTL or IPV6_MINHOPCOUNT). + */ + SKB_DROP_REASON_TCP_MINTTL, + /** @SKB_DROP_REASON_IPV6_BAD_EXTHDR: Bad IPv6 extension header. */ + SKB_DROP_REASON_IPV6_BAD_EXTHDR, + /** @SKB_DROP_REASON_IPV6_NDISC_FRAG: invalid frag (suppress_frag_ndisc). */ + SKB_DROP_REASON_IPV6_NDISC_FRAG, + /** @SKB_DROP_REASON_IPV6_NDISC_HOP_LIMIT: invalid hop limit. */ + SKB_DROP_REASON_IPV6_NDISC_HOP_LIMIT, + /** @SKB_DROP_REASON_IPV6_NDISC_BAD_CODE: invalid NDISC icmp6 code. */ + SKB_DROP_REASON_IPV6_NDISC_BAD_CODE, + /** @SKB_DROP_REASON_IPV6_NDISC_BAD_OPTIONS: invalid NDISC options. */ + SKB_DROP_REASON_IPV6_NDISC_BAD_OPTIONS, + /** + * @SKB_DROP_REASON_IPV6_NDISC_NS_OTHERHOST: NEIGHBOUR SOLICITATION + * for another host. + */ + SKB_DROP_REASON_IPV6_NDISC_NS_OTHERHOST, + /** + * @SKB_DROP_REASON_MAX: the maximum of core drop reasons, which + * shouldn't be used as a real 'reason' - only for tracing code gen + */ + SKB_DROP_REASON_MAX, + + /** + * @SKB_DROP_REASON_SUBSYS_MASK: subsystem mask in drop reasons, + * see &enum skb_drop_reason_subsys + */ + SKB_DROP_REASON_SUBSYS_MASK = 0xffff0000, +}; + +#define SKB_DROP_REASON_SUBSYS_SHIFT 16 + +#define SKB_DR_INIT(name, reason) \ + enum skb_drop_reason name = SKB_DROP_REASON_##reason +#define SKB_DR(name) \ + SKB_DR_INIT(name, NOT_SPECIFIED) +#define SKB_DR_SET(name, reason) \ + (name = SKB_DROP_REASON_##reason) +#define SKB_DR_OR(name, reason) \ + do { \ + if (name == SKB_DROP_REASON_NOT_SPECIFIED || \ + name == SKB_NOT_DROPPED_YET) \ + SKB_DR_SET(name, reason); \ + } while (0) + +#endif diff --git a/include/net/dropreason.h b/include/net/dropreason.h index c0a3ea806cd5..685fb37df8e8 100644 --- a/include/net/dropreason.h +++ b/include/net/dropreason.h @@ -2,362 +2,42 @@ #ifndef _LINUX_DROPREASON_H #define _LINUX_DROPREASON_H - -#define DEFINE_DROP_REASON(FN, FNe) \ - FN(NOT_SPECIFIED) \ - FN(NO_SOCKET) \ - FN(PKT_TOO_SMALL) \ - FN(TCP_CSUM) \ - FN(SOCKET_FILTER) \ - FN(UDP_CSUM) \ - FN(NETFILTER_DROP) \ - FN(OTHERHOST) \ - FN(IP_CSUM) \ - FN(IP_INHDR) \ - FN(IP_RPFILTER) \ - FN(UNICAST_IN_L2_MULTICAST) \ - FN(XFRM_POLICY) \ - FN(IP_NOPROTO) \ - FN(SOCKET_RCVBUFF) \ - FN(PROTO_MEM) \ - FN(TCP_MD5NOTFOUND) \ - FN(TCP_MD5UNEXPECTED) \ - FN(TCP_MD5FAILURE) \ - FN(SOCKET_BACKLOG) \ - FN(TCP_FLAGS) \ - FN(TCP_ZEROWINDOW) \ - FN(TCP_OLD_DATA) \ - FN(TCP_OVERWINDOW) \ - FN(TCP_OFOMERGE) \ - FN(TCP_RFC7323_PAWS) \ - FN(TCP_INVALID_SEQUENCE) \ - FN(TCP_RESET) \ - FN(TCP_INVALID_SYN) \ - FN(TCP_CLOSE) \ - FN(TCP_FASTOPEN) \ - FN(TCP_OLD_ACK) \ - FN(TCP_TOO_OLD_ACK) \ - FN(TCP_ACK_UNSENT_DATA) \ - FN(TCP_OFO_QUEUE_PRUNE) \ - FN(TCP_OFO_DROP) \ - FN(IP_OUTNOROUTES) \ - FN(BPF_CGROUP_EGRESS) \ - FN(IPV6DISABLED) \ - FN(NEIGH_CREATEFAIL) \ - FN(NEIGH_FAILED) \ - FN(NEIGH_QUEUEFULL) \ - FN(NEIGH_DEAD) \ - FN(TC_EGRESS) \ - FN(QDISC_DROP) \ - FN(CPU_BACKLOG) \ - FN(XDP) \ - FN(TC_INGRESS) \ - FN(UNHANDLED_PROTO) \ - FN(SKB_CSUM) \ - FN(SKB_GSO_SEG) \ - FN(SKB_UCOPY_FAULT) \ - FN(DEV_HDR) \ - FN(DEV_READY) \ - FN(FULL_RING) \ - FN(NOMEM) \ - FN(HDR_TRUNC) \ - FN(TAP_FILTER) \ - FN(TAP_TXFILTER) \ - FN(ICMP_CSUM) \ - FN(INVALID_PROTO) \ - FN(IP_INADDRERRORS) \ - FN(IP_INNOROUTES) \ - FN(PKT_TOO_BIG) \ - FN(DUP_FRAG) \ - FN(FRAG_REASM_TIMEOUT) \ - FN(FRAG_TOO_FAR) \ - FN(TCP_MINTTL) \ - FN(IPV6_BAD_EXTHDR) \ - FN(IPV6_NDISC_FRAG) \ - FN(IPV6_NDISC_HOP_LIMIT) \ - FN(IPV6_NDISC_BAD_CODE) \ - FN(IPV6_NDISC_BAD_OPTIONS) \ - FN(IPV6_NDISC_NS_OTHERHOST) \ - FNe(MAX) +#include <net/dropreason-core.h> /** - * enum skb_drop_reason - the reasons of skb drops - * - * The reason of skb drop, which is used in kfree_skb_reason(). + * enum skb_drop_reason_subsys - subsystem tag for (extended) drop reasons */ -enum skb_drop_reason { - /** - * @SKB_NOT_DROPPED_YET: skb is not dropped yet (used for no-drop case) - */ - SKB_NOT_DROPPED_YET = 0, - /** @SKB_CONSUMED: packet has been consumed */ - SKB_CONSUMED, - /** @SKB_DROP_REASON_NOT_SPECIFIED: drop reason is not specified */ - SKB_DROP_REASON_NOT_SPECIFIED, - /** @SKB_DROP_REASON_NO_SOCKET: socket not found */ - SKB_DROP_REASON_NO_SOCKET, - /** @SKB_DROP_REASON_PKT_TOO_SMALL: packet size is too small */ - SKB_DROP_REASON_PKT_TOO_SMALL, - /** @SKB_DROP_REASON_TCP_CSUM: TCP checksum error */ - SKB_DROP_REASON_TCP_CSUM, - /** @SKB_DROP_REASON_SOCKET_FILTER: dropped by socket filter */ - SKB_DROP_REASON_SOCKET_FILTER, - /** @SKB_DROP_REASON_UDP_CSUM: UDP checksum error */ - SKB_DROP_REASON_UDP_CSUM, - /** @SKB_DROP_REASON_NETFILTER_DROP: dropped by netfilter */ - SKB_DROP_REASON_NETFILTER_DROP, - /** - * @SKB_DROP_REASON_OTHERHOST: packet don't belong to current host - * (interface is in promisc mode) - */ - SKB_DROP_REASON_OTHERHOST, - /** @SKB_DROP_REASON_IP_CSUM: IP checksum error */ - SKB_DROP_REASON_IP_CSUM, - /** - * @SKB_DROP_REASON_IP_INHDR: there is something wrong with IP header (see - * IPSTATS_MIB_INHDRERRORS) - */ - SKB_DROP_REASON_IP_INHDR, - /** - * @SKB_DROP_REASON_IP_RPFILTER: IP rpfilter validate failed. see the - * document for rp_filter in ip-sysctl.rst for more information - */ - SKB_DROP_REASON_IP_RPFILTER, - /** - * @SKB_DROP_REASON_UNICAST_IN_L2_MULTICAST: destination address of L2 is - * multicast, but L3 is unicast. - */ - SKB_DROP_REASON_UNICAST_IN_L2_MULTICAST, - /** @SKB_DROP_REASON_XFRM_POLICY: xfrm policy check failed */ - SKB_DROP_REASON_XFRM_POLICY, - /** @SKB_DROP_REASON_IP_NOPROTO: no support for IP protocol */ - SKB_DROP_REASON_IP_NOPROTO, - /** @SKB_DROP_REASON_SOCKET_RCVBUFF: socket receive buff is full */ - SKB_DROP_REASON_SOCKET_RCVBUFF, - /** - * @SKB_DROP_REASON_PROTO_MEM: proto memory limition, such as udp packet - * drop out of udp_memory_allocated. - */ - SKB_DROP_REASON_PROTO_MEM, - /** - * @SKB_DROP_REASON_TCP_MD5NOTFOUND: no MD5 hash and one expected, - * corresponding to LINUX_MIB_TCPMD5NOTFOUND - */ - SKB_DROP_REASON_TCP_MD5NOTFOUND, - /** - * @SKB_DROP_REASON_TCP_MD5UNEXPECTED: MD5 hash and we're not expecting - * one, corresponding to LINUX_MIB_TCPMD5UNEXPECTED - */ - SKB_DROP_REASON_TCP_MD5UNEXPECTED, - /** - * @SKB_DROP_REASON_TCP_MD5FAILURE: MD5 hash and its wrong, corresponding - * to LINUX_MIB_TCPMD5FAILURE - */ - SKB_DROP_REASON_TCP_MD5FAILURE, - /** - * @SKB_DROP_REASON_SOCKET_BACKLOG: failed to add skb to socket backlog ( - * see LINUX_MIB_TCPBACKLOGDROP) - */ - SKB_DROP_REASON_SOCKET_BACKLOG, - /** @SKB_DROP_REASON_TCP_FLAGS: TCP flags invalid */ - SKB_DROP_REASON_TCP_FLAGS, - /** - * @SKB_DROP_REASON_TCP_ZEROWINDOW: TCP receive window size is zero, - * see LINUX_MIB_TCPZEROWINDOWDROP - */ - SKB_DROP_REASON_TCP_ZEROWINDOW, - /** - * @SKB_DROP_REASON_TCP_OLD_DATA: the TCP data reveived is already - * received before (spurious retrans may happened), see - * LINUX_MIB_DELAYEDACKLOST - */ - SKB_DROP_REASON_TCP_OLD_DATA, - /** - * @SKB_DROP_REASON_TCP_OVERWINDOW: the TCP data is out of window, - * the seq of the first byte exceed the right edges of receive - * window - */ - SKB_DROP_REASON_TCP_OVERWINDOW, - /** - * @SKB_DROP_REASON_TCP_OFOMERGE: the data of skb is already in the ofo - * queue, corresponding to LINUX_MIB_TCPOFOMERGE - */ - SKB_DROP_REASON_TCP_OFOMERGE, - /** - * @SKB_DROP_REASON_TCP_RFC7323_PAWS: PAWS check, corresponding to - * LINUX_MIB_PAWSESTABREJECTED - */ - SKB_DROP_REASON_TCP_RFC7323_PAWS, - /** @SKB_DROP_REASON_TCP_INVALID_SEQUENCE: Not acceptable SEQ field */ - SKB_DROP_REASON_TCP_INVALID_SEQUENCE, - /** @SKB_DROP_REASON_TCP_RESET: Invalid RST packet */ - SKB_DROP_REASON_TCP_RESET, - /** - * @SKB_DROP_REASON_TCP_INVALID_SYN: Incoming packet has unexpected - * SYN flag - */ - SKB_DROP_REASON_TCP_INVALID_SYN, - /** @SKB_DROP_REASON_TCP_CLOSE: TCP socket in CLOSE state */ - SKB_DROP_REASON_TCP_CLOSE, - /** @SKB_DROP_REASON_TCP_FASTOPEN: dropped by FASTOPEN request socket */ - SKB_DROP_REASON_TCP_FASTOPEN, - /** @SKB_DROP_REASON_TCP_OLD_ACK: TCP ACK is old, but in window */ - SKB_DROP_REASON_TCP_OLD_ACK, - /** @SKB_DROP_REASON_TCP_TOO_OLD_ACK: TCP ACK is too old */ - SKB_DROP_REASON_TCP_TOO_OLD_ACK, - /** - * @SKB_DROP_REASON_TCP_ACK_UNSENT_DATA: TCP ACK for data we haven't - * sent yet - */ - SKB_DROP_REASON_TCP_ACK_UNSENT_DATA, - /** @SKB_DROP_REASON_TCP_OFO_QUEUE_PRUNE: pruned from TCP OFO queue */ - SKB_DROP_REASON_TCP_OFO_QUEUE_PRUNE, - /** @SKB_DROP_REASON_TCP_OFO_DROP: data already in receive queue */ - SKB_DROP_REASON_TCP_OFO_DROP, - /** @SKB_DROP_REASON_IP_OUTNOROUTES: route lookup failed */ - SKB_DROP_REASON_IP_OUTNOROUTES, - /** - * @SKB_DROP_REASON_BPF_CGROUP_EGRESS: dropped by BPF_PROG_TYPE_CGROUP_SKB - * eBPF program - */ - SKB_DROP_REASON_BPF_CGROUP_EGRESS, - /** @SKB_DROP_REASON_IPV6DISABLED: IPv6 is disabled on the device */ - SKB_DROP_REASON_IPV6DISABLED, - /** @SKB_DROP_REASON_NEIGH_CREATEFAIL: failed to create neigh entry */ - SKB_DROP_REASON_NEIGH_CREATEFAIL, - /** @SKB_DROP_REASON_NEIGH_FAILED: neigh entry in failed state */ - SKB_DROP_REASON_NEIGH_FAILED, - /** @SKB_DROP_REASON_NEIGH_QUEUEFULL: arp_queue for neigh entry is full */ - SKB_DROP_REASON_NEIGH_QUEUEFULL, - /** @SKB_DROP_REASON_NEIGH_DEAD: neigh entry is dead */ - SKB_DROP_REASON_NEIGH_DEAD, - /** @SKB_DROP_REASON_TC_EGRESS: dropped in TC egress HOOK */ - SKB_DROP_REASON_TC_EGRESS, - /** - * @SKB_DROP_REASON_QDISC_DROP: dropped by qdisc when packet outputting ( - * failed to enqueue to current qdisc) - */ - SKB_DROP_REASON_QDISC_DROP, - /** - * @SKB_DROP_REASON_CPU_BACKLOG: failed to enqueue the skb to the per CPU - * backlog queue. This can be caused by backlog queue full (see - * netdev_max_backlog in net.rst) or RPS flow limit - */ - SKB_DROP_REASON_CPU_BACKLOG, - /** @SKB_DROP_REASON_XDP: dropped by XDP in input path */ - SKB_DROP_REASON_XDP, - /** @SKB_DROP_REASON_TC_INGRESS: dropped in TC ingress HOOK */ - SKB_DROP_REASON_TC_INGRESS, - /** @SKB_DROP_REASON_UNHANDLED_PROTO: protocol not implemented or not supported */ - SKB_DROP_REASON_UNHANDLED_PROTO, - /** @SKB_DROP_REASON_SKB_CSUM: sk_buff checksum computation error */ - SKB_DROP_REASON_SKB_CSUM, - /** @SKB_DROP_REASON_SKB_GSO_SEG: gso segmentation error */ - SKB_DROP_REASON_SKB_GSO_SEG, - /** - * @SKB_DROP_REASON_SKB_UCOPY_FAULT: failed to copy data from user space, - * e.g., via zerocopy_sg_from_iter() or skb_orphan_frags_rx() - */ - SKB_DROP_REASON_SKB_UCOPY_FAULT, - /** @SKB_DROP_REASON_DEV_HDR: device driver specific header/metadata is invalid */ - SKB_DROP_REASON_DEV_HDR, - /** - * @SKB_DROP_REASON_DEV_READY: the device is not ready to xmit/recv due to - * any of its data structure that is not up/ready/initialized, - * e.g., the IFF_UP is not set, or driver specific tun->tfiles[txq] - * is not initialized - */ - SKB_DROP_REASON_DEV_READY, - /** @SKB_DROP_REASON_FULL_RING: ring buffer is full */ - SKB_DROP_REASON_FULL_RING, - /** @SKB_DROP_REASON_NOMEM: error due to OOM */ - SKB_DROP_REASON_NOMEM, - /** - * @SKB_DROP_REASON_HDR_TRUNC: failed to trunc/extract the header from - * networking data, e.g., failed to pull the protocol header from - * frags via pskb_may_pull() - */ - SKB_DROP_REASON_HDR_TRUNC, - /** - * @SKB_DROP_REASON_TAP_FILTER: dropped by (ebpf) filter directly attached - * to tun/tap, e.g., via TUNSETFILTEREBPF - */ - SKB_DROP_REASON_TAP_FILTER, - /** - * @SKB_DROP_REASON_TAP_TXFILTER: dropped by tx filter implemented at - * tun/tap, e.g., check_filter() - */ - SKB_DROP_REASON_TAP_TXFILTER, - /** @SKB_DROP_REASON_ICMP_CSUM: ICMP checksum error */ - SKB_DROP_REASON_ICMP_CSUM, - /** - * @SKB_DROP_REASON_INVALID_PROTO: the packet doesn't follow RFC 2211, - * such as a broadcasts ICMP_TIMESTAMP - */ - SKB_DROP_REASON_INVALID_PROTO, - /** - * @SKB_DROP_REASON_IP_INADDRERRORS: host unreachable, corresponding to - * IPSTATS_MIB_INADDRERRORS - */ - SKB_DROP_REASON_IP_INADDRERRORS, - /** - * @SKB_DROP_REASON_IP_INNOROUTES: network unreachable, corresponding to - * IPSTATS_MIB_INADDRERRORS - */ - SKB_DROP_REASON_IP_INNOROUTES, - /** - * @SKB_DROP_REASON_PKT_TOO_BIG: packet size is too big (maybe exceed the - * MTU) - */ - SKB_DROP_REASON_PKT_TOO_BIG, - /** @SKB_DROP_REASON_DUP_FRAG: duplicate fragment */ - SKB_DROP_REASON_DUP_FRAG, - /** @SKB_DROP_REASON_FRAG_REASM_TIMEOUT: fragment reassembly timeout */ - SKB_DROP_REASON_FRAG_REASM_TIMEOUT, - /** - * @SKB_DROP_REASON_FRAG_TOO_FAR: ipv4 fragment too far. - * (/proc/sys/net/ipv4/ipfrag_max_dist) - */ - SKB_DROP_REASON_FRAG_TOO_FAR, +enum skb_drop_reason_subsys { + /** @SKB_DROP_REASON_SUBSYS_CORE: core drop reasons defined above */ + SKB_DROP_REASON_SUBSYS_CORE, + /** - * @SKB_DROP_REASON_TCP_MINTTL: ipv4 ttl or ipv6 hoplimit below - * the threshold (IP_MINTTL or IPV6_MINHOPCOUNT). - */ - SKB_DROP_REASON_TCP_MINTTL, - /** @SKB_DROP_REASON_IPV6_BAD_EXTHDR: Bad IPv6 extension header. */ - SKB_DROP_REASON_IPV6_BAD_EXTHDR, - /** @SKB_DROP_REASON_IPV6_NDISC_FRAG: invalid frag (suppress_frag_ndisc). */ - SKB_DROP_REASON_IPV6_NDISC_FRAG, - /** @SKB_DROP_REASON_IPV6_NDISC_HOP_LIMIT: invalid hop limit. */ - SKB_DROP_REASON_IPV6_NDISC_HOP_LIMIT, - /** @SKB_DROP_REASON_IPV6_NDISC_BAD_CODE: invalid NDISC icmp6 code. */ - SKB_DROP_REASON_IPV6_NDISC_BAD_CODE, - /** @SKB_DROP_REASON_IPV6_NDISC_BAD_OPTIONS: invalid NDISC options. */ - SKB_DROP_REASON_IPV6_NDISC_BAD_OPTIONS, - /** @SKB_DROP_REASON_IPV6_NDISC_NS_OTHERHOST: NEIGHBOUR SOLICITATION - * for another host. + * @SKB_DROP_REASON_SUBSYS_MAC80211_UNUSABLE: mac80211 drop reasons + * for unusable frames, see net/mac80211/drop.h */ - SKB_DROP_REASON_IPV6_NDISC_NS_OTHERHOST, + SKB_DROP_REASON_SUBSYS_MAC80211_UNUSABLE, + /** - * @SKB_DROP_REASON_MAX: the maximum of drop reason, which shouldn't be - * used as a real 'reason' + * @SKB_DROP_REASON_SUBSYS_MAC80211_MONITOR: mac80211 drop reasons + * for frames still going to monitor, see net/mac80211/drop.h */ - SKB_DROP_REASON_MAX, + SKB_DROP_REASON_SUBSYS_MAC80211_MONITOR, + + /** @SKB_DROP_REASON_SUBSYS_NUM: number of subsystems defined */ + SKB_DROP_REASON_SUBSYS_NUM +}; + +struct drop_reason_list { + const char * const *reasons; + size_t n_reasons; }; -#define SKB_DR_INIT(name, reason) \ - enum skb_drop_reason name = SKB_DROP_REASON_##reason -#define SKB_DR(name) \ - SKB_DR_INIT(name, NOT_SPECIFIED) -#define SKB_DR_SET(name, reason) \ - (name = SKB_DROP_REASON_##reason) -#define SKB_DR_OR(name, reason) \ - do { \ - if (name == SKB_DROP_REASON_NOT_SPECIFIED || \ - name == SKB_NOT_DROPPED_YET) \ - SKB_DR_SET(name, reason); \ - } while (0) +/* Note: due to dynamic registrations, access must be under RCU */ +extern const struct drop_reason_list __rcu * +drop_reasons_by_subsys[SKB_DROP_REASON_SUBSYS_NUM]; -extern const char * const drop_reasons[]; +void drop_reasons_register_subsys(enum skb_drop_reason_subsys subsys, + const struct drop_reason_list *list); +void drop_reasons_unregister_subsys(enum skb_drop_reason_subsys subsys); #endif diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h index 5ccf52ef8809..85b2281576ed 100644 --- a/include/net/flow_dissector.h +++ b/include/net/flow_dissector.h @@ -14,7 +14,9 @@ struct sk_buff; /** * struct flow_dissector_key_control: - * @thoff: Transport header offset + * @thoff: Transport header offset + * @addr_type: Type of key. One of FLOW_DISSECTOR_KEY_* + * @flags: Key flags. Any of FLOW_DIS_(IS_FRAGMENT|FIRST_FRAGENCAPSULATION) */ struct flow_dissector_key_control { u16 thoff; @@ -36,8 +38,9 @@ enum flow_dissect_ret { /** * struct flow_dissector_key_basic: - * @n_proto: Network header protocol (eg. IPv4/IPv6) + * @n_proto: Network header protocol (eg. IPv4/IPv6) * @ip_proto: Transport header protocol (eg. TCP/UDP) + * @padding: Unused */ struct flow_dissector_key_basic { __be16 n_proto; @@ -135,6 +138,7 @@ struct flow_dissector_key_tipc { * struct flow_dissector_key_addrs: * @v4addrs: IPv4 addresses * @v6addrs: IPv6 addresses + * @tipckey: TIPC key */ struct flow_dissector_key_addrs { union { @@ -145,14 +149,12 @@ struct flow_dissector_key_addrs { }; /** - * flow_dissector_key_arp: - * @ports: Operation, source and target addresses for an ARP header - * for Ethernet hardware addresses and IPv4 protocol addresses - * sip: Sender IP address - * tip: Target IP address - * op: Operation - * sha: Sender hardware address - * tpa: Target hardware address + * struct flow_dissector_key_arp: + * @sip: Sender IP address + * @tip: Target IP address + * @op: Operation + * @sha: Sender hardware address + * @tha: Target hardware address */ struct flow_dissector_key_arp { __u32 sip; @@ -163,10 +165,10 @@ struct flow_dissector_key_arp { }; /** - * flow_dissector_key_tp_ports: - * @ports: port numbers of Transport header - * src: source port number - * dst: destination port number + * struct flow_dissector_key_ports: + * @ports: port numbers of Transport header + * @src: source port number + * @dst: destination port number */ struct flow_dissector_key_ports { union { @@ -195,10 +197,10 @@ struct flow_dissector_key_ports_range { }; /** - * flow_dissector_key_icmp: - * type: ICMP type - * code: ICMP code - * id: session identifier + * struct flow_dissector_key_icmp: + * @type: ICMP type + * @code: ICMP code + * @id: Session identifier */ struct flow_dissector_key_icmp { struct { diff --git a/include/net/inet_frag.h b/include/net/inet_frag.h index b23ddec3cd5c..325ad893f624 100644 --- a/include/net/inet_frag.h +++ b/include/net/inet_frag.h @@ -7,7 +7,7 @@ #include <linux/in6.h> #include <linux/rbtree_types.h> #include <linux/refcount.h> -#include <net/dropreason.h> +#include <net/dropreason-core.h> /* Per netns frag queues directory */ struct fqdir { diff --git a/include/net/mac80211.h b/include/net/mac80211.h index 679421d37a42..ac0370e76874 100644 --- a/include/net/mac80211.h +++ b/include/net/mac80211.h @@ -3950,6 +3950,10 @@ struct ieee80211_prep_tx_info { * Note that vif can be NULL. * The callback can sleep. * + * @flush_sta: Flush or drop all pending frames from the hardware queue(s) for + * the given station, as it's about to be removed. + * The callback can sleep. + * * @channel_switch: Drivers that need (or want) to offload the channel * switch operation for CSAs received from the AP may implement this * callback. They must then call ieee80211_chswitch_done() to indicate @@ -4415,6 +4419,8 @@ struct ieee80211_ops { #endif void (*flush)(struct ieee80211_hw *hw, struct ieee80211_vif *vif, u32 queues, bool drop); + void (*flush_sta)(struct ieee80211_hw *hw, struct ieee80211_vif *vif, + struct ieee80211_sta *sta); void (*channel_switch)(struct ieee80211_hw *hw, struct ieee80211_vif *vif, struct ieee80211_channel_switch *ch_switch); @@ -5210,26 +5216,6 @@ void ieee80211_tx_status_irqsafe(struct ieee80211_hw *hw, struct sk_buff *skb); /** - * ieee80211_tx_status_8023 - transmit status callback for 802.3 frame format - * - * Call this function for all transmitted data frames after their transmit - * completion. This callback should only be called for data frames which - * are using driver's (or hardware's) offload capability of encap/decap - * 802.11 frames. - * - * This function may not be called in IRQ context. Calls to this function - * for a single hardware must be synchronized against each other and all - * calls in the same tx status family. - * - * @hw: the hardware the frame was transmitted by - * @vif: the interface for which the frame was transmitted - * @skb: the frame that was transmitted, owned by mac80211 after this call - */ -void ieee80211_tx_status_8023(struct ieee80211_hw *hw, - struct ieee80211_vif *vif, - struct sk_buff *skb); - -/** * ieee80211_report_low_ack - report non-responding station * * When operating in AP-mode, call this function to report a non-responding diff --git a/include/net/netfilter/nf_bpf_link.h b/include/net/netfilter/nf_bpf_link.h new file mode 100644 index 000000000000..6c984b0ea838 --- /dev/null +++ b/include/net/netfilter/nf_bpf_link.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +struct bpf_nf_ctx { + const struct nf_hook_state *state; + struct sk_buff *skb; +}; + +#if IS_ENABLED(CONFIG_NETFILTER_BPF_LINK) +int bpf_nf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog); +#else +static inline int bpf_nf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) +{ + return -EOPNOTSUPP; +} +#endif diff --git a/include/net/netfilter/nf_tables.h b/include/net/netfilter/nf_tables.h index 9430128aae99..1b8e305bb54a 100644 --- a/include/net/netfilter/nf_tables.h +++ b/include/net/netfilter/nf_tables.h @@ -1085,6 +1085,10 @@ struct nft_chain { }; int nft_chain_validate(const struct nft_ctx *ctx, const struct nft_chain *chain); +int nft_setelem_validate(const struct nft_ctx *ctx, struct nft_set *set, + const struct nft_set_iter *iter, + struct nft_set_elem *elem); +int nft_set_catchall_validate(const struct nft_ctx *ctx, struct nft_set *set); enum nft_chain_types { NFT_CHAIN_T_DEFAULT = 0, diff --git a/include/net/netns/ipv6.h b/include/net/netns/ipv6.h index b4af4837d80b..3cceb3e9320b 100644 --- a/include/net/netns/ipv6.h +++ b/include/net/netns/ipv6.h @@ -55,6 +55,7 @@ struct netns_sysctl_ipv6 { u64 ioam6_id_wide; bool skip_notify_on_dev_down; u8 fib_notify_on_flag_change; + u8 icmpv6_error_anycast_as_unicast; }; struct netns_ipv6 { diff --git a/include/net/page_pool.h b/include/net/page_pool.h index 91b808dade82..c8ec2f34722b 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool.h @@ -247,6 +247,7 @@ struct page_pool *page_pool_create(const struct page_pool_params *params); struct xdp_mem_info; #ifdef CONFIG_PAGE_POOL +void page_pool_unlink_napi(struct page_pool *pool); void page_pool_destroy(struct page_pool *pool); void page_pool_use_xdp_mem(struct page_pool *pool, void (*disconnect)(void *), struct xdp_mem_info *mem); @@ -254,6 +255,10 @@ void page_pool_release_page(struct page_pool *pool, struct page *page); void page_pool_put_page_bulk(struct page_pool *pool, void **data, int count); #else +static inline void page_pool_unlink_napi(struct page_pool *pool) +{ +} + static inline void page_pool_destroy(struct page_pool *pool) { } diff --git a/include/net/sctp/sctp.h b/include/net/sctp/sctp.h index c335dd01a597..2a67100b2a17 100644 --- a/include/net/sctp/sctp.h +++ b/include/net/sctp/sctp.h @@ -425,11 +425,11 @@ static inline bool sctp_chunk_pending(const struct sctp_chunk *chunk) * the chunk length to indicate when to stop. Make sure * there is room for a param header too. */ -#define sctp_walk_params(pos, chunk, member)\ -_sctp_walk_params((pos), (chunk), ntohs((chunk)->chunk_hdr.length), member) +#define sctp_walk_params(pos, chunk)\ +_sctp_walk_params((pos), (chunk), ntohs((chunk)->chunk_hdr.length)) -#define _sctp_walk_params(pos, chunk, end, member)\ -for (pos.v = chunk->member;\ +#define _sctp_walk_params(pos, chunk, end)\ +for (pos.v = (u8 *)(chunk + 1);\ (pos.v + offsetof(struct sctp_paramhdr, length) + sizeof(pos.p->length) <=\ (void *)chunk + end) &&\ pos.v <= (void *)chunk + end - ntohs(pos.p->length) &&\ @@ -452,8 +452,8 @@ for (err = (struct sctp_errhdr *)((void *)chunk_hdr + \ _sctp_walk_fwdtsn((pos), (chunk), ntohs((chunk)->chunk_hdr->length) - sizeof(struct sctp_fwdtsn_chunk)) #define _sctp_walk_fwdtsn(pos, chunk, end)\ -for (pos = chunk->subh.fwdtsn_hdr->skip;\ - (void *)pos <= (void *)chunk->subh.fwdtsn_hdr->skip + end - sizeof(struct sctp_fwdtsn_skip);\ +for (pos = (void *)(chunk->subh.fwdtsn_hdr + 1);\ + (void *)pos <= (void *)(chunk->subh.fwdtsn_hdr + 1) + end - sizeof(struct sctp_fwdtsn_skip);\ pos++) /* External references. */ diff --git a/include/net/sctp/structs.h b/include/net/sctp/structs.h index 070c9458fff4..5c72d1864dd6 100644 --- a/include/net/sctp/structs.h +++ b/include/net/sctp/structs.h @@ -332,7 +332,7 @@ struct sctp_cookie { * the association TCB is re-constructed from the cookie. */ __u32 raw_addr_list_len; - struct sctp_init_chunk peer_init[]; + /* struct sctp_init_chunk peer_init[]; */ }; diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 3823100b7934..1bb11a6ee667 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -986,6 +986,7 @@ enum bpf_prog_type { BPF_PROG_TYPE_LSM, BPF_PROG_TYPE_SK_LOOKUP, BPF_PROG_TYPE_SYSCALL, /* a program that can execute syscalls */ + BPF_PROG_TYPE_NETFILTER, }; enum bpf_attach_type { @@ -1050,6 +1051,7 @@ enum bpf_link_type { BPF_LINK_TYPE_PERF_EVENT = 7, BPF_LINK_TYPE_KPROBE_MULTI = 8, BPF_LINK_TYPE_STRUCT_OPS = 9, + BPF_LINK_TYPE_NETFILTER = 10, MAX_BPF_LINK_TYPE, }; @@ -1560,6 +1562,12 @@ union bpf_attr { */ __u64 cookie; } tracing; + struct { + __u32 pf; + __u32 hooknum; + __s32 priority; + __u32 flags; + } netfilter; }; } link_create; @@ -6410,6 +6418,12 @@ struct bpf_link_info { struct { __u32 map_id; } struct_ops; + struct { + __u32 pf; + __u32 hooknum; + __s32 priority; + __u32 flags; + } netfilter; }; } __attribute__((aligned(8))); @@ -6985,6 +6999,10 @@ struct bpf_rb_node { __u64 :64; } __attribute__((aligned(8))); +struct bpf_refcount { + __u32 :32; +} __attribute__((aligned(4))); + struct bpf_sysctl { __u32 write; /* Sysctl is being read (= 0) or written (= 1). * Allows 1,2,4-byte read, but no write. diff --git a/include/uapi/linux/if_bridge.h b/include/uapi/linux/if_bridge.h index c9d624f528c5..f95326fce6bb 100644 --- a/include/uapi/linux/if_bridge.h +++ b/include/uapi/linux/if_bridge.h @@ -525,6 +525,7 @@ enum { BRIDGE_VLANDB_ENTRY_MCAST_ROUTER, BRIDGE_VLANDB_ENTRY_MCAST_N_GROUPS, BRIDGE_VLANDB_ENTRY_MCAST_MAX_GROUPS, + BRIDGE_VLANDB_ENTRY_NEIGH_SUPPRESS, __BRIDGE_VLANDB_ENTRY_MAX, }; #define BRIDGE_VLANDB_ENTRY_MAX (__BRIDGE_VLANDB_ENTRY_MAX - 1) diff --git a/include/uapi/linux/if_link.h b/include/uapi/linux/if_link.h index 8d679688efe0..4ac1000b0ef2 100644 --- a/include/uapi/linux/if_link.h +++ b/include/uapi/linux/if_link.h @@ -569,6 +569,7 @@ enum { IFLA_BRPORT_MAB, IFLA_BRPORT_MCAST_N_GROUPS, IFLA_BRPORT_MCAST_MAX_GROUPS, + IFLA_BRPORT_NEIGH_VLAN_SUPPRESS, __IFLA_BRPORT_MAX }; #define IFLA_BRPORT_MAX (__IFLA_BRPORT_MAX - 1) diff --git a/include/uapi/linux/if_packet.h b/include/uapi/linux/if_packet.h index 78c981d6a9d4..9efc42382fdb 100644 --- a/include/uapi/linux/if_packet.h +++ b/include/uapi/linux/if_packet.h @@ -59,6 +59,7 @@ struct sockaddr_ll { #define PACKET_ROLLOVER_STATS 21 #define PACKET_FANOUT_DATA 22 #define PACKET_IGNORE_OUTGOING 23 +#define PACKET_VNET_HDR_SZ 24 #define PACKET_FANOUT_HASH 0 #define PACKET_FANOUT_LB 1 diff --git a/include/uapi/linux/netfilter/nfnetlink_hook.h b/include/uapi/linux/netfilter/nfnetlink_hook.h index bbcd285b22e1..84a561a74b98 100644 --- a/include/uapi/linux/netfilter/nfnetlink_hook.h +++ b/include/uapi/linux/netfilter/nfnetlink_hook.h @@ -32,8 +32,12 @@ enum nfnl_hook_attributes { /** * enum nfnl_hook_chain_info_attributes - chain description * - * NFNLA_HOOK_INFO_DESC: nft chain and table name (enum nft_table_attributes) (NLA_NESTED) - * NFNLA_HOOK_INFO_TYPE: chain type (enum nfnl_hook_chaintype) (NLA_U32) + * @NFNLA_HOOK_INFO_DESC: nft chain and table name (NLA_NESTED) + * @NFNLA_HOOK_INFO_TYPE: chain type (enum nfnl_hook_chaintype) (NLA_U32) + * + * NFNLA_HOOK_INFO_DESC depends on NFNLA_HOOK_INFO_TYPE value: + * NFNL_HOOK_TYPE_NFTABLES: enum nft_table_attributes + * NFNL_HOOK_TYPE_BPF: enum nfnl_hook_bpf_attributes */ enum nfnl_hook_chain_info_attributes { NFNLA_HOOK_INFO_UNSPEC, @@ -55,10 +59,24 @@ enum nfnl_hook_chain_desc_attributes { /** * enum nfnl_hook_chaintype - chain type * - * @NFNL_HOOK_TYPE_NFTABLES nf_tables base chain + * @NFNL_HOOK_TYPE_NFTABLES: nf_tables base chain + * @NFNL_HOOK_TYPE_BPF: bpf program */ enum nfnl_hook_chaintype { NFNL_HOOK_TYPE_NFTABLES = 0x1, + NFNL_HOOK_TYPE_BPF, +}; + +/** + * enum nfnl_hook_bpf_attributes - bpf prog description + * + * @NFNLA_HOOK_BPF_ID: bpf program id (NLA_U32) + */ +enum nfnl_hook_bpf_attributes { + NFNLA_HOOK_BPF_UNSPEC, + NFNLA_HOOK_BPF_ID, + __NFNLA_HOOK_BPF_MAX, }; +#define NFNLA_HOOK_BPF_MAX (__NFNLA_HOOK_BPF_MAX - 1) #endif /* _NFNL_HOOK_H */ diff --git a/init/initramfs.c b/init/initramfs.c index f6c112e30bd4..e7a01c2ccd1b 100644 --- a/init/initramfs.c +++ b/init/initramfs.c @@ -60,15 +60,8 @@ static void __init error(char *x) message = x; } -static void panic_show_mem(const char *fmt, ...) -{ - va_list args; - - show_mem(0, NULL); - va_start(args, fmt); - panic(fmt, args); - va_end(args); -} +#define panic_show_mem(fmt, ...) \ + ({ show_mem(0, NULL); panic(fmt, ##__VA_ARGS__); }) /* link hash */ diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 2a8b8c304d2a..4a865f0e85d0 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -998,7 +998,7 @@ static void __io_req_complete_post(struct io_kiocb *req) void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) { - if (req->ctx->task_complete && (issue_flags & IO_URING_F_IOWQ)) { + if (req->ctx->task_complete && req->ctx->submitter_task != current) { req->io_task_work.func = io_req_task_complete; io_req_task_work_add(req); } else if (!(issue_flags & IO_URING_F_UNLOCKED) || diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 2c2d1fb9f410..6b682b8e4b50 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -25,6 +25,9 @@ #include <linux/bsearch.h> #include <linux/kobject.h> #include <linux/sysfs.h> + +#include <net/netfilter/nf_bpf_link.h> + #include <net/sock.h> #include "../tools/lib/bpf/relo_core.h" @@ -212,6 +215,7 @@ enum btf_kfunc_hook { BTF_KFUNC_HOOK_SK_SKB, BTF_KFUNC_HOOK_SOCKET_FILTER, BTF_KFUNC_HOOK_LWT, + BTF_KFUNC_HOOK_NETFILTER, BTF_KFUNC_HOOK_MAX, }; @@ -577,8 +581,8 @@ static s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p) *btf_p = btf; return ret; } - spin_lock_bh(&btf_idr_lock); btf_put(btf); + spin_lock_bh(&btf_idr_lock); } spin_unlock_bh(&btf_idr_lock); return ret; @@ -1666,10 +1670,8 @@ static void btf_struct_metas_free(struct btf_struct_metas *tab) if (!tab) return; - for (i = 0; i < tab->cnt; i++) { + for (i = 0; i < tab->cnt; i++) btf_record_free(tab->types[i].record); - kfree(tab->types[i].field_offs); - } kfree(tab); } @@ -3393,6 +3395,7 @@ static int btf_get_field_type(const char *name, u32 field_mask, u32 *seen_mask, field_mask_test_name(BPF_LIST_NODE, "bpf_list_node"); field_mask_test_name(BPF_RB_ROOT, "bpf_rb_root"); field_mask_test_name(BPF_RB_NODE, "bpf_rb_node"); + field_mask_test_name(BPF_REFCOUNT, "bpf_refcount"); /* Only return BPF_KPTR when all other types with matchable names fail */ if (field_mask & BPF_KPTR) { @@ -3441,6 +3444,7 @@ static int btf_find_struct_field(const struct btf *btf, case BPF_TIMER: case BPF_LIST_NODE: case BPF_RB_NODE: + case BPF_REFCOUNT: ret = btf_find_struct(btf, member_type, off, sz, field_type, idx < info_cnt ? &info[idx] : &tmp); if (ret < 0) @@ -3506,6 +3510,7 @@ static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t, case BPF_TIMER: case BPF_LIST_NODE: case BPF_RB_NODE: + case BPF_REFCOUNT: ret = btf_find_struct(btf, var_type, off, sz, field_type, idx < info_cnt ? &info[idx] : &tmp); if (ret < 0) @@ -3700,12 +3705,24 @@ static int btf_parse_rb_root(const struct btf *btf, struct btf_field *field, __alignof__(struct bpf_rb_node)); } +static int btf_field_cmp(const void *_a, const void *_b, const void *priv) +{ + const struct btf_field *a = (const struct btf_field *)_a; + const struct btf_field *b = (const struct btf_field *)_b; + + if (a->offset < b->offset) + return -1; + else if (a->offset > b->offset) + return 1; + return 0; +} + struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type *t, u32 field_mask, u32 value_size) { struct btf_field_info info_arr[BTF_FIELDS_MAX]; + u32 next_off = 0, field_type_size; struct btf_record *rec; - u32 next_off = 0; int ret, i, cnt; ret = btf_find_field(btf, t, field_mask, info_arr, ARRAY_SIZE(info_arr)); @@ -3724,8 +3741,10 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type rec->spin_lock_off = -EINVAL; rec->timer_off = -EINVAL; + rec->refcount_off = -EINVAL; for (i = 0; i < cnt; i++) { - if (info_arr[i].off + btf_field_type_size(info_arr[i].type) > value_size) { + field_type_size = btf_field_type_size(info_arr[i].type); + if (info_arr[i].off + field_type_size > value_size) { WARN_ONCE(1, "verifier bug off %d size %d", info_arr[i].off, value_size); ret = -EFAULT; goto end; @@ -3734,11 +3753,12 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type ret = -EEXIST; goto end; } - next_off = info_arr[i].off + btf_field_type_size(info_arr[i].type); + next_off = info_arr[i].off + field_type_size; rec->field_mask |= info_arr[i].type; rec->fields[i].offset = info_arr[i].off; rec->fields[i].type = info_arr[i].type; + rec->fields[i].size = field_type_size; switch (info_arr[i].type) { case BPF_SPIN_LOCK: @@ -3751,6 +3771,11 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type /* Cache offset for faster lookup at runtime */ rec->timer_off = rec->fields[i].offset; break; + case BPF_REFCOUNT: + WARN_ON_ONCE(rec->refcount_off >= 0); + /* Cache offset for faster lookup at runtime */ + rec->refcount_off = rec->fields[i].offset; + break; case BPF_KPTR_UNREF: case BPF_KPTR_REF: ret = btf_parse_kptr(btf, &rec->fields[i], &info_arr[i]); @@ -3784,30 +3809,16 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type goto end; } - /* need collection identity for non-owning refs before allowing this - * - * Consider a node type w/ both list and rb_node fields: - * struct node { - * struct bpf_list_node l; - * struct bpf_rb_node r; - * } - * - * Used like so: - * struct node *n = bpf_obj_new(....); - * bpf_list_push_front(&list_head, &n->l); - * bpf_rbtree_remove(&rb_root, &n->r); - * - * It should not be possible to rbtree_remove the node since it hasn't - * been added to a tree. But push_front converts n to a non-owning - * reference, and rbtree_remove accepts the non-owning reference to - * a type w/ bpf_rb_node field. - */ - if (btf_record_has_field(rec, BPF_LIST_NODE) && + if (rec->refcount_off < 0 && + btf_record_has_field(rec, BPF_LIST_NODE) && btf_record_has_field(rec, BPF_RB_NODE)) { ret = -EINVAL; goto end; } + sort_r(rec->fields, rec->cnt, sizeof(struct btf_field), btf_field_cmp, + NULL, rec); + return rec; end: btf_record_free(rec); @@ -3889,61 +3900,6 @@ int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec) return 0; } -static int btf_field_offs_cmp(const void *_a, const void *_b, const void *priv) -{ - const u32 a = *(const u32 *)_a; - const u32 b = *(const u32 *)_b; - - if (a < b) - return -1; - else if (a > b) - return 1; - return 0; -} - -static void btf_field_offs_swap(void *_a, void *_b, int size, const void *priv) -{ - struct btf_field_offs *foffs = (void *)priv; - u32 *off_base = foffs->field_off; - u32 *a = _a, *b = _b; - u8 *sz_a, *sz_b; - - sz_a = foffs->field_sz + (a - off_base); - sz_b = foffs->field_sz + (b - off_base); - - swap(*a, *b); - swap(*sz_a, *sz_b); -} - -struct btf_field_offs *btf_parse_field_offs(struct btf_record *rec) -{ - struct btf_field_offs *foffs; - u32 i, *off; - u8 *sz; - - BUILD_BUG_ON(ARRAY_SIZE(foffs->field_off) != ARRAY_SIZE(foffs->field_sz)); - if (IS_ERR_OR_NULL(rec)) - return NULL; - - foffs = kzalloc(sizeof(*foffs), GFP_KERNEL | __GFP_NOWARN); - if (!foffs) - return ERR_PTR(-ENOMEM); - - off = foffs->field_off; - sz = foffs->field_sz; - for (i = 0; i < rec->cnt; i++) { - off[i] = rec->fields[i].offset; - sz[i] = btf_field_type_size(rec->fields[i].type); - } - foffs->cnt = rec->cnt; - - if (foffs->cnt == 1) - return foffs; - sort_r(foffs->field_off, foffs->cnt, sizeof(foffs->field_off[0]), - btf_field_offs_cmp, btf_field_offs_swap, foffs); - return foffs; -} - static void __btf_struct_show(const struct btf *btf, const struct btf_type *t, u32 type_id, void *data, u8 bits_offset, struct btf_show *show) @@ -5348,6 +5304,7 @@ static const char *alloc_obj_fields[] = { "bpf_list_node", "bpf_rb_root", "bpf_rb_node", + "bpf_refcount", }; static struct btf_struct_metas * @@ -5386,7 +5343,6 @@ btf_parse_struct_metas(struct bpf_verifier_log *log, struct btf *btf) for (i = 1; i < n; i++) { struct btf_struct_metas *new_tab; const struct btf_member *member; - struct btf_field_offs *foffs; struct btf_struct_meta *type; struct btf_record *record; const struct btf_type *t; @@ -5422,23 +5378,13 @@ btf_parse_struct_metas(struct bpf_verifier_log *log, struct btf *btf) type = &tab->types[tab->cnt]; type->btf_id = i; record = btf_parse_fields(btf, t, BPF_SPIN_LOCK | BPF_LIST_HEAD | BPF_LIST_NODE | - BPF_RB_ROOT | BPF_RB_NODE, t->size); + BPF_RB_ROOT | BPF_RB_NODE | BPF_REFCOUNT, t->size); /* The record cannot be unset, treat it as an error if so */ if (IS_ERR_OR_NULL(record)) { ret = PTR_ERR_OR_ZERO(record) ?: -EFAULT; goto free; } - foffs = btf_parse_field_offs(record); - /* We need the field_offs to be valid for a valid record, - * either both should be set or both should be unset. - */ - if (IS_ERR_OR_NULL(foffs)) { - btf_record_free(record); - ret = -EFAULT; - goto free; - } type->record = record; - type->field_offs = foffs; tab->cnt++; } return tab; @@ -6215,11 +6161,13 @@ again: if (off < moff) goto error; - /* Only allow structure for now, can be relaxed for - * other types later. - */ + /* allow structure and integer */ t = btf_type_skip_modifiers(btf, array_elem->type, NULL); + + if (btf_type_is_int(t)) + return WALK_SCALAR; + if (!btf_type_is_struct(t)) goto error; @@ -7858,6 +7806,8 @@ static int bpf_prog_type_to_kfunc_hook(enum bpf_prog_type prog_type) case BPF_PROG_TYPE_LWT_XMIT: case BPF_PROG_TYPE_LWT_SEG6LOCAL: return BTF_KFUNC_HOOK_LWT; + case BPF_PROG_TYPE_NETFILTER: + return BTF_KFUNC_HOOK_NETFILTER; default: return BTF_KFUNC_HOOK_MAX; } @@ -8410,12 +8360,10 @@ check_modules: btf_get(mod_btf); spin_unlock_bh(&btf_idr_lock); cands = bpf_core_add_cands(cands, mod_btf, btf_nr_types(main_btf)); - if (IS_ERR(cands)) { - btf_put(mod_btf); + btf_put(mod_btf); + if (IS_ERR(cands)) return ERR_CAST(cands); - } spin_lock_bh(&btf_idr_lock); - btf_put(mod_btf); } spin_unlock_bh(&btf_idr_lock); /* cands is a pointer to kmalloced memory here if cands->cnt > 0 diff --git a/kernel/bpf/cgroup.c b/kernel/bpf/cgroup.c index 53edb8ad2471..a06e118a9be5 100644 --- a/kernel/bpf/cgroup.c +++ b/kernel/bpf/cgroup.c @@ -1921,14 +1921,17 @@ int __cgroup_bpf_run_filter_getsockopt(struct sock *sk, int level, if (ret < 0) goto out; - if (ctx.optlen > max_optlen || ctx.optlen < 0) { + if (optval && (ctx.optlen > max_optlen || ctx.optlen < 0)) { ret = -EFAULT; goto out; } if (ctx.optlen != 0) { - if (copy_to_user(optval, ctx.optval, ctx.optlen) || - put_user(ctx.optlen, optlen)) { + if (optval && copy_to_user(optval, ctx.optval, ctx.optlen)) { + ret = -EFAULT; + goto out; + } + if (put_user(ctx.optlen, optlen)) { ret = -EFAULT; goto out; } diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index e2d256c82072..7421487422d4 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -1187,6 +1187,7 @@ int bpf_jit_get_func_addr(const struct bpf_prog *prog, s16 off = insn->off; s32 imm = insn->imm; u8 *addr; + int err; *func_addr_fixed = insn->src_reg != BPF_PSEUDO_CALL; if (!*func_addr_fixed) { @@ -1201,6 +1202,11 @@ int bpf_jit_get_func_addr(const struct bpf_prog *prog, addr = (u8 *)prog->aux->func[off]->bpf_func; else return -EINVAL; + } else if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL && + bpf_jit_supports_far_kfunc_call()) { + err = bpf_get_kfunc_addr(prog, insn->imm, insn->off, &addr); + if (err) + return err; } else { /* Address of a BPF helper call. Since part of the core * kernel, it's always at a fixed location. __bpf_call_base @@ -2732,6 +2738,11 @@ bool __weak bpf_jit_supports_kfunc_call(void) return false; } +bool __weak bpf_jit_supports_far_kfunc_call(void) +{ + return false; +} + /* To execute LD_ABS/LD_IND instructions __bpf_prog_run() may call * skb_copy_bits(), so provide a weak definition of it for NET-less config. */ diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index f04e60a4847f..8d368fa353f9 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -1798,6 +1798,8 @@ bpf_base_func_proto(enum bpf_func_id func_id) } } +void __bpf_obj_drop_impl(void *p, const struct btf_record *rec); + void bpf_list_head_free(const struct btf_field *field, void *list_head, struct bpf_spin_lock *spin_lock) { @@ -1828,13 +1830,8 @@ unlock: /* The contained type can also have resources, including a * bpf_list_head which needs to be freed. */ - bpf_obj_free_fields(field->graph_root.value_rec, obj); - /* bpf_mem_free requires migrate_disable(), since we can be - * called from map free path as well apart from BPF program (as - * part of map ops doing bpf_obj_free_fields). - */ migrate_disable(); - bpf_mem_free(&bpf_global_ma, obj); + __bpf_obj_drop_impl(obj, field->graph_root.value_rec); migrate_enable(); } } @@ -1871,10 +1868,9 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, obj = pos; obj -= field->graph_root.node_offset; - bpf_obj_free_fields(field->graph_root.value_rec, obj); migrate_disable(); - bpf_mem_free(&bpf_global_ma, obj); + __bpf_obj_drop_impl(obj, field->graph_root.value_rec); migrate_enable(); } } @@ -1893,12 +1889,21 @@ __bpf_kfunc void *bpf_obj_new_impl(u64 local_type_id__k, void *meta__ign) if (!p) return NULL; if (meta) - bpf_obj_init(meta->field_offs, p); + bpf_obj_init(meta->record, p); return p; } +/* Must be called under migrate_disable(), as required by bpf_mem_free */ void __bpf_obj_drop_impl(void *p, const struct btf_record *rec) { + if (rec && rec->refcount_off >= 0 && + !refcount_dec_and_test((refcount_t *)(p + rec->refcount_off))) { + /* Object is refcounted and refcount_dec didn't result in 0 + * refcount. Return without freeing the object + */ + return; + } + if (rec) bpf_obj_free_fields(rec, p); bpf_mem_free(&bpf_global_ma, p); @@ -1912,31 +1917,68 @@ __bpf_kfunc void bpf_obj_drop_impl(void *p__alloc, void *meta__ign) __bpf_obj_drop_impl(p, meta ? meta->record : NULL); } -static void __bpf_list_add(struct bpf_list_node *node, struct bpf_list_head *head, bool tail) +__bpf_kfunc void *bpf_refcount_acquire_impl(void *p__refcounted_kptr, void *meta__ign) +{ + struct btf_struct_meta *meta = meta__ign; + struct bpf_refcount *ref; + + /* Could just cast directly to refcount_t *, but need some code using + * bpf_refcount type so that it is emitted in vmlinux BTF + */ + ref = (struct bpf_refcount *)(p__refcounted_kptr + meta->record->refcount_off); + + refcount_inc((refcount_t *)ref); + return (void *)p__refcounted_kptr; +} + +static int __bpf_list_add(struct bpf_list_node *node, struct bpf_list_head *head, + bool tail, struct btf_record *rec, u64 off) { struct list_head *n = (void *)node, *h = (void *)head; + /* If list_head was 0-initialized by map, bpf_obj_init_field wasn't + * called on its fields, so init here + */ if (unlikely(!h->next)) INIT_LIST_HEAD(h); - if (unlikely(!n->next)) - INIT_LIST_HEAD(n); + if (!list_empty(n)) { + /* Only called from BPF prog, no need to migrate_disable */ + __bpf_obj_drop_impl(n - off, rec); + return -EINVAL; + } + tail ? list_add_tail(n, h) : list_add(n, h); + + return 0; } -__bpf_kfunc void bpf_list_push_front(struct bpf_list_head *head, struct bpf_list_node *node) +__bpf_kfunc int bpf_list_push_front_impl(struct bpf_list_head *head, + struct bpf_list_node *node, + void *meta__ign, u64 off) { - return __bpf_list_add(node, head, false); + struct btf_struct_meta *meta = meta__ign; + + return __bpf_list_add(node, head, false, + meta ? meta->record : NULL, off); } -__bpf_kfunc void bpf_list_push_back(struct bpf_list_head *head, struct bpf_list_node *node) +__bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head, + struct bpf_list_node *node, + void *meta__ign, u64 off) { - return __bpf_list_add(node, head, true); + struct btf_struct_meta *meta = meta__ign; + + return __bpf_list_add(node, head, true, + meta ? meta->record : NULL, off); } static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head, bool tail) { struct list_head *n, *h = (void *)head; + /* If list_head was 0-initialized by map, bpf_obj_init_field wasn't + * called on its fields, so init here + */ if (unlikely(!h->next)) INIT_LIST_HEAD(h); if (list_empty(h)) @@ -1962,6 +2004,9 @@ __bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root, struct rb_root_cached *r = (struct rb_root_cached *)root; struct rb_node *n = (struct rb_node *)node; + if (RB_EMPTY_NODE(n)) + return NULL; + rb_erase_cached(n, r); RB_CLEAR_NODE(n); return (struct bpf_rb_node *)n; @@ -1970,14 +2015,20 @@ __bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root, /* Need to copy rbtree_add_cached's logic here because our 'less' is a BPF * program */ -static void __bpf_rbtree_add(struct bpf_rb_root *root, struct bpf_rb_node *node, - void *less) +static int __bpf_rbtree_add(struct bpf_rb_root *root, struct bpf_rb_node *node, + void *less, struct btf_record *rec, u64 off) { struct rb_node **link = &((struct rb_root_cached *)root)->rb_root.rb_node; + struct rb_node *parent = NULL, *n = (struct rb_node *)node; bpf_callback_t cb = (bpf_callback_t)less; - struct rb_node *parent = NULL; bool leftmost = true; + if (!RB_EMPTY_NODE(n)) { + /* Only called from BPF prog, no need to migrate_disable */ + __bpf_obj_drop_impl(n - off, rec); + return -EINVAL; + } + while (*link) { parent = *link; if (cb((uintptr_t)node, (uintptr_t)parent, 0, 0, 0)) { @@ -1988,15 +2039,18 @@ static void __bpf_rbtree_add(struct bpf_rb_root *root, struct bpf_rb_node *node, } } - rb_link_node((struct rb_node *)node, parent, link); - rb_insert_color_cached((struct rb_node *)node, - (struct rb_root_cached *)root, leftmost); + rb_link_node(n, parent, link); + rb_insert_color_cached(n, (struct rb_root_cached *)root, leftmost); + return 0; } -__bpf_kfunc void bpf_rbtree_add(struct bpf_rb_root *root, struct bpf_rb_node *node, - bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b)) +__bpf_kfunc int bpf_rbtree_add_impl(struct bpf_rb_root *root, struct bpf_rb_node *node, + bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b), + void *meta__ign, u64 off) { - __bpf_rbtree_add(root, node, (void *)less); + struct btf_struct_meta *meta = meta__ign; + + return __bpf_rbtree_add(root, node, (void *)less, meta ? meta->record : NULL, off); } __bpf_kfunc struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root) @@ -2271,14 +2325,15 @@ BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE) #endif BTF_ID_FLAGS(func, bpf_obj_new_impl, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_obj_drop_impl, KF_RELEASE) -BTF_ID_FLAGS(func, bpf_list_push_front) -BTF_ID_FLAGS(func, bpf_list_push_back) +BTF_ID_FLAGS(func, bpf_refcount_acquire_impl, KF_ACQUIRE) +BTF_ID_FLAGS(func, bpf_list_push_front_impl) +BTF_ID_FLAGS(func, bpf_list_push_back_impl) BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE) -BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE) -BTF_ID_FLAGS(func, bpf_rbtree_add) +BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_rbtree_add_impl) BTF_ID_FLAGS(func, bpf_rbtree_first, KF_RET_NULL) #ifdef CONFIG_CGROUPS diff --git a/kernel/bpf/map_in_map.c b/kernel/bpf/map_in_map.c index 38136ec4e095..2c5c64c2a53b 100644 --- a/kernel/bpf/map_in_map.c +++ b/kernel/bpf/map_in_map.c @@ -56,18 +56,6 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd) ret = PTR_ERR(inner_map_meta->record); goto free; } - if (inner_map_meta->record) { - struct btf_field_offs *field_offs; - /* If btf_record is !IS_ERR_OR_NULL, then field_offs is always - * valid. - */ - field_offs = kmemdup(inner_map->field_offs, sizeof(*inner_map->field_offs), GFP_KERNEL | __GFP_NOWARN); - if (!field_offs) { - ret = -ENOMEM; - goto free_rec; - } - inner_map_meta->field_offs = field_offs; - } /* Note: We must use the same BTF, as we also used btf_record_dup above * which relies on BTF being same for both maps, as some members like * record->fields.list_head have pointers like value_rec pointing into @@ -88,8 +76,6 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd) fdput(f); return inner_map_meta; -free_rec: - btf_record_free(inner_map_meta->record); free: kfree(inner_map_meta); put: @@ -99,7 +85,6 @@ put: void bpf_map_meta_free(struct bpf_map *map_meta) { - kfree(map_meta->field_offs); bpf_map_free_record(map_meta); btf_put(map_meta->btf); kfree(map_meta); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 6d575505f89c..14f39c1e573e 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -35,6 +35,7 @@ #include <linux/rcupdate_trace.h> #include <linux/memcontrol.h> #include <linux/trace_events.h> +#include <net/netfilter/nf_bpf_link.h> #define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \ (map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \ @@ -552,6 +553,7 @@ void btf_record_free(struct btf_record *rec) case BPF_RB_NODE: case BPF_SPIN_LOCK: case BPF_TIMER: + case BPF_REFCOUNT: /* Nothing to release */ break; default: @@ -599,6 +601,7 @@ struct btf_record *btf_record_dup(const struct btf_record *rec) case BPF_RB_NODE: case BPF_SPIN_LOCK: case BPF_TIMER: + case BPF_REFCOUNT: /* Nothing to acquire */ break; default: @@ -705,6 +708,7 @@ void bpf_obj_free_fields(const struct btf_record *rec, void *obj) break; case BPF_LIST_NODE: case BPF_RB_NODE: + case BPF_REFCOUNT: break; default: WARN_ON_ONCE(1); @@ -717,14 +721,13 @@ void bpf_obj_free_fields(const struct btf_record *rec, void *obj) static void bpf_map_free_deferred(struct work_struct *work) { struct bpf_map *map = container_of(work, struct bpf_map, work); - struct btf_field_offs *foffs = map->field_offs; struct btf_record *rec = map->record; security_bpf_map_free(map); bpf_map_release_memcg(map); /* implementation dependent freeing */ map->ops->map_free(map); - /* Delay freeing of field_offs and btf_record for maps, as map_free + /* Delay freeing of btf_record for maps, as map_free * callback usually needs access to them. It is better to do it here * than require each callback to do the free itself manually. * @@ -733,7 +736,6 @@ static void bpf_map_free_deferred(struct work_struct *work) * eventually calls bpf_map_free_meta, since inner_map_meta is only a * template bpf_map struct used during verification. */ - kfree(foffs); btf_record_free(rec); } @@ -1034,7 +1036,7 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf, map->record = btf_parse_fields(btf, value_type, BPF_SPIN_LOCK | BPF_TIMER | BPF_KPTR | BPF_LIST_HEAD | - BPF_RB_ROOT, + BPF_RB_ROOT | BPF_REFCOUNT, map->value_size); if (!IS_ERR_OR_NULL(map->record)) { int i; @@ -1073,6 +1075,7 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf, break; case BPF_KPTR_UNREF: case BPF_KPTR_REF: + case BPF_REFCOUNT: if (map->map_type != BPF_MAP_TYPE_HASH && map->map_type != BPF_MAP_TYPE_PERCPU_HASH && map->map_type != BPF_MAP_TYPE_LRU_HASH && @@ -1125,7 +1128,6 @@ free_map_tab: static int map_create(union bpf_attr *attr) { int numa_node = bpf_map_attr_numa_node(attr); - struct btf_field_offs *foffs; struct bpf_map *map; int f_flags; int err; @@ -1205,17 +1207,9 @@ static int map_create(union bpf_attr *attr) attr->btf_vmlinux_value_type_id; } - - foffs = btf_parse_field_offs(map->record); - if (IS_ERR(foffs)) { - err = PTR_ERR(foffs); - goto free_map; - } - map->field_offs = foffs; - err = security_bpf_map_alloc(map); if (err) - goto free_map_field_offs; + goto free_map; err = bpf_map_alloc_id(map); if (err) @@ -1239,8 +1233,6 @@ static int map_create(union bpf_attr *attr) free_map_sec: security_bpf_map_free(map); -free_map_field_offs: - kfree(map->field_offs); free_map: btf_put(map->btf); map->ops->map_free(map); @@ -2463,7 +2455,6 @@ static bool is_net_admin_prog_type(enum bpf_prog_type prog_type) case BPF_PROG_TYPE_LWT_SEG6LOCAL: case BPF_PROG_TYPE_SK_SKB: case BPF_PROG_TYPE_SK_MSG: - case BPF_PROG_TYPE_LIRC_MODE2: case BPF_PROG_TYPE_FLOW_DISSECTOR: case BPF_PROG_TYPE_CGROUP_DEVICE: case BPF_PROG_TYPE_CGROUP_SOCK: @@ -2472,6 +2463,7 @@ static bool is_net_admin_prog_type(enum bpf_prog_type prog_type) case BPF_PROG_TYPE_CGROUP_SYSCTL: case BPF_PROG_TYPE_SOCK_OPS: case BPF_PROG_TYPE_EXT: /* extends any prog */ + case BPF_PROG_TYPE_NETFILTER: return true; case BPF_PROG_TYPE_CGROUP_SKB: /* always unpriv */ @@ -4598,6 +4590,7 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr) switch (prog->type) { case BPF_PROG_TYPE_EXT: + case BPF_PROG_TYPE_NETFILTER: break; case BPF_PROG_TYPE_PERF_EVENT: case BPF_PROG_TYPE_TRACEPOINT: @@ -4664,6 +4657,9 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr) case BPF_PROG_TYPE_XDP: ret = bpf_xdp_link_attach(attr, prog); break; + case BPF_PROG_TYPE_NETFILTER: + ret = bpf_nf_link_attach(attr, prog); + break; #endif case BPF_PROG_TYPE_PERF_EVENT: case BPF_PROG_TYPE_TRACEPOINT: diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d6db6de3e9ea..0d73139ee4d8 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -195,6 +195,8 @@ static void invalidate_non_owning_refs(struct bpf_verifier_env *env); static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env); static int ref_set_non_owning(struct bpf_verifier_env *env, struct bpf_reg_state *reg); +static void specialize_kfunc(struct bpf_verifier_env *env, + u32 func_id, u16 offset, unsigned long *addr); static bool bpf_map_ptr_poisoned(const struct bpf_insn_aux_data *aux) { @@ -271,6 +273,11 @@ struct bpf_call_arg_meta { struct btf_field *kptr_field; }; +struct btf_and_id { + struct btf *btf; + u32 btf_id; +}; + struct bpf_kfunc_call_arg_meta { /* In parameters */ struct btf *btf; @@ -289,10 +296,10 @@ struct bpf_kfunc_call_arg_meta { u64 value; bool found; } arg_constant; - struct { - struct btf *btf; - u32 btf_id; - } arg_obj_drop; + union { + struct btf_and_id arg_obj_drop; + struct btf_and_id arg_refcount_acquire; + }; struct { struct btf_field *field; } arg_list_head; @@ -2374,6 +2381,7 @@ struct bpf_kfunc_desc { u32 func_id; s32 imm; u16 offset; + unsigned long addr; }; struct bpf_kfunc_btf { @@ -2383,6 +2391,11 @@ struct bpf_kfunc_btf { }; struct bpf_kfunc_desc_tab { + /* Sorted by func_id (BTF ID) and offset (fd_array offset) during + * verification. JITs do lookups by bpf_insn, where func_id may not be + * available, therefore at the end of verification do_misc_fixups() + * sorts this by imm and offset. + */ struct bpf_kfunc_desc descs[MAX_KFUNC_DESCS]; u32 nr_descs; }; @@ -2423,6 +2436,19 @@ find_kfunc_desc(const struct bpf_prog *prog, u32 func_id, u16 offset) sizeof(tab->descs[0]), kfunc_desc_cmp_by_id_off); } +int bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id, + u16 btf_fd_idx, u8 **func_addr) +{ + const struct bpf_kfunc_desc *desc; + + desc = find_kfunc_desc(prog, func_id, btf_fd_idx); + if (!desc) + return -EFAULT; + + *func_addr = (u8 *)desc->addr; + return 0; +} + static struct btf *__find_kfunc_desc_btf(struct bpf_verifier_env *env, s16 offset) { @@ -2602,13 +2628,18 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset) func_name); return -EINVAL; } + specialize_kfunc(env, func_id, offset, &addr); - call_imm = BPF_CALL_IMM(addr); - /* Check whether or not the relative offset overflows desc->imm */ - if ((unsigned long)(s32)call_imm != call_imm) { - verbose(env, "address of kernel function %s is out of range\n", - func_name); - return -EINVAL; + if (bpf_jit_supports_far_kfunc_call()) { + call_imm = func_id; + } else { + call_imm = BPF_CALL_IMM(addr); + /* Check whether the relative offset overflows desc->imm */ + if ((unsigned long)(s32)call_imm != call_imm) { + verbose(env, "address of kernel function %s is out of range\n", + func_name); + return -EINVAL; + } } if (bpf_dev_bound_kfunc_id(func_id)) { @@ -2621,6 +2652,7 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset) desc->func_id = func_id; desc->imm = call_imm; desc->offset = offset; + desc->addr = addr; err = btf_distill_func_proto(&env->log, desc_btf, func_proto, func_name, &desc->func_model); @@ -2630,19 +2662,19 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset) return err; } -static int kfunc_desc_cmp_by_imm(const void *a, const void *b) +static int kfunc_desc_cmp_by_imm_off(const void *a, const void *b) { const struct bpf_kfunc_desc *d0 = a; const struct bpf_kfunc_desc *d1 = b; - if (d0->imm > d1->imm) - return 1; - else if (d0->imm < d1->imm) - return -1; + if (d0->imm != d1->imm) + return d0->imm < d1->imm ? -1 : 1; + if (d0->offset != d1->offset) + return d0->offset < d1->offset ? -1 : 1; return 0; } -static void sort_kfunc_descs_by_imm(struct bpf_prog *prog) +static void sort_kfunc_descs_by_imm_off(struct bpf_prog *prog) { struct bpf_kfunc_desc_tab *tab; @@ -2651,7 +2683,7 @@ static void sort_kfunc_descs_by_imm(struct bpf_prog *prog) return; sort(tab->descs, tab->nr_descs, sizeof(tab->descs[0]), - kfunc_desc_cmp_by_imm, NULL); + kfunc_desc_cmp_by_imm_off, NULL); } bool bpf_prog_has_kfunc_call(const struct bpf_prog *prog) @@ -2665,13 +2697,14 @@ bpf_jit_find_kfunc_model(const struct bpf_prog *prog, { const struct bpf_kfunc_desc desc = { .imm = insn->imm, + .offset = insn->off, }; const struct bpf_kfunc_desc *res; struct bpf_kfunc_desc_tab *tab; tab = prog->aux->kfunc_tab; res = bsearch(&desc, tab->descs, tab->nr_descs, - sizeof(tab->descs[0]), kfunc_desc_cmp_by_imm); + sizeof(tab->descs[0]), kfunc_desc_cmp_by_imm_off); return res ? &res->func_model : NULL; } @@ -3243,6 +3276,21 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, } } else if (opcode == BPF_EXIT) { return -ENOTSUPP; + } else if (BPF_SRC(insn->code) == BPF_X) { + if (!(*reg_mask & (dreg | sreg))) + return 0; + /* dreg <cond> sreg + * Both dreg and sreg need precision before + * this insn. If only sreg was marked precise + * before it would be equally necessary to + * propagate it to dreg. + */ + *reg_mask |= (sreg | dreg); + /* else dreg <cond> K + * Only dreg still needs precision before + * this insn, so for the K-based conditional + * there is nothing new to be marked. + */ } } else if (class == BPF_LD) { if (!(*reg_mask & dreg)) @@ -8467,10 +8515,10 @@ static int set_rbtree_add_callback_state(struct bpf_verifier_env *env, struct bpf_func_state *callee, int insn_idx) { - /* void bpf_rbtree_add(struct bpf_rb_root *root, struct bpf_rb_node *node, + /* void bpf_rbtree_add_impl(struct bpf_rb_root *root, struct bpf_rb_node *node, * bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b)); * - * 'struct bpf_rb_node *node' arg to bpf_rbtree_add is the same PTR_TO_BTF_ID w/ offset + * 'struct bpf_rb_node *node' arg to bpf_rbtree_add_impl is the same PTR_TO_BTF_ID w/ offset * that 'less' callback args will be receiving. However, 'node' arg was release_reference'd * by this point, so look at 'root' */ @@ -9306,11 +9354,6 @@ static bool is_kfunc_rcu(struct bpf_kfunc_call_arg_meta *meta) return meta->kfunc_flags & KF_RCU; } -static bool is_kfunc_arg_kptr_get(struct bpf_kfunc_call_arg_meta *meta, int arg) -{ - return arg == 0 && (meta->kfunc_flags & KF_KPTR_GET); -} - static bool __kfunc_param_match_suffix(const struct btf *btf, const struct btf_param *arg, const char *suffix) @@ -9375,6 +9418,11 @@ static bool is_kfunc_arg_uninit(const struct btf *btf, const struct btf_param *a return __kfunc_param_match_suffix(btf, arg, "__uninit"); } +static bool is_kfunc_arg_refcounted_kptr(const struct btf *btf, const struct btf_param *arg) +{ + return __kfunc_param_match_suffix(btf, arg, "__refcounted_kptr"); +} + static bool is_kfunc_arg_scalar_with_name(const struct btf *btf, const struct btf_param *arg, const char *name) @@ -9514,15 +9562,15 @@ static u32 *reg2btf_ids[__BPF_REG_TYPE_MAX] = { enum kfunc_ptr_arg_type { KF_ARG_PTR_TO_CTX, - KF_ARG_PTR_TO_ALLOC_BTF_ID, /* Allocated object */ - KF_ARG_PTR_TO_KPTR, /* PTR_TO_KPTR but type specific */ + KF_ARG_PTR_TO_ALLOC_BTF_ID, /* Allocated object */ + KF_ARG_PTR_TO_REFCOUNTED_KPTR, /* Refcounted local kptr */ KF_ARG_PTR_TO_DYNPTR, KF_ARG_PTR_TO_ITER, KF_ARG_PTR_TO_LIST_HEAD, KF_ARG_PTR_TO_LIST_NODE, - KF_ARG_PTR_TO_BTF_ID, /* Also covers reg2btf_ids conversions */ + KF_ARG_PTR_TO_BTF_ID, /* Also covers reg2btf_ids conversions */ KF_ARG_PTR_TO_MEM, - KF_ARG_PTR_TO_MEM_SIZE, /* Size derived from next argument, skip it */ + KF_ARG_PTR_TO_MEM_SIZE, /* Size derived from next argument, skip it */ KF_ARG_PTR_TO_CALLBACK, KF_ARG_PTR_TO_RB_ROOT, KF_ARG_PTR_TO_RB_NODE, @@ -9531,8 +9579,9 @@ enum kfunc_ptr_arg_type { enum special_kfunc_type { KF_bpf_obj_new_impl, KF_bpf_obj_drop_impl, - KF_bpf_list_push_front, - KF_bpf_list_push_back, + KF_bpf_refcount_acquire_impl, + KF_bpf_list_push_front_impl, + KF_bpf_list_push_back_impl, KF_bpf_list_pop_front, KF_bpf_list_pop_back, KF_bpf_cast_to_kern_ctx, @@ -9540,7 +9589,7 @@ enum special_kfunc_type { KF_bpf_rcu_read_lock, KF_bpf_rcu_read_unlock, KF_bpf_rbtree_remove, - KF_bpf_rbtree_add, + KF_bpf_rbtree_add_impl, KF_bpf_rbtree_first, KF_bpf_dynptr_from_skb, KF_bpf_dynptr_from_xdp, @@ -9551,14 +9600,15 @@ enum special_kfunc_type { BTF_SET_START(special_kfunc_set) BTF_ID(func, bpf_obj_new_impl) BTF_ID(func, bpf_obj_drop_impl) -BTF_ID(func, bpf_list_push_front) -BTF_ID(func, bpf_list_push_back) +BTF_ID(func, bpf_refcount_acquire_impl) +BTF_ID(func, bpf_list_push_front_impl) +BTF_ID(func, bpf_list_push_back_impl) BTF_ID(func, bpf_list_pop_front) BTF_ID(func, bpf_list_pop_back) BTF_ID(func, bpf_cast_to_kern_ctx) BTF_ID(func, bpf_rdonly_cast) BTF_ID(func, bpf_rbtree_remove) -BTF_ID(func, bpf_rbtree_add) +BTF_ID(func, bpf_rbtree_add_impl) BTF_ID(func, bpf_rbtree_first) BTF_ID(func, bpf_dynptr_from_skb) BTF_ID(func, bpf_dynptr_from_xdp) @@ -9569,8 +9619,9 @@ BTF_SET_END(special_kfunc_set) BTF_ID_LIST(special_kfunc_list) BTF_ID(func, bpf_obj_new_impl) BTF_ID(func, bpf_obj_drop_impl) -BTF_ID(func, bpf_list_push_front) -BTF_ID(func, bpf_list_push_back) +BTF_ID(func, bpf_refcount_acquire_impl) +BTF_ID(func, bpf_list_push_front_impl) +BTF_ID(func, bpf_list_push_back_impl) BTF_ID(func, bpf_list_pop_front) BTF_ID(func, bpf_list_pop_back) BTF_ID(func, bpf_cast_to_kern_ctx) @@ -9578,7 +9629,7 @@ BTF_ID(func, bpf_rdonly_cast) BTF_ID(func, bpf_rcu_read_lock) BTF_ID(func, bpf_rcu_read_unlock) BTF_ID(func, bpf_rbtree_remove) -BTF_ID(func, bpf_rbtree_add) +BTF_ID(func, bpf_rbtree_add_impl) BTF_ID(func, bpf_rbtree_first) BTF_ID(func, bpf_dynptr_from_skb) BTF_ID(func, bpf_dynptr_from_xdp) @@ -9621,20 +9672,8 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, if (is_kfunc_arg_alloc_obj(meta->btf, &args[argno])) return KF_ARG_PTR_TO_ALLOC_BTF_ID; - if (is_kfunc_arg_kptr_get(meta, argno)) { - if (!btf_type_is_ptr(ref_t)) { - verbose(env, "arg#0 BTF type must be a double pointer for kptr_get kfunc\n"); - return -EINVAL; - } - ref_t = btf_type_by_id(meta->btf, ref_t->type); - ref_tname = btf_name_by_offset(meta->btf, ref_t->name_off); - if (!btf_type_is_struct(ref_t)) { - verbose(env, "kernel function %s args#0 pointer type %s %s is not supported\n", - meta->func_name, btf_type_str(ref_t), ref_tname); - return -EINVAL; - } - return KF_ARG_PTR_TO_KPTR; - } + if (is_kfunc_arg_refcounted_kptr(meta->btf, &args[argno])) + return KF_ARG_PTR_TO_REFCOUNTED_KPTR; if (is_kfunc_arg_dynptr(meta->btf, &args[argno])) return KF_ARG_PTR_TO_DYNPTR; @@ -9749,40 +9788,6 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, return 0; } -static int process_kf_arg_ptr_to_kptr(struct bpf_verifier_env *env, - struct bpf_reg_state *reg, - const struct btf_type *ref_t, - const char *ref_tname, - struct bpf_kfunc_call_arg_meta *meta, - int argno) -{ - struct btf_field *kptr_field; - - /* check_func_arg_reg_off allows var_off for - * PTR_TO_MAP_VALUE, but we need fixed offset to find - * off_desc. - */ - if (!tnum_is_const(reg->var_off)) { - verbose(env, "arg#0 must have constant offset\n"); - return -EINVAL; - } - - kptr_field = btf_record_find(reg->map_ptr->record, reg->off + reg->var_off.value, BPF_KPTR); - if (!kptr_field || kptr_field->type != BPF_KPTR_REF) { - verbose(env, "arg#0 no referenced kptr at map value offset=%llu\n", - reg->off + reg->var_off.value); - return -EINVAL; - } - - if (!btf_struct_ids_match(&env->log, meta->btf, ref_t->type, 0, kptr_field->kptr.btf, - kptr_field->kptr.btf_id, true)) { - verbose(env, "kernel function %s args#%d expected pointer to %s %s\n", - meta->func_name, argno, btf_type_str(ref_t), ref_tname); - return -EINVAL; - } - return 0; -} - static int ref_set_non_owning(struct bpf_verifier_env *env, struct bpf_reg_state *reg) { struct bpf_verifier_state *state = env->cur_state; @@ -9909,27 +9914,28 @@ static int check_reg_allocation_locked(struct bpf_verifier_env *env, struct bpf_ static bool is_bpf_list_api_kfunc(u32 btf_id) { - return btf_id == special_kfunc_list[KF_bpf_list_push_front] || - btf_id == special_kfunc_list[KF_bpf_list_push_back] || + return btf_id == special_kfunc_list[KF_bpf_list_push_front_impl] || + btf_id == special_kfunc_list[KF_bpf_list_push_back_impl] || btf_id == special_kfunc_list[KF_bpf_list_pop_front] || btf_id == special_kfunc_list[KF_bpf_list_pop_back]; } static bool is_bpf_rbtree_api_kfunc(u32 btf_id) { - return btf_id == special_kfunc_list[KF_bpf_rbtree_add] || + return btf_id == special_kfunc_list[KF_bpf_rbtree_add_impl] || btf_id == special_kfunc_list[KF_bpf_rbtree_remove] || btf_id == special_kfunc_list[KF_bpf_rbtree_first]; } static bool is_bpf_graph_api_kfunc(u32 btf_id) { - return is_bpf_list_api_kfunc(btf_id) || is_bpf_rbtree_api_kfunc(btf_id); + return is_bpf_list_api_kfunc(btf_id) || is_bpf_rbtree_api_kfunc(btf_id) || + btf_id == special_kfunc_list[KF_bpf_refcount_acquire_impl]; } static bool is_callback_calling_kfunc(u32 btf_id) { - return btf_id == special_kfunc_list[KF_bpf_rbtree_add]; + return btf_id == special_kfunc_list[KF_bpf_rbtree_add_impl]; } static bool is_rbtree_lock_required_kfunc(u32 btf_id) @@ -9970,12 +9976,12 @@ static bool check_kfunc_is_graph_node_api(struct bpf_verifier_env *env, switch (node_field_type) { case BPF_LIST_NODE: - ret = (kfunc_btf_id == special_kfunc_list[KF_bpf_list_push_front] || - kfunc_btf_id == special_kfunc_list[KF_bpf_list_push_back]); + ret = (kfunc_btf_id == special_kfunc_list[KF_bpf_list_push_front_impl] || + kfunc_btf_id == special_kfunc_list[KF_bpf_list_push_back_impl]); break; case BPF_RB_NODE: ret = (kfunc_btf_id == special_kfunc_list[KF_bpf_rbtree_remove] || - kfunc_btf_id == special_kfunc_list[KF_bpf_rbtree_add]); + kfunc_btf_id == special_kfunc_list[KF_bpf_rbtree_add_impl]); break; default: verbose(env, "verifier internal error: unexpected graph node argument type %s\n", @@ -10143,6 +10149,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ const char *func_name = meta->func_name, *ref_tname; const struct btf *btf = meta->btf; const struct btf_param *args; + struct btf_record *rec; u32 i, nargs; int ret; @@ -10268,7 +10275,6 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ /* Trusted arguments have the same offset checks as release arguments */ arg_type |= OBJ_RELEASE; break; - case KF_ARG_PTR_TO_KPTR: case KF_ARG_PTR_TO_DYNPTR: case KF_ARG_PTR_TO_ITER: case KF_ARG_PTR_TO_LIST_HEAD: @@ -10278,6 +10284,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ case KF_ARG_PTR_TO_MEM: case KF_ARG_PTR_TO_MEM_SIZE: case KF_ARG_PTR_TO_CALLBACK: + case KF_ARG_PTR_TO_REFCOUNTED_KPTR: /* Trusted by default */ break; default: @@ -10320,15 +10327,6 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ meta->arg_obj_drop.btf_id = reg->btf_id; } break; - case KF_ARG_PTR_TO_KPTR: - if (reg->type != PTR_TO_MAP_VALUE) { - verbose(env, "arg#0 expected pointer to map value\n"); - return -EINVAL; - } - ret = process_kf_arg_ptr_to_kptr(env, reg, ref_t, ref_tname, meta, i); - if (ret < 0) - return ret; - break; case KF_ARG_PTR_TO_DYNPTR: { enum bpf_arg_type dynptr_arg_type = ARG_PTR_TO_DYNPTR; @@ -10495,6 +10493,26 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ case KF_ARG_PTR_TO_CALLBACK: meta->subprogno = reg->subprogno; break; + case KF_ARG_PTR_TO_REFCOUNTED_KPTR: + if (!type_is_ptr_alloc_obj(reg->type) && !type_is_non_owning_ref(reg->type)) { + verbose(env, "arg#%d is neither owning or non-owning ref\n", i); + return -EINVAL; + } + + rec = reg_btf_record(reg); + if (!rec) { + verbose(env, "verifier internal error: Couldn't find btf_record\n"); + return -EFAULT; + } + + if (rec->refcount_off < 0) { + verbose(env, "arg#%d doesn't point to a type with bpf_refcount field\n", i); + return -EINVAL; + } + + meta->arg_refcount_acquire.btf = reg->btf; + meta->arg_refcount_acquire.btf_id = reg->btf_id; + break; } } @@ -10634,10 +10652,11 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, } } - if (meta.func_id == special_kfunc_list[KF_bpf_list_push_front] || - meta.func_id == special_kfunc_list[KF_bpf_list_push_back] || - meta.func_id == special_kfunc_list[KF_bpf_rbtree_add]) { + if (meta.func_id == special_kfunc_list[KF_bpf_list_push_front_impl] || + meta.func_id == special_kfunc_list[KF_bpf_list_push_back_impl] || + meta.func_id == special_kfunc_list[KF_bpf_rbtree_add_impl]) { release_ref_obj_id = regs[BPF_REG_2].ref_obj_id; + insn_aux->insert_off = regs[BPF_REG_2].off; err = ref_convert_owning_non_owning(env, release_ref_obj_id); if (err) { verbose(env, "kfunc %s#%d conversion of owning ref to non-owning failed\n", @@ -10653,7 +10672,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, } } - if (meta.func_id == special_kfunc_list[KF_bpf_rbtree_add]) { + if (meta.func_id == special_kfunc_list[KF_bpf_rbtree_add_impl]) { err = __check_func_call(env, insn, insn_idx_p, meta.subprogno, set_rbtree_add_callback_state); if (err) { @@ -10671,7 +10690,9 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, if (is_kfunc_acquire(&meta) && !btf_type_is_struct_ptr(meta.btf, t)) { /* Only exception is bpf_obj_new_impl */ - if (meta.btf != btf_vmlinux || meta.func_id != special_kfunc_list[KF_bpf_obj_new_impl]) { + if (meta.btf != btf_vmlinux || + (meta.func_id != special_kfunc_list[KF_bpf_obj_new_impl] && + meta.func_id != special_kfunc_list[KF_bpf_refcount_acquire_impl])) { verbose(env, "acquire kernel function does not return PTR_TO_BTF_ID\n"); return -EINVAL; } @@ -10719,6 +10740,15 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, insn_aux->obj_new_size = ret_t->size; insn_aux->kptr_struct_meta = btf_find_struct_meta(ret_btf, ret_btf_id); + } else if (meta.func_id == special_kfunc_list[KF_bpf_refcount_acquire_impl]) { + mark_reg_known_zero(env, regs, BPF_REG_0); + regs[BPF_REG_0].type = PTR_TO_BTF_ID | MEM_ALLOC; + regs[BPF_REG_0].btf = meta.arg_refcount_acquire.btf; + regs[BPF_REG_0].btf_id = meta.arg_refcount_acquire.btf_id; + + insn_aux->kptr_struct_meta = + btf_find_struct_meta(meta.arg_refcount_acquire.btf, + meta.arg_refcount_acquire.btf_id); } else if (meta.func_id == special_kfunc_list[KF_bpf_list_pop_front] || meta.func_id == special_kfunc_list[KF_bpf_list_pop_back]) { struct btf_field *field = meta.arg_list_head.field; @@ -10842,9 +10872,6 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, ref_set_non_owning(env, ®s[BPF_REG_0]); } - if (meta.func_id == special_kfunc_list[KF_bpf_rbtree_remove]) - invalidate_non_owning_refs(env); - if (reg_may_point_to_spin_lock(®s[BPF_REG_0]) && !regs[BPF_REG_0].id) regs[BPF_REG_0].id = ++env->id_gen; } else if (btf_type_is_void(t)) { @@ -12409,12 +12436,17 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) insn->src_reg); return -EACCES; } else if (src_reg->type == SCALAR_VALUE) { + bool is_src_reg_u32 = src_reg->umax_value <= U32_MAX; + + if (is_src_reg_u32 && !src_reg->id) + src_reg->id = ++env->id_gen; copy_register_state(dst_reg, src_reg); - /* Make sure ID is cleared otherwise + /* Make sure ID is cleared if src_reg is not in u32 range otherwise * dst_reg min/max could be incorrectly * propagated into src_reg by find_equal_scalars() */ - dst_reg->id = 0; + if (!is_src_reg_u32) + dst_reg->id = 0; dst_reg->live |= REG_LIVE_WRITTEN; dst_reg->subreg_def = env->insn_idx + 1; } else { @@ -13799,6 +13831,9 @@ static int check_return_code(struct bpf_verifier_env *env) } break; + case BPF_PROG_TYPE_NETFILTER: + range = tnum_range(NF_DROP, NF_ACCEPT); + break; case BPF_PROG_TYPE_EXT: /* freplace program can return anything as its return value * depends on the to-be-replaced kernel func or bpf program. @@ -14685,7 +14720,7 @@ static bool regs_exact(const struct bpf_reg_state *rold, const struct bpf_reg_state *rcur, struct bpf_id_pair *idmap) { - return memcmp(rold, rcur, offsetof(struct bpf_reg_state, id)) == 0 && + return memcmp(rold, rcur, offsetof(struct bpf_reg_state, id)) == 0 && check_ids(rold->id, rcur->id, idmap) && check_ids(rold->ref_obj_id, rcur->ref_obj_id, idmap); } @@ -17293,11 +17328,62 @@ static int fixup_call_args(struct bpf_verifier_env *env) return err; } +/* replace a generic kfunc with a specialized version if necessary */ +static void specialize_kfunc(struct bpf_verifier_env *env, + u32 func_id, u16 offset, unsigned long *addr) +{ + struct bpf_prog *prog = env->prog; + bool seen_direct_write; + void *xdp_kfunc; + bool is_rdonly; + + if (bpf_dev_bound_kfunc_id(func_id)) { + xdp_kfunc = bpf_dev_bound_resolve_kfunc(prog, func_id); + if (xdp_kfunc) { + *addr = (unsigned long)xdp_kfunc; + return; + } + /* fallback to default kfunc when not supported by netdev */ + } + + if (offset) + return; + + if (func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) { + seen_direct_write = env->seen_direct_write; + is_rdonly = !may_access_direct_pkt_data(env, NULL, BPF_WRITE); + + if (is_rdonly) + *addr = (unsigned long)bpf_dynptr_from_skb_rdonly; + + /* restore env->seen_direct_write to its original value, since + * may_access_direct_pkt_data mutates it + */ + env->seen_direct_write = seen_direct_write; + } +} + +static void __fixup_collection_insert_kfunc(struct bpf_insn_aux_data *insn_aux, + u16 struct_meta_reg, + u16 node_offset_reg, + struct bpf_insn *insn, + struct bpf_insn *insn_buf, + int *cnt) +{ + struct btf_struct_meta *kptr_struct_meta = insn_aux->kptr_struct_meta; + struct bpf_insn addr[2] = { BPF_LD_IMM64(struct_meta_reg, (long)kptr_struct_meta) }; + + insn_buf[0] = addr[0]; + insn_buf[1] = addr[1]; + insn_buf[2] = BPF_MOV64_IMM(node_offset_reg, insn_aux->insert_off); + insn_buf[3] = *insn; + *cnt = 4; +} + static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, struct bpf_insn *insn_buf, int insn_idx, int *cnt) { const struct bpf_kfunc_desc *desc; - void *xdp_kfunc; if (!insn->imm) { verbose(env, "invalid kernel function call not eliminated in verifier pass\n"); @@ -17306,18 +17392,9 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, *cnt = 0; - if (bpf_dev_bound_kfunc_id(insn->imm)) { - xdp_kfunc = bpf_dev_bound_resolve_kfunc(env->prog, insn->imm); - if (xdp_kfunc) { - insn->imm = BPF_CALL_IMM(xdp_kfunc); - return 0; - } - - /* fallback to default kfunc when not supported by netdev */ - } - - /* insn->imm has the btf func_id. Replace it with - * an address (relative to __bpf_call_base). + /* insn->imm has the btf func_id. Replace it with an offset relative to + * __bpf_call_base, unless the JIT needs to call functions that are + * further than 32 bits away (bpf_jit_supports_far_kfunc_call()). */ desc = find_kfunc_desc(env->prog, insn->imm, insn->off); if (!desc) { @@ -17326,7 +17403,8 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, return -EFAULT; } - insn->imm = desc->imm; + if (!bpf_jit_supports_far_kfunc_call()) + insn->imm = BPF_CALL_IMM(desc->addr); if (insn->off) return 0; if (desc->func_id == special_kfunc_list[KF_bpf_obj_new_impl]) { @@ -17339,7 +17417,8 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, insn_buf[2] = addr[1]; insn_buf[3] = *insn; *cnt = 4; - } else if (desc->func_id == special_kfunc_list[KF_bpf_obj_drop_impl]) { + } else if (desc->func_id == special_kfunc_list[KF_bpf_obj_drop_impl] || + desc->func_id == special_kfunc_list[KF_bpf_refcount_acquire_impl]) { struct btf_struct_meta *kptr_struct_meta = env->insn_aux_data[insn_idx].kptr_struct_meta; struct bpf_insn addr[2] = { BPF_LD_IMM64(BPF_REG_2, (long)kptr_struct_meta) }; @@ -17347,21 +17426,24 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, insn_buf[1] = addr[1]; insn_buf[2] = *insn; *cnt = 3; + } else if (desc->func_id == special_kfunc_list[KF_bpf_list_push_back_impl] || + desc->func_id == special_kfunc_list[KF_bpf_list_push_front_impl] || + desc->func_id == special_kfunc_list[KF_bpf_rbtree_add_impl]) { + int struct_meta_reg = BPF_REG_3; + int node_offset_reg = BPF_REG_4; + + /* rbtree_add has extra 'less' arg, so args-to-fixup are in diff regs */ + if (desc->func_id == special_kfunc_list[KF_bpf_rbtree_add_impl]) { + struct_meta_reg = BPF_REG_4; + node_offset_reg = BPF_REG_5; + } + + __fixup_collection_insert_kfunc(&env->insn_aux_data[insn_idx], struct_meta_reg, + node_offset_reg, insn, insn_buf, cnt); } else if (desc->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) { insn_buf[0] = BPF_MOV64_REG(BPF_REG_0, BPF_REG_1); *cnt = 1; - } else if (desc->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) { - bool seen_direct_write = env->seen_direct_write; - bool is_rdonly = !may_access_direct_pkt_data(env, NULL, BPF_WRITE); - - if (is_rdonly) - insn->imm = BPF_CALL_IMM(bpf_dynptr_from_skb_rdonly); - - /* restore env->seen_direct_write to its original value, since - * may_access_direct_pkt_data mutates it - */ - env->seen_direct_write = seen_direct_write; } return 0; } @@ -17891,7 +17973,7 @@ patch_call_imm: } } - sort_kfunc_descs_by_imm(env->prog); + sort_kfunc_descs_by_imm_off(env->prog); return 0; } @@ -18582,6 +18664,10 @@ BTF_ID(func, migrate_enable) #if !defined CONFIG_PREEMPT_RCU && !defined CONFIG_TINY_RCU BTF_ID(func, rcu_read_unlock_strict) #endif +#if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_TRACE_PREEMPT_TOGGLE) +BTF_ID(func, preempt_count_add) +BTF_ID(func, preempt_count_sub) +#endif BTF_SET_END(btf_id_deny) static bool can_be_sleepable(struct bpf_prog *prog) diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c index 636f1c682ac0..505d86b16642 100644 --- a/kernel/cgroup/cpuset.c +++ b/kernel/cgroup/cpuset.c @@ -1513,7 +1513,7 @@ static int update_parent_subparts_cpumask(struct cpuset *cs, int cmd, spin_unlock_irq(&callback_lock); if (adding || deleting) - update_tasks_cpumask(parent, tmp->new_cpus); + update_tasks_cpumask(parent, tmp->addmask); /* * Set or clear CS_SCHED_LOAD_BALANCE when partcmd_update, if necessary. @@ -1770,10 +1770,13 @@ static int update_cpumask(struct cpuset *cs, struct cpuset *trialcs, /* * Use the cpumasks in trialcs for tmpmasks when they are pointers * to allocated cpumasks. + * + * Note that update_parent_subparts_cpumask() uses only addmask & + * delmask, but not new_cpus. */ tmp.addmask = trialcs->subparts_cpus; tmp.delmask = trialcs->effective_cpus; - tmp.new_cpus = trialcs->cpus_allowed; + tmp.new_cpus = NULL; #endif retval = validate_change(cs, trialcs); @@ -1838,6 +1841,11 @@ static int update_cpumask(struct cpuset *cs, struct cpuset *trialcs, } spin_unlock_irq(&callback_lock); +#ifdef CONFIG_CPUMASK_OFFSTACK + /* Now trialcs->cpus_allowed is available */ + tmp.new_cpus = trialcs->cpus_allowed; +#endif + /* effective_cpus will be updated here */ update_cpumasks_hier(cs, &tmp, false); @@ -2445,6 +2453,20 @@ static int fmeter_getrate(struct fmeter *fmp) static struct cpuset *cpuset_attach_old_cs; +/* + * Check to see if a cpuset can accept a new task + * For v1, cpus_allowed and mems_allowed can't be empty. + * For v2, effective_cpus can't be empty. + * Note that in v1, effective_cpus = cpus_allowed. + */ +static int cpuset_can_attach_check(struct cpuset *cs) +{ + if (cpumask_empty(cs->effective_cpus) || + (!is_in_v2_mode() && nodes_empty(cs->mems_allowed))) + return -ENOSPC; + return 0; +} + /* Called by cgroups to determine if a cpuset is usable; cpuset_rwsem held */ static int cpuset_can_attach(struct cgroup_taskset *tset) { @@ -2459,16 +2481,9 @@ static int cpuset_can_attach(struct cgroup_taskset *tset) percpu_down_write(&cpuset_rwsem); - /* allow moving tasks into an empty cpuset if on default hierarchy */ - ret = -ENOSPC; - if (!is_in_v2_mode() && - (cpumask_empty(cs->cpus_allowed) || nodes_empty(cs->mems_allowed))) - goto out_unlock; - - /* - * Task cannot be moved to a cpuset with empty effective cpus. - */ - if (cpumask_empty(cs->effective_cpus)) + /* Check to see if task is allowed in the cpuset */ + ret = cpuset_can_attach_check(cs); + if (ret) goto out_unlock; cgroup_taskset_for_each(task, css, tset) { @@ -2485,7 +2500,6 @@ static int cpuset_can_attach(struct cgroup_taskset *tset) * changes which zero cpus/mems_allowed. */ cs->attach_in_progress++; - ret = 0; out_unlock: percpu_up_write(&cpuset_rwsem); return ret; @@ -2494,25 +2508,47 @@ out_unlock: static void cpuset_cancel_attach(struct cgroup_taskset *tset) { struct cgroup_subsys_state *css; + struct cpuset *cs; cgroup_taskset_first(tset, &css); + cs = css_cs(css); percpu_down_write(&cpuset_rwsem); - css_cs(css)->attach_in_progress--; + cs->attach_in_progress--; + if (!cs->attach_in_progress) + wake_up(&cpuset_attach_wq); percpu_up_write(&cpuset_rwsem); } /* - * Protected by cpuset_rwsem. cpus_attach is used only by cpuset_attach() + * Protected by cpuset_rwsem. cpus_attach is used only by cpuset_attach_task() * but we can't allocate it dynamically there. Define it global and * allocate from cpuset_init(). */ static cpumask_var_t cpus_attach; +static nodemask_t cpuset_attach_nodemask_to; + +static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task) +{ + percpu_rwsem_assert_held(&cpuset_rwsem); + + if (cs != &top_cpuset) + guarantee_online_cpus(task, cpus_attach); + else + cpumask_andnot(cpus_attach, task_cpu_possible_mask(task), + cs->subparts_cpus); + /* + * can_attach beforehand should guarantee that this doesn't + * fail. TODO: have a better way to handle failure here + */ + WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach)); + + cpuset_change_task_nodemask(task, &cpuset_attach_nodemask_to); + cpuset_update_task_spread_flags(cs, task); +} static void cpuset_attach(struct cgroup_taskset *tset) { - /* static buf protected by cpuset_rwsem */ - static nodemask_t cpuset_attach_nodemask_to; struct task_struct *task; struct task_struct *leader; struct cgroup_subsys_state *css; @@ -2543,20 +2579,8 @@ static void cpuset_attach(struct cgroup_taskset *tset) guarantee_online_mems(cs, &cpuset_attach_nodemask_to); - cgroup_taskset_for_each(task, css, tset) { - if (cs != &top_cpuset) - guarantee_online_cpus(task, cpus_attach); - else - cpumask_copy(cpus_attach, task_cpu_possible_mask(task)); - /* - * can_attach beforehand should guarantee that this doesn't - * fail. TODO: have a better way to handle failure here - */ - WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach)); - - cpuset_change_task_nodemask(task, &cpuset_attach_nodemask_to); - cpuset_update_task_spread_flags(cs, task); - } + cgroup_taskset_for_each(task, css, tset) + cpuset_attach_task(cs, task); /* * Change mm for all threadgroup leaders. This is expensive and may @@ -3248,17 +3272,101 @@ static void cpuset_bind(struct cgroup_subsys_state *root_css) } /* + * In case the child is cloned into a cpuset different from its parent, + * additional checks are done to see if the move is allowed. + */ +static int cpuset_can_fork(struct task_struct *task, struct css_set *cset) +{ + struct cpuset *cs = css_cs(cset->subsys[cpuset_cgrp_id]); + bool same_cs; + int ret; + + rcu_read_lock(); + same_cs = (cs == task_cs(current)); + rcu_read_unlock(); + + if (same_cs) + return 0; + + lockdep_assert_held(&cgroup_mutex); + percpu_down_write(&cpuset_rwsem); + + /* Check to see if task is allowed in the cpuset */ + ret = cpuset_can_attach_check(cs); + if (ret) + goto out_unlock; + + ret = task_can_attach(task, cs->effective_cpus); + if (ret) + goto out_unlock; + + ret = security_task_setscheduler(task); + if (ret) + goto out_unlock; + + /* + * Mark attach is in progress. This makes validate_change() fail + * changes which zero cpus/mems_allowed. + */ + cs->attach_in_progress++; +out_unlock: + percpu_up_write(&cpuset_rwsem); + return ret; +} + +static void cpuset_cancel_fork(struct task_struct *task, struct css_set *cset) +{ + struct cpuset *cs = css_cs(cset->subsys[cpuset_cgrp_id]); + bool same_cs; + + rcu_read_lock(); + same_cs = (cs == task_cs(current)); + rcu_read_unlock(); + + if (same_cs) + return; + + percpu_down_write(&cpuset_rwsem); + cs->attach_in_progress--; + if (!cs->attach_in_progress) + wake_up(&cpuset_attach_wq); + percpu_up_write(&cpuset_rwsem); +} + +/* * Make sure the new task conform to the current state of its parent, * which could have been changed by cpuset just after it inherits the * state from the parent and before it sits on the cgroup's task list. */ static void cpuset_fork(struct task_struct *task) { - if (task_css_is_root(task, cpuset_cgrp_id)) + struct cpuset *cs; + bool same_cs; + + rcu_read_lock(); + cs = task_cs(task); + same_cs = (cs == task_cs(current)); + rcu_read_unlock(); + + if (same_cs) { + if (cs == &top_cpuset) + return; + + set_cpus_allowed_ptr(task, current->cpus_ptr); + task->mems_allowed = current->mems_allowed; return; + } + + /* CLONE_INTO_CGROUP */ + percpu_down_write(&cpuset_rwsem); + guarantee_online_mems(cs, &cpuset_attach_nodemask_to); + cpuset_attach_task(cs, task); + + cs->attach_in_progress--; + if (!cs->attach_in_progress) + wake_up(&cpuset_attach_wq); - set_cpus_allowed_ptr(task, current->cpus_ptr); - task->mems_allowed = current->mems_allowed; + percpu_up_write(&cpuset_rwsem); } struct cgroup_subsys cpuset_cgrp_subsys = { @@ -3271,6 +3379,8 @@ struct cgroup_subsys cpuset_cgrp_subsys = { .attach = cpuset_attach, .post_attach = cpuset_post_attach, .bind = cpuset_bind, + .can_fork = cpuset_can_fork, + .cancel_fork = cpuset_cancel_fork, .fork = cpuset_fork, .legacy_cftypes = legacy_files, .dfl_cftypes = dfl_files, diff --git a/kernel/cgroup/legacy_freezer.c b/kernel/cgroup/legacy_freezer.c index 1b6b21851e9d..936473203a6b 100644 --- a/kernel/cgroup/legacy_freezer.c +++ b/kernel/cgroup/legacy_freezer.c @@ -22,6 +22,7 @@ #include <linux/freezer.h> #include <linux/seq_file.h> #include <linux/mutex.h> +#include <linux/cpu.h> /* * A cgroup is freezing if any FREEZING flags are set. FREEZING_SELF is @@ -350,7 +351,7 @@ static void freezer_apply_state(struct freezer *freezer, bool freeze, if (freeze) { if (!(freezer->state & CGROUP_FREEZING)) - static_branch_inc(&freezer_active); + static_branch_inc_cpuslocked(&freezer_active); freezer->state |= state; freeze_cgroup(freezer); } else { @@ -361,7 +362,7 @@ static void freezer_apply_state(struct freezer *freezer, bool freeze, if (!(freezer->state & CGROUP_FREEZING)) { freezer->state &= ~CGROUP_FROZEN; if (was_freezing) - static_branch_dec(&freezer_active); + static_branch_dec_cpuslocked(&freezer_active); unfreeze_cgroup(freezer); } } @@ -379,6 +380,7 @@ static void freezer_change_state(struct freezer *freezer, bool freeze) { struct cgroup_subsys_state *pos; + cpus_read_lock(); /* * Update all its descendants in pre-order traversal. Each * descendant will try to inherit its parent's FREEZING state as @@ -407,6 +409,7 @@ static void freezer_change_state(struct freezer *freezer, bool freeze) } rcu_read_unlock(); mutex_unlock(&freezer_mutex); + cpus_read_unlock(); } static ssize_t freezer_write(struct kernfs_open_file *of, diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 831f1f472bb8..0a2b4967e333 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -457,9 +457,7 @@ static void root_cgroup_cputime(struct cgroup_base_stat *bstat) struct task_cputime *cputime = &bstat->cputime; int i; - cputime->stime = 0; - cputime->utime = 0; - cputime->sum_exec_runtime = 0; + memset(bstat, 0, sizeof(*bstat)); for_each_possible_cpu(i) { struct kernel_cpustat kcpustat; u64 *cpustat = kcpustat.cpustat; diff --git a/kernel/fork.c b/kernel/fork.c index 0c92f224c68c..ea332319dffe 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1174,6 +1174,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, fail_pcpu: while (i > 0) percpu_counter_destroy(&mm->rss_stat[--i]); + destroy_context(mm); fail_nocontext: mm_free_pgd(mm); fail_nopgd: diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 6986ea31c984..5f6587d94c1d 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -10238,6 +10238,16 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *s sds->avg_load = (sds->total_load * SCHED_CAPACITY_SCALE) / sds->total_capacity; + + /* + * If the local group is more loaded than the average system + * load, don't try to pull any tasks. + */ + if (local->avg_load >= sds->avg_load) { + env->imbalance = 0; + return; + } + } /* diff --git a/kernel/sys.c b/kernel/sys.c index 495cd87d9bf4..351de7916302 100644 --- a/kernel/sys.c +++ b/kernel/sys.c @@ -664,6 +664,7 @@ long __sys_setresuid(uid_t ruid, uid_t euid, uid_t suid) struct cred *new; int retval; kuid_t kruid, keuid, ksuid; + bool ruid_new, euid_new, suid_new; kruid = make_kuid(ns, ruid); keuid = make_kuid(ns, euid); @@ -678,25 +679,29 @@ long __sys_setresuid(uid_t ruid, uid_t euid, uid_t suid) if ((suid != (uid_t) -1) && !uid_valid(ksuid)) return -EINVAL; + old = current_cred(); + + /* check for no-op */ + if ((ruid == (uid_t) -1 || uid_eq(kruid, old->uid)) && + (euid == (uid_t) -1 || (uid_eq(keuid, old->euid) && + uid_eq(keuid, old->fsuid))) && + (suid == (uid_t) -1 || uid_eq(ksuid, old->suid))) + return 0; + + ruid_new = ruid != (uid_t) -1 && !uid_eq(kruid, old->uid) && + !uid_eq(kruid, old->euid) && !uid_eq(kruid, old->suid); + euid_new = euid != (uid_t) -1 && !uid_eq(keuid, old->uid) && + !uid_eq(keuid, old->euid) && !uid_eq(keuid, old->suid); + suid_new = suid != (uid_t) -1 && !uid_eq(ksuid, old->uid) && + !uid_eq(ksuid, old->euid) && !uid_eq(ksuid, old->suid); + if ((ruid_new || euid_new || suid_new) && + !ns_capable_setid(old->user_ns, CAP_SETUID)) + return -EPERM; + new = prepare_creds(); if (!new) return -ENOMEM; - old = current_cred(); - - retval = -EPERM; - if (!ns_capable_setid(old->user_ns, CAP_SETUID)) { - if (ruid != (uid_t) -1 && !uid_eq(kruid, old->uid) && - !uid_eq(kruid, old->euid) && !uid_eq(kruid, old->suid)) - goto error; - if (euid != (uid_t) -1 && !uid_eq(keuid, old->uid) && - !uid_eq(keuid, old->euid) && !uid_eq(keuid, old->suid)) - goto error; - if (suid != (uid_t) -1 && !uid_eq(ksuid, old->uid) && - !uid_eq(ksuid, old->euid) && !uid_eq(ksuid, old->suid)) - goto error; - } - if (ruid != (uid_t) -1) { new->uid = kruid; if (!uid_eq(kruid, old->uid)) { @@ -761,6 +766,7 @@ long __sys_setresgid(gid_t rgid, gid_t egid, gid_t sgid) struct cred *new; int retval; kgid_t krgid, kegid, ksgid; + bool rgid_new, egid_new, sgid_new; krgid = make_kgid(ns, rgid); kegid = make_kgid(ns, egid); @@ -773,23 +779,28 @@ long __sys_setresgid(gid_t rgid, gid_t egid, gid_t sgid) if ((sgid != (gid_t) -1) && !gid_valid(ksgid)) return -EINVAL; + old = current_cred(); + + /* check for no-op */ + if ((rgid == (gid_t) -1 || gid_eq(krgid, old->gid)) && + (egid == (gid_t) -1 || (gid_eq(kegid, old->egid) && + gid_eq(kegid, old->fsgid))) && + (sgid == (gid_t) -1 || gid_eq(ksgid, old->sgid))) + return 0; + + rgid_new = rgid != (gid_t) -1 && !gid_eq(krgid, old->gid) && + !gid_eq(krgid, old->egid) && !gid_eq(krgid, old->sgid); + egid_new = egid != (gid_t) -1 && !gid_eq(kegid, old->gid) && + !gid_eq(kegid, old->egid) && !gid_eq(kegid, old->sgid); + sgid_new = sgid != (gid_t) -1 && !gid_eq(ksgid, old->gid) && + !gid_eq(ksgid, old->egid) && !gid_eq(ksgid, old->sgid); + if ((rgid_new || egid_new || sgid_new) && + !ns_capable_setid(old->user_ns, CAP_SETGID)) + return -EPERM; + new = prepare_creds(); if (!new) return -ENOMEM; - old = current_cred(); - - retval = -EPERM; - if (!ns_capable_setid(old->user_ns, CAP_SETGID)) { - if (rgid != (gid_t) -1 && !gid_eq(krgid, old->gid) && - !gid_eq(krgid, old->egid) && !gid_eq(krgid, old->sgid)) - goto error; - if (egid != (gid_t) -1 && !gid_eq(kegid, old->gid) && - !gid_eq(kegid, old->egid) && !gid_eq(kegid, old->sgid)) - goto error; - if (sgid != (gid_t) -1 && !gid_eq(ksgid, old->gid) && - !gid_eq(ksgid, old->egid) && !gid_eq(ksgid, old->sgid)) - goto error; - } if (rgid != (gid_t) -1) new->gid = krgid; diff --git a/lib/maple_tree.c b/lib/maple_tree.c index db60edb55f2f..1281a40d5735 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1303,26 +1303,21 @@ static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp) node = mas->alloc; node->request_count = 0; while (requested) { - max_req = MAPLE_ALLOC_SLOTS; - if (node->node_count) { - unsigned int offset = node->node_count; - - slots = (void **)&node->slot[offset]; - max_req -= offset; - } else { - slots = (void **)&node->slot; - } - + max_req = MAPLE_ALLOC_SLOTS - node->node_count; + slots = (void **)&node->slot[node->node_count]; max_req = min(requested, max_req); count = mt_alloc_bulk(gfp, max_req, slots); if (!count) goto nomem_bulk; + if (node->node_count == 0) { + node->slot[0]->node_count = 0; + node->slot[0]->request_count = 0; + } + node->node_count += count; allocated += count; node = node->slot[0]; - node->node_count = 0; - node->request_count = 0; requested -= count; } mas->alloc->total = allocated; @@ -4970,7 +4965,8 @@ not_found: * Return: True if found in a leaf, false otherwise. * */ -static bool mas_rev_awalk(struct ma_state *mas, unsigned long size) +static bool mas_rev_awalk(struct ma_state *mas, unsigned long size, + unsigned long *gap_min, unsigned long *gap_max) { enum maple_type type = mte_node_type(mas->node); struct maple_node *node = mas_mn(mas); @@ -5035,8 +5031,8 @@ static bool mas_rev_awalk(struct ma_state *mas, unsigned long size) if (unlikely(ma_is_leaf(type))) { mas->offset = offset; - mas->min = min; - mas->max = min + gap - 1; + *gap_min = min; + *gap_max = min + gap - 1; return true; } @@ -5060,10 +5056,10 @@ static inline bool mas_anode_descend(struct ma_state *mas, unsigned long size) { enum maple_type type = mte_node_type(mas->node); unsigned long pivot, min, gap = 0; - unsigned char offset; - unsigned long *gaps; - unsigned long *pivots = ma_pivots(mas_mn(mas), type); - void __rcu **slots = ma_slots(mas_mn(mas), type); + unsigned char offset, data_end; + unsigned long *gaps, *pivots; + void __rcu **slots; + struct maple_node *node; bool found = false; if (ma_is_dense(type)) { @@ -5071,13 +5067,15 @@ static inline bool mas_anode_descend(struct ma_state *mas, unsigned long size) return true; } - gaps = ma_gaps(mte_to_node(mas->node), type); + node = mas_mn(mas); + pivots = ma_pivots(node, type); + slots = ma_slots(node, type); + gaps = ma_gaps(node, type); offset = mas->offset; min = mas_safe_min(mas, pivots, offset); - for (; offset < mt_slots[type]; offset++) { - pivot = mas_safe_pivot(mas, pivots, offset, type); - if (offset && !pivot) - break; + data_end = ma_data_end(node, type, pivots, mas->max); + for (; offset <= data_end; offset++) { + pivot = mas_logical_pivot(mas, pivots, offset, type); /* Not within lower bounds */ if (mas->index > pivot) @@ -5312,6 +5310,9 @@ int mas_empty_area(struct ma_state *mas, unsigned long min, unsigned long *pivots; enum maple_type mt; + if (min >= max) + return -EINVAL; + if (mas_is_start(mas)) mas_start(mas); else if (mas->offset >= 2) @@ -5366,6 +5367,9 @@ int mas_empty_area_rev(struct ma_state *mas, unsigned long min, { struct maple_enode *last = mas->node; + if (min >= max) + return -EINVAL; + if (mas_is_start(mas)) { mas_start(mas); mas->offset = mas_data_end(mas); @@ -5385,7 +5389,7 @@ int mas_empty_area_rev(struct ma_state *mas, unsigned long min, mas->index = min; mas->last = max; - while (!mas_rev_awalk(mas, size)) { + while (!mas_rev_awalk(mas, size, &min, &max)) { if (last == mas->node) { if (!mas_rewind_node(mas)) return -EBUSY; @@ -5400,17 +5404,9 @@ int mas_empty_area_rev(struct ma_state *mas, unsigned long min, if (unlikely(mas->offset == MAPLE_NODE_SLOTS)) return -EBUSY; - /* - * mas_rev_awalk() has set mas->min and mas->max to the gap values. If - * the maximum is outside the window we are searching, then use the last - * location in the search. - * mas->max and mas->min is the range of the gap. - * mas->index and mas->last are currently set to the search range. - */ - /* Trim the upper limit to the max. */ - if (mas->max <= mas->last) - mas->last = mas->max; + if (max <= mas->last) + mas->last = max; mas->index = mas->last - size + 1; return 0; diff --git a/mm/backing-dev.c b/mm/backing-dev.c index a53b9360b72e..30d2d0386fdb 100644 --- a/mm/backing-dev.c +++ b/mm/backing-dev.c @@ -507,6 +507,15 @@ static LIST_HEAD(offline_cgwbs); static void cleanup_offline_cgwbs_workfn(struct work_struct *work); static DECLARE_WORK(cleanup_offline_cgwbs_work, cleanup_offline_cgwbs_workfn); +static void cgwb_free_rcu(struct rcu_head *rcu_head) +{ + struct bdi_writeback *wb = container_of(rcu_head, + struct bdi_writeback, rcu); + + percpu_ref_exit(&wb->refcnt); + kfree(wb); +} + static void cgwb_release_workfn(struct work_struct *work) { struct bdi_writeback *wb = container_of(work, struct bdi_writeback, @@ -529,11 +538,10 @@ static void cgwb_release_workfn(struct work_struct *work) list_del(&wb->offline_node); spin_unlock_irq(&cgwb_lock); - percpu_ref_exit(&wb->refcnt); wb_exit(wb); bdi_put(bdi); WARN_ON_ONCE(!list_empty(&wb->b_attached)); - kfree_rcu(wb, rcu); + call_rcu(&wb->rcu, cgwb_free_rcu); } static void cgwb_release(struct percpu_ref *refcnt) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 032fb0ef9cd1..3fae2d2496ab 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1838,10 +1838,10 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, if (is_swap_pmd(*pmd)) { swp_entry_t entry = pmd_to_swp_entry(*pmd); struct page *page = pfn_swap_entry_to_page(entry); + pmd_t newpmd; VM_BUG_ON(!is_pmd_migration_entry(*pmd)); if (is_writable_migration_entry(entry)) { - pmd_t newpmd; /* * A protection check is difficult so * just be safe and disable write @@ -1855,8 +1855,16 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, newpmd = pmd_swp_mksoft_dirty(newpmd); if (pmd_swp_uffd_wp(*pmd)) newpmd = pmd_swp_mkuffd_wp(newpmd); - set_pmd_at(mm, addr, pmd, newpmd); + } else { + newpmd = *pmd; } + + if (uffd_wp) + newpmd = pmd_swp_mkuffd_wp(newpmd); + else if (uffd_wp_resolve) + newpmd = pmd_swp_clear_uffd_wp(newpmd); + if (!pmd_same(*pmd, newpmd)) + set_pmd_at(mm, addr, pmd, newpmd); goto unlock; } #endif @@ -2657,9 +2665,10 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); is_hzp = is_huge_zero_page(&folio->page); - VM_WARN_ON_ONCE_FOLIO(is_hzp, folio); - if (is_hzp) + if (is_hzp) { + pr_warn_ratelimited("Called split_huge_page for huge zero page\n"); return -EBUSY; + } if (folio_test_writeback(folio)) return -EBUSY; @@ -3251,6 +3260,8 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, pmdswp = swp_entry_to_pmd(entry); if (pmd_soft_dirty(pmdval)) pmdswp = pmd_swp_mksoft_dirty(pmdswp); + if (pmd_uffd_wp(pmdval)) + pmdswp = pmd_swp_mkuffd_wp(pmdswp); set_pmd_at(mm, address, pvmw->pmd, pmdswp); page_remove_rmap(page, vma, true); put_page(page); diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 92e6f56a932d..0ec69b96b497 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -572,6 +572,10 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, result = SCAN_PTE_NON_PRESENT; goto out; } + if (pte_uffd_wp(pteval)) { + result = SCAN_PTE_UFFD_WP; + goto out; + } page = vm_normal_page(vma, address, pteval); if (unlikely(!page) || unlikely(is_zone_device_page(page))) { result = SCAN_PAGE_NULL; diff --git a/mm/kmsan/hooks.c b/mm/kmsan/hooks.c index 3807502766a3..ec0da72e65aa 100644 --- a/mm/kmsan/hooks.c +++ b/mm/kmsan/hooks.c @@ -148,35 +148,74 @@ void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end) * into the virtual memory. If those physical pages already had shadow/origin, * those are ignored. */ -void kmsan_ioremap_page_range(unsigned long start, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int page_shift) +int kmsan_ioremap_page_range(unsigned long start, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int page_shift) { gfp_t gfp_mask = GFP_KERNEL | __GFP_ZERO; struct page *shadow, *origin; unsigned long off = 0; - int nr; + int nr, err = 0, clean = 0, mapped; if (!kmsan_enabled || kmsan_in_runtime()) - return; + return 0; nr = (end - start) / PAGE_SIZE; kmsan_enter_runtime(); - for (int i = 0; i < nr; i++, off += PAGE_SIZE) { + for (int i = 0; i < nr; i++, off += PAGE_SIZE, clean = i) { shadow = alloc_pages(gfp_mask, 1); origin = alloc_pages(gfp_mask, 1); - __vmap_pages_range_noflush( + if (!shadow || !origin) { + err = -ENOMEM; + goto ret; + } + mapped = __vmap_pages_range_noflush( vmalloc_shadow(start + off), vmalloc_shadow(start + off + PAGE_SIZE), prot, &shadow, PAGE_SHIFT); - __vmap_pages_range_noflush( + if (mapped) { + err = mapped; + goto ret; + } + shadow = NULL; + mapped = __vmap_pages_range_noflush( vmalloc_origin(start + off), vmalloc_origin(start + off + PAGE_SIZE), prot, &origin, PAGE_SHIFT); + if (mapped) { + __vunmap_range_noflush( + vmalloc_shadow(start + off), + vmalloc_shadow(start + off + PAGE_SIZE)); + err = mapped; + goto ret; + } + origin = NULL; + } + /* Page mapping loop finished normally, nothing to clean up. */ + clean = 0; + +ret: + if (clean > 0) { + /* + * Something went wrong. Clean up shadow/origin pages allocated + * on the last loop iteration, then delete mappings created + * during the previous iterations. + */ + if (shadow) + __free_pages(shadow, 1); + if (origin) + __free_pages(origin, 1); + __vunmap_range_noflush( + vmalloc_shadow(start), + vmalloc_shadow(start + clean * PAGE_SIZE)); + __vunmap_range_noflush( + vmalloc_origin(start), + vmalloc_origin(start + clean * PAGE_SIZE)); } flush_cache_vmap(vmalloc_shadow(start), vmalloc_shadow(end)); flush_cache_vmap(vmalloc_origin(start), vmalloc_origin(end)); kmsan_leave_runtime(); + return err; } void kmsan_iounmap_page_range(unsigned long start, unsigned long end) diff --git a/mm/kmsan/shadow.c b/mm/kmsan/shadow.c index a787c04e9583..b8bb95eea5e3 100644 --- a/mm/kmsan/shadow.c +++ b/mm/kmsan/shadow.c @@ -216,27 +216,29 @@ void kmsan_free_page(struct page *page, unsigned int order) kmsan_leave_runtime(); } -void kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end, - pgprot_t prot, struct page **pages, - unsigned int page_shift) +int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end, + pgprot_t prot, struct page **pages, + unsigned int page_shift) { unsigned long shadow_start, origin_start, shadow_end, origin_end; struct page **s_pages, **o_pages; - int nr, mapped; + int nr, mapped, err = 0; if (!kmsan_enabled) - return; + return 0; shadow_start = vmalloc_meta((void *)start, KMSAN_META_SHADOW); shadow_end = vmalloc_meta((void *)end, KMSAN_META_SHADOW); if (!shadow_start) - return; + return 0; nr = (end - start) / PAGE_SIZE; s_pages = kcalloc(nr, sizeof(*s_pages), GFP_KERNEL); o_pages = kcalloc(nr, sizeof(*o_pages), GFP_KERNEL); - if (!s_pages || !o_pages) + if (!s_pages || !o_pages) { + err = -ENOMEM; goto ret; + } for (int i = 0; i < nr; i++) { s_pages[i] = shadow_page_for(pages[i]); o_pages[i] = origin_page_for(pages[i]); @@ -249,10 +251,16 @@ void kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end, kmsan_enter_runtime(); mapped = __vmap_pages_range_noflush(shadow_start, shadow_end, prot, s_pages, page_shift); - KMSAN_WARN_ON(mapped); + if (mapped) { + err = mapped; + goto ret; + } mapped = __vmap_pages_range_noflush(origin_start, origin_end, prot, o_pages, page_shift); - KMSAN_WARN_ON(mapped); + if (mapped) { + err = mapped; + goto ret; + } kmsan_leave_runtime(); flush_tlb_kernel_range(shadow_start, shadow_end); flush_tlb_kernel_range(origin_start, origin_end); @@ -262,6 +270,7 @@ void kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end, ret: kfree(s_pages); kfree(o_pages); + return err; } /* Allocate metadata for pages allocated at boot time. */ diff --git a/mm/mempolicy.c b/mm/mempolicy.c index a256a241fd1d..2068b594dc88 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -790,61 +790,50 @@ static int vma_replace_policy(struct vm_area_struct *vma, return err; } -/* Step 2: apply policy to a range and do splits. */ -static int mbind_range(struct mm_struct *mm, unsigned long start, - unsigned long end, struct mempolicy *new_pol) +/* Split or merge the VMA (if required) and apply the new policy */ +static int mbind_range(struct vma_iterator *vmi, struct vm_area_struct *vma, + struct vm_area_struct **prev, unsigned long start, + unsigned long end, struct mempolicy *new_pol) { - VMA_ITERATOR(vmi, mm, start); - struct vm_area_struct *prev; - struct vm_area_struct *vma; - int err = 0; + struct vm_area_struct *merged; + unsigned long vmstart, vmend; pgoff_t pgoff; + int err; - prev = vma_prev(&vmi); - vma = vma_find(&vmi, end); - if (WARN_ON(!vma)) + vmend = min(end, vma->vm_end); + if (start > vma->vm_start) { + *prev = vma; + vmstart = start; + } else { + vmstart = vma->vm_start; + } + + if (mpol_equal(vma_policy(vma), new_pol)) return 0; - if (start > vma->vm_start) - prev = vma; - - do { - unsigned long vmstart = max(start, vma->vm_start); - unsigned long vmend = min(end, vma->vm_end); - - if (mpol_equal(vma_policy(vma), new_pol)) - goto next; - - pgoff = vma->vm_pgoff + - ((vmstart - vma->vm_start) >> PAGE_SHIFT); - prev = vma_merge(&vmi, mm, prev, vmstart, vmend, vma->vm_flags, - vma->anon_vma, vma->vm_file, pgoff, - new_pol, vma->vm_userfaultfd_ctx, - anon_vma_name(vma)); - if (prev) { - vma = prev; - goto replace; - } - if (vma->vm_start != vmstart) { - err = split_vma(&vmi, vma, vmstart, 1); - if (err) - goto out; - } - if (vma->vm_end != vmend) { - err = split_vma(&vmi, vma, vmend, 0); - if (err) - goto out; - } -replace: - err = vma_replace_policy(vma, new_pol); + pgoff = vma->vm_pgoff + ((vmstart - vma->vm_start) >> PAGE_SHIFT); + merged = vma_merge(vmi, vma->vm_mm, *prev, vmstart, vmend, vma->vm_flags, + vma->anon_vma, vma->vm_file, pgoff, new_pol, + vma->vm_userfaultfd_ctx, anon_vma_name(vma)); + if (merged) { + *prev = merged; + return vma_replace_policy(merged, new_pol); + } + + if (vma->vm_start != vmstart) { + err = split_vma(vmi, vma, vmstart, 1); if (err) - goto out; -next: - prev = vma; - } for_each_vma_range(vmi, vma, end); + return err; + } -out: - return err; + if (vma->vm_end != vmend) { + err = split_vma(vmi, vma, vmend, 0); + if (err) + return err; + } + + *prev = vma; + return vma_replace_policy(vma, new_pol); } /* Set the process memory policy */ @@ -1259,6 +1248,8 @@ static long do_mbind(unsigned long start, unsigned long len, nodemask_t *nmask, unsigned long flags) { struct mm_struct *mm = current->mm; + struct vm_area_struct *vma, *prev; + struct vma_iterator vmi; struct mempolicy *new; unsigned long end; int err; @@ -1328,7 +1319,13 @@ static long do_mbind(unsigned long start, unsigned long len, goto up_out; } - err = mbind_range(mm, start, end, new); + vma_iter_init(&vmi, mm, start); + prev = vma_prev(&vmi); + for_each_vma_range(vmi, vma, end) { + err = mbind_range(&vmi, vma, &prev, start, end, new); + if (err) + break; + } if (!err) { int nr_failed = 0; @@ -1489,10 +1486,8 @@ SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned long, le unsigned long, home_node, unsigned long, flags) { struct mm_struct *mm = current->mm; - struct vm_area_struct *vma; + struct vm_area_struct *vma, *prev; struct mempolicy *new, *old; - unsigned long vmstart; - unsigned long vmend; unsigned long end; int err = -ENOENT; VMA_ITERATOR(vmi, mm, start); @@ -1521,6 +1516,7 @@ SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned long, le if (end == start) return 0; mmap_write_lock(mm); + prev = vma_prev(&vmi); for_each_vma_range(vmi, vma, end) { /* * If any vma in the range got policy other than MPOL_BIND @@ -1541,9 +1537,7 @@ SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned long, le } new->home_node = home_node; - vmstart = max(start, vma->vm_start); - vmend = min(end, vma->vm_end); - err = mbind_range(mm, vmstart, vmend, new); + err = mbind_range(&vmi, vma, &prev, start, end, new); mpol_put(new); if (err) break; diff --git a/mm/mmap.c b/mm/mmap.c index ff68a67a2a7c..d5475fbf5729 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1518,7 +1518,8 @@ static inline int accountable_mapping(struct file *file, vm_flags_t vm_flags) */ static unsigned long unmapped_area(struct vm_unmapped_area_info *info) { - unsigned long length, gap; + unsigned long length, gap, low_limit; + struct vm_area_struct *tmp; MA_STATE(mas, ¤t->mm->mm_mt, 0, 0); @@ -1527,12 +1528,29 @@ static unsigned long unmapped_area(struct vm_unmapped_area_info *info) if (length < info->length) return -ENOMEM; - if (mas_empty_area(&mas, info->low_limit, info->high_limit - 1, - length)) + low_limit = info->low_limit; +retry: + if (mas_empty_area(&mas, low_limit, info->high_limit - 1, length)) return -ENOMEM; gap = mas.index; gap += (info->align_offset - gap) & info->align_mask; + tmp = mas_next(&mas, ULONG_MAX); + if (tmp && (tmp->vm_flags & VM_GROWSDOWN)) { /* Avoid prev check if possible */ + if (vm_start_gap(tmp) < gap + length - 1) { + low_limit = tmp->vm_end; + mas_reset(&mas); + goto retry; + } + } else { + tmp = mas_prev(&mas, 0); + if (tmp && vm_end_gap(tmp) > gap) { + low_limit = vm_end_gap(tmp); + mas_reset(&mas); + goto retry; + } + } + return gap; } @@ -1548,7 +1566,8 @@ static unsigned long unmapped_area(struct vm_unmapped_area_info *info) */ static unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info) { - unsigned long length, gap; + unsigned long length, gap, high_limit, gap_end; + struct vm_area_struct *tmp; MA_STATE(mas, ¤t->mm->mm_mt, 0, 0); /* Adjust search length to account for worst case alignment overhead */ @@ -1556,12 +1575,31 @@ static unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info) if (length < info->length) return -ENOMEM; - if (mas_empty_area_rev(&mas, info->low_limit, info->high_limit - 1, + high_limit = info->high_limit; +retry: + if (mas_empty_area_rev(&mas, info->low_limit, high_limit - 1, length)) return -ENOMEM; gap = mas.last + 1 - info->length; gap -= (gap - info->align_offset) & info->align_mask; + gap_end = mas.last; + tmp = mas_next(&mas, ULONG_MAX); + if (tmp && (tmp->vm_flags & VM_GROWSDOWN)) { /* Avoid prev check if possible */ + if (vm_start_gap(tmp) <= gap_end) { + high_limit = vm_start_gap(tmp); + mas_reset(&mas); + goto retry; + } + } else { + tmp = mas_prev(&mas, 0); + if (tmp && vm_end_gap(tmp) > gap) { + high_limit = tmp->vm_start; + mas_reset(&mas); + goto retry; + } + } + return gap; } diff --git a/mm/mprotect.c b/mm/mprotect.c index 13e84d8c0797..36351a00c0e8 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -838,7 +838,7 @@ static int do_mprotect_pkey(unsigned long start, size_t len, } tlb_finish_mmu(&tlb); - if (vma_iter_end(&vmi) < end) + if (!error && vma_iter_end(&vmi) < end) error = -ENOMEM; out: diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 7136c36c5d01..8e39705c7bdc 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6632,7 +6632,21 @@ static void __build_all_zonelists(void *data) int nid; int __maybe_unused cpu; pg_data_t *self = data; + unsigned long flags; + /* + * Explicitly disable this CPU's interrupts before taking seqlock + * to prevent any IRQ handler from calling into the page allocator + * (e.g. GFP_ATOMIC) that could hit zonelist_iter_begin and livelock. + */ + local_irq_save(flags); + /* + * Explicitly disable this CPU's synchronous printk() before taking + * seqlock to prevent any printk() from trying to hold port->lock, for + * tty_insert_flip_string_and_push_buffer() on other CPU might be + * calling kmalloc(GFP_ATOMIC | __GFP_NOWARN) with port->lock held. + */ + printk_deferred_enter(); write_seqlock(&zonelist_update_seq); #ifdef CONFIG_NUMA @@ -6671,6 +6685,8 @@ static void __build_all_zonelists(void *data) } write_sequnlock(&zonelist_update_seq); + printk_deferred_exit(); + local_irq_restore(flags); } static noinline void __init @@ -9450,6 +9466,9 @@ static bool pfn_range_valid_contig(struct zone *z, unsigned long start_pfn, if (PageReserved(page)) return false; + + if (PageHuge(page)) + return false; } return true; } diff --git a/mm/swap.c b/mm/swap.c index 57cb01b042f6..423199ee8478 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -222,7 +222,7 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) if (lruvec) unlock_page_lruvec_irqrestore(lruvec, flags); folios_put(fbatch->folios, folio_batch_count(fbatch)); - folio_batch_init(fbatch); + folio_batch_reinit(fbatch); } static void folio_batch_add_and_move(struct folio_batch *fbatch, diff --git a/mm/vmalloc.c b/mm/vmalloc.c index a50072066221..31ff782d368b 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -313,8 +313,8 @@ int ioremap_page_range(unsigned long addr, unsigned long end, ioremap_max_page_shift); flush_cache_vmap(addr, end); if (!err) - kmsan_ioremap_page_range(addr, end, phys_addr, prot, - ioremap_max_page_shift); + err = kmsan_ioremap_page_range(addr, end, phys_addr, prot, + ioremap_max_page_shift); return err; } @@ -605,7 +605,11 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, int vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { - kmsan_vmap_pages_range_noflush(addr, end, prot, pages, page_shift); + int ret = kmsan_vmap_pages_range_noflush(addr, end, prot, pages, + page_shift); + + if (ret) + return ret; return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift); } diff --git a/net/8021q/vlan_dev.c b/net/8021q/vlan_dev.c index 5920544e93e8..870e4935d6e6 100644 --- a/net/8021q/vlan_dev.c +++ b/net/8021q/vlan_dev.c @@ -26,6 +26,7 @@ #include <linux/ethtool.h> #include <linux/phy.h> #include <net/arp.h> +#include <net/macsec.h> #include "vlan.h" #include "vlanproc.h" @@ -572,6 +573,9 @@ static int vlan_dev_init(struct net_device *dev) NETIF_F_HIGHDMA | NETIF_F_SCTP_CRC | NETIF_F_ALL_FCOE; + if (real_dev->vlan_features & NETIF_F_HW_MACSEC) + dev->hw_features |= NETIF_F_HW_MACSEC; + dev->features |= dev->hw_features | NETIF_F_LLTX; netif_inherit_tso_max(dev, real_dev); if (dev->features & NETIF_F_VLAN_FEATURES) @@ -803,6 +807,241 @@ static int vlan_dev_fill_forward_path(struct net_device_path_ctx *ctx, return 0; } +#if IS_ENABLED(CONFIG_MACSEC) + +static const struct macsec_ops *vlan_get_macsec_ops(const struct macsec_context *ctx) +{ + return vlan_dev_priv(ctx->netdev)->real_dev->macsec_ops; +} + +static int vlan_macsec_offload(int (* const func)(struct macsec_context *), + struct macsec_context *ctx) +{ + if (unlikely(!func)) + return 0; + + return (*func)(ctx); +} + +static int vlan_macsec_dev_open(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_dev_open, ctx); +} + +static int vlan_macsec_dev_stop(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_dev_stop, ctx); +} + +static int vlan_macsec_add_secy(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_add_secy, ctx); +} + +static int vlan_macsec_upd_secy(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_upd_secy, ctx); +} + +static int vlan_macsec_del_secy(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_del_secy, ctx); +} + +static int vlan_macsec_add_rxsc(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_add_rxsc, ctx); +} + +static int vlan_macsec_upd_rxsc(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_upd_rxsc, ctx); +} + +static int vlan_macsec_del_rxsc(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_del_rxsc, ctx); +} + +static int vlan_macsec_add_rxsa(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_add_rxsa, ctx); +} + +static int vlan_macsec_upd_rxsa(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_upd_rxsa, ctx); +} + +static int vlan_macsec_del_rxsa(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_del_rxsa, ctx); +} + +static int vlan_macsec_add_txsa(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_add_txsa, ctx); +} + +static int vlan_macsec_upd_txsa(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_upd_txsa, ctx); +} + +static int vlan_macsec_del_txsa(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_del_txsa, ctx); +} + +static int vlan_macsec_get_dev_stats(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_get_dev_stats, ctx); +} + +static int vlan_macsec_get_tx_sc_stats(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_get_tx_sc_stats, ctx); +} + +static int vlan_macsec_get_tx_sa_stats(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_get_tx_sa_stats, ctx); +} + +static int vlan_macsec_get_rx_sc_stats(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_get_rx_sc_stats, ctx); +} + +static int vlan_macsec_get_rx_sa_stats(struct macsec_context *ctx) +{ + const struct macsec_ops *ops = vlan_get_macsec_ops(ctx); + + if (!ops) + return -EOPNOTSUPP; + + return vlan_macsec_offload(ops->mdo_get_rx_sa_stats, ctx); +} + +static const struct macsec_ops macsec_offload_ops = { + /* Device wide */ + .mdo_dev_open = vlan_macsec_dev_open, + .mdo_dev_stop = vlan_macsec_dev_stop, + /* SecY */ + .mdo_add_secy = vlan_macsec_add_secy, + .mdo_upd_secy = vlan_macsec_upd_secy, + .mdo_del_secy = vlan_macsec_del_secy, + /* Security channels */ + .mdo_add_rxsc = vlan_macsec_add_rxsc, + .mdo_upd_rxsc = vlan_macsec_upd_rxsc, + .mdo_del_rxsc = vlan_macsec_del_rxsc, + /* Security associations */ + .mdo_add_rxsa = vlan_macsec_add_rxsa, + .mdo_upd_rxsa = vlan_macsec_upd_rxsa, + .mdo_del_rxsa = vlan_macsec_del_rxsa, + .mdo_add_txsa = vlan_macsec_add_txsa, + .mdo_upd_txsa = vlan_macsec_upd_txsa, + .mdo_del_txsa = vlan_macsec_del_txsa, + /* Statistics */ + .mdo_get_dev_stats = vlan_macsec_get_dev_stats, + .mdo_get_tx_sc_stats = vlan_macsec_get_tx_sc_stats, + .mdo_get_tx_sa_stats = vlan_macsec_get_tx_sa_stats, + .mdo_get_rx_sc_stats = vlan_macsec_get_rx_sc_stats, + .mdo_get_rx_sa_stats = vlan_macsec_get_rx_sa_stats, +}; + +#endif + static const struct ethtool_ops vlan_ethtool_ops = { .get_link_ksettings = vlan_ethtool_get_link_ksettings, .get_drvinfo = vlan_ethtool_get_drvinfo, @@ -869,6 +1108,9 @@ void vlan_setup(struct net_device *dev) dev->priv_destructor = vlan_dev_free; dev->ethtool_ops = &vlan_ethtool_ops; +#if IS_ENABLED(CONFIG_MACSEC) + dev->macsec_ops = &macsec_offload_ops; +#endif dev->min_mtu = 0; dev->max_mtu = ETH_MAX_MTU; diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c index 0b9bd9b39990..e79e3a415ca9 100644 --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -19,7 +19,9 @@ #include <linux/error-injection.h> #include <linux/smp.h> #include <linux/sock_diag.h> +#include <linux/netfilter.h> #include <net/xdp.h> +#include <net/netfilter/nf_bpf_link.h> #define CREATE_TRACE_POINTS #include <trace/events/bpf_test_run.h> @@ -679,17 +681,6 @@ __bpf_kfunc void bpf_kfunc_call_int_mem_release(int *p) { } -__bpf_kfunc struct prog_test_ref_kfunc * -bpf_kfunc_call_test_kptr_get(struct prog_test_ref_kfunc **pp, int a, int b) -{ - struct prog_test_ref_kfunc *p = READ_ONCE(*pp); - - if (!p) - return NULL; - refcount_inc(&p->cnt); - return p; -} - struct prog_test_pass1 { int x0; struct { @@ -804,7 +795,6 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test_get_rdwr_mem, KF_RET_NULL) BTF_ID_FLAGS(func, bpf_kfunc_call_test_get_rdonly_mem, KF_RET_NULL) BTF_ID_FLAGS(func, bpf_kfunc_call_test_acq_rdonly_mem, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_kfunc_call_int_mem_release, KF_RELEASE) -BTF_ID_FLAGS(func, bpf_kfunc_call_test_kptr_get, KF_ACQUIRE | KF_RET_NULL | KF_KPTR_GET) BTF_ID_FLAGS(func, bpf_kfunc_call_test_pass_ctx) BTF_ID_FLAGS(func, bpf_kfunc_call_test_pass1) BTF_ID_FLAGS(func, bpf_kfunc_call_test_pass2) @@ -1703,6 +1693,162 @@ out: return err; } +static int verify_and_copy_hook_state(struct nf_hook_state *state, + const struct nf_hook_state *user, + struct net_device *dev) +{ + if (user->in || user->out) + return -EINVAL; + + if (user->net || user->sk || user->okfn) + return -EINVAL; + + switch (user->pf) { + case NFPROTO_IPV4: + case NFPROTO_IPV6: + switch (state->hook) { + case NF_INET_PRE_ROUTING: + state->in = dev; + break; + case NF_INET_LOCAL_IN: + state->in = dev; + break; + case NF_INET_FORWARD: + state->in = dev; + state->out = dev; + break; + case NF_INET_LOCAL_OUT: + state->out = dev; + break; + case NF_INET_POST_ROUTING: + state->out = dev; + break; + } + + break; + default: + return -EINVAL; + } + + state->pf = user->pf; + state->hook = user->hook; + + return 0; +} + +static __be16 nfproto_eth(int nfproto) +{ + switch (nfproto) { + case NFPROTO_IPV4: + return htons(ETH_P_IP); + case NFPROTO_IPV6: + break; + } + + return htons(ETH_P_IPV6); +} + +int bpf_prog_test_run_nf(struct bpf_prog *prog, + const union bpf_attr *kattr, + union bpf_attr __user *uattr) +{ + struct net *net = current->nsproxy->net_ns; + struct net_device *dev = net->loopback_dev; + struct nf_hook_state *user_ctx, hook_state = { + .pf = NFPROTO_IPV4, + .hook = NF_INET_LOCAL_OUT, + }; + u32 size = kattr->test.data_size_in; + u32 repeat = kattr->test.repeat; + struct bpf_nf_ctx ctx = { + .state = &hook_state, + }; + struct sk_buff *skb = NULL; + u32 retval, duration; + void *data; + int ret; + + if (kattr->test.flags || kattr->test.cpu || kattr->test.batch_size) + return -EINVAL; + + if (size < sizeof(struct iphdr)) + return -EINVAL; + + data = bpf_test_init(kattr, kattr->test.data_size_in, size, + NET_SKB_PAD + NET_IP_ALIGN, + SKB_DATA_ALIGN(sizeof(struct skb_shared_info))); + if (IS_ERR(data)) + return PTR_ERR(data); + + if (!repeat) + repeat = 1; + + user_ctx = bpf_ctx_init(kattr, sizeof(struct nf_hook_state)); + if (IS_ERR(user_ctx)) { + kfree(data); + return PTR_ERR(user_ctx); + } + + if (user_ctx) { + ret = verify_and_copy_hook_state(&hook_state, user_ctx, dev); + if (ret) + goto out; + } + + skb = slab_build_skb(data); + if (!skb) { + ret = -ENOMEM; + goto out; + } + + data = NULL; /* data released via kfree_skb */ + + skb_reserve(skb, NET_SKB_PAD + NET_IP_ALIGN); + __skb_put(skb, size); + + ret = -EINVAL; + + if (hook_state.hook != NF_INET_LOCAL_OUT) { + if (size < ETH_HLEN + sizeof(struct iphdr)) + goto out; + + skb->protocol = eth_type_trans(skb, dev); + switch (skb->protocol) { + case htons(ETH_P_IP): + if (hook_state.pf == NFPROTO_IPV4) + break; + goto out; + case htons(ETH_P_IPV6): + if (size < ETH_HLEN + sizeof(struct ipv6hdr)) + goto out; + if (hook_state.pf == NFPROTO_IPV6) + break; + goto out; + default: + ret = -EPROTO; + goto out; + } + + skb_reset_network_header(skb); + } else { + skb->protocol = nfproto_eth(hook_state.pf); + } + + ctx.skb = skb; + + ret = bpf_test_run(prog, &ctx, repeat, &retval, &duration, false); + if (ret) + goto out; + + ret = bpf_test_finish(kattr, uattr, NULL, NULL, 0, retval, duration); + +out: + kfree(user_ctx); + kfree_skb(skb); + kfree(data); + return ret; +} + static const struct btf_kfunc_id_set bpf_prog_test_kfunc_set = { .owner = THIS_MODULE, .set = &test_sk_check_kfunc_ids, diff --git a/net/bridge/br_arp_nd_proxy.c b/net/bridge/br_arp_nd_proxy.c index b45c00c01dea..c7869a286df4 100644 --- a/net/bridge/br_arp_nd_proxy.c +++ b/net/bridge/br_arp_nd_proxy.c @@ -30,7 +30,7 @@ void br_recalculate_neigh_suppress_enabled(struct net_bridge *br) bool neigh_suppress = false; list_for_each_entry(p, &br->port_list, list) { - if (p->flags & BR_NEIGH_SUPPRESS) { + if (p->flags & (BR_NEIGH_SUPPRESS | BR_NEIGH_VLAN_SUPPRESS)) { neigh_suppress = true; break; } @@ -158,7 +158,7 @@ void br_do_proxy_suppress_arp(struct sk_buff *skb, struct net_bridge *br, return; if (br_opt_get(br, BROPT_NEIGH_SUPPRESS_ENABLED)) { - if (p && (p->flags & BR_NEIGH_SUPPRESS)) + if (br_is_neigh_suppress_enabled(p, vid)) return; if (parp->ar_op != htons(ARPOP_RREQUEST) && parp->ar_op != htons(ARPOP_RREPLY) && @@ -202,8 +202,8 @@ void br_do_proxy_suppress_arp(struct sk_buff *skb, struct net_bridge *br, bool replied = false; if ((p && (p->flags & BR_PROXYARP)) || - (f->dst && (f->dst->flags & (BR_PROXYARP_WIFI | - BR_NEIGH_SUPPRESS)))) { + (f->dst && (f->dst->flags & BR_PROXYARP_WIFI)) || + br_is_neigh_suppress_enabled(f->dst, vid)) { if (!vid) br_arp_send(br, p, skb->dev, sip, tip, sha, n->ha, sha, 0, 0); @@ -407,7 +407,7 @@ void br_do_suppress_nd(struct sk_buff *skb, struct net_bridge *br, BR_INPUT_SKB_CB(skb)->proxyarp_replied = 0; - if (p && (p->flags & BR_NEIGH_SUPPRESS)) + if (br_is_neigh_suppress_enabled(p, vid)) return; if (msg->icmph.icmp6_type == NDISC_NEIGHBOUR_ADVERTISEMENT && @@ -461,7 +461,7 @@ void br_do_suppress_nd(struct sk_buff *skb, struct net_bridge *br, if (f) { bool replied = false; - if (f->dst && (f->dst->flags & BR_NEIGH_SUPPRESS)) { + if (br_is_neigh_suppress_enabled(f->dst, vid)) { if (vid != 0) br_nd_send(br, p, skb, n, skb->vlan_proto, @@ -483,3 +483,24 @@ void br_do_suppress_nd(struct sk_buff *skb, struct net_bridge *br, } } #endif + +bool br_is_neigh_suppress_enabled(const struct net_bridge_port *p, u16 vid) +{ + if (!p) + return false; + + if (!vid) + return !!(p->flags & BR_NEIGH_SUPPRESS); + + if (p->flags & BR_NEIGH_VLAN_SUPPRESS) { + struct net_bridge_vlan_group *vg = nbp_vlan_group_rcu(p); + struct net_bridge_vlan *v; + + v = br_vlan_find(vg, vid); + if (!v) + return false; + return !!(v->priv_flags & BR_VLFLAG_NEIGH_SUPPRESS_ENABLED); + } else { + return !!(p->flags & BR_NEIGH_SUPPRESS); + } +} diff --git a/net/bridge/br_device.c b/net/bridge/br_device.c index df47c876230e..8eca8a5c80c6 100644 --- a/net/bridge/br_device.c +++ b/net/bridge/br_device.c @@ -80,10 +80,10 @@ netdev_tx_t br_dev_xmit(struct sk_buff *skb, struct net_device *dev) dest = eth_hdr(skb)->h_dest; if (is_broadcast_ether_addr(dest)) { - br_flood(br, skb, BR_PKT_BROADCAST, false, true); + br_flood(br, skb, BR_PKT_BROADCAST, false, true, vid); } else if (is_multicast_ether_addr(dest)) { if (unlikely(netpoll_tx_running(dev))) { - br_flood(br, skb, BR_PKT_MULTICAST, false, true); + br_flood(br, skb, BR_PKT_MULTICAST, false, true, vid); goto out; } if (br_multicast_rcv(&brmctx, &pmctx_null, vlan, skb, vid)) { @@ -96,11 +96,11 @@ netdev_tx_t br_dev_xmit(struct sk_buff *skb, struct net_device *dev) br_multicast_querier_exists(brmctx, eth_hdr(skb), mdst)) br_multicast_flood(mdst, skb, brmctx, false, true); else - br_flood(br, skb, BR_PKT_MULTICAST, false, true); + br_flood(br, skb, BR_PKT_MULTICAST, false, true, vid); } else if ((dst = br_fdb_find_rcu(br, dest, vid)) != NULL) { br_forward(dst->dst, skb, false, true); } else { - br_flood(br, skb, BR_PKT_UNICAST, false, true); + br_flood(br, skb, BR_PKT_UNICAST, false, true, vid); } out: rcu_read_unlock(); diff --git a/net/bridge/br_forward.c b/net/bridge/br_forward.c index 02bb620d3b8d..57744704ff69 100644 --- a/net/bridge/br_forward.c +++ b/net/bridge/br_forward.c @@ -197,7 +197,8 @@ out: /* called under rcu_read_lock */ void br_flood(struct net_bridge *br, struct sk_buff *skb, - enum br_pkt_type pkt_type, bool local_rcv, bool local_orig) + enum br_pkt_type pkt_type, bool local_rcv, bool local_orig, + u16 vid) { struct net_bridge_port *prev = NULL; struct net_bridge_port *p; @@ -224,8 +225,9 @@ void br_flood(struct net_bridge *br, struct sk_buff *skb, /* Do not flood to ports that enable proxy ARP */ if (p->flags & BR_PROXYARP) continue; - if ((p->flags & (BR_PROXYARP_WIFI | BR_NEIGH_SUPPRESS)) && - BR_INPUT_SKB_CB(skb)->proxyarp_replied) + if (BR_INPUT_SKB_CB(skb)->proxyarp_replied && + ((p->flags & BR_PROXYARP_WIFI) || + br_is_neigh_suppress_enabled(p, vid))) continue; prev = maybe_deliver(prev, p, skb, local_orig); diff --git a/net/bridge/br_if.c b/net/bridge/br_if.c index 24f01ff113f0..3f04b40f6056 100644 --- a/net/bridge/br_if.c +++ b/net/bridge/br_if.c @@ -759,7 +759,7 @@ void br_port_flags_change(struct net_bridge_port *p, unsigned long mask) if (mask & BR_AUTO_MASK) nbp_update_port_count(br); - if (mask & BR_NEIGH_SUPPRESS) + if (mask & (BR_NEIGH_SUPPRESS | BR_NEIGH_VLAN_SUPPRESS)) br_recalculate_neigh_suppress_enabled(br); } diff --git a/net/bridge/br_input.c b/net/bridge/br_input.c index 3027e8f6be15..fc17b9fd93e6 100644 --- a/net/bridge/br_input.c +++ b/net/bridge/br_input.c @@ -207,7 +207,7 @@ int br_handle_frame_finish(struct net *net, struct sock *sk, struct sk_buff *skb br_forward(dst->dst, skb, local_rcv, false); } else { if (!mcast_hit) - br_flood(br, skb, pkt_type, local_rcv, false); + br_flood(br, skb, pkt_type, local_rcv, false, vid); else br_multicast_flood(mdst, skb, brmctx, local_rcv, false); } diff --git a/net/bridge/br_netfilter_hooks.c b/net/bridge/br_netfilter_hooks.c index 3e3065bc0465..1a801fab9543 100644 --- a/net/bridge/br_netfilter_hooks.c +++ b/net/bridge/br_netfilter_hooks.c @@ -869,12 +869,17 @@ static unsigned int ip_sabotage_in(void *priv, { struct nf_bridge_info *nf_bridge = nf_bridge_info_get(skb); - if (nf_bridge && !nf_bridge->in_prerouting && - !netif_is_l3_master(skb->dev) && - !netif_is_l3_slave(skb->dev)) { - nf_bridge_info_free(skb); - state->okfn(state->net, state->sk, skb); - return NF_STOLEN; + if (nf_bridge) { + if (nf_bridge->sabotage_in_done) + return NF_ACCEPT; + + if (!nf_bridge->in_prerouting && + !netif_is_l3_master(skb->dev) && + !netif_is_l3_slave(skb->dev)) { + nf_bridge->sabotage_in_done = 1; + state->okfn(state->net, state->sk, skb); + return NF_STOLEN; + } } return NF_ACCEPT; diff --git a/net/bridge/br_netlink.c b/net/bridge/br_netlink.c index fefb1c0e248b..05c5863d2e20 100644 --- a/net/bridge/br_netlink.c +++ b/net/bridge/br_netlink.c @@ -189,6 +189,7 @@ static inline size_t br_port_info_size(void) + nla_total_size(1) /* IFLA_BRPORT_ISOLATED */ + nla_total_size(1) /* IFLA_BRPORT_LOCKED */ + nla_total_size(1) /* IFLA_BRPORT_MAB */ + + nla_total_size(1) /* IFLA_BRPORT_NEIGH_VLAN_SUPPRESS */ + nla_total_size(sizeof(struct ifla_bridge_id)) /* IFLA_BRPORT_ROOT_ID */ + nla_total_size(sizeof(struct ifla_bridge_id)) /* IFLA_BRPORT_BRIDGE_ID */ + nla_total_size(sizeof(u16)) /* IFLA_BRPORT_DESIGNATED_PORT */ @@ -278,7 +279,9 @@ static int br_port_fill_attrs(struct sk_buff *skb, !!(p->flags & BR_MRP_LOST_IN_CONT)) || nla_put_u8(skb, IFLA_BRPORT_ISOLATED, !!(p->flags & BR_ISOLATED)) || nla_put_u8(skb, IFLA_BRPORT_LOCKED, !!(p->flags & BR_PORT_LOCKED)) || - nla_put_u8(skb, IFLA_BRPORT_MAB, !!(p->flags & BR_PORT_MAB))) + nla_put_u8(skb, IFLA_BRPORT_MAB, !!(p->flags & BR_PORT_MAB)) || + nla_put_u8(skb, IFLA_BRPORT_NEIGH_VLAN_SUPPRESS, + !!(p->flags & BR_NEIGH_VLAN_SUPPRESS))) return -EMSGSIZE; timerval = br_timer_value(&p->message_age_timer); @@ -891,6 +894,7 @@ static const struct nla_policy br_port_policy[IFLA_BRPORT_MAX + 1] = { [IFLA_BRPORT_MCAST_EHT_HOSTS_LIMIT] = { .type = NLA_U32 }, [IFLA_BRPORT_MCAST_N_GROUPS] = { .type = NLA_REJECT }, [IFLA_BRPORT_MCAST_MAX_GROUPS] = { .type = NLA_U32 }, + [IFLA_BRPORT_NEIGH_VLAN_SUPPRESS] = NLA_POLICY_MAX(NLA_U8, 1), }; /* Change the state of the port and notify spanning tree */ @@ -957,6 +961,8 @@ static int br_setport(struct net_bridge_port *p, struct nlattr *tb[], br_set_port_flag(p, tb, IFLA_BRPORT_ISOLATED, BR_ISOLATED); br_set_port_flag(p, tb, IFLA_BRPORT_LOCKED, BR_PORT_LOCKED); br_set_port_flag(p, tb, IFLA_BRPORT_MAB, BR_PORT_MAB); + br_set_port_flag(p, tb, IFLA_BRPORT_NEIGH_VLAN_SUPPRESS, + BR_NEIGH_VLAN_SUPPRESS); if ((p->flags & BR_PORT_MAB) && (!(p->flags & BR_PORT_LOCKED) || !(p->flags & BR_LEARNING))) { diff --git a/net/bridge/br_private.h b/net/bridge/br_private.h index 7264fd40f82f..2119729ded2b 100644 --- a/net/bridge/br_private.h +++ b/net/bridge/br_private.h @@ -178,6 +178,7 @@ enum { BR_VLFLAG_ADDED_BY_SWITCHDEV = BIT(1), BR_VLFLAG_MCAST_ENABLED = BIT(2), BR_VLFLAG_GLOBAL_MCAST_ENABLED = BIT(3), + BR_VLFLAG_NEIGH_SUPPRESS_ENABLED = BIT(4), }; /** @@ -849,7 +850,8 @@ void br_forward(const struct net_bridge_port *to, struct sk_buff *skb, bool local_rcv, bool local_orig); int br_forward_finish(struct net *net, struct sock *sk, struct sk_buff *skb); void br_flood(struct net_bridge *br, struct sk_buff *skb, - enum br_pkt_type pkt_type, bool local_rcv, bool local_orig); + enum br_pkt_type pkt_type, bool local_rcv, bool local_orig, + u16 vid); /* return true if both source port and dest port are isolated */ static inline bool br_skb_isolated(const struct net_bridge_port *to, @@ -2218,4 +2220,5 @@ void br_do_proxy_suppress_arp(struct sk_buff *skb, struct net_bridge *br, void br_do_suppress_nd(struct sk_buff *skb, struct net_bridge *br, u16 vid, struct net_bridge_port *p, struct nd_msg *msg); struct nd_msg *br_is_nd_neigh_msg(struct sk_buff *skb, struct nd_msg *m); +bool br_is_neigh_suppress_enabled(const struct net_bridge_port *p, u16 vid); #endif diff --git a/net/bridge/br_switchdev.c b/net/bridge/br_switchdev.c index de18e9c1d7a7..ba95c4d74a60 100644 --- a/net/bridge/br_switchdev.c +++ b/net/bridge/br_switchdev.c @@ -148,6 +148,17 @@ br_switchdev_fdb_notify(struct net_bridge *br, if (test_bit(BR_FDB_LOCKED, &fdb->flags)) return; + /* Entries with these flags were created using ndm_state == NUD_REACHABLE, + * ndm_flags == NTF_MASTER( | NTF_STICKY), ext_flags == 0 by something + * equivalent to 'bridge fdb add ... master dynamic (sticky)'. + * Drivers don't know how to deal with these, so don't notify them to + * avoid confusing them. + */ + if (test_bit(BR_FDB_ADDED_BY_USER, &fdb->flags) && + !test_bit(BR_FDB_STATIC, &fdb->flags) && + !test_bit(BR_FDB_ADDED_BY_EXT_LEARN, &fdb->flags)) + return; + br_switchdev_fdb_populate(br, &item, fdb, NULL); switch (type) { diff --git a/net/bridge/br_vlan.c b/net/bridge/br_vlan.c index 8a3dbc09ba38..15f44d026e75 100644 --- a/net/bridge/br_vlan.c +++ b/net/bridge/br_vlan.c @@ -2134,6 +2134,7 @@ static const struct nla_policy br_vlan_db_policy[BRIDGE_VLANDB_ENTRY_MAX + 1] = [BRIDGE_VLANDB_ENTRY_MCAST_ROUTER] = { .type = NLA_U8 }, [BRIDGE_VLANDB_ENTRY_MCAST_N_GROUPS] = { .type = NLA_REJECT }, [BRIDGE_VLANDB_ENTRY_MCAST_MAX_GROUPS] = { .type = NLA_U32 }, + [BRIDGE_VLANDB_ENTRY_NEIGH_SUPPRESS] = NLA_POLICY_MAX(NLA_U8, 1), }; static int br_vlan_rtm_process_one(struct net_device *dev, diff --git a/net/bridge/br_vlan_options.c b/net/bridge/br_vlan_options.c index e378c2f3a9e2..8fa89b04ee94 100644 --- a/net/bridge/br_vlan_options.c +++ b/net/bridge/br_vlan_options.c @@ -52,7 +52,9 @@ bool br_vlan_opts_fill(struct sk_buff *skb, const struct net_bridge_vlan *v, const struct net_bridge_port *p) { if (nla_put_u8(skb, BRIDGE_VLANDB_ENTRY_STATE, br_vlan_get_state(v)) || - !__vlan_tun_put(skb, v)) + !__vlan_tun_put(skb, v) || + nla_put_u8(skb, BRIDGE_VLANDB_ENTRY_NEIGH_SUPPRESS, + !!(v->priv_flags & BR_VLFLAG_NEIGH_SUPPRESS_ENABLED))) return false; #ifdef CONFIG_BRIDGE_IGMP_SNOOPING @@ -80,6 +82,7 @@ size_t br_vlan_opts_nl_size(void) + nla_total_size(sizeof(u32)) /* BRIDGE_VLANDB_ENTRY_MCAST_N_GROUPS */ + nla_total_size(sizeof(u32)) /* BRIDGE_VLANDB_ENTRY_MCAST_MAX_GROUPS */ #endif + + nla_total_size(sizeof(u8)) /* BRIDGE_VLANDB_ENTRY_NEIGH_SUPPRESS */ + 0; } @@ -239,6 +242,21 @@ static int br_vlan_process_one_opts(const struct net_bridge *br, } #endif + if (tb[BRIDGE_VLANDB_ENTRY_NEIGH_SUPPRESS]) { + bool enabled = v->priv_flags & BR_VLFLAG_NEIGH_SUPPRESS_ENABLED; + bool val = nla_get_u8(tb[BRIDGE_VLANDB_ENTRY_NEIGH_SUPPRESS]); + + if (!p) { + NL_SET_ERR_MSG_MOD(extack, "Can't set neigh_suppress for non-port vlans"); + return -EINVAL; + } + + if (val != enabled) { + v->priv_flags ^= BR_VLFLAG_NEIGH_SUPPRESS_ENABLED; + *changed = true; + } + } + return 0; } diff --git a/net/core/dev.c b/net/core/dev.c index 3fc4dba71f9d..1551aabac343 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -5040,7 +5040,8 @@ static __latent_entropy void net_tx_action(struct softirq_action *h) if (skb->fclone != SKB_FCLONE_UNAVAILABLE) __kfree_skb(skb); else - __kfree_skb_defer(skb); + __napi_kfree_skb(skb, + get_kfree_skb_cb(skb)->reason); } } diff --git a/net/core/drop_monitor.c b/net/core/drop_monitor.c index 5a782d1d8fd3..aff31cd944c2 100644 --- a/net/core/drop_monitor.c +++ b/net/core/drop_monitor.c @@ -21,6 +21,7 @@ #include <linux/workqueue.h> #include <linux/netlink.h> #include <linux/net_dropmon.h> +#include <linux/bitfield.h> #include <linux/percpu.h> #include <linux/timer.h> #include <linux/bitops.h> @@ -29,6 +30,7 @@ #include <net/genetlink.h> #include <net/netevent.h> #include <net/flow_offload.h> +#include <net/dropreason.h> #include <net/devlink.h> #include <trace/events/skb.h> @@ -504,8 +506,6 @@ static void net_dm_packet_trace_kfree_skb_hit(void *ignore, if (!nskb) return; - if (unlikely(reason >= SKB_DROP_REASON_MAX || reason <= 0)) - reason = SKB_DROP_REASON_NOT_SPECIFIED; cb = NET_DM_SKB_CB(nskb); cb->reason = reason; cb->pc = location; @@ -552,9 +552,9 @@ static size_t net_dm_in_port_size(void) } #define NET_DM_MAX_SYMBOL_LEN 40 +#define NET_DM_MAX_REASON_LEN 50 -static size_t net_dm_packet_report_size(size_t payload_len, - enum skb_drop_reason reason) +static size_t net_dm_packet_report_size(size_t payload_len) { size_t size; @@ -576,7 +576,7 @@ static size_t net_dm_packet_report_size(size_t payload_len, /* NET_DM_ATTR_PROTO */ nla_total_size(sizeof(u16)) + /* NET_DM_ATTR_REASON */ - nla_total_size(strlen(drop_reasons[reason]) + 1) + + nla_total_size(NET_DM_MAX_REASON_LEN + 1) + /* NET_DM_ATTR_PAYLOAD */ nla_total_size(payload_len); } @@ -610,6 +610,8 @@ static int net_dm_packet_report_fill(struct sk_buff *msg, struct sk_buff *skb, size_t payload_len) { struct net_dm_skb_cb *cb = NET_DM_SKB_CB(skb); + const struct drop_reason_list *list = NULL; + unsigned int subsys, subsys_reason; char buf[NET_DM_MAX_SYMBOL_LEN]; struct nlattr *attr; void *hdr; @@ -627,9 +629,24 @@ static int net_dm_packet_report_fill(struct sk_buff *msg, struct sk_buff *skb, NET_DM_ATTR_PAD)) goto nla_put_failure; + rcu_read_lock(); + subsys = u32_get_bits(cb->reason, SKB_DROP_REASON_SUBSYS_MASK); + if (subsys < SKB_DROP_REASON_SUBSYS_NUM) + list = rcu_dereference(drop_reasons_by_subsys[subsys]); + subsys_reason = cb->reason & ~SKB_DROP_REASON_SUBSYS_MASK; + if (!list || + subsys_reason >= list->n_reasons || + !list->reasons[subsys_reason] || + strlen(list->reasons[subsys_reason]) > NET_DM_MAX_REASON_LEN) { + list = rcu_dereference(drop_reasons_by_subsys[SKB_DROP_REASON_SUBSYS_CORE]); + subsys_reason = SKB_DROP_REASON_NOT_SPECIFIED; + } if (nla_put_string(msg, NET_DM_ATTR_REASON, - drop_reasons[cb->reason])) + list->reasons[subsys_reason])) { + rcu_read_unlock(); goto nla_put_failure; + } + rcu_read_unlock(); snprintf(buf, sizeof(buf), "%pS", cb->pc); if (nla_put_string(msg, NET_DM_ATTR_SYMBOL, buf)) @@ -687,9 +704,7 @@ static void net_dm_packet_report(struct sk_buff *skb) if (net_dm_trunc_len) payload_len = min_t(size_t, net_dm_trunc_len, payload_len); - msg = nlmsg_new(net_dm_packet_report_size(payload_len, - NET_DM_SKB_CB(skb)->reason), - GFP_KERNEL); + msg = nlmsg_new(net_dm_packet_report_size(payload_len), GFP_KERNEL); if (!msg) goto out; diff --git a/net/core/dst.c b/net/core/dst.c index 3247e84045ca..79d9306ad1ee 100644 --- a/net/core/dst.c +++ b/net/core/dst.c @@ -67,6 +67,7 @@ void dst_init(struct dst_entry *dst, struct dst_ops *ops, #endif dst->lwtstate = NULL; rcuref_init(&dst->__rcuref, initial_ref); + INIT_LIST_HEAD(&dst->rt_uncached); dst->__use = 0; dst->lastuse = jiffies; dst->flags = flags; diff --git a/net/core/filter.c b/net/core/filter.c index df0df59814ae..d9ce04ca22ce 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -2122,6 +2122,7 @@ static inline int __bpf_tx_skb(struct net_device *dev, struct sk_buff *skb) } skb->dev = dev; + skb_set_redirected_noclear(skb, skb_at_tc_ingress(skb)); skb_clear_tstamp(skb); dev_xmit_recursion_inc(); @@ -11716,6 +11717,7 @@ static int __init bpf_kfunc_init(void) ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_LWT_IN, &bpf_kfunc_set_skb); ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_LWT_XMIT, &bpf_kfunc_set_skb); ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_LWT_SEG6LOCAL, &bpf_kfunc_set_skb); + ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_NETFILTER, &bpf_kfunc_set_skb); return ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &bpf_kfunc_set_xdp); } late_initcall(bpf_kfunc_init); diff --git a/net/core/gro.c b/net/core/gro.c index a606705a0859..2d84165cb4f1 100644 --- a/net/core/gro.c +++ b/net/core/gro.c @@ -633,7 +633,7 @@ static gro_result_t napi_skb_finish(struct napi_struct *napi, else if (skb->fclone != SKB_FCLONE_UNAVAILABLE) __kfree_skb(skb); else - __kfree_skb_defer(skb); + __napi_kfree_skb(skb, SKB_CONSUMED); break; case GRO_HELD: diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 97f20f7ff4fc..e212e9d7edcb 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -839,6 +839,21 @@ void page_pool_use_xdp_mem(struct page_pool *pool, void (*disconnect)(void *), pool->xdp_mem_id = mem->id; } +void page_pool_unlink_napi(struct page_pool *pool) +{ + if (!pool->p.napi) + return; + + /* To avoid races with recycling and additional barriers make sure + * pool and NAPI are unlinked when NAPI is disabled. + */ + WARN_ON(!test_bit(NAPI_STATE_SCHED, &pool->p.napi->state) || + READ_ONCE(pool->p.napi->list_owner) != -1); + + WRITE_ONCE(pool->p.napi, NULL); +} +EXPORT_SYMBOL(page_pool_unlink_napi); + void page_pool_destroy(struct page_pool *pool) { if (!pool) @@ -847,6 +862,7 @@ void page_pool_destroy(struct page_pool *pool) if (!page_pool_put(pool)) return; + page_pool_unlink_napi(pool); page_pool_free_frag(pool); if (!page_pool_release(pool)) @@ -900,7 +916,7 @@ bool page_pool_return_skb_page(struct page *page, bool napi_safe) * in the same context as the consumer would run, so there's * no possible race. */ - napi = pp->p.napi; + napi = READ_ONCE(pp->p.napi); allow_direct = napi_safe && napi && READ_ONCE(napi->list_owner) == smp_processor_id(); diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c index e844d75220fb..653901a1bf75 100644 --- a/net/core/rtnetlink.c +++ b/net/core/rtnetlink.c @@ -61,7 +61,7 @@ #include "dev.h" #define RTNL_MAX_TYPE 50 -#define RTNL_SLAVE_MAX_TYPE 42 +#define RTNL_SLAVE_MAX_TYPE 43 struct rtnl_link { rtnl_doit_func doit; diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 768f9d04911f..0d998806b377 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -58,6 +58,7 @@ #include <linux/scatterlist.h> #include <linux/errqueue.h> #include <linux/prefetch.h> +#include <linux/bitfield.h> #include <linux/if_vlan.h> #include <linux/mpls.h> #include <linux/kcov.h> @@ -72,6 +73,7 @@ #include <net/mptcp.h> #include <net/mctp.h> #include <net/page_pool.h> +#include <net/dropreason.h> #include <linux/uaccess.h> #include <trace/events/skb.h> @@ -122,11 +124,59 @@ EXPORT_SYMBOL(sysctl_max_skb_frags); #undef FN #define FN(reason) [SKB_DROP_REASON_##reason] = #reason, -const char * const drop_reasons[] = { +static const char * const drop_reasons[] = { [SKB_CONSUMED] = "CONSUMED", DEFINE_DROP_REASON(FN, FN) }; -EXPORT_SYMBOL(drop_reasons); + +static const struct drop_reason_list drop_reasons_core = { + .reasons = drop_reasons, + .n_reasons = ARRAY_SIZE(drop_reasons), +}; + +const struct drop_reason_list __rcu * +drop_reasons_by_subsys[SKB_DROP_REASON_SUBSYS_NUM] = { + [SKB_DROP_REASON_SUBSYS_CORE] = RCU_INITIALIZER(&drop_reasons_core), +}; +EXPORT_SYMBOL(drop_reasons_by_subsys); + +/** + * drop_reasons_register_subsys - register another drop reason subsystem + * @subsys: the subsystem to register, must not be the core + * @list: the list of drop reasons within the subsystem, must point to + * a statically initialized list + */ +void drop_reasons_register_subsys(enum skb_drop_reason_subsys subsys, + const struct drop_reason_list *list) +{ + if (WARN(subsys <= SKB_DROP_REASON_SUBSYS_CORE || + subsys >= ARRAY_SIZE(drop_reasons_by_subsys), + "invalid subsystem %d\n", subsys)) + return; + + /* must point to statically allocated memory, so INIT is OK */ + RCU_INIT_POINTER(drop_reasons_by_subsys[subsys], list); +} +EXPORT_SYMBOL_GPL(drop_reasons_register_subsys); + +/** + * drop_reasons_unregister_subsys - unregister a drop reason subsystem + * @subsys: the subsystem to remove, must not be the core + * + * Note: This will synchronize_rcu() to ensure no users when it returns. + */ +void drop_reasons_unregister_subsys(enum skb_drop_reason_subsys subsys) +{ + if (WARN(subsys <= SKB_DROP_REASON_SUBSYS_CORE || + subsys >= ARRAY_SIZE(drop_reasons_by_subsys), + "invalid subsystem %d\n", subsys)) + return; + + RCU_INIT_POINTER(drop_reasons_by_subsys[subsys], NULL); + + synchronize_rcu(); +} +EXPORT_SYMBOL_GPL(drop_reasons_unregister_subsys); /** * skb_panic - private function for out-of-line support @@ -986,7 +1036,10 @@ bool __kfree_skb_reason(struct sk_buff *skb, enum skb_drop_reason reason) if (unlikely(!skb_unref(skb))) return false; - DEBUG_NET_WARN_ON_ONCE(reason <= 0 || reason >= SKB_DROP_REASON_MAX); + DEBUG_NET_WARN_ON_ONCE(reason == SKB_NOT_DROPPED_YET || + u32_get_bits(reason, + SKB_DROP_REASON_SUBSYS_MASK) >= + SKB_DROP_REASON_SUBSYS_NUM); if (reason == SKB_CONSUMED) trace_consume_skb(skb, __builtin_return_address(0)); @@ -1226,9 +1279,9 @@ static void napi_skb_cache_put(struct sk_buff *skb) } } -void __kfree_skb_defer(struct sk_buff *skb) +void __napi_kfree_skb(struct sk_buff *skb, enum skb_drop_reason reason) { - skb_release_all(skb, SKB_DROP_REASON_NOT_SPECIFIED, true); + skb_release_all(skb, reason, true); napi_skb_cache_put(skb); } diff --git a/net/ethtool/mm.c b/net/ethtool/mm.c index e00d7d5cea7e..4058a557b5a4 100644 --- a/net/ethtool/mm.c +++ b/net/ethtool/mm.c @@ -214,6 +214,16 @@ static int ethnl_set_mm(struct ethnl_req_info *req_info, struct genl_info *info) return -ERANGE; } + if (cfg.verify_enabled && !cfg.tx_enabled) { + NL_SET_ERR_MSG(extack, "Verification requires TX enabled"); + return -EINVAL; + } + + if (cfg.tx_enabled && !cfg.pmac_enabled) { + NL_SET_ERR_MSG(extack, "TX enabled requires pMAC enabled"); + return -EINVAL; + } + ret = dev->ethtool_ops->set_mm(dev, &cfg, extack); return ret < 0 ? ret : 1; } diff --git a/net/handshake/netlink.c b/net/handshake/netlink.c index 8ea0ff993f9f..35c9c445e0b8 100644 --- a/net/handshake/netlink.c +++ b/net/handshake/netlink.c @@ -249,7 +249,7 @@ static void __net_exit handshake_net_exit(struct net *net) } } -static struct pernet_operations __net_initdata handshake_genl_net_ops = { +static struct pernet_operations handshake_genl_net_ops = { .init = handshake_net_init, .exit = handshake_net_exit, .id = &handshake_net_id, diff --git a/net/ipv4/route.c b/net/ipv4/route.c index 2a3d14d95ada..98d7e6ba7493 100644 --- a/net/ipv4/route.c +++ b/net/ipv4/route.c @@ -1644,7 +1644,6 @@ struct rtable *rt_dst_alloc(struct net_device *dev, rt->rt_uses_gateway = 0; rt->rt_gw_family = 0; rt->rt_gw4 = 0; - INIT_LIST_HEAD(&rt->dst.rt_uncached); rt->dst.output = ip_output; if (flags & RTCF_LOCAL) @@ -1675,7 +1674,6 @@ struct rtable *rt_dst_clone(struct net_device *dev, struct rtable *rt) new_rt->rt_gw4 = rt->rt_gw4; else if (rt->rt_gw_family == AF_INET6) new_rt->rt_gw6 = rt->rt_gw6; - INIT_LIST_HEAD(&new_rt->dst.rt_uncached); new_rt->dst.input = rt->dst.input; new_rt->dst.output = rt->dst.output; @@ -2858,8 +2856,6 @@ struct dst_entry *ipv4_blackhole_route(struct net *net, struct dst_entry *dst_or rt->rt_gw4 = ort->rt_gw4; else if (rt->rt_gw_family == AF_INET6) rt->rt_gw6 = ort->rt_gw6; - - INIT_LIST_HEAD(&rt->dst.rt_uncached); } dst_release(dst_orig); diff --git a/net/ipv4/xfrm4_policy.c b/net/ipv4/xfrm4_policy.c index 47861c8b7340..9403bbaf1b61 100644 --- a/net/ipv4/xfrm4_policy.c +++ b/net/ipv4/xfrm4_policy.c @@ -91,7 +91,6 @@ static int xfrm4_fill_dst(struct xfrm_dst *xdst, struct net_device *dev, xdst->u.rt.rt_gw6 = rt->rt_gw6; xdst->u.rt.rt_pmtu = rt->rt_pmtu; xdst->u.rt.rt_mtu_locked = rt->rt_mtu_locked; - INIT_LIST_HEAD(&xdst->u.rt.dst.rt_uncached); rt_add_uncached_list(&xdst->u.rt); return 0; @@ -121,8 +120,7 @@ static void xfrm4_dst_destroy(struct dst_entry *dst) struct xfrm_dst *xdst = (struct xfrm_dst *)dst; dst_destroy_metrics_generic(dst); - if (xdst->u.rt.dst.rt_uncached_list) - rt_del_uncached_list(&xdst->u.rt); + rt_del_uncached_list(&xdst->u.rt); xfrm_dst_destroy(xdst); } diff --git a/net/ipv6/af_inet6.c b/net/ipv6/af_inet6.c index e1b679a590c9..2bbf13216a3d 100644 --- a/net/ipv6/af_inet6.c +++ b/net/ipv6/af_inet6.c @@ -952,6 +952,7 @@ static int __net_init inet6_net_init(struct net *net) net->ipv6.sysctl.icmpv6_echo_ignore_all = 0; net->ipv6.sysctl.icmpv6_echo_ignore_multicast = 0; net->ipv6.sysctl.icmpv6_echo_ignore_anycast = 0; + net->ipv6.sysctl.icmpv6_error_anycast_as_unicast = 0; /* By default, rate limit error messages. * Except for pmtu discovery, it would break it. diff --git a/net/ipv6/icmp.c b/net/ipv6/icmp.c index 1f53f2a74480..9edf1f45b1ed 100644 --- a/net/ipv6/icmp.c +++ b/net/ipv6/icmp.c @@ -362,9 +362,10 @@ static struct dst_entry *icmpv6_route_lookup(struct net *net, /* * We won't send icmp if the destination is known - * anycast. + * anycast unless we need to treat anycast as unicast. */ - if (ipv6_anycast_destination(dst, &fl6->daddr)) { + if (!READ_ONCE(net->ipv6.sysctl.icmpv6_error_anycast_as_unicast) && + ipv6_anycast_destination(dst, &fl6->daddr)) { net_dbg_ratelimited("icmp6_send: acast source\n"); dst_release(dst); return ERR_PTR(-EINVAL); @@ -1195,6 +1196,15 @@ static struct ctl_table ipv6_icmp_table_template[] = { .mode = 0644, .proc_handler = proc_do_large_bitmap, }, + { + .procname = "error_anycast_as_unicast", + .data = &init_net.ipv6.sysctl.icmpv6_error_anycast_as_unicast, + .maxlen = sizeof(u8), + .mode = 0644, + .proc_handler = proc_dou8vec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_ONE, + }, { }, }; @@ -1212,6 +1222,7 @@ struct ctl_table * __net_init ipv6_icmp_sysctl_init(struct net *net) table[2].data = &net->ipv6.sysctl.icmpv6_echo_ignore_multicast; table[3].data = &net->ipv6.sysctl.icmpv6_echo_ignore_anycast; table[4].data = &net->ipv6.sysctl.icmpv6_ratemask_ptr; + table[5].data = &net->ipv6.sysctl.icmpv6_error_anycast_as_unicast; } return table; } diff --git a/net/ipv6/route.c b/net/ipv6/route.c index 35085fc0cf15..e3aec46bd466 100644 --- a/net/ipv6/route.c +++ b/net/ipv6/route.c @@ -334,7 +334,6 @@ static const struct rt6_info ip6_blk_hole_entry_template = { static void rt6_info_init(struct rt6_info *rt) { memset_after(rt, 0, dst); - INIT_LIST_HEAD(&rt->dst.rt_uncached); } /* allocate dst with ip6_dst_ops */ diff --git a/net/ipv6/rpl.c b/net/ipv6/rpl.c index 488aec9e1a74..d1876f192225 100644 --- a/net/ipv6/rpl.c +++ b/net/ipv6/rpl.c @@ -32,7 +32,8 @@ static void *ipv6_rpl_segdata_pos(const struct ipv6_rpl_sr_hdr *hdr, int i) size_t ipv6_rpl_srh_size(unsigned char n, unsigned char cmpri, unsigned char cmpre) { - return (n * IPV6_PFXTAIL_LEN(cmpri)) + IPV6_PFXTAIL_LEN(cmpre); + return sizeof(struct ipv6_rpl_sr_hdr) + (n * IPV6_PFXTAIL_LEN(cmpri)) + + IPV6_PFXTAIL_LEN(cmpre); } void ipv6_rpl_srh_decompress(struct ipv6_rpl_sr_hdr *outhdr, diff --git a/net/ipv6/xfrm6_policy.c b/net/ipv6/xfrm6_policy.c index 2b493f8d0091..eecc5e59da17 100644 --- a/net/ipv6/xfrm6_policy.c +++ b/net/ipv6/xfrm6_policy.c @@ -89,7 +89,6 @@ static int xfrm6_fill_dst(struct xfrm_dst *xdst, struct net_device *dev, xdst->u.rt6.rt6i_gateway = rt->rt6i_gateway; xdst->u.rt6.rt6i_dst = rt->rt6i_dst; xdst->u.rt6.rt6i_src = rt->rt6i_src; - INIT_LIST_HEAD(&xdst->u.rt6.dst.rt_uncached); rt6_uncached_list_add(&xdst->u.rt6); return 0; @@ -121,8 +120,7 @@ static void xfrm6_dst_destroy(struct dst_entry *dst) if (likely(xdst->u.rt6.rt6i_idev)) in6_dev_put(xdst->u.rt6.rt6i_idev); dst_destroy_metrics_generic(dst); - if (xdst->u.rt6.dst.rt_uncached_list) - rt6_uncached_list_del(&xdst->u.rt6); + rt6_uncached_list_del(&xdst->u.rt6); xfrm_dst_destroy(xdst); } diff --git a/net/mac80211/cfg.c b/net/mac80211/cfg.c index 473915606715..7317e4a5d1ff 100644 --- a/net/mac80211/cfg.c +++ b/net/mac80211/cfg.c @@ -1342,6 +1342,10 @@ static int ieee80211_start_ap(struct wiphy *wiphy, struct net_device *dev, } if (params->eht_cap) { + if (!link_conf->he_support) + return -EOPNOTSUPP; + + link_conf->eht_support = true; link_conf->eht_puncturing = params->punct_bitmap; changed |= BSS_CHANGED_EHT_PUNCTURING; diff --git a/net/mac80211/debugfs.c b/net/mac80211/debugfs.c index dfb9f55e2685..207f772bd8ce 100644 --- a/net/mac80211/debugfs.c +++ b/net/mac80211/debugfs.c @@ -673,10 +673,6 @@ void debugfs_hw_add(struct ieee80211_local *local) statsd = debugfs_create_dir("statistics", phyd); - /* if the dir failed, don't put all the other things into the root! */ - if (!statsd) - return; - #ifdef CONFIG_MAC80211_DEBUG_COUNTERS DEBUGFS_STATS_ADD(dot11TransmittedFragmentCount); DEBUGFS_STATS_ADD(dot11MulticastTransmittedFrameCount); diff --git a/net/mac80211/driver-ops.h b/net/mac80211/driver-ops.h index 0bf208f5bbc5..45d3e53c7383 100644 --- a/net/mac80211/driver-ops.h +++ b/net/mac80211/driver-ops.h @@ -649,6 +649,21 @@ static inline void drv_flush(struct ieee80211_local *local, trace_drv_return_void(local); } +static inline void drv_flush_sta(struct ieee80211_local *local, + struct ieee80211_sub_if_data *sdata, + struct sta_info *sta) +{ + might_sleep(); + + if (sdata && !check_sdata_in_driver(sdata)) + return; + + trace_drv_flush_sta(local, sdata, &sta->sta); + if (local->ops->flush_sta) + local->ops->flush_sta(&local->hw, &sdata->vif, &sta->sta); + trace_drv_return_void(local); +} + static inline void drv_channel_switch(struct ieee80211_local *local, struct ieee80211_sub_if_data *sdata, struct ieee80211_channel_switch *ch_switch) diff --git a/net/mac80211/drop.h b/net/mac80211/drop.h new file mode 100644 index 000000000000..49dc809cab29 --- /dev/null +++ b/net/mac80211/drop.h @@ -0,0 +1,56 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * mac80211 drop reason list + * + * Copyright (C) 2023 Intel Corporation + */ + +#ifndef MAC80211_DROP_H +#define MAC80211_DROP_H +#include <net/dropreason.h> + +typedef unsigned int __bitwise ieee80211_rx_result; + +#define MAC80211_DROP_REASONS_MONITOR(R) \ + R(RX_DROP_M_UNEXPECTED_4ADDR_FRAME) \ + R(RX_DROP_M_BAD_BCN_KEYIDX) \ + R(RX_DROP_M_BAD_MGMT_KEYIDX) \ +/* this line for the trailing \ - add before this */ + +#define MAC80211_DROP_REASONS_UNUSABLE(R) \ + R(RX_DROP_U_MIC_FAIL) \ + R(RX_DROP_U_REPLAY) \ + R(RX_DROP_U_BAD_MMIE) \ +/* this line for the trailing \ - add before this */ + +/* having two enums allows for checking ieee80211_rx_result use with sparse */ +enum ___mac80211_drop_reason { +/* if we get to the end of handlers with RX_CONTINUE this will be the reason */ + ___RX_CONTINUE = SKB_CONSUMED, + +/* this never gets used as an argument to kfree_skb_reason() */ + ___RX_QUEUED = SKB_NOT_DROPPED_YET, + +#define ENUM(x) ___ ## x, + ___RX_DROP_MONITOR = SKB_DROP_REASON_SUBSYS_MAC80211_MONITOR << + SKB_DROP_REASON_SUBSYS_SHIFT, + MAC80211_DROP_REASONS_MONITOR(ENUM) + + ___RX_DROP_UNUSABLE = SKB_DROP_REASON_SUBSYS_MAC80211_UNUSABLE << + SKB_DROP_REASON_SUBSYS_SHIFT, + MAC80211_DROP_REASONS_UNUSABLE(ENUM) +#undef ENUM +}; + +enum mac80211_drop_reason { + RX_CONTINUE = (__force ieee80211_rx_result)___RX_CONTINUE, + RX_QUEUED = (__force ieee80211_rx_result)___RX_QUEUED, + RX_DROP_MONITOR = (__force ieee80211_rx_result)___RX_DROP_MONITOR, + RX_DROP_UNUSABLE = (__force ieee80211_rx_result)___RX_DROP_UNUSABLE, +#define DEF(x) x = (__force ieee80211_rx_result)___ ## x, + MAC80211_DROP_REASONS_MONITOR(DEF) + MAC80211_DROP_REASONS_UNUSABLE(DEF) +#undef DEF +}; + +#endif /* MAC80211_DROP_H */ diff --git a/net/mac80211/ieee80211_i.h b/net/mac80211/ieee80211_i.h index 9b7e184430b8..a0a7839cb961 100644 --- a/net/mac80211/ieee80211_i.h +++ b/net/mac80211/ieee80211_i.h @@ -33,6 +33,7 @@ #include "key.h" #include "sta_info.h" #include "debug.h" +#include "drop.h" extern const struct cfg80211_ops mac80211_config_ops; @@ -170,13 +171,6 @@ struct ieee80211_tx_data { unsigned int flags; }; - -typedef unsigned __bitwise ieee80211_rx_result; -#define RX_CONTINUE ((__force ieee80211_rx_result) 0u) -#define RX_DROP_UNUSABLE ((__force ieee80211_rx_result) 1u) -#define RX_DROP_MONITOR ((__force ieee80211_rx_result) 2u) -#define RX_QUEUED ((__force ieee80211_rx_result) 3u) - /** * enum ieee80211_packet_rx_flags - packet RX flags * @IEEE80211_RX_AMSDU: a-MSDU packet diff --git a/net/mac80211/main.c b/net/mac80211/main.c index ddf2b7811c55..55cdfaef0f5d 100644 --- a/net/mac80211/main.c +++ b/net/mac80211/main.c @@ -22,6 +22,7 @@ #include <linux/bitmap.h> #include <linux/inetdevice.h> #include <net/net_namespace.h> +#include <net/dropreason.h> #include <net/cfg80211.h> #include <net/addrconf.h> @@ -1542,6 +1543,28 @@ void ieee80211_free_hw(struct ieee80211_hw *hw) } EXPORT_SYMBOL(ieee80211_free_hw); +static const char * const drop_reasons_monitor[] = { +#define V(x) #x, + [0] = "RX_DROP_MONITOR", + MAC80211_DROP_REASONS_MONITOR(V) +}; + +static struct drop_reason_list drop_reason_list_monitor = { + .reasons = drop_reasons_monitor, + .n_reasons = ARRAY_SIZE(drop_reasons_monitor), +}; + +static const char * const drop_reasons_unusable[] = { + [0] = "RX_DROP_UNUSABLE", + MAC80211_DROP_REASONS_UNUSABLE(V) +#undef V +}; + +static struct drop_reason_list drop_reason_list_unusable = { + .reasons = drop_reasons_unusable, + .n_reasons = ARRAY_SIZE(drop_reasons_unusable), +}; + static int __init ieee80211_init(void) { struct sk_buff *skb; @@ -1559,6 +1582,11 @@ static int __init ieee80211_init(void) if (ret) goto err_netdev; + drop_reasons_register_subsys(SKB_DROP_REASON_SUBSYS_MAC80211_MONITOR, + &drop_reason_list_monitor); + drop_reasons_register_subsys(SKB_DROP_REASON_SUBSYS_MAC80211_UNUSABLE, + &drop_reason_list_unusable); + return 0; err_netdev: rc80211_minstrel_exit(); @@ -1574,6 +1602,9 @@ static void __exit ieee80211_exit(void) ieee80211_iface_exit(); + drop_reasons_unregister_subsys(SKB_DROP_REASON_SUBSYS_MAC80211_MONITOR); + drop_reasons_unregister_subsys(SKB_DROP_REASON_SUBSYS_MAC80211_UNUSABLE); + rcu_barrier(); } diff --git a/net/mac80211/rx.c b/net/mac80211/rx.c index db3451f5f2fb..58222c077898 100644 --- a/net/mac80211/rx.c +++ b/net/mac80211/rx.c @@ -1826,7 +1826,7 @@ ieee80211_rx_h_sta_process(struct ieee80211_rx_data *rx) cfg80211_rx_unexpected_4addr_frame( rx->sdata->dev, sta->sta.addr, GFP_ATOMIC); - return RX_DROP_MONITOR; + return RX_DROP_M_UNEXPECTED_4ADDR_FRAME; } /* * Update counter and free packet here to avoid @@ -1961,7 +1961,7 @@ ieee80211_rx_h_decrypt(struct ieee80211_rx_data *rx) cfg80211_rx_unprot_mlme_mgmt(rx->sdata->dev, skb->data, skb->len); - return RX_DROP_MONITOR; /* unexpected BIP keyidx */ + return RX_DROP_M_BAD_BCN_KEYIDX; } rx->key = ieee80211_rx_get_bigtk(rx, mmie_keyidx); @@ -1975,7 +1975,7 @@ ieee80211_rx_h_decrypt(struct ieee80211_rx_data *rx) if (mmie_keyidx < NUM_DEFAULT_KEYS || mmie_keyidx >= NUM_DEFAULT_KEYS + NUM_DEFAULT_MGMT_KEYS) - return RX_DROP_MONITOR; /* unexpected BIP keyidx */ + return RX_DROP_M_BAD_MGMT_KEYIDX; /* unexpected BIP keyidx */ if (rx->link_sta) { if (ieee80211_is_group_privacy_action(skb) && test_sta_flag(rx->sta, WLAN_STA_MFP)) @@ -3960,7 +3960,8 @@ ieee80211_rx_h_mgmt(struct ieee80211_rx_data *rx) } static void ieee80211_rx_cooked_monitor(struct ieee80211_rx_data *rx, - struct ieee80211_rate *rate) + struct ieee80211_rate *rate, + ieee80211_rx_result reason) { struct ieee80211_sub_if_data *sdata; struct ieee80211_local *local = rx->local; @@ -4024,42 +4025,38 @@ static void ieee80211_rx_cooked_monitor(struct ieee80211_rx_data *rx, } out_free_skb: - dev_kfree_skb(skb); + kfree_skb_reason(skb, (__force u32)reason); } static void ieee80211_rx_handlers_result(struct ieee80211_rx_data *rx, ieee80211_rx_result res) { - switch (res) { - case RX_DROP_MONITOR: - I802_DEBUG_INC(rx->sdata->local->rx_handlers_drop); - if (rx->sta) - rx->link_sta->rx_stats.dropped++; - fallthrough; - case RX_CONTINUE: { - struct ieee80211_rate *rate = NULL; - struct ieee80211_supported_band *sband; - struct ieee80211_rx_status *status; - - status = IEEE80211_SKB_RXCB((rx->skb)); + struct ieee80211_rx_status *status = IEEE80211_SKB_RXCB(rx->skb); + struct ieee80211_supported_band *sband; + struct ieee80211_rate *rate = NULL; - sband = rx->local->hw.wiphy->bands[status->band]; - if (status->encoding == RX_ENC_LEGACY) - rate = &sband->bitrates[status->rate_idx]; + if (res == RX_QUEUED) { + I802_DEBUG_INC(rx->sdata->local->rx_handlers_queued); + return; + } - ieee80211_rx_cooked_monitor(rx, rate); - break; - } - case RX_DROP_UNUSABLE: + if (res != RX_CONTINUE) { I802_DEBUG_INC(rx->sdata->local->rx_handlers_drop); if (rx->sta) rx->link_sta->rx_stats.dropped++; - dev_kfree_skb(rx->skb); - break; - case RX_QUEUED: - I802_DEBUG_INC(rx->sdata->local->rx_handlers_queued); - break; } + + if (u32_get_bits((__force u32)res, SKB_DROP_REASON_SUBSYS_MASK) == + SKB_DROP_REASON_SUBSYS_MAC80211_UNUSABLE) { + kfree_skb_reason(rx->skb, (__force u32)res); + return; + } + + sband = rx->local->hw.wiphy->bands[status->band]; + if (status->encoding == RX_ENC_LEGACY) + rate = &sband->bitrates[status->rate_idx]; + + ieee80211_rx_cooked_monitor(rx, rate, res); } static void ieee80211_rx_handlers(struct ieee80211_rx_data *rx, diff --git a/net/mac80211/sta_info.c b/net/mac80211/sta_info.c index 941bda9141fa..1400512e0dde 100644 --- a/net/mac80211/sta_info.c +++ b/net/mac80211/sta_info.c @@ -1294,6 +1294,18 @@ static void __sta_info_destroy_part2(struct sta_info *sta) WARN_ON_ONCE(ret); } + /* Flush queues before removing keys, as that might remove them + * from hardware, and then depending on the offload method, any + * frames sitting on hardware queues might be sent out without + * any encryption at all. + */ + if (local->ops->set_key) { + if (local->ops->flush_sta) + drv_flush_sta(local, sta->sdata, sta); + else + ieee80211_flush_queues(local, sta->sdata, false); + } + /* now keys can no longer be reached */ ieee80211_free_sta_keys(local, sta); diff --git a/net/mac80211/status.c b/net/mac80211/status.c index 3f9ddd7f04b6..2b13a52ce96c 100644 --- a/net/mac80211/status.c +++ b/net/mac80211/status.c @@ -1244,30 +1244,6 @@ void ieee80211_tx_rate_update(struct ieee80211_hw *hw, } EXPORT_SYMBOL(ieee80211_tx_rate_update); -void ieee80211_tx_status_8023(struct ieee80211_hw *hw, - struct ieee80211_vif *vif, - struct sk_buff *skb) -{ - struct ieee80211_sub_if_data *sdata; - struct ieee80211_tx_status status = { - .skb = skb, - .info = IEEE80211_SKB_CB(skb), - }; - struct sta_info *sta; - - sdata = vif_to_sdata(vif); - - rcu_read_lock(); - - if (!ieee80211_lookup_ra_sta(sdata, skb, &sta) && !IS_ERR(sta)) - status.sta = &sta->sta; - - ieee80211_tx_status_ext(hw, &status); - - rcu_read_unlock(); -} -EXPORT_SYMBOL(ieee80211_tx_status_8023); - void ieee80211_report_low_ack(struct ieee80211_sta *pubsta, u32 num_packets) { struct sta_info *sta = container_of(pubsta, struct sta_info, sta); diff --git a/net/mac80211/trace.h b/net/mac80211/trace.h index e0ccf5fe708a..de5d69f21306 100644 --- a/net/mac80211/trace.h +++ b/net/mac80211/trace.h @@ -1177,6 +1177,13 @@ TRACE_EVENT(drv_flush, ) ); +DEFINE_EVENT(sta_event, drv_flush_sta, + TP_PROTO(struct ieee80211_local *local, + struct ieee80211_sub_if_data *sdata, + struct ieee80211_sta *sta), + TP_ARGS(local, sdata, sta) +); + TRACE_EVENT(drv_channel_switch, TP_PROTO(struct ieee80211_local *local, struct ieee80211_sub_if_data *sdata, diff --git a/net/mac80211/tx.c b/net/mac80211/tx.c index dfe6b9c9b29e..1a3327407552 100644 --- a/net/mac80211/tx.c +++ b/net/mac80211/tx.c @@ -488,7 +488,7 @@ ieee80211_tx_h_unicast_ps_buf(struct ieee80211_tx_data *tx) int ac = skb_get_queue_mapping(tx->skb); if (ieee80211_is_mgmt(hdr->frame_control) && - !ieee80211_is_bufferable_mmpdu(hdr->frame_control)) { + !ieee80211_is_bufferable_mmpdu(tx->skb)) { info->flags |= IEEE80211_TX_CTL_NO_PS_BUFFER; return TX_CONTINUE; } @@ -1323,7 +1323,7 @@ static struct txq_info *ieee80211_get_txq(struct ieee80211_local *local, if (!(info->flags & IEEE80211_TX_CTL_HW_80211_ENCAP) && unlikely(!ieee80211_is_data_present(hdr->frame_control))) { if ((!ieee80211_is_mgmt(hdr->frame_control) || - ieee80211_is_bufferable_mmpdu(hdr->frame_control) || + ieee80211_is_bufferable_mmpdu(skb) || vif->type == NL80211_IFTYPE_STATION) && sta && sta->uploaded) { /* diff --git a/net/mac80211/wpa.c b/net/mac80211/wpa.c index 20f742b5503b..4133496da378 100644 --- a/net/mac80211/wpa.c +++ b/net/mac80211/wpa.c @@ -550,7 +550,7 @@ ieee80211_crypto_ccmp_decrypt(struct ieee80211_rx_data *rx, if (res < 0 || (!res && !(status->flag & RX_FLAG_ALLOW_SAME_PN))) { key->u.ccmp.replays++; - return RX_DROP_UNUSABLE; + return RX_DROP_U_REPLAY; } if (!(status->flag & RX_FLAG_DECRYPTED)) { @@ -564,7 +564,7 @@ ieee80211_crypto_ccmp_decrypt(struct ieee80211_rx_data *rx, skb->data + hdrlen + IEEE80211_CCMP_HDR_LEN, data_len, skb->data + skb->len - mic_len)) - return RX_DROP_UNUSABLE; + return RX_DROP_U_MIC_FAIL; } memcpy(key->u.ccmp.rx_pn[queue], pn, IEEE80211_CCMP_PN_LEN); @@ -746,7 +746,7 @@ ieee80211_crypto_gcmp_decrypt(struct ieee80211_rx_data *rx) if (res < 0 || (!res && !(status->flag & RX_FLAG_ALLOW_SAME_PN))) { key->u.gcmp.replays++; - return RX_DROP_UNUSABLE; + return RX_DROP_U_REPLAY; } if (!(status->flag & RX_FLAG_DECRYPTED)) { @@ -761,7 +761,7 @@ ieee80211_crypto_gcmp_decrypt(struct ieee80211_rx_data *rx) data_len, skb->data + skb->len - IEEE80211_GCMP_MIC_LEN)) - return RX_DROP_UNUSABLE; + return RX_DROP_U_MIC_FAIL; } memcpy(key->u.gcmp.rx_pn[queue], pn, IEEE80211_GCMP_PN_LEN); @@ -930,13 +930,13 @@ ieee80211_crypto_aes_cmac_decrypt(struct ieee80211_rx_data *rx) (skb->data + skb->len - sizeof(*mmie)); if (mmie->element_id != WLAN_EID_MMIE || mmie->length != sizeof(*mmie) - 2) - return RX_DROP_UNUSABLE; /* Invalid MMIE */ + return RX_DROP_U_BAD_MMIE; /* Invalid MMIE */ bip_ipn_swap(ipn, mmie->sequence_number); if (memcmp(ipn, key->u.aes_cmac.rx_pn, 6) <= 0) { key->u.aes_cmac.replays++; - return RX_DROP_UNUSABLE; + return RX_DROP_U_REPLAY; } if (!(status->flag & RX_FLAG_DECRYPTED)) { @@ -946,7 +946,7 @@ ieee80211_crypto_aes_cmac_decrypt(struct ieee80211_rx_data *rx) skb->data + 24, skb->len - 24, mic); if (crypto_memneq(mic, mmie->mic, sizeof(mmie->mic))) { key->u.aes_cmac.icverrors++; - return RX_DROP_UNUSABLE; + return RX_DROP_U_MIC_FAIL; } } @@ -986,7 +986,7 @@ ieee80211_crypto_aes_cmac_256_decrypt(struct ieee80211_rx_data *rx) if (memcmp(ipn, key->u.aes_cmac.rx_pn, 6) <= 0) { key->u.aes_cmac.replays++; - return RX_DROP_UNUSABLE; + return RX_DROP_U_REPLAY; } if (!(status->flag & RX_FLAG_DECRYPTED)) { @@ -996,7 +996,7 @@ ieee80211_crypto_aes_cmac_256_decrypt(struct ieee80211_rx_data *rx) skb->data + 24, skb->len - 24, mic); if (crypto_memneq(mic, mmie->mic, sizeof(mmie->mic))) { key->u.aes_cmac.icverrors++; - return RX_DROP_UNUSABLE; + return RX_DROP_U_MIC_FAIL; } } @@ -1079,13 +1079,13 @@ ieee80211_crypto_aes_gmac_decrypt(struct ieee80211_rx_data *rx) (skb->data + skb->len - sizeof(*mmie)); if (mmie->element_id != WLAN_EID_MMIE || mmie->length != sizeof(*mmie) - 2) - return RX_DROP_UNUSABLE; /* Invalid MMIE */ + return RX_DROP_U_BAD_MMIE; /* Invalid MMIE */ bip_ipn_swap(ipn, mmie->sequence_number); if (memcmp(ipn, key->u.aes_gmac.rx_pn, 6) <= 0) { key->u.aes_gmac.replays++; - return RX_DROP_UNUSABLE; + return RX_DROP_U_REPLAY; } if (!(status->flag & RX_FLAG_DECRYPTED)) { @@ -1104,7 +1104,7 @@ ieee80211_crypto_aes_gmac_decrypt(struct ieee80211_rx_data *rx) crypto_memneq(mic, mmie->mic, sizeof(mmie->mic))) { key->u.aes_gmac.icverrors++; kfree(mic); - return RX_DROP_UNUSABLE; + return RX_DROP_U_MIC_FAIL; } kfree(mic); } diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c index 1926b81a9538..08dc53f56bc2 100644 --- a/net/mptcp/protocol.c +++ b/net/mptcp/protocol.c @@ -2344,7 +2344,26 @@ static void __mptcp_close_ssk(struct sock *sk, struct sock *ssk, unsigned int flags) { struct mptcp_sock *msk = mptcp_sk(sk); - bool need_push, dispose_it; + bool dispose_it, need_push = false; + + /* If the first subflow moved to a close state before accept, e.g. due + * to an incoming reset, mptcp either: + * - if either the subflow or the msk are dead, destroy the context + * (the subflow socket is deleted by inet_child_forget) and the msk + * - otherwise do nothing at the moment and take action at accept and/or + * listener shutdown - user-space must be able to accept() the closed + * socket. + */ + if (msk->in_accept_queue && msk->first == ssk) { + if (!sock_flag(sk, SOCK_DEAD) && !sock_flag(ssk, SOCK_DEAD)) + return; + + /* ensure later check in mptcp_worker() will dispose the msk */ + sock_set_flag(sk, SOCK_DEAD); + lock_sock_nested(ssk, SINGLE_DEPTH_NESTING); + mptcp_subflow_drop_ctx(ssk); + goto out_release; + } dispose_it = !msk->subflow || ssk != msk->subflow->sk; if (dispose_it) @@ -2380,28 +2399,22 @@ static void __mptcp_close_ssk(struct sock *sk, struct sock *ssk, if (!inet_csk(ssk)->icsk_ulp_ops) { WARN_ON_ONCE(!sock_flag(ssk, SOCK_DEAD)); kfree_rcu(subflow, rcu); - } else if (msk->in_accept_queue && msk->first == ssk) { - /* if the first subflow moved to a close state, e.g. due to - * incoming reset and we reach here before inet_child_forget() - * the TCP stack could later try to close it via - * inet_csk_listen_stop(), or deliver it to the user space via - * accept(). - * We can't delete the subflow - or risk a double free - nor let - * the msk survive - or will be leaked in the non accept scenario: - * fallback and let TCP cope with the subflow cleanup. - */ - WARN_ON_ONCE(sock_flag(ssk, SOCK_DEAD)); - mptcp_subflow_drop_ctx(ssk); } else { /* otherwise tcp will dispose of the ssk and subflow ctx */ - if (ssk->sk_state == TCP_LISTEN) + if (ssk->sk_state == TCP_LISTEN) { + tcp_set_state(ssk, TCP_CLOSE); + mptcp_subflow_queue_clean(sk, ssk); + inet_csk_listen_stop(ssk); mptcp_event_pm_listener(ssk, MPTCP_EVENT_LISTENER_CLOSED); + } __tcp_close(ssk, 0); /* close acquired an extra ref */ __sock_put(ssk); } + +out_release: release_sock(ssk); sock_put(ssk); @@ -2456,21 +2469,14 @@ static void __mptcp_close_subflow(struct sock *sk) mptcp_close_ssk(sk, ssk, subflow); } - /* if the MPC subflow has been closed before the msk is accepted, - * msk will never be accept-ed, close it now - */ - if (!msk->first && msk->in_accept_queue) { - sock_set_flag(sk, SOCK_DEAD); - inet_sk_state_store(sk, TCP_CLOSE); - } } -static bool mptcp_check_close_timeout(const struct sock *sk) +static bool mptcp_should_close(const struct sock *sk) { s32 delta = tcp_jiffies32 - inet_csk(sk)->icsk_mtup.probe_timestamp; struct mptcp_subflow_context *subflow; - if (delta >= TCP_TIMEWAIT_LEN) + if (delta >= TCP_TIMEWAIT_LEN || mptcp_sk(sk)->in_accept_queue) return true; /* if all subflows are in closed status don't bother with additional @@ -2678,7 +2684,7 @@ static void mptcp_worker(struct work_struct *work) * even if it is orphaned and in FIN_WAIT2 state */ if (sock_flag(sk, SOCK_DEAD)) { - if (mptcp_check_close_timeout(sk)) { + if (mptcp_should_close(sk)) { inet_sk_state_store(sk, TCP_CLOSE); mptcp_do_fastclose(sk); } @@ -2920,6 +2926,14 @@ static void __mptcp_destroy_sock(struct sock *sk) sock_put(sk); } +void __mptcp_unaccepted_force_close(struct sock *sk) +{ + sock_set_flag(sk, SOCK_DEAD); + inet_sk_state_store(sk, TCP_CLOSE); + mptcp_do_fastclose(sk); + __mptcp_destroy_sock(sk); +} + static __poll_t mptcp_check_readable(struct mptcp_sock *msk) { /* Concurrent splices from sk_receive_queue into receive_queue will @@ -3764,6 +3778,18 @@ static int mptcp_stream_accept(struct socket *sock, struct socket *newsock, if (!ssk->sk_socket) mptcp_sock_graft(ssk, newsock); } + + /* Do late cleanup for the first subflow as necessary. Also + * deal with bad peers not doing a complete shutdown. + */ + if (msk->first && + unlikely(inet_sk_state_load(msk->first) == TCP_CLOSE)) { + __mptcp_close_ssk(newsk, msk->first, + mptcp_subflow_ctx(msk->first), 0); + if (unlikely(list_empty(&msk->conn_list))) + inet_sk_state_store(newsk, TCP_CLOSE); + } + release_sock(newsk); } diff --git a/net/mptcp/protocol.h b/net/mptcp/protocol.h index d91b852c2405..2d7b2c80a164 100644 --- a/net/mptcp/protocol.h +++ b/net/mptcp/protocol.h @@ -626,10 +626,12 @@ void mptcp_close_ssk(struct sock *sk, struct sock *ssk, struct mptcp_subflow_context *subflow); void __mptcp_subflow_send_ack(struct sock *ssk); void mptcp_subflow_reset(struct sock *ssk); +void mptcp_subflow_queue_clean(struct sock *sk, struct sock *ssk); void mptcp_sock_graft(struct sock *sk, struct socket *parent); struct socket *__mptcp_nmpc_socket(struct mptcp_sock *msk); bool __mptcp_close(struct sock *sk, long timeout); void mptcp_cancel_work(struct sock *sk); +void __mptcp_unaccepted_force_close(struct sock *sk); void mptcp_set_owner_r(struct sk_buff *skb, struct sock *sk); bool mptcp_addresses_equal(const struct mptcp_addr_info *a, diff --git a/net/mptcp/subflow.c b/net/mptcp/subflow.c index 80bbe96c0694..ba065b66551a 100644 --- a/net/mptcp/subflow.c +++ b/net/mptcp/subflow.c @@ -715,9 +715,12 @@ void mptcp_subflow_drop_ctx(struct sock *ssk) if (!ctx) return; - subflow_ulp_fallback(ssk, ctx); - if (ctx->conn) - sock_put(ctx->conn); + list_del(&mptcp_subflow_ctx(ssk)->node); + if (inet_csk(ssk)->icsk_ulp_ops) { + subflow_ulp_fallback(ssk, ctx); + if (ctx->conn) + sock_put(ctx->conn); + } kfree_rcu(ctx, rcu); } @@ -1802,6 +1805,77 @@ static void subflow_state_change(struct sock *sk) } } +void mptcp_subflow_queue_clean(struct sock *listener_sk, struct sock *listener_ssk) +{ + struct request_sock_queue *queue = &inet_csk(listener_ssk)->icsk_accept_queue; + struct mptcp_sock *msk, *next, *head = NULL; + struct request_sock *req; + struct sock *sk; + + /* build a list of all unaccepted mptcp sockets */ + spin_lock_bh(&queue->rskq_lock); + for (req = queue->rskq_accept_head; req; req = req->dl_next) { + struct mptcp_subflow_context *subflow; + struct sock *ssk = req->sk; + + if (!sk_is_mptcp(ssk)) + continue; + + subflow = mptcp_subflow_ctx(ssk); + if (!subflow || !subflow->conn) + continue; + + /* skip if already in list */ + sk = subflow->conn; + msk = mptcp_sk(sk); + if (msk->dl_next || msk == head) + continue; + + sock_hold(sk); + msk->dl_next = head; + head = msk; + } + spin_unlock_bh(&queue->rskq_lock); + if (!head) + return; + + /* can't acquire the msk socket lock under the subflow one, + * or will cause ABBA deadlock + */ + release_sock(listener_ssk); + + for (msk = head; msk; msk = next) { + sk = (struct sock *)msk; + + lock_sock_nested(sk, SINGLE_DEPTH_NESTING); + next = msk->dl_next; + msk->dl_next = NULL; + + __mptcp_unaccepted_force_close(sk); + release_sock(sk); + + /* lockdep will report a false positive ABBA deadlock + * between cancel_work_sync and the listener socket. + * The involved locks belong to different sockets WRT + * the existing AB chain. + * Using a per socket key is problematic as key + * deregistration requires process context and must be + * performed at socket disposal time, in atomic + * context. + * Just tell lockdep to consider the listener socket + * released here. + */ + mutex_release(&listener_sk->sk_lock.dep_map, _RET_IP_); + mptcp_cancel_work(sk); + mutex_acquire(&listener_sk->sk_lock.dep_map, 0, 0, _RET_IP_); + + sock_put(sk); + } + + /* we are still under the listener msk socket lock */ + lock_sock_nested(listener_ssk, SINGLE_DEPTH_NESTING); +} + static int subflow_ulp_init(struct sock *sk) { struct inet_connection_sock *icsk = inet_csk(sk); diff --git a/net/netfilter/Kconfig b/net/netfilter/Kconfig index d0bf630482c1..441d1f134110 100644 --- a/net/netfilter/Kconfig +++ b/net/netfilter/Kconfig @@ -30,6 +30,9 @@ config NETFILTER_FAMILY_BRIDGE config NETFILTER_FAMILY_ARP bool +config NETFILTER_BPF_LINK + def_bool BPF_SYSCALL + config NETFILTER_NETLINK_HOOK tristate "Netfilter base hook dump support" depends on NETFILTER_ADVANCED diff --git a/net/netfilter/Makefile b/net/netfilter/Makefile index 5ffef1cd6143..d4958e7e7631 100644 --- a/net/netfilter/Makefile +++ b/net/netfilter/Makefile @@ -22,6 +22,7 @@ nf_conntrack-$(CONFIG_DEBUG_INFO_BTF) += nf_conntrack_bpf.o endif obj-$(CONFIG_NETFILTER) = netfilter.o +obj-$(CONFIG_NETFILTER_BPF_LINK) += nf_bpf_link.o obj-$(CONFIG_NETFILTER_NETLINK) += nfnetlink.o obj-$(CONFIG_NETFILTER_NETLINK_ACCT) += nfnetlink_acct.o diff --git a/net/netfilter/core.c b/net/netfilter/core.c index 358220b58521..f0783e42108b 100644 --- a/net/netfilter/core.c +++ b/net/netfilter/core.c @@ -119,6 +119,18 @@ nf_hook_entries_grow(const struct nf_hook_entries *old, for (i = 0; i < old_entries; i++) { if (orig_ops[i] != &dummy_ops) alloc_entries++; + + /* Restrict BPF hook type to force a unique priority, not + * shared at attach time. + * + * This is mainly to avoid ordering issues between two + * different bpf programs, this doesn't prevent a normal + * hook at same priority as a bpf one (we don't want to + * prevent defrag, conntrack, iptables etc from attaching). + */ + if (reg->priority == orig_ops[i]->priority && + reg->hook_ops_type == NF_HOOK_OP_BPF) + return ERR_PTR(-EBUSY); } } diff --git a/net/netfilter/nf_bpf_link.c b/net/netfilter/nf_bpf_link.c new file mode 100644 index 000000000000..c36da56d756f --- /dev/null +++ b/net/netfilter/nf_bpf_link.c @@ -0,0 +1,228 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/bpf.h> +#include <linux/filter.h> +#include <linux/netfilter.h> + +#include <net/netfilter/nf_bpf_link.h> +#include <uapi/linux/netfilter_ipv4.h> + +static unsigned int nf_hook_run_bpf(void *bpf_prog, struct sk_buff *skb, + const struct nf_hook_state *s) +{ + const struct bpf_prog *prog = bpf_prog; + struct bpf_nf_ctx ctx = { + .state = s, + .skb = skb, + }; + + return bpf_prog_run(prog, &ctx); +} + +struct bpf_nf_link { + struct bpf_link link; + struct nf_hook_ops hook_ops; + struct net *net; + u32 dead; +}; + +static void bpf_nf_link_release(struct bpf_link *link) +{ + struct bpf_nf_link *nf_link = container_of(link, struct bpf_nf_link, link); + + if (nf_link->dead) + return; + + /* prevent hook-not-found warning splat from netfilter core when + * .detach was already called + */ + if (!cmpxchg(&nf_link->dead, 0, 1)) + nf_unregister_net_hook(nf_link->net, &nf_link->hook_ops); +} + +static void bpf_nf_link_dealloc(struct bpf_link *link) +{ + struct bpf_nf_link *nf_link = container_of(link, struct bpf_nf_link, link); + + kfree(nf_link); +} + +static int bpf_nf_link_detach(struct bpf_link *link) +{ + bpf_nf_link_release(link); + return 0; +} + +static void bpf_nf_link_show_info(const struct bpf_link *link, + struct seq_file *seq) +{ + struct bpf_nf_link *nf_link = container_of(link, struct bpf_nf_link, link); + + seq_printf(seq, "pf:\t%u\thooknum:\t%u\tprio:\t%d\n", + nf_link->hook_ops.pf, nf_link->hook_ops.hooknum, + nf_link->hook_ops.priority); +} + +static int bpf_nf_link_fill_link_info(const struct bpf_link *link, + struct bpf_link_info *info) +{ + struct bpf_nf_link *nf_link = container_of(link, struct bpf_nf_link, link); + + info->netfilter.pf = nf_link->hook_ops.pf; + info->netfilter.hooknum = nf_link->hook_ops.hooknum; + info->netfilter.priority = nf_link->hook_ops.priority; + info->netfilter.flags = 0; + + return 0; +} + +static int bpf_nf_link_update(struct bpf_link *link, struct bpf_prog *new_prog, + struct bpf_prog *old_prog) +{ + return -EOPNOTSUPP; +} + +static const struct bpf_link_ops bpf_nf_link_lops = { + .release = bpf_nf_link_release, + .dealloc = bpf_nf_link_dealloc, + .detach = bpf_nf_link_detach, + .show_fdinfo = bpf_nf_link_show_info, + .fill_link_info = bpf_nf_link_fill_link_info, + .update_prog = bpf_nf_link_update, +}; + +static int bpf_nf_check_pf_and_hooks(const union bpf_attr *attr) +{ + switch (attr->link_create.netfilter.pf) { + case NFPROTO_IPV4: + case NFPROTO_IPV6: + if (attr->link_create.netfilter.hooknum >= NF_INET_NUMHOOKS) + return -EPROTO; + break; + default: + return -EAFNOSUPPORT; + } + + if (attr->link_create.netfilter.flags) + return -EOPNOTSUPP; + + /* make sure conntrack confirm is always last. + * + * In the future, if userspace can e.g. request defrag, then + * "defrag_requested && prio before NF_IP_PRI_CONNTRACK_DEFRAG" + * should fail. + */ + switch (attr->link_create.netfilter.priority) { + case NF_IP_PRI_FIRST: return -ERANGE; /* sabotage_in and other warts */ + case NF_IP_PRI_LAST: return -ERANGE; /* e.g. conntrack confirm */ + } + + return 0; +} + +int bpf_nf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) +{ + struct net *net = current->nsproxy->net_ns; + struct bpf_link_primer link_primer; + struct bpf_nf_link *link; + int err; + + if (attr->link_create.flags) + return -EINVAL; + + err = bpf_nf_check_pf_and_hooks(attr); + if (err) + return err; + + link = kzalloc(sizeof(*link), GFP_USER); + if (!link) + return -ENOMEM; + + bpf_link_init(&link->link, BPF_LINK_TYPE_NETFILTER, &bpf_nf_link_lops, prog); + + link->hook_ops.hook = nf_hook_run_bpf; + link->hook_ops.hook_ops_type = NF_HOOK_OP_BPF; + link->hook_ops.priv = prog; + + link->hook_ops.pf = attr->link_create.netfilter.pf; + link->hook_ops.priority = attr->link_create.netfilter.priority; + link->hook_ops.hooknum = attr->link_create.netfilter.hooknum; + + link->net = net; + link->dead = false; + + err = bpf_link_prime(&link->link, &link_primer); + if (err) { + kfree(link); + return err; + } + + err = nf_register_net_hook(net, &link->hook_ops); + if (err) { + bpf_link_cleanup(&link_primer); + return err; + } + + return bpf_link_settle(&link_primer); +} + +const struct bpf_prog_ops netfilter_prog_ops = { + .test_run = bpf_prog_test_run_nf, +}; + +static bool nf_ptr_to_btf_id(struct bpf_insn_access_aux *info, const char *name) +{ + struct btf *btf; + s32 type_id; + + btf = bpf_get_btf_vmlinux(); + if (IS_ERR_OR_NULL(btf)) + return false; + + type_id = btf_find_by_name_kind(btf, name, BTF_KIND_STRUCT); + if (WARN_ON_ONCE(type_id < 0)) + return false; + + info->btf = btf; + info->btf_id = type_id; + info->reg_type = PTR_TO_BTF_ID | PTR_TRUSTED; + return true; +} + +static bool nf_is_valid_access(int off, int size, enum bpf_access_type type, + const struct bpf_prog *prog, + struct bpf_insn_access_aux *info) +{ + if (off < 0 || off >= sizeof(struct bpf_nf_ctx)) + return false; + + if (type == BPF_WRITE) + return false; + + switch (off) { + case bpf_ctx_range(struct bpf_nf_ctx, skb): + if (size != sizeof_field(struct bpf_nf_ctx, skb)) + return false; + + return nf_ptr_to_btf_id(info, "sk_buff"); + case bpf_ctx_range(struct bpf_nf_ctx, state): + if (size != sizeof_field(struct bpf_nf_ctx, state)) + return false; + + return nf_ptr_to_btf_id(info, "nf_hook_state"); + default: + return false; + } + + return false; +} + +static const struct bpf_func_proto * +bpf_nf_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) +{ + return bpf_base_func_proto(func_id); +} + +const struct bpf_verifier_ops netfilter_verifier_ops = { + .is_valid_access = nf_is_valid_access, + .get_func_proto = bpf_nf_func_proto, +}; diff --git a/net/netfilter/nf_tables_api.c b/net/netfilter/nf_tables_api.c index 6004d4b24451..e48ab8dfb541 100644 --- a/net/netfilter/nf_tables_api.c +++ b/net/netfilter/nf_tables_api.c @@ -3447,6 +3447,64 @@ static int nft_table_validate(struct net *net, const struct nft_table *table) return 0; } +int nft_setelem_validate(const struct nft_ctx *ctx, struct nft_set *set, + const struct nft_set_iter *iter, + struct nft_set_elem *elem) +{ + const struct nft_set_ext *ext = nft_set_elem_ext(set, elem->priv); + struct nft_ctx *pctx = (struct nft_ctx *)ctx; + const struct nft_data *data; + int err; + + if (nft_set_ext_exists(ext, NFT_SET_EXT_FLAGS) && + *nft_set_ext_flags(ext) & NFT_SET_ELEM_INTERVAL_END) + return 0; + + data = nft_set_ext_data(ext); + switch (data->verdict.code) { + case NFT_JUMP: + case NFT_GOTO: + pctx->level++; + err = nft_chain_validate(ctx, data->verdict.chain); + if (err < 0) + return err; + pctx->level--; + break; + default: + break; + } + + return 0; +} + +struct nft_set_elem_catchall { + struct list_head list; + struct rcu_head rcu; + void *elem; +}; + +int nft_set_catchall_validate(const struct nft_ctx *ctx, struct nft_set *set) +{ + u8 genmask = nft_genmask_next(ctx->net); + struct nft_set_elem_catchall *catchall; + struct nft_set_elem elem; + struct nft_set_ext *ext; + int ret = 0; + + list_for_each_entry_rcu(catchall, &set->catchall_list, list) { + ext = nft_set_elem_ext(set, catchall->elem); + if (!nft_set_elem_active(ext, genmask)) + continue; + + elem.priv = catchall->elem; + ret = nft_setelem_validate(ctx, set, NULL, &elem); + if (ret < 0) + return ret; + } + + return ret; +} + static struct nft_rule *nft_rule_lookup_byid(const struct net *net, const struct nft_chain *chain, const struct nlattr *nla); @@ -4759,12 +4817,6 @@ err_set_name: return err; } -struct nft_set_elem_catchall { - struct list_head list; - struct rcu_head rcu; - void *elem; -}; - static void nft_set_catchall_destroy(const struct nft_ctx *ctx, struct nft_set *set) { @@ -6056,7 +6108,8 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set, if (err < 0) return err; - if (!nla[NFTA_SET_ELEM_KEY] && !(flags & NFT_SET_ELEM_CATCHALL)) + if (((flags & NFT_SET_ELEM_CATCHALL) && nla[NFTA_SET_ELEM_KEY]) || + (!(flags & NFT_SET_ELEM_CATCHALL) && !nla[NFTA_SET_ELEM_KEY])) return -EINVAL; if (flags != 0) { @@ -7052,7 +7105,7 @@ static int nf_tables_newobj(struct sk_buff *skb, const struct nfnl_info *info, } if (nla[NFTA_OBJ_USERDATA]) { - obj->udata = nla_memdup(nla[NFTA_OBJ_USERDATA], GFP_KERNEL); + obj->udata = nla_memdup(nla[NFTA_OBJ_USERDATA], GFP_KERNEL_ACCOUNT); if (obj->udata == NULL) goto err_userdata; diff --git a/net/netfilter/nfnetlink_hook.c b/net/netfilter/nfnetlink_hook.c index 8120aadf6a0f..ade8ee1988b1 100644 --- a/net/netfilter/nfnetlink_hook.c +++ b/net/netfilter/nfnetlink_hook.c @@ -5,6 +5,7 @@ * Author: Florian Westphal <[email protected]> */ +#include <linux/bpf.h> #include <linux/module.h> #include <linux/kallsyms.h> #include <linux/kernel.h> @@ -57,35 +58,76 @@ struct nfnl_dump_hook_data { u8 hook; }; +static struct nlattr *nfnl_start_info_type(struct sk_buff *nlskb, enum nfnl_hook_chaintype t) +{ + struct nlattr *nest = nla_nest_start(nlskb, NFNLA_HOOK_CHAIN_INFO); + int ret; + + if (!nest) + return NULL; + + ret = nla_put_be32(nlskb, NFNLA_HOOK_INFO_TYPE, htonl(t)); + if (ret == 0) + return nest; + + nla_nest_cancel(nlskb, nest); + return NULL; +} + +static int nfnl_hook_put_bpf_prog_info(struct sk_buff *nlskb, + const struct nfnl_dump_hook_data *ctx, + unsigned int seq, + const struct bpf_prog *prog) +{ + struct nlattr *nest, *nest2; + int ret; + + if (!IS_ENABLED(CONFIG_NETFILTER_BPF_LINK)) + return 0; + + if (WARN_ON_ONCE(!prog)) + return 0; + + nest = nfnl_start_info_type(nlskb, NFNL_HOOK_TYPE_BPF); + if (!nest) + return -EMSGSIZE; + + nest2 = nla_nest_start(nlskb, NFNLA_HOOK_INFO_DESC); + if (!nest2) + goto cancel_nest; + + ret = nla_put_be32(nlskb, NFNLA_HOOK_BPF_ID, htonl(prog->aux->id)); + if (ret) + goto cancel_nest; + + nla_nest_end(nlskb, nest2); + nla_nest_end(nlskb, nest); + return 0; + +cancel_nest: + nla_nest_cancel(nlskb, nest); + return -EMSGSIZE; +} + static int nfnl_hook_put_nft_chain_info(struct sk_buff *nlskb, const struct nfnl_dump_hook_data *ctx, unsigned int seq, - const struct nf_hook_ops *ops) + struct nft_chain *chain) { struct net *net = sock_net(nlskb->sk); struct nlattr *nest, *nest2; - struct nft_chain *chain; int ret = 0; - if (ops->hook_ops_type != NF_HOOK_OP_NF_TABLES) - return 0; - - chain = ops->priv; if (WARN_ON_ONCE(!chain)) return 0; if (!nft_is_active(net, chain)) return 0; - nest = nla_nest_start(nlskb, NFNLA_HOOK_CHAIN_INFO); + nest = nfnl_start_info_type(nlskb, NFNL_HOOK_TYPE_NFTABLES); if (!nest) return -EMSGSIZE; - ret = nla_put_be32(nlskb, NFNLA_HOOK_INFO_TYPE, - htonl(NFNL_HOOK_TYPE_NFTABLES)); - if (ret) - goto cancel_nest; - nest2 = nla_nest_start(nlskb, NFNLA_HOOK_INFO_DESC); if (!nest2) goto cancel_nest; @@ -171,7 +213,20 @@ static int nfnl_hook_dump_one(struct sk_buff *nlskb, if (ret) goto nla_put_failure; - ret = nfnl_hook_put_nft_chain_info(nlskb, ctx, seq, ops); + switch (ops->hook_ops_type) { + case NF_HOOK_OP_NF_TABLES: + ret = nfnl_hook_put_nft_chain_info(nlskb, ctx, seq, ops->priv); + break; + case NF_HOOK_OP_BPF: + ret = nfnl_hook_put_bpf_prog_info(nlskb, ctx, seq, ops->priv); + break; + case NF_HOOK_OP_UNDEFINED: + break; + default: + WARN_ON_ONCE(1); + break; + } + if (ret) goto nla_put_failure; diff --git a/net/netfilter/nft_lookup.c b/net/netfilter/nft_lookup.c index cae5a6724163..cecf8ab90e58 100644 --- a/net/netfilter/nft_lookup.c +++ b/net/netfilter/nft_lookup.c @@ -199,37 +199,6 @@ nla_put_failure: return -1; } -static int nft_lookup_validate_setelem(const struct nft_ctx *ctx, - struct nft_set *set, - const struct nft_set_iter *iter, - struct nft_set_elem *elem) -{ - const struct nft_set_ext *ext = nft_set_elem_ext(set, elem->priv); - struct nft_ctx *pctx = (struct nft_ctx *)ctx; - const struct nft_data *data; - int err; - - if (nft_set_ext_exists(ext, NFT_SET_EXT_FLAGS) && - *nft_set_ext_flags(ext) & NFT_SET_ELEM_INTERVAL_END) - return 0; - - data = nft_set_ext_data(ext); - switch (data->verdict.code) { - case NFT_JUMP: - case NFT_GOTO: - pctx->level++; - err = nft_chain_validate(ctx, data->verdict.chain); - if (err < 0) - return err; - pctx->level--; - break; - default: - break; - } - - return 0; -} - static int nft_lookup_validate(const struct nft_ctx *ctx, const struct nft_expr *expr, const struct nft_data **d) @@ -245,9 +214,12 @@ static int nft_lookup_validate(const struct nft_ctx *ctx, iter.skip = 0; iter.count = 0; iter.err = 0; - iter.fn = nft_lookup_validate_setelem; + iter.fn = nft_setelem_validate; priv->set->ops->walk(ctx, priv->set, &iter); + if (!iter.err) + iter.err = nft_set_catchall_validate(ctx, priv->set); + if (iter.err < 0) return iter.err; diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c index 568f8d76e3c1..6080c0db0814 100644 --- a/net/packet/af_packet.c +++ b/net/packet/af_packet.c @@ -2090,18 +2090,18 @@ static unsigned int run_filter(struct sk_buff *skb, } static int packet_rcv_vnet(struct msghdr *msg, const struct sk_buff *skb, - size_t *len) + size_t *len, int vnet_hdr_sz) { - struct virtio_net_hdr vnet_hdr; + struct virtio_net_hdr_mrg_rxbuf vnet_hdr = { .num_buffers = 0 }; - if (*len < sizeof(vnet_hdr)) + if (*len < vnet_hdr_sz) return -EINVAL; - *len -= sizeof(vnet_hdr); + *len -= vnet_hdr_sz; - if (virtio_net_hdr_from_skb(skb, &vnet_hdr, vio_le(), true, 0)) + if (virtio_net_hdr_from_skb(skb, (struct virtio_net_hdr *)&vnet_hdr, vio_le(), true, 0)) return -EINVAL; - return memcpy_to_msg(msg, (void *)&vnet_hdr, sizeof(vnet_hdr)); + return memcpy_to_msg(msg, (void *)&vnet_hdr, vnet_hdr_sz); } /* @@ -2250,7 +2250,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev, __u32 ts_status; bool is_drop_n_account = false; unsigned int slot_id = 0; - bool do_vnet = false; + int vnet_hdr_sz = 0; /* struct tpacket{2,3}_hdr is aligned to a multiple of TPACKET_ALIGNMENT. * We may add members to them until current aligned size without forcing @@ -2308,10 +2308,9 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev, netoff = TPACKET_ALIGN(po->tp_hdrlen + (maclen < 16 ? 16 : maclen)) + po->tp_reserve; - if (packet_sock_flag(po, PACKET_SOCK_HAS_VNET_HDR)) { - netoff += sizeof(struct virtio_net_hdr); - do_vnet = true; - } + vnet_hdr_sz = READ_ONCE(po->vnet_hdr_sz); + if (vnet_hdr_sz) + netoff += vnet_hdr_sz; macoff = netoff - maclen; } if (netoff > USHRT_MAX) { @@ -2337,7 +2336,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev, snaplen = po->rx_ring.frame_size - macoff; if ((int)snaplen < 0) { snaplen = 0; - do_vnet = false; + vnet_hdr_sz = 0; } } } else if (unlikely(macoff + snaplen > @@ -2351,7 +2350,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev, if (unlikely((int)snaplen < 0)) { snaplen = 0; macoff = GET_PBDQC_FROM_RB(&po->rx_ring)->max_frame_len; - do_vnet = false; + vnet_hdr_sz = 0; } } spin_lock(&sk->sk_receive_queue.lock); @@ -2367,7 +2366,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev, __set_bit(slot_id, po->rx_ring.rx_owner_map); } - if (do_vnet && + if (vnet_hdr_sz && virtio_net_hdr_from_skb(skb, h.raw + macoff - sizeof(struct virtio_net_hdr), vio_le(), true, 0)) { @@ -2551,16 +2550,26 @@ static int __packet_snd_vnet_parse(struct virtio_net_hdr *vnet_hdr, size_t len) } static int packet_snd_vnet_parse(struct msghdr *msg, size_t *len, - struct virtio_net_hdr *vnet_hdr) + struct virtio_net_hdr *vnet_hdr, int vnet_hdr_sz) { - if (*len < sizeof(*vnet_hdr)) + int ret; + + if (*len < vnet_hdr_sz) return -EINVAL; - *len -= sizeof(*vnet_hdr); + *len -= vnet_hdr_sz; if (!copy_from_iter_full(vnet_hdr, sizeof(*vnet_hdr), &msg->msg_iter)) return -EFAULT; - return __packet_snd_vnet_parse(vnet_hdr, *len); + ret = __packet_snd_vnet_parse(vnet_hdr, *len); + if (ret) + return ret; + + /* move iter to point to the start of mac header */ + if (vnet_hdr_sz != sizeof(struct virtio_net_hdr)) + iov_iter_advance(&msg->msg_iter, vnet_hdr_sz - sizeof(struct virtio_net_hdr)); + + return 0; } static int tpacket_fill_skb(struct packet_sock *po, struct sk_buff *skb, @@ -2722,6 +2731,7 @@ static int tpacket_snd(struct packet_sock *po, struct msghdr *msg) void *ph; DECLARE_SOCKADDR(struct sockaddr_ll *, saddr, msg->msg_name); bool need_wait = !(msg->msg_flags & MSG_DONTWAIT); + int vnet_hdr_sz = READ_ONCE(po->vnet_hdr_sz); unsigned char *addr = NULL; int tp_len, size_max; void *data; @@ -2779,8 +2789,7 @@ static int tpacket_snd(struct packet_sock *po, struct msghdr *msg) size_max = po->tx_ring.frame_size - (po->tp_hdrlen - sizeof(struct sockaddr_ll)); - if ((size_max > dev->mtu + reserve + VLAN_HLEN) && - !packet_sock_flag(po, PACKET_SOCK_HAS_VNET_HDR)) + if ((size_max > dev->mtu + reserve + VLAN_HLEN) && !vnet_hdr_sz) size_max = dev->mtu + reserve + VLAN_HLEN; reinit_completion(&po->skb_completion); @@ -2809,10 +2818,10 @@ static int tpacket_snd(struct packet_sock *po, struct msghdr *msg) status = TP_STATUS_SEND_REQUEST; hlen = LL_RESERVED_SPACE(dev); tlen = dev->needed_tailroom; - if (packet_sock_flag(po, PACKET_SOCK_HAS_VNET_HDR)) { + if (vnet_hdr_sz) { vnet_hdr = data; - data += sizeof(*vnet_hdr); - tp_len -= sizeof(*vnet_hdr); + data += vnet_hdr_sz; + tp_len -= vnet_hdr_sz; if (tp_len < 0 || __packet_snd_vnet_parse(vnet_hdr, tp_len)) { tp_len = -EINVAL; @@ -2837,7 +2846,7 @@ static int tpacket_snd(struct packet_sock *po, struct msghdr *msg) addr, hlen, copylen, &sockc); if (likely(tp_len >= 0) && tp_len > dev->mtu + reserve && - !packet_sock_flag(po, PACKET_SOCK_HAS_VNET_HDR) && + !vnet_hdr_sz && !packet_extra_vlan_len_allowed(dev, skb)) tp_len = -EMSGSIZE; @@ -2856,7 +2865,7 @@ tpacket_error: } } - if (packet_sock_flag(po, PACKET_SOCK_HAS_VNET_HDR)) { + if (vnet_hdr_sz) { if (virtio_net_hdr_to_skb(skb, vnet_hdr, vio_le())) { tp_len = -EINVAL; goto tpacket_error; @@ -2946,7 +2955,7 @@ static int packet_snd(struct socket *sock, struct msghdr *msg, size_t len) struct virtio_net_hdr vnet_hdr = { 0 }; int offset = 0; struct packet_sock *po = pkt_sk(sk); - bool has_vnet_hdr = false; + int vnet_hdr_sz = READ_ONCE(po->vnet_hdr_sz); int hlen, tlen, linear; int extra_len = 0; @@ -2990,11 +2999,10 @@ static int packet_snd(struct socket *sock, struct msghdr *msg, size_t len) if (sock->type == SOCK_RAW) reserve = dev->hard_header_len; - if (packet_sock_flag(po, PACKET_SOCK_HAS_VNET_HDR)) { - err = packet_snd_vnet_parse(msg, &len, &vnet_hdr); + if (vnet_hdr_sz) { + err = packet_snd_vnet_parse(msg, &len, &vnet_hdr, vnet_hdr_sz); if (err) goto out_unlock; - has_vnet_hdr = true; } if (unlikely(sock_flag(sk, SOCK_NOFCS))) { @@ -3064,11 +3072,11 @@ static int packet_snd(struct socket *sock, struct msghdr *msg, size_t len) packet_parse_headers(skb, sock); - if (has_vnet_hdr) { + if (vnet_hdr_sz) { err = virtio_net_hdr_to_skb(skb, &vnet_hdr, vio_le()); if (err) goto out_free; - len += sizeof(vnet_hdr); + len += vnet_hdr_sz; virtio_net_hdr_set_proto(skb, &vnet_hdr); } @@ -3408,7 +3416,7 @@ static int packet_recvmsg(struct socket *sock, struct msghdr *msg, size_t len, struct sock *sk = sock->sk; struct sk_buff *skb; int copied, err; - int vnet_hdr_len = 0; + int vnet_hdr_len = READ_ONCE(pkt_sk(sk)->vnet_hdr_sz); unsigned int origlen = 0; err = -EINVAL; @@ -3449,11 +3457,10 @@ static int packet_recvmsg(struct socket *sock, struct msghdr *msg, size_t len, packet_rcv_try_clear_pressure(pkt_sk(sk)); - if (packet_sock_flag(pkt_sk(sk), PACKET_SOCK_HAS_VNET_HDR)) { - err = packet_rcv_vnet(msg, skb, &len); + if (vnet_hdr_len) { + err = packet_rcv_vnet(msg, skb, &len, vnet_hdr_len); if (err) goto out_free; - vnet_hdr_len = sizeof(struct virtio_net_hdr); } /* You lose any data beyond the buffer you gave. If it worries @@ -3915,8 +3922,9 @@ packet_setsockopt(struct socket *sock, int level, int optname, sockptr_t optval, return 0; } case PACKET_VNET_HDR: + case PACKET_VNET_HDR_SZ: { - int val; + int val, hdr_len; if (sock->type != SOCK_RAW) return -EINVAL; @@ -3925,11 +3933,19 @@ packet_setsockopt(struct socket *sock, int level, int optname, sockptr_t optval, if (copy_from_sockptr(&val, optval, sizeof(val))) return -EFAULT; + if (optname == PACKET_VNET_HDR_SZ) { + if (val && val != sizeof(struct virtio_net_hdr) && + val != sizeof(struct virtio_net_hdr_mrg_rxbuf)) + return -EINVAL; + hdr_len = val; + } else { + hdr_len = val ? sizeof(struct virtio_net_hdr) : 0; + } lock_sock(sk); if (po->rx_ring.pg_vec || po->tx_ring.pg_vec) { ret = -EBUSY; } else { - packet_sock_flag_set(po, PACKET_SOCK_HAS_VNET_HDR, val); + WRITE_ONCE(po->vnet_hdr_sz, hdr_len); ret = 0; } release_sock(sk); @@ -4062,7 +4078,10 @@ static int packet_getsockopt(struct socket *sock, int level, int optname, val = packet_sock_flag(po, PACKET_SOCK_ORIGDEV); break; case PACKET_VNET_HDR: - val = packet_sock_flag(po, PACKET_SOCK_HAS_VNET_HDR); + val = !!READ_ONCE(po->vnet_hdr_sz); + break; + case PACKET_VNET_HDR_SZ: + val = READ_ONCE(po->vnet_hdr_sz); break; case PACKET_VERSION: val = po->tp_version; diff --git a/net/packet/diag.c b/net/packet/diag.c index de4ced5cf3e8..d0c4eda4cdc6 100644 --- a/net/packet/diag.c +++ b/net/packet/diag.c @@ -27,7 +27,7 @@ static int pdiag_put_info(const struct packet_sock *po, struct sk_buff *nlskb) pinfo.pdi_flags |= PDI_AUXDATA; if (packet_sock_flag(po, PACKET_SOCK_ORIGDEV)) pinfo.pdi_flags |= PDI_ORIGDEV; - if (packet_sock_flag(po, PACKET_SOCK_HAS_VNET_HDR)) + if (READ_ONCE(po->vnet_hdr_sz)) pinfo.pdi_flags |= PDI_VNETHDR; if (packet_sock_flag(po, PACKET_SOCK_TP_LOSS)) pinfo.pdi_flags |= PDI_LOSS; diff --git a/net/packet/internal.h b/net/packet/internal.h index 27930f69f368..63f4865202c1 100644 --- a/net/packet/internal.h +++ b/net/packet/internal.h @@ -118,6 +118,7 @@ struct packet_sock { struct mutex pg_vec_lock; unsigned long flags; int ifindex; /* bound device */ + u8 vnet_hdr_sz; __be16 num; struct packet_rollover *rollover; struct packet_mclist *mclist; @@ -139,7 +140,6 @@ enum packet_sock_flags { PACKET_SOCK_AUXDATA, PACKET_SOCK_TX_HAS_OFF, PACKET_SOCK_TP_LOSS, - PACKET_SOCK_HAS_VNET_HDR, PACKET_SOCK_RUNNING, PACKET_SOCK_PRESSURE, PACKET_SOCK_QDISC_BYPASS, diff --git a/net/sched/cls_api.c b/net/sched/cls_api.c index 2a6b6be0811b..35785a36c802 100644 --- a/net/sched/cls_api.c +++ b/net/sched/cls_api.c @@ -3235,6 +3235,9 @@ int tcf_exts_init_ex(struct tcf_exts *exts, struct net *net, int action, err_miss_alloc: tcf_exts_destroy(exts); +#ifdef CONFIG_NET_CLS_ACT + exts->actions = NULL; +#endif return err; } EXPORT_SYMBOL(tcf_exts_init_ex); diff --git a/net/sched/sch_qfq.c b/net/sched/sch_qfq.c index cf5ebe43b3b4..02098a02943e 100644 --- a/net/sched/sch_qfq.c +++ b/net/sched/sch_qfq.c @@ -421,15 +421,16 @@ static int qfq_change_class(struct Qdisc *sch, u32 classid, u32 parentid, } else weight = 1; - if (tb[TCA_QFQ_LMAX]) { + if (tb[TCA_QFQ_LMAX]) lmax = nla_get_u32(tb[TCA_QFQ_LMAX]); - if (lmax < QFQ_MIN_LMAX || lmax > (1UL << QFQ_MTU_SHIFT)) { - pr_notice("qfq: invalid max length %u\n", lmax); - return -EINVAL; - } - } else + else lmax = psched_mtu(qdisc_dev(sch)); + if (lmax < QFQ_MIN_LMAX || lmax > (1UL << QFQ_MTU_SHIFT)) { + pr_notice("qfq: invalid max length %u\n", lmax); + return -EINVAL; + } + inv_w = ONE_FP / weight; weight = ONE_FP / inv_w; diff --git a/net/sctp/associola.c b/net/sctp/associola.c index 63ba5551c13f..796529167e8d 100644 --- a/net/sctp/associola.c +++ b/net/sctp/associola.c @@ -1597,9 +1597,10 @@ int sctp_assoc_set_bind_addr_from_cookie(struct sctp_association *asoc, struct sctp_cookie *cookie, gfp_t gfp) { - int var_size2 = ntohs(cookie->peer_init->chunk_hdr.length); + struct sctp_init_chunk *peer_init = (struct sctp_init_chunk *)(cookie + 1); + int var_size2 = ntohs(peer_init->chunk_hdr.length); int var_size3 = cookie->raw_addr_list_len; - __u8 *raw = (__u8 *)cookie->peer_init + var_size2; + __u8 *raw = (__u8 *)peer_init + var_size2; return sctp_raw_to_bind_addrs(&asoc->base.bind_addr, raw, var_size3, asoc->ep->base.bind_addr.port, gfp); diff --git a/net/sctp/auth.c b/net/sctp/auth.c index 34964145514e..c58fffc86a0c 100644 --- a/net/sctp/auth.c +++ b/net/sctp/auth.c @@ -738,7 +738,7 @@ void sctp_auth_calculate_hmac(const struct sctp_association *asoc, tfm = asoc->ep->auth_hmacs[hmac_id]; - digest = auth->auth_hdr.hmac; + digest = (u8 *)(&auth->auth_hdr + 1); if (crypto_shash_setkey(tfm, &asoc_key->data[0], asoc_key->len)) goto free; diff --git a/net/sctp/input.c b/net/sctp/input.c index 127bf28a6033..2613c4d74b16 100644 --- a/net/sctp/input.c +++ b/net/sctp/input.c @@ -1150,7 +1150,7 @@ static struct sctp_association *__sctp_rcv_init_lookup(struct net *net, init = (struct sctp_init_chunk *)skb->data; /* Walk the parameters looking for embedded addresses. */ - sctp_walk_params(params, init, init_hdr.params) { + sctp_walk_params(params, init) { /* Note: Ignoring hostname addresses. */ af = sctp_get_af_specific(param_type2af(params.p->type)); diff --git a/net/sctp/outqueue.c b/net/sctp/outqueue.c index 20831079fb09..0dc6b8ab9963 100644 --- a/net/sctp/outqueue.c +++ b/net/sctp/outqueue.c @@ -1231,7 +1231,7 @@ static void sctp_sack_update_unack_data(struct sctp_association *assoc, unack_data = assoc->next_tsn - assoc->ctsn_ack_point - 1; - frags = sack->variable; + frags = (union sctp_sack_variable *)(sack + 1); for (i = 0; i < ntohs(sack->num_gap_ack_blocks); i++) { unack_data -= ((ntohs(frags[i].gab.end) - ntohs(frags[i].gab.start) + 1)); @@ -1252,7 +1252,6 @@ int sctp_outq_sack(struct sctp_outq *q, struct sctp_chunk *chunk) struct sctp_transport *transport; struct sctp_chunk *tchunk = NULL; struct list_head *lchunk, *transport_list, *temp; - union sctp_sack_variable *frags = sack->variable; __u32 sack_ctsn, ctsn, tsn; __u32 highest_tsn, highest_new_tsn; __u32 sack_a_rwnd; @@ -1313,8 +1312,12 @@ int sctp_outq_sack(struct sctp_outq *q, struct sctp_chunk *chunk) /* Get the highest TSN in the sack. */ highest_tsn = sack_ctsn; - if (gap_ack_blocks) + if (gap_ack_blocks) { + union sctp_sack_variable *frags = + (union sctp_sack_variable *)(sack + 1); + highest_tsn += ntohs(frags[gap_ack_blocks - 1].gab.end); + } if (TSN_lt(asoc->highest_sacked, highest_tsn)) asoc->highest_sacked = highest_tsn; @@ -1789,7 +1792,7 @@ static int sctp_acked(struct sctp_sackhdr *sack, __u32 tsn) * Block are assumed to have been received correctly. */ - frags = sack->variable; + frags = (union sctp_sack_variable *)(sack + 1); blocks = ntohs(sack->num_gap_ack_blocks); tsn_offset = tsn - ctsn; for (i = 0; i < blocks; ++i) { diff --git a/net/sctp/sm_make_chunk.c b/net/sctp/sm_make_chunk.c index c8f4ec5d5f98..08527d882e56 100644 --- a/net/sctp/sm_make_chunk.c +++ b/net/sctp/sm_make_chunk.c @@ -1707,11 +1707,11 @@ static struct sctp_cookie_param *sctp_pack_cookie( ktime_get_real()); /* Copy the peer's init packet. */ - memcpy(&cookie->c.peer_init[0], init_chunk->chunk_hdr, + memcpy(cookie + 1, init_chunk->chunk_hdr, ntohs(init_chunk->chunk_hdr->length)); /* Copy the raw local address list of the association. */ - memcpy((__u8 *)&cookie->c.peer_init[0] + + memcpy((__u8 *)(cookie + 1) + ntohs(init_chunk->chunk_hdr->length), raw_addrs, addrs_len); if (sctp_sk(ep->base.sk)->hmac) { @@ -2306,7 +2306,7 @@ int sctp_verify_init(struct net *net, const struct sctp_endpoint *ep, ntohl(peer_init->init_hdr.a_rwnd) < SCTP_DEFAULT_MINWINDOW) return sctp_process_inv_mandatory(asoc, chunk, errp); - sctp_walk_params(param, peer_init, init_hdr.params) { + sctp_walk_params(param, peer_init) { if (param.p->type == SCTP_PARAM_STATE_COOKIE) has_cookie = true; } @@ -2329,7 +2329,7 @@ int sctp_verify_init(struct net *net, const struct sctp_endpoint *ep, chunk, errp); /* Verify all the variable length parameters */ - sctp_walk_params(param, peer_init, init_hdr.params) { + sctp_walk_params(param, peer_init) { result = sctp_verify_param(net, ep, asoc, param, cid, chunk, errp); switch (result) { @@ -2381,7 +2381,7 @@ int sctp_process_init(struct sctp_association *asoc, struct sctp_chunk *chunk, src_match = 1; /* Process the initialization parameters. */ - sctp_walk_params(param, peer_init, init_hdr.params) { + sctp_walk_params(param, peer_init) { if (!src_match && (param.p->type == SCTP_PARAM_IPV4_ADDRESS || param.p->type == SCTP_PARAM_IPV6_ADDRESS)) { @@ -3202,7 +3202,7 @@ bool sctp_verify_asconf(const struct sctp_association *asoc, union sctp_params param; addip = (struct sctp_addip_chunk *)chunk->chunk_hdr; - sctp_walk_params(param, addip, addip_hdr.params) { + sctp_walk_params(param, addip) { size_t length = ntohs(param.p->length); *errp = param.p; @@ -3215,14 +3215,14 @@ bool sctp_verify_asconf(const struct sctp_association *asoc, /* ensure there is only one addr param and it's in the * beginning of addip_hdr params, or we reject it. */ - if (param.v != addip->addip_hdr.params) + if (param.v != (addip + 1)) return false; addr_param_seen = true; break; case SCTP_PARAM_IPV6_ADDRESS: if (length != sizeof(struct sctp_ipv6addr_param)) return false; - if (param.v != addip->addip_hdr.params) + if (param.v != (addip + 1)) return false; addr_param_seen = true; break; @@ -3302,7 +3302,7 @@ struct sctp_chunk *sctp_process_asconf(struct sctp_association *asoc, goto done; /* Process the TLVs contained within the ASCONF chunk. */ - sctp_walk_params(param, addip, addip_hdr.params) { + sctp_walk_params(param, addip) { /* Skip preceeding address parameters. */ if (param.p->type == SCTP_PARAM_IPV4_ADDRESS || param.p->type == SCTP_PARAM_IPV6_ADDRESS) @@ -3636,7 +3636,7 @@ static struct sctp_chunk *sctp_make_reconf(const struct sctp_association *asoc, return NULL; reconf = (struct sctp_reconf_chunk *)retval->chunk_hdr; - retval->param_hdr.v = reconf->params; + retval->param_hdr.v = (u8 *)(reconf + 1); return retval; } @@ -3878,7 +3878,7 @@ bool sctp_verify_reconf(const struct sctp_association *asoc, __u16 cnt = 0; hdr = (struct sctp_reconf_chunk *)chunk->chunk_hdr; - sctp_walk_params(param, hdr, params) { + sctp_walk_params(param, hdr) { __u16 length = ntohs(param.p->length); *errp = param.p; diff --git a/net/sctp/sm_sideeffect.c b/net/sctp/sm_sideeffect.c index 463c4a58d2c3..7fbeb99d8d32 100644 --- a/net/sctp/sm_sideeffect.c +++ b/net/sctp/sm_sideeffect.c @@ -984,8 +984,7 @@ static void sctp_cmd_process_operr(struct sctp_cmd_seq *cmds, { struct sctp_chunkhdr *unk_chunk_hdr; - unk_chunk_hdr = (struct sctp_chunkhdr *) - err_hdr->variable; + unk_chunk_hdr = (struct sctp_chunkhdr *)(err_hdr + 1); switch (unk_chunk_hdr->type) { /* ADDIP 4.1 A9) If the peer responds to an ASCONF with * an ERROR chunk reporting that it did not recognized diff --git a/net/sctp/sm_statefuns.c b/net/sctp/sm_statefuns.c index ce5426171206..97f1155a2045 100644 --- a/net/sctp/sm_statefuns.c +++ b/net/sctp/sm_statefuns.c @@ -794,8 +794,7 @@ enum sctp_disposition sctp_sf_do_5_1D_ce(struct net *net, /* This is a brand-new association, so these are not yet side * effects--it is safe to run them here. */ - peer_init = &chunk->subh.cookie_hdr->c.peer_init[0]; - + peer_init = (struct sctp_init_chunk *)(chunk->subh.cookie_hdr + 1); if (!sctp_process_init(new_asoc, chunk, &chunk->subh.cookie_hdr->c.peer_addr, peer_init, GFP_ATOMIC)) @@ -1337,7 +1336,7 @@ static int sctp_sf_send_restart_abort(struct net *net, union sctp_addr *ssa, * throughout the code today. */ errhdr = (struct sctp_errhdr *)buffer; - addrparm = (union sctp_addr_param *)errhdr->variable; + addrparm = (union sctp_addr_param *)(errhdr + 1); /* Copy into a parm format. */ len = af->to_addr_param(ssa, addrparm); @@ -1869,8 +1868,7 @@ static enum sctp_disposition sctp_sf_do_dupcook_a( /* new_asoc is a brand-new association, so these are not yet * side effects--it is safe to run them here. */ - peer_init = &chunk->subh.cookie_hdr->c.peer_init[0]; - + peer_init = (struct sctp_init_chunk *)(chunk->subh.cookie_hdr + 1); if (!sctp_process_init(new_asoc, chunk, sctp_source(chunk), peer_init, GFP_ATOMIC)) goto nomem; @@ -1990,7 +1988,7 @@ static enum sctp_disposition sctp_sf_do_dupcook_b( /* new_asoc is a brand-new association, so these are not yet * side effects--it is safe to run them here. */ - peer_init = &chunk->subh.cookie_hdr->c.peer_init[0]; + peer_init = (struct sctp_init_chunk *)(chunk->subh.cookie_hdr + 1); if (!sctp_process_init(new_asoc, chunk, sctp_source(chunk), peer_init, GFP_ATOMIC)) goto nomem; @@ -4142,7 +4140,7 @@ enum sctp_disposition sctp_sf_do_reconf(struct net *net, (void *)err_param, commands); hdr = (struct sctp_reconf_chunk *)chunk->chunk_hdr; - sctp_walk_params(param, hdr, params) { + sctp_walk_params(param, hdr) { struct sctp_chunk *reply = NULL; struct sctp_ulpevent *ev = NULL; @@ -4393,7 +4391,7 @@ static enum sctp_ierror sctp_sf_authenticate( * 3. Compute the new digest * 4. Compare saved and new digests. */ - digest = auth_hdr->hmac; + digest = (u8 *)(auth_hdr + 1); skb_pull(chunk->skb, sig_len); save_digest = kmemdup(digest, sig_len, GFP_ATOMIC); diff --git a/net/sctp/stream.c b/net/sctp/stream.c index ee6514af830f..c241cc552e8d 100644 --- a/net/sctp/stream.c +++ b/net/sctp/stream.c @@ -491,7 +491,7 @@ static struct sctp_paramhdr *sctp_chunk_lookup_strreset_param( return NULL; hdr = (struct sctp_reconf_chunk *)chunk->chunk_hdr; - sctp_walk_params(param, hdr, params) { + sctp_walk_params(param, hdr) { /* sctp_strreset_tsnreq is actually the basic structure * of all stream reconf params, so it's safe to use it * to access request_seq. diff --git a/net/sctp/stream_interleave.c b/net/sctp/stream_interleave.c index b046b11200c9..840f24045ae2 100644 --- a/net/sctp/stream_interleave.c +++ b/net/sctp/stream_interleave.c @@ -1153,8 +1153,8 @@ static void sctp_generate_iftsn(struct sctp_outq *q, __u32 ctsn) } #define _sctp_walk_ifwdtsn(pos, chunk, end) \ - for (pos = chunk->subh.ifwdtsn_hdr->skip; \ - (void *)pos <= (void *)chunk->subh.ifwdtsn_hdr->skip + (end) - \ + for (pos = (void *)(chunk->subh.ifwdtsn_hdr + 1); \ + (void *)pos <= (void *)(chunk->subh.ifwdtsn_hdr + 1) + (end) - \ sizeof(struct sctp_ifwdtsn_skip); pos++) #define sctp_walk_ifwdtsn(pos, ch) \ diff --git a/net/sunrpc/auth_gss/gss_krb5_test.c b/net/sunrpc/auth_gss/gss_krb5_test.c index ce0541e32fc9..95ca783795c5 100644 --- a/net/sunrpc/auth_gss/gss_krb5_test.c +++ b/net/sunrpc/auth_gss/gss_krb5_test.c @@ -73,7 +73,6 @@ static void checksum_case(struct kunit *test) { const struct gss_krb5_test_param *param = test->param_value; struct xdr_buf buf = { - .head[0].iov_base = param->plaintext->data, .head[0].iov_len = param->plaintext->len, .len = param->plaintext->len, }; @@ -99,6 +98,10 @@ static void checksum_case(struct kunit *test) err = crypto_ahash_setkey(tfm, Kc.data, Kc.len); KUNIT_ASSERT_EQ(test, err, 0); + buf.head[0].iov_base = kunit_kzalloc(test, buf.head[0].iov_len, GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf.head[0].iov_base); + memcpy(buf.head[0].iov_base, param->plaintext->data, buf.head[0].iov_len); + checksum.len = gk5e->cksumlength; checksum.data = kunit_kzalloc(test, checksum.len, GFP_KERNEL); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, checksum.data); @@ -1327,6 +1330,7 @@ static void rfc6803_encrypt_case(struct kunit *test) if (!gk5e) kunit_skip(test, "Encryption type is not available"); + memset(usage_data, 0, sizeof(usage_data)); usage.data[3] = param->constant; Ke.len = gk5e->Ke_length; diff --git a/rust/Makefile b/rust/Makefile index f88d108fbef0..aef85e9e8eeb 100644 --- a/rust/Makefile +++ b/rust/Makefile @@ -262,6 +262,20 @@ BINDGEN_TARGET := $(BINDGEN_TARGET_$(SRCARCH)) # some configurations, with new GCC versions, etc. bindgen_extra_c_flags = -w --target=$(BINDGEN_TARGET) +# Auto variable zero-initialization requires an additional special option with +# clang that is going to be removed sometime in the future (likely in +# clang-18), so make sure to pass this option only if clang supports it +# (libclang major version < 16). +# +# https://github.com/llvm/llvm-project/issues/44842 +# https://github.com/llvm/llvm-project/blob/llvmorg-16.0.0-rc2/clang/docs/ReleaseNotes.rst#deprecated-compiler-flags +ifdef CONFIG_INIT_STACK_ALL_ZERO +libclang_maj_ver=$(shell $(BINDGEN) $(srctree)/scripts/rust_is_available_bindgen_libclang.h 2>&1 | sed -ne 's/.*clang version \([0-9]*\).*/\1/p') +ifeq ($(shell expr $(libclang_maj_ver) \< 16), 1) +bindgen_extra_c_flags += -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang +endif +endif + bindgen_c_flags = $(filter-out $(bindgen_skip_c_flags), $(c_flags)) \ $(bindgen_extra_c_flags) endif @@ -283,7 +297,7 @@ quiet_cmd_bindgen = BINDGEN $@ $(bindgen_target_cflags) $(bindgen_target_extra) $(obj)/bindings/bindings_generated.rs: private bindgen_target_flags = \ - $(shell grep -v '^\#\|^$$' $(srctree)/$(src)/bindgen_parameters) + $(shell grep -v '^#\|^$$' $(srctree)/$(src)/bindgen_parameters) $(obj)/bindings/bindings_generated.rs: $(src)/bindings/bindings_helper.h \ $(src)/bindgen_parameters FORCE $(call if_changed_dep,bindgen) diff --git a/rust/kernel/print.rs b/rust/kernel/print.rs index 30103325696d..8009184bf6d7 100644 --- a/rust/kernel/print.rs +++ b/rust/kernel/print.rs @@ -18,7 +18,11 @@ use crate::bindings; // Called from `vsprintf` with format specifier `%pA`. #[no_mangle] -unsafe fn rust_fmt_argument(buf: *mut c_char, end: *mut c_char, ptr: *const c_void) -> *mut c_char { +unsafe extern "C" fn rust_fmt_argument( + buf: *mut c_char, + end: *mut c_char, + ptr: *const c_void, +) -> *mut c_char { use fmt::Write; // SAFETY: The C contract guarantees that `buf` is valid if it's less than `end`. let mut w = unsafe { RawFormatter::from_ptrs(buf.cast(), end.cast()) }; diff --git a/rust/kernel/str.rs b/rust/kernel/str.rs index b771310fa4a4..cd3d2a6cf1fc 100644 --- a/rust/kernel/str.rs +++ b/rust/kernel/str.rs @@ -408,7 +408,7 @@ impl RawFormatter { /// If `pos` is less than `end`, then the region between `pos` (inclusive) and `end` /// (exclusive) must be valid for writes for the lifetime of the returned [`RawFormatter`]. pub(crate) unsafe fn from_ptrs(pos: *mut u8, end: *mut u8) -> Self { - // INVARIANT: The safety requierments guarantee the type invariants. + // INVARIANT: The safety requirements guarantee the type invariants. Self { beg: pos as _, pos: pos as _, diff --git a/samples/bpf/sampleip_user.c b/samples/bpf/sampleip_user.c index 921c505bb567..9283f47844fb 100644 --- a/samples/bpf/sampleip_user.c +++ b/samples/bpf/sampleip_user.c @@ -21,10 +21,10 @@ #define DEFAULT_FREQ 99 #define DEFAULT_SECS 5 #define MAX_IPS 8192 -#define PAGE_OFFSET 0xffff880000000000 static int map_fd; static int nr_cpus; +static long _text_addr; static void usage(void) { @@ -108,7 +108,7 @@ static void print_ip_map(int fd) /* sort and print */ qsort(counts, max, sizeof(struct ipcount), count_cmp); for (i = 0; i < max; i++) { - if (counts[i].ip > PAGE_OFFSET) { + if (counts[i].ip > _text_addr) { sym = ksym_search(counts[i].ip); if (!sym) { printf("ksym not found. Is kallsyms loaded?\n"); @@ -169,6 +169,13 @@ int main(int argc, char **argv) return 2; } + /* used to determine whether the address is kernel space */ + _text_addr = ksym_get_addr("_text"); + if (!_text_addr) { + fprintf(stderr, "ERROR: no '_text' in /proc/kallsyms\n"); + return 3; + } + /* create perf FDs for each CPU */ nr_cpus = sysconf(_SC_NPROCESSORS_ONLN); links = calloc(nr_cpus, sizeof(struct bpf_link *)); diff --git a/scripts/Makefile.package b/scripts/Makefile.package index 61f72eb8d9be..4d90691505b1 100644 --- a/scripts/Makefile.package +++ b/scripts/Makefile.package @@ -27,21 +27,6 @@ fi ; \ tar -I $(KGZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \ --transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3) -# tarball compression -# --------------------------------------------------------------------------- - -%.tar.gz: %.tar - $(call cmd,gzip) - -%.tar.bz2: %.tar - $(call cmd,bzip2) - -%.tar.xz: %.tar - $(call cmd,xzmisc) - -%.tar.zst: %.tar - $(call cmd,zstd) - # Git # --------------------------------------------------------------------------- @@ -57,16 +42,24 @@ check-git: false; \ fi +git-config-tar.gz = -c tar.tar.gz.command="$(KGZIP)" +git-config-tar.bz2 = -c tar.tar.bz2.command="$(KBZIP2)" +git-config-tar.xz = -c tar.tar.xz.command="$(XZ)" +git-config-tar.zst = -c tar.tar.zst.command="$(ZSTD)" + +quiet_cmd_archive = ARCHIVE $@ + cmd_archive = git -C $(srctree) $(git-config-tar$(suffix $@)) archive \ + --output=$$(realpath $@) --prefix=$(basename $@)/ $(archive-args) + # Linux source tarball # --------------------------------------------------------------------------- -quiet_cmd_archive_linux = ARCHIVE $@ - cmd_archive_linux = \ - git -C $(srctree) archive --output=$$(realpath $@) --prefix=$(basename $@)/ $$(cat $<) +linux-tarballs := $(addprefix linux, .tar.gz) -targets += linux.tar -linux.tar: .tmp_HEAD FORCE - $(call if_changed,archive_linux) +targets += $(linux-tarballs) +$(linux-tarballs): archive-args = $$(cat $<) +$(linux-tarballs): .tmp_HEAD FORCE + $(call if_changed,archive) # rpm-pkg # --------------------------------------------------------------------------- @@ -94,7 +87,7 @@ binrpm-pkg: $(UTS_MACHINE)-linux -bb $(objtree)/binkernel.spec quiet_cmd_debianize = GEN $@ - cmd_debianize = $(srctree)/scripts/package/mkdebian + cmd_debianize = $(srctree)/scripts/package/mkdebian $(mkdebian-opts) debian: FORCE $(call cmd,debianize) @@ -103,6 +96,7 @@ PHONY += debian-orig debian-orig: private source = $(shell dpkg-parsechangelog -S Source) debian-orig: private version = $(shell dpkg-parsechangelog -S Version | sed 's/-[^-]*$$//') debian-orig: private orig-name = $(source)_$(version).orig.tar.gz +debian-orig: mkdebian-opts = --need-source debian-orig: linux.tar.gz debian $(Q)if [ "$(df --output=target .. 2>/dev/null)" = "$(df --output=target $< 2>/dev/null)" ]; then \ ln -f $< ../$(orig-name); \ @@ -145,10 +139,17 @@ tar-install: FORCE $(Q)$(MAKE) -f $(srctree)/Makefile +$(Q)$(srctree)/scripts/package/buildtar $@ +compress-tar.gz = -I "$(KGZIP)" +compress-tar.bz2 = -I "$(KBZIP2)" +compress-tar.xz = -I "$(XZ)" +compress-tar.zst = -I "$(ZSTD)" + quiet_cmd_tar = TAR $@ - cmd_tar = cd $<; tar cf ../$@ --owner=root --group=root --sort=name * + cmd_tar = cd $<; tar cf ../$@ $(compress-tar$(suffix $@)) --owner=root --group=root --sort=name * -linux-$(KERNELRELEASE)-$(ARCH).tar: tar-install +dir-tarballs := $(addprefix linux-$(KERNELRELEASE)-$(ARCH), .tar .tar.gz .tar.bz2 .tar.xz .tar.zst) + +$(dir-tarballs): tar-install $(call cmd,tar) PHONY += dir-pkg @@ -180,16 +181,17 @@ quiet_cmd_perf_version_file = GEN $@ .tmp_perf/PERF-VERSION-FILE: .tmp_HEAD $(srctree)/tools/perf/util/PERF-VERSION-GEN | .tmp_perf $(call cmd,perf_version_file) -quiet_cmd_archive_perf = ARCHIVE $@ - cmd_archive_perf = \ - git -C $(srctree) archive --output=$$(realpath $@) --prefix=$(basename $@)/ \ - --add-file=$$(realpath $(word 2, $^)) \ +perf-archive-args = --add-file=$$(realpath $(word 2, $^)) \ --add-file=$$(realpath $(word 3, $^)) \ $$(cat $(word 2, $^))^{tree} $$(cat $<) -targets += perf-$(KERNELVERSION).tar -perf-$(KERNELVERSION).tar: tools/perf/MANIFEST .tmp_perf/HEAD .tmp_perf/PERF-VERSION-FILE FORCE - $(call if_changed,archive_perf) + +perf-tarballs := $(addprefix perf-$(KERNELVERSION), .tar .tar.gz .tar.bz2 .tar.xz .tar.zst) + +targets += $(perf-tarballs) +$(perf-tarballs): archive-args = $(perf-archive-args) +$(perf-tarballs): tools/perf/MANIFEST .tmp_perf/HEAD .tmp_perf/PERF-VERSION-FILE FORCE + $(call if_changed,archive) PHONY += perf-tar-src-pkg perf-tar-src-pkg: perf-$(KERNELVERSION).tar diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py index ecc7ea9a4dcf..946e250c1b2a 100755 --- a/scripts/generate_rust_analyzer.py +++ b/scripts/generate_rust_analyzer.py @@ -104,7 +104,10 @@ def generate_crates(srctree, objtree, sysroot_src): name = path.name.replace(".rs", "") # Skip those that are not crate roots. - if f"{name}.o" not in open(path.parent / "Makefile").read(): + try: + if f"{name}.o" not in open(path.parent / "Makefile").read(): + continue + except FileNotFoundError: continue logging.info("Adding %s", name) diff --git a/scripts/is_rust_module.sh b/scripts/is_rust_module.sh index 28b3831a7593..464761a7cf7f 100755 --- a/scripts/is_rust_module.sh +++ b/scripts/is_rust_module.sh @@ -13,4 +13,4 @@ set -e # # In the future, checking for the `.comment` section may be another # option, see https://github.com/rust-lang/rust/pull/97550. -${NM} "$*" | grep -qE '^[0-9a-fA-F]+ r _R[^[:space:]]+16___IS_RUST_MODULE[^[:space:]]*$' +${NM} "$*" | grep -qE '^[0-9a-fA-F]+ [Rr] _R[^[:space:]]+16___IS_RUST_MODULE[^[:space:]]*$' diff --git a/scripts/package/gen-diff-patch b/scripts/package/gen-diff-patch index f842ab50a780..8a98b7bb78a0 100755 --- a/scripts/package/gen-diff-patch +++ b/scripts/package/gen-diff-patch @@ -1,44 +1,36 @@ #!/bin/sh # SPDX-License-Identifier: GPL-2.0-only -diff_patch="${1}" -untracked_patch="${2}" -srctree=$(dirname $0)/../.. +diff_patch=$1 -rm -f ${diff_patch} ${untracked_patch} +mkdir -p "$(dirname "${diff_patch}")" -if ! ${srctree}/scripts/check-git; then - exit -fi - -mkdir -p "$(dirname ${diff_patch})" "$(dirname ${untracked_patch})" +git -C "${srctree:-.}" diff HEAD > "${diff_patch}" -git -C "${srctree}" diff HEAD > "${diff_patch}" - -if [ ! -s "${diff_patch}" ]; then - rm -f "${diff_patch}" +if [ ! -s "${diff_patch}" ] || + [ -z "$(git -C "${srctree:-.}" ls-files --other --exclude-standard | head -n1)" ]; then exit fi -git -C ${srctree} status --porcelain --untracked-files=all | -while read stat path -do - if [ "${stat}" = '??' ]; then - - if ! diff -u /dev/null "${srctree}/${path}" > .tmp_diff && - ! head -n1 .tmp_diff | grep -q "Binary files"; then - { - echo "--- /dev/null" - echo "+++ linux/$path" - cat .tmp_diff | tail -n +3 - } >> ${untracked_patch} - fi - fi -done - -rm -f .tmp_diff - -if [ ! -s "${diff_patch}" ]; then - rm -f "${diff_patch}" - exit -fi +# The source tarball, which is generated by 'git archive', contains everything +# you committed in the repository. If you have local diff ('git diff HEAD'), +# it will go into ${diff_patch}. If untracked files are remaining, the resulting +# source package may not be correct. +# +# Examples: +# - You modified a source file to add #include "new-header.h" +# but forgot to add new-header.h +# - You modified a Makefile to add 'obj-$(CONFIG_FOO) += new-dirver.o' +# but you forgot to add new-driver.c +# +# You need to commit them, or at least stage them by 'git add'. +# +# This script does not take care of untracked files because doing so would +# introduce additional complexity. Instead, print a warning message here if +# untracked files are found. +# If all untracked files are just garbage, you can ignore this warning. +echo >&2 "============================ WARNING ============================" +echo >&2 "Your working tree has diff from HEAD, and also untracked file(s)." +echo >&2 "Please make sure you did 'git add' for all new files you need in" +echo >&2 "the source package." +echo >&2 "=================================================================" diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian index e20a2b5be9eb..a4c2c2276223 100755 --- a/scripts/package/mkdebian +++ b/scripts/package/mkdebian @@ -84,7 +84,66 @@ set_debarch() { fi } +# Create debian/source/ if it is a source package build +gen_source () +{ + mkdir -p debian/source + + echo "3.0 (quilt)" > debian/source/format + + { + echo "diff-ignore" + echo "extend-diff-ignore = .*" + } > debian/source/local-options + + # Add .config as a patch + mkdir -p debian/patches + { + echo "Subject: Add .config" + echo "Author: ${maintainer}" + echo + echo "--- /dev/null" + echo "+++ linux/.config" + diff -u /dev/null "${KCONFIG_CONFIG}" | tail -n +3 + } > debian/patches/config.patch + echo config.patch > debian/patches/series + + "${srctree}/scripts/package/gen-diff-patch" debian/patches/diff.patch + if [ -s debian/patches/diff.patch ]; then + sed -i " + 1iSubject: Add local diff + 1iAuthor: ${maintainer} + 1i + " debian/patches/diff.patch + + echo diff.patch >> debian/patches/series + else + rm -f debian/patches/diff.patch + fi +} + rm -rf debian +mkdir debian + +email=${DEBEMAIL-$EMAIL} + +# use email string directly if it contains <email> +if echo "${email}" | grep -q '<.*>'; then + maintainer=${email} +else + # or construct the maintainer string + user=${KBUILD_BUILD_USER-$(id -nu)} + name=${DEBFULLNAME-${user}} + if [ -z "${email}" ]; then + buildhost=${KBUILD_BUILD_HOST-$(hostname -f 2>/dev/null || hostname)} + email="${user}@${buildhost}" + fi + maintainer="${name} <${email}>" +fi + +if [ "$1" = --need-source ]; then + gen_source +fi # Some variables and settings used throughout the script version=$KERNELRELEASE @@ -104,22 +163,6 @@ fi debarch= set_debarch -email=${DEBEMAIL-$EMAIL} - -# use email string directly if it contains <email> -if echo $email | grep -q '<.*>'; then - maintainer=$email -else - # or construct the maintainer string - user=${KBUILD_BUILD_USER-$(id -nu)} - name=${DEBFULLNAME-$user} - if [ -z "$email" ]; then - buildhost=${KBUILD_BUILD_HOST-$(hostname -f 2>/dev/null || hostname)} - email="$user@$buildhost" - fi - maintainer="$name <$email>" -fi - # Try to determine distribution if [ -n "$KDEB_CHANGELOG_DIST" ]; then distribution=$KDEB_CHANGELOG_DIST @@ -132,34 +175,6 @@ else echo >&2 "Install lsb-release or set \$KDEB_CHANGELOG_DIST explicitly" fi -mkdir -p debian/source/ -echo "3.0 (quilt)" > debian/source/format - -{ - echo "diff-ignore" - echo "extend-diff-ignore = .*" -} > debian/source/local-options - -# Add .config as a patch -mkdir -p debian/patches -{ - echo "Subject: Add .config" - echo "Author: ${maintainer}" - echo - echo "--- /dev/null" - echo "+++ linux/.config" - diff -u /dev/null "${KCONFIG_CONFIG}" | tail -n +3 -} > debian/patches/config -echo config > debian/patches/series - -$(dirname $0)/gen-diff-patch debian/patches/diff.patch debian/patches/untracked.patch -if [ -f debian/patches/diff.patch ]; then - echo diff.patch >> debian/patches/series -fi -if [ -f debian/patches/untracked.patch ]; then - echo untracked.patch >> debian/patches/series -fi - echo $debarch > debian/arch extra_build_depends=", $(if_enabled_echo CONFIG_UNWINDER_ORC libelf-dev:native)" extra_build_depends="$extra_build_depends, $(if_enabled_echo CONFIG_SYSTEM_TRUSTED_KEYRING libssl-dev:native)" diff --git a/scripts/package/mkspec b/scripts/package/mkspec index b7d1dc28a5d6..fc8ad3fbc0a9 100755 --- a/scripts/package/mkspec +++ b/scripts/package/mkspec @@ -19,8 +19,7 @@ else mkdir -p rpmbuild/SOURCES cp linux.tar.gz rpmbuild/SOURCES cp "${KCONFIG_CONFIG}" rpmbuild/SOURCES/config - $(dirname $0)/gen-diff-patch rpmbuild/SOURCES/diff.patch rpmbuild/SOURCES/untracked.patch - touch rpmbuild/SOURCES/diff.patch rpmbuild/SOURCES/untracked.patch + "${srctree}/scripts/package/gen-diff-patch" rpmbuild/SOURCES/diff.patch fi if grep -q CONFIG_MODULES=y include/config/auto.conf; then @@ -56,7 +55,6 @@ sed -e '/^DEL/d' -e 's/^\t*//' <<EOF $S Source0: linux.tar.gz $S Source1: config $S Source2: diff.patch -$S Source3: untracked.patch Provides: $PROVIDES $S BuildRequires: bc binutils bison dwarves $S BuildRequires: (elfutils-libelf-devel or libelf-devel) flex @@ -94,12 +92,7 @@ $S$M $S %prep $S %setup -q -n linux $S cp %{SOURCE1} .config -$S if [ -s %{SOURCE2} ]; then -$S patch -p1 < %{SOURCE2} -$S fi -$S if [ -s %{SOURCE3} ]; then -$S patch -p1 < %{SOURCE3} -$S fi +$S patch -p1 < %{SOURCE2} $S $S %build $S $MAKE %{?_smp_mflags} KERNELRELEASE=$KERNELRELEASE KBUILD_BUILD_VERSION=%{release} diff --git a/sound/firewire/tascam/tascam-stream.c b/sound/firewire/tascam/tascam-stream.c index 53e094cc411f..dfe783d01d7d 100644 --- a/sound/firewire/tascam/tascam-stream.c +++ b/sound/firewire/tascam/tascam-stream.c @@ -490,7 +490,7 @@ int snd_tscm_stream_start_duplex(struct snd_tscm *tscm, unsigned int rate) // packet is important for media clock recovery. err = amdtp_domain_start(&tscm->domain, tx_init_skip_cycles, true, true); if (err < 0) - return err; + goto error; if (!amdtp_domain_wait_ready(&tscm->domain, READY_TIMEOUT_MS)) { err = -ETIMEDOUT; diff --git a/sound/i2c/cs8427.c b/sound/i2c/cs8427.c index 65012af6a36e..f58b14b49045 100644 --- a/sound/i2c/cs8427.c +++ b/sound/i2c/cs8427.c @@ -561,10 +561,13 @@ int snd_cs8427_iec958_active(struct snd_i2c_device *cs8427, int active) if (snd_BUG_ON(!cs8427)) return -ENXIO; chip = cs8427->private_data; - if (active) + if (active) { memcpy(chip->playback.pcm_status, chip->playback.def_status, 24); - chip->playback.pcm_ctl->vd[0].access &= ~SNDRV_CTL_ELEM_ACCESS_INACTIVE; + chip->playback.pcm_ctl->vd[0].access &= ~SNDRV_CTL_ELEM_ACCESS_INACTIVE; + } else { + chip->playback.pcm_ctl->vd[0].access |= SNDRV_CTL_ELEM_ACCESS_INACTIVE; + } snd_ctl_notify(cs8427->bus->card, SNDRV_CTL_EVENT_MASK_VALUE | SNDRV_CTL_EVENT_MASK_INFO, &chip->playback.pcm_ctl->id); diff --git a/sound/pci/emu10k1/emupcm.c b/sound/pci/emu10k1/emupcm.c index 48af77ae8020..6ec394fb1846 100644 --- a/sound/pci/emu10k1/emupcm.c +++ b/sound/pci/emu10k1/emupcm.c @@ -1236,7 +1236,7 @@ static int snd_emu10k1_capture_mic_close(struct snd_pcm_substream *substream) { struct snd_emu10k1 *emu = snd_pcm_substream_chip(substream); - emu->capture_interrupt = NULL; + emu->capture_mic_interrupt = NULL; emu->pcm_capture_mic_substream = NULL; return 0; } @@ -1344,7 +1344,7 @@ static int snd_emu10k1_capture_efx_close(struct snd_pcm_substream *substream) { struct snd_emu10k1 *emu = snd_pcm_substream_chip(substream); - emu->capture_interrupt = NULL; + emu->capture_efx_interrupt = NULL; emu->pcm_capture_efx_substream = NULL; return 0; } @@ -1781,17 +1781,21 @@ int snd_emu10k1_pcm_efx(struct snd_emu10k1 *emu, int device) struct snd_kcontrol *kctl; int err; - err = snd_pcm_new(emu->card, "emu10k1 efx", device, 8, 1, &pcm); + err = snd_pcm_new(emu->card, "emu10k1 efx", device, emu->audigy ? 0 : 8, 1, &pcm); if (err < 0) return err; pcm->private_data = emu; - snd_pcm_set_ops(pcm, SNDRV_PCM_STREAM_PLAYBACK, &snd_emu10k1_fx8010_playback_ops); + if (!emu->audigy) + snd_pcm_set_ops(pcm, SNDRV_PCM_STREAM_PLAYBACK, &snd_emu10k1_fx8010_playback_ops); snd_pcm_set_ops(pcm, SNDRV_PCM_STREAM_CAPTURE, &snd_emu10k1_capture_efx_ops); pcm->info_flags = 0; - strcpy(pcm->name, "Multichannel Capture/PT Playback"); + if (emu->audigy) + strcpy(pcm->name, "Multichannel Capture"); + else + strcpy(pcm->name, "Multichannel Capture/PT Playback"); emu->pcm_efx = pcm; /* EFX capture - record the "FXBUS2" channels, by default we connect the EXTINs diff --git a/sound/pci/hda/patch_hdmi.c b/sound/pci/hda/patch_hdmi.c index 4ffa3a59f419..5c6980394dce 100644 --- a/sound/pci/hda/patch_hdmi.c +++ b/sound/pci/hda/patch_hdmi.c @@ -4604,7 +4604,7 @@ HDA_CODEC_ENTRY(0x80862814, "DG1 HDMI", patch_i915_tgl_hdmi), HDA_CODEC_ENTRY(0x80862815, "Alderlake HDMI", patch_i915_tgl_hdmi), HDA_CODEC_ENTRY(0x80862816, "Rocketlake HDMI", patch_i915_tgl_hdmi), HDA_CODEC_ENTRY(0x80862818, "Raptorlake HDMI", patch_i915_tgl_hdmi), -HDA_CODEC_ENTRY(0x80862819, "DG2 HDMI", patch_i915_adlp_hdmi), +HDA_CODEC_ENTRY(0x80862819, "DG2 HDMI", patch_i915_tgl_hdmi), HDA_CODEC_ENTRY(0x8086281a, "Jasperlake HDMI", patch_i915_icl_hdmi), HDA_CODEC_ENTRY(0x8086281b, "Elkhartlake HDMI", patch_i915_icl_hdmi), HDA_CODEC_ENTRY(0x8086281c, "Alderlake-P HDMI", patch_i915_adlp_hdmi), diff --git a/sound/pci/hda/patch_realtek.c b/sound/pci/hda/patch_realtek.c index 26187f5d56b5..3b9f077a227f 100644 --- a/sound/pci/hda/patch_realtek.c +++ b/sound/pci/hda/patch_realtek.c @@ -6960,6 +6960,8 @@ enum { ALC269_FIXUP_DELL_M101Z, ALC269_FIXUP_SKU_IGNORE, ALC269_FIXUP_ASUS_G73JW, + ALC269_FIXUP_ASUS_N7601ZM_PINS, + ALC269_FIXUP_ASUS_N7601ZM, ALC269_FIXUP_LENOVO_EAPD, ALC275_FIXUP_SONY_HWEQ, ALC275_FIXUP_SONY_DISABLE_AAMIX, @@ -7256,6 +7258,29 @@ static const struct hda_fixup alc269_fixups[] = { { } } }, + [ALC269_FIXUP_ASUS_N7601ZM_PINS] = { + .type = HDA_FIXUP_PINS, + .v.pins = (const struct hda_pintbl[]) { + { 0x19, 0x03A11050 }, + { 0x1a, 0x03A11C30 }, + { 0x21, 0x03211420 }, + { } + } + }, + [ALC269_FIXUP_ASUS_N7601ZM] = { + .type = HDA_FIXUP_VERBS, + .v.verbs = (const struct hda_verb[]) { + {0x20, AC_VERB_SET_COEF_INDEX, 0x62}, + {0x20, AC_VERB_SET_PROC_COEF, 0xa007}, + {0x20, AC_VERB_SET_COEF_INDEX, 0x10}, + {0x20, AC_VERB_SET_PROC_COEF, 0x8420}, + {0x20, AC_VERB_SET_COEF_INDEX, 0x0f}, + {0x20, AC_VERB_SET_PROC_COEF, 0x7774}, + { } + }, + .chained = true, + .chain_id = ALC269_FIXUP_ASUS_N7601ZM_PINS, + }, [ALC269_FIXUP_LENOVO_EAPD] = { .type = HDA_FIXUP_VERBS, .v.verbs = (const struct hda_verb[]) { @@ -9466,6 +9491,7 @@ static const struct snd_pci_quirk alc269_fixup_tbl[] = { SND_PCI_QUIRK(0x1043, 0x1271, "ASUS X430UN", ALC256_FIXUP_ASUS_MIC_NO_PRESENCE), SND_PCI_QUIRK(0x1043, 0x1290, "ASUS X441SA", ALC233_FIXUP_EAPD_COEF_AND_MIC_NO_PRESENCE), SND_PCI_QUIRK(0x1043, 0x12a0, "ASUS X441UV", ALC233_FIXUP_EAPD_COEF_AND_MIC_NO_PRESENCE), + SND_PCI_QUIRK(0x1043, 0x12a3, "Asus N7691ZM", ALC269_FIXUP_ASUS_N7601ZM), SND_PCI_QUIRK(0x1043, 0x12af, "ASUS UX582ZS", ALC245_FIXUP_CS35L41_SPI_2), SND_PCI_QUIRK(0x1043, 0x12e0, "ASUS X541SA", ALC256_FIXUP_ASUS_MIC), SND_PCI_QUIRK(0x1043, 0x12f0, "ASUS X541UV", ALC256_FIXUP_ASUS_MIC), @@ -9663,6 +9689,9 @@ static const struct snd_pci_quirk alc269_fixup_tbl[] = { SND_PCI_QUIRK(0x17aa, 0x22f1, "Thinkpad", ALC287_FIXUP_CS35L41_I2C_2), SND_PCI_QUIRK(0x17aa, 0x22f2, "Thinkpad", ALC287_FIXUP_CS35L41_I2C_2), SND_PCI_QUIRK(0x17aa, 0x22f3, "Thinkpad", ALC287_FIXUP_CS35L41_I2C_2), + SND_PCI_QUIRK(0x17aa, 0x2318, "Thinkpad Z13 Gen2", ALC287_FIXUP_CS35L41_I2C_2), + SND_PCI_QUIRK(0x17aa, 0x2319, "Thinkpad Z16 Gen2", ALC287_FIXUP_CS35L41_I2C_2), + SND_PCI_QUIRK(0x17aa, 0x231a, "Thinkpad Z16 Gen2", ALC287_FIXUP_CS35L41_I2C_2), SND_PCI_QUIRK(0x17aa, 0x30bb, "ThinkCentre AIO", ALC233_FIXUP_LENOVO_LINE2_MIC_HOTKEY), SND_PCI_QUIRK(0x17aa, 0x30e2, "ThinkCentre AIO", ALC233_FIXUP_LENOVO_LINE2_MIC_HOTKEY), SND_PCI_QUIRK(0x17aa, 0x310c, "ThinkCentre Station", ALC294_FIXUP_LENOVO_MIC_LOCATION), diff --git a/sound/pci/hda/patch_sigmatel.c b/sound/pci/hda/patch_sigmatel.c index a794a01a68ca..61258b0aac8d 100644 --- a/sound/pci/hda/patch_sigmatel.c +++ b/sound/pci/hda/patch_sigmatel.c @@ -1707,6 +1707,7 @@ static const struct snd_pci_quirk stac925x_fixup_tbl[] = { }; static const struct hda_pintbl ref92hd73xx_pin_configs[] = { + // Port A-H { 0x0a, 0x02214030 }, { 0x0b, 0x02a19040 }, { 0x0c, 0x01a19020 }, @@ -1715,9 +1716,12 @@ static const struct hda_pintbl ref92hd73xx_pin_configs[] = { { 0x0f, 0x01014010 }, { 0x10, 0x01014020 }, { 0x11, 0x01014030 }, + // CD in { 0x12, 0x02319040 }, + // Digial Mic ins { 0x13, 0x90a000f0 }, { 0x14, 0x90a000f0 }, + // Digital outs { 0x22, 0x01452050 }, { 0x23, 0x01452050 }, {} @@ -1758,6 +1762,7 @@ static const struct hda_pintbl alienware_m17x_pin_configs[] = { }; static const struct hda_pintbl intel_dg45id_pin_configs[] = { + // Analog outputs { 0x0a, 0x02214230 }, { 0x0b, 0x02A19240 }, { 0x0c, 0x01013214 }, @@ -1765,6 +1770,9 @@ static const struct hda_pintbl intel_dg45id_pin_configs[] = { { 0x0e, 0x01A19250 }, { 0x0f, 0x01011212 }, { 0x10, 0x01016211 }, + // Digital output + { 0x22, 0x01451380 }, + { 0x23, 0x40f000f0 }, {} }; @@ -1955,6 +1963,8 @@ static const struct snd_pci_quirk stac92hd73xx_fixup_tbl[] = { "DFI LanParty", STAC_92HD73XX_REF), SND_PCI_QUIRK(PCI_VENDOR_ID_DFI, 0x3101, "DFI LanParty", STAC_92HD73XX_REF), + SND_PCI_QUIRK(PCI_VENDOR_ID_INTEL, 0x5001, + "Intel DP45SG", STAC_92HD73XX_INTEL), SND_PCI_QUIRK(PCI_VENDOR_ID_INTEL, 0x5002, "Intel DG45ID", STAC_92HD73XX_INTEL), SND_PCI_QUIRK(PCI_VENDOR_ID_INTEL, 0x5003, diff --git a/tools/Makefile b/tools/Makefile index e497875fc7e3..37e9f6804832 100644 --- a/tools/Makefile +++ b/tools/Makefile @@ -39,7 +39,7 @@ help: @echo ' turbostat - Intel CPU idle stats and freq reporting tool' @echo ' usb - USB testing tools' @echo ' virtio - vhost test module' - @echo ' vm - misc vm tools' + @echo ' mm - misc mm tools' @echo ' wmi - WMI interface examples' @echo ' x86_energy_perf_policy - Intel energy policy tool' @echo '' @@ -69,7 +69,7 @@ acpi: FORCE cpupower: FORCE $(call descend,power/$@) -cgroup counter firewire hv guest bootconfig spi usb virtio vm bpf iio gpio objtool leds wmi pci firmware debugging tracing: FORCE +cgroup counter firewire hv guest bootconfig spi usb virtio mm bpf iio gpio objtool leds wmi pci firmware debugging tracing: FORCE $(call descend,$@) bpf/%: FORCE @@ -118,7 +118,7 @@ kvm_stat: FORCE all: acpi cgroup counter cpupower gpio hv firewire \ perf selftests bootconfig spi turbostat usb \ - virtio vm bpf x86_energy_perf_policy \ + virtio mm bpf x86_energy_perf_policy \ tmon freefall iio objtool kvm_stat wmi \ pci debugging tracing thermal thermometer thermal-engine @@ -128,7 +128,7 @@ acpi_install: cpupower_install: $(call descend,power/$(@:_install=),install) -cgroup_install counter_install firewire_install gpio_install hv_install iio_install perf_install bootconfig_install spi_install usb_install virtio_install vm_install bpf_install objtool_install wmi_install pci_install debugging_install tracing_install: +cgroup_install counter_install firewire_install gpio_install hv_install iio_install perf_install bootconfig_install spi_install usb_install virtio_install mm_install bpf_install objtool_install wmi_install pci_install debugging_install tracing_install: $(call descend,$(@:_install=),install) selftests_install: @@ -158,7 +158,7 @@ kvm_stat_install: install: acpi_install cgroup_install counter_install cpupower_install gpio_install \ hv_install firewire_install iio_install \ perf_install selftests_install turbostat_install usb_install \ - virtio_install vm_install bpf_install x86_energy_perf_policy_install \ + virtio_install mm_install bpf_install x86_energy_perf_policy_install \ tmon_install freefall_install objtool_install kvm_stat_install \ wmi_install pci_install debugging_install intel-speed-select_install \ tracing_install thermometer_install thermal-engine_install @@ -169,7 +169,7 @@ acpi_clean: cpupower_clean: $(call descend,power/cpupower,clean) -cgroup_clean counter_clean hv_clean firewire_clean bootconfig_clean spi_clean usb_clean virtio_clean vm_clean wmi_clean bpf_clean iio_clean gpio_clean objtool_clean leds_clean pci_clean firmware_clean debugging_clean tracing_clean: +cgroup_clean counter_clean hv_clean firewire_clean bootconfig_clean spi_clean usb_clean virtio_clean mm_clean wmi_clean bpf_clean iio_clean gpio_clean objtool_clean leds_clean pci_clean firmware_clean debugging_clean tracing_clean: $(call descend,$(@:_clean=),clean) libapi_clean: @@ -211,7 +211,7 @@ build_clean: clean: acpi_clean cgroup_clean counter_clean cpupower_clean hv_clean firewire_clean \ perf_clean selftests_clean turbostat_clean bootconfig_clean spi_clean usb_clean virtio_clean \ - vm_clean bpf_clean iio_clean x86_energy_perf_policy_clean tmon_clean \ + mm_clean bpf_clean iio_clean x86_energy_perf_policy_clean tmon_clean \ freefall_clean build_clean libbpf_clean libsubcmd_clean \ gpio_clean objtool_clean leds_clean wmi_clean pci_clean firmware_clean debugging_clean \ intel-speed-select_clean tracing_clean thermal_clean thermometer_clean thermal-engine_clean diff --git a/tools/arch/loongarch/include/uapi/asm/bitsperlong.h b/tools/arch/loongarch/include/uapi/asm/bitsperlong.h index d4e32b3d4843..00b4ba1e5cdf 100644 --- a/tools/arch/loongarch/include/uapi/asm/bitsperlong.h +++ b/tools/arch/loongarch/include/uapi/asm/bitsperlong.h @@ -2,7 +2,7 @@ #ifndef __ASM_LOONGARCH_BITSPERLONG_H #define __ASM_LOONGARCH_BITSPERLONG_H -#define __BITS_PER_LONG (__SIZEOF_POINTER__ * 8) +#define __BITS_PER_LONG (__SIZEOF_LONG__ * 8) #include <asm-generic/bitsperlong.h> diff --git a/tools/bpf/bpftool/Documentation/bpftool-struct_ops.rst b/tools/bpf/bpftool/Documentation/bpftool-struct_ops.rst index ee53a122c0c7..8022b5321dbe 100644 --- a/tools/bpf/bpftool/Documentation/bpftool-struct_ops.rst +++ b/tools/bpf/bpftool/Documentation/bpftool-struct_ops.rst @@ -26,7 +26,7 @@ STRUCT_OPS COMMANDS | **bpftool** **struct_ops { show | list }** [*STRUCT_OPS_MAP*] | **bpftool** **struct_ops dump** [*STRUCT_OPS_MAP*] -| **bpftool** **struct_ops register** *OBJ* +| **bpftool** **struct_ops register** *OBJ* [*LINK_DIR*] | **bpftool** **struct_ops unregister** *STRUCT_OPS_MAP* | **bpftool** **struct_ops help** | @@ -51,10 +51,14 @@ DESCRIPTION for the given struct_ops. Otherwise, it dumps all struct_ops currently existing in the system. - **bpftool struct_ops register** *OBJ* + **bpftool struct_ops register** *OBJ* [*LINK_DIR*] Register bpf struct_ops from *OBJ*. All struct_ops under - the ELF section ".struct_ops" will be registered to - its kernel subsystem. + the ELF section ".struct_ops" and ".struct_ops.link" will + be registered to its kernel subsystem. For each + struct_ops in the ".struct_ops.link" section, a link + will be created. You can give *LINK_DIR* to provide a + directory path where these links will be pinned with the + same name as their corresponding map name. **bpftool struct_ops unregister** *STRUCT_OPS_MAP* Unregister the *STRUCT_OPS_MAP* from the kernel subsystem. diff --git a/tools/bpf/bpftool/btf_dumper.c b/tools/bpf/bpftool/btf_dumper.c index 6c5e0e82da22..294de231db99 100644 --- a/tools/bpf/bpftool/btf_dumper.c +++ b/tools/bpf/bpftool/btf_dumper.c @@ -835,7 +835,7 @@ static void dotlabel_puts(const char *s) case '|': case ' ': putchar('\\'); - __fallthrough; + /* fallthrough */ default: putchar(*s); } diff --git a/tools/bpf/bpftool/common.c b/tools/bpf/bpftool/common.c index 5a73ccf14332..1360c82ae732 100644 --- a/tools/bpf/bpftool/common.c +++ b/tools/bpf/bpftool/common.c @@ -1091,3 +1091,17 @@ const char *bpf_attach_type_input_str(enum bpf_attach_type t) default: return libbpf_bpf_attach_type_str(t); } } + +int pathname_concat(char *buf, int buf_sz, const char *path, + const char *name) +{ + int len; + + len = snprintf(buf, buf_sz, "%s/%s", path, name); + if (len < 0) + return -EINVAL; + if (len >= buf_sz) + return -ENAMETOOLONG; + + return 0; +} diff --git a/tools/bpf/bpftool/link.c b/tools/bpf/bpftool/link.c index f985b79cca27..d98dbc50cf4c 100644 --- a/tools/bpf/bpftool/link.c +++ b/tools/bpf/bpftool/link.c @@ -3,6 +3,8 @@ #include <errno.h> #include <linux/err.h> +#include <linux/netfilter.h> +#include <linux/netfilter_arp.h> #include <net/if.h> #include <stdio.h> #include <unistd.h> @@ -135,6 +137,18 @@ static void show_iter_json(struct bpf_link_info *info, json_writer_t *wtr) } } +void netfilter_dump_json(const struct bpf_link_info *info, json_writer_t *wtr) +{ + jsonw_uint_field(json_wtr, "pf", + info->netfilter.pf); + jsonw_uint_field(json_wtr, "hook", + info->netfilter.hooknum); + jsonw_int_field(json_wtr, "prio", + info->netfilter.priority); + jsonw_uint_field(json_wtr, "flags", + info->netfilter.flags); +} + static int get_prog_info(int prog_id, struct bpf_prog_info *info) { __u32 len = sizeof(*info); @@ -195,6 +209,10 @@ static int show_link_close_json(int fd, struct bpf_link_info *info) info->netns.netns_ino); show_link_attach_type_json(info->netns.attach_type, json_wtr); break; + case BPF_LINK_TYPE_NETFILTER: + netfilter_dump_json(info, json_wtr); + break; + default: break; } @@ -263,6 +281,68 @@ static void show_iter_plain(struct bpf_link_info *info) } } +static const char * const pf2name[] = { + [NFPROTO_INET] = "inet", + [NFPROTO_IPV4] = "ip", + [NFPROTO_ARP] = "arp", + [NFPROTO_NETDEV] = "netdev", + [NFPROTO_BRIDGE] = "bridge", + [NFPROTO_IPV6] = "ip6", +}; + +static const char * const inethook2name[] = { + [NF_INET_PRE_ROUTING] = "prerouting", + [NF_INET_LOCAL_IN] = "input", + [NF_INET_FORWARD] = "forward", + [NF_INET_LOCAL_OUT] = "output", + [NF_INET_POST_ROUTING] = "postrouting", +}; + +static const char * const arphook2name[] = { + [NF_ARP_IN] = "input", + [NF_ARP_OUT] = "output", +}; + +void netfilter_dump_plain(const struct bpf_link_info *info) +{ + const char *hookname = NULL, *pfname = NULL; + unsigned int hook = info->netfilter.hooknum; + unsigned int pf = info->netfilter.pf; + + if (pf < ARRAY_SIZE(pf2name)) + pfname = pf2name[pf]; + + switch (pf) { + case NFPROTO_BRIDGE: /* bridge shares numbers with enum nf_inet_hooks */ + case NFPROTO_IPV4: + case NFPROTO_IPV6: + case NFPROTO_INET: + if (hook < ARRAY_SIZE(inethook2name)) + hookname = inethook2name[hook]; + break; + case NFPROTO_ARP: + if (hook < ARRAY_SIZE(arphook2name)) + hookname = arphook2name[hook]; + default: + break; + } + + if (pfname) + printf("\n\t%s", pfname); + else + printf("\n\tpf: %d", pf); + + if (hookname) + printf(" %s", hookname); + else + printf(", hook %u,", hook); + + printf(" prio %d", info->netfilter.priority); + + if (info->netfilter.flags) + printf(" flags 0x%x", info->netfilter.flags); +} + static int show_link_close_plain(int fd, struct bpf_link_info *info) { struct bpf_prog_info prog_info; @@ -301,6 +381,9 @@ static int show_link_close_plain(int fd, struct bpf_link_info *info) printf("\n\tnetns_ino %u ", info->netns.netns_ino); show_link_attach_type_plain(info->netns.attach_type); break; + case BPF_LINK_TYPE_NETFILTER: + netfilter_dump_plain(info); + break; default: break; } diff --git a/tools/bpf/bpftool/main.h b/tools/bpf/bpftool/main.h index 00d11ca6d3f2..a49534d7eafa 100644 --- a/tools/bpf/bpftool/main.h +++ b/tools/bpf/bpftool/main.h @@ -264,4 +264,10 @@ static inline bool hashmap__empty(struct hashmap *map) return map ? hashmap__size(map) == 0 : true; } +int pathname_concat(char *buf, int buf_sz, const char *path, + const char *name); + +/* print netfilter bpf_link info */ +void netfilter_dump_plain(const struct bpf_link_info *info); +void netfilter_dump_json(const struct bpf_link_info *info, json_writer_t *wtr); #endif diff --git a/tools/bpf/bpftool/net.c b/tools/bpf/bpftool/net.c index c40e44c938ae..26a49965bf71 100644 --- a/tools/bpf/bpftool/net.c +++ b/tools/bpf/bpftool/net.c @@ -647,6 +647,108 @@ static int do_detach(int argc, char **argv) return 0; } +static int netfilter_link_compar(const void *a, const void *b) +{ + const struct bpf_link_info *nfa = a; + const struct bpf_link_info *nfb = b; + int delta; + + delta = nfa->netfilter.pf - nfb->netfilter.pf; + if (delta) + return delta; + + delta = nfa->netfilter.hooknum - nfb->netfilter.hooknum; + if (delta) + return delta; + + if (nfa->netfilter.priority < nfb->netfilter.priority) + return -1; + if (nfa->netfilter.priority > nfb->netfilter.priority) + return 1; + + return nfa->netfilter.flags - nfb->netfilter.flags; +} + +static void show_link_netfilter(void) +{ + unsigned int nf_link_len = 0, nf_link_count = 0; + struct bpf_link_info *nf_link_info = NULL; + __u32 id = 0; + + while (true) { + struct bpf_link_info info; + int fd, err; + __u32 len; + + err = bpf_link_get_next_id(id, &id); + if (err) { + if (errno == ENOENT) + break; + p_err("can't get next link: %s (id %d)", strerror(errno), id); + break; + } + + fd = bpf_link_get_fd_by_id(id); + if (fd < 0) { + p_err("can't get link by id (%u): %s", id, strerror(errno)); + continue; + } + + memset(&info, 0, sizeof(info)); + len = sizeof(info); + + err = bpf_link_get_info_by_fd(fd, &info, &len); + + close(fd); + + if (err) { + p_err("can't get link info for fd %d: %s", fd, strerror(errno)); + continue; + } + + if (info.type != BPF_LINK_TYPE_NETFILTER) + continue; + + if (nf_link_count >= nf_link_len) { + static const unsigned int max_link_count = INT_MAX / sizeof(info); + struct bpf_link_info *expand; + + if (nf_link_count > max_link_count) { + p_err("cannot handle more than %u links\n", max_link_count); + break; + } + + nf_link_len += 16; + + expand = realloc(nf_link_info, nf_link_len * sizeof(info)); + if (!expand) { + p_err("realloc: %s", strerror(errno)); + break; + } + + nf_link_info = expand; + } + + nf_link_info[nf_link_count] = info; + nf_link_count++; + } + + qsort(nf_link_info, nf_link_count, sizeof(*nf_link_info), netfilter_link_compar); + + for (id = 0; id < nf_link_count; id++) { + NET_START_OBJECT; + if (json_output) + netfilter_dump_json(&nf_link_info[id], json_wtr); + else + netfilter_dump_plain(&nf_link_info[id]); + + NET_DUMP_UINT("id", " prog_id %u", nf_link_info[id].prog_id); + NET_END_OBJECT; + } + + free(nf_link_info); +} + static int do_show(int argc, char **argv) { struct bpf_attach_info attach_info = {}; @@ -701,6 +803,10 @@ static int do_show(int argc, char **argv) NET_DUMP_UINT("id", "id %u", attach_info.flow_dissector_id); NET_END_ARRAY("\n"); + NET_START_ARRAY("netfilter", "%s:\n"); + show_link_netfilter(); + NET_END_ARRAY("\n"); + NET_END_OBJECT; if (json_output) jsonw_end_array(json_wtr); diff --git a/tools/bpf/bpftool/prog.c b/tools/bpf/bpftool/prog.c index e5b613a7974c..91b6075b2db3 100644 --- a/tools/bpf/bpftool/prog.c +++ b/tools/bpf/bpftool/prog.c @@ -1476,19 +1476,6 @@ auto_attach_program(struct bpf_program *prog, const char *path) return err; } -static int pathname_concat(char *buf, size_t buf_sz, const char *path, const char *name) -{ - int len; - - len = snprintf(buf, buf_sz, "%s/%s", path, name); - if (len < 0) - return -EINVAL; - if ((size_t)len >= buf_sz) - return -ENAMETOOLONG; - - return 0; -} - static int auto_attach_programs(struct bpf_object *obj, const char *path) { diff --git a/tools/bpf/bpftool/struct_ops.c b/tools/bpf/bpftool/struct_ops.c index b389f4830e11..57c3da70aa31 100644 --- a/tools/bpf/bpftool/struct_ops.c +++ b/tools/bpf/bpftool/struct_ops.c @@ -475,21 +475,44 @@ static int do_unregister(int argc, char **argv) return cmd_retval(&res, true); } +static int pin_link(struct bpf_link *link, const char *pindir, + const char *name) +{ + char pinfile[PATH_MAX]; + int err; + + err = pathname_concat(pinfile, sizeof(pinfile), pindir, name); + if (err) + return -1; + + return bpf_link__pin(link, pinfile); +} + static int do_register(int argc, char **argv) { LIBBPF_OPTS(bpf_object_open_opts, open_opts); + __u32 link_info_len = sizeof(struct bpf_link_info); + struct bpf_link_info link_info = {}; struct bpf_map_info info = {}; __u32 info_len = sizeof(info); int nr_errs = 0, nr_maps = 0; + const char *linkdir = NULL; struct bpf_object *obj; struct bpf_link *link; struct bpf_map *map; const char *file; - if (argc != 1) + if (argc != 1 && argc != 2) usage(); file = GET_ARG(); + if (argc == 1) + linkdir = GET_ARG(); + + if (linkdir && mount_bpffs_for_pin(linkdir)) { + p_err("can't mount bpffs for pinning"); + return -1; + } if (verifier_logs) /* log_level1 + log_level2 + stats, but not stable UAPI */ @@ -519,21 +542,44 @@ static int do_register(int argc, char **argv) } nr_maps++; - bpf_link__disconnect(link); - bpf_link__destroy(link); - - if (!bpf_map_get_info_by_fd(bpf_map__fd(map), &info, - &info_len)) - p_info("Registered %s %s id %u", - get_kern_struct_ops_name(&info), - bpf_map__name(map), - info.id); - else + if (bpf_map_get_info_by_fd(bpf_map__fd(map), &info, + &info_len)) { /* Not p_err. The struct_ops was attached * successfully. */ p_info("Registered %s but can't find id: %s", bpf_map__name(map), strerror(errno)); + goto clean_link; + } + if (!(bpf_map__map_flags(map) & BPF_F_LINK)) { + p_info("Registered %s %s id %u", + get_kern_struct_ops_name(&info), + info.name, + info.id); + goto clean_link; + } + if (bpf_link_get_info_by_fd(bpf_link__fd(link), + &link_info, + &link_info_len)) { + p_err("Registered %s but can't find link id: %s", + bpf_map__name(map), strerror(errno)); + nr_errs++; + goto clean_link; + } + if (linkdir && pin_link(link, linkdir, info.name)) { + p_err("can't pin link %u for %s: %s", + link_info.id, info.name, + strerror(errno)); + nr_errs++; + goto clean_link; + } + p_info("Registered %s %s map id %u link id %u", + get_kern_struct_ops_name(&info), + info.name, info.id, link_info.id); + +clean_link: + bpf_link__disconnect(link); + bpf_link__destroy(link); } bpf_object__close(obj); @@ -562,7 +608,7 @@ static int do_help(int argc, char **argv) fprintf(stderr, "Usage: %1$s %2$s { show | list } [STRUCT_OPS_MAP]\n" " %1$s %2$s dump [STRUCT_OPS_MAP]\n" - " %1$s %2$s register OBJ\n" + " %1$s %2$s register OBJ [LINK_DIR]\n" " %1$s %2$s unregister STRUCT_OPS_MAP\n" " %1$s %2$s help\n" "\n" diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 3823100b7934..1bb11a6ee667 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -986,6 +986,7 @@ enum bpf_prog_type { BPF_PROG_TYPE_LSM, BPF_PROG_TYPE_SK_LOOKUP, BPF_PROG_TYPE_SYSCALL, /* a program that can execute syscalls */ + BPF_PROG_TYPE_NETFILTER, }; enum bpf_attach_type { @@ -1050,6 +1051,7 @@ enum bpf_link_type { BPF_LINK_TYPE_PERF_EVENT = 7, BPF_LINK_TYPE_KPROBE_MULTI = 8, BPF_LINK_TYPE_STRUCT_OPS = 9, + BPF_LINK_TYPE_NETFILTER = 10, MAX_BPF_LINK_TYPE, }; @@ -1560,6 +1562,12 @@ union bpf_attr { */ __u64 cookie; } tracing; + struct { + __u32 pf; + __u32 hooknum; + __s32 priority; + __u32 flags; + } netfilter; }; } link_create; @@ -6410,6 +6418,12 @@ struct bpf_link_info { struct { __u32 map_id; } struct_ops; + struct { + __u32 pf; + __u32 hooknum; + __s32 priority; + __u32 flags; + } netfilter; }; } __attribute__((aligned(8))); @@ -6985,6 +6999,10 @@ struct bpf_rb_node { __u64 :64; } __attribute__((aligned(8))); +struct bpf_refcount { + __u32 :32; +} __attribute__((aligned(4))); + struct bpf_sysctl { __u32 write; /* Sysctl is being read (= 0) or written (= 1). * Allows 1,2,4-byte read, but no write. diff --git a/tools/lib/bpf/bpf_helpers.h b/tools/lib/bpf/bpf_helpers.h index e7e1a8acc299..929a3baca8ef 100644 --- a/tools/lib/bpf/bpf_helpers.h +++ b/tools/lib/bpf/bpf_helpers.h @@ -291,4 +291,107 @@ enum libbpf_tristate { /* Helper macro to print out debug messages */ #define bpf_printk(fmt, args...) ___bpf_pick_printk(args)(fmt, ##args) +struct bpf_iter_num; + +extern int bpf_iter_num_new(struct bpf_iter_num *it, int start, int end) __weak __ksym; +extern int *bpf_iter_num_next(struct bpf_iter_num *it) __weak __ksym; +extern void bpf_iter_num_destroy(struct bpf_iter_num *it) __weak __ksym; + +#ifndef bpf_for_each +/* bpf_for_each(iter_type, cur_elem, args...) provides generic construct for + * using BPF open-coded iterators without having to write mundane explicit + * low-level loop logic. Instead, it provides for()-like generic construct + * that can be used pretty naturally. E.g., for some hypothetical cgroup + * iterator, you'd write: + * + * struct cgroup *cg, *parent_cg = <...>; + * + * bpf_for_each(cgroup, cg, parent_cg, CG_ITER_CHILDREN) { + * bpf_printk("Child cgroup id = %d", cg->cgroup_id); + * if (cg->cgroup_id == 123) + * break; + * } + * + * I.e., it looks almost like high-level for each loop in other languages, + * supports continue/break, and is verifiable by BPF verifier. + * + * For iterating integers, the difference betwen bpf_for_each(num, i, N, M) + * and bpf_for(i, N, M) is in that bpf_for() provides additional proof to + * verifier that i is in [N, M) range, and in bpf_for_each() case i is `int + * *`, not just `int`. So for integers bpf_for() is more convenient. + * + * Note: this macro relies on C99 feature of allowing to declare variables + * inside for() loop, bound to for() loop lifetime. It also utilizes GCC + * extension: __attribute__((cleanup(<func>))), supported by both GCC and + * Clang. + */ +#define bpf_for_each(type, cur, args...) for ( \ + /* initialize and define destructor */ \ + struct bpf_iter_##type ___it __attribute__((aligned(8), /* enforce, just in case */, \ + cleanup(bpf_iter_##type##_destroy))), \ + /* ___p pointer is just to call bpf_iter_##type##_new() *once* to init ___it */ \ + *___p __attribute__((unused)) = ( \ + bpf_iter_##type##_new(&___it, ##args), \ + /* this is a workaround for Clang bug: it currently doesn't emit BTF */ \ + /* for bpf_iter_##type##_destroy() when used from cleanup() attribute */ \ + (void)bpf_iter_##type##_destroy, (void *)0); \ + /* iteration and termination check */ \ + (((cur) = bpf_iter_##type##_next(&___it))); \ +) +#endif /* bpf_for_each */ + +#ifndef bpf_for +/* bpf_for(i, start, end) implements a for()-like looping construct that sets + * provided integer variable *i* to values starting from *start* through, + * but not including, *end*. It also proves to BPF verifier that *i* belongs + * to range [start, end), so this can be used for accessing arrays without + * extra checks. + * + * Note: *start* and *end* are assumed to be expressions with no side effects + * and whose values do not change throughout bpf_for() loop execution. They do + * not have to be statically known or constant, though. + * + * Note: similarly to bpf_for_each(), it relies on C99 feature of declaring for() + * loop bound variables and cleanup attribute, supported by GCC and Clang. + */ +#define bpf_for(i, start, end) for ( \ + /* initialize and define destructor */ \ + struct bpf_iter_num ___it __attribute__((aligned(8), /* enforce, just in case */ \ + cleanup(bpf_iter_num_destroy))), \ + /* ___p pointer is necessary to call bpf_iter_num_new() *once* to init ___it */ \ + *___p __attribute__((unused)) = ( \ + bpf_iter_num_new(&___it, (start), (end)), \ + /* this is a workaround for Clang bug: it currently doesn't emit BTF */ \ + /* for bpf_iter_num_destroy() when used from cleanup() attribute */ \ + (void)bpf_iter_num_destroy, (void *)0); \ + ({ \ + /* iteration step */ \ + int *___t = bpf_iter_num_next(&___it); \ + /* termination and bounds check */ \ + (___t && ((i) = *___t, (i) >= (start) && (i) < (end))); \ + }); \ +) +#endif /* bpf_for */ + +#ifndef bpf_repeat +/* bpf_repeat(N) performs N iterations without exposing iteration number + * + * Note: similarly to bpf_for_each(), it relies on C99 feature of declaring for() + * loop bound variables and cleanup attribute, supported by GCC and Clang. + */ +#define bpf_repeat(N) for ( \ + /* initialize and define destructor */ \ + struct bpf_iter_num ___it __attribute__((aligned(8), /* enforce, just in case */ \ + cleanup(bpf_iter_num_destroy))), \ + /* ___p pointer is necessary to call bpf_iter_num_new() *once* to init ___it */ \ + *___p __attribute__((unused)) = ( \ + bpf_iter_num_new(&___it, 0, (N)), \ + /* this is a workaround for Clang bug: it currently doesn't emit BTF */ \ + /* for bpf_iter_num_destroy() when used from cleanup() attribute */ \ + (void)bpf_iter_num_destroy, (void *)0); \ + bpf_iter_num_next(&___it); \ + /* nothing here */ \ +) +#endif /* bpf_repeat */ + #endif diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 49cd304ae3bc..1cbacf9e71f3 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -130,6 +130,7 @@ static const char * const link_type_name[] = { [BPF_LINK_TYPE_PERF_EVENT] = "perf_event", [BPF_LINK_TYPE_KPROBE_MULTI] = "kprobe_multi", [BPF_LINK_TYPE_STRUCT_OPS] = "struct_ops", + [BPF_LINK_TYPE_NETFILTER] = "netfilter", }; static const char * const map_type_name[] = { @@ -201,6 +202,7 @@ static const char * const prog_type_name[] = { [BPF_PROG_TYPE_LSM] = "lsm", [BPF_PROG_TYPE_SK_LOOKUP] = "sk_lookup", [BPF_PROG_TYPE_SYSCALL] = "syscall", + [BPF_PROG_TYPE_NETFILTER] = "netfilter", }; static int __base_pr(enum libbpf_print_level level, const char *format, @@ -333,6 +335,7 @@ struct reloc_desc { struct { int map_idx; int sym_off; + int ext_idx; }; }; }; @@ -4042,7 +4045,7 @@ static int bpf_program__record_reloc(struct bpf_program *prog, else reloc_desc->type = RELO_EXTERN_LD64; reloc_desc->insn_idx = insn_idx; - reloc_desc->sym_off = i; /* sym_off stores extern index */ + reloc_desc->ext_idx = i; return 0; } @@ -5811,8 +5814,8 @@ out: } /* base map load ldimm64 special constant, used also for log fixup logic */ -#define MAP_LDIMM64_POISON_BASE 2001000000 -#define MAP_LDIMM64_POISON_PFX "200100" +#define POISON_LDIMM64_MAP_BASE 2001000000 +#define POISON_LDIMM64_MAP_PFX "200100" static void poison_map_ldimm64(struct bpf_program *prog, int relo_idx, int insn_idx, struct bpf_insn *insn, @@ -5834,12 +5837,36 @@ static void poison_map_ldimm64(struct bpf_program *prog, int relo_idx, * invalid func unknown#2001000123 * where lower 123 is map index into obj->maps[] array */ - insn->imm = MAP_LDIMM64_POISON_BASE + map_idx; + insn->imm = POISON_LDIMM64_MAP_BASE + map_idx; insn++; } } +/* unresolved kfunc call special constant, used also for log fixup logic */ +#define POISON_CALL_KFUNC_BASE 2002000000 +#define POISON_CALL_KFUNC_PFX "2002" + +static void poison_kfunc_call(struct bpf_program *prog, int relo_idx, + int insn_idx, struct bpf_insn *insn, + int ext_idx, const struct extern_desc *ext) +{ + pr_debug("prog '%s': relo #%d: poisoning insn #%d that calls kfunc '%s'\n", + prog->name, relo_idx, insn_idx, ext->name); + + /* we turn kfunc call into invalid helper call with identifiable constant */ + insn->code = BPF_JMP | BPF_CALL; + insn->dst_reg = 0; + insn->src_reg = 0; + insn->off = 0; + /* if this instruction is reachable (not a dead code), + * verifier will complain with something like: + * invalid func unknown#2001000123 + * where lower 123 is extern index into obj->externs[] array + */ + insn->imm = POISON_CALL_KFUNC_BASE + ext_idx; +} + /* Relocate data references within program code: * - map references; * - global variable references; @@ -5885,7 +5912,7 @@ bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog) } break; case RELO_EXTERN_LD64: - ext = &obj->externs[relo->sym_off]; + ext = &obj->externs[relo->ext_idx]; if (ext->type == EXT_KCFG) { if (obj->gen_loader) { insn[0].src_reg = BPF_PSEUDO_MAP_IDX_VALUE; @@ -5907,14 +5934,14 @@ bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog) } break; case RELO_EXTERN_CALL: - ext = &obj->externs[relo->sym_off]; + ext = &obj->externs[relo->ext_idx]; insn[0].src_reg = BPF_PSEUDO_KFUNC_CALL; if (ext->is_set) { insn[0].imm = ext->ksym.kernel_btf_id; insn[0].off = ext->ksym.btf_fd_idx; - } else { /* unresolved weak kfunc */ - insn[0].imm = 0; - insn[0].off = 0; + } else { /* unresolved weak kfunc call */ + poison_kfunc_call(prog, i, relo->insn_idx, insn, + relo->ext_idx, ext); } break; case RELO_SUBPROG_ADDR: @@ -7022,13 +7049,13 @@ static void fixup_log_missing_map_load(struct bpf_program *prog, char *buf, size_t buf_sz, size_t log_sz, char *line1, char *line2, char *line3) { - /* Expected log for failed and not properly guarded CO-RE relocation: + /* Expected log for failed and not properly guarded map reference: * line1 -> 123: (85) call unknown#2001000345 * line2 -> invalid func unknown#2001000345 * line3 -> <anything else or end of buffer> * * "123" is the index of the instruction that was poisoned. - * "345" in "2001000345" are map index in obj->maps to fetch map name. + * "345" in "2001000345" is a map index in obj->maps to fetch map name. */ struct bpf_object *obj = prog->obj; const struct bpf_map *map; @@ -7038,7 +7065,7 @@ static void fixup_log_missing_map_load(struct bpf_program *prog, if (sscanf(line1, "%d: (%*d) call unknown#%d\n", &insn_idx, &map_idx) != 2) return; - map_idx -= MAP_LDIMM64_POISON_BASE; + map_idx -= POISON_LDIMM64_MAP_BASE; if (map_idx < 0 || map_idx >= obj->nr_maps) return; map = &obj->maps[map_idx]; @@ -7051,6 +7078,39 @@ static void fixup_log_missing_map_load(struct bpf_program *prog, patch_log(buf, buf_sz, log_sz, line1, line3 - line1, patch); } +static void fixup_log_missing_kfunc_call(struct bpf_program *prog, + char *buf, size_t buf_sz, size_t log_sz, + char *line1, char *line2, char *line3) +{ + /* Expected log for failed and not properly guarded kfunc call: + * line1 -> 123: (85) call unknown#2002000345 + * line2 -> invalid func unknown#2002000345 + * line3 -> <anything else or end of buffer> + * + * "123" is the index of the instruction that was poisoned. + * "345" in "2002000345" is an extern index in obj->externs to fetch kfunc name. + */ + struct bpf_object *obj = prog->obj; + const struct extern_desc *ext; + int insn_idx, ext_idx; + char patch[128]; + + if (sscanf(line1, "%d: (%*d) call unknown#%d\n", &insn_idx, &ext_idx) != 2) + return; + + ext_idx -= POISON_CALL_KFUNC_BASE; + if (ext_idx < 0 || ext_idx >= obj->nr_extern) + return; + ext = &obj->externs[ext_idx]; + + snprintf(patch, sizeof(patch), + "%d: <invalid kfunc call>\n" + "kfunc '%s' is referenced but wasn't resolved\n", + insn_idx, ext->name); + + patch_log(buf, buf_sz, log_sz, line1, line3 - line1, patch); +} + static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_sz) { /* look for familiar error patterns in last N lines of the log */ @@ -7070,23 +7130,33 @@ static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s if (!cur_line) return; - /* failed CO-RE relocation case */ if (str_has_pfx(cur_line, "invalid func unknown#195896080\n")) { prev_line = find_prev_line(buf, cur_line); if (!prev_line) continue; + /* failed CO-RE relocation case */ fixup_log_failed_core_relo(prog, buf, buf_sz, log_sz, prev_line, cur_line, next_line); return; - } else if (str_has_pfx(cur_line, "invalid func unknown#"MAP_LDIMM64_POISON_PFX)) { + } else if (str_has_pfx(cur_line, "invalid func unknown#"POISON_LDIMM64_MAP_PFX)) { prev_line = find_prev_line(buf, cur_line); if (!prev_line) continue; + /* reference to uncreated BPF map */ fixup_log_missing_map_load(prog, buf, buf_sz, log_sz, prev_line, cur_line, next_line); return; + } else if (str_has_pfx(cur_line, "invalid func unknown#"POISON_CALL_KFUNC_PFX)) { + prev_line = find_prev_line(buf, cur_line); + if (!prev_line) + continue; + + /* reference to unresolved kfunc */ + fixup_log_missing_kfunc_call(prog, buf, buf_sz, log_sz, + prev_line, cur_line, next_line); + return; } } } @@ -7098,7 +7168,7 @@ static int bpf_program_record_relos(struct bpf_program *prog) for (i = 0; i < prog->nr_reloc; i++) { struct reloc_desc *relo = &prog->reloc_desc[i]; - struct extern_desc *ext = &obj->externs[relo->sym_off]; + struct extern_desc *ext = &obj->externs[relo->ext_idx]; int kind; switch (relo->type) { @@ -7536,8 +7606,9 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj, ret = bpf_core_types_are_compat(obj->btf, local_func_proto_id, kern_btf, kfunc_proto_id); if (ret <= 0) { - pr_warn("extern (func ksym) '%s': func_proto [%d] incompatible with kernel [%d]\n", - ext->name, local_func_proto_id, kfunc_proto_id); + pr_warn("extern (func ksym) '%s': func_proto [%d] incompatible with %s [%d]\n", + ext->name, local_func_proto_id, + mod_btf ? mod_btf->name : "vmlinux", kfunc_proto_id); return -EINVAL; } @@ -7571,8 +7642,8 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj, * {kernel_btf_id, kernel_btf_obj_fd} -> fixup ld_imm64. */ ext->ksym.kernel_btf_obj_fd = mod_btf ? mod_btf->fd : 0; - pr_debug("extern (func ksym) '%s': resolved to kernel [%d]\n", - ext->name, kfunc_id); + pr_debug("extern (func ksym) '%s': resolved to %s [%d]\n", + ext->name, mod_btf ? mod_btf->name : "vmlinux", kfunc_id); return 0; } @@ -8641,6 +8712,7 @@ static const struct bpf_sec_def section_defs[] = { SEC_DEF("struct_ops+", STRUCT_OPS, 0, SEC_NONE), SEC_DEF("struct_ops.s+", STRUCT_OPS, 0, SEC_SLEEPABLE), SEC_DEF("sk_lookup", SK_LOOKUP, BPF_SK_LOOKUP, SEC_ATTACHABLE), + SEC_DEF("netfilter", NETFILTER, 0, SEC_NONE), }; static size_t custom_sec_def_cnt; diff --git a/tools/lib/bpf/libbpf_probes.c b/tools/lib/bpf/libbpf_probes.c index 4f3bc968ff8e..6065f408a59c 100644 --- a/tools/lib/bpf/libbpf_probes.c +++ b/tools/lib/bpf/libbpf_probes.c @@ -180,6 +180,7 @@ static int probe_prog_load(enum bpf_prog_type prog_type, case BPF_PROG_TYPE_SK_REUSEPORT: case BPF_PROG_TYPE_FLOW_DISSECTOR: case BPF_PROG_TYPE_CGROUP_SYSCTL: + case BPF_PROG_TYPE_NETFILTER: break; default: return -EOPNOTSUPP; diff --git a/tools/mm/page_owner_sort.c b/tools/mm/page_owner_sort.c index 7c2ac124cdc8..99798894b879 100644 --- a/tools/mm/page_owner_sort.c +++ b/tools/mm/page_owner_sort.c @@ -857,7 +857,7 @@ int main(int argc, char **argv) if (cull & CULL_PID || filter & FILTER_PID) fprintf(fout, ", PID %d", list[i].pid); if (cull & CULL_TGID || filter & FILTER_TGID) - fprintf(fout, ", TGID %d", list[i].pid); + fprintf(fout, ", TGID %d", list[i].tgid); if (cull & CULL_COMM || filter & FILTER_COMM) fprintf(fout, ", task_comm_name: %s", list[i].comm); if (cull & CULL_ALLOCATOR) { diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing/selftests/bpf/bpf_experimental.h index dbd2c729781a..209811b1993a 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -14,7 +14,8 @@ * type ID of a struct in program BTF. * * The 'local_type_id' parameter must be a known constant. - * The 'meta' parameter is a hidden argument that is ignored. + * The 'meta' parameter is rewritten by the verifier, no need for BPF + * program to set it. * Returns * A pointer to an object of the type corresponding to the passed in * 'local_type_id', or NULL on failure. @@ -28,7 +29,8 @@ extern void *bpf_obj_new_impl(__u64 local_type_id, void *meta) __ksym; * Free an allocated object. All fields of the object that require * destruction will be destructed before the storage is freed. * - * The 'meta' parameter is a hidden argument that is ignored. + * The 'meta' parameter is rewritten by the verifier, no need for BPF + * program to set it. * Returns * Void. */ @@ -38,18 +40,50 @@ extern void bpf_obj_drop_impl(void *kptr, void *meta) __ksym; #define bpf_obj_drop(kptr) bpf_obj_drop_impl(kptr, NULL) /* Description + * Increment the refcount on a refcounted local kptr, turning the + * non-owning reference input into an owning reference in the process. + * + * The 'meta' parameter is rewritten by the verifier, no need for BPF + * program to set it. + * Returns + * An owning reference to the object pointed to by 'kptr' + */ +extern void *bpf_refcount_acquire_impl(void *kptr, void *meta) __ksym; + +/* Convenience macro to wrap over bpf_refcount_acquire_impl */ +#define bpf_refcount_acquire(kptr) bpf_refcount_acquire_impl(kptr, NULL) + +/* Description * Add a new entry to the beginning of the BPF linked list. + * + * The 'meta' and 'off' parameters are rewritten by the verifier, no need + * for BPF programs to set them * Returns - * Void. + * 0 if the node was successfully added + * -EINVAL if the node wasn't added because it's already in a list */ -extern void bpf_list_push_front(struct bpf_list_head *head, struct bpf_list_node *node) __ksym; +extern int bpf_list_push_front_impl(struct bpf_list_head *head, + struct bpf_list_node *node, + void *meta, __u64 off) __ksym; + +/* Convenience macro to wrap over bpf_list_push_front_impl */ +#define bpf_list_push_front(head, node) bpf_list_push_front_impl(head, node, NULL, 0) /* Description * Add a new entry to the end of the BPF linked list. + * + * The 'meta' and 'off' parameters are rewritten by the verifier, no need + * for BPF programs to set them * Returns - * Void. + * 0 if the node was successfully added + * -EINVAL if the node wasn't added because it's already in a list */ -extern void bpf_list_push_back(struct bpf_list_head *head, struct bpf_list_node *node) __ksym; +extern int bpf_list_push_back_impl(struct bpf_list_head *head, + struct bpf_list_node *node, + void *meta, __u64 off) __ksym; + +/* Convenience macro to wrap over bpf_list_push_back_impl */ +#define bpf_list_push_back(head, node) bpf_list_push_back_impl(head, node, NULL, 0) /* Description * Remove the entry at the beginning of the BPF linked list. @@ -75,11 +109,19 @@ extern struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root, /* Description * Add 'node' to rbtree with root 'root' using comparator 'less' + * + * The 'meta' and 'off' parameters are rewritten by the verifier, no need + * for BPF programs to set them * Returns - * Nothing + * 0 if the node was successfully added + * -EINVAL if the node wasn't added because it's already in a tree */ -extern void bpf_rbtree_add(struct bpf_rb_root *root, struct bpf_rb_node *node, - bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b)) __ksym; +extern int bpf_rbtree_add_impl(struct bpf_rb_root *root, struct bpf_rb_node *node, + bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b), + void *meta, __u64 off) __ksym; + +/* Convenience macro to wrap over bpf_rbtree_add_impl */ +#define bpf_rbtree_add(head, node, less) bpf_rbtree_add_impl(head, node, less, NULL, 0) /* Description * Return the first (leftmost) node in input tree diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c index fe847ebfb731..52785ba671e6 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c @@ -28,6 +28,11 @@ struct bpf_testmod_struct_arg_2 { long b; }; +struct bpf_testmod_struct_arg_3 { + int a; + int b[]; +}; + __diag_push(); __diag_ignore_all("-Wmissing-prototypes", "Global functions as their definitions will be in bpf_testmod.ko BTF"); @@ -63,6 +68,12 @@ bpf_testmod_test_struct_arg_5(void) { return bpf_testmod_test_struct_arg_result; } +noinline int +bpf_testmod_test_struct_arg_6(struct bpf_testmod_struct_arg_3 *a) { + bpf_testmod_test_struct_arg_result = a->b[0]; + return bpf_testmod_test_struct_arg_result; +} + __bpf_kfunc void bpf_testmod_test_mod_kfunc(int i) { @@ -195,6 +206,7 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj, }; struct bpf_testmod_struct_arg_1 struct_arg1 = {10}; struct bpf_testmod_struct_arg_2 struct_arg2 = {2, 3}; + struct bpf_testmod_struct_arg_3 *struct_arg3; int i = 1; while (bpf_testmod_return_ptr(i)) @@ -206,6 +218,14 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj, (void)bpf_testmod_test_struct_arg_4(struct_arg1, 1, 2, 3, struct_arg2); (void)bpf_testmod_test_struct_arg_5(); + struct_arg3 = kmalloc((sizeof(struct bpf_testmod_struct_arg_3) + + sizeof(int)), GFP_KERNEL); + if (struct_arg3 != NULL) { + struct_arg3->b[0] = 1; + (void)bpf_testmod_test_struct_arg_6(struct_arg3); + kfree(struct_arg3); + } + /* This is always true. Use the check to make sure the compiler * doesn't remove bpf_testmod_loop_test. */ diff --git a/tools/testing/selftests/bpf/prog_tests/access_variable_array.c b/tools/testing/selftests/bpf/prog_tests/access_variable_array.c new file mode 100644 index 000000000000..08131782437c --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/access_variable_array.c @@ -0,0 +1,16 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2022 Bytedance */ + +#include <test_progs.h> +#include "test_access_variable_array.skel.h" + +void test_access_variable_array(void) +{ + struct test_access_variable_array *skel; + + skel = test_access_variable_array__open_and_load(); + if (!ASSERT_OK_PTR(skel, "test_access_variable_array__open_and_load")) + return; + + test_access_variable_array__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/prog_tests/linked_list.c b/tools/testing/selftests/bpf/prog_tests/linked_list.c index 0ed8132ce1c3..f63309fd0e28 100644 --- a/tools/testing/selftests/bpf/prog_tests/linked_list.c +++ b/tools/testing/selftests/bpf/prog_tests/linked_list.c @@ -84,11 +84,11 @@ static struct { { "double_push_back", "arg#1 expected pointer to allocated object" }, { "no_node_value_type", "bpf_list_node not found at offset=0" }, { "incorrect_value_type", - "operation on bpf_list_head expects arg#1 bpf_list_node at offset=0 in struct foo, " + "operation on bpf_list_head expects arg#1 bpf_list_node at offset=40 in struct foo, " "but arg is at offset=0 in struct bar" }, { "incorrect_node_var_off", "variable ptr_ access var_off=(0x0; 0xffffffff) disallowed" }, - { "incorrect_node_off1", "bpf_list_node not found at offset=1" }, - { "incorrect_node_off2", "arg#1 offset=40, but expected bpf_list_node at offset=0 in struct foo" }, + { "incorrect_node_off1", "bpf_list_node not found at offset=41" }, + { "incorrect_node_off2", "arg#1 offset=0, but expected bpf_list_node at offset=40 in struct foo" }, { "no_head_type", "bpf_list_head not found at offset=0" }, { "incorrect_head_var_off1", "R1 doesn't have constant offset" }, { "incorrect_head_var_off2", "variable ptr_ access var_off=(0x0; 0xffffffff) disallowed" }, @@ -266,6 +266,59 @@ end: return NULL; } +static void list_and_rb_node_same_struct(bool refcount_field) +{ + int bpf_rb_node_btf_id, bpf_refcount_btf_id, foo_btf_id; + struct btf *btf; + int id, err; + + btf = init_btf(); + if (!ASSERT_OK_PTR(btf, "init_btf")) + return; + + bpf_rb_node_btf_id = btf__add_struct(btf, "bpf_rb_node", 24); + if (!ASSERT_GT(bpf_rb_node_btf_id, 0, "btf__add_struct bpf_rb_node")) + return; + + if (refcount_field) { + bpf_refcount_btf_id = btf__add_struct(btf, "bpf_refcount", 4); + if (!ASSERT_GT(bpf_refcount_btf_id, 0, "btf__add_struct bpf_refcount")) + return; + } + + id = btf__add_struct(btf, "bar", refcount_field ? 44 : 40); + if (!ASSERT_GT(id, 0, "btf__add_struct bar")) + return; + err = btf__add_field(btf, "a", LIST_NODE, 0, 0); + if (!ASSERT_OK(err, "btf__add_field bar::a")) + return; + err = btf__add_field(btf, "c", bpf_rb_node_btf_id, 128, 0); + if (!ASSERT_OK(err, "btf__add_field bar::c")) + return; + if (refcount_field) { + err = btf__add_field(btf, "ref", bpf_refcount_btf_id, 320, 0); + if (!ASSERT_OK(err, "btf__add_field bar::ref")) + return; + } + + foo_btf_id = btf__add_struct(btf, "foo", 20); + if (!ASSERT_GT(foo_btf_id, 0, "btf__add_struct foo")) + return; + err = btf__add_field(btf, "a", LIST_HEAD, 0, 0); + if (!ASSERT_OK(err, "btf__add_field foo::a")) + return; + err = btf__add_field(btf, "b", SPIN_LOCK, 128, 0); + if (!ASSERT_OK(err, "btf__add_field foo::b")) + return; + id = btf__add_decl_tag(btf, "contains:bar:a", foo_btf_id, 0); + if (!ASSERT_GT(id, 0, "btf__add_decl_tag contains:bar:a")) + return; + + err = btf__load_into_kernel(btf); + ASSERT_EQ(err, refcount_field ? 0 : -EINVAL, "check btf"); + btf__free(btf); +} + static void test_btf(void) { struct btf *btf = NULL; @@ -717,39 +770,12 @@ static void test_btf(void) } while (test__start_subtest("btf: list_node and rb_node in same struct")) { - btf = init_btf(); - if (!ASSERT_OK_PTR(btf, "init_btf")) - break; - - id = btf__add_struct(btf, "bpf_rb_node", 24); - if (!ASSERT_EQ(id, 5, "btf__add_struct bpf_rb_node")) - break; - id = btf__add_struct(btf, "bar", 40); - if (!ASSERT_EQ(id, 6, "btf__add_struct bar")) - break; - err = btf__add_field(btf, "a", LIST_NODE, 0, 0); - if (!ASSERT_OK(err, "btf__add_field bar::a")) - break; - err = btf__add_field(btf, "c", 5, 128, 0); - if (!ASSERT_OK(err, "btf__add_field bar::c")) - break; - - id = btf__add_struct(btf, "foo", 20); - if (!ASSERT_EQ(id, 7, "btf__add_struct foo")) - break; - err = btf__add_field(btf, "a", LIST_HEAD, 0, 0); - if (!ASSERT_OK(err, "btf__add_field foo::a")) - break; - err = btf__add_field(btf, "b", SPIN_LOCK, 128, 0); - if (!ASSERT_OK(err, "btf__add_field foo::b")) - break; - id = btf__add_decl_tag(btf, "contains:bar:a", 7, 0); - if (!ASSERT_EQ(id, 8, "btf__add_decl_tag contains:bar:a")) - break; + list_and_rb_node_same_struct(true); + break; + } - err = btf__load_into_kernel(btf); - ASSERT_EQ(err, -EINVAL, "check btf"); - btf__free(btf); + while (test__start_subtest("btf: list_node and rb_node in same struct, no bpf_refcount")) { + list_and_rb_node_same_struct(false); break; } } diff --git a/tools/testing/selftests/bpf/prog_tests/log_fixup.c b/tools/testing/selftests/bpf/prog_tests/log_fixup.c index bc27170bdeb0..dba71d98a227 100644 --- a/tools/testing/selftests/bpf/prog_tests/log_fixup.c +++ b/tools/testing/selftests/bpf/prog_tests/log_fixup.c @@ -135,6 +135,35 @@ cleanup: test_log_fixup__destroy(skel); } +static void missing_kfunc(void) +{ + char log_buf[8 * 1024]; + struct test_log_fixup* skel; + int err; + + skel = test_log_fixup__open(); + if (!ASSERT_OK_PTR(skel, "skel_open")) + return; + + bpf_program__set_autoload(skel->progs.use_missing_kfunc, true); + bpf_program__set_log_buf(skel->progs.use_missing_kfunc, log_buf, sizeof(log_buf)); + + err = test_log_fixup__load(skel); + if (!ASSERT_ERR(err, "load_fail")) + goto cleanup; + + ASSERT_HAS_SUBSTR(log_buf, + "0: <invalid kfunc call>\n" + "kfunc 'bpf_nonexistent_kfunc' is referenced but wasn't resolved\n", + "log_buf"); + + if (env.verbosity > VERBOSE_NONE) + printf("LOG: \n=================\n%s=================\n", log_buf); + +cleanup: + test_log_fixup__destroy(skel); +} + void test_log_fixup(void) { if (test__start_subtest("bad_core_relo_trunc_none")) @@ -147,4 +176,6 @@ void test_log_fixup(void) bad_core_relo_subprog(); if (test__start_subtest("missing_map")) missing_map(); + if (test__start_subtest("missing_kfunc")) + missing_kfunc(); } diff --git a/tools/testing/selftests/bpf/prog_tests/rbtree.c b/tools/testing/selftests/bpf/prog_tests/rbtree.c index 156fa95c42f6..e9300c96607d 100644 --- a/tools/testing/selftests/bpf/prog_tests/rbtree.c +++ b/tools/testing/selftests/bpf/prog_tests/rbtree.c @@ -77,6 +77,29 @@ static void test_rbtree_first_and_remove(void) rbtree__destroy(skel); } +static void test_rbtree_api_release_aliasing(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, opts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 1, + ); + struct rbtree *skel; + int ret; + + skel = rbtree__open_and_load(); + if (!ASSERT_OK_PTR(skel, "rbtree__open_and_load")) + return; + + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.rbtree_api_release_aliasing), &opts); + ASSERT_OK(ret, "rbtree_api_release_aliasing"); + ASSERT_OK(opts.retval, "rbtree_api_release_aliasing retval"); + ASSERT_EQ(skel->data->first_data[0], 42, "rbtree_api_release_aliasing first rbtree_remove()"); + ASSERT_EQ(skel->data->first_data[1], -1, "rbtree_api_release_aliasing second rbtree_remove()"); + + rbtree__destroy(skel); +} + void test_rbtree_success(void) { if (test__start_subtest("rbtree_add_nodes")) @@ -85,6 +108,8 @@ void test_rbtree_success(void) test_rbtree_add_and_remove(); if (test__start_subtest("rbtree_first_and_remove")) test_rbtree_first_and_remove(); + if (test__start_subtest("rbtree_api_release_aliasing")) + test_rbtree_api_release_aliasing(); } #define BTF_FAIL_TEST(suffix) \ diff --git a/tools/testing/selftests/bpf/prog_tests/refcounted_kptr.c b/tools/testing/selftests/bpf/prog_tests/refcounted_kptr.c new file mode 100644 index 000000000000..2ab23832062d --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/refcounted_kptr.c @@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */ + +#include <test_progs.h> +#include <network_helpers.h> + +#include "refcounted_kptr.skel.h" +#include "refcounted_kptr_fail.skel.h" + +void test_refcounted_kptr(void) +{ + RUN_TESTS(refcounted_kptr); +} + +void test_refcounted_kptr_fail(void) +{ + RUN_TESTS(refcounted_kptr_fail); +} diff --git a/tools/testing/selftests/bpf/prog_tests/sockmap_listen.c b/tools/testing/selftests/bpf/prog_tests/sockmap_listen.c index 8f09e1ea3ba7..141c1e5944ee 100644 --- a/tools/testing/selftests/bpf/prog_tests/sockmap_listen.c +++ b/tools/testing/selftests/bpf/prog_tests/sockmap_listen.c @@ -20,6 +20,11 @@ #include <unistd.h> #include <linux/vm_sockets.h> +/* workaround for older vm_sockets.h */ +#ifndef VMADDR_CID_LOCAL +#define VMADDR_CID_LOCAL 1 +#endif + #include <bpf/bpf.h> #include <bpf/libbpf.h> diff --git a/tools/testing/selftests/bpf/prog_tests/sockopt_sk.c b/tools/testing/selftests/bpf/prog_tests/sockopt_sk.c index 60d952719d27..4512dd808c33 100644 --- a/tools/testing/selftests/bpf/prog_tests/sockopt_sk.c +++ b/tools/testing/selftests/bpf/prog_tests/sockopt_sk.c @@ -3,6 +3,7 @@ #include "cgroup_helpers.h" #include <linux/tcp.h> +#include <linux/netlink.h> #include "sockopt_sk.skel.h" #ifndef SOL_TCP @@ -183,6 +184,33 @@ static int getsetsockopt(void) goto err; } + /* optval=NULL case is handled correctly */ + + close(fd); + fd = socket(AF_NETLINK, SOCK_RAW, 0); + if (fd < 0) { + log_err("Failed to create AF_NETLINK socket"); + return -1; + } + + buf.u32 = 1; + optlen = sizeof(__u32); + err = setsockopt(fd, SOL_NETLINK, NETLINK_ADD_MEMBERSHIP, &buf, optlen); + if (err) { + log_err("Unexpected getsockopt(NETLINK_ADD_MEMBERSHIP) err=%d errno=%d", + err, errno); + goto err; + } + + optlen = 0; + err = getsockopt(fd, SOL_NETLINK, NETLINK_LIST_MEMBERSHIPS, NULL, &optlen); + if (err) { + log_err("Unexpected getsockopt(NETLINK_LIST_MEMBERSHIPS) err=%d errno=%d", + err, errno); + goto err; + } + ASSERT_EQ(optlen, 4, "Unexpected NETLINK_LIST_MEMBERSHIPS value"); + free(big_buf); close(fd); return 0; diff --git a/tools/testing/selftests/bpf/prog_tests/tracing_struct.c b/tools/testing/selftests/bpf/prog_tests/tracing_struct.c index 48dc9472e160..1c75a32186d6 100644 --- a/tools/testing/selftests/bpf/prog_tests/tracing_struct.c +++ b/tools/testing/selftests/bpf/prog_tests/tracing_struct.c @@ -53,6 +53,8 @@ static void test_fentry(void) ASSERT_EQ(skel->bss->t5_ret, 1, "t5 ret"); + ASSERT_EQ(skel->bss->t6, 1, "t6 ret"); + tracing_struct__detach(skel); destroy_skel: tracing_struct__destroy(skel); diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c index 73dff693d411..c8bab8b1a6a4 100644 --- a/tools/testing/selftests/bpf/prog_tests/verifier.c +++ b/tools/testing/selftests/bpf/prog_tests/verifier.c @@ -6,15 +6,21 @@ #include "verifier_and.skel.h" #include "verifier_array_access.skel.h" #include "verifier_basic_stack.skel.h" +#include "verifier_bounds.skel.h" #include "verifier_bounds_deduction.skel.h" #include "verifier_bounds_deduction_non_const.skel.h" #include "verifier_bounds_mix_sign_unsign.skel.h" +#include "verifier_bpf_get_stack.skel.h" +#include "verifier_btf_ctx_access.skel.h" #include "verifier_cfg.skel.h" #include "verifier_cgroup_inv_retcode.skel.h" #include "verifier_cgroup_skb.skel.h" #include "verifier_cgroup_storage.skel.h" #include "verifier_const_or.skel.h" +#include "verifier_ctx.skel.h" #include "verifier_ctx_sk_msg.skel.h" +#include "verifier_d_path.skel.h" +#include "verifier_direct_packet_access.skel.h" #include "verifier_direct_stack_access_wraparound.skel.h" #include "verifier_div0.skel.h" #include "verifier_div_overflow.skel.h" @@ -23,28 +29,56 @@ #include "verifier_helper_restricted.skel.h" #include "verifier_helper_value_access.skel.h" #include "verifier_int_ptr.skel.h" +#include "verifier_jeq_infer_not_null.skel.h" #include "verifier_ld_ind.skel.h" #include "verifier_leak_ptr.skel.h" +#include "verifier_loops1.skel.h" +#include "verifier_lwt.skel.h" +#include "verifier_map_in_map.skel.h" #include "verifier_map_ptr.skel.h" +#include "verifier_map_ptr_mixing.skel.h" #include "verifier_map_ret_val.skel.h" #include "verifier_masking.skel.h" #include "verifier_meta_access.skel.h" +#include "verifier_netfilter_ctx.skel.h" +#include "verifier_netfilter_retcode.skel.h" #include "verifier_raw_stack.skel.h" #include "verifier_raw_tp_writable.skel.h" +#include "verifier_reg_equal.skel.h" +#include "verifier_ref_tracking.skel.h" +#include "verifier_regalloc.skel.h" #include "verifier_ringbuf.skel.h" +#include "verifier_runtime_jit.skel.h" +#include "verifier_search_pruning.skel.h" +#include "verifier_sock.skel.h" #include "verifier_spill_fill.skel.h" +#include "verifier_spin_lock.skel.h" #include "verifier_stack_ptr.skel.h" +#include "verifier_subreg.skel.h" #include "verifier_uninit.skel.h" +#include "verifier_unpriv.skel.h" +#include "verifier_unpriv_perf.skel.h" #include "verifier_value_adj_spill.skel.h" #include "verifier_value.skel.h" +#include "verifier_value_illegal_alu.skel.h" #include "verifier_value_or_null.skel.h" +#include "verifier_value_ptr_arith.skel.h" #include "verifier_var_off.skel.h" #include "verifier_xadd.skel.h" #include "verifier_xdp.skel.h" #include "verifier_xdp_direct_packet_access.skel.h" +#define MAX_ENTRIES 11 + +struct test_val { + unsigned int index; + int foo[MAX_ENTRIES]; +}; + __maybe_unused -static void run_tests_aux(const char *skel_name, skel_elf_bytes_fn elf_bytes_factory) +static void run_tests_aux(const char *skel_name, + skel_elf_bytes_fn elf_bytes_factory, + pre_execution_cb pre_execution_cb) { struct test_loader tester = {}; __u64 old_caps; @@ -57,6 +91,7 @@ static void run_tests_aux(const char *skel_name, skel_elf_bytes_fn elf_bytes_fac return; } + test_loader__set_pre_execution_cb(&tester, pre_execution_cb); test_loader__run_subtests(&tester, skel_name, elf_bytes_factory); test_loader_fini(&tester); @@ -65,20 +100,25 @@ static void run_tests_aux(const char *skel_name, skel_elf_bytes_fn elf_bytes_fac PRINT_FAIL("failed to restore CAP_SYS_ADMIN: %i, %s\n", err, strerror(err)); } -#define RUN(skel) run_tests_aux(#skel, skel##__elf_bytes) +#define RUN(skel) run_tests_aux(#skel, skel##__elf_bytes, NULL) void test_verifier_and(void) { RUN(verifier_and); } -void test_verifier_array_access(void) { RUN(verifier_array_access); } void test_verifier_basic_stack(void) { RUN(verifier_basic_stack); } +void test_verifier_bounds(void) { RUN(verifier_bounds); } void test_verifier_bounds_deduction(void) { RUN(verifier_bounds_deduction); } void test_verifier_bounds_deduction_non_const(void) { RUN(verifier_bounds_deduction_non_const); } void test_verifier_bounds_mix_sign_unsign(void) { RUN(verifier_bounds_mix_sign_unsign); } +void test_verifier_bpf_get_stack(void) { RUN(verifier_bpf_get_stack); } +void test_verifier_btf_ctx_access(void) { RUN(verifier_btf_ctx_access); } void test_verifier_cfg(void) { RUN(verifier_cfg); } void test_verifier_cgroup_inv_retcode(void) { RUN(verifier_cgroup_inv_retcode); } void test_verifier_cgroup_skb(void) { RUN(verifier_cgroup_skb); } void test_verifier_cgroup_storage(void) { RUN(verifier_cgroup_storage); } void test_verifier_const_or(void) { RUN(verifier_const_or); } +void test_verifier_ctx(void) { RUN(verifier_ctx); } void test_verifier_ctx_sk_msg(void) { RUN(verifier_ctx_sk_msg); } +void test_verifier_d_path(void) { RUN(verifier_d_path); } +void test_verifier_direct_packet_access(void) { RUN(verifier_direct_packet_access); } void test_verifier_direct_stack_access_wraparound(void) { RUN(verifier_direct_stack_access_wraparound); } void test_verifier_div0(void) { RUN(verifier_div0); } void test_verifier_div_overflow(void) { RUN(verifier_div_overflow); } @@ -87,22 +127,88 @@ void test_verifier_helper_packet_access(void) { RUN(verifier_helper_packet_acces void test_verifier_helper_restricted(void) { RUN(verifier_helper_restricted); } void test_verifier_helper_value_access(void) { RUN(verifier_helper_value_access); } void test_verifier_int_ptr(void) { RUN(verifier_int_ptr); } +void test_verifier_jeq_infer_not_null(void) { RUN(verifier_jeq_infer_not_null); } void test_verifier_ld_ind(void) { RUN(verifier_ld_ind); } void test_verifier_leak_ptr(void) { RUN(verifier_leak_ptr); } +void test_verifier_loops1(void) { RUN(verifier_loops1); } +void test_verifier_lwt(void) { RUN(verifier_lwt); } +void test_verifier_map_in_map(void) { RUN(verifier_map_in_map); } void test_verifier_map_ptr(void) { RUN(verifier_map_ptr); } +void test_verifier_map_ptr_mixing(void) { RUN(verifier_map_ptr_mixing); } void test_verifier_map_ret_val(void) { RUN(verifier_map_ret_val); } void test_verifier_masking(void) { RUN(verifier_masking); } void test_verifier_meta_access(void) { RUN(verifier_meta_access); } +void test_verifier_netfilter_ctx(void) { RUN(verifier_netfilter_ctx); } +void test_verifier_netfilter_retcode(void) { RUN(verifier_netfilter_retcode); } void test_verifier_raw_stack(void) { RUN(verifier_raw_stack); } void test_verifier_raw_tp_writable(void) { RUN(verifier_raw_tp_writable); } +void test_verifier_reg_equal(void) { RUN(verifier_reg_equal); } +void test_verifier_ref_tracking(void) { RUN(verifier_ref_tracking); } +void test_verifier_regalloc(void) { RUN(verifier_regalloc); } void test_verifier_ringbuf(void) { RUN(verifier_ringbuf); } +void test_verifier_runtime_jit(void) { RUN(verifier_runtime_jit); } +void test_verifier_search_pruning(void) { RUN(verifier_search_pruning); } +void test_verifier_sock(void) { RUN(verifier_sock); } void test_verifier_spill_fill(void) { RUN(verifier_spill_fill); } +void test_verifier_spin_lock(void) { RUN(verifier_spin_lock); } void test_verifier_stack_ptr(void) { RUN(verifier_stack_ptr); } +void test_verifier_subreg(void) { RUN(verifier_subreg); } void test_verifier_uninit(void) { RUN(verifier_uninit); } +void test_verifier_unpriv(void) { RUN(verifier_unpriv); } +void test_verifier_unpriv_perf(void) { RUN(verifier_unpriv_perf); } void test_verifier_value_adj_spill(void) { RUN(verifier_value_adj_spill); } void test_verifier_value(void) { RUN(verifier_value); } +void test_verifier_value_illegal_alu(void) { RUN(verifier_value_illegal_alu); } void test_verifier_value_or_null(void) { RUN(verifier_value_or_null); } void test_verifier_var_off(void) { RUN(verifier_var_off); } void test_verifier_xadd(void) { RUN(verifier_xadd); } void test_verifier_xdp(void) { RUN(verifier_xdp); } void test_verifier_xdp_direct_packet_access(void) { RUN(verifier_xdp_direct_packet_access); } + +static int init_test_val_map(struct bpf_object *obj, char *map_name) +{ + struct test_val value = { + .index = (6 + 1) * sizeof(int), + .foo[6] = 0xabcdef12, + }; + struct bpf_map *map; + int err, key = 0; + + map = bpf_object__find_map_by_name(obj, map_name); + if (!map) { + PRINT_FAIL("Can't find map '%s'\n", map_name); + return -EINVAL; + } + + err = bpf_map_update_elem(bpf_map__fd(map), &key, &value, 0); + if (err) { + PRINT_FAIL("Error while updating map '%s': %d\n", map_name, err); + return err; + } + + return 0; +} + +static int init_array_access_maps(struct bpf_object *obj) +{ + return init_test_val_map(obj, "map_array_ro"); +} + +void test_verifier_array_access(void) +{ + run_tests_aux("verifier_array_access", + verifier_array_access__elf_bytes, + init_array_access_maps); +} + +static int init_value_ptr_arith_maps(struct bpf_object *obj) +{ + return init_test_val_map(obj, "map_array_48b"); +} + +void test_verifier_value_ptr_arith(void) +{ + run_tests_aux("verifier_value_ptr_arith", + verifier_value_ptr_arith__elf_bytes, + init_value_ptr_arith_maps); +} diff --git a/tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c b/tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c index c94eb63b7b77..498d3bdaa4b0 100644 --- a/tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c +++ b/tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c @@ -175,8 +175,8 @@ void test_xdp_do_redirect(void) goto out; /* Enable GRO */ - SYS("ethtool -K veth_src gro on"); - SYS("ethtool -K veth_dst gro on"); + SYS(out, "ethtool -K veth_src gro on"); + SYS(out, "ethtool -K veth_dst gro on"); err = bpf_xdp_query(ifindex_src, XDP_FLAGS_DRV_MODE, &query_opts); if (!ASSERT_OK(err, "veth_src bpf_xdp_query gro on")) diff --git a/tools/testing/selftests/bpf/progs/bpf_misc.h b/tools/testing/selftests/bpf/progs/bpf_misc.h index 6e3b4903c541..d3c1217ba79a 100644 --- a/tools/testing/selftests/bpf/progs/bpf_misc.h +++ b/tools/testing/selftests/bpf/progs/bpf_misc.h @@ -53,6 +53,10 @@ * - A numeric value. * Multiple __flag attributes could be specified, the final flags * value is derived by applying binary "or" to all specified values. + * + * __auxiliary Annotated program is not a separate test, but used as auxiliary + * for some other test cases and should always be loaded. + * __auxiliary_unpriv Same, but load program in unprivileged mode. */ #define __msg(msg) __attribute__((btf_decl_tag("comment:test_expect_msg=" msg))) #define __failure __attribute__((btf_decl_tag("comment:test_expect_failure"))) @@ -65,6 +69,8 @@ #define __flag(flag) __attribute__((btf_decl_tag("comment:test_prog_flags="#flag))) #define __retval(val) __attribute__((btf_decl_tag("comment:test_retval="#val))) #define __retval_unpriv(val) __attribute__((btf_decl_tag("comment:test_retval_unpriv="#val))) +#define __auxiliary __attribute__((btf_decl_tag("comment:test_auxiliary"))) +#define __auxiliary_unpriv __attribute__((btf_decl_tag("comment:test_auxiliary_unpriv"))) /* Convenience macro for use with 'asm volatile' blocks */ #define __naked __attribute__((naked)) @@ -121,107 +127,4 @@ /* make it look to compiler like value is read and written */ #define __sink(expr) asm volatile("" : "+g"(expr)) -struct bpf_iter_num; - -extern int bpf_iter_num_new(struct bpf_iter_num *it, int start, int end) __ksym; -extern int *bpf_iter_num_next(struct bpf_iter_num *it) __ksym; -extern void bpf_iter_num_destroy(struct bpf_iter_num *it) __ksym; - -#ifndef bpf_for_each -/* bpf_for_each(iter_type, cur_elem, args...) provides generic construct for - * using BPF open-coded iterators without having to write mundane explicit - * low-level loop logic. Instead, it provides for()-like generic construct - * that can be used pretty naturally. E.g., for some hypothetical cgroup - * iterator, you'd write: - * - * struct cgroup *cg, *parent_cg = <...>; - * - * bpf_for_each(cgroup, cg, parent_cg, CG_ITER_CHILDREN) { - * bpf_printk("Child cgroup id = %d", cg->cgroup_id); - * if (cg->cgroup_id == 123) - * break; - * } - * - * I.e., it looks almost like high-level for each loop in other languages, - * supports continue/break, and is verifiable by BPF verifier. - * - * For iterating integers, the difference betwen bpf_for_each(num, i, N, M) - * and bpf_for(i, N, M) is in that bpf_for() provides additional proof to - * verifier that i is in [N, M) range, and in bpf_for_each() case i is `int - * *`, not just `int`. So for integers bpf_for() is more convenient. - * - * Note: this macro relies on C99 feature of allowing to declare variables - * inside for() loop, bound to for() loop lifetime. It also utilizes GCC - * extension: __attribute__((cleanup(<func>))), supported by both GCC and - * Clang. - */ -#define bpf_for_each(type, cur, args...) for ( \ - /* initialize and define destructor */ \ - struct bpf_iter_##type ___it __attribute__((aligned(8), /* enforce, just in case */, \ - cleanup(bpf_iter_##type##_destroy))), \ - /* ___p pointer is just to call bpf_iter_##type##_new() *once* to init ___it */ \ - *___p __attribute__((unused)) = ( \ - bpf_iter_##type##_new(&___it, ##args), \ - /* this is a workaround for Clang bug: it currently doesn't emit BTF */ \ - /* for bpf_iter_##type##_destroy() when used from cleanup() attribute */ \ - (void)bpf_iter_##type##_destroy, (void *)0); \ - /* iteration and termination check */ \ - (((cur) = bpf_iter_##type##_next(&___it))); \ -) -#endif /* bpf_for_each */ - -#ifndef bpf_for -/* bpf_for(i, start, end) implements a for()-like looping construct that sets - * provided integer variable *i* to values starting from *start* through, - * but not including, *end*. It also proves to BPF verifier that *i* belongs - * to range [start, end), so this can be used for accessing arrays without - * extra checks. - * - * Note: *start* and *end* are assumed to be expressions with no side effects - * and whose values do not change throughout bpf_for() loop execution. They do - * not have to be statically known or constant, though. - * - * Note: similarly to bpf_for_each(), it relies on C99 feature of declaring for() - * loop bound variables and cleanup attribute, supported by GCC and Clang. - */ -#define bpf_for(i, start, end) for ( \ - /* initialize and define destructor */ \ - struct bpf_iter_num ___it __attribute__((aligned(8), /* enforce, just in case */ \ - cleanup(bpf_iter_num_destroy))), \ - /* ___p pointer is necessary to call bpf_iter_num_new() *once* to init ___it */ \ - *___p __attribute__((unused)) = ( \ - bpf_iter_num_new(&___it, (start), (end)), \ - /* this is a workaround for Clang bug: it currently doesn't emit BTF */ \ - /* for bpf_iter_num_destroy() when used from cleanup() attribute */ \ - (void)bpf_iter_num_destroy, (void *)0); \ - ({ \ - /* iteration step */ \ - int *___t = bpf_iter_num_next(&___it); \ - /* termination and bounds check */ \ - (___t && ((i) = *___t, (i) >= (start) && (i) < (end))); \ - }); \ -) -#endif /* bpf_for */ - -#ifndef bpf_repeat -/* bpf_repeat(N) performs N iterations without exposing iteration number - * - * Note: similarly to bpf_for_each(), it relies on C99 feature of declaring for() - * loop bound variables and cleanup attribute, supported by GCC and Clang. - */ -#define bpf_repeat(N) for ( \ - /* initialize and define destructor */ \ - struct bpf_iter_num ___it __attribute__((aligned(8), /* enforce, just in case */ \ - cleanup(bpf_iter_num_destroy))), \ - /* ___p pointer is necessary to call bpf_iter_num_new() *once* to init ___it */ \ - *___p __attribute__((unused)) = ( \ - bpf_iter_num_new(&___it, 0, (N)), \ - /* this is a workaround for Clang bug: it currently doesn't emit BTF */ \ - /* for bpf_iter_num_destroy() when used from cleanup() attribute */ \ - (void)bpf_iter_num_destroy, (void *)0); \ - bpf_iter_num_next(&___it); \ - /* nothing here */ \ -) -#endif /* bpf_repeat */ - #endif diff --git a/tools/testing/selftests/bpf/progs/linked_list.c b/tools/testing/selftests/bpf/progs/linked_list.c index 53ded51a3abb..57440a554304 100644 --- a/tools/testing/selftests/bpf/progs/linked_list.c +++ b/tools/testing/selftests/bpf/progs/linked_list.c @@ -25,7 +25,7 @@ int list_push_pop(struct bpf_spin_lock *lock, struct bpf_list_head *head, bool l n = bpf_list_pop_front(head); bpf_spin_unlock(lock); if (n) { - bpf_obj_drop(container_of(n, struct foo, node)); + bpf_obj_drop(container_of(n, struct foo, node2)); bpf_obj_drop(f); return 3; } @@ -34,7 +34,7 @@ int list_push_pop(struct bpf_spin_lock *lock, struct bpf_list_head *head, bool l n = bpf_list_pop_back(head); bpf_spin_unlock(lock); if (n) { - bpf_obj_drop(container_of(n, struct foo, node)); + bpf_obj_drop(container_of(n, struct foo, node2)); bpf_obj_drop(f); return 4; } @@ -42,7 +42,7 @@ int list_push_pop(struct bpf_spin_lock *lock, struct bpf_list_head *head, bool l bpf_spin_lock(lock); f->data = 42; - bpf_list_push_front(head, &f->node); + bpf_list_push_front(head, &f->node2); bpf_spin_unlock(lock); if (leave_in_map) return 0; @@ -51,7 +51,7 @@ int list_push_pop(struct bpf_spin_lock *lock, struct bpf_list_head *head, bool l bpf_spin_unlock(lock); if (!n) return 5; - f = container_of(n, struct foo, node); + f = container_of(n, struct foo, node2); if (f->data != 42) { bpf_obj_drop(f); return 6; @@ -59,14 +59,14 @@ int list_push_pop(struct bpf_spin_lock *lock, struct bpf_list_head *head, bool l bpf_spin_lock(lock); f->data = 13; - bpf_list_push_front(head, &f->node); + bpf_list_push_front(head, &f->node2); bpf_spin_unlock(lock); bpf_spin_lock(lock); n = bpf_list_pop_front(head); bpf_spin_unlock(lock); if (!n) return 7; - f = container_of(n, struct foo, node); + f = container_of(n, struct foo, node2); if (f->data != 13) { bpf_obj_drop(f); return 8; @@ -77,7 +77,7 @@ int list_push_pop(struct bpf_spin_lock *lock, struct bpf_list_head *head, bool l n = bpf_list_pop_front(head); bpf_spin_unlock(lock); if (n) { - bpf_obj_drop(container_of(n, struct foo, node)); + bpf_obj_drop(container_of(n, struct foo, node2)); return 9; } @@ -85,7 +85,7 @@ int list_push_pop(struct bpf_spin_lock *lock, struct bpf_list_head *head, bool l n = bpf_list_pop_back(head); bpf_spin_unlock(lock); if (n) { - bpf_obj_drop(container_of(n, struct foo, node)); + bpf_obj_drop(container_of(n, struct foo, node2)); return 10; } return 0; @@ -119,8 +119,8 @@ int list_push_pop_multiple(struct bpf_spin_lock *lock, struct bpf_list_head *hea f[i + 1]->data = i + 1; bpf_spin_lock(lock); - bpf_list_push_front(head, &f[i]->node); - bpf_list_push_front(head, &f[i + 1]->node); + bpf_list_push_front(head, &f[i]->node2); + bpf_list_push_front(head, &f[i + 1]->node2); bpf_spin_unlock(lock); } @@ -130,13 +130,13 @@ int list_push_pop_multiple(struct bpf_spin_lock *lock, struct bpf_list_head *hea bpf_spin_unlock(lock); if (!n) return 3; - pf = container_of(n, struct foo, node); + pf = container_of(n, struct foo, node2); if (pf->data != (ARRAY_SIZE(f) - i - 1)) { bpf_obj_drop(pf); return 4; } bpf_spin_lock(lock); - bpf_list_push_back(head, &pf->node); + bpf_list_push_back(head, &pf->node2); bpf_spin_unlock(lock); } @@ -149,7 +149,7 @@ int list_push_pop_multiple(struct bpf_spin_lock *lock, struct bpf_list_head *hea bpf_spin_unlock(lock); if (!n) return 5; - pf = container_of(n, struct foo, node); + pf = container_of(n, struct foo, node2); if (pf->data != i) { bpf_obj_drop(pf); return 6; @@ -160,7 +160,7 @@ int list_push_pop_multiple(struct bpf_spin_lock *lock, struct bpf_list_head *hea n = bpf_list_pop_back(head); bpf_spin_unlock(lock); if (n) { - bpf_obj_drop(container_of(n, struct foo, node)); + bpf_obj_drop(container_of(n, struct foo, node2)); return 7; } @@ -168,7 +168,7 @@ int list_push_pop_multiple(struct bpf_spin_lock *lock, struct bpf_list_head *hea n = bpf_list_pop_front(head); bpf_spin_unlock(lock); if (n) { - bpf_obj_drop(container_of(n, struct foo, node)); + bpf_obj_drop(container_of(n, struct foo, node2)); return 8; } return 0; @@ -199,7 +199,7 @@ int list_in_list(struct bpf_spin_lock *lock, struct bpf_list_head *head, bool le bpf_spin_lock(lock); f->data = 42; - bpf_list_push_front(head, &f->node); + bpf_list_push_front(head, &f->node2); bpf_spin_unlock(lock); if (leave_in_map) @@ -210,7 +210,7 @@ int list_in_list(struct bpf_spin_lock *lock, struct bpf_list_head *head, bool le bpf_spin_unlock(lock); if (!n) return 4; - f = container_of(n, struct foo, node); + f = container_of(n, struct foo, node2); if (f->data != 42) { bpf_obj_drop(f); return 5; diff --git a/tools/testing/selftests/bpf/progs/linked_list.h b/tools/testing/selftests/bpf/progs/linked_list.h index 3fb2412552fc..c0f3609a7ffa 100644 --- a/tools/testing/selftests/bpf/progs/linked_list.h +++ b/tools/testing/selftests/bpf/progs/linked_list.h @@ -22,7 +22,7 @@ struct foo { struct map_value { struct bpf_spin_lock lock; int data; - struct bpf_list_head head __contains(foo, node); + struct bpf_list_head head __contains(foo, node2); }; struct array_map { @@ -50,7 +50,7 @@ struct { #define private(name) SEC(".bss." #name) __hidden __attribute__((aligned(8))) private(A) struct bpf_spin_lock glock; -private(A) struct bpf_list_head ghead __contains(foo, node); +private(A) struct bpf_list_head ghead __contains(foo, node2); private(B) struct bpf_spin_lock glock2; #endif diff --git a/tools/testing/selftests/bpf/progs/linked_list_fail.c b/tools/testing/selftests/bpf/progs/linked_list_fail.c index 41978b46f58e..f4c63daba229 100644 --- a/tools/testing/selftests/bpf/progs/linked_list_fail.c +++ b/tools/testing/selftests/bpf/progs/linked_list_fail.c @@ -73,22 +73,21 @@ CHECK(inner_map, pop_back, &iv->head); int test##_missing_lock_##op(void *ctx) \ { \ INIT; \ - void (*p)(void *, void *) = (void *)&bpf_list_##op; \ - p(hexpr, nexpr); \ + bpf_list_##op(hexpr, nexpr); \ return 0; \ } -CHECK(kptr, push_front, &f->head, b); -CHECK(kptr, push_back, &f->head, b); +CHECK(kptr, push_front, &f->head, &b->node); +CHECK(kptr, push_back, &f->head, &b->node); -CHECK(global, push_front, &ghead, f); -CHECK(global, push_back, &ghead, f); +CHECK(global, push_front, &ghead, &f->node2); +CHECK(global, push_back, &ghead, &f->node2); -CHECK(map, push_front, &v->head, f); -CHECK(map, push_back, &v->head, f); +CHECK(map, push_front, &v->head, &f->node2); +CHECK(map, push_back, &v->head, &f->node2); -CHECK(inner_map, push_front, &iv->head, f); -CHECK(inner_map, push_back, &iv->head, f); +CHECK(inner_map, push_front, &iv->head, &f->node2); +CHECK(inner_map, push_back, &iv->head, &f->node2); #undef CHECK @@ -135,32 +134,31 @@ CHECK_OP(pop_back); int test##_incorrect_lock_##op(void *ctx) \ { \ INIT; \ - void (*p)(void *, void*) = (void *)&bpf_list_##op; \ bpf_spin_lock(lexpr); \ - p(hexpr, nexpr); \ + bpf_list_##op(hexpr, nexpr); \ return 0; \ } #define CHECK_OP(op) \ - CHECK(kptr_kptr, op, &f1->lock, &f2->head, b); \ - CHECK(kptr_global, op, &f1->lock, &ghead, f); \ - CHECK(kptr_map, op, &f1->lock, &v->head, f); \ - CHECK(kptr_inner_map, op, &f1->lock, &iv->head, f); \ + CHECK(kptr_kptr, op, &f1->lock, &f2->head, &b->node); \ + CHECK(kptr_global, op, &f1->lock, &ghead, &f->node2); \ + CHECK(kptr_map, op, &f1->lock, &v->head, &f->node2); \ + CHECK(kptr_inner_map, op, &f1->lock, &iv->head, &f->node2); \ \ - CHECK(global_global, op, &glock2, &ghead, f); \ - CHECK(global_kptr, op, &glock, &f1->head, b); \ - CHECK(global_map, op, &glock, &v->head, f); \ - CHECK(global_inner_map, op, &glock, &iv->head, f); \ + CHECK(global_global, op, &glock2, &ghead, &f->node2); \ + CHECK(global_kptr, op, &glock, &f1->head, &b->node); \ + CHECK(global_map, op, &glock, &v->head, &f->node2); \ + CHECK(global_inner_map, op, &glock, &iv->head, &f->node2); \ \ - CHECK(map_map, op, &v->lock, &v2->head, f); \ - CHECK(map_kptr, op, &v->lock, &f2->head, b); \ - CHECK(map_global, op, &v->lock, &ghead, f); \ - CHECK(map_inner_map, op, &v->lock, &iv->head, f); \ + CHECK(map_map, op, &v->lock, &v2->head, &f->node2); \ + CHECK(map_kptr, op, &v->lock, &f2->head, &b->node); \ + CHECK(map_global, op, &v->lock, &ghead, &f->node2); \ + CHECK(map_inner_map, op, &v->lock, &iv->head, &f->node2); \ \ - CHECK(inner_map_inner_map, op, &iv->lock, &iv2->head, f); \ - CHECK(inner_map_kptr, op, &iv->lock, &f2->head, b); \ - CHECK(inner_map_global, op, &iv->lock, &ghead, f); \ - CHECK(inner_map_map, op, &iv->lock, &v->head, f); + CHECK(inner_map_inner_map, op, &iv->lock, &iv2->head, &f->node2);\ + CHECK(inner_map_kptr, op, &iv->lock, &f2->head, &b->node); \ + CHECK(inner_map_global, op, &iv->lock, &ghead, &f->node2); \ + CHECK(inner_map_map, op, &iv->lock, &v->head, &f->node2); CHECK_OP(push_front); CHECK_OP(push_back); @@ -340,7 +338,7 @@ int direct_read_node(void *ctx) f = bpf_obj_new(typeof(*f)); if (!f) return 0; - return *(int *)&f->node; + return *(int *)&f->node2; } SEC("?tc") @@ -351,12 +349,12 @@ int direct_write_node(void *ctx) f = bpf_obj_new(typeof(*f)); if (!f) return 0; - *(int *)&f->node = 0; + *(int *)&f->node2 = 0; return 0; } static __always_inline -int use_after_unlock(void (*op)(void *head, void *node)) +int use_after_unlock(bool push_front) { struct foo *f; @@ -365,7 +363,10 @@ int use_after_unlock(void (*op)(void *head, void *node)) return 0; bpf_spin_lock(&glock); f->data = 42; - op(&ghead, &f->node); + if (push_front) + bpf_list_push_front(&ghead, &f->node2); + else + bpf_list_push_back(&ghead, &f->node2); bpf_spin_unlock(&glock); return f->data; @@ -374,17 +375,17 @@ int use_after_unlock(void (*op)(void *head, void *node)) SEC("?tc") int use_after_unlock_push_front(void *ctx) { - return use_after_unlock((void *)bpf_list_push_front); + return use_after_unlock(true); } SEC("?tc") int use_after_unlock_push_back(void *ctx) { - return use_after_unlock((void *)bpf_list_push_back); + return use_after_unlock(false); } static __always_inline -int list_double_add(void (*op)(void *head, void *node)) +int list_double_add(bool push_front) { struct foo *f; @@ -392,8 +393,13 @@ int list_double_add(void (*op)(void *head, void *node)) if (!f) return 0; bpf_spin_lock(&glock); - op(&ghead, &f->node); - op(&ghead, &f->node); + if (push_front) { + bpf_list_push_front(&ghead, &f->node2); + bpf_list_push_front(&ghead, &f->node2); + } else { + bpf_list_push_back(&ghead, &f->node2); + bpf_list_push_back(&ghead, &f->node2); + } bpf_spin_unlock(&glock); return 0; @@ -402,13 +408,13 @@ int list_double_add(void (*op)(void *head, void *node)) SEC("?tc") int double_push_front(void *ctx) { - return list_double_add((void *)bpf_list_push_front); + return list_double_add(true); } SEC("?tc") int double_push_back(void *ctx) { - return list_double_add((void *)bpf_list_push_back); + return list_double_add(false); } SEC("?tc") @@ -450,7 +456,7 @@ int incorrect_node_var_off(struct __sk_buff *ctx) if (!f) return 0; bpf_spin_lock(&glock); - bpf_list_push_front(&ghead, (void *)&f->node + ctx->protocol); + bpf_list_push_front(&ghead, (void *)&f->node2 + ctx->protocol); bpf_spin_unlock(&glock); return 0; @@ -465,7 +471,7 @@ int incorrect_node_off1(void *ctx) if (!f) return 0; bpf_spin_lock(&glock); - bpf_list_push_front(&ghead, (void *)&f->node + 1); + bpf_list_push_front(&ghead, (void *)&f->node2 + 1); bpf_spin_unlock(&glock); return 0; @@ -480,7 +486,7 @@ int incorrect_node_off2(void *ctx) if (!f) return 0; bpf_spin_lock(&glock); - bpf_list_push_front(&ghead, &f->node2); + bpf_list_push_front(&ghead, &f->node); bpf_spin_unlock(&glock); return 0; @@ -510,7 +516,7 @@ int incorrect_head_var_off1(struct __sk_buff *ctx) if (!f) return 0; bpf_spin_lock(&glock); - bpf_list_push_front((void *)&ghead + ctx->protocol, &f->node); + bpf_list_push_front((void *)&ghead + ctx->protocol, &f->node2); bpf_spin_unlock(&glock); return 0; @@ -525,7 +531,7 @@ int incorrect_head_var_off2(struct __sk_buff *ctx) if (!f) return 0; bpf_spin_lock(&glock); - bpf_list_push_front((void *)&f->head + ctx->protocol, &f->node); + bpf_list_push_front((void *)&f->head + ctx->protocol, &f->node2); bpf_spin_unlock(&glock); return 0; @@ -563,7 +569,7 @@ int incorrect_head_off2(void *ctx) return 0; bpf_spin_lock(&glock); - bpf_list_push_front((void *)&ghead + 1, &f->node); + bpf_list_push_front((void *)&ghead + 1, &f->node2); bpf_spin_unlock(&glock); return 0; diff --git a/tools/testing/selftests/bpf/progs/map_kptr.c b/tools/testing/selftests/bpf/progs/map_kptr.c index dae5dab1bbf7..d7150041e5d1 100644 --- a/tools/testing/selftests/bpf/progs/map_kptr.c +++ b/tools/testing/selftests/bpf/progs/map_kptr.c @@ -115,8 +115,6 @@ DEFINE_MAP_OF_MAP(BPF_MAP_TYPE_HASH_OF_MAPS, hash_malloc_map, hash_of_hash_mallo DEFINE_MAP_OF_MAP(BPF_MAP_TYPE_HASH_OF_MAPS, lru_hash_map, hash_of_lru_hash_maps); extern struct prog_test_ref_kfunc *bpf_kfunc_call_test_acquire(unsigned long *sp) __ksym; -extern struct prog_test_ref_kfunc * -bpf_kfunc_call_test_kptr_get(struct prog_test_ref_kfunc **p, int a, int b) __ksym; extern void bpf_kfunc_call_test_release(struct prog_test_ref_kfunc *p) __ksym; void bpf_kfunc_call_test_ref(struct prog_test_ref_kfunc *p) __ksym; @@ -187,25 +185,10 @@ static void test_kptr_ref(struct map_value *v) bpf_kfunc_call_test_release(p); } -static void test_kptr_get(struct map_value *v) -{ - struct prog_test_ref_kfunc *p; - - p = bpf_kfunc_call_test_kptr_get(&v->ref_ptr, 0, 0); - if (!p) - return; - if (p->a + p->b > 100) { - bpf_kfunc_call_test_release(p); - return; - } - bpf_kfunc_call_test_release(p); -} - static void test_kptr(struct map_value *v) { test_kptr_unref(v); test_kptr_ref(v); - test_kptr_get(v); } SEC("tc") @@ -338,38 +321,25 @@ int test_map_kptr_ref_pre(struct map_value *v) if (p_st->cnt.refs.counter != ref) return 4; - p = bpf_kfunc_call_test_kptr_get(&v->ref_ptr, 0, 0); - if (!p) - return 5; - ref++; - if (p_st->cnt.refs.counter != ref) { - ret = 6; - goto end; - } - bpf_kfunc_call_test_release(p); - ref--; - if (p_st->cnt.refs.counter != ref) - return 7; - p = bpf_kptr_xchg(&v->ref_ptr, NULL); if (!p) - return 8; + return 5; bpf_kfunc_call_test_release(p); ref--; if (p_st->cnt.refs.counter != ref) - return 9; + return 6; p = bpf_kfunc_call_test_acquire(&arg); if (!p) - return 10; + return 7; ref++; p = bpf_kptr_xchg(&v->ref_ptr, p); if (p) { - ret = 11; + ret = 8; goto end; } if (p_st->cnt.refs.counter != ref) - return 12; + return 9; /* Leave in map */ return 0; diff --git a/tools/testing/selftests/bpf/progs/map_kptr_fail.c b/tools/testing/selftests/bpf/progs/map_kptr_fail.c index 15bf3127dba3..da8c724f839b 100644 --- a/tools/testing/selftests/bpf/progs/map_kptr_fail.c +++ b/tools/testing/selftests/bpf/progs/map_kptr_fail.c @@ -21,8 +21,6 @@ struct array_map { extern struct prog_test_ref_kfunc *bpf_kfunc_call_test_acquire(unsigned long *sp) __ksym; extern void bpf_kfunc_call_test_release(struct prog_test_ref_kfunc *p) __ksym; -extern struct prog_test_ref_kfunc * -bpf_kfunc_call_test_kptr_get(struct prog_test_ref_kfunc **p, int a, int b) __ksym; SEC("?tc") __failure __msg("kptr access size must be BPF_DW") @@ -221,67 +219,6 @@ int reject_kptr_xchg_on_unref(struct __sk_buff *ctx) } SEC("?tc") -__failure __msg("arg#0 expected pointer to map value") -int reject_kptr_get_no_map_val(struct __sk_buff *ctx) -{ - bpf_kfunc_call_test_kptr_get((void *)&ctx, 0, 0); - return 0; -} - -SEC("?tc") -__failure __msg("arg#0 expected pointer to map value") -int reject_kptr_get_no_null_map_val(struct __sk_buff *ctx) -{ - bpf_kfunc_call_test_kptr_get(bpf_map_lookup_elem(&array_map, &(int){0}), 0, 0); - return 0; -} - -SEC("?tc") -__failure __msg("arg#0 no referenced kptr at map value offset=0") -int reject_kptr_get_no_kptr(struct __sk_buff *ctx) -{ - struct map_value *v; - int key = 0; - - v = bpf_map_lookup_elem(&array_map, &key); - if (!v) - return 0; - - bpf_kfunc_call_test_kptr_get((void *)v, 0, 0); - return 0; -} - -SEC("?tc") -__failure __msg("arg#0 no referenced kptr at map value offset=8") -int reject_kptr_get_on_unref(struct __sk_buff *ctx) -{ - struct map_value *v; - int key = 0; - - v = bpf_map_lookup_elem(&array_map, &key); - if (!v) - return 0; - - bpf_kfunc_call_test_kptr_get(&v->unref_ptr, 0, 0); - return 0; -} - -SEC("?tc") -__failure __msg("kernel function bpf_kfunc_call_test_kptr_get args#0") -int reject_kptr_get_bad_type_match(struct __sk_buff *ctx) -{ - struct map_value *v; - int key = 0; - - v = bpf_map_lookup_elem(&array_map, &key); - if (!v) - return 0; - - bpf_kfunc_call_test_kptr_get((void *)&v->ref_memb_ptr, 0, 0); - return 0; -} - -SEC("?tc") __failure __msg("R1 type=rcu_ptr_or_null_ expected=percpu_ptr_") int mark_ref_as_untrusted_or_null(struct __sk_buff *ctx) { @@ -429,21 +366,6 @@ int kptr_xchg_ref_state(struct __sk_buff *ctx) } SEC("?tc") -__failure __msg("Unreleased reference id=3 alloc_insn=") -int kptr_get_ref_state(struct __sk_buff *ctx) -{ - struct map_value *v; - int key = 0; - - v = bpf_map_lookup_elem(&array_map, &key); - if (!v) - return 0; - - bpf_kfunc_call_test_kptr_get(&v->ref_ptr, 0, 0); - return 0; -} - -SEC("?tc") __failure __msg("Possibly NULL pointer passed to helper arg2") int kptr_xchg_possibly_null(struct __sk_buff *ctx) { diff --git a/tools/testing/selftests/bpf/progs/rbtree.c b/tools/testing/selftests/bpf/progs/rbtree.c index 4c90aa6abddd..b09f4fffe57c 100644 --- a/tools/testing/selftests/bpf/progs/rbtree.c +++ b/tools/testing/selftests/bpf/progs/rbtree.c @@ -93,9 +93,11 @@ long rbtree_add_and_remove(void *ctx) res = bpf_rbtree_remove(&groot, &n->node); bpf_spin_unlock(&glock); + if (!res) + return 1; + n = container_of(res, struct node_data, node); removed_key = n->key; - bpf_obj_drop(n); return 0; @@ -148,9 +150,11 @@ long rbtree_first_and_remove(void *ctx) res = bpf_rbtree_remove(&groot, &o->node); bpf_spin_unlock(&glock); + if (!res) + return 5; + o = container_of(res, struct node_data, node); removed_key = o->key; - bpf_obj_drop(o); bpf_spin_lock(&glock); @@ -173,4 +177,70 @@ err_out: return 1; } +SEC("tc") +long rbtree_api_release_aliasing(void *ctx) +{ + struct node_data *n, *m, *o; + struct bpf_rb_node *res, *res2; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 1; + n->key = 41; + n->data = 42; + + bpf_spin_lock(&glock); + bpf_rbtree_add(&groot, &n->node, less); + bpf_spin_unlock(&glock); + + bpf_spin_lock(&glock); + + /* m and o point to the same node, + * but verifier doesn't know this + */ + res = bpf_rbtree_first(&groot); + if (!res) + goto err_out; + o = container_of(res, struct node_data, node); + + res = bpf_rbtree_first(&groot); + if (!res) + goto err_out; + m = container_of(res, struct node_data, node); + + res = bpf_rbtree_remove(&groot, &m->node); + /* Retval of previous remove returns an owning reference to m, + * which is the same node non-owning ref o is pointing at. + * We can safely try to remove o as the second rbtree_remove will + * return NULL since the node isn't in a tree. + * + * Previously we relied on the verifier type system + rbtree_remove + * invalidating non-owning refs to ensure that rbtree_remove couldn't + * fail, but now rbtree_remove does runtime checking so we no longer + * invalidate non-owning refs after remove. + */ + res2 = bpf_rbtree_remove(&groot, &o->node); + + bpf_spin_unlock(&glock); + + if (res) { + o = container_of(res, struct node_data, node); + first_data[0] = o->data; + bpf_obj_drop(o); + } + if (res2) { + /* The second remove fails, so res2 is null and this doesn't + * execute + */ + m = container_of(res2, struct node_data, node); + first_data[1] = m->data; + bpf_obj_drop(m); + } + return 0; + +err_out: + bpf_spin_unlock(&glock); + return 1; +} + char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/rbtree_fail.c b/tools/testing/selftests/bpf/progs/rbtree_fail.c index 46d7d18a218f..3fecf1c6dfe5 100644 --- a/tools/testing/selftests/bpf/progs/rbtree_fail.c +++ b/tools/testing/selftests/bpf/progs/rbtree_fail.c @@ -105,7 +105,7 @@ long rbtree_api_remove_unadded_node(void *ctx) } SEC("?tc") -__failure __msg("Unreleased reference id=2 alloc_insn=10") +__failure __msg("Unreleased reference id=3 alloc_insn=10") long rbtree_api_remove_no_drop(void *ctx) { struct bpf_rb_node *res; @@ -118,11 +118,13 @@ long rbtree_api_remove_no_drop(void *ctx) res = bpf_rbtree_remove(&groot, res); - n = container_of(res, struct node_data, node); - __sink(n); + if (res) { + n = container_of(res, struct node_data, node); + __sink(n); + } bpf_spin_unlock(&glock); - /* bpf_obj_drop(n) is missing here */ + /* if (res) { bpf_obj_drop(n); } is missing here */ return 0; unlock_err: @@ -150,35 +152,36 @@ long rbtree_api_add_to_multiple_trees(void *ctx) } SEC("?tc") -__failure __msg("rbtree_remove node input must be non-owning ref") -long rbtree_api_add_release_unlock_escape(void *ctx) +__failure __msg("dereference of modified ptr_or_null_ ptr R2 off=16 disallowed") +long rbtree_api_use_unchecked_remove_retval(void *ctx) { - struct node_data *n; - - n = bpf_obj_new(typeof(*n)); - if (!n) - return 1; + struct bpf_rb_node *res; bpf_spin_lock(&glock); - bpf_rbtree_add(&groot, &n->node, less); + + res = bpf_rbtree_first(&groot); + if (!res) + goto err_out; + res = bpf_rbtree_remove(&groot, res); + bpf_spin_unlock(&glock); bpf_spin_lock(&glock); - /* After add() in previous critical section, n should be - * release_on_unlock and released after previous spin_unlock, - * so should not be possible to use it here - */ - bpf_rbtree_remove(&groot, &n->node); + /* Must check res for NULL before using in rbtree_add below */ + bpf_rbtree_add(&groot, res, less); bpf_spin_unlock(&glock); return 0; + +err_out: + bpf_spin_unlock(&glock); + return 1; } SEC("?tc") __failure __msg("rbtree_remove node input must be non-owning ref") -long rbtree_api_release_aliasing(void *ctx) +long rbtree_api_add_release_unlock_escape(void *ctx) { - struct node_data *n, *m, *o; - struct bpf_rb_node *res; + struct node_data *n; n = bpf_obj_new(typeof(*n)); if (!n) @@ -189,37 +192,11 @@ long rbtree_api_release_aliasing(void *ctx) bpf_spin_unlock(&glock); bpf_spin_lock(&glock); - - /* m and o point to the same node, - * but verifier doesn't know this - */ - res = bpf_rbtree_first(&groot); - if (!res) - return 1; - o = container_of(res, struct node_data, node); - - res = bpf_rbtree_first(&groot); - if (!res) - return 1; - m = container_of(res, struct node_data, node); - - bpf_rbtree_remove(&groot, &m->node); - /* This second remove shouldn't be possible. Retval of previous - * remove returns owning reference to m, which is the same - * node o's non-owning ref is pointing at - * - * In order to preserve property - * * owning ref must not be in rbtree - * * non-owning ref must be in rbtree - * - * o's ref must be invalidated after previous remove. Otherwise - * we'd have non-owning ref to node that isn't in rbtree, and - * verifier wouldn't be able to use type system to prevent remove - * of ref that already isn't in any tree. Would have to do runtime - * checks in that case. + /* After add() in previous critical section, n should be + * release_on_unlock and released after previous spin_unlock, + * so should not be possible to use it here */ - bpf_rbtree_remove(&groot, &o->node); - + bpf_rbtree_remove(&groot, &n->node); bpf_spin_unlock(&glock); return 0; } diff --git a/tools/testing/selftests/bpf/progs/refcounted_kptr.c b/tools/testing/selftests/bpf/progs/refcounted_kptr.c new file mode 100644 index 000000000000..1d348a225140 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/refcounted_kptr.c @@ -0,0 +1,406 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */ + +#include <vmlinux.h> +#include <bpf/bpf_tracing.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_core_read.h> +#include "bpf_misc.h" +#include "bpf_experimental.h" + +struct node_data { + long key; + long list_data; + struct bpf_rb_node r; + struct bpf_list_node l; + struct bpf_refcount ref; +}; + +struct map_value { + struct node_data __kptr *node; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, int); + __type(value, struct map_value); + __uint(max_entries, 1); +} stashed_nodes SEC(".maps"); + +struct node_acquire { + long key; + long data; + struct bpf_rb_node node; + struct bpf_refcount refcount; +}; + +#define private(name) SEC(".bss." #name) __hidden __attribute__((aligned(8))) +private(A) struct bpf_spin_lock lock; +private(A) struct bpf_rb_root root __contains(node_data, r); +private(A) struct bpf_list_head head __contains(node_data, l); + +private(B) struct bpf_spin_lock alock; +private(B) struct bpf_rb_root aroot __contains(node_acquire, node); + +static bool less(struct bpf_rb_node *node_a, const struct bpf_rb_node *node_b) +{ + struct node_data *a; + struct node_data *b; + + a = container_of(node_a, struct node_data, r); + b = container_of(node_b, struct node_data, r); + + return a->key < b->key; +} + +static bool less_a(struct bpf_rb_node *a, const struct bpf_rb_node *b) +{ + struct node_acquire *node_a; + struct node_acquire *node_b; + + node_a = container_of(a, struct node_acquire, node); + node_b = container_of(b, struct node_acquire, node); + + return node_a->key < node_b->key; +} + +static long __insert_in_tree_and_list(struct bpf_list_head *head, + struct bpf_rb_root *root, + struct bpf_spin_lock *lock) +{ + struct node_data *n, *m; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return -1; + + m = bpf_refcount_acquire(n); + m->key = 123; + m->list_data = 456; + + bpf_spin_lock(lock); + if (bpf_rbtree_add(root, &n->r, less)) { + /* Failure to insert - unexpected */ + bpf_spin_unlock(lock); + bpf_obj_drop(m); + return -2; + } + bpf_spin_unlock(lock); + + bpf_spin_lock(lock); + if (bpf_list_push_front(head, &m->l)) { + /* Failure to insert - unexpected */ + bpf_spin_unlock(lock); + return -3; + } + bpf_spin_unlock(lock); + return 0; +} + +static long __stash_map_insert_tree(int idx, int val, struct bpf_rb_root *root, + struct bpf_spin_lock *lock) +{ + struct map_value *mapval; + struct node_data *n, *m; + + mapval = bpf_map_lookup_elem(&stashed_nodes, &idx); + if (!mapval) + return -1; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return -2; + + n->key = val; + m = bpf_refcount_acquire(n); + + n = bpf_kptr_xchg(&mapval->node, n); + if (n) { + bpf_obj_drop(n); + bpf_obj_drop(m); + return -3; + } + + bpf_spin_lock(lock); + if (bpf_rbtree_add(root, &m->r, less)) { + /* Failure to insert - unexpected */ + bpf_spin_unlock(lock); + return -4; + } + bpf_spin_unlock(lock); + return 0; +} + +static long __read_from_tree(struct bpf_rb_root *root, + struct bpf_spin_lock *lock, + bool remove_from_tree) +{ + struct bpf_rb_node *rb; + struct node_data *n; + long res = -99; + + bpf_spin_lock(lock); + + rb = bpf_rbtree_first(root); + if (!rb) { + bpf_spin_unlock(lock); + return -1; + } + + n = container_of(rb, struct node_data, r); + res = n->key; + + if (!remove_from_tree) { + bpf_spin_unlock(lock); + return res; + } + + rb = bpf_rbtree_remove(root, rb); + bpf_spin_unlock(lock); + if (!rb) + return -2; + n = container_of(rb, struct node_data, r); + bpf_obj_drop(n); + return res; +} + +static long __read_from_list(struct bpf_list_head *head, + struct bpf_spin_lock *lock, + bool remove_from_list) +{ + struct bpf_list_node *l; + struct node_data *n; + long res = -99; + + bpf_spin_lock(lock); + + l = bpf_list_pop_front(head); + if (!l) { + bpf_spin_unlock(lock); + return -1; + } + + n = container_of(l, struct node_data, l); + res = n->list_data; + + if (!remove_from_list) { + if (bpf_list_push_back(head, &n->l)) { + bpf_spin_unlock(lock); + return -2; + } + } + + bpf_spin_unlock(lock); + + if (remove_from_list) + bpf_obj_drop(n); + return res; +} + +static long __read_from_unstash(int idx) +{ + struct node_data *n = NULL; + struct map_value *mapval; + long val = -99; + + mapval = bpf_map_lookup_elem(&stashed_nodes, &idx); + if (!mapval) + return -1; + + n = bpf_kptr_xchg(&mapval->node, n); + if (!n) + return -2; + + val = n->key; + bpf_obj_drop(n); + return val; +} + +#define INSERT_READ_BOTH(rem_tree, rem_list, desc) \ +SEC("tc") \ +__description(desc) \ +__success __retval(579) \ +long insert_and_remove_tree_##rem_tree##_list_##rem_list(void *ctx) \ +{ \ + long err, tree_data, list_data; \ + \ + err = __insert_in_tree_and_list(&head, &root, &lock); \ + if (err) \ + return err; \ + \ + err = __read_from_tree(&root, &lock, rem_tree); \ + if (err < 0) \ + return err; \ + else \ + tree_data = err; \ + \ + err = __read_from_list(&head, &lock, rem_list); \ + if (err < 0) \ + return err; \ + else \ + list_data = err; \ + \ + return tree_data + list_data; \ +} + +/* After successful insert of struct node_data into both collections: + * - it should have refcount = 2 + * - removing / not removing the node_data from a collection after + * reading should have no effect on ability to read / remove from + * the other collection + */ +INSERT_READ_BOTH(true, true, "insert_read_both: remove from tree + list"); +INSERT_READ_BOTH(false, false, "insert_read_both: remove from neither"); +INSERT_READ_BOTH(true, false, "insert_read_both: remove from tree"); +INSERT_READ_BOTH(false, true, "insert_read_both: remove from list"); + +#undef INSERT_READ_BOTH +#define INSERT_READ_BOTH(rem_tree, rem_list, desc) \ +SEC("tc") \ +__description(desc) \ +__success __retval(579) \ +long insert_and_remove_lf_tree_##rem_tree##_list_##rem_list(void *ctx) \ +{ \ + long err, tree_data, list_data; \ + \ + err = __insert_in_tree_and_list(&head, &root, &lock); \ + if (err) \ + return err; \ + \ + err = __read_from_list(&head, &lock, rem_list); \ + if (err < 0) \ + return err; \ + else \ + list_data = err; \ + \ + err = __read_from_tree(&root, &lock, rem_tree); \ + if (err < 0) \ + return err; \ + else \ + tree_data = err; \ + \ + return tree_data + list_data; \ +} + +/* Similar to insert_read_both, but list data is read and possibly removed + * first + * + * Results should be no different than reading and possibly removing rbtree + * node first + */ +INSERT_READ_BOTH(true, true, "insert_read_both_list_first: remove from tree + list"); +INSERT_READ_BOTH(false, false, "insert_read_both_list_first: remove from neither"); +INSERT_READ_BOTH(true, false, "insert_read_both_list_first: remove from tree"); +INSERT_READ_BOTH(false, true, "insert_read_both_list_first: remove from list"); + +#define INSERT_DOUBLE_READ_AND_DEL(read_fn, read_root, desc) \ +SEC("tc") \ +__description(desc) \ +__success __retval(-1) \ +long insert_double_##read_fn##_and_del_##read_root(void *ctx) \ +{ \ + long err, list_data; \ + \ + err = __insert_in_tree_and_list(&head, &root, &lock); \ + if (err) \ + return err; \ + \ + err = read_fn(&read_root, &lock, true); \ + if (err < 0) \ + return err; \ + else \ + list_data = err; \ + \ + err = read_fn(&read_root, &lock, true); \ + if (err < 0) \ + return err; \ + \ + return err + list_data; \ +} + +/* Insert into both tree and list, then try reading-and-removing from either twice + * + * The second read-and-remove should fail on read step since the node has + * already been removed + */ +INSERT_DOUBLE_READ_AND_DEL(__read_from_tree, root, "insert_double_del: 2x read-and-del from tree"); +INSERT_DOUBLE_READ_AND_DEL(__read_from_list, head, "insert_double_del: 2x read-and-del from list"); + +#define INSERT_STASH_READ(rem_tree, desc) \ +SEC("tc") \ +__description(desc) \ +__success __retval(84) \ +long insert_rbtree_and_stash__del_tree_##rem_tree(void *ctx) \ +{ \ + long err, tree_data, map_data; \ + \ + err = __stash_map_insert_tree(0, 42, &root, &lock); \ + if (err) \ + return err; \ + \ + err = __read_from_tree(&root, &lock, rem_tree); \ + if (err < 0) \ + return err; \ + else \ + tree_data = err; \ + \ + err = __read_from_unstash(0); \ + if (err < 0) \ + return err; \ + else \ + map_data = err; \ + \ + return tree_data + map_data; \ +} + +/* Stash a refcounted node in map_val, insert same node into tree, then try + * reading data from tree then unstashed map_val, possibly removing from tree + * + * Removing from tree should have no effect on map_val kptr validity + */ +INSERT_STASH_READ(true, "insert_stash_read: remove from tree"); +INSERT_STASH_READ(false, "insert_stash_read: don't remove from tree"); + +SEC("tc") +__success +long rbtree_refcounted_node_ref_escapes(void *ctx) +{ + struct node_acquire *n, *m; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 1; + + bpf_spin_lock(&alock); + bpf_rbtree_add(&aroot, &n->node, less_a); + m = bpf_refcount_acquire(n); + bpf_spin_unlock(&alock); + + m->key = 2; + bpf_obj_drop(m); + return 0; +} + +SEC("tc") +__success +long rbtree_refcounted_node_ref_escapes_owning_input(void *ctx) +{ + struct node_acquire *n, *m; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 1; + + m = bpf_refcount_acquire(n); + m->key = 2; + + bpf_spin_lock(&alock); + bpf_rbtree_add(&aroot, &n->node, less_a); + bpf_spin_unlock(&alock); + + bpf_obj_drop(m); + + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c b/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c new file mode 100644 index 000000000000..efcb308f80ad --- /dev/null +++ b/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <vmlinux.h> +#include <bpf/bpf_tracing.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_core_read.h> +#include "bpf_experimental.h" +#include "bpf_misc.h" + +struct node_acquire { + long key; + long data; + struct bpf_rb_node node; + struct bpf_refcount refcount; +}; + +#define private(name) SEC(".data." #name) __hidden __attribute__((aligned(8))) +private(A) struct bpf_spin_lock glock; +private(A) struct bpf_rb_root groot __contains(node_acquire, node); + +static bool less(struct bpf_rb_node *a, const struct bpf_rb_node *b) +{ + struct node_acquire *node_a; + struct node_acquire *node_b; + + node_a = container_of(a, struct node_acquire, node); + node_b = container_of(b, struct node_acquire, node); + + return node_a->key < node_b->key; +} + +SEC("?tc") +__failure __msg("Unreleased reference id=3 alloc_insn=21") +long rbtree_refcounted_node_ref_escapes(void *ctx) +{ + struct node_acquire *n, *m; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 1; + + bpf_spin_lock(&glock); + bpf_rbtree_add(&groot, &n->node, less); + /* m becomes an owning ref but is never drop'd or added to a tree */ + m = bpf_refcount_acquire(n); + bpf_spin_unlock(&glock); + + m->key = 2; + return 0; +} + +SEC("?tc") +__failure __msg("Unreleased reference id=3 alloc_insn=9") +long rbtree_refcounted_node_ref_escapes_owning_input(void *ctx) +{ + struct node_acquire *n, *m; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 1; + + /* m becomes an owning ref but is never drop'd or added to a tree */ + m = bpf_refcount_acquire(n); + m->key = 2; + + bpf_spin_lock(&glock); + bpf_rbtree_add(&groot, &n->node, less); + bpf_spin_unlock(&glock); + + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/sockopt_sk.c b/tools/testing/selftests/bpf/progs/sockopt_sk.c index c8d810010a94..fe1df4cd206e 100644 --- a/tools/testing/selftests/bpf/progs/sockopt_sk.c +++ b/tools/testing/selftests/bpf/progs/sockopt_sk.c @@ -32,6 +32,12 @@ int _getsockopt(struct bpf_sockopt *ctx) __u8 *optval_end = ctx->optval_end; __u8 *optval = ctx->optval; struct sockopt_sk *storage; + struct bpf_sock *sk; + + /* Bypass AF_NETLINK. */ + sk = ctx->sk; + if (sk && sk->family == AF_NETLINK) + return 1; /* Make sure bpf_get_netns_cookie is callable. */ @@ -131,6 +137,12 @@ int _setsockopt(struct bpf_sockopt *ctx) __u8 *optval_end = ctx->optval_end; __u8 *optval = ctx->optval; struct sockopt_sk *storage; + struct bpf_sock *sk; + + /* Bypass AF_NETLINK. */ + sk = ctx->sk; + if (sk && sk->family == AF_NETLINK) + return 1; /* Make sure bpf_get_netns_cookie is callable. */ diff --git a/tools/testing/selftests/bpf/progs/test_access_variable_array.c b/tools/testing/selftests/bpf/progs/test_access_variable_array.c new file mode 100644 index 000000000000..808c49b79889 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_access_variable_array.c @@ -0,0 +1,19 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Bytedance */ + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> + +unsigned long span = 0; + +SEC("fentry/load_balance") +int BPF_PROG(fentry_fentry, int this_cpu, struct rq *this_rq, + struct sched_domain *sd) +{ + span = sd->span[0]; + + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/test_log_fixup.c b/tools/testing/selftests/bpf/progs/test_log_fixup.c index 60450cb0e72e..1bd48feaaa42 100644 --- a/tools/testing/selftests/bpf/progs/test_log_fixup.c +++ b/tools/testing/selftests/bpf/progs/test_log_fixup.c @@ -61,4 +61,14 @@ int use_missing_map(const void *ctx) return value != NULL; } +extern int bpf_nonexistent_kfunc(void) __ksym __weak; + +SEC("?raw_tp/sys_enter") +int use_missing_kfunc(const void *ctx) +{ + bpf_nonexistent_kfunc(); + + return 0; +} + char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/tracing_struct.c b/tools/testing/selftests/bpf/progs/tracing_struct.c index e718f0ebee7d..c435a3a8328a 100644 --- a/tools/testing/selftests/bpf/progs/tracing_struct.c +++ b/tools/testing/selftests/bpf/progs/tracing_struct.c @@ -13,12 +13,18 @@ struct bpf_testmod_struct_arg_2 { long b; }; +struct bpf_testmod_struct_arg_3 { + int a; + int b[]; +}; + long t1_a_a, t1_a_b, t1_b, t1_c, t1_ret, t1_nregs; __u64 t1_reg0, t1_reg1, t1_reg2, t1_reg3; long t2_a, t2_b_a, t2_b_b, t2_c, t2_ret; long t3_a, t3_b, t3_c_a, t3_c_b, t3_ret; long t4_a_a, t4_b, t4_c, t4_d, t4_e_a, t4_e_b, t4_ret; long t5_ret; +int t6; SEC("fentry/bpf_testmod_test_struct_arg_1") int BPF_PROG2(test_struct_arg_1, struct bpf_testmod_struct_arg_2, a, int, b, int, c) @@ -117,4 +123,11 @@ int BPF_PROG2(test_struct_arg_10, int, ret) return 0; } +SEC("fentry/bpf_testmod_test_struct_arg_6") +int BPF_PROG2(test_struct_arg_11, struct bpf_testmod_struct_arg_3 *, a) +{ + t6 = a->b[0]; + return 0; +} + char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds.c b/tools/testing/selftests/bpf/progs/verifier_bounds.c new file mode 100644 index 000000000000..c5588a14fe2e --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_bounds.c @@ -0,0 +1,1076 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/bounds.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 1); + __type(key, long long); + __type(value, long long); +} map_hash_8b SEC(".maps"); + +SEC("socket") +__description("subtraction bounds (map value) variant 1") +__failure __msg("R0 max value is outside of the allowed memory range") +__failure_unpriv +__naked void bounds_map_value_variant_1(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = *(u8*)(r0 + 0); \ + if r1 > 0xff goto l0_%=; \ + r3 = *(u8*)(r0 + 1); \ + if r3 > 0xff goto l0_%=; \ + r1 -= r3; \ + r1 >>= 56; \ + r0 += r1; \ + r0 = *(u8*)(r0 + 0); \ + exit; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("subtraction bounds (map value) variant 2") +__failure +__msg("R0 min value is negative, either use unsigned index or do a if (index >=0) check.") +__msg_unpriv("R1 has unknown scalar with mixed signed bounds") +__naked void bounds_map_value_variant_2(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = *(u8*)(r0 + 0); \ + if r1 > 0xff goto l0_%=; \ + r3 = *(u8*)(r0 + 1); \ + if r3 > 0xff goto l0_%=; \ + r1 -= r3; \ + r0 += r1; \ + r0 = *(u8*)(r0 + 0); \ + exit; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("check subtraction on pointers for unpriv") +__success __failure_unpriv __msg_unpriv("R9 pointer -= pointer prohibited") +__retval(0) +__naked void subtraction_on_pointers_for_unpriv(void) +{ + asm volatile (" \ + r0 = 0; \ + r1 = %[map_hash_8b] ll; \ + r2 = r10; \ + r2 += -8; \ + r6 = 9; \ + *(u64*)(r2 + 0) = r6; \ + call %[bpf_map_lookup_elem]; \ + r9 = r10; \ + r9 -= r0; \ + r1 = %[map_hash_8b] ll; \ + r2 = r10; \ + r2 += -8; \ + r6 = 0; \ + *(u64*)(r2 + 0) = r6; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: *(u64*)(r0 + 0) = r9; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds check based on zero-extended MOV") +__success __success_unpriv __retval(0) +__naked void based_on_zero_extended_mov(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + /* r2 = 0x0000'0000'ffff'ffff */ \ + w2 = 0xffffffff; \ + /* r2 = 0 */ \ + r2 >>= 32; \ + /* no-op */ \ + r0 += r2; \ + /* access at offset 0 */ \ + r0 = *(u8*)(r0 + 0); \ +l0_%=: /* exit */ \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds check based on sign-extended MOV. test1") +__failure __msg("map_value pointer and 4294967295") +__failure_unpriv +__naked void on_sign_extended_mov_test1(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + /* r2 = 0xffff'ffff'ffff'ffff */ \ + r2 = 0xffffffff; \ + /* r2 = 0xffff'ffff */ \ + r2 >>= 32; \ + /* r0 = <oob pointer> */ \ + r0 += r2; \ + /* access to OOB pointer */ \ + r0 = *(u8*)(r0 + 0); \ +l0_%=: /* exit */ \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds check based on sign-extended MOV. test2") +__failure __msg("R0 min value is outside of the allowed memory range") +__failure_unpriv +__naked void on_sign_extended_mov_test2(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + /* r2 = 0xffff'ffff'ffff'ffff */ \ + r2 = 0xffffffff; \ + /* r2 = 0xfff'ffff */ \ + r2 >>= 36; \ + /* r0 = <oob pointer> */ \ + r0 += r2; \ + /* access to OOB pointer */ \ + r0 = *(u8*)(r0 + 0); \ +l0_%=: /* exit */ \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("tc") +__description("bounds check based on reg_off + var_off + insn_off. test1") +__failure __msg("value_size=8 off=1073741825") +__naked void var_off_insn_off_test1(void) +{ + asm volatile (" \ + r6 = *(u32*)(r1 + %[__sk_buff_mark]); \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r6 &= 1; \ + r6 += %[__imm_0]; \ + r0 += r6; \ + r0 += %[__imm_0]; \ +l0_%=: r0 = *(u8*)(r0 + 3); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b), + __imm_const(__imm_0, (1 << 29) - 1), + __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark)) + : __clobber_all); +} + +SEC("tc") +__description("bounds check based on reg_off + var_off + insn_off. test2") +__failure __msg("value 1073741823") +__naked void var_off_insn_off_test2(void) +{ + asm volatile (" \ + r6 = *(u32*)(r1 + %[__sk_buff_mark]); \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r6 &= 1; \ + r6 += %[__imm_0]; \ + r0 += r6; \ + r0 += %[__imm_1]; \ +l0_%=: r0 = *(u8*)(r0 + 3); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b), + __imm_const(__imm_0, (1 << 30) - 1), + __imm_const(__imm_1, (1 << 29) - 1), + __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark)) + : __clobber_all); +} + +SEC("socket") +__description("bounds check after truncation of non-boundary-crossing range") +__success __success_unpriv __retval(0) +__naked void of_non_boundary_crossing_range(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + /* r1 = [0x00, 0xff] */ \ + r1 = *(u8*)(r0 + 0); \ + r2 = 1; \ + /* r2 = 0x10'0000'0000 */ \ + r2 <<= 36; \ + /* r1 = [0x10'0000'0000, 0x10'0000'00ff] */ \ + r1 += r2; \ + /* r1 = [0x10'7fff'ffff, 0x10'8000'00fe] */ \ + r1 += 0x7fffffff; \ + /* r1 = [0x00, 0xff] */ \ + w1 -= 0x7fffffff; \ + /* r1 = 0 */ \ + r1 >>= 8; \ + /* no-op */ \ + r0 += r1; \ + /* access at offset 0 */ \ + r0 = *(u8*)(r0 + 0); \ +l0_%=: /* exit */ \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds check after truncation of boundary-crossing range (1)") +__failure +/* not actually fully unbounded, but the bound is very high */ +__msg("value -4294967168 makes map_value pointer be out of bounds") +__failure_unpriv +__naked void of_boundary_crossing_range_1(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + /* r1 = [0x00, 0xff] */ \ + r1 = *(u8*)(r0 + 0); \ + r1 += %[__imm_0]; \ + /* r1 = [0xffff'ff80, 0x1'0000'007f] */ \ + r1 += %[__imm_0]; \ + /* r1 = [0xffff'ff80, 0xffff'ffff] or \ + * [0x0000'0000, 0x0000'007f] \ + */ \ + w1 += 0; \ + r1 -= %[__imm_0]; \ + /* r1 = [0x00, 0xff] or \ + * [0xffff'ffff'0000'0080, 0xffff'ffff'ffff'ffff]\ + */ \ + r1 -= %[__imm_0]; \ + /* error on OOB pointer computation */ \ + r0 += r1; \ + /* exit */ \ + r0 = 0; \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b), + __imm_const(__imm_0, 0xffffff80 >> 1) + : __clobber_all); +} + +SEC("socket") +__description("bounds check after truncation of boundary-crossing range (2)") +__failure __msg("value -4294967168 makes map_value pointer be out of bounds") +__failure_unpriv +__naked void of_boundary_crossing_range_2(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + /* r1 = [0x00, 0xff] */ \ + r1 = *(u8*)(r0 + 0); \ + r1 += %[__imm_0]; \ + /* r1 = [0xffff'ff80, 0x1'0000'007f] */ \ + r1 += %[__imm_0]; \ + /* r1 = [0xffff'ff80, 0xffff'ffff] or \ + * [0x0000'0000, 0x0000'007f] \ + * difference to previous test: truncation via MOV32\ + * instead of ALU32. \ + */ \ + w1 = w1; \ + r1 -= %[__imm_0]; \ + /* r1 = [0x00, 0xff] or \ + * [0xffff'ffff'0000'0080, 0xffff'ffff'ffff'ffff]\ + */ \ + r1 -= %[__imm_0]; \ + /* error on OOB pointer computation */ \ + r0 += r1; \ + /* exit */ \ + r0 = 0; \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b), + __imm_const(__imm_0, 0xffffff80 >> 1) + : __clobber_all); +} + +SEC("socket") +__description("bounds check after wrapping 32-bit addition") +__success __success_unpriv __retval(0) +__naked void after_wrapping_32_bit_addition(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + /* r1 = 0x7fff'ffff */ \ + r1 = 0x7fffffff; \ + /* r1 = 0xffff'fffe */ \ + r1 += 0x7fffffff; \ + /* r1 = 0 */ \ + w1 += 2; \ + /* no-op */ \ + r0 += r1; \ + /* access at offset 0 */ \ + r0 = *(u8*)(r0 + 0); \ +l0_%=: /* exit */ \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds check after shift with oversized count operand") +__failure __msg("R0 max value is outside of the allowed memory range") +__failure_unpriv +__naked void shift_with_oversized_count_operand(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r2 = 32; \ + r1 = 1; \ + /* r1 = (u32)1 << (u32)32 = ? */ \ + w1 <<= w2; \ + /* r1 = [0x0000, 0xffff] */ \ + r1 &= 0xffff; \ + /* computes unknown pointer, potentially OOB */ \ + r0 += r1; \ + /* potentially OOB access */ \ + r0 = *(u8*)(r0 + 0); \ +l0_%=: /* exit */ \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds check after right shift of maybe-negative number") +__failure __msg("R0 unbounded memory access") +__failure_unpriv +__naked void shift_of_maybe_negative_number(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + /* r1 = [0x00, 0xff] */ \ + r1 = *(u8*)(r0 + 0); \ + /* r1 = [-0x01, 0xfe] */ \ + r1 -= 1; \ + /* r1 = 0 or 0xff'ffff'ffff'ffff */ \ + r1 >>= 8; \ + /* r1 = 0 or 0xffff'ffff'ffff */ \ + r1 >>= 8; \ + /* computes unknown pointer, potentially OOB */ \ + r0 += r1; \ + /* potentially OOB access */ \ + r0 = *(u8*)(r0 + 0); \ +l0_%=: /* exit */ \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds check after 32-bit right shift with 64-bit input") +__failure __msg("math between map_value pointer and 4294967294 is not allowed") +__failure_unpriv +__naked void shift_with_64_bit_input(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = 2; \ + /* r1 = 1<<32 */ \ + r1 <<= 31; \ + /* r1 = 0 (NOT 2!) */ \ + w1 >>= 31; \ + /* r1 = 0xffff'fffe (NOT 0!) */ \ + w1 -= 2; \ + /* error on computing OOB pointer */ \ + r0 += r1; \ + /* exit */ \ + r0 = 0; \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds check map access with off+size signed 32bit overflow. test1") +__failure __msg("map_value pointer and 2147483646") +__failure_unpriv +__naked void size_signed_32bit_overflow_test1(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r0 += 0x7ffffffe; \ + r0 = *(u64*)(r0 + 0); \ + goto l1_%=; \ +l1_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds check map access with off+size signed 32bit overflow. test2") +__failure __msg("pointer offset 1073741822") +__msg_unpriv("R0 pointer arithmetic of map value goes out of range") +__naked void size_signed_32bit_overflow_test2(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r0 += 0x1fffffff; \ + r0 += 0x1fffffff; \ + r0 += 0x1fffffff; \ + r0 = *(u64*)(r0 + 0); \ + goto l1_%=; \ +l1_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds check map access with off+size signed 32bit overflow. test3") +__failure __msg("pointer offset -1073741822") +__msg_unpriv("R0 pointer arithmetic of map value goes out of range") +__naked void size_signed_32bit_overflow_test3(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r0 -= 0x1fffffff; \ + r0 -= 0x1fffffff; \ + r0 = *(u64*)(r0 + 2); \ + goto l1_%=; \ +l1_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds check map access with off+size signed 32bit overflow. test4") +__failure __msg("map_value pointer and 1000000000000") +__failure_unpriv +__naked void size_signed_32bit_overflow_test4(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r1 = 1000000; \ + r1 *= 1000000; \ + r0 += r1; \ + r0 = *(u64*)(r0 + 2); \ + goto l1_%=; \ +l1_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds check mixed 32bit and 64bit arithmetic. test1") +__success __failure_unpriv __msg_unpriv("R0 invalid mem access 'scalar'") +__retval(0) +__naked void _32bit_and_64bit_arithmetic_test1(void) +{ + asm volatile (" \ + r0 = 0; \ + r1 = -1; \ + r1 <<= 32; \ + r1 += 1; \ + /* r1 = 0xffffFFFF00000001 */ \ + if w1 > 1 goto l0_%=; \ + /* check ALU64 op keeps 32bit bounds */ \ + r1 += 1; \ + if w1 > 2 goto l0_%=; \ + goto l1_%=; \ +l0_%=: /* invalid ldx if bounds are lost above */ \ + r0 = *(u64*)(r0 - 1); \ +l1_%=: exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("bounds check mixed 32bit and 64bit arithmetic. test2") +__success __failure_unpriv __msg_unpriv("R0 invalid mem access 'scalar'") +__retval(0) +__naked void _32bit_and_64bit_arithmetic_test2(void) +{ + asm volatile (" \ + r0 = 0; \ + r1 = -1; \ + r1 <<= 32; \ + r1 += 1; \ + /* r1 = 0xffffFFFF00000001 */ \ + r2 = 3; \ + /* r1 = 0x2 */ \ + w1 += 1; \ + /* check ALU32 op zero extends 64bit bounds */ \ + if r1 > r2 goto l0_%=; \ + goto l1_%=; \ +l0_%=: /* invalid ldx if bounds are lost above */ \ + r0 = *(u64*)(r0 - 1); \ +l1_%=: exit; \ +" ::: __clobber_all); +} + +SEC("tc") +__description("assigning 32bit bounds to 64bit for wA = 0, wB = wA") +__success __retval(0) __flag(BPF_F_ANY_ALIGNMENT) +__naked void for_wa_0_wb_wa(void) +{ + asm volatile (" \ + r8 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r7 = *(u32*)(r1 + %[__sk_buff_data]); \ + w9 = 0; \ + w2 = w9; \ + r6 = r7; \ + r6 += r2; \ + r3 = r6; \ + r3 += 8; \ + if r3 > r8 goto l0_%=; \ + r5 = *(u32*)(r6 + 0); \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("socket") +__description("bounds check for reg = 0, reg xor 1") +__success __failure_unpriv +__msg_unpriv("R0 min value is outside of the allowed memory range") +__retval(0) +__naked void reg_0_reg_xor_1(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r1 = 0; \ + r1 ^= 1; \ + if r1 != 0 goto l1_%=; \ + r0 = *(u64*)(r0 + 8); \ +l1_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds check for reg32 = 0, reg32 xor 1") +__success __failure_unpriv +__msg_unpriv("R0 min value is outside of the allowed memory range") +__retval(0) +__naked void reg32_0_reg32_xor_1(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: w1 = 0; \ + w1 ^= 1; \ + if w1 != 0 goto l1_%=; \ + r0 = *(u64*)(r0 + 8); \ +l1_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds check for reg = 2, reg xor 3") +__success __failure_unpriv +__msg_unpriv("R0 min value is outside of the allowed memory range") +__retval(0) +__naked void reg_2_reg_xor_3(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r1 = 2; \ + r1 ^= 3; \ + if r1 > 0 goto l1_%=; \ + r0 = *(u64*)(r0 + 8); \ +l1_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds check for reg = any, reg xor 3") +__failure __msg("invalid access to map value") +__msg_unpriv("invalid access to map value") +__naked void reg_any_reg_xor_3(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r1 = *(u64*)(r0 + 0); \ + r1 ^= 3; \ + if r1 != 0 goto l1_%=; \ + r0 = *(u64*)(r0 + 8); \ +l1_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds check for reg32 = any, reg32 xor 3") +__failure __msg("invalid access to map value") +__msg_unpriv("invalid access to map value") +__naked void reg32_any_reg32_xor_3(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r1 = *(u64*)(r0 + 0); \ + w1 ^= 3; \ + if w1 != 0 goto l1_%=; \ + r0 = *(u64*)(r0 + 8); \ +l1_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds check for reg > 0, reg xor 3") +__success __failure_unpriv +__msg_unpriv("R0 min value is outside of the allowed memory range") +__retval(0) +__naked void reg_0_reg_xor_3(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r1 = *(u64*)(r0 + 0); \ + if r1 <= 0 goto l1_%=; \ + r1 ^= 3; \ + if r1 >= 0 goto l1_%=; \ + r0 = *(u64*)(r0 + 8); \ +l1_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds check for reg32 > 0, reg32 xor 3") +__success __failure_unpriv +__msg_unpriv("R0 min value is outside of the allowed memory range") +__retval(0) +__naked void reg32_0_reg32_xor_3(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r1 = *(u64*)(r0 + 0); \ + if w1 <= 0 goto l1_%=; \ + w1 ^= 3; \ + if w1 >= 0 goto l1_%=; \ + r0 = *(u64*)(r0 + 8); \ +l1_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds checks after 32-bit truncation. test 1") +__success __failure_unpriv __msg_unpriv("R0 leaks addr") +__retval(0) +__naked void _32_bit_truncation_test_1(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = *(u32*)(r0 + 0); \ + /* This used to reduce the max bound to 0x7fffffff */\ + if r1 == 0 goto l1_%=; \ + if r1 > 0x7fffffff goto l0_%=; \ +l1_%=: r0 = 0; \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("bounds checks after 32-bit truncation. test 2") +__success __failure_unpriv __msg_unpriv("R0 leaks addr") +__retval(0) +__naked void _32_bit_truncation_test_2(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = *(u32*)(r0 + 0); \ + if r1 s< 1 goto l1_%=; \ + if w1 s< 0 goto l0_%=; \ +l1_%=: r0 = 0; \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("xdp") +__description("bound check with JMP_JLT for crossing 64-bit signed boundary") +__success __retval(0) +__naked void crossing_64_bit_signed_boundary_1(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[xdp_md_data]); \ + r3 = *(u32*)(r1 + %[xdp_md_data_end]); \ + r1 = r2; \ + r1 += 1; \ + if r1 > r3 goto l0_%=; \ + r1 = *(u8*)(r2 + 0); \ + r0 = 0x7fffffffffffff10 ll; \ + r1 += r0; \ + r0 = 0x8000000000000000 ll; \ +l1_%=: r0 += 1; \ + /* r1 unsigned range is [0x7fffffffffffff10, 0x800000000000000f] */\ + if r0 < r1 goto l1_%=; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(xdp_md_data, offsetof(struct xdp_md, data)), + __imm_const(xdp_md_data_end, offsetof(struct xdp_md, data_end)) + : __clobber_all); +} + +SEC("xdp") +__description("bound check with JMP_JSLT for crossing 64-bit signed boundary") +__success __retval(0) +__naked void crossing_64_bit_signed_boundary_2(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[xdp_md_data]); \ + r3 = *(u32*)(r1 + %[xdp_md_data_end]); \ + r1 = r2; \ + r1 += 1; \ + if r1 > r3 goto l0_%=; \ + r1 = *(u8*)(r2 + 0); \ + r0 = 0x7fffffffffffff10 ll; \ + r1 += r0; \ + r2 = 0x8000000000000fff ll; \ + r0 = 0x8000000000000000 ll; \ +l1_%=: r0 += 1; \ + if r0 s> r2 goto l0_%=; \ + /* r1 signed range is [S64_MIN, S64_MAX] */ \ + if r0 s< r1 goto l1_%=; \ + r0 = 1; \ + exit; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(xdp_md_data, offsetof(struct xdp_md, data)), + __imm_const(xdp_md_data_end, offsetof(struct xdp_md, data_end)) + : __clobber_all); +} + +SEC("xdp") +__description("bound check for loop upper bound greater than U32_MAX") +__success __retval(0) +__naked void bound_greater_than_u32_max(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[xdp_md_data]); \ + r3 = *(u32*)(r1 + %[xdp_md_data_end]); \ + r1 = r2; \ + r1 += 1; \ + if r1 > r3 goto l0_%=; \ + r1 = *(u8*)(r2 + 0); \ + r0 = 0x100000000 ll; \ + r1 += r0; \ + r0 = 0x100000000 ll; \ +l1_%=: r0 += 1; \ + if r0 < r1 goto l1_%=; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(xdp_md_data, offsetof(struct xdp_md, data)), + __imm_const(xdp_md_data_end, offsetof(struct xdp_md, data_end)) + : __clobber_all); +} + +SEC("xdp") +__description("bound check with JMP32_JLT for crossing 32-bit signed boundary") +__success __retval(0) +__naked void crossing_32_bit_signed_boundary_1(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[xdp_md_data]); \ + r3 = *(u32*)(r1 + %[xdp_md_data_end]); \ + r1 = r2; \ + r1 += 1; \ + if r1 > r3 goto l0_%=; \ + r1 = *(u8*)(r2 + 0); \ + w0 = 0x7fffff10; \ + w1 += w0; \ + w0 = 0x80000000; \ +l1_%=: w0 += 1; \ + /* r1 unsigned range is [0, 0x8000000f] */ \ + if w0 < w1 goto l1_%=; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(xdp_md_data, offsetof(struct xdp_md, data)), + __imm_const(xdp_md_data_end, offsetof(struct xdp_md, data_end)) + : __clobber_all); +} + +SEC("xdp") +__description("bound check with JMP32_JSLT for crossing 32-bit signed boundary") +__success __retval(0) +__naked void crossing_32_bit_signed_boundary_2(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[xdp_md_data]); \ + r3 = *(u32*)(r1 + %[xdp_md_data_end]); \ + r1 = r2; \ + r1 += 1; \ + if r1 > r3 goto l0_%=; \ + r1 = *(u8*)(r2 + 0); \ + w0 = 0x7fffff10; \ + w1 += w0; \ + w2 = 0x80000fff; \ + w0 = 0x80000000; \ +l1_%=: w0 += 1; \ + if w0 s> w2 goto l0_%=; \ + /* r1 signed range is [S32_MIN, S32_MAX] */ \ + if w0 s< w1 goto l1_%=; \ + r0 = 1; \ + exit; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(xdp_md_data, offsetof(struct xdp_md, data)), + __imm_const(xdp_md_data_end, offsetof(struct xdp_md, data_end)) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_bpf_get_stack.c b/tools/testing/selftests/bpf/progs/verifier_bpf_get_stack.c new file mode 100644 index 000000000000..325a2bab4a71 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_bpf_get_stack.c @@ -0,0 +1,124 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/bpf_get_stack.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +#define MAX_ENTRIES 11 + +struct test_val { + unsigned int index; + int foo[MAX_ENTRIES]; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, int); + __type(value, struct test_val); +} map_array_48b SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 1); + __type(key, long long); + __type(value, struct test_val); +} map_hash_48b SEC(".maps"); + +SEC("tracepoint") +__description("bpf_get_stack return R0 within range") +__success +__naked void stack_return_r0_within_range(void) +{ + asm volatile (" \ + r6 = r1; \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r7 = r0; \ + r9 = %[__imm_0]; \ + r1 = r6; \ + r2 = r7; \ + r3 = %[__imm_0]; \ + r4 = 256; \ + call %[bpf_get_stack]; \ + r1 = 0; \ + r8 = r0; \ + r8 <<= 32; \ + r8 s>>= 32; \ + if r1 s> r8 goto l0_%=; \ + r9 -= r8; \ + r2 = r7; \ + r2 += r8; \ + r1 = r9; \ + r1 <<= 32; \ + r1 s>>= 32; \ + r3 = r2; \ + r3 += r1; \ + r1 = r7; \ + r5 = %[__imm_0]; \ + r1 += r5; \ + if r3 >= r1 goto l0_%=; \ + r1 = r6; \ + r3 = r9; \ + r4 = 0; \ + call %[bpf_get_stack]; \ +l0_%=: exit; \ +" : + : __imm(bpf_get_stack), + __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_48b), + __imm_const(__imm_0, sizeof(struct test_val) / 2) + : __clobber_all); +} + +SEC("iter/task") +__description("bpf_get_task_stack return R0 range is refined") +__success +__naked void return_r0_range_is_refined(void) +{ + asm volatile (" \ + r6 = *(u64*)(r1 + 0); \ + r6 = *(u64*)(r6 + 0); /* ctx->meta->seq */\ + r7 = *(u64*)(r1 + 8); /* ctx->task */\ + r1 = %[map_array_48b] ll; /* fixup_map_array_48b */\ + r2 = 0; \ + *(u64*)(r10 - 8) = r2; \ + r2 = r10; \ + r2 += -8; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: if r7 != 0 goto l1_%=; \ + r0 = 0; \ + exit; \ +l1_%=: r1 = r7; \ + r2 = r0; \ + r9 = r0; /* keep buf for seq_write */\ + r3 = 48; \ + r4 = 0; \ + call %[bpf_get_task_stack]; \ + if r0 s> 0 goto l2_%=; \ + r0 = 0; \ + exit; \ +l2_%=: r1 = r6; \ + r2 = r9; \ + r3 = r0; \ + call %[bpf_seq_write]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_task_stack), + __imm(bpf_map_lookup_elem), + __imm(bpf_seq_write), + __imm_addr(map_array_48b) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_btf_ctx_access.c b/tools/testing/selftests/bpf/progs/verifier_btf_ctx_access.c new file mode 100644 index 000000000000..a570e48b917a --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_btf_ctx_access.c @@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/btf_ctx_access.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +SEC("fentry/bpf_modify_return_test") +__description("btf_ctx_access accept") +__success __retval(0) +__naked void btf_ctx_access_accept(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + 8); /* load 2nd argument value (int pointer) */\ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("fentry/bpf_fentry_test9") +__description("btf_ctx_access u32 pointer accept") +__success __retval(0) +__naked void ctx_access_u32_pointer_accept(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + 0); /* load 1nd argument value (u32 pointer) */\ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_ctx.c b/tools/testing/selftests/bpf/progs/verifier_ctx.c new file mode 100644 index 000000000000..a83809a1dbbf --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_ctx.c @@ -0,0 +1,221 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/ctx.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +SEC("tc") +__description("context stores via BPF_ATOMIC") +__failure __msg("BPF_ATOMIC stores into R1 ctx is not allowed") +__naked void context_stores_via_bpf_atomic(void) +{ + asm volatile (" \ + r0 = 0; \ + lock *(u32 *)(r1 + %[__sk_buff_mark]) += w0; \ + exit; \ +" : + : __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark)) + : __clobber_all); +} + +SEC("tc") +__description("arithmetic ops make PTR_TO_CTX unusable") +__failure __msg("dereference of modified ctx ptr") +__naked void make_ptr_to_ctx_unusable(void) +{ + asm volatile (" \ + r1 += %[__imm_0]; \ + r0 = *(u32*)(r1 + %[__sk_buff_mark]); \ + exit; \ +" : + : __imm_const(__imm_0, + offsetof(struct __sk_buff, data) - offsetof(struct __sk_buff, mark)), + __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark)) + : __clobber_all); +} + +SEC("tc") +__description("pass unmodified ctx pointer to helper") +__success __retval(0) +__naked void unmodified_ctx_pointer_to_helper(void) +{ + asm volatile (" \ + r2 = 0; \ + call %[bpf_csum_update]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_csum_update) + : __clobber_all); +} + +SEC("tc") +__description("pass modified ctx pointer to helper, 1") +__failure __msg("negative offset ctx ptr R1 off=-612 disallowed") +__naked void ctx_pointer_to_helper_1(void) +{ + asm volatile (" \ + r1 += -612; \ + r2 = 0; \ + call %[bpf_csum_update]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_csum_update) + : __clobber_all); +} + +SEC("socket") +__description("pass modified ctx pointer to helper, 2") +__failure __msg("negative offset ctx ptr R1 off=-612 disallowed") +__failure_unpriv __msg_unpriv("negative offset ctx ptr R1 off=-612 disallowed") +__naked void ctx_pointer_to_helper_2(void) +{ + asm volatile (" \ + r1 += -612; \ + call %[bpf_get_socket_cookie]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_socket_cookie) + : __clobber_all); +} + +SEC("tc") +__description("pass modified ctx pointer to helper, 3") +__failure __msg("variable ctx access var_off=(0x0; 0x4)") +__naked void ctx_pointer_to_helper_3(void) +{ + asm volatile (" \ + r3 = *(u32*)(r1 + 0); \ + r3 &= 4; \ + r1 += r3; \ + r2 = 0; \ + call %[bpf_csum_update]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_csum_update) + : __clobber_all); +} + +SEC("cgroup/sendmsg6") +__description("pass ctx or null check, 1: ctx") +__success +__naked void or_null_check_1_ctx(void) +{ + asm volatile (" \ + call %[bpf_get_netns_cookie]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_netns_cookie) + : __clobber_all); +} + +SEC("cgroup/sendmsg6") +__description("pass ctx or null check, 2: null") +__success +__naked void or_null_check_2_null(void) +{ + asm volatile (" \ + r1 = 0; \ + call %[bpf_get_netns_cookie]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_netns_cookie) + : __clobber_all); +} + +SEC("cgroup/sendmsg6") +__description("pass ctx or null check, 3: 1") +__failure __msg("R1 type=scalar expected=ctx") +__naked void or_null_check_3_1(void) +{ + asm volatile (" \ + r1 = 1; \ + call %[bpf_get_netns_cookie]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_netns_cookie) + : __clobber_all); +} + +SEC("cgroup/sendmsg6") +__description("pass ctx or null check, 4: ctx - const") +__failure __msg("negative offset ctx ptr R1 off=-612 disallowed") +__naked void null_check_4_ctx_const(void) +{ + asm volatile (" \ + r1 += -612; \ + call %[bpf_get_netns_cookie]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_netns_cookie) + : __clobber_all); +} + +SEC("cgroup/connect4") +__description("pass ctx or null check, 5: null (connect)") +__success +__naked void null_check_5_null_connect(void) +{ + asm volatile (" \ + r1 = 0; \ + call %[bpf_get_netns_cookie]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_netns_cookie) + : __clobber_all); +} + +SEC("cgroup/post_bind4") +__description("pass ctx or null check, 6: null (bind)") +__success +__naked void null_check_6_null_bind(void) +{ + asm volatile (" \ + r1 = 0; \ + call %[bpf_get_netns_cookie]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_netns_cookie) + : __clobber_all); +} + +SEC("cgroup/post_bind4") +__description("pass ctx or null check, 7: ctx (bind)") +__success +__naked void null_check_7_ctx_bind(void) +{ + asm volatile (" \ + call %[bpf_get_socket_cookie]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_socket_cookie) + : __clobber_all); +} + +SEC("cgroup/post_bind4") +__description("pass ctx or null check, 8: null (bind)") +__failure __msg("R1 type=scalar expected=ctx") +__naked void null_check_8_null_bind(void) +{ + asm volatile (" \ + r1 = 0; \ + call %[bpf_get_socket_cookie]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_socket_cookie) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_d_path.c b/tools/testing/selftests/bpf/progs/verifier_d_path.c new file mode 100644 index 000000000000..ec79cbcfde91 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_d_path.c @@ -0,0 +1,48 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/d_path.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +SEC("fentry/dentry_open") +__description("d_path accept") +__success __retval(0) +__naked void d_path_accept(void) +{ + asm volatile (" \ + r1 = *(u32*)(r1 + 0); \ + r2 = r10; \ + r2 += -8; \ + r6 = 0; \ + *(u64*)(r2 + 0) = r6; \ + r3 = 8 ll; \ + call %[bpf_d_path]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_d_path) + : __clobber_all); +} + +SEC("fentry/d_path") +__description("d_path reject") +__failure __msg("helper call is not allowed in probe") +__naked void d_path_reject(void) +{ + asm volatile (" \ + r1 = *(u32*)(r1 + 0); \ + r2 = r10; \ + r2 += -8; \ + r6 = 0; \ + *(u64*)(r2 + 0) = r6; \ + r3 = 8 ll; \ + call %[bpf_d_path]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_d_path) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_direct_packet_access.c b/tools/testing/selftests/bpf/progs/verifier_direct_packet_access.c new file mode 100644 index 000000000000..99a23dea8233 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_direct_packet_access.c @@ -0,0 +1,803 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/direct_packet_access.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +SEC("tc") +__description("pkt_end - pkt_start is allowed") +__success __retval(TEST_DATA_LEN) +__naked void end_pkt_start_is_allowed(void) +{ + asm volatile (" \ + r0 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r0 -= r2; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test1") +__success __retval(0) +__naked void direct_packet_access_test1(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r0 > r3 goto l0_%=; \ + r0 = *(u8*)(r2 + 0); \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test2") +__success __retval(0) +__naked void direct_packet_access_test2(void) +{ + asm volatile (" \ + r0 = 1; \ + r4 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data]); \ + r5 = r3; \ + r5 += 14; \ + if r5 > r4 goto l0_%=; \ + r0 = *(u8*)(r3 + 7); \ + r4 = *(u8*)(r3 + 12); \ + r4 *= 14; \ + r3 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 += r4; \ + r2 = *(u32*)(r1 + %[__sk_buff_len]); \ + r2 <<= 49; \ + r2 >>= 49; \ + r3 += r2; \ + r2 = r3; \ + r2 += 8; \ + r1 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + if r2 > r1 goto l1_%=; \ + r1 = *(u8*)(r3 + 4); \ +l1_%=: r0 = 0; \ +l0_%=: exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)), + __imm_const(__sk_buff_len, offsetof(struct __sk_buff, len)) + : __clobber_all); +} + +SEC("socket") +__description("direct packet access: test3") +__failure __msg("invalid bpf_context access off=76") +__failure_unpriv +__naked void direct_packet_access_test3(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test4 (write)") +__success __retval(0) +__naked void direct_packet_access_test4_write(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r0 > r3 goto l0_%=; \ + *(u8*)(r2 + 0) = r2; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test5 (pkt_end >= reg, good access)") +__success __retval(0) +__naked void pkt_end_reg_good_access(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r3 >= r0 goto l0_%=; \ + r0 = 1; \ + exit; \ +l0_%=: r0 = *(u8*)(r2 + 0); \ + r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test6 (pkt_end >= reg, bad access)") +__failure __msg("invalid access to packet") +__naked void pkt_end_reg_bad_access(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r3 >= r0 goto l0_%=; \ + r0 = *(u8*)(r2 + 0); \ + r0 = 1; \ + exit; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test7 (pkt_end >= reg, both accesses)") +__failure __msg("invalid access to packet") +__naked void pkt_end_reg_both_accesses(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r3 >= r0 goto l0_%=; \ + r0 = *(u8*)(r2 + 0); \ + r0 = 1; \ + exit; \ +l0_%=: r0 = *(u8*)(r2 + 0); \ + r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test8 (double test, variant 1)") +__success __retval(0) +__naked void test8_double_test_variant_1(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r3 >= r0 goto l0_%=; \ + if r0 > r3 goto l1_%=; \ + r0 = *(u8*)(r2 + 0); \ +l1_%=: r0 = 1; \ + exit; \ +l0_%=: r0 = *(u8*)(r2 + 0); \ + r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test9 (double test, variant 2)") +__success __retval(0) +__naked void test9_double_test_variant_2(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r3 >= r0 goto l0_%=; \ + r0 = 1; \ + exit; \ +l0_%=: if r0 > r3 goto l1_%=; \ + r0 = *(u8*)(r2 + 0); \ +l1_%=: r0 = *(u8*)(r2 + 0); \ + r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test10 (write invalid)") +__failure __msg("invalid access to packet") +__naked void packet_access_test10_write_invalid(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r0 > r3 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: *(u8*)(r2 + 0) = r2; \ + r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test11 (shift, good access)") +__success __retval(1) +__naked void access_test11_shift_good_access(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 22; \ + if r0 > r3 goto l0_%=; \ + r3 = 144; \ + r5 = r3; \ + r5 += 23; \ + r5 >>= 3; \ + r6 = r2; \ + r6 += r5; \ + r0 = 1; \ + exit; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test12 (and, good access)") +__success __retval(1) +__naked void access_test12_and_good_access(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 22; \ + if r0 > r3 goto l0_%=; \ + r3 = 144; \ + r5 = r3; \ + r5 += 23; \ + r5 &= 15; \ + r6 = r2; \ + r6 += r5; \ + r0 = 1; \ + exit; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test13 (branches, good access)") +__success __retval(1) +__naked void access_test13_branches_good_access(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 22; \ + if r0 > r3 goto l0_%=; \ + r3 = *(u32*)(r1 + %[__sk_buff_mark]); \ + r4 = 1; \ + if r3 > r4 goto l1_%=; \ + r3 = 14; \ + goto l2_%=; \ +l1_%=: r3 = 24; \ +l2_%=: r5 = r3; \ + r5 += 23; \ + r5 &= 15; \ + r6 = r2; \ + r6 += r5; \ + r0 = 1; \ + exit; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)), + __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test14 (pkt_ptr += 0, CONST_IMM, good access)") +__success __retval(1) +__naked void _0_const_imm_good_access(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 22; \ + if r0 > r3 goto l0_%=; \ + r5 = 12; \ + r5 >>= 4; \ + r6 = r2; \ + r6 += r5; \ + r0 = *(u8*)(r6 + 0); \ + r0 = 1; \ + exit; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test15 (spill with xadd)") +__failure __msg("R2 invalid mem access 'scalar'") +__flag(BPF_F_ANY_ALIGNMENT) +__naked void access_test15_spill_with_xadd(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r0 > r3 goto l0_%=; \ + r5 = 4096; \ + r4 = r10; \ + r4 += -8; \ + *(u64*)(r4 + 0) = r2; \ + lock *(u64 *)(r4 + 0) += r5; \ + r2 = *(u64*)(r4 + 0); \ + *(u32*)(r2 + 0) = r5; \ + r0 = 0; \ +l0_%=: exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test16 (arith on data_end)") +__failure __msg("R3 pointer arithmetic on pkt_end") +__naked void test16_arith_on_data_end(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + r3 += 16; \ + if r0 > r3 goto l0_%=; \ + *(u8*)(r2 + 0) = r2; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test17 (pruning, alignment)") +__failure __msg("misaligned packet access off 2+(0x0; 0x0)+15+-4 size 4") +__flag(BPF_F_STRICT_ALIGNMENT) +__naked void packet_access_test17_pruning_alignment(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r7 = *(u32*)(r1 + %[__sk_buff_mark]); \ + r0 = r2; \ + r0 += 14; \ + if r7 > 1 goto l0_%=; \ +l2_%=: if r0 > r3 goto l1_%=; \ + *(u32*)(r0 - 4) = r0; \ +l1_%=: r0 = 0; \ + exit; \ +l0_%=: r0 += 1; \ + goto l2_%=; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)), + __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test18 (imm += pkt_ptr, 1)") +__success __retval(0) +__naked void test18_imm_pkt_ptr_1(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = 8; \ + r0 += r2; \ + if r0 > r3 goto l0_%=; \ + *(u8*)(r2 + 0) = r2; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test19 (imm += pkt_ptr, 2)") +__success __retval(0) +__naked void test19_imm_pkt_ptr_2(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r0 > r3 goto l0_%=; \ + r4 = 4; \ + r4 += r2; \ + *(u8*)(r4 + 0) = r4; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test20 (x += pkt_ptr, 1)") +__success __retval(0) __flag(BPF_F_ANY_ALIGNMENT) +__naked void test20_x_pkt_ptr_1(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = 0xffffffff; \ + *(u64*)(r10 - 8) = r0; \ + r0 = *(u64*)(r10 - 8); \ + r0 &= 0x7fff; \ + r4 = r0; \ + r4 += r2; \ + r5 = r4; \ + r4 += %[__imm_0]; \ + if r4 > r3 goto l0_%=; \ + *(u64*)(r5 + 0) = r4; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__imm_0, 0x7fff - 1), + __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test21 (x += pkt_ptr, 2)") +__success __retval(0) __flag(BPF_F_ANY_ALIGNMENT) +__naked void test21_x_pkt_ptr_2(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r0 > r3 goto l0_%=; \ + r4 = 0xffffffff; \ + *(u64*)(r10 - 8) = r4; \ + r4 = *(u64*)(r10 - 8); \ + r4 &= 0x7fff; \ + r4 += r2; \ + r5 = r4; \ + r4 += %[__imm_0]; \ + if r4 > r3 goto l0_%=; \ + *(u64*)(r5 + 0) = r4; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__imm_0, 0x7fff - 1), + __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test22 (x += pkt_ptr, 3)") +__success __retval(0) __flag(BPF_F_ANY_ALIGNMENT) +__naked void test22_x_pkt_ptr_3(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + *(u64*)(r10 - 8) = r2; \ + *(u64*)(r10 - 16) = r3; \ + r3 = *(u64*)(r10 - 16); \ + if r0 > r3 goto l0_%=; \ + r2 = *(u64*)(r10 - 8); \ + r4 = 0xffffffff; \ + lock *(u64 *)(r10 - 8) += r4; \ + r4 = *(u64*)(r10 - 8); \ + r4 >>= 49; \ + r4 += r2; \ + r0 = r4; \ + r0 += 2; \ + if r0 > r3 goto l0_%=; \ + r2 = 1; \ + *(u16*)(r4 + 0) = r2; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test23 (x += pkt_ptr, 4)") +__failure __msg("invalid access to packet, off=0 size=8, R5(id=2,off=0,r=0)") +__flag(BPF_F_ANY_ALIGNMENT) +__naked void test23_x_pkt_ptr_4(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = *(u32*)(r1 + %[__sk_buff_mark]); \ + *(u64*)(r10 - 8) = r0; \ + r0 = *(u64*)(r10 - 8); \ + r0 &= 0xffff; \ + r4 = r0; \ + r0 = 31; \ + r0 += r4; \ + r0 += r2; \ + r5 = r0; \ + r0 += %[__imm_0]; \ + if r0 > r3 goto l0_%=; \ + *(u64*)(r5 + 0) = r0; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__imm_0, 0xffff - 1), + __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)), + __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test24 (x += pkt_ptr, 5)") +__success __retval(0) __flag(BPF_F_ANY_ALIGNMENT) +__naked void test24_x_pkt_ptr_5(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = 0xffffffff; \ + *(u64*)(r10 - 8) = r0; \ + r0 = *(u64*)(r10 - 8); \ + r0 &= 0xff; \ + r4 = r0; \ + r0 = 64; \ + r0 += r4; \ + r0 += r2; \ + r5 = r0; \ + r0 += %[__imm_0]; \ + if r0 > r3 goto l0_%=; \ + *(u64*)(r5 + 0) = r0; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__imm_0, 0x7fff - 1), + __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test25 (marking on <, good access)") +__success __retval(0) +__naked void test25_marking_on_good_access(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r0 < r3 goto l0_%=; \ +l1_%=: r0 = 0; \ + exit; \ +l0_%=: r0 = *(u8*)(r2 + 0); \ + goto l1_%=; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test26 (marking on <, bad access)") +__failure __msg("invalid access to packet") +__naked void test26_marking_on_bad_access(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r0 < r3 goto l0_%=; \ + r0 = *(u8*)(r2 + 0); \ +l1_%=: r0 = 0; \ + exit; \ +l0_%=: goto l1_%=; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test27 (marking on <=, good access)") +__success __retval(1) +__naked void test27_marking_on_good_access(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r3 <= r0 goto l0_%=; \ + r0 = *(u8*)(r2 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test28 (marking on <=, bad access)") +__failure __msg("invalid access to packet") +__naked void test28_marking_on_bad_access(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r3 <= r0 goto l0_%=; \ +l1_%=: r0 = 1; \ + exit; \ +l0_%=: r0 = *(u8*)(r2 + 0); \ + goto l1_%=; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("direct packet access: test29 (reg > pkt_end in subprog)") +__success __retval(0) +__naked void reg_pkt_end_in_subprog(void) +{ + asm volatile (" \ + r6 = *(u32*)(r1 + %[__sk_buff_data]); \ + r2 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r3 = r6; \ + r3 += 8; \ + call reg_pkt_end_in_subprog__1; \ + if r0 == 0 goto l0_%=; \ + r0 = *(u8*)(r6 + 0); \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void reg_pkt_end_in_subprog__1(void) +{ + asm volatile (" \ + r0 = 0; \ + if r3 > r2 goto l0_%=; \ + r0 = 1; \ +l0_%=: exit; \ +" ::: __clobber_all); +} + +SEC("tc") +__description("direct packet access: test30 (check_id() in regsafe(), bad access)") +__failure __msg("invalid access to packet, off=0 size=1, R2") +__flag(BPF_F_TEST_STATE_FREQ) +__naked void id_in_regsafe_bad_access(void) +{ + asm volatile (" \ + /* r9 = ctx */ \ + r9 = r1; \ + /* r7 = ktime_get_ns() */ \ + call %[bpf_ktime_get_ns]; \ + r7 = r0; \ + /* r6 = ktime_get_ns() */ \ + call %[bpf_ktime_get_ns]; \ + r6 = r0; \ + /* r2 = ctx->data \ + * r3 = ctx->data \ + * r4 = ctx->data_end \ + */ \ + r2 = *(u32*)(r9 + %[__sk_buff_data]); \ + r3 = *(u32*)(r9 + %[__sk_buff_data]); \ + r4 = *(u32*)(r9 + %[__sk_buff_data_end]); \ + /* if r6 > 100 goto exit \ + * if r7 > 100 goto exit \ + */ \ + if r6 > 100 goto l0_%=; \ + if r7 > 100 goto l0_%=; \ + /* r2 += r6 ; this forces assignment of ID to r2\ + * r2 += 1 ; get some fixed off for r2\ + * r3 += r7 ; this forces assignment of ID to r3\ + * r3 += 1 ; get some fixed off for r3\ + */ \ + r2 += r6; \ + r2 += 1; \ + r3 += r7; \ + r3 += 1; \ + /* if r6 > r7 goto +1 ; no new information about the state is derived from\ + * ; this check, thus produced verifier states differ\ + * ; only in 'insn_idx' \ + * r2 = r3 ; optionally share ID between r2 and r3\ + */ \ + if r6 != r7 goto l1_%=; \ + r2 = r3; \ +l1_%=: /* if r3 > ctx->data_end goto exit */ \ + if r3 > r4 goto l0_%=; \ + /* r5 = *(u8 *) (r2 - 1) ; access packet memory using r2,\ + * ; this is not always safe\ + */ \ + r5 = *(u8*)(r2 - 1); \ +l0_%=: /* exit(0) */ \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_ktime_get_ns), + __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c b/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c new file mode 100644 index 000000000000..bf16b00502f2 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c @@ -0,0 +1,213 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/jeq_infer_not_null.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +struct { + __uint(type, BPF_MAP_TYPE_XSKMAP); + __uint(max_entries, 1); + __type(key, int); + __type(value, int); +} map_xskmap SEC(".maps"); + +/* This is equivalent to the following program: + * + * r6 = skb->sk; + * r7 = sk_fullsock(r6); + * r0 = sk_fullsock(r6); + * if (r0 == 0) return 0; (a) + * if (r0 != r7) return 0; (b) + * *r7->type; (c) + * return 0; + * + * It is safe to dereference r7 at point (c), because of (a) and (b). + * The test verifies that relation r0 == r7 is propagated from (b) to (c). + */ +SEC("cgroup/skb") +__description("jne/jeq infer not null, PTR_TO_SOCKET_OR_NULL -> PTR_TO_SOCKET for JNE false branch") +__success __failure_unpriv __msg_unpriv("R7 pointer comparison") +__retval(0) +__naked void socket_for_jne_false_branch(void) +{ + asm volatile (" \ + /* r6 = skb->sk; */ \ + r6 = *(u64*)(r1 + %[__sk_buff_sk]); \ + /* if (r6 == 0) return 0; */ \ + if r6 == 0 goto l0_%=; \ + /* r7 = sk_fullsock(skb); */ \ + r1 = r6; \ + call %[bpf_sk_fullsock]; \ + r7 = r0; \ + /* r0 = sk_fullsock(skb); */ \ + r1 = r6; \ + call %[bpf_sk_fullsock]; \ + /* if (r0 == null) return 0; */ \ + if r0 == 0 goto l0_%=; \ + /* if (r0 == r7) r0 = *(r7->type); */ \ + if r0 != r7 goto l0_%=; /* Use ! JNE ! */\ + r0 = *(u32*)(r7 + %[bpf_sock_type]); \ +l0_%=: /* return 0 */ \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_sock_type, offsetof(struct bpf_sock, type)) + : __clobber_all); +} + +/* Same as above, but verify that another branch of JNE still + * prohibits access to PTR_MAYBE_NULL. + */ +SEC("cgroup/skb") +__description("jne/jeq infer not null, PTR_TO_SOCKET_OR_NULL unchanged for JNE true branch") +__failure __msg("R7 invalid mem access 'sock_or_null'") +__failure_unpriv __msg_unpriv("R7 pointer comparison") +__naked void unchanged_for_jne_true_branch(void) +{ + asm volatile (" \ + /* r6 = skb->sk */ \ + r6 = *(u64*)(r1 + %[__sk_buff_sk]); \ + /* if (r6 == 0) return 0; */ \ + if r6 == 0 goto l0_%=; \ + /* r7 = sk_fullsock(skb); */ \ + r1 = r6; \ + call %[bpf_sk_fullsock]; \ + r7 = r0; \ + /* r0 = sk_fullsock(skb); */ \ + r1 = r6; \ + call %[bpf_sk_fullsock]; \ + /* if (r0 == null) return 0; */ \ + if r0 != 0 goto l0_%=; \ + /* if (r0 == r7) return 0; */ \ + if r0 != r7 goto l1_%=; /* Use ! JNE ! */\ + goto l0_%=; \ +l1_%=: /* r0 = *(r7->type); */ \ + r0 = *(u32*)(r7 + %[bpf_sock_type]); \ +l0_%=: /* return 0 */ \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_sock_type, offsetof(struct bpf_sock, type)) + : __clobber_all); +} + +/* Same as a first test, but not null should be inferred for JEQ branch */ +SEC("cgroup/skb") +__description("jne/jeq infer not null, PTR_TO_SOCKET_OR_NULL -> PTR_TO_SOCKET for JEQ true branch") +__success __failure_unpriv __msg_unpriv("R7 pointer comparison") +__retval(0) +__naked void socket_for_jeq_true_branch(void) +{ + asm volatile (" \ + /* r6 = skb->sk; */ \ + r6 = *(u64*)(r1 + %[__sk_buff_sk]); \ + /* if (r6 == null) return 0; */ \ + if r6 == 0 goto l0_%=; \ + /* r7 = sk_fullsock(skb); */ \ + r1 = r6; \ + call %[bpf_sk_fullsock]; \ + r7 = r0; \ + /* r0 = sk_fullsock(skb); */ \ + r1 = r6; \ + call %[bpf_sk_fullsock]; \ + /* if (r0 == null) return 0; */ \ + if r0 == 0 goto l0_%=; \ + /* if (r0 != r7) return 0; */ \ + if r0 == r7 goto l1_%=; /* Use ! JEQ ! */\ + goto l0_%=; \ +l1_%=: /* r0 = *(r7->type); */ \ + r0 = *(u32*)(r7 + %[bpf_sock_type]); \ +l0_%=: /* return 0; */ \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_sock_type, offsetof(struct bpf_sock, type)) + : __clobber_all); +} + +/* Same as above, but verify that another branch of JNE still + * prohibits access to PTR_MAYBE_NULL. + */ +SEC("cgroup/skb") +__description("jne/jeq infer not null, PTR_TO_SOCKET_OR_NULL unchanged for JEQ false branch") +__failure __msg("R7 invalid mem access 'sock_or_null'") +__failure_unpriv __msg_unpriv("R7 pointer comparison") +__naked void unchanged_for_jeq_false_branch(void) +{ + asm volatile (" \ + /* r6 = skb->sk; */ \ + r6 = *(u64*)(r1 + %[__sk_buff_sk]); \ + /* if (r6 == null) return 0; */ \ + if r6 == 0 goto l0_%=; \ + /* r7 = sk_fullsock(skb); */ \ + r1 = r6; \ + call %[bpf_sk_fullsock]; \ + r7 = r0; \ + /* r0 = sk_fullsock(skb); */ \ + r1 = r6; \ + call %[bpf_sk_fullsock]; \ + /* if (r0 == null) return 0; */ \ + if r0 == 0 goto l0_%=; \ + /* if (r0 != r7) r0 = *(r7->type); */ \ + if r0 == r7 goto l0_%=; /* Use ! JEQ ! */\ + r0 = *(u32*)(r7 + %[bpf_sock_type]); \ +l0_%=: /* return 0; */ \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_sock_type, offsetof(struct bpf_sock, type)) + : __clobber_all); +} + +/* Maps are treated in a different branch of `mark_ptr_not_null_reg`, + * so separate test for maps case. + */ +SEC("xdp") +__description("jne/jeq infer not null, PTR_TO_MAP_VALUE_OR_NULL -> PTR_TO_MAP_VALUE") +__success __retval(0) +__naked void null_ptr_to_map_value(void) +{ + asm volatile (" \ + /* r9 = &some stack to use as key */ \ + r1 = 0; \ + *(u32*)(r10 - 8) = r1; \ + r9 = r10; \ + r9 += -8; \ + /* r8 = process local map */ \ + r8 = %[map_xskmap] ll; \ + /* r6 = map_lookup_elem(r8, r9); */ \ + r1 = r8; \ + r2 = r9; \ + call %[bpf_map_lookup_elem]; \ + r6 = r0; \ + /* r7 = map_lookup_elem(r8, r9); */ \ + r1 = r8; \ + r2 = r9; \ + call %[bpf_map_lookup_elem]; \ + r7 = r0; \ + /* if (r6 == 0) return 0; */ \ + if r6 == 0 goto l0_%=; \ + /* if (r6 != r7) return 0; */ \ + if r6 != r7 goto l0_%=; \ + /* read *r7; */ \ + r0 = *(u32*)(r7 + %[bpf_xdp_sock_queue_id]); \ +l0_%=: /* return 0; */ \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_xskmap), + __imm_const(bpf_xdp_sock_queue_id, offsetof(struct bpf_xdp_sock, queue_id)) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_loops1.c b/tools/testing/selftests/bpf/progs/verifier_loops1.c new file mode 100644 index 000000000000..5bc86af80a9a --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_loops1.c @@ -0,0 +1,259 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/loops1.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +SEC("xdp") +__description("bounded loop, count to 4") +__success __retval(4) +__naked void bounded_loop_count_to_4(void) +{ + asm volatile (" \ + r0 = 0; \ +l0_%=: r0 += 1; \ + if r0 < 4 goto l0_%=; \ + exit; \ +" ::: __clobber_all); +} + +SEC("tracepoint") +__description("bounded loop, count to 20") +__success +__naked void bounded_loop_count_to_20(void) +{ + asm volatile (" \ + r0 = 0; \ +l0_%=: r0 += 3; \ + if r0 < 20 goto l0_%=; \ + exit; \ +" ::: __clobber_all); +} + +SEC("tracepoint") +__description("bounded loop, count from positive unknown to 4") +__success +__naked void from_positive_unknown_to_4(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + if r0 s< 0 goto l0_%=; \ +l1_%=: r0 += 1; \ + if r0 < 4 goto l1_%=; \ +l0_%=: exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("tracepoint") +__description("bounded loop, count from totally unknown to 4") +__success +__naked void from_totally_unknown_to_4(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ +l0_%=: r0 += 1; \ + if r0 < 4 goto l0_%=; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("tracepoint") +__description("bounded loop, count to 4 with equality") +__success +__naked void count_to_4_with_equality(void) +{ + asm volatile (" \ + r0 = 0; \ +l0_%=: r0 += 1; \ + if r0 != 4 goto l0_%=; \ + exit; \ +" ::: __clobber_all); +} + +SEC("tracepoint") +__description("bounded loop, start in the middle") +__failure __msg("back-edge") +__naked void loop_start_in_the_middle(void) +{ + asm volatile (" \ + r0 = 0; \ + goto l0_%=; \ +l1_%=: r0 += 1; \ +l0_%=: if r0 < 4 goto l1_%=; \ + exit; \ +" ::: __clobber_all); +} + +SEC("xdp") +__description("bounded loop containing a forward jump") +__success __retval(4) +__naked void loop_containing_a_forward_jump(void) +{ + asm volatile (" \ + r0 = 0; \ +l1_%=: r0 += 1; \ + if r0 == r0 goto l0_%=; \ +l0_%=: if r0 < 4 goto l1_%=; \ + exit; \ +" ::: __clobber_all); +} + +SEC("tracepoint") +__description("bounded loop that jumps out rather than in") +__success +__naked void jumps_out_rather_than_in(void) +{ + asm volatile (" \ + r6 = 0; \ +l1_%=: r6 += 1; \ + if r6 > 10000 goto l0_%=; \ + call %[bpf_get_prandom_u32]; \ + goto l1_%=; \ +l0_%=: exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("tracepoint") +__description("infinite loop after a conditional jump") +__failure __msg("program is too large") +__naked void loop_after_a_conditional_jump(void) +{ + asm volatile (" \ + r0 = 5; \ + if r0 < 4 goto l0_%=; \ +l1_%=: r0 += 1; \ + goto l1_%=; \ +l0_%=: exit; \ +" ::: __clobber_all); +} + +SEC("tracepoint") +__description("bounded recursion") +__failure __msg("back-edge") +__naked void bounded_recursion(void) +{ + asm volatile (" \ + r1 = 0; \ + call bounded_recursion__1; \ + exit; \ +" ::: __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void bounded_recursion__1(void) +{ + asm volatile (" \ + r1 += 1; \ + r0 = r1; \ + if r1 < 4 goto l0_%=; \ + exit; \ +l0_%=: call bounded_recursion__1; \ + exit; \ +" ::: __clobber_all); +} + +SEC("tracepoint") +__description("infinite loop in two jumps") +__failure __msg("loop detected") +__naked void infinite_loop_in_two_jumps(void) +{ + asm volatile (" \ + r0 = 0; \ +l1_%=: goto l0_%=; \ +l0_%=: if r0 < 4 goto l1_%=; \ + exit; \ +" ::: __clobber_all); +} + +SEC("tracepoint") +__description("infinite loop: three-jump trick") +__failure __msg("loop detected") +__naked void infinite_loop_three_jump_trick(void) +{ + asm volatile (" \ + r0 = 0; \ +l2_%=: r0 += 1; \ + r0 &= 1; \ + if r0 < 2 goto l0_%=; \ + exit; \ +l0_%=: r0 += 1; \ + r0 &= 1; \ + if r0 < 2 goto l1_%=; \ + exit; \ +l1_%=: r0 += 1; \ + r0 &= 1; \ + if r0 < 2 goto l2_%=; \ + exit; \ +" ::: __clobber_all); +} + +SEC("xdp") +__description("not-taken loop with back jump to 1st insn") +__success __retval(123) +__naked void back_jump_to_1st_insn_1(void) +{ + asm volatile (" \ +l0_%=: r0 = 123; \ + if r0 == 4 goto l0_%=; \ + exit; \ +" ::: __clobber_all); +} + +SEC("xdp") +__description("taken loop with back jump to 1st insn") +__success __retval(55) +__naked void back_jump_to_1st_insn_2(void) +{ + asm volatile (" \ + r1 = 10; \ + r2 = 0; \ + call back_jump_to_1st_insn_2__1; \ + exit; \ +" ::: __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void back_jump_to_1st_insn_2__1(void) +{ + asm volatile (" \ +l0_%=: r2 += r1; \ + r1 -= 1; \ + if r1 != 0 goto l0_%=; \ + r0 = r2; \ + exit; \ +" ::: __clobber_all); +} + +SEC("xdp") +__description("taken loop with back jump to 1st insn, 2") +__success __retval(55) +__naked void jump_to_1st_insn_2(void) +{ + asm volatile (" \ + r1 = 10; \ + r2 = 0; \ + call jump_to_1st_insn_2__1; \ + exit; \ +" ::: __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void jump_to_1st_insn_2__1(void) +{ + asm volatile (" \ +l0_%=: r2 += r1; \ + r1 -= 1; \ + if w1 != 0 goto l0_%=; \ + r0 = r2; \ + exit; \ +" ::: __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_lwt.c b/tools/testing/selftests/bpf/progs/verifier_lwt.c new file mode 100644 index 000000000000..5ab746307309 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_lwt.c @@ -0,0 +1,234 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/lwt.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +SEC("lwt_in") +__description("invalid direct packet write for LWT_IN") +__failure __msg("cannot write into packet") +__naked void packet_write_for_lwt_in(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r0 > r3 goto l0_%=; \ + *(u8*)(r2 + 0) = r2; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("lwt_out") +__description("invalid direct packet write for LWT_OUT") +__failure __msg("cannot write into packet") +__naked void packet_write_for_lwt_out(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r0 > r3 goto l0_%=; \ + *(u8*)(r2 + 0) = r2; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("lwt_xmit") +__description("direct packet write for LWT_XMIT") +__success __retval(0) +__naked void packet_write_for_lwt_xmit(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r0 > r3 goto l0_%=; \ + *(u8*)(r2 + 0) = r2; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("lwt_in") +__description("direct packet read for LWT_IN") +__success __retval(0) +__naked void packet_read_for_lwt_in(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r0 > r3 goto l0_%=; \ + r0 = *(u8*)(r2 + 0); \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("lwt_out") +__description("direct packet read for LWT_OUT") +__success __retval(0) +__naked void packet_read_for_lwt_out(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r0 > r3 goto l0_%=; \ + r0 = *(u8*)(r2 + 0); \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("lwt_xmit") +__description("direct packet read for LWT_XMIT") +__success __retval(0) +__naked void packet_read_for_lwt_xmit(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r0 > r3 goto l0_%=; \ + r0 = *(u8*)(r2 + 0); \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("lwt_xmit") +__description("overlapping checks for direct packet access") +__success __retval(0) +__naked void checks_for_direct_packet_access(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 8; \ + if r0 > r3 goto l0_%=; \ + r1 = r2; \ + r1 += 6; \ + if r1 > r3 goto l0_%=; \ + r0 = *(u16*)(r2 + 6); \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("lwt_xmit") +__description("make headroom for LWT_XMIT") +__success __retval(0) +__naked void make_headroom_for_lwt_xmit(void) +{ + asm volatile (" \ + r6 = r1; \ + r2 = 34; \ + r3 = 0; \ + call %[bpf_skb_change_head]; \ + /* split for s390 to succeed */ \ + r1 = r6; \ + r2 = 42; \ + r3 = 0; \ + call %[bpf_skb_change_head]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_skb_change_head) + : __clobber_all); +} + +SEC("socket") +__description("invalid access of tc_classid for LWT_IN") +__failure __msg("invalid bpf_context access") +__failure_unpriv +__naked void tc_classid_for_lwt_in(void) +{ + asm volatile (" \ + r0 = *(u32*)(r1 + %[__sk_buff_tc_classid]); \ + exit; \ +" : + : __imm_const(__sk_buff_tc_classid, offsetof(struct __sk_buff, tc_classid)) + : __clobber_all); +} + +SEC("socket") +__description("invalid access of tc_classid for LWT_OUT") +__failure __msg("invalid bpf_context access") +__failure_unpriv +__naked void tc_classid_for_lwt_out(void) +{ + asm volatile (" \ + r0 = *(u32*)(r1 + %[__sk_buff_tc_classid]); \ + exit; \ +" : + : __imm_const(__sk_buff_tc_classid, offsetof(struct __sk_buff, tc_classid)) + : __clobber_all); +} + +SEC("socket") +__description("invalid access of tc_classid for LWT_XMIT") +__failure __msg("invalid bpf_context access") +__failure_unpriv +__naked void tc_classid_for_lwt_xmit(void) +{ + asm volatile (" \ + r0 = *(u32*)(r1 + %[__sk_buff_tc_classid]); \ + exit; \ +" : + : __imm_const(__sk_buff_tc_classid, offsetof(struct __sk_buff, tc_classid)) + : __clobber_all); +} + +SEC("lwt_in") +__description("check skb->tc_classid half load not permitted for lwt prog") +__failure __msg("invalid bpf_context access") +__naked void not_permitted_for_lwt_prog(void) +{ + asm volatile ( + "r0 = 0;" +#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ + "r0 = *(u16*)(r1 + %[__sk_buff_tc_classid]);" +#else + "r0 = *(u16*)(r1 + %[__imm_0]);" +#endif + "exit;" + : + : __imm_const(__imm_0, offsetof(struct __sk_buff, tc_classid) + 2), + __imm_const(__sk_buff_tc_classid, offsetof(struct __sk_buff, tc_classid)) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_map_in_map.c b/tools/testing/selftests/bpf/progs/verifier_map_in_map.c new file mode 100644 index 000000000000..4eaab1468eb7 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_map_in_map.c @@ -0,0 +1,142 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/map_in_map.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS); + __uint(max_entries, 1); + __type(key, int); + __type(value, int); + __array(values, struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, int); + __type(value, int); + }); +} map_in_map SEC(".maps"); + +SEC("socket") +__description("map in map access") +__success __success_unpriv __retval(0) +__naked void map_in_map_access(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_in_map] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = r0; \ + call %[bpf_map_lookup_elem]; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_in_map) + : __clobber_all); +} + +SEC("xdp") +__description("map in map state pruning") +__success __msg("processed 26 insns") +__log_level(2) __retval(0) __flag(BPF_F_TEST_STATE_FREQ) +__naked void map_in_map_state_pruning(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r6 = r10; \ + r6 += -4; \ + r2 = r6; \ + r1 = %[map_in_map] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r2 = r6; \ + r1 = r0; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l1_%=; \ + r2 = r6; \ + r1 = %[map_in_map] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l2_%=; \ + exit; \ +l2_%=: r2 = r6; \ + r1 = r0; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l1_%=; \ + exit; \ +l1_%=: r0 = *(u32*)(r0 + 0); \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_in_map) + : __clobber_all); +} + +SEC("socket") +__description("invalid inner map pointer") +__failure __msg("R1 pointer arithmetic on map_ptr prohibited") +__failure_unpriv +__naked void invalid_inner_map_pointer(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_in_map] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = r0; \ + r1 += 8; \ + call %[bpf_map_lookup_elem]; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_in_map) + : __clobber_all); +} + +SEC("socket") +__description("forgot null checking on the inner map pointer") +__failure __msg("R1 type=map_value_or_null expected=map_ptr") +__failure_unpriv +__naked void on_the_inner_map_pointer(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_in_map] ll; \ + call %[bpf_map_lookup_elem]; \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = r0; \ + call %[bpf_map_lookup_elem]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_in_map) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_map_ptr_mixing.c b/tools/testing/selftests/bpf/progs/verifier_map_ptr_mixing.c new file mode 100644 index 000000000000..c5a7c1ddc562 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_map_ptr_mixing.c @@ -0,0 +1,265 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/map_ptr_mixing.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +#define MAX_ENTRIES 11 + +struct test_val { + unsigned int index; + int foo[MAX_ENTRIES]; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, int); + __type(value, struct test_val); +} map_array_48b SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 1); + __type(key, long long); + __type(value, struct test_val); +} map_hash_48b SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS); + __uint(max_entries, 1); + __type(key, int); + __type(value, int); + __array(values, struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, int); + __type(value, int); + }); +} map_in_map SEC(".maps"); + +void dummy_prog_42_socket(void); +void dummy_prog_24_socket(void); +void dummy_prog_loop1_socket(void); +void dummy_prog_loop2_socket(void); + +struct { + __uint(type, BPF_MAP_TYPE_PROG_ARRAY); + __uint(max_entries, 4); + __uint(key_size, sizeof(int)); + __array(values, void (void)); +} map_prog1_socket SEC(".maps") = { + .values = { + [0] = (void *)&dummy_prog_42_socket, + [1] = (void *)&dummy_prog_loop1_socket, + [2] = (void *)&dummy_prog_24_socket, + }, +}; + +struct { + __uint(type, BPF_MAP_TYPE_PROG_ARRAY); + __uint(max_entries, 8); + __uint(key_size, sizeof(int)); + __array(values, void (void)); +} map_prog2_socket SEC(".maps") = { + .values = { + [1] = (void *)&dummy_prog_loop2_socket, + [2] = (void *)&dummy_prog_24_socket, + [7] = (void *)&dummy_prog_42_socket, + }, +}; + +SEC("socket") +__auxiliary __auxiliary_unpriv +__naked void dummy_prog_42_socket(void) +{ + asm volatile ("r0 = 42; exit;"); +} + +SEC("socket") +__auxiliary __auxiliary_unpriv +__naked void dummy_prog_24_socket(void) +{ + asm volatile ("r0 = 24; exit;"); +} + +SEC("socket") +__auxiliary __auxiliary_unpriv +__naked void dummy_prog_loop1_socket(void) +{ + asm volatile (" \ + r3 = 1; \ + r2 = %[map_prog1_socket] ll; \ + call %[bpf_tail_call]; \ + r0 = 41; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_socket) + : __clobber_all); +} + +SEC("socket") +__auxiliary __auxiliary_unpriv +__naked void dummy_prog_loop2_socket(void) +{ + asm volatile (" \ + r3 = 1; \ + r2 = %[map_prog2_socket] ll; \ + call %[bpf_tail_call]; \ + r0 = 41; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog2_socket) + : __clobber_all); +} + +SEC("tc") +__description("calls: two calls returning different map pointers for lookup (hash, array)") +__success __retval(1) +__naked void pointers_for_lookup_hash_array(void) +{ + asm volatile (" \ + /* main prog */ \ + if r1 != 0 goto l0_%=; \ + call pointers_for_lookup_hash_array__1; \ + goto l1_%=; \ +l0_%=: call pointers_for_lookup_hash_array__2; \ +l1_%=: r1 = r0; \ + r2 = 0; \ + *(u64*)(r10 - 8) = r2; \ + r2 = r10; \ + r2 += -8; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l2_%=; \ + r1 = %[test_val_foo]; \ + *(u64*)(r0 + 0) = r1; \ + r0 = 1; \ +l2_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_const(test_val_foo, offsetof(struct test_val, foo)) + : __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void pointers_for_lookup_hash_array__1(void) +{ + asm volatile (" \ + r0 = %[map_hash_48b] ll; \ + exit; \ +" : + : __imm_addr(map_hash_48b) + : __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void pointers_for_lookup_hash_array__2(void) +{ + asm volatile (" \ + r0 = %[map_array_48b] ll; \ + exit; \ +" : + : __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("tc") +__description("calls: two calls returning different map pointers for lookup (hash, map in map)") +__failure __msg("only read from bpf_array is supported") +__naked void lookup_hash_map_in_map(void) +{ + asm volatile (" \ + /* main prog */ \ + if r1 != 0 goto l0_%=; \ + call lookup_hash_map_in_map__1; \ + goto l1_%=; \ +l0_%=: call lookup_hash_map_in_map__2; \ +l1_%=: r1 = r0; \ + r2 = 0; \ + *(u64*)(r10 - 8) = r2; \ + r2 = r10; \ + r2 += -8; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l2_%=; \ + r1 = %[test_val_foo]; \ + *(u64*)(r0 + 0) = r1; \ + r0 = 1; \ +l2_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_const(test_val_foo, offsetof(struct test_val, foo)) + : __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void lookup_hash_map_in_map__1(void) +{ + asm volatile (" \ + r0 = %[map_array_48b] ll; \ + exit; \ +" : + : __imm_addr(map_array_48b) + : __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void lookup_hash_map_in_map__2(void) +{ + asm volatile (" \ + r0 = %[map_in_map] ll; \ + exit; \ +" : + : __imm_addr(map_in_map) + : __clobber_all); +} + +SEC("socket") +__description("cond: two branches returning different map pointers for lookup (tail, tail)") +__success __failure_unpriv __msg_unpriv("tail_call abusing map_ptr") +__retval(42) +__naked void pointers_for_lookup_tail_tail_1(void) +{ + asm volatile (" \ + r6 = *(u32*)(r1 + %[__sk_buff_mark]); \ + if r6 != 0 goto l0_%=; \ + r2 = %[map_prog2_socket] ll; \ + goto l1_%=; \ +l0_%=: r2 = %[map_prog1_socket] ll; \ +l1_%=: r3 = 7; \ + call %[bpf_tail_call]; \ + r0 = 1; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_socket), + __imm_addr(map_prog2_socket), + __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark)) + : __clobber_all); +} + +SEC("socket") +__description("cond: two branches returning same map pointers for lookup (tail, tail)") +__success __success_unpriv __retval(42) +__naked void pointers_for_lookup_tail_tail_2(void) +{ + asm volatile (" \ + r6 = *(u32*)(r1 + %[__sk_buff_mark]); \ + if r6 == 0 goto l0_%=; \ + r2 = %[map_prog2_socket] ll; \ + goto l1_%=; \ +l0_%=: r2 = %[map_prog2_socket] ll; \ +l1_%=: r3 = 7; \ + call %[bpf_tail_call]; \ + r0 = 1; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog2_socket), + __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark)) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_netfilter_ctx.c b/tools/testing/selftests/bpf/progs/verifier_netfilter_ctx.c new file mode 100644 index 000000000000..65bba330e7e5 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_netfilter_ctx.c @@ -0,0 +1,121 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include "vmlinux.h" + +#include "bpf_misc.h" + +#include <bpf/bpf_endian.h> +#include <bpf/bpf_tracing.h> +#include <bpf/bpf_helpers.h> + +SEC("netfilter") +__description("netfilter invalid context access, size too short") +__failure __msg("invalid bpf_context access") +__naked void with_invalid_ctx_access_test1(void) +{ + asm volatile (" \ + r2 = *(u8*)(r1 + %[__bpf_nf_ctx_state]); \ + r0 = 0; \ + exit; \ +" : + : __imm_const(__bpf_nf_ctx_state, offsetof(struct bpf_nf_ctx, state)) + : __clobber_all); +} + +SEC("netfilter") +__description("netfilter invalid context access, size too short") +__failure __msg("invalid bpf_context access") +__naked void with_invalid_ctx_access_test2(void) +{ + asm volatile (" \ + r2 = *(u16*)(r1 + %[__bpf_nf_ctx_skb]); \ + r0 = 0; \ + exit; \ +" : + : __imm_const(__bpf_nf_ctx_skb, offsetof(struct bpf_nf_ctx, skb)) + : __clobber_all); +} + +SEC("netfilter") +__description("netfilter invalid context access, past end of ctx") +__failure __msg("invalid bpf_context access") +__naked void with_invalid_ctx_access_test3(void) +{ + asm volatile (" \ + r2 = *(u64*)(r1 + %[__bpf_nf_ctx_size]); \ + r0 = 0; \ + exit; \ +" : + : __imm_const(__bpf_nf_ctx_size, sizeof(struct bpf_nf_ctx)) + : __clobber_all); +} + +SEC("netfilter") +__description("netfilter invalid context, write") +__failure __msg("invalid bpf_context access") +__naked void with_invalid_ctx_access_test4(void) +{ + asm volatile (" \ + r2 = r1; \ + *(u64*)(r2 + 0) = r1; \ + r0 = 1; \ + exit; \ +" : + : __imm_const(__bpf_nf_ctx_skb, offsetof(struct bpf_nf_ctx, skb)) + : __clobber_all); +} + +#define NF_DROP 0 +#define NF_ACCEPT 1 + +SEC("netfilter") +__description("netfilter valid context read and invalid write") +__failure __msg("only read is supported") +int with_invalid_ctx_access_test5(struct bpf_nf_ctx *ctx) +{ + struct nf_hook_state *state = (void *)ctx->state; + + state->sk = NULL; + return NF_ACCEPT; +} + +extern int bpf_dynptr_from_skb(struct sk_buff *skb, __u64 flags, + struct bpf_dynptr *ptr__uninit) __ksym; +extern void *bpf_dynptr_slice(const struct bpf_dynptr *ptr, uint32_t offset, + void *buffer, uint32_t buffer__sz) __ksym; + +SEC("netfilter") +__description("netfilter test prog with skb and state read access") +__success __failure_unpriv +__retval(0) +int with_valid_ctx_access_test6(struct bpf_nf_ctx *ctx) +{ + const struct nf_hook_state *state = ctx->state; + struct sk_buff *skb = ctx->skb; + const struct iphdr *iph; + const struct tcphdr *th; + u8 buffer_iph[20] = {}; + u8 buffer_th[40] = {}; + struct bpf_dynptr ptr; + uint8_t ihl; + + if (skb->len <= 20 || bpf_dynptr_from_skb(skb, 0, &ptr)) + return NF_ACCEPT; + + iph = bpf_dynptr_slice(&ptr, 0, buffer_iph, sizeof(buffer_iph)); + if (!iph) + return NF_ACCEPT; + + if (state->pf != 2) + return NF_ACCEPT; + + ihl = iph->ihl << 2; + + th = bpf_dynptr_slice(&ptr, ihl, buffer_th, sizeof(buffer_th)); + if (!th) + return NF_ACCEPT; + + return th->dest == bpf_htons(22) ? NF_ACCEPT : NF_DROP; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_netfilter_retcode.c b/tools/testing/selftests/bpf/progs/verifier_netfilter_retcode.c new file mode 100644 index 000000000000..353ae6da00e1 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_netfilter_retcode.c @@ -0,0 +1,49 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +SEC("netfilter") +__description("bpf_exit with invalid return code. test1") +__failure __msg("R0 is not a known value") +__naked void with_invalid_return_code_test1(void) +{ + asm volatile (" \ + r0 = *(u64*)(r1 + 0); \ + exit; \ +" ::: __clobber_all); +} + +SEC("netfilter") +__description("bpf_exit with valid return code. test2") +__success +__naked void with_valid_return_code_test2(void) +{ + asm volatile (" \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("netfilter") +__description("bpf_exit with valid return code. test3") +__success +__naked void with_valid_return_code_test3(void) +{ + asm volatile (" \ + r0 = 1; \ + exit; \ +" ::: __clobber_all); +} + +SEC("netfilter") +__description("bpf_exit with invalid return code. test4") +__failure __msg("R0 has value (0x2; 0x0)") +__naked void with_invalid_return_code_test4(void) +{ + asm volatile (" \ + r0 = 2; \ + exit; \ +" ::: __clobber_all); +} diff --git a/tools/testing/selftests/bpf/progs/verifier_ref_tracking.c b/tools/testing/selftests/bpf/progs/verifier_ref_tracking.c new file mode 100644 index 000000000000..c4c6da21265e --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_ref_tracking.c @@ -0,0 +1,1495 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/ref_tracking.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "../../../include/linux/filter.h" +#include "bpf_misc.h" + +#define BPF_SK_LOOKUP(func) \ + /* struct bpf_sock_tuple tuple = {} */ \ + "r2 = 0;" \ + "*(u32*)(r10 - 8) = r2;" \ + "*(u64*)(r10 - 16) = r2;" \ + "*(u64*)(r10 - 24) = r2;" \ + "*(u64*)(r10 - 32) = r2;" \ + "*(u64*)(r10 - 40) = r2;" \ + "*(u64*)(r10 - 48) = r2;" \ + /* sk = func(ctx, &tuple, sizeof tuple, 0, 0) */ \ + "r2 = r10;" \ + "r2 += -48;" \ + "r3 = %[sizeof_bpf_sock_tuple];"\ + "r4 = 0;" \ + "r5 = 0;" \ + "call %[" #func "];" + +struct bpf_key {} __attribute__((preserve_access_index)); + +extern void bpf_key_put(struct bpf_key *key) __ksym; +extern struct bpf_key *bpf_lookup_system_key(__u64 id) __ksym; +extern struct bpf_key *bpf_lookup_user_key(__u32 serial, __u64 flags) __ksym; + +/* BTF FUNC records are not generated for kfuncs referenced + * from inline assembly. These records are necessary for + * libbpf to link the program. The function below is a hack + * to ensure that BTF FUNC records are generated. + */ +void __kfunc_btf_root(void) +{ + bpf_key_put(0); + bpf_lookup_system_key(0); + bpf_lookup_user_key(0, 0); +} + +#define MAX_ENTRIES 11 + +struct test_val { + unsigned int index; + int foo[MAX_ENTRIES]; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, int); + __type(value, struct test_val); +} map_array_48b SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_RINGBUF); + __uint(max_entries, 4096); +} map_ringbuf SEC(".maps"); + +void dummy_prog_42_tc(void); +void dummy_prog_24_tc(void); +void dummy_prog_loop1_tc(void); + +struct { + __uint(type, BPF_MAP_TYPE_PROG_ARRAY); + __uint(max_entries, 4); + __uint(key_size, sizeof(int)); + __array(values, void (void)); +} map_prog1_tc SEC(".maps") = { + .values = { + [0] = (void *)&dummy_prog_42_tc, + [1] = (void *)&dummy_prog_loop1_tc, + [2] = (void *)&dummy_prog_24_tc, + }, +}; + +SEC("tc") +__auxiliary +__naked void dummy_prog_42_tc(void) +{ + asm volatile ("r0 = 42; exit;"); +} + +SEC("tc") +__auxiliary +__naked void dummy_prog_24_tc(void) +{ + asm volatile ("r0 = 24; exit;"); +} + +SEC("tc") +__auxiliary +__naked void dummy_prog_loop1_tc(void) +{ + asm volatile (" \ + r3 = 1; \ + r2 = %[map_prog1_tc] ll; \ + call %[bpf_tail_call]; \ + r0 = 41; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_tc) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: leak potential reference") +__failure __msg("Unreleased reference") +__naked void reference_tracking_leak_potential_reference(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r6 = r0; /* leak reference */ \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: leak potential reference to sock_common") +__failure __msg("Unreleased reference") +__naked void potential_reference_to_sock_common_1(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_skc_lookup_tcp) +" r6 = r0; /* leak reference */ \ + exit; \ +" : + : __imm(bpf_skc_lookup_tcp), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: leak potential reference on stack") +__failure __msg("Unreleased reference") +__naked void leak_potential_reference_on_stack(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r4 = r10; \ + r4 += -8; \ + *(u64*)(r4 + 0) = r0; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: leak potential reference on stack 2") +__failure __msg("Unreleased reference") +__naked void potential_reference_on_stack_2(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r4 = r10; \ + r4 += -8; \ + *(u64*)(r4 + 0) = r0; \ + r0 = 0; \ + r1 = 0; \ + *(u64*)(r4 + 0) = r1; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: zero potential reference") +__failure __msg("Unreleased reference") +__naked void reference_tracking_zero_potential_reference(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r0 = 0; /* leak reference */ \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: zero potential reference to sock_common") +__failure __msg("Unreleased reference") +__naked void potential_reference_to_sock_common_2(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_skc_lookup_tcp) +" r0 = 0; /* leak reference */ \ + exit; \ +" : + : __imm(bpf_skc_lookup_tcp), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: copy and zero potential references") +__failure __msg("Unreleased reference") +__naked void copy_and_zero_potential_references(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r7 = r0; \ + r0 = 0; \ + r7 = 0; /* leak reference */ \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("lsm.s/bpf") +__description("reference tracking: acquire/release user key reference") +__success +__naked void acquire_release_user_key_reference(void) +{ + asm volatile (" \ + r1 = -3; \ + r2 = 0; \ + call %[bpf_lookup_user_key]; \ + if r0 == 0 goto l0_%=; \ + r1 = r0; \ + call %[bpf_key_put]; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_key_put), + __imm(bpf_lookup_user_key) + : __clobber_all); +} + +SEC("lsm.s/bpf") +__description("reference tracking: acquire/release system key reference") +__success +__naked void acquire_release_system_key_reference(void) +{ + asm volatile (" \ + r1 = 1; \ + call %[bpf_lookup_system_key]; \ + if r0 == 0 goto l0_%=; \ + r1 = r0; \ + call %[bpf_key_put]; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_key_put), + __imm(bpf_lookup_system_key) + : __clobber_all); +} + +SEC("lsm.s/bpf") +__description("reference tracking: release user key reference without check") +__failure __msg("Possibly NULL pointer passed to trusted arg0") +__naked void user_key_reference_without_check(void) +{ + asm volatile (" \ + r1 = -3; \ + r2 = 0; \ + call %[bpf_lookup_user_key]; \ + r1 = r0; \ + call %[bpf_key_put]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_key_put), + __imm(bpf_lookup_user_key) + : __clobber_all); +} + +SEC("lsm.s/bpf") +__description("reference tracking: release system key reference without check") +__failure __msg("Possibly NULL pointer passed to trusted arg0") +__naked void system_key_reference_without_check(void) +{ + asm volatile (" \ + r1 = 1; \ + call %[bpf_lookup_system_key]; \ + r1 = r0; \ + call %[bpf_key_put]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_key_put), + __imm(bpf_lookup_system_key) + : __clobber_all); +} + +SEC("lsm.s/bpf") +__description("reference tracking: release with NULL key pointer") +__failure __msg("Possibly NULL pointer passed to trusted arg0") +__naked void release_with_null_key_pointer(void) +{ + asm volatile (" \ + r1 = 0; \ + call %[bpf_key_put]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_key_put) + : __clobber_all); +} + +SEC("lsm.s/bpf") +__description("reference tracking: leak potential reference to user key") +__failure __msg("Unreleased reference") +__naked void potential_reference_to_user_key(void) +{ + asm volatile (" \ + r1 = -3; \ + r2 = 0; \ + call %[bpf_lookup_user_key]; \ + exit; \ +" : + : __imm(bpf_lookup_user_key) + : __clobber_all); +} + +SEC("lsm.s/bpf") +__description("reference tracking: leak potential reference to system key") +__failure __msg("Unreleased reference") +__naked void potential_reference_to_system_key(void) +{ + asm volatile (" \ + r1 = 1; \ + call %[bpf_lookup_system_key]; \ + exit; \ +" : + : __imm(bpf_lookup_system_key) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: release reference without check") +__failure __msg("type=sock_or_null expected=sock") +__naked void tracking_release_reference_without_check(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" /* reference in r0 may be NULL */ \ + r1 = r0; \ + r2 = 0; \ + call %[bpf_sk_release]; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: release reference to sock_common without check") +__failure __msg("type=sock_common_or_null expected=sock") +__naked void to_sock_common_without_check(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_skc_lookup_tcp) +" /* reference in r0 may be NULL */ \ + r1 = r0; \ + r2 = 0; \ + call %[bpf_sk_release]; \ + exit; \ +" : + : __imm(bpf_sk_release), + __imm(bpf_skc_lookup_tcp), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: release reference") +__success __retval(0) +__naked void reference_tracking_release_reference(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r1 = r0; \ + if r0 == 0 goto l0_%=; \ + call %[bpf_sk_release]; \ +l0_%=: exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: release reference to sock_common") +__success __retval(0) +__naked void release_reference_to_sock_common(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_skc_lookup_tcp) +" r1 = r0; \ + if r0 == 0 goto l0_%=; \ + call %[bpf_sk_release]; \ +l0_%=: exit; \ +" : + : __imm(bpf_sk_release), + __imm(bpf_skc_lookup_tcp), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: release reference 2") +__success __retval(0) +__naked void reference_tracking_release_reference_2(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r1 = r0; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: call %[bpf_sk_release]; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: release reference twice") +__failure __msg("type=scalar expected=sock") +__naked void reference_tracking_release_reference_twice(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r1 = r0; \ + r6 = r0; \ + if r0 == 0 goto l0_%=; \ + call %[bpf_sk_release]; \ +l0_%=: r1 = r6; \ + call %[bpf_sk_release]; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: release reference twice inside branch") +__failure __msg("type=scalar expected=sock") +__naked void release_reference_twice_inside_branch(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r1 = r0; \ + r6 = r0; \ + if r0 == 0 goto l0_%=; /* goto end */ \ + call %[bpf_sk_release]; \ + r1 = r6; \ + call %[bpf_sk_release]; \ +l0_%=: exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: alloc, check, free in one subbranch") +__failure __msg("Unreleased reference") +__flag(BPF_F_ANY_ALIGNMENT) +__naked void check_free_in_one_subbranch(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 16; \ + /* if (offsetof(skb, mark) > data_len) exit; */ \ + if r0 <= r3 goto l0_%=; \ + exit; \ +l0_%=: r6 = *(u32*)(r2 + %[__sk_buff_mark]); \ +" BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" if r6 == 0 goto l1_%=; /* mark == 0? */\ + /* Leak reference in R0 */ \ + exit; \ +l1_%=: if r0 == 0 goto l2_%=; /* sk NULL? */ \ + r1 = r0; \ + call %[bpf_sk_release]; \ +l2_%=: exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)), + __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark)), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: alloc, check, free in both subbranches") +__success __retval(0) __flag(BPF_F_ANY_ALIGNMENT) +__naked void check_free_in_both_subbranches(void) +{ + asm volatile (" \ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 16; \ + /* if (offsetof(skb, mark) > data_len) exit; */ \ + if r0 <= r3 goto l0_%=; \ + exit; \ +l0_%=: r6 = *(u32*)(r2 + %[__sk_buff_mark]); \ +" BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" if r6 == 0 goto l1_%=; /* mark == 0? */\ + if r0 == 0 goto l2_%=; /* sk NULL? */ \ + r1 = r0; \ + call %[bpf_sk_release]; \ +l2_%=: exit; \ +l1_%=: if r0 == 0 goto l3_%=; /* sk NULL? */ \ + r1 = r0; \ + call %[bpf_sk_release]; \ +l3_%=: exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)), + __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark)), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking in call: free reference in subprog") +__success __retval(0) +__naked void call_free_reference_in_subprog(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r1 = r0; /* unchecked reference */ \ + call call_free_reference_in_subprog__1; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void call_free_reference_in_subprog__1(void) +{ + asm volatile (" \ + /* subprog 1 */ \ + r2 = r1; \ + if r2 == 0 goto l0_%=; \ + call %[bpf_sk_release]; \ +l0_%=: exit; \ +" : + : __imm(bpf_sk_release) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking in call: free reference in subprog and outside") +__failure __msg("type=scalar expected=sock") +__naked void reference_in_subprog_and_outside(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r1 = r0; /* unchecked reference */ \ + r6 = r0; \ + call reference_in_subprog_and_outside__1; \ + r1 = r6; \ + call %[bpf_sk_release]; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void reference_in_subprog_and_outside__1(void) +{ + asm volatile (" \ + /* subprog 1 */ \ + r2 = r1; \ + if r2 == 0 goto l0_%=; \ + call %[bpf_sk_release]; \ +l0_%=: exit; \ +" : + : __imm(bpf_sk_release) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking in call: alloc & leak reference in subprog") +__failure __msg("Unreleased reference") +__naked void alloc_leak_reference_in_subprog(void) +{ + asm volatile (" \ + r4 = r10; \ + r4 += -8; \ + call alloc_leak_reference_in_subprog__1; \ + r1 = r0; \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void alloc_leak_reference_in_subprog__1(void) +{ + asm volatile (" \ + /* subprog 1 */ \ + r6 = r4; \ +" BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" /* spill unchecked sk_ptr into stack of caller */\ + *(u64*)(r6 + 0) = r0; \ + r1 = r0; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking in call: alloc in subprog, release outside") +__success __retval(POINTER_VALUE) +__naked void alloc_in_subprog_release_outside(void) +{ + asm volatile (" \ + r4 = r10; \ + call alloc_in_subprog_release_outside__1; \ + r1 = r0; \ + if r0 == 0 goto l0_%=; \ + call %[bpf_sk_release]; \ +l0_%=: exit; \ +" : + : __imm(bpf_sk_release) + : __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void alloc_in_subprog_release_outside__1(void) +{ + asm volatile (" \ + /* subprog 1 */ \ +" BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" exit; /* return sk */ \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking in call: sk_ptr leak into caller stack") +__failure __msg("Unreleased reference") +__naked void ptr_leak_into_caller_stack(void) +{ + asm volatile (" \ + r4 = r10; \ + r4 += -8; \ + call ptr_leak_into_caller_stack__1; \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void ptr_leak_into_caller_stack__1(void) +{ + asm volatile (" \ + /* subprog 1 */ \ + r5 = r10; \ + r5 += -8; \ + *(u64*)(r5 + 0) = r4; \ + call ptr_leak_into_caller_stack__2; \ + /* spill unchecked sk_ptr into stack of caller */\ + r5 = r10; \ + r5 += -8; \ + r4 = *(u64*)(r5 + 0); \ + *(u64*)(r4 + 0) = r0; \ + exit; \ +" ::: __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void ptr_leak_into_caller_stack__2(void) +{ + asm volatile (" \ + /* subprog 2 */ \ +" BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking in call: sk_ptr spill into caller stack") +__success __retval(0) +__naked void ptr_spill_into_caller_stack(void) +{ + asm volatile (" \ + r4 = r10; \ + r4 += -8; \ + call ptr_spill_into_caller_stack__1; \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void ptr_spill_into_caller_stack__1(void) +{ + asm volatile (" \ + /* subprog 1 */ \ + r5 = r10; \ + r5 += -8; \ + *(u64*)(r5 + 0) = r4; \ + call ptr_spill_into_caller_stack__2; \ + /* spill unchecked sk_ptr into stack of caller */\ + r5 = r10; \ + r5 += -8; \ + r4 = *(u64*)(r5 + 0); \ + *(u64*)(r4 + 0) = r0; \ + if r0 == 0 goto l0_%=; \ + /* now the sk_ptr is verified, free the reference */\ + r1 = *(u64*)(r4 + 0); \ + call %[bpf_sk_release]; \ +l0_%=: exit; \ +" : + : __imm(bpf_sk_release) + : __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void ptr_spill_into_caller_stack__2(void) +{ + asm volatile (" \ + /* subprog 2 */ \ +" BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: allow LD_ABS") +__success __retval(0) +__naked void reference_tracking_allow_ld_abs(void) +{ + asm volatile (" \ + r6 = r1; \ +" BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r1 = r0; \ + if r0 == 0 goto l0_%=; \ + call %[bpf_sk_release]; \ +l0_%=: r0 = *(u8*)skb[0]; \ + r0 = *(u16*)skb[0]; \ + r0 = *(u32*)skb[0]; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: forbid LD_ABS while holding reference") +__failure __msg("BPF_LD_[ABS|IND] cannot be mixed with socket references") +__naked void ld_abs_while_holding_reference(void) +{ + asm volatile (" \ + r6 = r1; \ +" BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r0 = *(u8*)skb[0]; \ + r0 = *(u16*)skb[0]; \ + r0 = *(u32*)skb[0]; \ + r1 = r0; \ + if r0 == 0 goto l0_%=; \ + call %[bpf_sk_release]; \ +l0_%=: exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: allow LD_IND") +__success __retval(1) +__naked void reference_tracking_allow_ld_ind(void) +{ + asm volatile (" \ + r6 = r1; \ +" BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r1 = r0; \ + if r0 == 0 goto l0_%=; \ + call %[bpf_sk_release]; \ +l0_%=: r7 = 1; \ + .8byte %[ld_ind]; \ + r0 = r7; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)), + __imm_insn(ld_ind, BPF_LD_IND(BPF_W, BPF_REG_7, -0x200000)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: forbid LD_IND while holding reference") +__failure __msg("BPF_LD_[ABS|IND] cannot be mixed with socket references") +__naked void ld_ind_while_holding_reference(void) +{ + asm volatile (" \ + r6 = r1; \ +" BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r4 = r0; \ + r7 = 1; \ + .8byte %[ld_ind]; \ + r0 = r7; \ + r1 = r4; \ + if r1 == 0 goto l0_%=; \ + call %[bpf_sk_release]; \ +l0_%=: exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)), + __imm_insn(ld_ind, BPF_LD_IND(BPF_W, BPF_REG_7, -0x200000)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: check reference or tail call") +__success __retval(0) +__naked void check_reference_or_tail_call(void) +{ + asm volatile (" \ + r7 = r1; \ +" BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" /* if (sk) bpf_sk_release() */ \ + r1 = r0; \ + if r1 != 0 goto l0_%=; \ + /* bpf_tail_call() */ \ + r3 = 3; \ + r2 = %[map_prog1_tc] ll; \ + r1 = r7; \ + call %[bpf_tail_call]; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_release]; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm(bpf_tail_call), + __imm_addr(map_prog1_tc), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: release reference then tail call") +__success __retval(0) +__naked void release_reference_then_tail_call(void) +{ + asm volatile (" \ + r7 = r1; \ +" BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" /* if (sk) bpf_sk_release() */ \ + r1 = r0; \ + if r1 == 0 goto l0_%=; \ + call %[bpf_sk_release]; \ +l0_%=: /* bpf_tail_call() */ \ + r3 = 3; \ + r2 = %[map_prog1_tc] ll; \ + r1 = r7; \ + call %[bpf_tail_call]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm(bpf_tail_call), + __imm_addr(map_prog1_tc), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: leak possible reference over tail call") +__failure __msg("tail_call would lead to reference leak") +__naked void possible_reference_over_tail_call(void) +{ + asm volatile (" \ + r7 = r1; \ + /* Look up socket and store in REG_6 */ \ +" BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" /* bpf_tail_call() */ \ + r6 = r0; \ + r3 = 3; \ + r2 = %[map_prog1_tc] ll; \ + r1 = r7; \ + call %[bpf_tail_call]; \ + r0 = 0; \ + /* if (sk) bpf_sk_release() */ \ + r1 = r6; \ + if r1 == 0 goto l0_%=; \ + call %[bpf_sk_release]; \ +l0_%=: exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm(bpf_tail_call), + __imm_addr(map_prog1_tc), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: leak checked reference over tail call") +__failure __msg("tail_call would lead to reference leak") +__naked void checked_reference_over_tail_call(void) +{ + asm volatile (" \ + r7 = r1; \ + /* Look up socket and store in REG_6 */ \ +" BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r6 = r0; \ + /* if (!sk) goto end */ \ + if r0 == 0 goto l0_%=; \ + /* bpf_tail_call() */ \ + r3 = 0; \ + r2 = %[map_prog1_tc] ll; \ + r1 = r7; \ + call %[bpf_tail_call]; \ + r0 = 0; \ + r1 = r6; \ +l0_%=: call %[bpf_sk_release]; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm(bpf_tail_call), + __imm_addr(map_prog1_tc), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: mangle and release sock_or_null") +__failure __msg("R1 pointer arithmetic on sock_or_null prohibited") +__naked void and_release_sock_or_null(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r1 = r0; \ + r1 += 5; \ + if r0 == 0 goto l0_%=; \ + call %[bpf_sk_release]; \ +l0_%=: exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: mangle and release sock") +__failure __msg("R1 pointer arithmetic on sock prohibited") +__naked void tracking_mangle_and_release_sock(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r1 = r0; \ + if r0 == 0 goto l0_%=; \ + r1 += 5; \ + call %[bpf_sk_release]; \ +l0_%=: exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: access member") +__success __retval(0) +__naked void reference_tracking_access_member(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r6 = r0; \ + if r0 == 0 goto l0_%=; \ + r2 = *(u32*)(r0 + 4); \ + r1 = r6; \ + call %[bpf_sk_release]; \ +l0_%=: exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: write to member") +__failure __msg("cannot write into sock") +__naked void reference_tracking_write_to_member(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r6 = r0; \ + if r0 == 0 goto l0_%=; \ + r1 = r6; \ + r2 = 42 ll; \ + *(u32*)(r1 + %[bpf_sock_mark]) = r2; \ + r1 = r6; \ +l0_%=: call %[bpf_sk_release]; \ + r0 = 0 ll; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(bpf_sock_mark, offsetof(struct bpf_sock, mark)), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: invalid 64-bit access of member") +__failure __msg("invalid sock access off=0 size=8") +__naked void _64_bit_access_of_member(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r6 = r0; \ + if r0 == 0 goto l0_%=; \ + r2 = *(u64*)(r0 + 0); \ + r1 = r6; \ + call %[bpf_sk_release]; \ +l0_%=: exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: access after release") +__failure __msg("!read_ok") +__naked void reference_tracking_access_after_release(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r1 = r0; \ + if r0 == 0 goto l0_%=; \ + call %[bpf_sk_release]; \ + r2 = *(u32*)(r1 + 0); \ +l0_%=: exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: direct access for lookup") +__success __retval(0) +__naked void tracking_direct_access_for_lookup(void) +{ + asm volatile (" \ + /* Check that the packet is at least 64B long */\ + r2 = *(u32*)(r1 + %[__sk_buff_data]); \ + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r0 = r2; \ + r0 += 64; \ + if r0 > r3 goto l0_%=; \ + /* sk = sk_lookup_tcp(ctx, skb->data, ...) */ \ + r3 = %[sizeof_bpf_sock_tuple]; \ + r4 = 0; \ + r5 = 0; \ + call %[bpf_sk_lookup_tcp]; \ + r6 = r0; \ + if r0 == 0 goto l0_%=; \ + r2 = *(u32*)(r0 + 4); \ + r1 = r6; \ + call %[bpf_sk_release]; \ +l0_%=: exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: use ptr from bpf_tcp_sock() after release") +__failure __msg("invalid mem access") +__flag(BPF_F_ANY_ALIGNMENT) +__naked void bpf_tcp_sock_after_release(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r1 = r0; \ + call %[bpf_tcp_sock]; \ + if r0 != 0 goto l1_%=; \ + r1 = r6; \ + call %[bpf_sk_release]; \ + exit; \ +l1_%=: r7 = r0; \ + r1 = r6; \ + call %[bpf_sk_release]; \ + r0 = *(u32*)(r7 + %[bpf_tcp_sock_snd_cwnd]); \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm(bpf_tcp_sock), + __imm_const(bpf_tcp_sock_snd_cwnd, offsetof(struct bpf_tcp_sock, snd_cwnd)), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: use ptr from bpf_sk_fullsock() after release") +__failure __msg("invalid mem access") +__flag(BPF_F_ANY_ALIGNMENT) +__naked void bpf_sk_fullsock_after_release(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r1 = r0; \ + call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + r1 = r6; \ + call %[bpf_sk_release]; \ + exit; \ +l1_%=: r7 = r0; \ + r1 = r6; \ + call %[bpf_sk_release]; \ + r0 = *(u32*)(r7 + %[bpf_sock_type]); \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(bpf_sock_type, offsetof(struct bpf_sock, type)), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: use ptr from bpf_sk_fullsock(tp) after release") +__failure __msg("invalid mem access") +__flag(BPF_F_ANY_ALIGNMENT) +__naked void sk_fullsock_tp_after_release(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r1 = r0; \ + call %[bpf_tcp_sock]; \ + if r0 != 0 goto l1_%=; \ + r1 = r6; \ + call %[bpf_sk_release]; \ + exit; \ +l1_%=: r1 = r0; \ + call %[bpf_sk_fullsock]; \ + r1 = r6; \ + r6 = r0; \ + call %[bpf_sk_release]; \ + if r6 != 0 goto l2_%=; \ + exit; \ +l2_%=: r0 = *(u32*)(r6 + %[bpf_sock_type]); \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm(bpf_tcp_sock), + __imm_const(bpf_sock_type, offsetof(struct bpf_sock, type)), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: use sk after bpf_sk_release(tp)") +__failure __msg("invalid mem access") +__flag(BPF_F_ANY_ALIGNMENT) +__naked void after_bpf_sk_release_tp(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r1 = r0; \ + call %[bpf_tcp_sock]; \ + if r0 != 0 goto l1_%=; \ + r1 = r6; \ + call %[bpf_sk_release]; \ + exit; \ +l1_%=: r1 = r0; \ + call %[bpf_sk_release]; \ + r0 = *(u32*)(r6 + %[bpf_sock_type]); \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm(bpf_tcp_sock), + __imm_const(bpf_sock_type, offsetof(struct bpf_sock, type)), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: use ptr from bpf_get_listener_sock() after bpf_sk_release(sk)") +__success __retval(0) +__naked void after_bpf_sk_release_sk(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r1 = r0; \ + call %[bpf_get_listener_sock]; \ + if r0 != 0 goto l1_%=; \ + r1 = r6; \ + call %[bpf_sk_release]; \ + exit; \ +l1_%=: r1 = r6; \ + r6 = r0; \ + call %[bpf_sk_release]; \ + r0 = *(u32*)(r6 + %[bpf_sock_src_port]); \ + exit; \ +" : + : __imm(bpf_get_listener_sock), + __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(bpf_sock_src_port, offsetof(struct bpf_sock, src_port)), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: bpf_sk_release(listen_sk)") +__failure __msg("R1 must be referenced when passed to release function") +__naked void bpf_sk_release_listen_sk(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r1 = r0; \ + call %[bpf_get_listener_sock]; \ + if r0 != 0 goto l1_%=; \ + r1 = r6; \ + call %[bpf_sk_release]; \ + exit; \ +l1_%=: r1 = r0; \ + call %[bpf_sk_release]; \ + r0 = *(u32*)(r6 + %[bpf_sock_type]); \ + r1 = r6; \ + call %[bpf_sk_release]; \ + exit; \ +" : + : __imm(bpf_get_listener_sock), + __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(bpf_sock_type, offsetof(struct bpf_sock, type)), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +/* !bpf_sk_fullsock(sk) is checked but !bpf_tcp_sock(sk) is not checked */ +SEC("tc") +__description("reference tracking: tp->snd_cwnd after bpf_sk_fullsock(sk) and bpf_tcp_sock(sk)") +__failure __msg("invalid mem access") +__naked void and_bpf_tcp_sock_sk(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r1 = r0; \ + call %[bpf_sk_fullsock]; \ + r7 = r0; \ + r1 = r6; \ + call %[bpf_tcp_sock]; \ + r8 = r0; \ + if r7 != 0 goto l1_%=; \ + r1 = r6; \ + call %[bpf_sk_release]; \ + exit; \ +l1_%=: r0 = *(u32*)(r8 + %[bpf_tcp_sock_snd_cwnd]); \ + r1 = r6; \ + call %[bpf_sk_release]; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm(bpf_tcp_sock), + __imm_const(bpf_tcp_sock_snd_cwnd, offsetof(struct bpf_tcp_sock, snd_cwnd)), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: branch tracking valid pointer null comparison") +__success __retval(0) +__naked void tracking_valid_pointer_null_comparison(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r6 = r0; \ + r3 = 1; \ + if r6 != 0 goto l0_%=; \ + r3 = 0; \ +l0_%=: if r6 == 0 goto l1_%=; \ + r1 = r6; \ + call %[bpf_sk_release]; \ +l1_%=: exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: branch tracking valid pointer value comparison") +__failure __msg("Unreleased reference") +__naked void tracking_valid_pointer_value_comparison(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r6 = r0; \ + r3 = 1; \ + if r6 == 0 goto l0_%=; \ + r3 = 0; \ + if r6 == 1234 goto l0_%=; \ + r1 = r6; \ + call %[bpf_sk_release]; \ +l0_%=: exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: bpf_sk_release(btf_tcp_sock)") +__success +__retval(0) +__naked void sk_release_btf_tcp_sock(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r1 = r0; \ + call %[bpf_skc_to_tcp_sock]; \ + if r0 != 0 goto l1_%=; \ + r1 = r6; \ + call %[bpf_sk_release]; \ + exit; \ +l1_%=: r1 = r0; \ + call %[bpf_sk_release]; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm(bpf_skc_to_tcp_sock), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("reference tracking: use ptr from bpf_skc_to_tcp_sock() after release") +__failure __msg("invalid mem access") +__naked void to_tcp_sock_after_release(void) +{ + asm volatile ( + BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r1 = r0; \ + call %[bpf_skc_to_tcp_sock]; \ + if r0 != 0 goto l1_%=; \ + r1 = r6; \ + call %[bpf_sk_release]; \ + exit; \ +l1_%=: r7 = r0; \ + r1 = r6; \ + call %[bpf_sk_release]; \ + r0 = *(u8*)(r7 + 0); \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm(bpf_skc_to_tcp_sock), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("socket") +__description("reference tracking: try to leak released ptr reg") +__success __failure_unpriv __msg_unpriv("R8 !read_ok") +__retval(0) +__naked void to_leak_released_ptr_reg(void) +{ + asm volatile (" \ + r0 = 0; \ + *(u32*)(r10 - 4) = r0; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r9 = r0; \ + r0 = 0; \ + r1 = %[map_ringbuf] ll; \ + r2 = 8; \ + r3 = 0; \ + call %[bpf_ringbuf_reserve]; \ + if r0 != 0 goto l1_%=; \ + exit; \ +l1_%=: r8 = r0; \ + r1 = r8; \ + r2 = 0; \ + call %[bpf_ringbuf_discard]; \ + r0 = 0; \ + *(u64*)(r9 + 0) = r8; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm(bpf_ringbuf_discard), + __imm(bpf_ringbuf_reserve), + __imm_addr(map_array_48b), + __imm_addr(map_ringbuf) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_reg_equal.c b/tools/testing/selftests/bpf/progs/verifier_reg_equal.c new file mode 100644 index 000000000000..dc1d8c30fb0e --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_reg_equal.c @@ -0,0 +1,58 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +SEC("socket") +__description("check w reg equal if r reg upper32 bits 0") +__success +__naked void subreg_equality_1(void) +{ + asm volatile (" \ + call %[bpf_ktime_get_ns]; \ + *(u64 *)(r10 - 8) = r0; \ + r2 = *(u32 *)(r10 - 8); \ + /* At this point upper 4-bytes of r2 are 0, \ + * thus insn w3 = w2 should propagate reg id, \ + * and w2 < 9 comparison would also propagate \ + * the range for r3. \ + */ \ + w3 = w2; \ + if w2 < 9 goto l0_%=; \ + exit; \ +l0_%=: if r3 < 9 goto l1_%=; \ + /* r1 read is illegal at this point */ \ + r0 -= r1; \ +l1_%=: exit; \ +" : + : __imm(bpf_ktime_get_ns) + : __clobber_all); +} + +SEC("socket") +__description("check w reg not equal if r reg upper32 bits not 0") +__failure __msg("R1 !read_ok") +__naked void subreg_equality_2(void) +{ + asm volatile (" \ + call %[bpf_ktime_get_ns]; \ + r2 = r0; \ + /* Upper 4-bytes of r2 may not be 0, thus insn \ + * w3 = w2 should not propagate reg id, and \ + * w2 < 9 comparison should not propagate \ + * the range for r3 either. \ + */ \ + w3 = w2; \ + if w2 < 9 goto l0_%=; \ + exit; \ +l0_%=: if r3 < 9 goto l1_%=; \ + /* r1 read is illegal at this point */ \ + r0 -= r1; \ +l1_%=: exit; \ +" : + : __imm(bpf_ktime_get_ns) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_regalloc.c b/tools/testing/selftests/bpf/progs/verifier_regalloc.c new file mode 100644 index 000000000000..ee5ddea87c91 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_regalloc.c @@ -0,0 +1,364 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/regalloc.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +#define MAX_ENTRIES 11 + +struct test_val { + unsigned int index; + int foo[MAX_ENTRIES]; +}; + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 1); + __type(key, long long); + __type(value, struct test_val); +} map_hash_48b SEC(".maps"); + +SEC("tracepoint") +__description("regalloc basic") +__success __flag(BPF_F_ANY_ALIGNMENT) +__naked void regalloc_basic(void) +{ + asm volatile (" \ + r6 = r1; \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r7 = r0; \ + call %[bpf_get_prandom_u32]; \ + r2 = r0; \ + if r0 s> 20 goto l0_%=; \ + if r2 s< 0 goto l0_%=; \ + r7 += r0; \ + r7 += r2; \ + r0 = *(u64*)(r7 + 0); \ +l0_%=: exit; \ +" : + : __imm(bpf_get_prandom_u32), + __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_48b) + : __clobber_all); +} + +SEC("tracepoint") +__description("regalloc negative") +__failure __msg("invalid access to map value, value_size=48 off=48 size=1") +__naked void regalloc_negative(void) +{ + asm volatile (" \ + r6 = r1; \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r7 = r0; \ + call %[bpf_get_prandom_u32]; \ + r2 = r0; \ + if r0 s> 24 goto l0_%=; \ + if r2 s< 0 goto l0_%=; \ + r7 += r0; \ + r7 += r2; \ + r0 = *(u8*)(r7 + 0); \ +l0_%=: exit; \ +" : + : __imm(bpf_get_prandom_u32), + __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_48b) + : __clobber_all); +} + +SEC("tracepoint") +__description("regalloc src_reg mark") +__success __flag(BPF_F_ANY_ALIGNMENT) +__naked void regalloc_src_reg_mark(void) +{ + asm volatile (" \ + r6 = r1; \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r7 = r0; \ + call %[bpf_get_prandom_u32]; \ + r2 = r0; \ + if r0 s> 20 goto l0_%=; \ + r3 = 0; \ + if r3 s>= r2 goto l0_%=; \ + r7 += r0; \ + r7 += r2; \ + r0 = *(u64*)(r7 + 0); \ +l0_%=: exit; \ +" : + : __imm(bpf_get_prandom_u32), + __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_48b) + : __clobber_all); +} + +SEC("tracepoint") +__description("regalloc src_reg negative") +__failure __msg("invalid access to map value, value_size=48 off=44 size=8") +__flag(BPF_F_ANY_ALIGNMENT) +__naked void regalloc_src_reg_negative(void) +{ + asm volatile (" \ + r6 = r1; \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r7 = r0; \ + call %[bpf_get_prandom_u32]; \ + r2 = r0; \ + if r0 s> 22 goto l0_%=; \ + r3 = 0; \ + if r3 s>= r2 goto l0_%=; \ + r7 += r0; \ + r7 += r2; \ + r0 = *(u64*)(r7 + 0); \ +l0_%=: exit; \ +" : + : __imm(bpf_get_prandom_u32), + __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_48b) + : __clobber_all); +} + +SEC("tracepoint") +__description("regalloc and spill") +__success __flag(BPF_F_ANY_ALIGNMENT) +__naked void regalloc_and_spill(void) +{ + asm volatile (" \ + r6 = r1; \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r7 = r0; \ + call %[bpf_get_prandom_u32]; \ + r2 = r0; \ + if r0 s> 20 goto l0_%=; \ + /* r0 has upper bound that should propagate into r2 */\ + *(u64*)(r10 - 8) = r2; /* spill r2 */ \ + r0 = 0; \ + r2 = 0; /* clear r0 and r2 */\ + r3 = *(u64*)(r10 - 8); /* fill r3 */ \ + if r0 s>= r3 goto l0_%=; \ + /* r3 has lower and upper bounds */ \ + r7 += r3; \ + r0 = *(u64*)(r7 + 0); \ +l0_%=: exit; \ +" : + : __imm(bpf_get_prandom_u32), + __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_48b) + : __clobber_all); +} + +SEC("tracepoint") +__description("regalloc and spill negative") +__failure __msg("invalid access to map value, value_size=48 off=48 size=8") +__flag(BPF_F_ANY_ALIGNMENT) +__naked void regalloc_and_spill_negative(void) +{ + asm volatile (" \ + r6 = r1; \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r7 = r0; \ + call %[bpf_get_prandom_u32]; \ + r2 = r0; \ + if r0 s> 48 goto l0_%=; \ + /* r0 has upper bound that should propagate into r2 */\ + *(u64*)(r10 - 8) = r2; /* spill r2 */ \ + r0 = 0; \ + r2 = 0; /* clear r0 and r2 */\ + r3 = *(u64*)(r10 - 8); /* fill r3 */\ + if r0 s>= r3 goto l0_%=; \ + /* r3 has lower and upper bounds */ \ + r7 += r3; \ + r0 = *(u64*)(r7 + 0); \ +l0_%=: exit; \ +" : + : __imm(bpf_get_prandom_u32), + __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_48b) + : __clobber_all); +} + +SEC("tracepoint") +__description("regalloc three regs") +__success __flag(BPF_F_ANY_ALIGNMENT) +__naked void regalloc_three_regs(void) +{ + asm volatile (" \ + r6 = r1; \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r7 = r0; \ + call %[bpf_get_prandom_u32]; \ + r2 = r0; \ + r4 = r2; \ + if r0 s> 12 goto l0_%=; \ + if r2 s< 0 goto l0_%=; \ + r7 += r0; \ + r7 += r2; \ + r7 += r4; \ + r0 = *(u64*)(r7 + 0); \ +l0_%=: exit; \ +" : + : __imm(bpf_get_prandom_u32), + __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_48b) + : __clobber_all); +} + +SEC("tracepoint") +__description("regalloc after call") +__success __flag(BPF_F_ANY_ALIGNMENT) +__naked void regalloc_after_call(void) +{ + asm volatile (" \ + r6 = r1; \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r7 = r0; \ + call %[bpf_get_prandom_u32]; \ + r8 = r0; \ + r9 = r0; \ + call regalloc_after_call__1; \ + if r8 s> 20 goto l0_%=; \ + if r9 s< 0 goto l0_%=; \ + r7 += r8; \ + r7 += r9; \ + r0 = *(u64*)(r7 + 0); \ +l0_%=: exit; \ +" : + : __imm(bpf_get_prandom_u32), + __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_48b) + : __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void regalloc_after_call__1(void) +{ + asm volatile (" \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("tracepoint") +__description("regalloc in callee") +__success __flag(BPF_F_ANY_ALIGNMENT) +__naked void regalloc_in_callee(void) +{ + asm volatile (" \ + r6 = r1; \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r7 = r0; \ + call %[bpf_get_prandom_u32]; \ + r1 = r0; \ + r2 = r0; \ + r3 = r7; \ + call regalloc_in_callee__1; \ +l0_%=: exit; \ +" : + : __imm(bpf_get_prandom_u32), + __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_48b) + : __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void regalloc_in_callee__1(void) +{ + asm volatile (" \ + if r1 s> 20 goto l0_%=; \ + if r2 s< 0 goto l0_%=; \ + r3 += r1; \ + r3 += r2; \ + r0 = *(u64*)(r3 + 0); \ + exit; \ +l0_%=: r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("tracepoint") +__description("regalloc, spill, JEQ") +__success +__naked void regalloc_spill_jeq(void) +{ + asm volatile (" \ + r6 = r1; \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + *(u64*)(r10 - 8) = r0; /* spill r0 */ \ + if r0 == 0 goto l0_%=; \ +l0_%=: /* The verifier will walk the rest twice with r0 == 0 and r0 == map_value */\ + call %[bpf_get_prandom_u32]; \ + r2 = r0; \ + if r2 == 20 goto l1_%=; \ +l1_%=: /* The verifier will walk the rest two more times with r0 == 20 and r0 == unknown */\ + r3 = *(u64*)(r10 - 8); /* fill r3 with map_value */\ + if r3 == 0 goto l2_%=; /* skip ldx if map_value == NULL */\ + /* Buggy verifier will think that r3 == 20 here */\ + r0 = *(u64*)(r3 + 0); /* read from map_value */\ +l2_%=: exit; \ +" : + : __imm(bpf_get_prandom_u32), + __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_48b) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_runtime_jit.c b/tools/testing/selftests/bpf/progs/verifier_runtime_jit.c new file mode 100644 index 000000000000..27ebfc1fd9ee --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_runtime_jit.c @@ -0,0 +1,360 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/runtime_jit.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +void dummy_prog_42_socket(void); +void dummy_prog_24_socket(void); +void dummy_prog_loop1_socket(void); +void dummy_prog_loop2_socket(void); + +struct { + __uint(type, BPF_MAP_TYPE_PROG_ARRAY); + __uint(max_entries, 4); + __uint(key_size, sizeof(int)); + __array(values, void (void)); +} map_prog1_socket SEC(".maps") = { + .values = { + [0] = (void *)&dummy_prog_42_socket, + [1] = (void *)&dummy_prog_loop1_socket, + [2] = (void *)&dummy_prog_24_socket, + }, +}; + +struct { + __uint(type, BPF_MAP_TYPE_PROG_ARRAY); + __uint(max_entries, 8); + __uint(key_size, sizeof(int)); + __array(values, void (void)); +} map_prog2_socket SEC(".maps") = { + .values = { + [1] = (void *)&dummy_prog_loop2_socket, + [2] = (void *)&dummy_prog_24_socket, + [7] = (void *)&dummy_prog_42_socket, + }, +}; + +SEC("socket") +__auxiliary __auxiliary_unpriv +__naked void dummy_prog_42_socket(void) +{ + asm volatile ("r0 = 42; exit;"); +} + +SEC("socket") +__auxiliary __auxiliary_unpriv +__naked void dummy_prog_24_socket(void) +{ + asm volatile ("r0 = 24; exit;"); +} + +SEC("socket") +__auxiliary __auxiliary_unpriv +__naked void dummy_prog_loop1_socket(void) +{ + asm volatile (" \ + r3 = 1; \ + r2 = %[map_prog1_socket] ll; \ + call %[bpf_tail_call]; \ + r0 = 41; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_socket) + : __clobber_all); +} + +SEC("socket") +__auxiliary __auxiliary_unpriv +__naked void dummy_prog_loop2_socket(void) +{ + asm volatile (" \ + r3 = 1; \ + r2 = %[map_prog2_socket] ll; \ + call %[bpf_tail_call]; \ + r0 = 41; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog2_socket) + : __clobber_all); +} + +SEC("socket") +__description("runtime/jit: tail_call within bounds, prog once") +__success __success_unpriv __retval(42) +__naked void call_within_bounds_prog_once(void) +{ + asm volatile (" \ + r3 = 0; \ + r2 = %[map_prog1_socket] ll; \ + call %[bpf_tail_call]; \ + r0 = 1; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_socket) + : __clobber_all); +} + +SEC("socket") +__description("runtime/jit: tail_call within bounds, prog loop") +__success __success_unpriv __retval(41) +__naked void call_within_bounds_prog_loop(void) +{ + asm volatile (" \ + r3 = 1; \ + r2 = %[map_prog1_socket] ll; \ + call %[bpf_tail_call]; \ + r0 = 1; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_socket) + : __clobber_all); +} + +SEC("socket") +__description("runtime/jit: tail_call within bounds, no prog") +__success __success_unpriv __retval(1) +__naked void call_within_bounds_no_prog(void) +{ + asm volatile (" \ + r3 = 3; \ + r2 = %[map_prog1_socket] ll; \ + call %[bpf_tail_call]; \ + r0 = 1; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_socket) + : __clobber_all); +} + +SEC("socket") +__description("runtime/jit: tail_call within bounds, key 2") +__success __success_unpriv __retval(24) +__naked void call_within_bounds_key_2(void) +{ + asm volatile (" \ + r3 = 2; \ + r2 = %[map_prog1_socket] ll; \ + call %[bpf_tail_call]; \ + r0 = 1; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_socket) + : __clobber_all); +} + +SEC("socket") +__description("runtime/jit: tail_call within bounds, key 2 / key 2, first branch") +__success __success_unpriv __retval(24) +__naked void _2_key_2_first_branch(void) +{ + asm volatile (" \ + r0 = 13; \ + *(u8*)(r1 + %[__sk_buff_cb_0]) = r0; \ + r0 = *(u8*)(r1 + %[__sk_buff_cb_0]); \ + if r0 == 13 goto l0_%=; \ + r3 = 2; \ + r2 = %[map_prog1_socket] ll; \ + goto l1_%=; \ +l0_%=: r3 = 2; \ + r2 = %[map_prog1_socket] ll; \ +l1_%=: call %[bpf_tail_call]; \ + r0 = 1; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_socket), + __imm_const(__sk_buff_cb_0, offsetof(struct __sk_buff, cb[0])) + : __clobber_all); +} + +SEC("socket") +__description("runtime/jit: tail_call within bounds, key 2 / key 2, second branch") +__success __success_unpriv __retval(24) +__naked void _2_key_2_second_branch(void) +{ + asm volatile (" \ + r0 = 14; \ + *(u8*)(r1 + %[__sk_buff_cb_0]) = r0; \ + r0 = *(u8*)(r1 + %[__sk_buff_cb_0]); \ + if r0 == 13 goto l0_%=; \ + r3 = 2; \ + r2 = %[map_prog1_socket] ll; \ + goto l1_%=; \ +l0_%=: r3 = 2; \ + r2 = %[map_prog1_socket] ll; \ +l1_%=: call %[bpf_tail_call]; \ + r0 = 1; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_socket), + __imm_const(__sk_buff_cb_0, offsetof(struct __sk_buff, cb[0])) + : __clobber_all); +} + +SEC("socket") +__description("runtime/jit: tail_call within bounds, key 0 / key 2, first branch") +__success __success_unpriv __retval(24) +__naked void _0_key_2_first_branch(void) +{ + asm volatile (" \ + r0 = 13; \ + *(u8*)(r1 + %[__sk_buff_cb_0]) = r0; \ + r0 = *(u8*)(r1 + %[__sk_buff_cb_0]); \ + if r0 == 13 goto l0_%=; \ + r3 = 0; \ + r2 = %[map_prog1_socket] ll; \ + goto l1_%=; \ +l0_%=: r3 = 2; \ + r2 = %[map_prog1_socket] ll; \ +l1_%=: call %[bpf_tail_call]; \ + r0 = 1; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_socket), + __imm_const(__sk_buff_cb_0, offsetof(struct __sk_buff, cb[0])) + : __clobber_all); +} + +SEC("socket") +__description("runtime/jit: tail_call within bounds, key 0 / key 2, second branch") +__success __success_unpriv __retval(42) +__naked void _0_key_2_second_branch(void) +{ + asm volatile (" \ + r0 = 14; \ + *(u8*)(r1 + %[__sk_buff_cb_0]) = r0; \ + r0 = *(u8*)(r1 + %[__sk_buff_cb_0]); \ + if r0 == 13 goto l0_%=; \ + r3 = 0; \ + r2 = %[map_prog1_socket] ll; \ + goto l1_%=; \ +l0_%=: r3 = 2; \ + r2 = %[map_prog1_socket] ll; \ +l1_%=: call %[bpf_tail_call]; \ + r0 = 1; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_socket), + __imm_const(__sk_buff_cb_0, offsetof(struct __sk_buff, cb[0])) + : __clobber_all); +} + +SEC("socket") +__description("runtime/jit: tail_call within bounds, different maps, first branch") +__success __failure_unpriv __msg_unpriv("tail_call abusing map_ptr") +__retval(1) +__naked void bounds_different_maps_first_branch(void) +{ + asm volatile (" \ + r0 = 13; \ + *(u8*)(r1 + %[__sk_buff_cb_0]) = r0; \ + r0 = *(u8*)(r1 + %[__sk_buff_cb_0]); \ + if r0 == 13 goto l0_%=; \ + r3 = 0; \ + r2 = %[map_prog1_socket] ll; \ + goto l1_%=; \ +l0_%=: r3 = 0; \ + r2 = %[map_prog2_socket] ll; \ +l1_%=: call %[bpf_tail_call]; \ + r0 = 1; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_socket), + __imm_addr(map_prog2_socket), + __imm_const(__sk_buff_cb_0, offsetof(struct __sk_buff, cb[0])) + : __clobber_all); +} + +SEC("socket") +__description("runtime/jit: tail_call within bounds, different maps, second branch") +__success __failure_unpriv __msg_unpriv("tail_call abusing map_ptr") +__retval(42) +__naked void bounds_different_maps_second_branch(void) +{ + asm volatile (" \ + r0 = 14; \ + *(u8*)(r1 + %[__sk_buff_cb_0]) = r0; \ + r0 = *(u8*)(r1 + %[__sk_buff_cb_0]); \ + if r0 == 13 goto l0_%=; \ + r3 = 0; \ + r2 = %[map_prog1_socket] ll; \ + goto l1_%=; \ +l0_%=: r3 = 0; \ + r2 = %[map_prog2_socket] ll; \ +l1_%=: call %[bpf_tail_call]; \ + r0 = 1; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_socket), + __imm_addr(map_prog2_socket), + __imm_const(__sk_buff_cb_0, offsetof(struct __sk_buff, cb[0])) + : __clobber_all); +} + +SEC("socket") +__description("runtime/jit: tail_call out of bounds") +__success __success_unpriv __retval(2) +__naked void tail_call_out_of_bounds(void) +{ + asm volatile (" \ + r3 = 256; \ + r2 = %[map_prog1_socket] ll; \ + call %[bpf_tail_call]; \ + r0 = 2; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_socket) + : __clobber_all); +} + +SEC("socket") +__description("runtime/jit: pass negative index to tail_call") +__success __success_unpriv __retval(2) +__naked void negative_index_to_tail_call(void) +{ + asm volatile (" \ + r3 = -1; \ + r2 = %[map_prog1_socket] ll; \ + call %[bpf_tail_call]; \ + r0 = 2; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_socket) + : __clobber_all); +} + +SEC("socket") +__description("runtime/jit: pass > 32bit index to tail_call") +__success __success_unpriv __retval(42) +/* Verifier rewrite for unpriv skips tail call here. */ +__retval_unpriv(2) +__naked void _32bit_index_to_tail_call(void) +{ + asm volatile (" \ + r3 = 0x100000000 ll; \ + r2 = %[map_prog1_socket] ll; \ + call %[bpf_tail_call]; \ + r0 = 2; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_socket) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_search_pruning.c b/tools/testing/selftests/bpf/progs/verifier_search_pruning.c new file mode 100644 index 000000000000..5a14498d352f --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_search_pruning.c @@ -0,0 +1,339 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/search_pruning.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +#define MAX_ENTRIES 11 + +struct test_val { + unsigned int index; + int foo[MAX_ENTRIES]; +}; + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 1); + __type(key, long long); + __type(value, struct test_val); +} map_hash_48b SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 1); + __type(key, long long); + __type(value, long long); +} map_hash_8b SEC(".maps"); + +SEC("socket") +__description("pointer/scalar confusion in state equality check (way 1)") +__success __failure_unpriv __msg_unpriv("R0 leaks addr as return value") +__retval(POINTER_VALUE) +__naked void state_equality_check_way_1(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r0 = *(u64*)(r0 + 0); \ + goto l1_%=; \ +l0_%=: r0 = r10; \ +l1_%=: goto l2_%=; \ +l2_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("pointer/scalar confusion in state equality check (way 2)") +__success __failure_unpriv __msg_unpriv("R0 leaks addr as return value") +__retval(POINTER_VALUE) +__naked void state_equality_check_way_2(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + r0 = r10; \ + goto l1_%=; \ +l0_%=: r0 = *(u64*)(r0 + 0); \ +l1_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("lwt_in") +__description("liveness pruning and write screening") +__failure __msg("R0 !read_ok") +__naked void liveness_pruning_and_write_screening(void) +{ + asm volatile (" \ + /* Get an unknown value */ \ + r2 = *(u32*)(r1 + 0); \ + /* branch conditions teach us nothing about R2 */\ + if r2 >= 0 goto l0_%=; \ + r0 = 0; \ +l0_%=: if r2 >= 0 goto l1_%=; \ + r0 = 0; \ +l1_%=: exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("varlen_map_value_access pruning") +__failure __msg("R0 unbounded memory access") +__failure_unpriv __msg_unpriv("R0 leaks addr") +__flag(BPF_F_ANY_ALIGNMENT) +__naked void varlen_map_value_access_pruning(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = *(u64*)(r0 + 0); \ + w2 = %[max_entries]; \ + if r2 s> r1 goto l1_%=; \ + w1 = 0; \ +l1_%=: w1 <<= 2; \ + r0 += r1; \ + goto l2_%=; \ +l2_%=: r1 = %[test_val_foo]; \ + *(u64*)(r0 + 0) = r1; \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_48b), + __imm_const(max_entries, MAX_ENTRIES), + __imm_const(test_val_foo, offsetof(struct test_val, foo)) + : __clobber_all); +} + +SEC("tracepoint") +__description("search pruning: all branches should be verified (nop operation)") +__failure __msg("R6 invalid mem access 'scalar'") +__naked void should_be_verified_nop_operation(void) +{ + asm volatile (" \ + r2 = r10; \ + r2 += -8; \ + r1 = 0; \ + *(u64*)(r2 + 0) = r1; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r3 = *(u64*)(r0 + 0); \ + if r3 == 0xbeef goto l1_%=; \ + r4 = 0; \ + goto l2_%=; \ +l1_%=: r4 = 1; \ +l2_%=: *(u64*)(r10 - 16) = r4; \ + call %[bpf_ktime_get_ns]; \ + r5 = *(u64*)(r10 - 16); \ + if r5 == 0 goto l0_%=; \ + r6 = 0; \ + r1 = 0xdead; \ + *(u64*)(r6 + 0) = r1; \ +l0_%=: exit; \ +" : + : __imm(bpf_ktime_get_ns), + __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("search pruning: all branches should be verified (invalid stack access)") +/* in privileged mode reads from uninitialized stack locations are permitted */ +__success __failure_unpriv +__msg_unpriv("invalid read from stack off -16+0 size 8") +__retval(0) +__naked void be_verified_invalid_stack_access(void) +{ + asm volatile (" \ + r2 = r10; \ + r2 += -8; \ + r1 = 0; \ + *(u64*)(r2 + 0) = r1; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r3 = *(u64*)(r0 + 0); \ + r4 = 0; \ + if r3 == 0xbeef goto l1_%=; \ + *(u64*)(r10 - 16) = r4; \ + goto l2_%=; \ +l1_%=: *(u64*)(r10 - 24) = r4; \ +l2_%=: call %[bpf_ktime_get_ns]; \ + r5 = *(u64*)(r10 - 16); \ +l0_%=: exit; \ +" : + : __imm(bpf_ktime_get_ns), + __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("tracepoint") +__description("precision tracking for u32 spill/fill") +__failure __msg("R0 min value is outside of the allowed memory range") +__naked void tracking_for_u32_spill_fill(void) +{ + asm volatile (" \ + r7 = r1; \ + call %[bpf_get_prandom_u32]; \ + w6 = 32; \ + if r0 == 0 goto l0_%=; \ + w6 = 4; \ +l0_%=: /* Additional insns to introduce a pruning point. */\ + call %[bpf_get_prandom_u32]; \ + r3 = 0; \ + r3 = 0; \ + if r0 == 0 goto l1_%=; \ + r3 = 0; \ +l1_%=: /* u32 spill/fill */ \ + *(u32*)(r10 - 8) = r6; \ + r8 = *(u32*)(r10 - 8); \ + /* out-of-bound map value access for r6=32 */ \ + r1 = 0; \ + *(u64*)(r10 - 16) = r1; \ + r2 = r10; \ + r2 += -16; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l2_%=; \ + r0 += r8; \ + r1 = *(u32*)(r0 + 0); \ +l2_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32), + __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("tracepoint") +__description("precision tracking for u32 spills, u64 fill") +__failure __msg("div by zero") +__naked void for_u32_spills_u64_fill(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r6 = r0; \ + w7 = 0xffffffff; \ + /* Additional insns to introduce a pruning point. */\ + r3 = 1; \ + r3 = 1; \ + r3 = 1; \ + r3 = 1; \ + call %[bpf_get_prandom_u32]; \ + if r0 == 0 goto l0_%=; \ + r3 = 1; \ +l0_%=: w3 /= 0; \ + /* u32 spills, u64 fill */ \ + *(u32*)(r10 - 4) = r6; \ + *(u32*)(r10 - 8) = r7; \ + r8 = *(u64*)(r10 - 8); \ + /* if r8 != X goto pc+1 r8 known in fallthrough branch */\ + if r8 != 0xffffffff goto l1_%=; \ + r3 = 1; \ +l1_%=: /* if r8 == X goto pc+1 condition always true on first\ + * traversal, so starts backtracking to mark r8 as requiring\ + * precision. r7 marked as needing precision. r6 not marked\ + * since it's not tracked. \ + */ \ + if r8 == 0xffffffff goto l2_%=; \ + /* fails if r8 correctly marked unknown after fill. */\ + w3 /= 0; \ +l2_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("allocated_stack") +__success __msg("processed 15 insns") +__success_unpriv __msg_unpriv("") __log_level(1) __retval(0) +__naked void allocated_stack(void) +{ + asm volatile (" \ + r6 = r1; \ + call %[bpf_get_prandom_u32]; \ + r7 = r0; \ + if r0 == 0 goto l0_%=; \ + r0 = 0; \ + *(u64*)(r10 - 8) = r6; \ + r6 = *(u64*)(r10 - 8); \ + *(u8*)(r10 - 9) = r7; \ + r7 = *(u8*)(r10 - 9); \ +l0_%=: if r0 != 0 goto l1_%=; \ +l1_%=: if r0 != 0 goto l2_%=; \ +l2_%=: if r0 != 0 goto l3_%=; \ +l3_%=: if r0 != 0 goto l4_%=; \ +l4_%=: exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* The test performs a conditional 64-bit write to a stack location + * fp[-8], this is followed by an unconditional 8-bit write to fp[-8], + * then data is read from fp[-8]. This sequence is unsafe. + * + * The test would be mistakenly marked as safe w/o dst register parent + * preservation in verifier.c:copy_register_state() function. + * + * Note the usage of BPF_F_TEST_STATE_FREQ to force creation of the + * checkpoint state after conditional 64-bit assignment. + */ + +SEC("socket") +__description("write tracking and register parent chain bug") +/* in privileged mode reads from uninitialized stack locations are permitted */ +__success __failure_unpriv +__msg_unpriv("invalid read from stack off -8+1 size 8") +__retval(0) __flag(BPF_F_TEST_STATE_FREQ) +__naked void and_register_parent_chain_bug(void) +{ + asm volatile (" \ + /* r6 = ktime_get_ns() */ \ + call %[bpf_ktime_get_ns]; \ + r6 = r0; \ + /* r0 = ktime_get_ns() */ \ + call %[bpf_ktime_get_ns]; \ + /* if r0 > r6 goto +1 */ \ + if r0 > r6 goto l0_%=; \ + /* *(u64 *)(r10 - 8) = 0xdeadbeef */ \ + r0 = 0xdeadbeef; \ + *(u64*)(r10 - 8) = r0; \ +l0_%=: r1 = 42; \ + *(u8*)(r10 - 8) = r1; \ + r2 = *(u64*)(r10 - 8); \ + /* exit(0) */ \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_ktime_get_ns) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_sock.c b/tools/testing/selftests/bpf/progs/verifier_sock.c new file mode 100644 index 000000000000..ee76b51005ab --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_sock.c @@ -0,0 +1,980 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/sock.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +#define sizeof_field(TYPE, MEMBER) sizeof((((TYPE *)0)->MEMBER)) +#define offsetofend(TYPE, MEMBER) \ + (offsetof(TYPE, MEMBER) + sizeof_field(TYPE, MEMBER)) + +struct { + __uint(type, BPF_MAP_TYPE_REUSEPORT_SOCKARRAY); + __uint(max_entries, 1); + __type(key, __u32); + __type(value, __u64); +} map_reuseport_array SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_SOCKHASH); + __uint(max_entries, 1); + __type(key, int); + __type(value, int); +} map_sockhash SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_SOCKMAP); + __uint(max_entries, 1); + __type(key, int); + __type(value, int); +} map_sockmap SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_XSKMAP); + __uint(max_entries, 1); + __type(key, int); + __type(value, int); +} map_xskmap SEC(".maps"); + +struct val { + int cnt; + struct bpf_spin_lock l; +}; + +struct { + __uint(type, BPF_MAP_TYPE_SK_STORAGE); + __uint(max_entries, 0); + __type(key, int); + __type(value, struct val); + __uint(map_flags, BPF_F_NO_PREALLOC); +} sk_storage_map SEC(".maps"); + +SEC("cgroup/skb") +__description("skb->sk: no NULL check") +__failure __msg("invalid mem access 'sock_common_or_null'") +__failure_unpriv +__naked void skb_sk_no_null_check(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + r0 = *(u32*)(r1 + 0); \ + r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("skb->sk: sk->family [non fullsock field]") +__success __success_unpriv __retval(0) +__naked void sk_family_non_fullsock_field_1(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: r0 = *(u32*)(r1 + %[bpf_sock_family]); \ + r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_sock_family, offsetof(struct bpf_sock, family)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("skb->sk: sk->type [fullsock field]") +__failure __msg("invalid sock_common access") +__failure_unpriv +__naked void sk_sk_type_fullsock_field_1(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: r0 = *(u32*)(r1 + %[bpf_sock_type]); \ + r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_sock_type, offsetof(struct bpf_sock, type)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("bpf_sk_fullsock(skb->sk): no !skb->sk check") +__failure __msg("type=sock_common_or_null expected=sock_common") +__failure_unpriv +__naked void sk_no_skb_sk_check_1(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + call %[bpf_sk_fullsock]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("sk_fullsock(skb->sk): no NULL check on ret") +__failure __msg("invalid mem access 'sock_or_null'") +__failure_unpriv +__naked void no_null_check_on_ret_1(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + r0 = *(u32*)(r0 + %[bpf_sock_type]); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_sock_type, offsetof(struct bpf_sock, type)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("sk_fullsock(skb->sk): sk->type [fullsock field]") +__success __success_unpriv __retval(0) +__naked void sk_sk_type_fullsock_field_2(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + r0 = 0; \ + exit; \ +l1_%=: r0 = *(u32*)(r0 + %[bpf_sock_type]); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_sock_type, offsetof(struct bpf_sock, type)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("sk_fullsock(skb->sk): sk->family [non fullsock field]") +__success __success_unpriv __retval(0) +__naked void sk_family_non_fullsock_field_2(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + exit; \ +l1_%=: r0 = *(u32*)(r0 + %[bpf_sock_family]); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_sock_family, offsetof(struct bpf_sock, family)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("sk_fullsock(skb->sk): sk->state [narrow load]") +__success __success_unpriv __retval(0) +__naked void sk_sk_state_narrow_load(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + r0 = 0; \ + exit; \ +l1_%=: r0 = *(u8*)(r0 + %[bpf_sock_state]); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_sock_state, offsetof(struct bpf_sock, state)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("sk_fullsock(skb->sk): sk->dst_port [word load] (backward compatibility)") +__success __success_unpriv __retval(0) +__naked void port_word_load_backward_compatibility(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + r0 = 0; \ + exit; \ +l1_%=: r0 = *(u32*)(r0 + %[bpf_sock_dst_port]); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_sock_dst_port, offsetof(struct bpf_sock, dst_port)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("sk_fullsock(skb->sk): sk->dst_port [half load]") +__success __success_unpriv __retval(0) +__naked void sk_dst_port_half_load(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + r0 = 0; \ + exit; \ +l1_%=: r0 = *(u16*)(r0 + %[bpf_sock_dst_port]); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_sock_dst_port, offsetof(struct bpf_sock, dst_port)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("sk_fullsock(skb->sk): sk->dst_port [half load] (invalid)") +__failure __msg("invalid sock access") +__failure_unpriv +__naked void dst_port_half_load_invalid_1(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + r0 = 0; \ + exit; \ +l1_%=: r0 = *(u16*)(r0 + %[__imm_0]); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__imm_0, offsetof(struct bpf_sock, dst_port) + 2), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("sk_fullsock(skb->sk): sk->dst_port [byte load]") +__success __success_unpriv __retval(0) +__naked void sk_dst_port_byte_load(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + r0 = 0; \ + exit; \ +l1_%=: r2 = *(u8*)(r0 + %[bpf_sock_dst_port]); \ + r2 = *(u8*)(r0 + %[__imm_0]); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__imm_0, offsetof(struct bpf_sock, dst_port) + 1), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_sock_dst_port, offsetof(struct bpf_sock, dst_port)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("sk_fullsock(skb->sk): sk->dst_port [byte load] (invalid)") +__failure __msg("invalid sock access") +__failure_unpriv +__naked void dst_port_byte_load_invalid(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + r0 = 0; \ + exit; \ +l1_%=: r0 = *(u8*)(r0 + %[__imm_0]); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__imm_0, offsetof(struct bpf_sock, dst_port) + 2), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("sk_fullsock(skb->sk): past sk->dst_port [half load] (invalid)") +__failure __msg("invalid sock access") +__failure_unpriv +__naked void dst_port_half_load_invalid_2(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + r0 = 0; \ + exit; \ +l1_%=: r0 = *(u16*)(r0 + %[bpf_sock_dst_port__end]); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_sock_dst_port__end, offsetofend(struct bpf_sock, dst_port)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("sk_fullsock(skb->sk): sk->dst_ip6 [load 2nd byte]") +__success __success_unpriv __retval(0) +__naked void dst_ip6_load_2nd_byte(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + r0 = 0; \ + exit; \ +l1_%=: r0 = *(u8*)(r0 + %[__imm_0]); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__imm_0, offsetof(struct bpf_sock, dst_ip6[0]) + 1), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("sk_fullsock(skb->sk): sk->type [narrow load]") +__success __success_unpriv __retval(0) +__naked void sk_sk_type_narrow_load(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + r0 = 0; \ + exit; \ +l1_%=: r0 = *(u8*)(r0 + %[bpf_sock_type]); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_sock_type, offsetof(struct bpf_sock, type)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("sk_fullsock(skb->sk): sk->protocol [narrow load]") +__success __success_unpriv __retval(0) +__naked void sk_sk_protocol_narrow_load(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + r0 = 0; \ + exit; \ +l1_%=: r0 = *(u8*)(r0 + %[bpf_sock_protocol]); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_sock_protocol, offsetof(struct bpf_sock, protocol)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("sk_fullsock(skb->sk): beyond last field") +__failure __msg("invalid sock access") +__failure_unpriv +__naked void skb_sk_beyond_last_field_1(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + r0 = 0; \ + exit; \ +l1_%=: r0 = *(u32*)(r0 + %[bpf_sock_rx_queue_mapping__end]);\ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_sock_rx_queue_mapping__end, offsetofend(struct bpf_sock, rx_queue_mapping)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("bpf_tcp_sock(skb->sk): no !skb->sk check") +__failure __msg("type=sock_common_or_null expected=sock_common") +__failure_unpriv +__naked void sk_no_skb_sk_check_2(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + call %[bpf_tcp_sock]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_tcp_sock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("bpf_tcp_sock(skb->sk): no NULL check on ret") +__failure __msg("invalid mem access 'tcp_sock_or_null'") +__failure_unpriv +__naked void no_null_check_on_ret_2(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_tcp_sock]; \ + r0 = *(u32*)(r0 + %[bpf_tcp_sock_snd_cwnd]); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_tcp_sock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_tcp_sock_snd_cwnd, offsetof(struct bpf_tcp_sock, snd_cwnd)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("bpf_tcp_sock(skb->sk): tp->snd_cwnd") +__success __success_unpriv __retval(0) +__naked void skb_sk_tp_snd_cwnd_1(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_tcp_sock]; \ + if r0 != 0 goto l1_%=; \ + exit; \ +l1_%=: r0 = *(u32*)(r0 + %[bpf_tcp_sock_snd_cwnd]); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_tcp_sock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_tcp_sock_snd_cwnd, offsetof(struct bpf_tcp_sock, snd_cwnd)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("bpf_tcp_sock(skb->sk): tp->bytes_acked") +__success __success_unpriv __retval(0) +__naked void skb_sk_tp_bytes_acked(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_tcp_sock]; \ + if r0 != 0 goto l1_%=; \ + exit; \ +l1_%=: r0 = *(u64*)(r0 + %[bpf_tcp_sock_bytes_acked]); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_tcp_sock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_tcp_sock_bytes_acked, offsetof(struct bpf_tcp_sock, bytes_acked)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("bpf_tcp_sock(skb->sk): beyond last field") +__failure __msg("invalid tcp_sock access") +__failure_unpriv +__naked void skb_sk_beyond_last_field_2(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_tcp_sock]; \ + if r0 != 0 goto l1_%=; \ + exit; \ +l1_%=: r0 = *(u64*)(r0 + %[bpf_tcp_sock_bytes_acked__end]);\ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_tcp_sock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_tcp_sock_bytes_acked__end, offsetofend(struct bpf_tcp_sock, bytes_acked)) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("bpf_tcp_sock(bpf_sk_fullsock(skb->sk)): tp->snd_cwnd") +__success __success_unpriv __retval(0) +__naked void skb_sk_tp_snd_cwnd_2(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + exit; \ +l1_%=: r1 = r0; \ + call %[bpf_tcp_sock]; \ + if r0 != 0 goto l2_%=; \ + exit; \ +l2_%=: r0 = *(u32*)(r0 + %[bpf_tcp_sock_snd_cwnd]); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm(bpf_tcp_sock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)), + __imm_const(bpf_tcp_sock_snd_cwnd, offsetof(struct bpf_tcp_sock, snd_cwnd)) + : __clobber_all); +} + +SEC("tc") +__description("bpf_sk_release(skb->sk)") +__failure __msg("R1 must be referenced when passed to release function") +__naked void bpf_sk_release_skb_sk(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 == 0 goto l0_%=; \ + call %[bpf_sk_release]; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_release), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)) + : __clobber_all); +} + +SEC("tc") +__description("bpf_sk_release(bpf_sk_fullsock(skb->sk))") +__failure __msg("R1 must be referenced when passed to release function") +__naked void bpf_sk_fullsock_skb_sk(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + exit; \ +l1_%=: r1 = r0; \ + call %[bpf_sk_release]; \ + r0 = 1; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm(bpf_sk_release), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)) + : __clobber_all); +} + +SEC("tc") +__description("bpf_sk_release(bpf_tcp_sock(skb->sk))") +__failure __msg("R1 must be referenced when passed to release function") +__naked void bpf_tcp_sock_skb_sk(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_tcp_sock]; \ + if r0 != 0 goto l1_%=; \ + exit; \ +l1_%=: r1 = r0; \ + call %[bpf_sk_release]; \ + r0 = 1; \ + exit; \ +" : + : __imm(bpf_sk_release), + __imm(bpf_tcp_sock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)) + : __clobber_all); +} + +SEC("tc") +__description("sk_storage_get(map, skb->sk, NULL, 0): value == NULL") +__success __retval(0) +__naked void sk_null_0_value_null(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + r0 = 0; \ + exit; \ +l1_%=: r4 = 0; \ + r3 = 0; \ + r2 = r0; \ + r1 = %[sk_storage_map] ll; \ + call %[bpf_sk_storage_get]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm(bpf_sk_storage_get), + __imm_addr(sk_storage_map), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)) + : __clobber_all); +} + +SEC("tc") +__description("sk_storage_get(map, skb->sk, 1, 1): value == 1") +__failure __msg("R3 type=scalar expected=fp") +__naked void sk_1_1_value_1(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + r0 = 0; \ + exit; \ +l1_%=: r4 = 1; \ + r3 = 1; \ + r2 = r0; \ + r1 = %[sk_storage_map] ll; \ + call %[bpf_sk_storage_get]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm(bpf_sk_storage_get), + __imm_addr(sk_storage_map), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)) + : __clobber_all); +} + +SEC("tc") +__description("sk_storage_get(map, skb->sk, &stack_value, 1): stack_value") +__success __retval(0) +__naked void stack_value_1_stack_value(void) +{ + asm volatile (" \ + r2 = 0; \ + *(u64*)(r10 - 8) = r2; \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: call %[bpf_sk_fullsock]; \ + if r0 != 0 goto l1_%=; \ + r0 = 0; \ + exit; \ +l1_%=: r4 = 1; \ + r3 = r10; \ + r3 += -8; \ + r2 = r0; \ + r1 = %[sk_storage_map] ll; \ + call %[bpf_sk_storage_get]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_fullsock), + __imm(bpf_sk_storage_get), + __imm_addr(sk_storage_map), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)) + : __clobber_all); +} + +SEC("tc") +__description("bpf_map_lookup_elem(smap, &key)") +__failure __msg("cannot pass map_type 24 into func bpf_map_lookup_elem") +__naked void map_lookup_elem_smap_key(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[sk_storage_map] ll; \ + call %[bpf_map_lookup_elem]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(sk_storage_map) + : __clobber_all); +} + +SEC("xdp") +__description("bpf_map_lookup_elem(xskmap, &key); xs->queue_id") +__success __retval(0) +__naked void xskmap_key_xs_queue_id(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_xskmap] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r0 = *(u32*)(r0 + %[bpf_xdp_sock_queue_id]); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_xskmap), + __imm_const(bpf_xdp_sock_queue_id, offsetof(struct bpf_xdp_sock, queue_id)) + : __clobber_all); +} + +SEC("sk_skb") +__description("bpf_map_lookup_elem(sockmap, &key)") +__failure __msg("Unreleased reference id=2 alloc_insn=6") +__naked void map_lookup_elem_sockmap_key(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_sockmap] ll; \ + call %[bpf_map_lookup_elem]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_sockmap) + : __clobber_all); +} + +SEC("sk_skb") +__description("bpf_map_lookup_elem(sockhash, &key)") +__failure __msg("Unreleased reference id=2 alloc_insn=6") +__naked void map_lookup_elem_sockhash_key(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_sockhash] ll; \ + call %[bpf_map_lookup_elem]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_sockhash) + : __clobber_all); +} + +SEC("sk_skb") +__description("bpf_map_lookup_elem(sockmap, &key); sk->type [fullsock field]; bpf_sk_release(sk)") +__success +__naked void field_bpf_sk_release_sk_1(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_sockmap] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r1 = r0; \ + r0 = *(u32*)(r0 + %[bpf_sock_type]); \ + call %[bpf_sk_release]; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm(bpf_sk_release), + __imm_addr(map_sockmap), + __imm_const(bpf_sock_type, offsetof(struct bpf_sock, type)) + : __clobber_all); +} + +SEC("sk_skb") +__description("bpf_map_lookup_elem(sockhash, &key); sk->type [fullsock field]; bpf_sk_release(sk)") +__success +__naked void field_bpf_sk_release_sk_2(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_sockhash] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r1 = r0; \ + r0 = *(u32*)(r0 + %[bpf_sock_type]); \ + call %[bpf_sk_release]; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm(bpf_sk_release), + __imm_addr(map_sockhash), + __imm_const(bpf_sock_type, offsetof(struct bpf_sock, type)) + : __clobber_all); +} + +SEC("sk_reuseport") +__description("bpf_sk_select_reuseport(ctx, reuseport_array, &key, flags)") +__success +__naked void ctx_reuseport_array_key_flags(void) +{ + asm volatile (" \ + r4 = 0; \ + r2 = 0; \ + *(u32*)(r10 - 4) = r2; \ + r3 = r10; \ + r3 += -4; \ + r2 = %[map_reuseport_array] ll; \ + call %[bpf_sk_select_reuseport]; \ + exit; \ +" : + : __imm(bpf_sk_select_reuseport), + __imm_addr(map_reuseport_array) + : __clobber_all); +} + +SEC("sk_reuseport") +__description("bpf_sk_select_reuseport(ctx, sockmap, &key, flags)") +__success +__naked void reuseport_ctx_sockmap_key_flags(void) +{ + asm volatile (" \ + r4 = 0; \ + r2 = 0; \ + *(u32*)(r10 - 4) = r2; \ + r3 = r10; \ + r3 += -4; \ + r2 = %[map_sockmap] ll; \ + call %[bpf_sk_select_reuseport]; \ + exit; \ +" : + : __imm(bpf_sk_select_reuseport), + __imm_addr(map_sockmap) + : __clobber_all); +} + +SEC("sk_reuseport") +__description("bpf_sk_select_reuseport(ctx, sockhash, &key, flags)") +__success +__naked void reuseport_ctx_sockhash_key_flags(void) +{ + asm volatile (" \ + r4 = 0; \ + r2 = 0; \ + *(u32*)(r10 - 4) = r2; \ + r3 = r10; \ + r3 += -4; \ + r2 = %[map_sockmap] ll; \ + call %[bpf_sk_select_reuseport]; \ + exit; \ +" : + : __imm(bpf_sk_select_reuseport), + __imm_addr(map_sockmap) + : __clobber_all); +} + +SEC("tc") +__description("mark null check on return value of bpf_skc_to helpers") +__failure __msg("invalid mem access") +__naked void of_bpf_skc_to_helpers(void) +{ + asm volatile (" \ + r1 = *(u64*)(r1 + %[__sk_buff_sk]); \ + if r1 != 0 goto l0_%=; \ + r0 = 0; \ + exit; \ +l0_%=: r6 = r1; \ + call %[bpf_skc_to_tcp_sock]; \ + r7 = r0; \ + r1 = r6; \ + call %[bpf_skc_to_tcp_request_sock]; \ + r8 = r0; \ + if r8 != 0 goto l1_%=; \ + r0 = 0; \ + exit; \ +l1_%=: r0 = *(u8*)(r7 + 0); \ + exit; \ +" : + : __imm(bpf_skc_to_tcp_request_sock), + __imm(bpf_skc_to_tcp_sock), + __imm_const(__sk_buff_sk, offsetof(struct __sk_buff, sk)) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_spin_lock.c b/tools/testing/selftests/bpf/progs/verifier_spin_lock.c new file mode 100644 index 000000000000..9c1aa69650f8 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_spin_lock.c @@ -0,0 +1,533 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/spin_lock.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +struct val { + int cnt; + struct bpf_spin_lock l; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, int); + __type(value, struct val); +} map_spin_lock SEC(".maps"); + +SEC("cgroup/skb") +__description("spin_lock: test1 success") +__success __failure_unpriv __msg_unpriv("") +__retval(0) +__naked void spin_lock_test1_success(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_spin_lock] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r1 = r0; \ + r1 += 4; \ + call %[bpf_spin_lock]; \ + r1 = r6; \ + r1 += 4; \ + r0 = *(u32*)(r6 + 0); \ + call %[bpf_spin_unlock]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm(bpf_spin_lock), + __imm(bpf_spin_unlock), + __imm_addr(map_spin_lock) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("spin_lock: test2 direct ld/st") +__failure __msg("cannot be accessed directly") +__failure_unpriv __msg_unpriv("") +__naked void lock_test2_direct_ld_st(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_spin_lock] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r1 = r0; \ + r1 += 4; \ + call %[bpf_spin_lock]; \ + r1 = r6; \ + r1 += 4; \ + r0 = *(u32*)(r1 + 0); \ + call %[bpf_spin_unlock]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm(bpf_spin_lock), + __imm(bpf_spin_unlock), + __imm_addr(map_spin_lock) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("spin_lock: test3 direct ld/st") +__failure __msg("cannot be accessed directly") +__failure_unpriv __msg_unpriv("") +__flag(BPF_F_ANY_ALIGNMENT) +__naked void lock_test3_direct_ld_st(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_spin_lock] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r1 = r0; \ + r1 += 4; \ + call %[bpf_spin_lock]; \ + r1 = r6; \ + r1 += 4; \ + r0 = *(u32*)(r6 + 1); \ + call %[bpf_spin_unlock]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm(bpf_spin_lock), + __imm(bpf_spin_unlock), + __imm_addr(map_spin_lock) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("spin_lock: test4 direct ld/st") +__failure __msg("cannot be accessed directly") +__failure_unpriv __msg_unpriv("") +__flag(BPF_F_ANY_ALIGNMENT) +__naked void lock_test4_direct_ld_st(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_spin_lock] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r1 = r0; \ + r1 += 4; \ + call %[bpf_spin_lock]; \ + r1 = r6; \ + r1 += 4; \ + r0 = *(u16*)(r6 + 3); \ + call %[bpf_spin_unlock]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm(bpf_spin_lock), + __imm(bpf_spin_unlock), + __imm_addr(map_spin_lock) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("spin_lock: test5 call within a locked region") +__failure __msg("calls are not allowed") +__failure_unpriv __msg_unpriv("") +__naked void call_within_a_locked_region(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_spin_lock] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r1 = r0; \ + r1 += 4; \ + call %[bpf_spin_lock]; \ + call %[bpf_get_prandom_u32]; \ + r1 = r6; \ + r1 += 4; \ + call %[bpf_spin_unlock]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32), + __imm(bpf_map_lookup_elem), + __imm(bpf_spin_lock), + __imm(bpf_spin_unlock), + __imm_addr(map_spin_lock) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("spin_lock: test6 missing unlock") +__failure __msg("unlock is missing") +__failure_unpriv __msg_unpriv("") +__naked void spin_lock_test6_missing_unlock(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_spin_lock] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r1 = r0; \ + r1 += 4; \ + call %[bpf_spin_lock]; \ + r1 = r6; \ + r1 += 4; \ + r0 = *(u32*)(r6 + 0); \ + if r0 != 0 goto l1_%=; \ + call %[bpf_spin_unlock]; \ +l1_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm(bpf_spin_lock), + __imm(bpf_spin_unlock), + __imm_addr(map_spin_lock) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("spin_lock: test7 unlock without lock") +__failure __msg("without taking a lock") +__failure_unpriv __msg_unpriv("") +__naked void lock_test7_unlock_without_lock(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_spin_lock] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r1 = r0; \ + r1 += 4; \ + if r1 != 0 goto l1_%=; \ + call %[bpf_spin_lock]; \ +l1_%=: r1 = r6; \ + r1 += 4; \ + r0 = *(u32*)(r6 + 0); \ + call %[bpf_spin_unlock]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm(bpf_spin_lock), + __imm(bpf_spin_unlock), + __imm_addr(map_spin_lock) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("spin_lock: test8 double lock") +__failure __msg("calls are not allowed") +__failure_unpriv __msg_unpriv("") +__naked void spin_lock_test8_double_lock(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_spin_lock] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r1 = r0; \ + r1 += 4; \ + call %[bpf_spin_lock]; \ + r1 = r6; \ + r1 += 4; \ + call %[bpf_spin_lock]; \ + r1 = r6; \ + r1 += 4; \ + r0 = *(u32*)(r6 + 0); \ + call %[bpf_spin_unlock]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm(bpf_spin_lock), + __imm(bpf_spin_unlock), + __imm_addr(map_spin_lock) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("spin_lock: test9 different lock") +__failure __msg("unlock of different lock") +__failure_unpriv __msg_unpriv("") +__naked void spin_lock_test9_different_lock(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_spin_lock] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_spin_lock] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l1_%=; \ + exit; \ +l1_%=: r7 = r0; \ + r1 = r6; \ + r1 += 4; \ + call %[bpf_spin_lock]; \ + r1 = r7; \ + r1 += 4; \ + call %[bpf_spin_unlock]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm(bpf_spin_lock), + __imm(bpf_spin_unlock), + __imm_addr(map_spin_lock) + : __clobber_all); +} + +SEC("cgroup/skb") +__description("spin_lock: test10 lock in subprog without unlock") +__failure __msg("unlock is missing") +__failure_unpriv __msg_unpriv("") +__naked void lock_in_subprog_without_unlock(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_spin_lock] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r6 = r0; \ + r1 = r0; \ + r1 += 4; \ + call lock_in_subprog_without_unlock__1; \ + r1 = r6; \ + r1 += 4; \ + call %[bpf_spin_unlock]; \ + r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm(bpf_spin_unlock), + __imm_addr(map_spin_lock) + : __clobber_all); +} + +static __naked __noinline __attribute__((used)) +void lock_in_subprog_without_unlock__1(void) +{ + asm volatile (" \ + call %[bpf_spin_lock]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_spin_lock) + : __clobber_all); +} + +SEC("tc") +__description("spin_lock: test11 ld_abs under lock") +__failure __msg("inside bpf_spin_lock") +__naked void test11_ld_abs_under_lock(void) +{ + asm volatile (" \ + r6 = r1; \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_spin_lock] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r7 = r0; \ + r1 = r0; \ + r1 += 4; \ + call %[bpf_spin_lock]; \ + r0 = *(u8*)skb[0]; \ + r1 = r7; \ + r1 += 4; \ + call %[bpf_spin_unlock]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm(bpf_spin_lock), + __imm(bpf_spin_unlock), + __imm_addr(map_spin_lock) + : __clobber_all); +} + +SEC("tc") +__description("spin_lock: regsafe compare reg->id for map value") +__failure __msg("bpf_spin_unlock of different lock") +__flag(BPF_F_TEST_STATE_FREQ) +__naked void reg_id_for_map_value(void) +{ + asm volatile (" \ + r6 = r1; \ + r6 = *(u32*)(r6 + %[__sk_buff_mark]); \ + r1 = %[map_spin_lock] ll; \ + r9 = r1; \ + r2 = 0; \ + *(u32*)(r10 - 4) = r2; \ + r2 = r10; \ + r2 += -4; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r7 = r0; \ + r1 = r9; \ + r2 = r10; \ + r2 += -4; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l1_%=; \ + exit; \ +l1_%=: r8 = r0; \ + r1 = r7; \ + r1 += 4; \ + call %[bpf_spin_lock]; \ + if r6 == 0 goto l2_%=; \ + goto l3_%=; \ +l2_%=: r7 = r8; \ +l3_%=: r1 = r7; \ + r1 += 4; \ + call %[bpf_spin_unlock]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm(bpf_spin_lock), + __imm(bpf_spin_unlock), + __imm_addr(map_spin_lock), + __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark)) + : __clobber_all); +} + +/* Make sure that regsafe() compares ids for spin lock records using + * check_ids(): + * 1: r9 = map_lookup_elem(...) ; r9.id == 1 + * 2: r8 = map_lookup_elem(...) ; r8.id == 2 + * 3: r7 = ktime_get_ns() + * 4: r6 = ktime_get_ns() + * 5: if r6 > r7 goto <9> + * 6: spin_lock(r8) + * 7: r9 = r8 + * 8: goto <10> + * 9: spin_lock(r9) + * 10: spin_unlock(r9) ; r9.id == 1 || r9.id == 2 and lock is active, + * ; second visit to (10) should be considered safe + * ; if check_ids() is used. + * 11: exit(0) + */ + +SEC("cgroup/skb") +__description("spin_lock: regsafe() check_ids() similar id mappings") +__success __msg("29: safe") +__failure_unpriv __msg_unpriv("") +__log_level(2) __retval(0) __flag(BPF_F_TEST_STATE_FREQ) +__naked void check_ids_similar_id_mappings(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u32*)(r10 - 4) = r1; \ + /* r9 = map_lookup_elem(...) */ \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_spin_lock] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r9 = r0; \ + /* r8 = map_lookup_elem(...) */ \ + r2 = r10; \ + r2 += -4; \ + r1 = %[map_spin_lock] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l1_%=; \ + r8 = r0; \ + /* r7 = ktime_get_ns() */ \ + call %[bpf_ktime_get_ns]; \ + r7 = r0; \ + /* r6 = ktime_get_ns() */ \ + call %[bpf_ktime_get_ns]; \ + r6 = r0; \ + /* if r6 > r7 goto +5 ; no new information about the state is derived from\ + * ; this check, thus produced verifier states differ\ + * ; only in 'insn_idx' \ + * spin_lock(r8) \ + * r9 = r8 \ + * goto unlock \ + */ \ + if r6 > r7 goto l2_%=; \ + r1 = r8; \ + r1 += 4; \ + call %[bpf_spin_lock]; \ + r9 = r8; \ + goto l3_%=; \ +l2_%=: /* spin_lock(r9) */ \ + r1 = r9; \ + r1 += 4; \ + call %[bpf_spin_lock]; \ +l3_%=: /* spin_unlock(r9) */ \ + r1 = r9; \ + r1 += 4; \ + call %[bpf_spin_unlock]; \ +l0_%=: /* exit(0) */ \ + r0 = 0; \ +l1_%=: exit; \ +" : + : __imm(bpf_ktime_get_ns), + __imm(bpf_map_lookup_elem), + __imm(bpf_spin_lock), + __imm(bpf_spin_unlock), + __imm_addr(map_spin_lock) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_subreg.c b/tools/testing/selftests/bpf/progs/verifier_subreg.c new file mode 100644 index 000000000000..8613ea160dcd --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_subreg.c @@ -0,0 +1,673 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/subreg.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +/* This file contains sub-register zero extension checks for insns defining + * sub-registers, meaning: + * - All insns under BPF_ALU class. Their BPF_ALU32 variants or narrow width + * forms (BPF_END) could define sub-registers. + * - Narrow direct loads, BPF_B/H/W | BPF_LDX. + * - BPF_LD is not exposed to JIT back-ends, so no need for testing. + * + * "get_prandom_u32" is used to initialize low 32-bit of some registers to + * prevent potential optimizations done by verifier or JIT back-ends which could + * optimize register back into constant when range info shows one register is a + * constant. + */ + +SEC("socket") +__description("add32 reg zero extend check") +__success __success_unpriv __retval(0) +__naked void add32_reg_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = r0; \ + r0 = 0x100000000 ll; \ + w0 += w1; \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("add32 imm zero extend check") +__success __success_unpriv __retval(0) +__naked void add32_imm_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + /* An insn could have no effect on the low 32-bit, for example:\ + * a = a + 0 \ + * a = a | 0 \ + * a = a & -1 \ + * But, they should still zero high 32-bit. \ + */ \ + w0 += 0; \ + r0 >>= 32; \ + r6 = r0; \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 += -2; \ + r0 >>= 32; \ + r0 |= r6; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("sub32 reg zero extend check") +__success __success_unpriv __retval(0) +__naked void sub32_reg_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = r0; \ + r0 = 0x1ffffffff ll; \ + w0 -= w1; \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("sub32 imm zero extend check") +__success __success_unpriv __retval(0) +__naked void sub32_imm_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 -= 0; \ + r0 >>= 32; \ + r6 = r0; \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 -= 1; \ + r0 >>= 32; \ + r0 |= r6; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("mul32 reg zero extend check") +__success __success_unpriv __retval(0) +__naked void mul32_reg_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = r0; \ + r0 = 0x100000001 ll; \ + w0 *= w1; \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("mul32 imm zero extend check") +__success __success_unpriv __retval(0) +__naked void mul32_imm_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 *= 1; \ + r0 >>= 32; \ + r6 = r0; \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 *= -1; \ + r0 >>= 32; \ + r0 |= r6; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("div32 reg zero extend check") +__success __success_unpriv __retval(0) +__naked void div32_reg_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = r0; \ + r0 = -1; \ + w0 /= w1; \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("div32 imm zero extend check") +__success __success_unpriv __retval(0) +__naked void div32_imm_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 /= 1; \ + r0 >>= 32; \ + r6 = r0; \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 /= 2; \ + r0 >>= 32; \ + r0 |= r6; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("or32 reg zero extend check") +__success __success_unpriv __retval(0) +__naked void or32_reg_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = r0; \ + r0 = 0x100000001 ll; \ + w0 |= w1; \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("or32 imm zero extend check") +__success __success_unpriv __retval(0) +__naked void or32_imm_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 |= 0; \ + r0 >>= 32; \ + r6 = r0; \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 |= 1; \ + r0 >>= 32; \ + r0 |= r6; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("and32 reg zero extend check") +__success __success_unpriv __retval(0) +__naked void and32_reg_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x100000000 ll; \ + r1 |= r0; \ + r0 = 0x1ffffffff ll; \ + w0 &= w1; \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("and32 imm zero extend check") +__success __success_unpriv __retval(0) +__naked void and32_imm_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 &= -1; \ + r0 >>= 32; \ + r6 = r0; \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 &= -2; \ + r0 >>= 32; \ + r0 |= r6; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("lsh32 reg zero extend check") +__success __success_unpriv __retval(0) +__naked void lsh32_reg_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x100000000 ll; \ + r0 |= r1; \ + r1 = 1; \ + w0 <<= w1; \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("lsh32 imm zero extend check") +__success __success_unpriv __retval(0) +__naked void lsh32_imm_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 <<= 0; \ + r0 >>= 32; \ + r6 = r0; \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 <<= 1; \ + r0 >>= 32; \ + r0 |= r6; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("rsh32 reg zero extend check") +__success __success_unpriv __retval(0) +__naked void rsh32_reg_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + r1 = 1; \ + w0 >>= w1; \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("rsh32 imm zero extend check") +__success __success_unpriv __retval(0) +__naked void rsh32_imm_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 >>= 0; \ + r0 >>= 32; \ + r6 = r0; \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 >>= 1; \ + r0 >>= 32; \ + r0 |= r6; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("neg32 reg zero extend check") +__success __success_unpriv __retval(0) +__naked void neg32_reg_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 = -w0; \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("mod32 reg zero extend check") +__success __success_unpriv __retval(0) +__naked void mod32_reg_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = r0; \ + r0 = -1; \ + w0 %%= w1; \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("mod32 imm zero extend check") +__success __success_unpriv __retval(0) +__naked void mod32_imm_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 %%= 1; \ + r0 >>= 32; \ + r6 = r0; \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 %%= 2; \ + r0 >>= 32; \ + r0 |= r6; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("xor32 reg zero extend check") +__success __success_unpriv __retval(0) +__naked void xor32_reg_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = r0; \ + r0 = 0x100000000 ll; \ + w0 ^= w1; \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("xor32 imm zero extend check") +__success __success_unpriv __retval(0) +__naked void xor32_imm_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 ^= 1; \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("mov32 reg zero extend check") +__success __success_unpriv __retval(0) +__naked void mov32_reg_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x100000000 ll; \ + r1 |= r0; \ + r0 = 0x100000000 ll; \ + w0 = w1; \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("mov32 imm zero extend check") +__success __success_unpriv __retval(0) +__naked void mov32_imm_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 = 0; \ + r0 >>= 32; \ + r6 = r0; \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 = 1; \ + r0 >>= 32; \ + r0 |= r6; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("arsh32 reg zero extend check") +__success __success_unpriv __retval(0) +__naked void arsh32_reg_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + r1 = 1; \ + w0 s>>= w1; \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("arsh32 imm zero extend check") +__success __success_unpriv __retval(0) +__naked void arsh32_imm_zero_extend_check(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 s>>= 0; \ + r0 >>= 32; \ + r6 = r0; \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + w0 s>>= 1; \ + r0 >>= 32; \ + r0 |= r6; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("end16 (to_le) reg zero extend check") +__success __success_unpriv __retval(0) +__naked void le_reg_zero_extend_check_1(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r6 = r0; \ + r6 <<= 32; \ + call %[bpf_get_prandom_u32]; \ + r0 |= r6; \ + r0 = le16 r0; \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("end32 (to_le) reg zero extend check") +__success __success_unpriv __retval(0) +__naked void le_reg_zero_extend_check_2(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r6 = r0; \ + r6 <<= 32; \ + call %[bpf_get_prandom_u32]; \ + r0 |= r6; \ + r0 = le32 r0; \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("end16 (to_be) reg zero extend check") +__success __success_unpriv __retval(0) +__naked void be_reg_zero_extend_check_1(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r6 = r0; \ + r6 <<= 32; \ + call %[bpf_get_prandom_u32]; \ + r0 |= r6; \ + r0 = be16 r0; \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("end32 (to_be) reg zero extend check") +__success __success_unpriv __retval(0) +__naked void be_reg_zero_extend_check_2(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r6 = r0; \ + r6 <<= 32; \ + call %[bpf_get_prandom_u32]; \ + r0 |= r6; \ + r0 = be32 r0; \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("ldx_b zero extend check") +__success __success_unpriv __retval(0) +__naked void ldx_b_zero_extend_check(void) +{ + asm volatile (" \ + r6 = r10; \ + r6 += -4; \ + r7 = 0xfaceb00c; \ + *(u32*)(r6 + 0) = r7; \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + r0 = *(u8*)(r6 + 0); \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("ldx_h zero extend check") +__success __success_unpriv __retval(0) +__naked void ldx_h_zero_extend_check(void) +{ + asm volatile (" \ + r6 = r10; \ + r6 += -4; \ + r7 = 0xfaceb00c; \ + *(u32*)(r6 + 0) = r7; \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + r0 = *(u16*)(r6 + 0); \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("ldx_w zero extend check") +__success __success_unpriv __retval(0) +__naked void ldx_w_zero_extend_check(void) +{ + asm volatile (" \ + r6 = r10; \ + r6 += -4; \ + r7 = 0xfaceb00c; \ + *(u32*)(r6 + 0) = r7; \ + call %[bpf_get_prandom_u32]; \ + r1 = 0x1000000000 ll; \ + r0 |= r1; \ + r0 = *(u32*)(r6 + 0); \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_unpriv.c b/tools/testing/selftests/bpf/progs/verifier_unpriv.c new file mode 100644 index 000000000000..7ea535bfbacd --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_unpriv.c @@ -0,0 +1,726 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/unpriv.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "../../../include/linux/filter.h" +#include "bpf_misc.h" + +#define BPF_SK_LOOKUP(func) \ + /* struct bpf_sock_tuple tuple = {} */ \ + "r2 = 0;" \ + "*(u32*)(r10 - 8) = r2;" \ + "*(u64*)(r10 - 16) = r2;" \ + "*(u64*)(r10 - 24) = r2;" \ + "*(u64*)(r10 - 32) = r2;" \ + "*(u64*)(r10 - 40) = r2;" \ + "*(u64*)(r10 - 48) = r2;" \ + /* sk = func(ctx, &tuple, sizeof tuple, 0, 0) */ \ + "r2 = r10;" \ + "r2 += -48;" \ + "r3 = %[sizeof_bpf_sock_tuple];"\ + "r4 = 0;" \ + "r5 = 0;" \ + "call %[" #func "];" + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 1); + __type(key, long long); + __type(value, long long); +} map_hash_8b SEC(".maps"); + +void dummy_prog_42_socket(void); +void dummy_prog_24_socket(void); +void dummy_prog_loop1_socket(void); + +struct { + __uint(type, BPF_MAP_TYPE_PROG_ARRAY); + __uint(max_entries, 4); + __uint(key_size, sizeof(int)); + __array(values, void (void)); +} map_prog1_socket SEC(".maps") = { + .values = { + [0] = (void *)&dummy_prog_42_socket, + [1] = (void *)&dummy_prog_loop1_socket, + [2] = (void *)&dummy_prog_24_socket, + }, +}; + +SEC("socket") +__auxiliary __auxiliary_unpriv +__naked void dummy_prog_42_socket(void) +{ + asm volatile ("r0 = 42; exit;"); +} + +SEC("socket") +__auxiliary __auxiliary_unpriv +__naked void dummy_prog_24_socket(void) +{ + asm volatile ("r0 = 24; exit;"); +} + +SEC("socket") +__auxiliary __auxiliary_unpriv +__naked void dummy_prog_loop1_socket(void) +{ + asm volatile (" \ + r3 = 1; \ + r2 = %[map_prog1_socket] ll; \ + call %[bpf_tail_call]; \ + r0 = 41; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_socket) + : __clobber_all); +} + +SEC("socket") +__description("unpriv: return pointer") +__success __failure_unpriv __msg_unpriv("R0 leaks addr") +__retval(POINTER_VALUE) +__naked void unpriv_return_pointer(void) +{ + asm volatile (" \ + r0 = r10; \ + exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("unpriv: add const to pointer") +__success __success_unpriv __retval(0) +__naked void unpriv_add_const_to_pointer(void) +{ + asm volatile (" \ + r1 += 8; \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("unpriv: add pointer to pointer") +__failure __msg("R1 pointer += pointer") +__failure_unpriv +__naked void unpriv_add_pointer_to_pointer(void) +{ + asm volatile (" \ + r1 += r10; \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("unpriv: neg pointer") +__success __failure_unpriv __msg_unpriv("R1 pointer arithmetic") +__retval(0) +__naked void unpriv_neg_pointer(void) +{ + asm volatile (" \ + r1 = -r1; \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("unpriv: cmp pointer with const") +__success __failure_unpriv __msg_unpriv("R1 pointer comparison") +__retval(0) +__naked void unpriv_cmp_pointer_with_const(void) +{ + asm volatile (" \ + if r1 == 0 goto l0_%=; \ +l0_%=: r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("unpriv: cmp pointer with pointer") +__success __failure_unpriv __msg_unpriv("R10 pointer comparison") +__retval(0) +__naked void unpriv_cmp_pointer_with_pointer(void) +{ + asm volatile (" \ + if r1 == r10 goto l0_%=; \ +l0_%=: r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("tracepoint") +__description("unpriv: check that printk is disallowed") +__success +__naked void check_that_printk_is_disallowed(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r1 = r10; \ + r1 += -8; \ + r2 = 8; \ + r3 = r1; \ + call %[bpf_trace_printk]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_trace_printk) + : __clobber_all); +} + +SEC("socket") +__description("unpriv: pass pointer to helper function") +__success __failure_unpriv __msg_unpriv("R4 leaks addr") +__retval(0) +__naked void pass_pointer_to_helper_function(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + r3 = r2; \ + r4 = r2; \ + call %[bpf_map_update_elem]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_update_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("unpriv: indirectly pass pointer on stack to helper function") +__success __failure_unpriv +__msg_unpriv("invalid indirect read from stack R2 off -8+0 size 8") +__retval(0) +__naked void on_stack_to_helper_function(void) +{ + asm volatile (" \ + *(u64*)(r10 - 8) = r10; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("unpriv: mangle pointer on stack 1") +__success __failure_unpriv __msg_unpriv("attempt to corrupt spilled") +__retval(0) +__naked void mangle_pointer_on_stack_1(void) +{ + asm volatile (" \ + *(u64*)(r10 - 8) = r10; \ + r0 = 0; \ + *(u32*)(r10 - 8) = r0; \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("unpriv: mangle pointer on stack 2") +__success __failure_unpriv __msg_unpriv("attempt to corrupt spilled") +__retval(0) +__naked void mangle_pointer_on_stack_2(void) +{ + asm volatile (" \ + *(u64*)(r10 - 8) = r10; \ + r0 = 0; \ + *(u8*)(r10 - 1) = r0; \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("unpriv: read pointer from stack in small chunks") +__failure __msg("invalid size") +__failure_unpriv +__naked void from_stack_in_small_chunks(void) +{ + asm volatile (" \ + *(u64*)(r10 - 8) = r10; \ + r0 = *(u32*)(r10 - 8); \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("unpriv: write pointer into ctx") +__failure __msg("invalid bpf_context access") +__failure_unpriv __msg_unpriv("R1 leaks addr") +__naked void unpriv_write_pointer_into_ctx(void) +{ + asm volatile (" \ + *(u64*)(r1 + 0) = r1; \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("unpriv: spill/fill of ctx") +__success __success_unpriv __retval(0) +__naked void unpriv_spill_fill_of_ctx(void) +{ + asm volatile (" \ + r6 = r10; \ + r6 += -8; \ + *(u64*)(r6 + 0) = r1; \ + r1 = *(u64*)(r6 + 0); \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("tc") +__description("unpriv: spill/fill of ctx 2") +__success __retval(0) +__naked void spill_fill_of_ctx_2(void) +{ + asm volatile (" \ + r6 = r10; \ + r6 += -8; \ + *(u64*)(r6 + 0) = r1; \ + r1 = *(u64*)(r6 + 0); \ + call %[bpf_get_hash_recalc]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_hash_recalc) + : __clobber_all); +} + +SEC("tc") +__description("unpriv: spill/fill of ctx 3") +__failure __msg("R1 type=fp expected=ctx") +__naked void spill_fill_of_ctx_3(void) +{ + asm volatile (" \ + r6 = r10; \ + r6 += -8; \ + *(u64*)(r6 + 0) = r1; \ + *(u64*)(r6 + 0) = r10; \ + r1 = *(u64*)(r6 + 0); \ + call %[bpf_get_hash_recalc]; \ + exit; \ +" : + : __imm(bpf_get_hash_recalc) + : __clobber_all); +} + +SEC("tc") +__description("unpriv: spill/fill of ctx 4") +__failure __msg("R1 type=scalar expected=ctx") +__naked void spill_fill_of_ctx_4(void) +{ + asm volatile (" \ + r6 = r10; \ + r6 += -8; \ + *(u64*)(r6 + 0) = r1; \ + r0 = 1; \ + lock *(u64 *)(r10 - 8) += r0; \ + r1 = *(u64*)(r6 + 0); \ + call %[bpf_get_hash_recalc]; \ + exit; \ +" : + : __imm(bpf_get_hash_recalc) + : __clobber_all); +} + +SEC("tc") +__description("unpriv: spill/fill of different pointers stx") +__failure __msg("same insn cannot be used with different pointers") +__naked void fill_of_different_pointers_stx(void) +{ + asm volatile (" \ + r3 = 42; \ + r6 = r10; \ + r6 += -8; \ + if r1 == 0 goto l0_%=; \ + r2 = r10; \ + r2 += -16; \ + *(u64*)(r6 + 0) = r2; \ +l0_%=: if r1 != 0 goto l1_%=; \ + *(u64*)(r6 + 0) = r1; \ +l1_%=: r1 = *(u64*)(r6 + 0); \ + *(u32*)(r1 + %[__sk_buff_mark]) = r3; \ + r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark)) + : __clobber_all); +} + +/* Same as above, but use BPF_ST_MEM to save 42 + * instead of BPF_STX_MEM. + */ +SEC("tc") +__description("unpriv: spill/fill of different pointers st") +__failure __msg("same insn cannot be used with different pointers") +__naked void fill_of_different_pointers_st(void) +{ + asm volatile (" \ + r6 = r10; \ + r6 += -8; \ + if r1 == 0 goto l0_%=; \ + r2 = r10; \ + r2 += -16; \ + *(u64*)(r6 + 0) = r2; \ +l0_%=: if r1 != 0 goto l1_%=; \ + *(u64*)(r6 + 0) = r1; \ +l1_%=: r1 = *(u64*)(r6 + 0); \ + .8byte %[st_mem]; \ + r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark)), + __imm_insn(st_mem, + BPF_ST_MEM(BPF_W, BPF_REG_1, offsetof(struct __sk_buff, mark), 42)) + : __clobber_all); +} + +SEC("tc") +__description("unpriv: spill/fill of different pointers stx - ctx and sock") +__failure __msg("type=ctx expected=sock") +__naked void pointers_stx_ctx_and_sock(void) +{ + asm volatile (" \ + r8 = r1; \ + /* struct bpf_sock *sock = bpf_sock_lookup(...); */\ +" BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r2 = r0; \ + /* u64 foo; */ \ + /* void *target = &foo; */ \ + r6 = r10; \ + r6 += -8; \ + r1 = r8; \ + /* if (skb == NULL) *target = sock; */ \ + if r1 == 0 goto l0_%=; \ + *(u64*)(r6 + 0) = r2; \ +l0_%=: /* else *target = skb; */ \ + if r1 != 0 goto l1_%=; \ + *(u64*)(r6 + 0) = r1; \ +l1_%=: /* struct __sk_buff *skb = *target; */ \ + r1 = *(u64*)(r6 + 0); \ + /* skb->mark = 42; */ \ + r3 = 42; \ + *(u32*)(r1 + %[__sk_buff_mark]) = r3; \ + /* if (sk) bpf_sk_release(sk) */ \ + if r1 == 0 goto l2_%=; \ + call %[bpf_sk_release]; \ +l2_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark)), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("unpriv: spill/fill of different pointers stx - leak sock") +__failure +//.errstr = "same insn cannot be used with different pointers", +__msg("Unreleased reference") +__naked void different_pointers_stx_leak_sock(void) +{ + asm volatile (" \ + r8 = r1; \ + /* struct bpf_sock *sock = bpf_sock_lookup(...); */\ +" BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r2 = r0; \ + /* u64 foo; */ \ + /* void *target = &foo; */ \ + r6 = r10; \ + r6 += -8; \ + r1 = r8; \ + /* if (skb == NULL) *target = sock; */ \ + if r1 == 0 goto l0_%=; \ + *(u64*)(r6 + 0) = r2; \ +l0_%=: /* else *target = skb; */ \ + if r1 != 0 goto l1_%=; \ + *(u64*)(r6 + 0) = r1; \ +l1_%=: /* struct __sk_buff *skb = *target; */ \ + r1 = *(u64*)(r6 + 0); \ + /* skb->mark = 42; */ \ + r3 = 42; \ + *(u32*)(r1 + %[__sk_buff_mark]) = r3; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark)), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("unpriv: spill/fill of different pointers stx - sock and ctx (read)") +__failure __msg("same insn cannot be used with different pointers") +__naked void stx_sock_and_ctx_read(void) +{ + asm volatile (" \ + r8 = r1; \ + /* struct bpf_sock *sock = bpf_sock_lookup(...); */\ +" BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r2 = r0; \ + /* u64 foo; */ \ + /* void *target = &foo; */ \ + r6 = r10; \ + r6 += -8; \ + r1 = r8; \ + /* if (skb) *target = skb */ \ + if r1 == 0 goto l0_%=; \ + *(u64*)(r6 + 0) = r1; \ +l0_%=: /* else *target = sock */ \ + if r1 != 0 goto l1_%=; \ + *(u64*)(r6 + 0) = r2; \ +l1_%=: /* struct bpf_sock *sk = *target; */ \ + r1 = *(u64*)(r6 + 0); \ + /* if (sk) u32 foo = sk->mark; bpf_sk_release(sk); */\ + if r1 == 0 goto l2_%=; \ + r3 = *(u32*)(r1 + %[bpf_sock_mark]); \ + call %[bpf_sk_release]; \ +l2_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(bpf_sock_mark, offsetof(struct bpf_sock, mark)), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("tc") +__description("unpriv: spill/fill of different pointers stx - sock and ctx (write)") +__failure +//.errstr = "same insn cannot be used with different pointers", +__msg("cannot write into sock") +__naked void stx_sock_and_ctx_write(void) +{ + asm volatile (" \ + r8 = r1; \ + /* struct bpf_sock *sock = bpf_sock_lookup(...); */\ +" BPF_SK_LOOKUP(bpf_sk_lookup_tcp) +" r2 = r0; \ + /* u64 foo; */ \ + /* void *target = &foo; */ \ + r6 = r10; \ + r6 += -8; \ + r1 = r8; \ + /* if (skb) *target = skb */ \ + if r1 == 0 goto l0_%=; \ + *(u64*)(r6 + 0) = r1; \ +l0_%=: /* else *target = sock */ \ + if r1 != 0 goto l1_%=; \ + *(u64*)(r6 + 0) = r2; \ +l1_%=: /* struct bpf_sock *sk = *target; */ \ + r1 = *(u64*)(r6 + 0); \ + /* if (sk) sk->mark = 42; bpf_sk_release(sk); */\ + if r1 == 0 goto l2_%=; \ + r3 = 42; \ + *(u32*)(r1 + %[bpf_sock_mark]) = r3; \ + call %[bpf_sk_release]; \ +l2_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_sk_lookup_tcp), + __imm(bpf_sk_release), + __imm_const(bpf_sock_mark, offsetof(struct bpf_sock, mark)), + __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple)) + : __clobber_all); +} + +SEC("socket") +__description("unpriv: write pointer into map elem value") +__success __failure_unpriv __msg_unpriv("R0 leaks addr") +__retval(0) +__naked void pointer_into_map_elem_value(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + *(u64*)(r0 + 0) = r0; \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("alu32: mov u32 const") +__success __failure_unpriv __msg_unpriv("R7 invalid mem access 'scalar'") +__retval(0) +__naked void alu32_mov_u32_const(void) +{ + asm volatile (" \ + w7 = 0; \ + w7 &= 1; \ + w0 = w7; \ + if r0 == 0 goto l0_%=; \ + r0 = *(u64*)(r7 + 0); \ +l0_%=: exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("unpriv: partial copy of pointer") +__success __failure_unpriv __msg_unpriv("R10 partial copy") +__retval(0) +__naked void unpriv_partial_copy_of_pointer(void) +{ + asm volatile (" \ + w1 = w10; \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("unpriv: pass pointer to tail_call") +__success __failure_unpriv __msg_unpriv("R3 leaks addr into helper") +__retval(0) +__naked void pass_pointer_to_tail_call(void) +{ + asm volatile (" \ + r3 = r1; \ + r2 = %[map_prog1_socket] ll; \ + call %[bpf_tail_call]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_tail_call), + __imm_addr(map_prog1_socket) + : __clobber_all); +} + +SEC("socket") +__description("unpriv: cmp map pointer with zero") +__success __failure_unpriv __msg_unpriv("R1 pointer comparison") +__retval(0) +__naked void cmp_map_pointer_with_zero(void) +{ + asm volatile (" \ + r1 = 0; \ + r1 = %[map_hash_8b] ll; \ + if r1 == 0 goto l0_%=; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_addr(map_hash_8b) + : __clobber_all); +} + +SEC("socket") +__description("unpriv: write into frame pointer") +__failure __msg("frame pointer is read only") +__failure_unpriv +__naked void unpriv_write_into_frame_pointer(void) +{ + asm volatile (" \ + r10 = r1; \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("unpriv: spill/fill frame pointer") +__failure __msg("frame pointer is read only") +__failure_unpriv +__naked void unpriv_spill_fill_frame_pointer(void) +{ + asm volatile (" \ + r6 = r10; \ + r6 += -8; \ + *(u64*)(r6 + 0) = r10; \ + r10 = *(u64*)(r6 + 0); \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("unpriv: cmp of frame pointer") +__success __failure_unpriv __msg_unpriv("R10 pointer comparison") +__retval(0) +__naked void unpriv_cmp_of_frame_pointer(void) +{ + asm volatile (" \ + if r10 == 0 goto l0_%=; \ +l0_%=: r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("unpriv: adding of fp, reg") +__success __failure_unpriv +__msg_unpriv("R1 stack pointer arithmetic goes out of range") +__retval(0) +__naked void unpriv_adding_of_fp_reg(void) +{ + asm volatile (" \ + r0 = 0; \ + r1 = 0; \ + r1 += r10; \ + *(u64*)(r1 - 8) = r0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("unpriv: adding of fp, imm") +__success __failure_unpriv +__msg_unpriv("R1 stack pointer arithmetic goes out of range") +__retval(0) +__naked void unpriv_adding_of_fp_imm(void) +{ + asm volatile (" \ + r0 = 0; \ + r1 = r10; \ + r1 += 0; \ + *(u64*)(r1 - 8) = r0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("unpriv: cmp of stack pointer") +__success __failure_unpriv __msg_unpriv("R2 pointer comparison") +__retval(0) +__naked void unpriv_cmp_of_stack_pointer(void) +{ + asm volatile (" \ + r2 = r10; \ + r2 += -8; \ + if r2 == 0 goto l0_%=; \ +l0_%=: r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_unpriv_perf.c b/tools/testing/selftests/bpf/progs/verifier_unpriv_perf.c new file mode 100644 index 000000000000..4d77407a0a79 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_unpriv_perf.c @@ -0,0 +1,34 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/unpriv.c */ + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +SEC("perf_event") +__description("unpriv: spill/fill of different pointers ldx") +__failure __msg("same insn cannot be used with different pointers") +__naked void fill_of_different_pointers_ldx(void) +{ + asm volatile (" \ + r6 = r10; \ + r6 += -8; \ + if r1 == 0 goto l0_%=; \ + r2 = r10; \ + r2 += %[__imm_0]; \ + *(u64*)(r6 + 0) = r2; \ +l0_%=: if r1 != 0 goto l1_%=; \ + *(u64*)(r6 + 0) = r1; \ +l1_%=: r1 = *(u64*)(r6 + 0); \ + r1 = *(u64*)(r1 + %[sample_period]); \ + r0 = 0; \ + exit; \ +" : + : __imm_const(__imm_0, + -(__s32) offsetof(struct bpf_perf_event_data, sample_period) - 8), + __imm_const(sample_period, + offsetof(struct bpf_perf_event_data, sample_period)) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_value_illegal_alu.c b/tools/testing/selftests/bpf/progs/verifier_value_illegal_alu.c new file mode 100644 index 000000000000..71814a753216 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_value_illegal_alu.c @@ -0,0 +1,149 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/value_illegal_alu.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +#define MAX_ENTRIES 11 + +struct test_val { + unsigned int index; + int foo[MAX_ENTRIES]; +}; + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 1); + __type(key, long long); + __type(value, struct test_val); +} map_hash_48b SEC(".maps"); + +SEC("socket") +__description("map element value illegal alu op, 1") +__failure __msg("R0 bitwise operator &= on pointer") +__failure_unpriv +__naked void value_illegal_alu_op_1(void) +{ + asm volatile (" \ + r2 = r10; \ + r2 += -8; \ + r1 = 0; \ + *(u64*)(r2 + 0) = r1; \ + r1 = %[map_hash_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r0 &= 8; \ + r1 = 22; \ + *(u64*)(r0 + 0) = r1; \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_48b) + : __clobber_all); +} + +SEC("socket") +__description("map element value illegal alu op, 2") +__failure __msg("R0 32-bit pointer arithmetic prohibited") +__failure_unpriv +__naked void value_illegal_alu_op_2(void) +{ + asm volatile (" \ + r2 = r10; \ + r2 += -8; \ + r1 = 0; \ + *(u64*)(r2 + 0) = r1; \ + r1 = %[map_hash_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + w0 += 0; \ + r1 = 22; \ + *(u64*)(r0 + 0) = r1; \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_48b) + : __clobber_all); +} + +SEC("socket") +__description("map element value illegal alu op, 3") +__failure __msg("R0 pointer arithmetic with /= operator") +__failure_unpriv +__naked void value_illegal_alu_op_3(void) +{ + asm volatile (" \ + r2 = r10; \ + r2 += -8; \ + r1 = 0; \ + *(u64*)(r2 + 0) = r1; \ + r1 = %[map_hash_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r0 /= 42; \ + r1 = 22; \ + *(u64*)(r0 + 0) = r1; \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_48b) + : __clobber_all); +} + +SEC("socket") +__description("map element value illegal alu op, 4") +__failure __msg("invalid mem access 'scalar'") +__failure_unpriv __msg_unpriv("R0 pointer arithmetic prohibited") +__flag(BPF_F_ANY_ALIGNMENT) +__naked void value_illegal_alu_op_4(void) +{ + asm volatile (" \ + r2 = r10; \ + r2 += -8; \ + r1 = 0; \ + *(u64*)(r2 + 0) = r1; \ + r1 = %[map_hash_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r0 = be64 r0; \ + r1 = 22; \ + *(u64*)(r0 + 0) = r1; \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_48b) + : __clobber_all); +} + +SEC("socket") +__description("map element value illegal alu op, 5") +__failure __msg("R0 invalid mem access 'scalar'") +__msg_unpriv("leaking pointer from stack off -8") +__flag(BPF_F_ANY_ALIGNMENT) +__naked void value_illegal_alu_op_5(void) +{ + asm volatile (" \ + r2 = r10; \ + r2 += -8; \ + r1 = 0; \ + *(u64*)(r2 + 0) = r1; \ + r1 = %[map_hash_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r3 = 4096; \ + r2 = r10; \ + r2 += -8; \ + *(u64*)(r2 + 0) = r0; \ + lock *(u64 *)(r2 + 0) += r3; \ + r0 = *(u64*)(r2 + 0); \ + r1 = 22; \ + *(u64*)(r0 + 0) = r1; \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_48b) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c b/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c new file mode 100644 index 000000000000..5ba6e53571c8 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c @@ -0,0 +1,1423 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Converted from tools/testing/selftests/bpf/verifier/value_ptr_arith.c */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include <errno.h> +#include "bpf_misc.h" + +#define MAX_ENTRIES 11 + +struct test_val { + unsigned int index; + int foo[MAX_ENTRIES]; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, int); + __type(value, struct test_val); +} map_array_48b SEC(".maps"); + +struct other_val { + long long foo; + long long bar; +}; + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 1); + __type(key, long long); + __type(value, struct other_val); +} map_hash_16b SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 1); + __type(key, long long); + __type(value, struct test_val); +} map_hash_48b SEC(".maps"); + +SEC("socket") +__description("map access: known scalar += value_ptr unknown vs const") +__success __failure_unpriv +__msg_unpriv("R1 tried to add from different maps, paths or scalars") +__retval(1) +__naked void value_ptr_unknown_vs_const(void) +{ + asm volatile (" \ + r0 = *(u32*)(r1 + %[__sk_buff_len]); \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + if r0 == 1 goto l0_%=; \ + r1 = %[map_hash_16b] ll; \ + if r0 != 1 goto l1_%=; \ +l0_%=: r1 = %[map_array_48b] ll; \ +l1_%=: call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l2_%=; \ + r4 = *(u8*)(r0 + 0); \ + if r4 == 1 goto l3_%=; \ + r1 = 6; \ + r1 = -r1; \ + r1 &= 0x7; \ + goto l4_%=; \ +l3_%=: r1 = 3; \ +l4_%=: r1 += r0; \ + r0 = *(u8*)(r1 + 0); \ +l2_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b), + __imm_addr(map_hash_16b), + __imm_const(__sk_buff_len, offsetof(struct __sk_buff, len)) + : __clobber_all); +} + +SEC("socket") +__description("map access: known scalar += value_ptr const vs unknown") +__success __failure_unpriv +__msg_unpriv("R1 tried to add from different maps, paths or scalars") +__retval(1) +__naked void value_ptr_const_vs_unknown(void) +{ + asm volatile (" \ + r0 = *(u32*)(r1 + %[__sk_buff_len]); \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + if r0 == 1 goto l0_%=; \ + r1 = %[map_hash_16b] ll; \ + if r0 != 1 goto l1_%=; \ +l0_%=: r1 = %[map_array_48b] ll; \ +l1_%=: call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l2_%=; \ + r4 = *(u8*)(r0 + 0); \ + if r4 == 1 goto l3_%=; \ + r1 = 3; \ + goto l4_%=; \ +l3_%=: r1 = 6; \ + r1 = -r1; \ + r1 &= 0x7; \ +l4_%=: r1 += r0; \ + r0 = *(u8*)(r1 + 0); \ +l2_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b), + __imm_addr(map_hash_16b), + __imm_const(__sk_buff_len, offsetof(struct __sk_buff, len)) + : __clobber_all); +} + +SEC("socket") +__description("map access: known scalar += value_ptr const vs const (ne)") +__success __failure_unpriv +__msg_unpriv("R1 tried to add from different maps, paths or scalars") +__retval(1) +__naked void ptr_const_vs_const_ne(void) +{ + asm volatile (" \ + r0 = *(u32*)(r1 + %[__sk_buff_len]); \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + if r0 == 1 goto l0_%=; \ + r1 = %[map_hash_16b] ll; \ + if r0 != 1 goto l1_%=; \ +l0_%=: r1 = %[map_array_48b] ll; \ +l1_%=: call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l2_%=; \ + r4 = *(u8*)(r0 + 0); \ + if r4 == 1 goto l3_%=; \ + r1 = 3; \ + goto l4_%=; \ +l3_%=: r1 = 5; \ +l4_%=: r1 += r0; \ + r0 = *(u8*)(r1 + 0); \ +l2_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b), + __imm_addr(map_hash_16b), + __imm_const(__sk_buff_len, offsetof(struct __sk_buff, len)) + : __clobber_all); +} + +SEC("socket") +__description("map access: known scalar += value_ptr const vs const (eq)") +__success __success_unpriv __retval(1) +__naked void ptr_const_vs_const_eq(void) +{ + asm volatile (" \ + r0 = *(u32*)(r1 + %[__sk_buff_len]); \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + if r0 == 1 goto l0_%=; \ + r1 = %[map_hash_16b] ll; \ + if r0 != 1 goto l1_%=; \ +l0_%=: r1 = %[map_array_48b] ll; \ +l1_%=: call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l2_%=; \ + r4 = *(u8*)(r0 + 0); \ + if r4 == 1 goto l3_%=; \ + r1 = 5; \ + goto l4_%=; \ +l3_%=: r1 = 5; \ +l4_%=: r1 += r0; \ + r0 = *(u8*)(r1 + 0); \ +l2_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b), + __imm_addr(map_hash_16b), + __imm_const(__sk_buff_len, offsetof(struct __sk_buff, len)) + : __clobber_all); +} + +SEC("socket") +__description("map access: known scalar += value_ptr unknown vs unknown (eq)") +__success __success_unpriv __retval(1) +__naked void ptr_unknown_vs_unknown_eq(void) +{ + asm volatile (" \ + r0 = *(u32*)(r1 + %[__sk_buff_len]); \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + if r0 == 1 goto l0_%=; \ + r1 = %[map_hash_16b] ll; \ + if r0 != 1 goto l1_%=; \ +l0_%=: r1 = %[map_array_48b] ll; \ +l1_%=: call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l2_%=; \ + r4 = *(u8*)(r0 + 0); \ + if r4 == 1 goto l3_%=; \ + r1 = 6; \ + r1 = -r1; \ + r1 &= 0x7; \ + goto l4_%=; \ +l3_%=: r1 = 6; \ + r1 = -r1; \ + r1 &= 0x7; \ +l4_%=: r1 += r0; \ + r0 = *(u8*)(r1 + 0); \ +l2_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b), + __imm_addr(map_hash_16b), + __imm_const(__sk_buff_len, offsetof(struct __sk_buff, len)) + : __clobber_all); +} + +SEC("socket") +__description("map access: known scalar += value_ptr unknown vs unknown (lt)") +__success __failure_unpriv +__msg_unpriv("R1 tried to add from different maps, paths or scalars") +__retval(1) +__naked void ptr_unknown_vs_unknown_lt(void) +{ + asm volatile (" \ + r0 = *(u32*)(r1 + %[__sk_buff_len]); \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + if r0 == 1 goto l0_%=; \ + r1 = %[map_hash_16b] ll; \ + if r0 != 1 goto l1_%=; \ +l0_%=: r1 = %[map_array_48b] ll; \ +l1_%=: call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l2_%=; \ + r4 = *(u8*)(r0 + 0); \ + if r4 == 1 goto l3_%=; \ + r1 = 6; \ + r1 = -r1; \ + r1 &= 0x3; \ + goto l4_%=; \ +l3_%=: r1 = 6; \ + r1 = -r1; \ + r1 &= 0x7; \ +l4_%=: r1 += r0; \ + r0 = *(u8*)(r1 + 0); \ +l2_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b), + __imm_addr(map_hash_16b), + __imm_const(__sk_buff_len, offsetof(struct __sk_buff, len)) + : __clobber_all); +} + +SEC("socket") +__description("map access: known scalar += value_ptr unknown vs unknown (gt)") +__success __failure_unpriv +__msg_unpriv("R1 tried to add from different maps, paths or scalars") +__retval(1) +__naked void ptr_unknown_vs_unknown_gt(void) +{ + asm volatile (" \ + r0 = *(u32*)(r1 + %[__sk_buff_len]); \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + if r0 == 1 goto l0_%=; \ + r1 = %[map_hash_16b] ll; \ + if r0 != 1 goto l1_%=; \ +l0_%=: r1 = %[map_array_48b] ll; \ +l1_%=: call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l2_%=; \ + r4 = *(u8*)(r0 + 0); \ + if r4 == 1 goto l3_%=; \ + r1 = 6; \ + r1 = -r1; \ + r1 &= 0x7; \ + goto l4_%=; \ +l3_%=: r1 = 6; \ + r1 = -r1; \ + r1 &= 0x3; \ +l4_%=: r1 += r0; \ + r0 = *(u8*)(r1 + 0); \ +l2_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b), + __imm_addr(map_hash_16b), + __imm_const(__sk_buff_len, offsetof(struct __sk_buff, len)) + : __clobber_all); +} + +SEC("socket") +__description("map access: known scalar += value_ptr from different maps") +__success __success_unpriv __retval(1) +__naked void value_ptr_from_different_maps(void) +{ + asm volatile (" \ + r0 = *(u32*)(r1 + %[__sk_buff_len]); \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + if r0 == 1 goto l0_%=; \ + r1 = %[map_hash_16b] ll; \ + if r0 != 1 goto l1_%=; \ +l0_%=: r1 = %[map_array_48b] ll; \ +l1_%=: call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l2_%=; \ + r1 = 4; \ + r1 += r0; \ + r0 = *(u8*)(r1 + 0); \ +l2_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b), + __imm_addr(map_hash_16b), + __imm_const(__sk_buff_len, offsetof(struct __sk_buff, len)) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr -= known scalar from different maps") +__success __failure_unpriv +__msg_unpriv("R0 min value is outside of the allowed memory range") +__retval(1) +__naked void known_scalar_from_different_maps(void) +{ + asm volatile (" \ + r0 = *(u32*)(r1 + %[__sk_buff_len]); \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + if r0 == 1 goto l0_%=; \ + r1 = %[map_hash_16b] ll; \ + if r0 != 1 goto l1_%=; \ +l0_%=: r1 = %[map_array_48b] ll; \ +l1_%=: call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l2_%=; \ + r1 = 4; \ + r0 -= r1; \ + r0 += r1; \ + r0 = *(u8*)(r0 + 0); \ +l2_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b), + __imm_addr(map_hash_16b), + __imm_const(__sk_buff_len, offsetof(struct __sk_buff, len)) + : __clobber_all); +} + +SEC("socket") +__description("map access: known scalar += value_ptr from different maps, but same value properties") +__success __success_unpriv __retval(1) +__naked void maps_but_same_value_properties(void) +{ + asm volatile (" \ + r0 = *(u32*)(r1 + %[__sk_buff_len]); \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + if r0 == 1 goto l0_%=; \ + r1 = %[map_hash_48b] ll; \ + if r0 != 1 goto l1_%=; \ +l0_%=: r1 = %[map_array_48b] ll; \ +l1_%=: call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l2_%=; \ + r1 = 4; \ + r1 += r0; \ + r0 = *(u8*)(r1 + 0); \ +l2_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b), + __imm_addr(map_hash_48b), + __imm_const(__sk_buff_len, offsetof(struct __sk_buff, len)) + : __clobber_all); +} + +SEC("socket") +__description("map access: mixing value pointer and scalar, 1") +__success __failure_unpriv __msg_unpriv("R2 pointer comparison prohibited") +__retval(0) +__naked void value_pointer_and_scalar_1(void) +{ + asm volatile (" \ + /* load map value pointer into r0 and r2 */ \ + r0 = 1; \ + r1 = %[map_array_48b] ll; \ + r2 = r10; \ + r2 += -16; \ + r6 = 0; \ + *(u64*)(r10 - 16) = r6; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: /* load some number from the map into r1 */ \ + r1 = *(u8*)(r0 + 0); \ + /* depending on r1, branch: */ \ + if r1 != 0 goto l1_%=; \ + /* branch A */ \ + r2 = r0; \ + r3 = 0; \ + goto l2_%=; \ +l1_%=: /* branch B */ \ + r2 = 0; \ + r3 = 0x100000; \ +l2_%=: /* common instruction */ \ + r2 += r3; \ + /* depending on r1, branch: */ \ + if r1 != 0 goto l3_%=; \ + /* branch A */ \ + goto l4_%=; \ +l3_%=: /* branch B */ \ + r0 = 0x13371337; \ + /* verifier follows fall-through */ \ + if r2 != 0x100000 goto l4_%=; \ + r0 = 0; \ + exit; \ +l4_%=: /* fake-dead code; targeted from branch A to \ + * prevent dead code sanitization \ + */ \ + r0 = *(u8*)(r0 + 0); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: mixing value pointer and scalar, 2") +__success __failure_unpriv __msg_unpriv("R0 invalid mem access 'scalar'") +__retval(0) +__naked void value_pointer_and_scalar_2(void) +{ + asm volatile (" \ + /* load map value pointer into r0 and r2 */ \ + r0 = 1; \ + r1 = %[map_array_48b] ll; \ + r2 = r10; \ + r2 += -16; \ + r6 = 0; \ + *(u64*)(r10 - 16) = r6; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: /* load some number from the map into r1 */ \ + r1 = *(u8*)(r0 + 0); \ + /* depending on r1, branch: */ \ + if r1 == 0 goto l1_%=; \ + /* branch A */ \ + r2 = 0; \ + r3 = 0x100000; \ + goto l2_%=; \ +l1_%=: /* branch B */ \ + r2 = r0; \ + r3 = 0; \ +l2_%=: /* common instruction */ \ + r2 += r3; \ + /* depending on r1, branch: */ \ + if r1 != 0 goto l3_%=; \ + /* branch A */ \ + goto l4_%=; \ +l3_%=: /* branch B */ \ + r0 = 0x13371337; \ + /* verifier follows fall-through */ \ + if r2 != 0x100000 goto l4_%=; \ + r0 = 0; \ + exit; \ +l4_%=: /* fake-dead code; targeted from branch A to \ + * prevent dead code sanitization, rejected \ + * via branch B however \ + */ \ + r0 = *(u8*)(r0 + 0); \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("sanitation: alu with different scalars 1") +__success __success_unpriv __retval(0x100000) +__naked void alu_with_different_scalars_1(void) +{ + asm volatile (" \ + r0 = 1; \ + r1 = %[map_array_48b] ll; \ + r2 = r10; \ + r2 += -16; \ + r6 = 0; \ + *(u64*)(r10 - 16) = r6; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r1 = *(u32*)(r0 + 0); \ + if r1 == 0 goto l1_%=; \ + r2 = 0; \ + r3 = 0x100000; \ + goto l2_%=; \ +l1_%=: r2 = 42; \ + r3 = 0x100001; \ +l2_%=: r2 += r3; \ + r0 = r2; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("sanitation: alu with different scalars 2") +__success __success_unpriv __retval(0) +__naked void alu_with_different_scalars_2(void) +{ + asm volatile (" \ + r0 = 1; \ + r1 = %[map_array_48b] ll; \ + r6 = r1; \ + r2 = r10; \ + r2 += -16; \ + r7 = 0; \ + *(u64*)(r10 - 16) = r7; \ + call %[bpf_map_delete_elem]; \ + r7 = r0; \ + r1 = r6; \ + r2 = r10; \ + r2 += -16; \ + call %[bpf_map_delete_elem]; \ + r6 = r0; \ + r8 = r6; \ + r8 += r7; \ + r0 = r8; \ + r0 += %[einval]; \ + r0 += %[einval]; \ + exit; \ +" : + : __imm(bpf_map_delete_elem), + __imm_addr(map_array_48b), + __imm_const(einval, EINVAL) + : __clobber_all); +} + +SEC("socket") +__description("sanitation: alu with different scalars 3") +__success __success_unpriv __retval(0) +__naked void alu_with_different_scalars_3(void) +{ + asm volatile (" \ + r0 = %[einval]; \ + r0 *= -1; \ + r7 = r0; \ + r0 = %[einval]; \ + r0 *= -1; \ + r6 = r0; \ + r8 = r6; \ + r8 += r7; \ + r0 = r8; \ + r0 += %[einval]; \ + r0 += %[einval]; \ + exit; \ +" : + : __imm_const(einval, EINVAL) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr += known scalar, upper oob arith, test 1") +__success __failure_unpriv +__msg_unpriv("R0 pointer arithmetic of map value goes out of range") +__retval(1) +__naked void upper_oob_arith_test_1(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = 48; \ + r0 += r1; \ + r0 -= r1; \ + r0 = *(u8*)(r0 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr += known scalar, upper oob arith, test 2") +__success __failure_unpriv +__msg_unpriv("R0 pointer arithmetic of map value goes out of range") +__retval(1) +__naked void upper_oob_arith_test_2(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = 49; \ + r0 += r1; \ + r0 -= r1; \ + r0 = *(u8*)(r0 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr += known scalar, upper oob arith, test 3") +__success __success_unpriv __retval(1) +__naked void upper_oob_arith_test_3(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = 47; \ + r0 += r1; \ + r0 -= r1; \ + r0 = *(u8*)(r0 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr -= known scalar, lower oob arith, test 1") +__failure __msg("R0 min value is outside of the allowed memory range") +__failure_unpriv +__msg_unpriv("R0 pointer arithmetic of map value goes out of range") +__naked void lower_oob_arith_test_1(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = 47; \ + r0 += r1; \ + r1 = 48; \ + r0 -= r1; \ + r0 = *(u8*)(r0 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr -= known scalar, lower oob arith, test 2") +__success __failure_unpriv +__msg_unpriv("R0 pointer arithmetic of map value goes out of range") +__retval(1) +__naked void lower_oob_arith_test_2(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = 47; \ + r0 += r1; \ + r1 = 48; \ + r0 -= r1; \ + r1 = 1; \ + r0 += r1; \ + r0 = *(u8*)(r0 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr -= known scalar, lower oob arith, test 3") +__success __success_unpriv __retval(1) +__naked void lower_oob_arith_test_3(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = 47; \ + r0 += r1; \ + r1 = 47; \ + r0 -= r1; \ + r0 = *(u8*)(r0 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: known scalar += value_ptr") +__success __success_unpriv __retval(1) +__naked void access_known_scalar_value_ptr_1(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = 4; \ + r1 += r0; \ + r0 = *(u8*)(r1 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr += known scalar, 1") +__success __success_unpriv __retval(1) +__naked void value_ptr_known_scalar_1(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = 4; \ + r0 += r1; \ + r1 = *(u8*)(r0 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr += known scalar, 2") +__failure __msg("invalid access to map value") +__failure_unpriv +__naked void value_ptr_known_scalar_2_1(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = 49; \ + r0 += r1; \ + r1 = *(u8*)(r0 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr += known scalar, 3") +__failure __msg("invalid access to map value") +__failure_unpriv +__naked void value_ptr_known_scalar_3(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = -1; \ + r0 += r1; \ + r1 = *(u8*)(r0 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr += known scalar, 4") +__success __success_unpriv __retval(1) +__naked void value_ptr_known_scalar_4(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = 5; \ + r0 += r1; \ + r1 = -2; \ + r0 += r1; \ + r1 = -1; \ + r0 += r1; \ + r1 = *(u8*)(r0 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr += known scalar, 5") +__success __success_unpriv __retval(0xabcdef12) +__naked void value_ptr_known_scalar_5(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = %[__imm_0]; \ + r1 += r0; \ + r0 = *(u32*)(r1 + 0); \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b), + __imm_const(__imm_0, (6 + 1) * sizeof(int)) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr += known scalar, 6") +__success __success_unpriv __retval(0xabcdef12) +__naked void value_ptr_known_scalar_6(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = %[__imm_0]; \ + r0 += r1; \ + r1 = %[__imm_1]; \ + r0 += r1; \ + r0 = *(u32*)(r0 + 0); \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b), + __imm_const(__imm_0, (3 + 1) * sizeof(int)), + __imm_const(__imm_1, 3 * sizeof(int)) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr += N, value_ptr -= N known scalar") +__success __success_unpriv __retval(0x12345678) +__naked void value_ptr_n_known_scalar(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + w1 = 0x12345678; \ + *(u32*)(r0 + 0) = r1; \ + r0 += 2; \ + r1 = 2; \ + r0 -= r1; \ + r0 = *(u32*)(r0 + 0); \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: unknown scalar += value_ptr, 1") +__success __success_unpriv __retval(1) +__naked void unknown_scalar_value_ptr_1(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = *(u8*)(r0 + 0); \ + r1 &= 0xf; \ + r1 += r0; \ + r0 = *(u8*)(r1 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: unknown scalar += value_ptr, 2") +__success __success_unpriv __retval(0xabcdef12) __flag(BPF_F_ANY_ALIGNMENT) +__naked void unknown_scalar_value_ptr_2(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = *(u32*)(r0 + 0); \ + r1 &= 31; \ + r1 += r0; \ + r0 = *(u32*)(r1 + 0); \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: unknown scalar += value_ptr, 3") +__success __failure_unpriv +__msg_unpriv("R0 pointer arithmetic of map value goes out of range") +__retval(0xabcdef12) __flag(BPF_F_ANY_ALIGNMENT) +__naked void unknown_scalar_value_ptr_3(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = -1; \ + r0 += r1; \ + r1 = 1; \ + r0 += r1; \ + r1 = *(u32*)(r0 + 0); \ + r1 &= 31; \ + r1 += r0; \ + r0 = *(u32*)(r1 + 0); \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: unknown scalar += value_ptr, 4") +__failure __msg("R1 max value is outside of the allowed memory range") +__msg_unpriv("R1 pointer arithmetic of map value goes out of range") +__flag(BPF_F_ANY_ALIGNMENT) +__naked void unknown_scalar_value_ptr_4(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = 19; \ + r0 += r1; \ + r1 = *(u32*)(r0 + 0); \ + r1 &= 31; \ + r1 += r0; \ + r0 = *(u32*)(r1 + 0); \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr += unknown scalar, 1") +__success __success_unpriv __retval(1) +__naked void value_ptr_unknown_scalar_1(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = *(u8*)(r0 + 0); \ + r1 &= 0xf; \ + r0 += r1; \ + r1 = *(u8*)(r0 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr += unknown scalar, 2") +__success __success_unpriv __retval(0xabcdef12) __flag(BPF_F_ANY_ALIGNMENT) +__naked void value_ptr_unknown_scalar_2_1(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = *(u32*)(r0 + 0); \ + r1 &= 31; \ + r0 += r1; \ + r0 = *(u32*)(r0 + 0); \ +l0_%=: exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr += unknown scalar, 3") +__success __success_unpriv __retval(1) +__naked void value_ptr_unknown_scalar_3(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = *(u64*)(r0 + 0); \ + r2 = *(u64*)(r0 + 8); \ + r3 = *(u64*)(r0 + 16); \ + r1 &= 0xf; \ + r3 &= 1; \ + r3 |= 1; \ + if r2 > r3 goto l0_%=; \ + r0 += r3; \ + r0 = *(u8*)(r0 + 0); \ + r0 = 1; \ +l1_%=: exit; \ +l0_%=: r0 = 2; \ + goto l1_%=; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr += value_ptr") +__failure __msg("R0 pointer += pointer prohibited") +__failure_unpriv +__naked void access_value_ptr_value_ptr_1(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r0 += r0; \ + r1 = *(u8*)(r0 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: known scalar -= value_ptr") +__failure __msg("R1 tried to subtract pointer from scalar") +__failure_unpriv +__naked void access_known_scalar_value_ptr_2(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = 4; \ + r1 -= r0; \ + r0 = *(u8*)(r1 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr -= known scalar") +__failure __msg("R0 min value is outside of the allowed memory range") +__failure_unpriv +__naked void access_value_ptr_known_scalar(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = 4; \ + r0 -= r1; \ + r1 = *(u8*)(r0 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr -= known scalar, 2") +__success __success_unpriv __retval(1) +__naked void value_ptr_known_scalar_2_2(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = 6; \ + r2 = 4; \ + r0 += r1; \ + r0 -= r2; \ + r1 = *(u8*)(r0 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: unknown scalar -= value_ptr") +__failure __msg("R1 tried to subtract pointer from scalar") +__failure_unpriv +__naked void access_unknown_scalar_value_ptr(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = *(u8*)(r0 + 0); \ + r1 &= 0xf; \ + r1 -= r0; \ + r0 = *(u8*)(r1 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr -= unknown scalar") +__failure __msg("R0 min value is negative") +__failure_unpriv +__naked void access_value_ptr_unknown_scalar(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = *(u8*)(r0 + 0); \ + r1 &= 0xf; \ + r0 -= r1; \ + r1 = *(u8*)(r0 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr -= unknown scalar, 2") +__success __failure_unpriv +__msg_unpriv("R0 pointer arithmetic of map value goes out of range") +__retval(1) +__naked void value_ptr_unknown_scalar_2_2(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = *(u8*)(r0 + 0); \ + r1 &= 0xf; \ + r1 |= 0x7; \ + r0 += r1; \ + r1 = *(u8*)(r0 + 0); \ + r1 &= 0x7; \ + r0 -= r1; \ + r1 = *(u8*)(r0 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: value_ptr -= value_ptr") +__failure __msg("R0 invalid mem access 'scalar'") +__msg_unpriv("R0 pointer -= pointer prohibited") +__naked void access_value_ptr_value_ptr_2(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r0 -= r0; \ + r1 = *(u8*)(r0 + 0); \ +l0_%=: r0 = 1; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("socket") +__description("map access: trying to leak tainted dst reg") +__failure __msg("math between map_value pointer and 4294967295 is not allowed") +__failure_unpriv +__naked void to_leak_tainted_dst_reg(void) +{ + asm volatile (" \ + r0 = 0; \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_array_48b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 != 0 goto l0_%=; \ + exit; \ +l0_%=: r2 = r0; \ + w1 = 0xFFFFFFFF; \ + w1 = w1; \ + r2 -= r1; \ + *(u64*)(r0 + 0) = r2; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_array_48b) + : __clobber_all); +} + +SEC("tc") +__description("32bit pkt_ptr -= scalar") +__success __retval(0) __flag(BPF_F_ANY_ALIGNMENT) +__naked void _32bit_pkt_ptr_scalar(void) +{ + asm volatile (" \ + r8 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r7 = *(u32*)(r1 + %[__sk_buff_data]); \ + r6 = r7; \ + r6 += 40; \ + if r6 > r8 goto l0_%=; \ + w4 = w7; \ + w6 -= w4; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +SEC("tc") +__description("32bit scalar -= pkt_ptr") +__success __retval(0) __flag(BPF_F_ANY_ALIGNMENT) +__naked void _32bit_scalar_pkt_ptr(void) +{ + asm volatile (" \ + r8 = *(u32*)(r1 + %[__sk_buff_data_end]); \ + r7 = *(u32*)(r1 + %[__sk_buff_data]); \ + r6 = r7; \ + r6 += 40; \ + if r6 > r8 goto l0_%=; \ + w4 = w6; \ + w4 -= w7; \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)), + __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_loader.c b/tools/testing/selftests/bpf/test_loader.c index 47e9e076bc8f..b4edd8454934 100644 --- a/tools/testing/selftests/bpf/test_loader.c +++ b/tools/testing/selftests/bpf/test_loader.c @@ -25,6 +25,8 @@ #define TEST_TAG_DESCRIPTION_PFX "comment:test_description=" #define TEST_TAG_RETVAL_PFX "comment:test_retval=" #define TEST_TAG_RETVAL_PFX_UNPRIV "comment:test_retval_unpriv=" +#define TEST_TAG_AUXILIARY "comment:test_auxiliary" +#define TEST_TAG_AUXILIARY_UNPRIV "comment:test_auxiliary_unpriv" /* Warning: duplicated in bpf_misc.h */ #define POINTER_VALUE 0xcafe4all @@ -59,6 +61,8 @@ struct test_spec { int log_level; int prog_flags; int mode_mask; + bool auxiliary; + bool valid; }; static int tester_init(struct test_loader *tester) @@ -87,6 +91,11 @@ static void free_test_spec(struct test_spec *spec) free(spec->unpriv.name); free(spec->priv.expect_msgs); free(spec->unpriv.expect_msgs); + + spec->priv.name = NULL; + spec->unpriv.name = NULL; + spec->priv.expect_msgs = NULL; + spec->unpriv.expect_msgs = NULL; } static int push_msg(const char *msg, struct test_subspec *subspec) @@ -204,6 +213,12 @@ static int parse_test_spec(struct test_loader *tester, spec->unpriv.expect_failure = false; spec->mode_mask |= UNPRIV; has_unpriv_result = true; + } else if (strcmp(s, TEST_TAG_AUXILIARY) == 0) { + spec->auxiliary = true; + spec->mode_mask |= PRIV; + } else if (strcmp(s, TEST_TAG_AUXILIARY_UNPRIV) == 0) { + spec->auxiliary = true; + spec->mode_mask |= UNPRIV; } else if (str_has_pfx(s, TEST_TAG_EXPECT_MSG_PFX)) { msg = s + sizeof(TEST_TAG_EXPECT_MSG_PFX) - 1; err = push_msg(msg, &spec->priv); @@ -314,6 +329,8 @@ static int parse_test_spec(struct test_loader *tester, } } + spec->valid = true; + return 0; cleanup: @@ -516,16 +533,18 @@ void run_subtest(struct test_loader *tester, struct bpf_object_open_opts *open_opts, const void *obj_bytes, size_t obj_byte_cnt, + struct test_spec *specs, struct test_spec *spec, bool unpriv) { struct test_subspec *subspec = unpriv ? &spec->unpriv : &spec->priv; + struct bpf_program *tprog, *tprog_iter; + struct test_spec *spec_iter; struct cap_state caps = {}; - struct bpf_program *tprog; struct bpf_object *tobj; struct bpf_map *map; - int retval; - int err; + int retval, err, i; + bool should_load; if (!test__start_subtest(subspec->name)) return; @@ -546,15 +565,23 @@ void run_subtest(struct test_loader *tester, if (!ASSERT_OK_PTR(tobj, "obj_open_mem")) /* shouldn't happen */ goto subtest_cleanup; - bpf_object__for_each_program(tprog, tobj) - bpf_program__set_autoload(tprog, false); + i = 0; + bpf_object__for_each_program(tprog_iter, tobj) { + spec_iter = &specs[i++]; + should_load = false; - bpf_object__for_each_program(tprog, tobj) { - /* only load specified program */ - if (strcmp(bpf_program__name(tprog), spec->prog_name) == 0) { - bpf_program__set_autoload(tprog, true); - break; + if (spec_iter->valid) { + if (strcmp(bpf_program__name(tprog_iter), spec->prog_name) == 0) { + tprog = tprog_iter; + should_load = true; + } + + if (spec_iter->auxiliary && + spec_iter->mode_mask & (unpriv ? UNPRIV : PRIV)) + should_load = true; } + + bpf_program__set_autoload(tprog_iter, should_load); } prepare_case(tester, spec, tobj, tprog); @@ -587,9 +614,17 @@ void run_subtest(struct test_loader *tester, /* For some reason test_verifier executes programs * with all capabilities restored. Do the same here. */ - if (!restore_capabilities(&caps)) + if (restore_capabilities(&caps)) goto tobj_cleanup; + if (tester->pre_execution_cb) { + err = tester->pre_execution_cb(tobj); + if (err) { + PRINT_FAIL("pre_execution_cb failed: %d\n", err); + goto tobj_cleanup; + } + } + do_prog_test_run(bpf_program__fd(tprog), &retval); if (retval != subspec->retval && subspec->retval != POINTER_VALUE) { PRINT_FAIL("Unexpected retval: %d != %d\n", retval, subspec->retval); @@ -609,11 +644,12 @@ static void process_subtest(struct test_loader *tester, skel_elf_bytes_fn elf_bytes_factory) { LIBBPF_OPTS(bpf_object_open_opts, open_opts, .object_name = skel_name); + struct test_spec *specs = NULL; struct bpf_object *obj = NULL; struct bpf_program *prog; const void *obj_bytes; + int err, i, nr_progs; size_t obj_byte_cnt; - int err; if (tester_init(tester) < 0) return; /* failed to initialize tester */ @@ -623,25 +659,42 @@ static void process_subtest(struct test_loader *tester, if (!ASSERT_OK_PTR(obj, "obj_open_mem")) return; - bpf_object__for_each_program(prog, obj) { - struct test_spec spec; + nr_progs = 0; + bpf_object__for_each_program(prog, obj) + ++nr_progs; - /* if we can't derive test specification, go to the next test */ - err = parse_test_spec(tester, obj, prog, &spec); - if (err) { + specs = calloc(nr_progs, sizeof(struct test_spec)); + if (!ASSERT_OK_PTR(specs, "Can't alloc specs array")) + return; + + i = 0; + bpf_object__for_each_program(prog, obj) { + /* ignore tests for which we can't derive test specification */ + err = parse_test_spec(tester, obj, prog, &specs[i++]); + if (err) PRINT_FAIL("Can't parse test spec for program '%s'\n", bpf_program__name(prog)); + } + + i = 0; + bpf_object__for_each_program(prog, obj) { + struct test_spec *spec = &specs[i++]; + + if (!spec->valid || spec->auxiliary) continue; - } - if (spec.mode_mask & PRIV) - run_subtest(tester, &open_opts, obj_bytes, obj_byte_cnt, &spec, false); - if (spec.mode_mask & UNPRIV) - run_subtest(tester, &open_opts, obj_bytes, obj_byte_cnt, &spec, true); + if (spec->mode_mask & PRIV) + run_subtest(tester, &open_opts, obj_bytes, obj_byte_cnt, + specs, spec, false); + if (spec->mode_mask & UNPRIV) + run_subtest(tester, &open_opts, obj_bytes, obj_byte_cnt, + specs, spec, true); - free_test_spec(&spec); } + for (i = 0; i < nr_progs; ++i) + free_test_spec(&specs[i]); + free(specs); bpf_object__close(obj); } diff --git a/tools/testing/selftests/bpf/test_progs.h b/tools/testing/selftests/bpf/test_progs.h index 10ba43250668..0ed3134333d4 100644 --- a/tools/testing/selftests/bpf/test_progs.h +++ b/tools/testing/selftests/bpf/test_progs.h @@ -424,14 +424,23 @@ int get_bpf_max_tramp_links(void); #define BPF_TESTMOD_TEST_FILE "/sys/kernel/bpf_testmod" +typedef int (*pre_execution_cb)(struct bpf_object *obj); + struct test_loader { char *log_buf; size_t log_buf_sz; size_t next_match_pos; + pre_execution_cb pre_execution_cb; struct bpf_object *obj; }; +static inline void test_loader__set_pre_execution_cb(struct test_loader *tester, + pre_execution_cb cb) +{ + tester->pre_execution_cb = cb; +} + typedef const void *(*skel_elf_bytes_fn)(size_t *sz); extern void test_loader__run_subtests(struct test_loader *tester, diff --git a/tools/testing/selftests/bpf/verifier/bounds.c b/tools/testing/selftests/bpf/verifier/bounds.c deleted file mode 100644 index 43942ce8cf15..000000000000 --- a/tools/testing/selftests/bpf/verifier/bounds.c +++ /dev/null @@ -1,884 +0,0 @@ -{ - "subtraction bounds (map value) variant 1", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 9), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_JMP_IMM(BPF_JGT, BPF_REG_1, 0xff, 7), - BPF_LDX_MEM(BPF_B, BPF_REG_3, BPF_REG_0, 1), - BPF_JMP_IMM(BPF_JGT, BPF_REG_3, 0xff, 5), - BPF_ALU64_REG(BPF_SUB, BPF_REG_1, BPF_REG_3), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_1, 56), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr = "R0 max value is outside of the allowed memory range", - .result = REJECT, -}, -{ - "subtraction bounds (map value) variant 2", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 8), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_JMP_IMM(BPF_JGT, BPF_REG_1, 0xff, 6), - BPF_LDX_MEM(BPF_B, BPF_REG_3, BPF_REG_0, 1), - BPF_JMP_IMM(BPF_JGT, BPF_REG_3, 0xff, 4), - BPF_ALU64_REG(BPF_SUB, BPF_REG_1, BPF_REG_3), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr = "R0 min value is negative, either use unsigned index or do a if (index >=0) check.", - .errstr_unpriv = "R1 has unknown scalar with mixed signed bounds", - .result = REJECT, -}, -{ - "check subtraction on pointers for unpriv", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_LD_MAP_FD(BPF_REG_ARG1, 0), - BPF_MOV64_REG(BPF_REG_ARG2, BPF_REG_FP), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_ARG2, -8), - BPF_ST_MEM(BPF_DW, BPF_REG_ARG2, 0, 9), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_MOV64_REG(BPF_REG_9, BPF_REG_FP), - BPF_ALU64_REG(BPF_SUB, BPF_REG_9, BPF_REG_0), - BPF_LD_MAP_FD(BPF_REG_ARG1, 0), - BPF_MOV64_REG(BPF_REG_ARG2, BPF_REG_FP), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_ARG2, -8), - BPF_ST_MEM(BPF_DW, BPF_REG_ARG2, 0, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_STX_MEM(BPF_DW, BPF_REG_0, BPF_REG_9, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 1, 9 }, - .result = ACCEPT, - .result_unpriv = REJECT, - .errstr_unpriv = "R9 pointer -= pointer prohibited", -}, -{ - "bounds check based on zero-extended MOV", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4), - /* r2 = 0x0000'0000'ffff'ffff */ - BPF_MOV32_IMM(BPF_REG_2, 0xffffffff), - /* r2 = 0 */ - BPF_ALU64_IMM(BPF_RSH, BPF_REG_2, 32), - /* no-op */ - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_2), - /* access at offset 0 */ - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - /* exit */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .result = ACCEPT -}, -{ - "bounds check based on sign-extended MOV. test1", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4), - /* r2 = 0xffff'ffff'ffff'ffff */ - BPF_MOV64_IMM(BPF_REG_2, 0xffffffff), - /* r2 = 0xffff'ffff */ - BPF_ALU64_IMM(BPF_RSH, BPF_REG_2, 32), - /* r0 = <oob pointer> */ - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_2), - /* access to OOB pointer */ - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - /* exit */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr = "map_value pointer and 4294967295", - .result = REJECT -}, -{ - "bounds check based on sign-extended MOV. test2", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4), - /* r2 = 0xffff'ffff'ffff'ffff */ - BPF_MOV64_IMM(BPF_REG_2, 0xffffffff), - /* r2 = 0xfff'ffff */ - BPF_ALU64_IMM(BPF_RSH, BPF_REG_2, 36), - /* r0 = <oob pointer> */ - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_2), - /* access to OOB pointer */ - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - /* exit */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr = "R0 min value is outside of the allowed memory range", - .result = REJECT -}, -{ - "bounds check based on reg_off + var_off + insn_off. test1", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_1, - offsetof(struct __sk_buff, mark)), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4), - BPF_ALU64_IMM(BPF_AND, BPF_REG_6, 1), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, (1 << 29) - 1), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_6), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, (1 << 29) - 1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 3), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 4 }, - .errstr = "value_size=8 off=1073741825", - .result = REJECT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "bounds check based on reg_off + var_off + insn_off. test2", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_1, - offsetof(struct __sk_buff, mark)), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4), - BPF_ALU64_IMM(BPF_AND, BPF_REG_6, 1), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, (1 << 30) - 1), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_6), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, (1 << 29) - 1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 3), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 4 }, - .errstr = "value 1073741823", - .result = REJECT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "bounds check after truncation of non-boundary-crossing range", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 9), - /* r1 = [0x00, 0xff] */ - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_2, 1), - /* r2 = 0x10'0000'0000 */ - BPF_ALU64_IMM(BPF_LSH, BPF_REG_2, 36), - /* r1 = [0x10'0000'0000, 0x10'0000'00ff] */ - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_2), - /* r1 = [0x10'7fff'ffff, 0x10'8000'00fe] */ - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 0x7fffffff), - /* r1 = [0x00, 0xff] */ - BPF_ALU32_IMM(BPF_SUB, BPF_REG_1, 0x7fffffff), - /* r1 = 0 */ - BPF_ALU64_IMM(BPF_RSH, BPF_REG_1, 8), - /* no-op */ - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - /* access at offset 0 */ - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - /* exit */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .result = ACCEPT -}, -{ - "bounds check after truncation of boundary-crossing range (1)", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 8), - /* r1 = [0x00, 0xff] */ - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 0xffffff80 >> 1), - /* r1 = [0xffff'ff80, 0x1'0000'007f] */ - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 0xffffff80 >> 1), - /* r1 = [0xffff'ff80, 0xffff'ffff] or - * [0x0000'0000, 0x0000'007f] - */ - BPF_ALU32_IMM(BPF_ADD, BPF_REG_1, 0), - BPF_ALU64_IMM(BPF_SUB, BPF_REG_1, 0xffffff80 >> 1), - /* r1 = [0x00, 0xff] or - * [0xffff'ffff'0000'0080, 0xffff'ffff'ffff'ffff] - */ - BPF_ALU64_IMM(BPF_SUB, BPF_REG_1, 0xffffff80 >> 1), - /* error on OOB pointer computation */ - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - /* exit */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - /* not actually fully unbounded, but the bound is very high */ - .errstr = "value -4294967168 makes map_value pointer be out of bounds", - .result = REJECT, -}, -{ - "bounds check after truncation of boundary-crossing range (2)", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 8), - /* r1 = [0x00, 0xff] */ - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 0xffffff80 >> 1), - /* r1 = [0xffff'ff80, 0x1'0000'007f] */ - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 0xffffff80 >> 1), - /* r1 = [0xffff'ff80, 0xffff'ffff] or - * [0x0000'0000, 0x0000'007f] - * difference to previous test: truncation via MOV32 - * instead of ALU32. - */ - BPF_MOV32_REG(BPF_REG_1, BPF_REG_1), - BPF_ALU64_IMM(BPF_SUB, BPF_REG_1, 0xffffff80 >> 1), - /* r1 = [0x00, 0xff] or - * [0xffff'ffff'0000'0080, 0xffff'ffff'ffff'ffff] - */ - BPF_ALU64_IMM(BPF_SUB, BPF_REG_1, 0xffffff80 >> 1), - /* error on OOB pointer computation */ - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - /* exit */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr = "value -4294967168 makes map_value pointer be out of bounds", - .result = REJECT, -}, -{ - "bounds check after wrapping 32-bit addition", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 5), - /* r1 = 0x7fff'ffff */ - BPF_MOV64_IMM(BPF_REG_1, 0x7fffffff), - /* r1 = 0xffff'fffe */ - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 0x7fffffff), - /* r1 = 0 */ - BPF_ALU32_IMM(BPF_ADD, BPF_REG_1, 2), - /* no-op */ - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - /* access at offset 0 */ - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - /* exit */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .result = ACCEPT -}, -{ - "bounds check after shift with oversized count operand", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 6), - BPF_MOV64_IMM(BPF_REG_2, 32), - BPF_MOV64_IMM(BPF_REG_1, 1), - /* r1 = (u32)1 << (u32)32 = ? */ - BPF_ALU32_REG(BPF_LSH, BPF_REG_1, BPF_REG_2), - /* r1 = [0x0000, 0xffff] */ - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 0xffff), - /* computes unknown pointer, potentially OOB */ - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - /* potentially OOB access */ - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - /* exit */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr = "R0 max value is outside of the allowed memory range", - .result = REJECT -}, -{ - "bounds check after right shift of maybe-negative number", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 6), - /* r1 = [0x00, 0xff] */ - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - /* r1 = [-0x01, 0xfe] */ - BPF_ALU64_IMM(BPF_SUB, BPF_REG_1, 1), - /* r1 = 0 or 0xff'ffff'ffff'ffff */ - BPF_ALU64_IMM(BPF_RSH, BPF_REG_1, 8), - /* r1 = 0 or 0xffff'ffff'ffff */ - BPF_ALU64_IMM(BPF_RSH, BPF_REG_1, 8), - /* computes unknown pointer, potentially OOB */ - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - /* potentially OOB access */ - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - /* exit */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr = "R0 unbounded memory access", - .result = REJECT -}, -{ - "bounds check after 32-bit right shift with 64-bit input", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 6), - /* r1 = 2 */ - BPF_MOV64_IMM(BPF_REG_1, 2), - /* r1 = 1<<32 */ - BPF_ALU64_IMM(BPF_LSH, BPF_REG_1, 31), - /* r1 = 0 (NOT 2!) */ - BPF_ALU32_IMM(BPF_RSH, BPF_REG_1, 31), - /* r1 = 0xffff'fffe (NOT 0!) */ - BPF_ALU32_IMM(BPF_SUB, BPF_REG_1, 2), - /* error on computing OOB pointer */ - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - /* exit */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr = "math between map_value pointer and 4294967294 is not allowed", - .result = REJECT, -}, -{ - "bounds check map access with off+size signed 32bit overflow. test1", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 0x7ffffffe), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), - BPF_JMP_A(0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr = "map_value pointer and 2147483646", - .result = REJECT -}, -{ - "bounds check map access with off+size signed 32bit overflow. test2", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 0x1fffffff), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 0x1fffffff), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 0x1fffffff), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), - BPF_JMP_A(0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr = "pointer offset 1073741822", - .errstr_unpriv = "R0 pointer arithmetic of map value goes out of range", - .result = REJECT -}, -{ - "bounds check map access with off+size signed 32bit overflow. test3", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_ALU64_IMM(BPF_SUB, BPF_REG_0, 0x1fffffff), - BPF_ALU64_IMM(BPF_SUB, BPF_REG_0, 0x1fffffff), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 2), - BPF_JMP_A(0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr = "pointer offset -1073741822", - .errstr_unpriv = "R0 pointer arithmetic of map value goes out of range", - .result = REJECT -}, -{ - "bounds check map access with off+size signed 32bit overflow. test4", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_IMM(BPF_REG_1, 1000000), - BPF_ALU64_IMM(BPF_MUL, BPF_REG_1, 1000000), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 2), - BPF_JMP_A(0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr = "map_value pointer and 1000000000000", - .result = REJECT -}, -{ - "bounds check mixed 32bit and 64bit arithmetic. test1", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_1, -1), - BPF_ALU64_IMM(BPF_LSH, BPF_REG_1, 32), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 1), - /* r1 = 0xffffFFFF00000001 */ - BPF_JMP32_IMM(BPF_JGT, BPF_REG_1, 1, 3), - /* check ALU64 op keeps 32bit bounds */ - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 1), - BPF_JMP32_IMM(BPF_JGT, BPF_REG_1, 2, 1), - BPF_JMP_A(1), - /* invalid ldx if bounds are lost above */ - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, -1), - BPF_EXIT_INSN(), - }, - .errstr_unpriv = "R0 invalid mem access 'scalar'", - .result_unpriv = REJECT, - .result = ACCEPT -}, -{ - "bounds check mixed 32bit and 64bit arithmetic. test2", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_1, -1), - BPF_ALU64_IMM(BPF_LSH, BPF_REG_1, 32), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 1), - /* r1 = 0xffffFFFF00000001 */ - BPF_MOV64_IMM(BPF_REG_2, 3), - /* r1 = 0x2 */ - BPF_ALU32_IMM(BPF_ADD, BPF_REG_1, 1), - /* check ALU32 op zero extends 64bit bounds */ - BPF_JMP_REG(BPF_JGT, BPF_REG_1, BPF_REG_2, 1), - BPF_JMP_A(1), - /* invalid ldx if bounds are lost above */ - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, -1), - BPF_EXIT_INSN(), - }, - .errstr_unpriv = "R0 invalid mem access 'scalar'", - .result_unpriv = REJECT, - .result = ACCEPT -}, -{ - "assigning 32bit bounds to 64bit for wA = 0, wB = wA", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_8, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_LDX_MEM(BPF_W, BPF_REG_7, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_MOV32_IMM(BPF_REG_9, 0), - BPF_MOV32_REG(BPF_REG_2, BPF_REG_9), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_7), - BPF_ALU64_REG(BPF_ADD, BPF_REG_6, BPF_REG_2), - BPF_MOV64_REG(BPF_REG_3, BPF_REG_6), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, 8), - BPF_JMP_REG(BPF_JGT, BPF_REG_3, BPF_REG_8, 1), - BPF_LDX_MEM(BPF_W, BPF_REG_5, BPF_REG_6, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "bounds check for reg = 0, reg xor 1", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_IMM(BPF_REG_1, 0), - BPF_ALU64_IMM(BPF_XOR, BPF_REG_1, 1), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 1), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 8), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr_unpriv = "R0 min value is outside of the allowed memory range", - .result_unpriv = REJECT, - .fixup_map_hash_8b = { 3 }, - .result = ACCEPT, -}, -{ - "bounds check for reg32 = 0, reg32 xor 1", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV32_IMM(BPF_REG_1, 0), - BPF_ALU32_IMM(BPF_XOR, BPF_REG_1, 1), - BPF_JMP32_IMM(BPF_JNE, BPF_REG_1, 0, 1), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 8), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr_unpriv = "R0 min value is outside of the allowed memory range", - .result_unpriv = REJECT, - .fixup_map_hash_8b = { 3 }, - .result = ACCEPT, -}, -{ - "bounds check for reg = 2, reg xor 3", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_IMM(BPF_REG_1, 2), - BPF_ALU64_IMM(BPF_XOR, BPF_REG_1, 3), - BPF_JMP_IMM(BPF_JGT, BPF_REG_1, 0, 1), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 8), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr_unpriv = "R0 min value is outside of the allowed memory range", - .result_unpriv = REJECT, - .fixup_map_hash_8b = { 3 }, - .result = ACCEPT, -}, -{ - "bounds check for reg = any, reg xor 3", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_XOR, BPF_REG_1, 3), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 1), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 8), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .result = REJECT, - .errstr = "invalid access to map value", - .errstr_unpriv = "invalid access to map value", -}, -{ - "bounds check for reg32 = any, reg32 xor 3", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, 0), - BPF_ALU32_IMM(BPF_XOR, BPF_REG_1, 3), - BPF_JMP32_IMM(BPF_JNE, BPF_REG_1, 0, 1), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 8), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .result = REJECT, - .errstr = "invalid access to map value", - .errstr_unpriv = "invalid access to map value", -}, -{ - "bounds check for reg > 0, reg xor 3", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, 0), - BPF_JMP_IMM(BPF_JLE, BPF_REG_1, 0, 3), - BPF_ALU64_IMM(BPF_XOR, BPF_REG_1, 3), - BPF_JMP_IMM(BPF_JGE, BPF_REG_1, 0, 1), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 8), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr_unpriv = "R0 min value is outside of the allowed memory range", - .result_unpriv = REJECT, - .fixup_map_hash_8b = { 3 }, - .result = ACCEPT, -}, -{ - "bounds check for reg32 > 0, reg32 xor 3", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, 0), - BPF_JMP32_IMM(BPF_JLE, BPF_REG_1, 0, 3), - BPF_ALU32_IMM(BPF_XOR, BPF_REG_1, 3), - BPF_JMP32_IMM(BPF_JGE, BPF_REG_1, 0, 1), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 8), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr_unpriv = "R0 min value is outside of the allowed memory range", - .result_unpriv = REJECT, - .fixup_map_hash_8b = { 3 }, - .result = ACCEPT, -}, -{ - "bounds checks after 32-bit truncation. test 1", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4), - BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0), - /* This used to reduce the max bound to 0x7fffffff */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 1), - BPF_JMP_IMM(BPF_JGT, BPF_REG_1, 0x7fffffff, 1), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr_unpriv = "R0 leaks addr", - .result_unpriv = REJECT, - .result = ACCEPT, -}, -{ - "bounds checks after 32-bit truncation. test 2", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4), - BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0), - BPF_JMP_IMM(BPF_JSLT, BPF_REG_1, 1, 1), - BPF_JMP32_IMM(BPF_JSLT, BPF_REG_1, 0, 1), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr_unpriv = "R0 leaks addr", - .result_unpriv = REJECT, - .result = ACCEPT, -}, -{ - "bound check with JMP_JLT for crossing 64-bit signed boundary", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, offsetof(struct xdp_md, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, offsetof(struct xdp_md, data_end)), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 1), - BPF_JMP_REG(BPF_JGT, BPF_REG_1, BPF_REG_3, 8), - - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_2, 0), - BPF_LD_IMM64(BPF_REG_0, 0x7fffffffffffff10), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - - BPF_LD_IMM64(BPF_REG_0, 0x8000000000000000), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 1), - /* r1 unsigned range is [0x7fffffffffffff10, 0x800000000000000f] */ - BPF_JMP_REG(BPF_JLT, BPF_REG_0, BPF_REG_1, -2), - - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_XDP, -}, -{ - "bound check with JMP_JSLT for crossing 64-bit signed boundary", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, offsetof(struct xdp_md, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, offsetof(struct xdp_md, data_end)), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 1), - BPF_JMP_REG(BPF_JGT, BPF_REG_1, BPF_REG_3, 13), - - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_2, 0), - BPF_LD_IMM64(BPF_REG_0, 0x7fffffffffffff10), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - - BPF_LD_IMM64(BPF_REG_2, 0x8000000000000fff), - BPF_LD_IMM64(BPF_REG_0, 0x8000000000000000), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 1), - BPF_JMP_REG(BPF_JSGT, BPF_REG_0, BPF_REG_2, 3), - /* r1 signed range is [S64_MIN, S64_MAX] */ - BPF_JMP_REG(BPF_JSLT, BPF_REG_0, BPF_REG_1, -3), - - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_XDP, -}, -{ - "bound check for loop upper bound greater than U32_MAX", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, offsetof(struct xdp_md, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, offsetof(struct xdp_md, data_end)), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 1), - BPF_JMP_REG(BPF_JGT, BPF_REG_1, BPF_REG_3, 8), - - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_2, 0), - BPF_LD_IMM64(BPF_REG_0, 0x100000000), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - - BPF_LD_IMM64(BPF_REG_0, 0x100000000), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 1), - BPF_JMP_REG(BPF_JLT, BPF_REG_0, BPF_REG_1, -2), - - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_XDP, -}, -{ - "bound check with JMP32_JLT for crossing 32-bit signed boundary", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, offsetof(struct xdp_md, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, offsetof(struct xdp_md, data_end)), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 1), - BPF_JMP_REG(BPF_JGT, BPF_REG_1, BPF_REG_3, 6), - - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_2, 0), - BPF_MOV32_IMM(BPF_REG_0, 0x7fffff10), - BPF_ALU32_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - - BPF_MOV32_IMM(BPF_REG_0, 0x80000000), - BPF_ALU32_IMM(BPF_ADD, BPF_REG_0, 1), - /* r1 unsigned range is [0, 0x8000000f] */ - BPF_JMP32_REG(BPF_JLT, BPF_REG_0, BPF_REG_1, -2), - - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_XDP, -}, -{ - "bound check with JMP32_JSLT for crossing 32-bit signed boundary", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, offsetof(struct xdp_md, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, offsetof(struct xdp_md, data_end)), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 1), - BPF_JMP_REG(BPF_JGT, BPF_REG_1, BPF_REG_3, 10), - - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_2, 0), - BPF_MOV32_IMM(BPF_REG_0, 0x7fffff10), - BPF_ALU32_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - - BPF_MOV32_IMM(BPF_REG_2, 0x80000fff), - BPF_MOV32_IMM(BPF_REG_0, 0x80000000), - BPF_ALU32_IMM(BPF_ADD, BPF_REG_0, 1), - BPF_JMP32_REG(BPF_JSGT, BPF_REG_0, BPF_REG_2, 3), - /* r1 signed range is [S32_MIN, S32_MAX] */ - BPF_JMP32_REG(BPF_JSLT, BPF_REG_0, BPF_REG_1, -3), - - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_XDP, -}, diff --git a/tools/testing/selftests/bpf/verifier/bpf_get_stack.c b/tools/testing/selftests/bpf/verifier/bpf_get_stack.c deleted file mode 100644 index 3e024c891178..000000000000 --- a/tools/testing/selftests/bpf/verifier/bpf_get_stack.c +++ /dev/null @@ -1,87 +0,0 @@ -{ - "bpf_get_stack return R0 within range", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 28), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_MOV64_IMM(BPF_REG_9, sizeof(struct test_val)/2), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_7), - BPF_MOV64_IMM(BPF_REG_3, sizeof(struct test_val)/2), - BPF_MOV64_IMM(BPF_REG_4, 256), - BPF_EMIT_CALL(BPF_FUNC_get_stack), - BPF_MOV64_IMM(BPF_REG_1, 0), - BPF_MOV64_REG(BPF_REG_8, BPF_REG_0), - BPF_ALU64_IMM(BPF_LSH, BPF_REG_8, 32), - BPF_ALU64_IMM(BPF_ARSH, BPF_REG_8, 32), - BPF_JMP_REG(BPF_JSGT, BPF_REG_1, BPF_REG_8, 16), - BPF_ALU64_REG(BPF_SUB, BPF_REG_9, BPF_REG_8), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_7), - BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_8), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_9), - BPF_ALU64_IMM(BPF_LSH, BPF_REG_1, 32), - BPF_ALU64_IMM(BPF_ARSH, BPF_REG_1, 32), - BPF_MOV64_REG(BPF_REG_3, BPF_REG_2), - BPF_ALU64_REG(BPF_ADD, BPF_REG_3, BPF_REG_1), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), - BPF_MOV64_IMM(BPF_REG_5, sizeof(struct test_val)/2), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_5), - BPF_JMP_REG(BPF_JGE, BPF_REG_3, BPF_REG_1, 4), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_MOV64_REG(BPF_REG_3, BPF_REG_9), - BPF_MOV64_IMM(BPF_REG_4, 0), - BPF_EMIT_CALL(BPF_FUNC_get_stack), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_48b = { 4 }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACEPOINT, -}, -{ - "bpf_get_task_stack return R0 range is refined", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0), - BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_6, 0), // ctx->meta->seq - BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_1, 8), // ctx->task - BPF_LD_MAP_FD(BPF_REG_1, 0), // fixup_map_array_48b - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_JMP_IMM(BPF_JNE, BPF_REG_7, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - - BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_9, BPF_REG_0), // keep buf for seq_write - BPF_MOV64_IMM(BPF_REG_3, 48), - BPF_MOV64_IMM(BPF_REG_4, 0), - BPF_EMIT_CALL(BPF_FUNC_get_task_stack), - BPF_JMP_IMM(BPF_JSGT, BPF_REG_0, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_9), - BPF_MOV64_REG(BPF_REG_3, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_seq_write), - - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACING, - .expected_attach_type = BPF_TRACE_ITER, - .kfunc = "task", - .runs = -1, // Don't run, just load - .fixup_map_array_48b = { 3 }, -}, diff --git a/tools/testing/selftests/bpf/verifier/btf_ctx_access.c b/tools/testing/selftests/bpf/verifier/btf_ctx_access.c deleted file mode 100644 index 0484d3de040d..000000000000 --- a/tools/testing/selftests/bpf/verifier/btf_ctx_access.c +++ /dev/null @@ -1,25 +0,0 @@ -{ - "btf_ctx_access accept", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, 8), /* load 2nd argument value (int pointer) */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACING, - .expected_attach_type = BPF_TRACE_FENTRY, - .kfunc = "bpf_modify_return_test", -}, - -{ - "btf_ctx_access u32 pointer accept", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, 0), /* load 1nd argument value (u32 pointer) */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACING, - .expected_attach_type = BPF_TRACE_FENTRY, - .kfunc = "bpf_fentry_test9", -}, diff --git a/tools/testing/selftests/bpf/verifier/ctx.c b/tools/testing/selftests/bpf/verifier/ctx.c deleted file mode 100644 index 2fd31612c0b8..000000000000 --- a/tools/testing/selftests/bpf/verifier/ctx.c +++ /dev/null @@ -1,186 +0,0 @@ -{ - "context stores via BPF_ATOMIC", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_ATOMIC_OP(BPF_W, BPF_ADD, BPF_REG_1, BPF_REG_0, offsetof(struct __sk_buff, mark)), - BPF_EXIT_INSN(), - }, - .errstr = "BPF_ATOMIC stores into R1 ctx is not allowed", - .result = REJECT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "arithmetic ops make PTR_TO_CTX unusable", - .insns = { - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, - offsetof(struct __sk_buff, data) - - offsetof(struct __sk_buff, mark)), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, mark)), - BPF_EXIT_INSN(), - }, - .errstr = "dereference of modified ctx ptr", - .result = REJECT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "pass unmodified ctx pointer to helper", - .insns = { - BPF_MOV64_IMM(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, - BPF_FUNC_csum_update), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, -}, -{ - "pass modified ctx pointer to helper, 1", - .insns = { - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, -612), - BPF_MOV64_IMM(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, - BPF_FUNC_csum_update), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = REJECT, - .errstr = "negative offset ctx ptr R1 off=-612 disallowed", -}, -{ - "pass modified ctx pointer to helper, 2", - .insns = { - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, -612), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, - BPF_FUNC_get_socket_cookie), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result_unpriv = REJECT, - .result = REJECT, - .errstr_unpriv = "negative offset ctx ptr R1 off=-612 disallowed", - .errstr = "negative offset ctx ptr R1 off=-612 disallowed", -}, -{ - "pass modified ctx pointer to helper, 3", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_3, 4), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_3), - BPF_MOV64_IMM(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, - BPF_FUNC_csum_update), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = REJECT, - .errstr = "variable ctx access var_off=(0x0; 0x4)", -}, -{ - "pass ctx or null check, 1: ctx", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, - BPF_FUNC_get_netns_cookie), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SOCK_ADDR, - .expected_attach_type = BPF_CGROUP_UDP6_SENDMSG, - .result = ACCEPT, -}, -{ - "pass ctx or null check, 2: null", - .insns = { - BPF_MOV64_IMM(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, - BPF_FUNC_get_netns_cookie), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SOCK_ADDR, - .expected_attach_type = BPF_CGROUP_UDP6_SENDMSG, - .result = ACCEPT, -}, -{ - "pass ctx or null check, 3: 1", - .insns = { - BPF_MOV64_IMM(BPF_REG_1, 1), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, - BPF_FUNC_get_netns_cookie), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SOCK_ADDR, - .expected_attach_type = BPF_CGROUP_UDP6_SENDMSG, - .result = REJECT, - .errstr = "R1 type=scalar expected=ctx", -}, -{ - "pass ctx or null check, 4: ctx - const", - .insns = { - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, -612), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, - BPF_FUNC_get_netns_cookie), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SOCK_ADDR, - .expected_attach_type = BPF_CGROUP_UDP6_SENDMSG, - .result = REJECT, - .errstr = "negative offset ctx ptr R1 off=-612 disallowed", -}, -{ - "pass ctx or null check, 5: null (connect)", - .insns = { - BPF_MOV64_IMM(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, - BPF_FUNC_get_netns_cookie), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SOCK_ADDR, - .expected_attach_type = BPF_CGROUP_INET4_CONNECT, - .result = ACCEPT, -}, -{ - "pass ctx or null check, 6: null (bind)", - .insns = { - BPF_MOV64_IMM(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, - BPF_FUNC_get_netns_cookie), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SOCK, - .expected_attach_type = BPF_CGROUP_INET4_POST_BIND, - .result = ACCEPT, -}, -{ - "pass ctx or null check, 7: ctx (bind)", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, - BPF_FUNC_get_socket_cookie), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SOCK, - .expected_attach_type = BPF_CGROUP_INET4_POST_BIND, - .result = ACCEPT, -}, -{ - "pass ctx or null check, 8: null (bind)", - .insns = { - BPF_MOV64_IMM(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, - BPF_FUNC_get_socket_cookie), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SOCK, - .expected_attach_type = BPF_CGROUP_INET4_POST_BIND, - .result = REJECT, - .errstr = "R1 type=scalar expected=ctx", -}, diff --git a/tools/testing/selftests/bpf/verifier/d_path.c b/tools/testing/selftests/bpf/verifier/d_path.c deleted file mode 100644 index b988396379a7..000000000000 --- a/tools/testing/selftests/bpf/verifier/d_path.c +++ /dev/null @@ -1,37 +0,0 @@ -{ - "d_path accept", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_1, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_MOV64_IMM(BPF_REG_6, 0), - BPF_STX_MEM(BPF_DW, BPF_REG_2, BPF_REG_6, 0), - BPF_LD_IMM64(BPF_REG_3, 8), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_d_path), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACING, - .expected_attach_type = BPF_TRACE_FENTRY, - .kfunc = "dentry_open", -}, -{ - "d_path reject", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_1, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_MOV64_IMM(BPF_REG_6, 0), - BPF_STX_MEM(BPF_DW, BPF_REG_2, BPF_REG_6, 0), - BPF_LD_IMM64(BPF_REG_3, 8), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_d_path), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr = "helper call is not allowed in probe", - .result = REJECT, - .prog_type = BPF_PROG_TYPE_TRACING, - .expected_attach_type = BPF_TRACE_FENTRY, - .kfunc = "d_path", -}, diff --git a/tools/testing/selftests/bpf/verifier/direct_packet_access.c b/tools/testing/selftests/bpf/verifier/direct_packet_access.c deleted file mode 100644 index dce2e28aeb43..000000000000 --- a/tools/testing/selftests/bpf/verifier/direct_packet_access.c +++ /dev/null @@ -1,710 +0,0 @@ -{ - "pkt_end - pkt_start is allowed", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_2), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = TEST_DATA_LEN, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "direct packet access: test1", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "direct packet access: test2", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_LDX_MEM(BPF_W, BPF_REG_4, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_MOV64_REG(BPF_REG_5, BPF_REG_3), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_5, 14), - BPF_JMP_REG(BPF_JGT, BPF_REG_5, BPF_REG_4, 15), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_3, 7), - BPF_LDX_MEM(BPF_B, BPF_REG_4, BPF_REG_3, 12), - BPF_ALU64_IMM(BPF_MUL, BPF_REG_4, 14), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_ALU64_REG(BPF_ADD, BPF_REG_3, BPF_REG_4), - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, len)), - BPF_ALU64_IMM(BPF_LSH, BPF_REG_2, 49), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_2, 49), - BPF_ALU64_REG(BPF_ADD, BPF_REG_3, BPF_REG_2), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_3), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, 8), - BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_JMP_REG(BPF_JGT, BPF_REG_2, BPF_REG_1, 1), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_3, 4), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "direct packet access: test3", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr = "invalid bpf_context access off=76", - .result = REJECT, - .prog_type = BPF_PROG_TYPE_SOCKET_FILTER, -}, -{ - "direct packet access: test4 (write)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 1), - BPF_STX_MEM(BPF_B, BPF_REG_2, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "direct packet access: test5 (pkt_end >= reg, good access)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JGE, BPF_REG_3, BPF_REG_0, 2), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "direct packet access: test6 (pkt_end >= reg, bad access)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JGE, BPF_REG_3, BPF_REG_0, 3), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr = "invalid access to packet", - .result = REJECT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "direct packet access: test7 (pkt_end >= reg, both accesses)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JGE, BPF_REG_3, BPF_REG_0, 3), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr = "invalid access to packet", - .result = REJECT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "direct packet access: test8 (double test, variant 1)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JGE, BPF_REG_3, BPF_REG_0, 4), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "direct packet access: test9 (double test, variant 2)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JGE, BPF_REG_3, BPF_REG_0, 2), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_2, 0), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "direct packet access: test10 (write invalid)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_STX_MEM(BPF_B, BPF_REG_2, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr = "invalid access to packet", - .result = REJECT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "direct packet access: test11 (shift, good access)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 22), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 8), - BPF_MOV64_IMM(BPF_REG_3, 144), - BPF_MOV64_REG(BPF_REG_5, BPF_REG_3), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_5, 23), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_5, 3), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_2), - BPF_ALU64_REG(BPF_ADD, BPF_REG_6, BPF_REG_5), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .retval = 1, -}, -{ - "direct packet access: test12 (and, good access)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 22), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 8), - BPF_MOV64_IMM(BPF_REG_3, 144), - BPF_MOV64_REG(BPF_REG_5, BPF_REG_3), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_5, 23), - BPF_ALU64_IMM(BPF_AND, BPF_REG_5, 15), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_2), - BPF_ALU64_REG(BPF_ADD, BPF_REG_6, BPF_REG_5), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .retval = 1, -}, -{ - "direct packet access: test13 (branches, good access)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 22), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 13), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, mark)), - BPF_MOV64_IMM(BPF_REG_4, 1), - BPF_JMP_REG(BPF_JGT, BPF_REG_3, BPF_REG_4, 2), - BPF_MOV64_IMM(BPF_REG_3, 14), - BPF_JMP_IMM(BPF_JA, 0, 0, 1), - BPF_MOV64_IMM(BPF_REG_3, 24), - BPF_MOV64_REG(BPF_REG_5, BPF_REG_3), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_5, 23), - BPF_ALU64_IMM(BPF_AND, BPF_REG_5, 15), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_2), - BPF_ALU64_REG(BPF_ADD, BPF_REG_6, BPF_REG_5), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .retval = 1, -}, -{ - "direct packet access: test14 (pkt_ptr += 0, CONST_IMM, good access)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 22), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 7), - BPF_MOV64_IMM(BPF_REG_5, 12), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_5, 4), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_2), - BPF_ALU64_REG(BPF_ADD, BPF_REG_6, BPF_REG_5), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_6, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .retval = 1, -}, -{ - "direct packet access: test15 (spill with xadd)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 8), - BPF_MOV64_IMM(BPF_REG_5, 4096), - BPF_MOV64_REG(BPF_REG_4, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, -8), - BPF_STX_MEM(BPF_DW, BPF_REG_4, BPF_REG_2, 0), - BPF_ATOMIC_OP(BPF_DW, BPF_ADD, BPF_REG_4, BPF_REG_5, 0), - BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_4, 0), - BPF_STX_MEM(BPF_W, BPF_REG_2, BPF_REG_5, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr = "R2 invalid mem access 'scalar'", - .result = REJECT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "direct packet access: test16 (arith on data_end)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, 16), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 1), - BPF_STX_MEM(BPF_B, BPF_REG_2, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr = "R3 pointer arithmetic on pkt_end", - .result = REJECT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "direct packet access: test17 (pruning, alignment)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_LDX_MEM(BPF_W, BPF_REG_7, BPF_REG_1, - offsetof(struct __sk_buff, mark)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 14), - BPF_JMP_IMM(BPF_JGT, BPF_REG_7, 1, 4), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 1), - BPF_STX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, -4), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 1), - BPF_JMP_A(-6), - }, - .errstr = "misaligned packet access off 2+(0x0; 0x0)+15+-4 size 4", - .result = REJECT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .flags = F_LOAD_WITH_STRICT_ALIGNMENT, -}, -{ - "direct packet access: test18 (imm += pkt_ptr, 1)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_IMM(BPF_REG_0, 8), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_2), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 1), - BPF_STX_MEM(BPF_B, BPF_REG_2, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "direct packet access: test19 (imm += pkt_ptr, 2)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 3), - BPF_MOV64_IMM(BPF_REG_4, 4), - BPF_ALU64_REG(BPF_ADD, BPF_REG_4, BPF_REG_2), - BPF_STX_MEM(BPF_B, BPF_REG_4, BPF_REG_4, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "direct packet access: test20 (x += pkt_ptr, 1)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_IMM(BPF_REG_0, 0xffffffff), - BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -8), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_10, -8), - BPF_ALU64_IMM(BPF_AND, BPF_REG_0, 0x7fff), - BPF_MOV64_REG(BPF_REG_4, BPF_REG_0), - BPF_ALU64_REG(BPF_ADD, BPF_REG_4, BPF_REG_2), - BPF_MOV64_REG(BPF_REG_5, BPF_REG_4), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, 0x7fff - 1), - BPF_JMP_REG(BPF_JGT, BPF_REG_4, BPF_REG_3, 1), - BPF_STX_MEM(BPF_DW, BPF_REG_5, BPF_REG_4, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "direct packet access: test21 (x += pkt_ptr, 2)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 9), - BPF_MOV64_IMM(BPF_REG_4, 0xffffffff), - BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_4, -8), - BPF_LDX_MEM(BPF_DW, BPF_REG_4, BPF_REG_10, -8), - BPF_ALU64_IMM(BPF_AND, BPF_REG_4, 0x7fff), - BPF_ALU64_REG(BPF_ADD, BPF_REG_4, BPF_REG_2), - BPF_MOV64_REG(BPF_REG_5, BPF_REG_4), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, 0x7fff - 1), - BPF_JMP_REG(BPF_JGT, BPF_REG_4, BPF_REG_3, 1), - BPF_STX_MEM(BPF_DW, BPF_REG_5, BPF_REG_4, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "direct packet access: test22 (x += pkt_ptr, 3)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_2, -8), - BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_3, -16), - BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_10, -16), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 11), - BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_10, -8), - BPF_MOV64_IMM(BPF_REG_4, 0xffffffff), - BPF_ATOMIC_OP(BPF_DW, BPF_ADD, BPF_REG_10, BPF_REG_4, -8), - BPF_LDX_MEM(BPF_DW, BPF_REG_4, BPF_REG_10, -8), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_4, 49), - BPF_ALU64_REG(BPF_ADD, BPF_REG_4, BPF_REG_2), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_4), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 2), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 2), - BPF_MOV64_IMM(BPF_REG_2, 1), - BPF_STX_MEM(BPF_H, BPF_REG_4, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "direct packet access: test23 (x += pkt_ptr, 4)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, mark)), - BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -8), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_10, -8), - BPF_ALU64_IMM(BPF_AND, BPF_REG_0, 0xffff), - BPF_MOV64_REG(BPF_REG_4, BPF_REG_0), - BPF_MOV64_IMM(BPF_REG_0, 31), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_4), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_2), - BPF_MOV64_REG(BPF_REG_5, BPF_REG_0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 0xffff - 1), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 1), - BPF_STX_MEM(BPF_DW, BPF_REG_5, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = REJECT, - .errstr = "invalid access to packet, off=0 size=8, R5(id=2,off=0,r=0)", - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "direct packet access: test24 (x += pkt_ptr, 5)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_IMM(BPF_REG_0, 0xffffffff), - BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -8), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_10, -8), - BPF_ALU64_IMM(BPF_AND, BPF_REG_0, 0xff), - BPF_MOV64_REG(BPF_REG_4, BPF_REG_0), - BPF_MOV64_IMM(BPF_REG_0, 64), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_4), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_2), - BPF_MOV64_REG(BPF_REG_5, BPF_REG_0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 0x7fff - 1), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 1), - BPF_STX_MEM(BPF_DW, BPF_REG_5, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "direct packet access: test25 (marking on <, good access)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JLT, BPF_REG_0, BPF_REG_3, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_2, 0), - BPF_JMP_IMM(BPF_JA, 0, 0, -4), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "direct packet access: test26 (marking on <, bad access)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JLT, BPF_REG_0, BPF_REG_3, 3), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_JMP_IMM(BPF_JA, 0, 0, -3), - }, - .result = REJECT, - .errstr = "invalid access to packet", - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "direct packet access: test27 (marking on <=, good access)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JLE, BPF_REG_3, BPF_REG_0, 1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .retval = 1, -}, -{ - "direct packet access: test28 (marking on <=, bad access)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JLE, BPF_REG_3, BPF_REG_0, 2), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_2, 0), - BPF_JMP_IMM(BPF_JA, 0, 0, -4), - }, - .result = REJECT, - .errstr = "invalid access to packet", - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "direct packet access: test29 (reg > pkt_end in subprog)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_3, BPF_REG_6), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, 8), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 4), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_6, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_JMP_REG(BPF_JGT, BPF_REG_3, BPF_REG_2, 1), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "direct packet access: test30 (check_id() in regsafe(), bad access)", - .insns = { - /* r9 = ctx */ - BPF_MOV64_REG(BPF_REG_9, BPF_REG_1), - /* r7 = ktime_get_ns() */ - BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - /* r6 = ktime_get_ns() */ - BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - /* r2 = ctx->data - * r3 = ctx->data - * r4 = ctx->data_end - */ - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_9, offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_9, offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_4, BPF_REG_9, offsetof(struct __sk_buff, data_end)), - /* if r6 > 100 goto exit - * if r7 > 100 goto exit - */ - BPF_JMP_IMM(BPF_JGT, BPF_REG_6, 100, 9), - BPF_JMP_IMM(BPF_JGT, BPF_REG_7, 100, 8), - /* r2 += r6 ; this forces assignment of ID to r2 - * r2 += 1 ; get some fixed off for r2 - * r3 += r7 ; this forces assignment of ID to r3 - * r3 += 1 ; get some fixed off for r3 - */ - BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_6), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, 1), - BPF_ALU64_REG(BPF_ADD, BPF_REG_3, BPF_REG_7), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, 1), - /* if r6 > r7 goto +1 ; no new information about the state is derived from - * ; this check, thus produced verifier states differ - * ; only in 'insn_idx' - * r2 = r3 ; optionally share ID between r2 and r3 - */ - BPF_JMP_REG(BPF_JNE, BPF_REG_6, BPF_REG_7, 1), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_3), - /* if r3 > ctx->data_end goto exit */ - BPF_JMP_REG(BPF_JGT, BPF_REG_3, BPF_REG_4, 1), - /* r5 = *(u8 *) (r2 - 1) ; access packet memory using r2, - * ; this is not always safe - */ - BPF_LDX_MEM(BPF_B, BPF_REG_5, BPF_REG_2, -1), - /* exit(0) */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .flags = BPF_F_TEST_STATE_FREQ, - .result = REJECT, - .errstr = "invalid access to packet, off=0 size=1, R2", - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, diff --git a/tools/testing/selftests/bpf/verifier/jeq_infer_not_null.c b/tools/testing/selftests/bpf/verifier/jeq_infer_not_null.c deleted file mode 100644 index 67a1c07ead34..000000000000 --- a/tools/testing/selftests/bpf/verifier/jeq_infer_not_null.c +++ /dev/null @@ -1,174 +0,0 @@ -{ - /* This is equivalent to the following program: - * - * r6 = skb->sk; - * r7 = sk_fullsock(r6); - * r0 = sk_fullsock(r6); - * if (r0 == 0) return 0; (a) - * if (r0 != r7) return 0; (b) - * *r7->type; (c) - * return 0; - * - * It is safe to dereference r7 at point (c), because of (a) and (b). - * The test verifies that relation r0 == r7 is propagated from (b) to (c). - */ - "jne/jeq infer not null, PTR_TO_SOCKET_OR_NULL -> PTR_TO_SOCKET for JNE false branch", - .insns = { - /* r6 = skb->sk; */ - BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, offsetof(struct __sk_buff, sk)), - /* if (r6 == 0) return 0; */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_6, 0, 8), - /* r7 = sk_fullsock(skb); */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - /* r0 = sk_fullsock(skb); */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - /* if (r0 == null) return 0; */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), - /* if (r0 == r7) r0 = *(r7->type); */ - BPF_JMP_REG(BPF_JNE, BPF_REG_0, BPF_REG_7, 1), /* Use ! JNE ! */ - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_7, offsetof(struct bpf_sock, type)), - /* return 0 */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = ACCEPT, - .result_unpriv = REJECT, - .errstr_unpriv = "R7 pointer comparison", -}, -{ - /* Same as above, but verify that another branch of JNE still - * prohibits access to PTR_MAYBE_NULL. - */ - "jne/jeq infer not null, PTR_TO_SOCKET_OR_NULL unchanged for JNE true branch", - .insns = { - /* r6 = skb->sk */ - BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, offsetof(struct __sk_buff, sk)), - /* if (r6 == 0) return 0; */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_6, 0, 9), - /* r7 = sk_fullsock(skb); */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - /* r0 = sk_fullsock(skb); */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - /* if (r0 == null) return 0; */ - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 3), - /* if (r0 == r7) return 0; */ - BPF_JMP_REG(BPF_JNE, BPF_REG_0, BPF_REG_7, 1), /* Use ! JNE ! */ - BPF_JMP_IMM(BPF_JA, 0, 0, 1), - /* r0 = *(r7->type); */ - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_7, offsetof(struct bpf_sock, type)), - /* return 0 */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = REJECT, - .errstr = "R7 invalid mem access 'sock_or_null'", - .result_unpriv = REJECT, - .errstr_unpriv = "R7 pointer comparison", -}, -{ - /* Same as a first test, but not null should be inferred for JEQ branch */ - "jne/jeq infer not null, PTR_TO_SOCKET_OR_NULL -> PTR_TO_SOCKET for JEQ true branch", - .insns = { - /* r6 = skb->sk; */ - BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, offsetof(struct __sk_buff, sk)), - /* if (r6 == null) return 0; */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_6, 0, 9), - /* r7 = sk_fullsock(skb); */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - /* r0 = sk_fullsock(skb); */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - /* if (r0 == null) return 0; */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3), - /* if (r0 != r7) return 0; */ - BPF_JMP_REG(BPF_JEQ, BPF_REG_0, BPF_REG_7, 1), /* Use ! JEQ ! */ - BPF_JMP_IMM(BPF_JA, 0, 0, 1), - /* r0 = *(r7->type); */ - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_7, offsetof(struct bpf_sock, type)), - /* return 0; */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = ACCEPT, - .result_unpriv = REJECT, - .errstr_unpriv = "R7 pointer comparison", -}, -{ - /* Same as above, but verify that another branch of JNE still - * prohibits access to PTR_MAYBE_NULL. - */ - "jne/jeq infer not null, PTR_TO_SOCKET_OR_NULL unchanged for JEQ false branch", - .insns = { - /* r6 = skb->sk; */ - BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, offsetof(struct __sk_buff, sk)), - /* if (r6 == null) return 0; */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_6, 0, 8), - /* r7 = sk_fullsock(skb); */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - /* r0 = sk_fullsock(skb); */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - /* if (r0 == null) return 0; */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), - /* if (r0 != r7) r0 = *(r7->type); */ - BPF_JMP_REG(BPF_JEQ, BPF_REG_0, BPF_REG_7, 1), /* Use ! JEQ ! */ - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_7, offsetof(struct bpf_sock, type)), - /* return 0; */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = REJECT, - .errstr = "R7 invalid mem access 'sock_or_null'", - .result_unpriv = REJECT, - .errstr_unpriv = "R7 pointer comparison", -}, -{ - /* Maps are treated in a different branch of `mark_ptr_not_null_reg`, - * so separate test for maps case. - */ - "jne/jeq infer not null, PTR_TO_MAP_VALUE_OR_NULL -> PTR_TO_MAP_VALUE", - .insns = { - /* r9 = &some stack to use as key */ - BPF_ST_MEM(BPF_W, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_9, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_9, -8), - /* r8 = process local map */ - BPF_LD_MAP_FD(BPF_REG_8, 0), - /* r6 = map_lookup_elem(r8, r9); */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_8), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_9), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - /* r7 = map_lookup_elem(r8, r9); */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_8), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_9), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - /* if (r6 == 0) return 0; */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_6, 0, 2), - /* if (r6 != r7) return 0; */ - BPF_JMP_REG(BPF_JNE, BPF_REG_6, BPF_REG_7, 1), - /* read *r7; */ - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_7, offsetof(struct bpf_xdp_sock, queue_id)), - /* return 0; */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_xskmap = { 3 }, - .prog_type = BPF_PROG_TYPE_XDP, - .result = ACCEPT, -}, diff --git a/tools/testing/selftests/bpf/verifier/loops1.c b/tools/testing/selftests/bpf/verifier/loops1.c deleted file mode 100644 index 1af37187dc12..000000000000 --- a/tools/testing/selftests/bpf/verifier/loops1.c +++ /dev/null @@ -1,206 +0,0 @@ -{ - "bounded loop, count to 4", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 1), - BPF_JMP_IMM(BPF_JLT, BPF_REG_0, 4, -2), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACEPOINT, - .retval = 4, -}, -{ - "bounded loop, count to 20", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 3), - BPF_JMP_IMM(BPF_JLT, BPF_REG_0, 20, -2), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACEPOINT, -}, -{ - "bounded loop, count from positive unknown to 4", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_JMP_IMM(BPF_JSLT, BPF_REG_0, 0, 2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 1), - BPF_JMP_IMM(BPF_JLT, BPF_REG_0, 4, -2), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACEPOINT, - .retval = 4, -}, -{ - "bounded loop, count from totally unknown to 4", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 1), - BPF_JMP_IMM(BPF_JLT, BPF_REG_0, 4, -2), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACEPOINT, -}, -{ - "bounded loop, count to 4 with equality", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 1), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 4, -2), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACEPOINT, -}, -{ - "bounded loop, start in the middle", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_JMP_A(1), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 1), - BPF_JMP_IMM(BPF_JLT, BPF_REG_0, 4, -2), - BPF_EXIT_INSN(), - }, - .result = REJECT, - .errstr = "back-edge", - .prog_type = BPF_PROG_TYPE_TRACEPOINT, - .retval = 4, -}, -{ - "bounded loop containing a forward jump", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 1), - BPF_JMP_REG(BPF_JEQ, BPF_REG_0, BPF_REG_0, 0), - BPF_JMP_IMM(BPF_JLT, BPF_REG_0, 4, -3), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACEPOINT, - .retval = 4, -}, -{ - "bounded loop that jumps out rather than in", - .insns = { - BPF_MOV64_IMM(BPF_REG_6, 0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, 1), - BPF_JMP_IMM(BPF_JGT, BPF_REG_6, 10000, 2), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_JMP_A(-4), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACEPOINT, -}, -{ - "infinite loop after a conditional jump", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 5), - BPF_JMP_IMM(BPF_JLT, BPF_REG_0, 4, 2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 1), - BPF_JMP_A(-2), - BPF_EXIT_INSN(), - }, - .result = REJECT, - .errstr = "program is too large", - .prog_type = BPF_PROG_TYPE_TRACEPOINT, -}, -{ - "bounded recursion", - .insns = { - BPF_MOV64_IMM(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 1), - BPF_EXIT_INSN(), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 1), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_1), - BPF_JMP_IMM(BPF_JLT, BPF_REG_1, 4, 1), - BPF_EXIT_INSN(), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, -5), - BPF_EXIT_INSN(), - }, - .result = REJECT, - .errstr = "back-edge", - .prog_type = BPF_PROG_TYPE_TRACEPOINT, -}, -{ - "infinite loop in two jumps", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_JMP_A(0), - BPF_JMP_IMM(BPF_JLT, BPF_REG_0, 4, -2), - BPF_EXIT_INSN(), - }, - .result = REJECT, - .errstr = "loop detected", - .prog_type = BPF_PROG_TYPE_TRACEPOINT, -}, -{ - "infinite loop: three-jump trick", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 1), - BPF_ALU64_IMM(BPF_AND, BPF_REG_0, 1), - BPF_JMP_IMM(BPF_JLT, BPF_REG_0, 2, 1), - BPF_EXIT_INSN(), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 1), - BPF_ALU64_IMM(BPF_AND, BPF_REG_0, 1), - BPF_JMP_IMM(BPF_JLT, BPF_REG_0, 2, 1), - BPF_EXIT_INSN(), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 1), - BPF_ALU64_IMM(BPF_AND, BPF_REG_0, 1), - BPF_JMP_IMM(BPF_JLT, BPF_REG_0, 2, -11), - BPF_EXIT_INSN(), - }, - .result = REJECT, - .errstr = "loop detected", - .prog_type = BPF_PROG_TYPE_TRACEPOINT, -}, -{ - "not-taken loop with back jump to 1st insn", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 123), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 4, -2), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_XDP, - .retval = 123, -}, -{ - "taken loop with back jump to 1st insn", - .insns = { - BPF_MOV64_IMM(BPF_REG_1, 10), - BPF_MOV64_IMM(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 1), - BPF_EXIT_INSN(), - BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1), - BPF_ALU64_IMM(BPF_SUB, BPF_REG_1, 1), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, -3), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_XDP, - .retval = 55, -}, -{ - "taken loop with back jump to 1st insn, 2", - .insns = { - BPF_MOV64_IMM(BPF_REG_1, 10), - BPF_MOV64_IMM(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 1), - BPF_EXIT_INSN(), - BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1), - BPF_ALU64_IMM(BPF_SUB, BPF_REG_1, 1), - BPF_JMP32_IMM(BPF_JNE, BPF_REG_1, 0, -3), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_XDP, - .retval = 55, -}, diff --git a/tools/testing/selftests/bpf/verifier/lwt.c b/tools/testing/selftests/bpf/verifier/lwt.c deleted file mode 100644 index 5c8944d0b091..000000000000 --- a/tools/testing/selftests/bpf/verifier/lwt.c +++ /dev/null @@ -1,189 +0,0 @@ -{ - "invalid direct packet write for LWT_IN", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 1), - BPF_STX_MEM(BPF_B, BPF_REG_2, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr = "cannot write into packet", - .result = REJECT, - .prog_type = BPF_PROG_TYPE_LWT_IN, -}, -{ - "invalid direct packet write for LWT_OUT", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 1), - BPF_STX_MEM(BPF_B, BPF_REG_2, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr = "cannot write into packet", - .result = REJECT, - .prog_type = BPF_PROG_TYPE_LWT_OUT, -}, -{ - "direct packet write for LWT_XMIT", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 1), - BPF_STX_MEM(BPF_B, BPF_REG_2, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_LWT_XMIT, -}, -{ - "direct packet read for LWT_IN", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_LWT_IN, -}, -{ - "direct packet read for LWT_OUT", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_LWT_OUT, -}, -{ - "direct packet read for LWT_XMIT", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_LWT_XMIT, -}, -{ - "overlapping checks for direct packet access", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 4), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 6), - BPF_JMP_REG(BPF_JGT, BPF_REG_1, BPF_REG_3, 1), - BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_2, 6), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_LWT_XMIT, -}, -{ - "make headroom for LWT_XMIT", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_MOV64_IMM(BPF_REG_2, 34), - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_EMIT_CALL(BPF_FUNC_skb_change_head), - /* split for s390 to succeed */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_MOV64_IMM(BPF_REG_2, 42), - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_EMIT_CALL(BPF_FUNC_skb_change_head), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_LWT_XMIT, -}, -{ - "invalid access of tc_classid for LWT_IN", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, tc_classid)), - BPF_EXIT_INSN(), - }, - .result = REJECT, - .errstr = "invalid bpf_context access", -}, -{ - "invalid access of tc_classid for LWT_OUT", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, tc_classid)), - BPF_EXIT_INSN(), - }, - .result = REJECT, - .errstr = "invalid bpf_context access", -}, -{ - "invalid access of tc_classid for LWT_XMIT", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, tc_classid)), - BPF_EXIT_INSN(), - }, - .result = REJECT, - .errstr = "invalid bpf_context access", -}, -{ - "check skb->tc_classid half load not permitted for lwt prog", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 0), -#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ - BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, tc_classid)), -#else - BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, tc_classid) + 2), -#endif - BPF_EXIT_INSN(), - }, - .result = REJECT, - .errstr = "invalid bpf_context access", - .prog_type = BPF_PROG_TYPE_LWT_IN, -}, diff --git a/tools/testing/selftests/bpf/verifier/map_in_map.c b/tools/testing/selftests/bpf/verifier/map_in_map.c deleted file mode 100644 index 128a348b762d..000000000000 --- a/tools/testing/selftests/bpf/verifier/map_in_map.c +++ /dev/null @@ -1,96 +0,0 @@ -{ - "map in map access", - .insns = { - BPF_ST_MEM(0, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 5), - BPF_ST_MEM(0, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_in_map = { 3 }, - .result = ACCEPT, -}, -{ - "map in map state pruning", - .insns = { - BPF_ST_MEM(0, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -4), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_6), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_6), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 11), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_6), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_6), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_in_map = { 4, 14 }, - .flags = BPF_F_TEST_STATE_FREQ, - .result = VERBOSE_ACCEPT, - .errstr = "processed 25 insns", - .prog_type = BPF_PROG_TYPE_XDP, -}, -{ - "invalid inner map pointer", - .insns = { - BPF_ST_MEM(0, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 6), - BPF_ST_MEM(0, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 8), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_in_map = { 3 }, - .errstr = "R1 pointer arithmetic on map_ptr prohibited", - .result = REJECT, -}, -{ - "forgot null checking on the inner map pointer", - .insns = { - BPF_ST_MEM(0, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_ST_MEM(0, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_in_map = { 3 }, - .errstr = "R1 type=map_value_or_null expected=map_ptr", - .result = REJECT, -}, diff --git a/tools/testing/selftests/bpf/verifier/map_kptr.c b/tools/testing/selftests/bpf/verifier/map_kptr.c index d775ccb01989..a0cfc06d75bc 100644 --- a/tools/testing/selftests/bpf/verifier/map_kptr.c +++ b/tools/testing/selftests/bpf/verifier/map_kptr.c @@ -288,33 +288,6 @@ .result = REJECT, .errstr = "off=0 kptr isn't referenced kptr", }, -{ - "map_kptr: unref: bpf_kfunc_call_test_kptr_get rejected", - .insns = { - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_LD_MAP_FD(BPF_REG_6, 0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_ST_MEM(BPF_W, BPF_REG_2, 0, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_MOV64_IMM(BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .fixup_map_kptr = { 1 }, - .result = REJECT, - .errstr = "arg#0 no referenced kptr at map value offset=0", - .fixup_kfunc_btf_id = { - { "bpf_kfunc_call_test_kptr_get", 13 }, - } -}, /* Tests for referenced PTR_TO_BTF_ID */ { "map_kptr: ref: loaded pointer marked as untrusted", diff --git a/tools/testing/selftests/bpf/verifier/map_ptr_mixing.c b/tools/testing/selftests/bpf/verifier/map_ptr_mixing.c deleted file mode 100644 index 1f2b8c4cb26d..000000000000 --- a/tools/testing/selftests/bpf/verifier/map_ptr_mixing.c +++ /dev/null @@ -1,100 +0,0 @@ -{ - "calls: two calls returning different map pointers for lookup (hash, array)", - .insns = { - /* main prog */ - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_CALL_REL(11), - BPF_JMP_IMM(BPF_JA, 0, 0, 1), - BPF_CALL_REL(12), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), - BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, offsetof(struct test_val, foo)), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - /* subprog 1 */ - BPF_LD_MAP_FD(BPF_REG_0, 0), - BPF_EXIT_INSN(), - /* subprog 2 */ - BPF_LD_MAP_FD(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .fixup_map_hash_48b = { 13 }, - .fixup_map_array_48b = { 16 }, - .result = ACCEPT, - .retval = 1, -}, -{ - "calls: two calls returning different map pointers for lookup (hash, map in map)", - .insns = { - /* main prog */ - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_CALL_REL(11), - BPF_JMP_IMM(BPF_JA, 0, 0, 1), - BPF_CALL_REL(12), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), - BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, offsetof(struct test_val, foo)), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - /* subprog 1 */ - BPF_LD_MAP_FD(BPF_REG_0, 0), - BPF_EXIT_INSN(), - /* subprog 2 */ - BPF_LD_MAP_FD(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .fixup_map_in_map = { 16 }, - .fixup_map_array_48b = { 13 }, - .result = REJECT, - .errstr = "only read from bpf_array is supported", -}, -{ - "cond: two branches returning different map pointers for lookup (tail, tail)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_1, - offsetof(struct __sk_buff, mark)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_6, 0, 3), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_JMP_IMM(BPF_JA, 0, 0, 2), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_3, 7), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 5 }, - .fixup_prog2 = { 2 }, - .result_unpriv = REJECT, - .errstr_unpriv = "tail_call abusing map_ptr", - .result = ACCEPT, - .retval = 42, -}, -{ - "cond: two branches returning same map pointers for lookup (tail, tail)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_1, - offsetof(struct __sk_buff, mark)), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_6, 0, 3), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_JMP_IMM(BPF_JA, 0, 0, 2), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_3, 7), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_prog2 = { 2, 5 }, - .result_unpriv = ACCEPT, - .result = ACCEPT, - .retval = 42, -}, diff --git a/tools/testing/selftests/bpf/verifier/ref_tracking.c b/tools/testing/selftests/bpf/verifier/ref_tracking.c deleted file mode 100644 index 5a2e154dd1e0..000000000000 --- a/tools/testing/selftests/bpf/verifier/ref_tracking.c +++ /dev/null @@ -1,1082 +0,0 @@ -{ - "reference tracking: leak potential reference", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), /* leak reference */ - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "Unreleased reference", - .result = REJECT, -}, -{ - "reference tracking: leak potential reference to sock_common", - .insns = { - BPF_SK_LOOKUP(skc_lookup_tcp), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), /* leak reference */ - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "Unreleased reference", - .result = REJECT, -}, -{ - "reference tracking: leak potential reference on stack", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_4, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, -8), - BPF_STX_MEM(BPF_DW, BPF_REG_4, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "Unreleased reference", - .result = REJECT, -}, -{ - "reference tracking: leak potential reference on stack 2", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_4, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, -8), - BPF_STX_MEM(BPF_DW, BPF_REG_4, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_ST_MEM(BPF_DW, BPF_REG_4, 0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "Unreleased reference", - .result = REJECT, -}, -{ - "reference tracking: zero potential reference", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_IMM(BPF_REG_0, 0), /* leak reference */ - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "Unreleased reference", - .result = REJECT, -}, -{ - "reference tracking: zero potential reference to sock_common", - .insns = { - BPF_SK_LOOKUP(skc_lookup_tcp), - BPF_MOV64_IMM(BPF_REG_0, 0), /* leak reference */ - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "Unreleased reference", - .result = REJECT, -}, -{ - "reference tracking: copy and zero potential references", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_7, 0), /* leak reference */ - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "Unreleased reference", - .result = REJECT, -}, -{ - "reference tracking: acquire/release user key reference", - .insns = { - BPF_MOV64_IMM(BPF_REG_1, -3), - BPF_MOV64_IMM(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, 0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_LSM, - .kfunc = "bpf", - .expected_attach_type = BPF_LSM_MAC, - .flags = BPF_F_SLEEPABLE, - .fixup_kfunc_btf_id = { - { "bpf_lookup_user_key", 2 }, - { "bpf_key_put", 5 }, - }, - .result = ACCEPT, -}, -{ - "reference tracking: acquire/release system key reference", - .insns = { - BPF_MOV64_IMM(BPF_REG_1, 1), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, 0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_LSM, - .kfunc = "bpf", - .expected_attach_type = BPF_LSM_MAC, - .flags = BPF_F_SLEEPABLE, - .fixup_kfunc_btf_id = { - { "bpf_lookup_system_key", 1 }, - { "bpf_key_put", 4 }, - }, - .result = ACCEPT, -}, -{ - "reference tracking: release user key reference without check", - .insns = { - BPF_MOV64_IMM(BPF_REG_1, -3), - BPF_MOV64_IMM(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, 0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_LSM, - .kfunc = "bpf", - .expected_attach_type = BPF_LSM_MAC, - .flags = BPF_F_SLEEPABLE, - .errstr = "Possibly NULL pointer passed to trusted arg0", - .fixup_kfunc_btf_id = { - { "bpf_lookup_user_key", 2 }, - { "bpf_key_put", 4 }, - }, - .result = REJECT, -}, -{ - "reference tracking: release system key reference without check", - .insns = { - BPF_MOV64_IMM(BPF_REG_1, 1), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, 0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_LSM, - .kfunc = "bpf", - .expected_attach_type = BPF_LSM_MAC, - .flags = BPF_F_SLEEPABLE, - .errstr = "Possibly NULL pointer passed to trusted arg0", - .fixup_kfunc_btf_id = { - { "bpf_lookup_system_key", 1 }, - { "bpf_key_put", 3 }, - }, - .result = REJECT, -}, -{ - "reference tracking: release with NULL key pointer", - .insns = { - BPF_MOV64_IMM(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_LSM, - .kfunc = "bpf", - .expected_attach_type = BPF_LSM_MAC, - .flags = BPF_F_SLEEPABLE, - .errstr = "Possibly NULL pointer passed to trusted arg0", - .fixup_kfunc_btf_id = { - { "bpf_key_put", 1 }, - }, - .result = REJECT, -}, -{ - "reference tracking: leak potential reference to user key", - .insns = { - BPF_MOV64_IMM(BPF_REG_1, -3), - BPF_MOV64_IMM(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_LSM, - .kfunc = "bpf", - .expected_attach_type = BPF_LSM_MAC, - .flags = BPF_F_SLEEPABLE, - .errstr = "Unreleased reference", - .fixup_kfunc_btf_id = { - { "bpf_lookup_user_key", 2 }, - }, - .result = REJECT, -}, -{ - "reference tracking: leak potential reference to system key", - .insns = { - BPF_MOV64_IMM(BPF_REG_1, 1), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_LSM, - .kfunc = "bpf", - .expected_attach_type = BPF_LSM_MAC, - .flags = BPF_F_SLEEPABLE, - .errstr = "Unreleased reference", - .fixup_kfunc_btf_id = { - { "bpf_lookup_system_key", 1 }, - }, - .result = REJECT, -}, -{ - "reference tracking: release reference without check", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - /* reference in r0 may be NULL */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_MOV64_IMM(BPF_REG_2, 0), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "type=sock_or_null expected=sock", - .result = REJECT, -}, -{ - "reference tracking: release reference to sock_common without check", - .insns = { - BPF_SK_LOOKUP(skc_lookup_tcp), - /* reference in r0 may be NULL */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_MOV64_IMM(BPF_REG_2, 0), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "type=sock_common_or_null expected=sock", - .result = REJECT, -}, -{ - "reference tracking: release reference", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, -}, -{ - "reference tracking: release reference to sock_common", - .insns = { - BPF_SK_LOOKUP(skc_lookup_tcp), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, -}, -{ - "reference tracking: release reference 2", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, -}, -{ - "reference tracking: release reference twice", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "type=scalar expected=sock", - .result = REJECT, -}, -{ - "reference tracking: release reference twice inside branch", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3), /* goto end */ - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "type=scalar expected=sock", - .result = REJECT, -}, -{ - "reference tracking: alloc, check, free in one subbranch", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 16), - /* if (offsetof(skb, mark) > data_len) exit; */ - BPF_JMP_REG(BPF_JLE, BPF_REG_0, BPF_REG_3, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_2, - offsetof(struct __sk_buff, mark)), - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_6, 0, 1), /* mark == 0? */ - /* Leak reference in R0 */ - BPF_EXIT_INSN(), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), /* sk NULL? */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "Unreleased reference", - .result = REJECT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "reference tracking: alloc, check, free in both subbranches", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 16), - /* if (offsetof(skb, mark) > data_len) exit; */ - BPF_JMP_REG(BPF_JLE, BPF_REG_0, BPF_REG_3, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_2, - offsetof(struct __sk_buff, mark)), - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_6, 0, 4), /* mark == 0? */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), /* sk NULL? */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), /* sk NULL? */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "reference tracking in call: free reference in subprog", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), /* unchecked reference */ - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - - /* subprog 1 */ - BPF_MOV64_REG(BPF_REG_2, BPF_REG_1), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_2, 0, 1), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, -}, -{ - "reference tracking in call: free reference in subprog and outside", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), /* unchecked reference */ - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 3), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - - /* subprog 1 */ - BPF_MOV64_REG(BPF_REG_2, BPF_REG_1), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_2, 0, 1), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "type=scalar expected=sock", - .result = REJECT, -}, -{ - "reference tracking in call: alloc & leak reference in subprog", - .insns = { - BPF_MOV64_REG(BPF_REG_4, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, -8), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 3), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - - /* subprog 1 */ - BPF_MOV64_REG(BPF_REG_6, BPF_REG_4), - BPF_SK_LOOKUP(sk_lookup_tcp), - /* spill unchecked sk_ptr into stack of caller */ - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_0, 0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "Unreleased reference", - .result = REJECT, -}, -{ - "reference tracking in call: alloc in subprog, release outside", - .insns = { - BPF_MOV64_REG(BPF_REG_4, BPF_REG_10), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 4), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - - /* subprog 1 */ - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_EXIT_INSN(), /* return sk */ - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .retval = POINTER_VALUE, - .result = ACCEPT, -}, -{ - "reference tracking in call: sk_ptr leak into caller stack", - .insns = { - BPF_MOV64_REG(BPF_REG_4, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, -8), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - - /* subprog 1 */ - BPF_MOV64_REG(BPF_REG_5, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_5, -8), - BPF_STX_MEM(BPF_DW, BPF_REG_5, BPF_REG_4, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 5), - /* spill unchecked sk_ptr into stack of caller */ - BPF_MOV64_REG(BPF_REG_5, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_5, -8), - BPF_LDX_MEM(BPF_DW, BPF_REG_4, BPF_REG_5, 0), - BPF_STX_MEM(BPF_DW, BPF_REG_4, BPF_REG_0, 0), - BPF_EXIT_INSN(), - - /* subprog 2 */ - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "Unreleased reference", - .result = REJECT, -}, -{ - "reference tracking in call: sk_ptr spill into caller stack", - .insns = { - BPF_MOV64_REG(BPF_REG_4, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, -8), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - - /* subprog 1 */ - BPF_MOV64_REG(BPF_REG_5, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_5, -8), - BPF_STX_MEM(BPF_DW, BPF_REG_5, BPF_REG_4, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 8), - /* spill unchecked sk_ptr into stack of caller */ - BPF_MOV64_REG(BPF_REG_5, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_5, -8), - BPF_LDX_MEM(BPF_DW, BPF_REG_4, BPF_REG_5, 0), - BPF_STX_MEM(BPF_DW, BPF_REG_4, BPF_REG_0, 0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), - /* now the sk_ptr is verified, free the reference */ - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_4, 0), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - - /* subprog 2 */ - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, -}, -{ - "reference tracking: allow LD_ABS", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_LD_ABS(BPF_B, 0), - BPF_LD_ABS(BPF_H, 0), - BPF_LD_ABS(BPF_W, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, -}, -{ - "reference tracking: forbid LD_ABS while holding reference", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_LD_ABS(BPF_B, 0), - BPF_LD_ABS(BPF_H, 0), - BPF_LD_ABS(BPF_W, 0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "BPF_LD_[ABS|IND] cannot be mixed with socket references", - .result = REJECT, -}, -{ - "reference tracking: allow LD_IND", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_MOV64_IMM(BPF_REG_7, 1), - BPF_LD_IND(BPF_W, BPF_REG_7, -0x200000), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_7), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, - .retval = 1, -}, -{ - "reference tracking: forbid LD_IND while holding reference", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_4, BPF_REG_0), - BPF_MOV64_IMM(BPF_REG_7, 1), - BPF_LD_IND(BPF_W, BPF_REG_7, -0x200000), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_7), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_4), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 1), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "BPF_LD_[ABS|IND] cannot be mixed with socket references", - .result = REJECT, -}, -{ - "reference tracking: check reference or tail call", - .insns = { - BPF_MOV64_REG(BPF_REG_7, BPF_REG_1), - BPF_SK_LOOKUP(sk_lookup_tcp), - /* if (sk) bpf_sk_release() */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 7), - /* bpf_tail_call() */ - BPF_MOV64_IMM(BPF_REG_3, 3), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 17 }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, -}, -{ - "reference tracking: release reference then tail call", - .insns = { - BPF_MOV64_REG(BPF_REG_7, BPF_REG_1), - BPF_SK_LOOKUP(sk_lookup_tcp), - /* if (sk) bpf_sk_release() */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 1), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - /* bpf_tail_call() */ - BPF_MOV64_IMM(BPF_REG_3, 3), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 18 }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, -}, -{ - "reference tracking: leak possible reference over tail call", - .insns = { - BPF_MOV64_REG(BPF_REG_7, BPF_REG_1), - /* Look up socket and store in REG_6 */ - BPF_SK_LOOKUP(sk_lookup_tcp), - /* bpf_tail_call() */ - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_IMM(BPF_REG_3, 3), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 0), - /* if (sk) bpf_sk_release() */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 1), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 16 }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "tail_call would lead to reference leak", - .result = REJECT, -}, -{ - "reference tracking: leak checked reference over tail call", - .insns = { - BPF_MOV64_REG(BPF_REG_7, BPF_REG_1), - /* Look up socket and store in REG_6 */ - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - /* if (!sk) goto end */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 7), - /* bpf_tail_call() */ - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 17 }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "tail_call would lead to reference leak", - .result = REJECT, -}, -{ - "reference tracking: mangle and release sock_or_null", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 5), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "R1 pointer arithmetic on sock_or_null prohibited", - .result = REJECT, -}, -{ - "reference tracking: mangle and release sock", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 5), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "R1 pointer arithmetic on sock prohibited", - .result = REJECT, -}, -{ - "reference tracking: access member", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3), - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_0, 4), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, -}, -{ - "reference tracking: write to member", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 5), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_LD_IMM64(BPF_REG_2, 42), - BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_2, - offsetof(struct bpf_sock, mark)), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_LD_IMM64(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "cannot write into sock", - .result = REJECT, -}, -{ - "reference tracking: invalid 64-bit access of member", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3), - BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_0, 0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "invalid sock access off=0 size=8", - .result = REJECT, -}, -{ - "reference tracking: access after release", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "!read_ok", - .result = REJECT, -}, -{ - "reference tracking: direct access for lookup", - .insns = { - /* Check that the packet is at least 64B long */ - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 64), - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 9), - /* sk = sk_lookup_tcp(ctx, skb->data, ...) */ - BPF_MOV64_IMM(BPF_REG_3, sizeof(struct bpf_sock_tuple)), - BPF_MOV64_IMM(BPF_REG_4, 0), - BPF_MOV64_IMM(BPF_REG_5, 0), - BPF_EMIT_CALL(BPF_FUNC_sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3), - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_0, 4), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, -}, -{ - "reference tracking: use ptr from bpf_tcp_sock() after release", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_tcp_sock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 3), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_7, offsetof(struct bpf_tcp_sock, snd_cwnd)), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = REJECT, - .errstr = "invalid mem access", - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "reference tracking: use ptr from bpf_sk_fullsock() after release", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 3), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_7, offsetof(struct bpf_sock, type)), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = REJECT, - .errstr = "invalid mem access", - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "reference tracking: use ptr from bpf_sk_fullsock(tp) after release", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_tcp_sock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 3), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_JMP_IMM(BPF_JNE, BPF_REG_6, 0, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, offsetof(struct bpf_sock, type)), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = REJECT, - .errstr = "invalid mem access", - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "reference tracking: use sk after bpf_sk_release(tp)", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_tcp_sock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 3), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, offsetof(struct bpf_sock, type)), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = REJECT, - .errstr = "invalid mem access", - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "reference tracking: use ptr from bpf_get_listener_sock() after bpf_sk_release(sk)", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_get_listener_sock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 3), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, offsetof(struct bpf_sock, src_port)), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, -}, -{ - "reference tracking: bpf_sk_release(listen_sk)", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_get_listener_sock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 3), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, offsetof(struct bpf_sock, type)), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = REJECT, - .errstr = "R1 must be referenced when passed to release function", -}, -{ - /* !bpf_sk_fullsock(sk) is checked but !bpf_tcp_sock(sk) is not checked */ - "reference tracking: tp->snd_cwnd after bpf_sk_fullsock(sk) and bpf_tcp_sock(sk)", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_tcp_sock), - BPF_MOV64_REG(BPF_REG_8, BPF_REG_0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_7, 0, 3), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_8, offsetof(struct bpf_tcp_sock, snd_cwnd)), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = REJECT, - .errstr = "invalid mem access", -}, -{ - "reference tracking: branch tracking valid pointer null comparison", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_IMM(BPF_REG_3, 1), - BPF_JMP_IMM(BPF_JNE, BPF_REG_6, 0, 1), - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_6, 0, 2), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, -}, -{ - "reference tracking: branch tracking valid pointer value comparison", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_IMM(BPF_REG_3, 1), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_6, 0, 4), - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_6, 1234, 2), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .errstr = "Unreleased reference", - .result = REJECT, -}, -{ - "reference tracking: bpf_sk_release(btf_tcp_sock)", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_skc_to_tcp_sock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 3), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, - .result_unpriv = REJECT, - .errstr_unpriv = "unknown func", -}, -{ - "reference tracking: use ptr from bpf_skc_to_tcp_sock() after release", - .insns = { - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_skc_to_tcp_sock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 3), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_7, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = REJECT, - .errstr = "invalid mem access", - .result_unpriv = REJECT, - .errstr_unpriv = "unknown func", -}, -{ - "reference tracking: try to leak released ptr reg", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_0, -4), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_9, BPF_REG_0), - - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_MOV64_IMM(BPF_REG_2, 8), - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_EMIT_CALL(BPF_FUNC_ringbuf_reserve), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_8, BPF_REG_0), - - BPF_MOV64_REG(BPF_REG_1, BPF_REG_8), - BPF_MOV64_IMM(BPF_REG_2, 0), - BPF_EMIT_CALL(BPF_FUNC_ringbuf_discard), - BPF_MOV64_IMM(BPF_REG_0, 0), - - BPF_STX_MEM(BPF_DW, BPF_REG_9, BPF_REG_8, 0), - BPF_EXIT_INSN() - }, - .fixup_map_array_48b = { 4 }, - .fixup_map_ringbuf = { 11 }, - .result = ACCEPT, - .result_unpriv = REJECT, - .errstr_unpriv = "R8 !read_ok" -}, diff --git a/tools/testing/selftests/bpf/verifier/regalloc.c b/tools/testing/selftests/bpf/verifier/regalloc.c deleted file mode 100644 index bb0dd89dd212..000000000000 --- a/tools/testing/selftests/bpf/verifier/regalloc.c +++ /dev/null @@ -1,277 +0,0 @@ -{ - "regalloc basic", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 8), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - BPF_JMP_IMM(BPF_JSGT, BPF_REG_0, 20, 4), - BPF_JMP_IMM(BPF_JSLT, BPF_REG_2, 0, 3), - BPF_ALU64_REG(BPF_ADD, BPF_REG_7, BPF_REG_0), - BPF_ALU64_REG(BPF_ADD, BPF_REG_7, BPF_REG_2), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_7, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_48b = { 4 }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACEPOINT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "regalloc negative", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 8), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - BPF_JMP_IMM(BPF_JSGT, BPF_REG_0, 24, 4), - BPF_JMP_IMM(BPF_JSLT, BPF_REG_2, 0, 3), - BPF_ALU64_REG(BPF_ADD, BPF_REG_7, BPF_REG_0), - BPF_ALU64_REG(BPF_ADD, BPF_REG_7, BPF_REG_2), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_7, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_48b = { 4 }, - .result = REJECT, - .errstr = "invalid access to map value, value_size=48 off=48 size=1", - .prog_type = BPF_PROG_TYPE_TRACEPOINT, -}, -{ - "regalloc src_reg mark", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 9), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - BPF_JMP_IMM(BPF_JSGT, BPF_REG_0, 20, 5), - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_JMP_REG(BPF_JSGE, BPF_REG_3, BPF_REG_2, 3), - BPF_ALU64_REG(BPF_ADD, BPF_REG_7, BPF_REG_0), - BPF_ALU64_REG(BPF_ADD, BPF_REG_7, BPF_REG_2), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_7, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_48b = { 4 }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACEPOINT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "regalloc src_reg negative", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 9), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - BPF_JMP_IMM(BPF_JSGT, BPF_REG_0, 22, 5), - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_JMP_REG(BPF_JSGE, BPF_REG_3, BPF_REG_2, 3), - BPF_ALU64_REG(BPF_ADD, BPF_REG_7, BPF_REG_0), - BPF_ALU64_REG(BPF_ADD, BPF_REG_7, BPF_REG_2), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_7, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_48b = { 4 }, - .result = REJECT, - .errstr = "invalid access to map value, value_size=48 off=44 size=8", - .prog_type = BPF_PROG_TYPE_TRACEPOINT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "regalloc and spill", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 11), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - BPF_JMP_IMM(BPF_JSGT, BPF_REG_0, 20, 7), - /* r0 has upper bound that should propagate into r2 */ - BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_2, -8), /* spill r2 */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_2, 0), /* clear r0 and r2 */ - BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_10, -8), /* fill r3 */ - BPF_JMP_REG(BPF_JSGE, BPF_REG_0, BPF_REG_3, 2), - /* r3 has lower and upper bounds */ - BPF_ALU64_REG(BPF_ADD, BPF_REG_7, BPF_REG_3), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_7, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_48b = { 4 }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACEPOINT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "regalloc and spill negative", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 11), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - BPF_JMP_IMM(BPF_JSGT, BPF_REG_0, 48, 7), - /* r0 has upper bound that should propagate into r2 */ - BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_2, -8), /* spill r2 */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_2, 0), /* clear r0 and r2 */ - BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_10, -8), /* fill r3 */ - BPF_JMP_REG(BPF_JSGE, BPF_REG_0, BPF_REG_3, 2), - /* r3 has lower and upper bounds */ - BPF_ALU64_REG(BPF_ADD, BPF_REG_7, BPF_REG_3), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_7, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_48b = { 4 }, - .result = REJECT, - .errstr = "invalid access to map value, value_size=48 off=48 size=8", - .prog_type = BPF_PROG_TYPE_TRACEPOINT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "regalloc three regs", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 10), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_4, BPF_REG_2), - BPF_JMP_IMM(BPF_JSGT, BPF_REG_0, 12, 5), - BPF_JMP_IMM(BPF_JSLT, BPF_REG_2, 0, 4), - BPF_ALU64_REG(BPF_ADD, BPF_REG_7, BPF_REG_0), - BPF_ALU64_REG(BPF_ADD, BPF_REG_7, BPF_REG_2), - BPF_ALU64_REG(BPF_ADD, BPF_REG_7, BPF_REG_4), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_7, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_48b = { 4 }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACEPOINT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "regalloc after call", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 10), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_8, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_9, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 6), - BPF_JMP_IMM(BPF_JSGT, BPF_REG_8, 20, 4), - BPF_JMP_IMM(BPF_JSLT, BPF_REG_9, 0, 3), - BPF_ALU64_REG(BPF_ADD, BPF_REG_7, BPF_REG_8), - BPF_ALU64_REG(BPF_ADD, BPF_REG_7, BPF_REG_9), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_7, 0), - BPF_EXIT_INSN(), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_48b = { 4 }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACEPOINT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "regalloc in callee", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 6), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_3, BPF_REG_7), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 1), - BPF_EXIT_INSN(), - BPF_JMP_IMM(BPF_JSGT, BPF_REG_1, 20, 5), - BPF_JMP_IMM(BPF_JSLT, BPF_REG_2, 0, 4), - BPF_ALU64_REG(BPF_ADD, BPF_REG_3, BPF_REG_1), - BPF_ALU64_REG(BPF_ADD, BPF_REG_3, BPF_REG_2), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_3, 0), - BPF_EXIT_INSN(), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_48b = { 4 }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACEPOINT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "regalloc, spill, JEQ", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -8), /* spill r0 */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 0), - /* The verifier will walk the rest twice with r0 == 0 and r0 == map_value */ - BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_2, 20, 0), - /* The verifier will walk the rest two more times with r0 == 20 and r0 == unknown */ - BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_10, -8), /* fill r3 with map_value */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_3, 0, 1), /* skip ldx if map_value == NULL */ - /* Buggy verifier will think that r3 == 20 here */ - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_3, 0), /* read from map_value */ - BPF_EXIT_INSN(), - }, - .fixup_map_hash_48b = { 4 }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACEPOINT, -}, diff --git a/tools/testing/selftests/bpf/verifier/runtime_jit.c b/tools/testing/selftests/bpf/verifier/runtime_jit.c deleted file mode 100644 index 94c399d1faca..000000000000 --- a/tools/testing/selftests/bpf/verifier/runtime_jit.c +++ /dev/null @@ -1,231 +0,0 @@ -{ - "runtime/jit: tail_call within bounds, prog once", - .insns = { - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 1 }, - .result = ACCEPT, - .retval = 42, -}, -{ - "runtime/jit: tail_call within bounds, prog loop", - .insns = { - BPF_MOV64_IMM(BPF_REG_3, 1), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 1 }, - .result = ACCEPT, - .retval = 41, -}, -{ - "runtime/jit: tail_call within bounds, no prog", - .insns = { - BPF_MOV64_IMM(BPF_REG_3, 3), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 1 }, - .result = ACCEPT, - .retval = 1, -}, -{ - "runtime/jit: tail_call within bounds, key 2", - .insns = { - BPF_MOV64_IMM(BPF_REG_3, 2), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 1 }, - .result = ACCEPT, - .retval = 24, -}, -{ - "runtime/jit: tail_call within bounds, key 2 / key 2, first branch", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 13), - BPF_STX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, - offsetof(struct __sk_buff, cb[0])), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, cb[0])), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 13, 4), - BPF_MOV64_IMM(BPF_REG_3, 2), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_JMP_IMM(BPF_JA, 0, 0, 3), - BPF_MOV64_IMM(BPF_REG_3, 2), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 5, 9 }, - .result = ACCEPT, - .retval = 24, -}, -{ - "runtime/jit: tail_call within bounds, key 2 / key 2, second branch", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 14), - BPF_STX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, - offsetof(struct __sk_buff, cb[0])), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, cb[0])), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 13, 4), - BPF_MOV64_IMM(BPF_REG_3, 2), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_JMP_IMM(BPF_JA, 0, 0, 3), - BPF_MOV64_IMM(BPF_REG_3, 2), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 5, 9 }, - .result = ACCEPT, - .retval = 24, -}, -{ - "runtime/jit: tail_call within bounds, key 0 / key 2, first branch", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 13), - BPF_STX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, - offsetof(struct __sk_buff, cb[0])), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, cb[0])), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 13, 4), - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_JMP_IMM(BPF_JA, 0, 0, 3), - BPF_MOV64_IMM(BPF_REG_3, 2), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 5, 9 }, - .result = ACCEPT, - .retval = 24, -}, -{ - "runtime/jit: tail_call within bounds, key 0 / key 2, second branch", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 14), - BPF_STX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, - offsetof(struct __sk_buff, cb[0])), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, cb[0])), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 13, 4), - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_JMP_IMM(BPF_JA, 0, 0, 3), - BPF_MOV64_IMM(BPF_REG_3, 2), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 5, 9 }, - .result = ACCEPT, - .retval = 42, -}, -{ - "runtime/jit: tail_call within bounds, different maps, first branch", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 13), - BPF_STX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, - offsetof(struct __sk_buff, cb[0])), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, cb[0])), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 13, 4), - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_JMP_IMM(BPF_JA, 0, 0, 3), - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 5 }, - .fixup_prog2 = { 9 }, - .result_unpriv = REJECT, - .errstr_unpriv = "tail_call abusing map_ptr", - .result = ACCEPT, - .retval = 1, -}, -{ - "runtime/jit: tail_call within bounds, different maps, second branch", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 14), - BPF_STX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, - offsetof(struct __sk_buff, cb[0])), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, cb[0])), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 13, 4), - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_JMP_IMM(BPF_JA, 0, 0, 3), - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 5 }, - .fixup_prog2 = { 9 }, - .result_unpriv = REJECT, - .errstr_unpriv = "tail_call abusing map_ptr", - .result = ACCEPT, - .retval = 42, -}, -{ - "runtime/jit: tail_call out of bounds", - .insns = { - BPF_MOV64_IMM(BPF_REG_3, 256), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 2), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 1 }, - .result = ACCEPT, - .retval = 2, -}, -{ - "runtime/jit: pass negative index to tail_call", - .insns = { - BPF_MOV64_IMM(BPF_REG_3, -1), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 2), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 1 }, - .result = ACCEPT, - .retval = 2, -}, -{ - "runtime/jit: pass > 32bit index to tail_call", - .insns = { - BPF_LD_IMM64(BPF_REG_3, 0x100000000ULL), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 2), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 2 }, - .result = ACCEPT, - .retval = 42, - /* Verifier rewrite for unpriv skips tail call here. */ - .retval_unpriv = 2, -}, diff --git a/tools/testing/selftests/bpf/verifier/search_pruning.c b/tools/testing/selftests/bpf/verifier/search_pruning.c deleted file mode 100644 index 745d6b5842fd..000000000000 --- a/tools/testing/selftests/bpf/verifier/search_pruning.c +++ /dev/null @@ -1,266 +0,0 @@ -{ - "pointer/scalar confusion in state equality check (way 1)", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), - BPF_JMP_A(1), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_10), - BPF_JMP_A(0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .result = ACCEPT, - .retval = POINTER_VALUE, - .result_unpriv = REJECT, - .errstr_unpriv = "R0 leaks addr as return value" -}, -{ - "pointer/scalar confusion in state equality check (way 2)", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_10), - BPF_JMP_A(1), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .result = ACCEPT, - .retval = POINTER_VALUE, - .result_unpriv = REJECT, - .errstr_unpriv = "R0 leaks addr as return value" -}, -{ - "liveness pruning and write screening", - .insns = { - /* Get an unknown value */ - BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, 0), - /* branch conditions teach us nothing about R2 */ - BPF_JMP_IMM(BPF_JGE, BPF_REG_2, 0, 1), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_JMP_IMM(BPF_JGE, BPF_REG_2, 0, 1), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr = "R0 !read_ok", - .result = REJECT, - .prog_type = BPF_PROG_TYPE_LWT_IN, -}, -{ - "varlen_map_value_access pruning", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 8), - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, 0), - BPF_MOV32_IMM(BPF_REG_2, MAX_ENTRIES), - BPF_JMP_REG(BPF_JSGT, BPF_REG_2, BPF_REG_1, 1), - BPF_MOV32_IMM(BPF_REG_1, 0), - BPF_ALU32_IMM(BPF_LSH, BPF_REG_1, 2), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_JMP_IMM(BPF_JA, 0, 0, 0), - BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, offsetof(struct test_val, foo)), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_48b = { 3 }, - .errstr_unpriv = "R0 leaks addr", - .errstr = "R0 unbounded memory access", - .result_unpriv = REJECT, - .result = REJECT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "search pruning: all branches should be verified (nop operation)", - .insns = { - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_ST_MEM(BPF_DW, BPF_REG_2, 0, 0), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 11), - BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_0, 0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_3, 0xbeef, 2), - BPF_MOV64_IMM(BPF_REG_4, 0), - BPF_JMP_A(1), - BPF_MOV64_IMM(BPF_REG_4, 1), - BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_4, -16), - BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), - BPF_LDX_MEM(BPF_DW, BPF_REG_5, BPF_REG_10, -16), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_5, 0, 2), - BPF_MOV64_IMM(BPF_REG_6, 0), - BPF_ST_MEM(BPF_DW, BPF_REG_6, 0, 0xdead), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr = "R6 invalid mem access 'scalar'", - .result = REJECT, - .prog_type = BPF_PROG_TYPE_TRACEPOINT, -}, -{ - "search pruning: all branches should be verified (invalid stack access)", - .insns = { - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_ST_MEM(BPF_DW, BPF_REG_2, 0, 0), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 8), - BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_4, 0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_3, 0xbeef, 2), - BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_4, -16), - BPF_JMP_A(1), - BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_4, -24), - BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), - BPF_LDX_MEM(BPF_DW, BPF_REG_5, BPF_REG_10, -16), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr_unpriv = "invalid read from stack off -16+0 size 8", - .result_unpriv = REJECT, - /* in privileged mode reads from uninitialized stack locations are permitted */ - .result = ACCEPT, -}, -{ - "precision tracking for u32 spill/fill", - .insns = { - BPF_MOV64_REG(BPF_REG_7, BPF_REG_1), - BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32), - BPF_MOV32_IMM(BPF_REG_6, 32), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), - BPF_MOV32_IMM(BPF_REG_6, 4), - /* Additional insns to introduce a pruning point. */ - BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32), - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), - BPF_MOV64_IMM(BPF_REG_3, 0), - /* u32 spill/fill */ - BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_6, -8), - BPF_LDX_MEM(BPF_W, BPF_REG_8, BPF_REG_10, -8), - /* out-of-bound map value access for r6=32 */ - BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -16), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_8), - BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 15 }, - .result = REJECT, - .errstr = "R0 min value is outside of the allowed memory range", - .prog_type = BPF_PROG_TYPE_TRACEPOINT, -}, -{ - "precision tracking for u32 spills, u64 fill", - .insns = { - BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV32_IMM(BPF_REG_7, 0xffffffff), - /* Additional insns to introduce a pruning point. */ - BPF_MOV64_IMM(BPF_REG_3, 1), - BPF_MOV64_IMM(BPF_REG_3, 1), - BPF_MOV64_IMM(BPF_REG_3, 1), - BPF_MOV64_IMM(BPF_REG_3, 1), - BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), - BPF_MOV64_IMM(BPF_REG_3, 1), - BPF_ALU32_IMM(BPF_DIV, BPF_REG_3, 0), - /* u32 spills, u64 fill */ - BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_6, -4), - BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_7, -8), - BPF_LDX_MEM(BPF_DW, BPF_REG_8, BPF_REG_10, -8), - /* if r8 != X goto pc+1 r8 known in fallthrough branch */ - BPF_JMP_IMM(BPF_JNE, BPF_REG_8, 0xffffffff, 1), - BPF_MOV64_IMM(BPF_REG_3, 1), - /* if r8 == X goto pc+1 condition always true on first - * traversal, so starts backtracking to mark r8 as requiring - * precision. r7 marked as needing precision. r6 not marked - * since it's not tracked. - */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_8, 0xffffffff, 1), - /* fails if r8 correctly marked unknown after fill. */ - BPF_ALU32_IMM(BPF_DIV, BPF_REG_3, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = REJECT, - .errstr = "div by zero", - .prog_type = BPF_PROG_TYPE_TRACEPOINT, -}, -{ - "allocated_stack", - .insns = { - BPF_ALU64_REG(BPF_MOV, BPF_REG_6, BPF_REG_1), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_ALU64_REG(BPF_MOV, BPF_REG_7, BPF_REG_0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 5), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_6, -8), - BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_10, -8), - BPF_STX_MEM(BPF_B, BPF_REG_10, BPF_REG_7, -9), - BPF_LDX_MEM(BPF_B, BPF_REG_7, BPF_REG_10, -9), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .result_unpriv = ACCEPT, - .insn_processed = 15, -}, -/* The test performs a conditional 64-bit write to a stack location - * fp[-8], this is followed by an unconditional 8-bit write to fp[-8], - * then data is read from fp[-8]. This sequence is unsafe. - * - * The test would be mistakenly marked as safe w/o dst register parent - * preservation in verifier.c:copy_register_state() function. - * - * Note the usage of BPF_F_TEST_STATE_FREQ to force creation of the - * checkpoint state after conditional 64-bit assignment. - */ -{ - "write tracking and register parent chain bug", - .insns = { - /* r6 = ktime_get_ns() */ - BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - /* r0 = ktime_get_ns() */ - BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), - /* if r0 > r6 goto +1 */ - BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_6, 1), - /* *(u64 *)(r10 - 8) = 0xdeadbeef */ - BPF_ST_MEM(BPF_DW, BPF_REG_FP, -8, 0xdeadbeef), - /* r1 = 42 */ - BPF_MOV64_IMM(BPF_REG_1, 42), - /* *(u8 *)(r10 - 8) = r1 */ - BPF_STX_MEM(BPF_B, BPF_REG_FP, BPF_REG_1, -8), - /* r2 = *(u64 *)(r10 - 8) */ - BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_FP, -8), - /* exit(0) */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .flags = BPF_F_TEST_STATE_FREQ, - .errstr_unpriv = "invalid read from stack off -8+1 size 8", - .result_unpriv = REJECT, - /* in privileged mode reads from uninitialized stack locations are permitted */ - .result = ACCEPT, -}, diff --git a/tools/testing/selftests/bpf/verifier/sock.c b/tools/testing/selftests/bpf/verifier/sock.c deleted file mode 100644 index 108dd3ee1edd..000000000000 --- a/tools/testing/selftests/bpf/verifier/sock.c +++ /dev/null @@ -1,706 +0,0 @@ -{ - "skb->sk: no NULL check", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = REJECT, - .errstr = "invalid mem access 'sock_common_or_null'", -}, -{ - "skb->sk: sk->family [non fullsock field]", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, offsetof(struct bpf_sock, family)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = ACCEPT, -}, -{ - "skb->sk: sk->type [fullsock field]", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, offsetof(struct bpf_sock, type)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = REJECT, - .errstr = "invalid sock_common access", -}, -{ - "bpf_sk_fullsock(skb->sk): no !skb->sk check", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = REJECT, - .errstr = "type=sock_common_or_null expected=sock_common", -}, -{ - "sk_fullsock(skb->sk): no NULL check on ret", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, offsetof(struct bpf_sock, type)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = REJECT, - .errstr = "invalid mem access 'sock_or_null'", -}, -{ - "sk_fullsock(skb->sk): sk->type [fullsock field]", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, offsetof(struct bpf_sock, type)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = ACCEPT, -}, -{ - "sk_fullsock(skb->sk): sk->family [non fullsock field]", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, offsetof(struct bpf_sock, family)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = ACCEPT, -}, -{ - "sk_fullsock(skb->sk): sk->state [narrow load]", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, offsetof(struct bpf_sock, state)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = ACCEPT, -}, -{ - "sk_fullsock(skb->sk): sk->dst_port [word load] (backward compatibility)", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, offsetof(struct bpf_sock, dst_port)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = ACCEPT, -}, -{ - "sk_fullsock(skb->sk): sk->dst_port [half load]", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_0, offsetof(struct bpf_sock, dst_port)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = ACCEPT, -}, -{ - "sk_fullsock(skb->sk): sk->dst_port [half load] (invalid)", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_0, offsetof(struct bpf_sock, dst_port) + 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = REJECT, - .errstr = "invalid sock access", -}, -{ - "sk_fullsock(skb->sk): sk->dst_port [byte load]", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_B, BPF_REG_2, BPF_REG_0, offsetof(struct bpf_sock, dst_port)), - BPF_LDX_MEM(BPF_B, BPF_REG_2, BPF_REG_0, offsetof(struct bpf_sock, dst_port) + 1), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = ACCEPT, -}, -{ - "sk_fullsock(skb->sk): sk->dst_port [byte load] (invalid)", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, offsetof(struct bpf_sock, dst_port) + 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = REJECT, - .errstr = "invalid sock access", -}, -{ - "sk_fullsock(skb->sk): past sk->dst_port [half load] (invalid)", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_0, offsetofend(struct bpf_sock, dst_port)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = REJECT, - .errstr = "invalid sock access", -}, -{ - "sk_fullsock(skb->sk): sk->dst_ip6 [load 2nd byte]", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, offsetof(struct bpf_sock, dst_ip6[0]) + 1), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = ACCEPT, -}, -{ - "sk_fullsock(skb->sk): sk->type [narrow load]", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, offsetof(struct bpf_sock, type)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = ACCEPT, -}, -{ - "sk_fullsock(skb->sk): sk->protocol [narrow load]", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, offsetof(struct bpf_sock, protocol)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = ACCEPT, -}, -{ - "sk_fullsock(skb->sk): beyond last field", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, offsetofend(struct bpf_sock, rx_queue_mapping)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = REJECT, - .errstr = "invalid sock access", -}, -{ - "bpf_tcp_sock(skb->sk): no !skb->sk check", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_EMIT_CALL(BPF_FUNC_tcp_sock), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = REJECT, - .errstr = "type=sock_common_or_null expected=sock_common", -}, -{ - "bpf_tcp_sock(skb->sk): no NULL check on ret", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_tcp_sock), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, offsetof(struct bpf_tcp_sock, snd_cwnd)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = REJECT, - .errstr = "invalid mem access 'tcp_sock_or_null'", -}, -{ - "bpf_tcp_sock(skb->sk): tp->snd_cwnd", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_tcp_sock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, offsetof(struct bpf_tcp_sock, snd_cwnd)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = ACCEPT, -}, -{ - "bpf_tcp_sock(skb->sk): tp->bytes_acked", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_tcp_sock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, offsetof(struct bpf_tcp_sock, bytes_acked)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = ACCEPT, -}, -{ - "bpf_tcp_sock(skb->sk): beyond last field", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_tcp_sock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, offsetofend(struct bpf_tcp_sock, bytes_acked)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = REJECT, - .errstr = "invalid tcp_sock access", -}, -{ - "bpf_tcp_sock(bpf_sk_fullsock(skb->sk)): tp->snd_cwnd", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_tcp_sock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, offsetof(struct bpf_tcp_sock, snd_cwnd)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .result = ACCEPT, -}, -{ - "bpf_sk_release(skb->sk)", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 1), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = REJECT, - .errstr = "R1 must be referenced when passed to release function", -}, -{ - "bpf_sk_release(bpf_sk_fullsock(skb->sk))", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = REJECT, - .errstr = "R1 must be referenced when passed to release function", -}, -{ - "bpf_sk_release(bpf_tcp_sock(skb->sk))", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_tcp_sock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = REJECT, - .errstr = "R1 must be referenced when passed to release function", -}, -{ - "sk_storage_get(map, skb->sk, NULL, 0): value == NULL", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_MOV64_IMM(BPF_REG_4, 0), - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_sk_storage_get), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_sk_storage_map = { 11 }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, -}, -{ - "sk_storage_get(map, skb->sk, 1, 1): value == 1", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_MOV64_IMM(BPF_REG_4, 1), - BPF_MOV64_IMM(BPF_REG_3, 1), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_sk_storage_get), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_sk_storage_map = { 11 }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = REJECT, - .errstr = "R3 type=scalar expected=fp", -}, -{ - "sk_storage_get(map, skb->sk, &stack_value, 1): stack_value", - .insns = { - BPF_MOV64_IMM(BPF_REG_2, 0), - BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_2, -8), - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_MOV64_IMM(BPF_REG_4, 1), - BPF_MOV64_REG(BPF_REG_3, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, -8), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_sk_storage_get), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_sk_storage_map = { 14 }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, -}, -{ - "bpf_map_lookup_elem(smap, &key)", - .insns = { - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_sk_storage_map = { 3 }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = REJECT, - .errstr = "cannot pass map_type 24 into func bpf_map_lookup_elem", -}, -{ - "bpf_map_lookup_elem(xskmap, &key); xs->queue_id", - .insns = { - BPF_ST_MEM(BPF_W, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, offsetof(struct bpf_xdp_sock, queue_id)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_xskmap = { 3 }, - .prog_type = BPF_PROG_TYPE_XDP, - .result = ACCEPT, -}, -{ - "bpf_map_lookup_elem(sockmap, &key)", - .insns = { - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_sockmap = { 3 }, - .prog_type = BPF_PROG_TYPE_SK_SKB, - .result = REJECT, - .errstr = "Unreleased reference id=2 alloc_insn=5", -}, -{ - "bpf_map_lookup_elem(sockhash, &key)", - .insns = { - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_sockhash = { 3 }, - .prog_type = BPF_PROG_TYPE_SK_SKB, - .result = REJECT, - .errstr = "Unreleased reference id=2 alloc_insn=5", -}, -{ - "bpf_map_lookup_elem(sockmap, &key); sk->type [fullsock field]; bpf_sk_release(sk)", - .insns = { - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, offsetof(struct bpf_sock, type)), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .fixup_map_sockmap = { 3 }, - .prog_type = BPF_PROG_TYPE_SK_SKB, - .result = ACCEPT, -}, -{ - "bpf_map_lookup_elem(sockhash, &key); sk->type [fullsock field]; bpf_sk_release(sk)", - .insns = { - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, offsetof(struct bpf_sock, type)), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_EXIT_INSN(), - }, - .fixup_map_sockhash = { 3 }, - .prog_type = BPF_PROG_TYPE_SK_SKB, - .result = ACCEPT, -}, -{ - "bpf_sk_select_reuseport(ctx, reuseport_array, &key, flags)", - .insns = { - BPF_MOV64_IMM(BPF_REG_4, 0), - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_3, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, -4), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_EMIT_CALL(BPF_FUNC_sk_select_reuseport), - BPF_EXIT_INSN(), - }, - .fixup_map_reuseport_array = { 4 }, - .prog_type = BPF_PROG_TYPE_SK_REUSEPORT, - .result = ACCEPT, -}, -{ - "bpf_sk_select_reuseport(ctx, sockmap, &key, flags)", - .insns = { - BPF_MOV64_IMM(BPF_REG_4, 0), - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_3, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, -4), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_EMIT_CALL(BPF_FUNC_sk_select_reuseport), - BPF_EXIT_INSN(), - }, - .fixup_map_sockmap = { 4 }, - .prog_type = BPF_PROG_TYPE_SK_REUSEPORT, - .result = ACCEPT, -}, -{ - "bpf_sk_select_reuseport(ctx, sockhash, &key, flags)", - .insns = { - BPF_MOV64_IMM(BPF_REG_4, 0), - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_3, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, -4), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_EMIT_CALL(BPF_FUNC_sk_select_reuseport), - BPF_EXIT_INSN(), - }, - .fixup_map_sockmap = { 4 }, - .prog_type = BPF_PROG_TYPE_SK_REUSEPORT, - .result = ACCEPT, -}, -{ - "mark null check on return value of bpf_skc_to helpers", - .insns = { - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_EMIT_CALL(BPF_FUNC_skc_to_tcp_sock), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_EMIT_CALL(BPF_FUNC_skc_to_tcp_request_sock), - BPF_MOV64_REG(BPF_REG_8, BPF_REG_0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_8, 0, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_7, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = REJECT, - .errstr = "invalid mem access", - .result_unpriv = REJECT, - .errstr_unpriv = "unknown func", -}, diff --git a/tools/testing/selftests/bpf/verifier/spin_lock.c b/tools/testing/selftests/bpf/verifier/spin_lock.c deleted file mode 100644 index eaf114f07e2e..000000000000 --- a/tools/testing/selftests/bpf/verifier/spin_lock.c +++ /dev/null @@ -1,447 +0,0 @@ -{ - "spin_lock: test1 success", - .insns = { - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, - 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_lock), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_unlock), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_spin_lock = { 3 }, - .result = ACCEPT, - .result_unpriv = REJECT, - .errstr_unpriv = "", - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, -}, -{ - "spin_lock: test2 direct ld/st", - .insns = { - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, - 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_lock), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_unlock), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_spin_lock = { 3 }, - .result = REJECT, - .errstr = "cannot be accessed directly", - .result_unpriv = REJECT, - .errstr_unpriv = "", - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, -}, -{ - "spin_lock: test3 direct ld/st", - .insns = { - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, - 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_lock), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, 1), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_unlock), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_spin_lock = { 3 }, - .result = REJECT, - .errstr = "cannot be accessed directly", - .result_unpriv = REJECT, - .errstr_unpriv = "", - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "spin_lock: test4 direct ld/st", - .insns = { - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, - 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_lock), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_6, 3), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_unlock), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_spin_lock = { 3 }, - .result = REJECT, - .errstr = "cannot be accessed directly", - .result_unpriv = REJECT, - .errstr_unpriv = "", - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "spin_lock: test5 call within a locked region", - .insns = { - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, - 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_lock), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_unlock), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_spin_lock = { 3 }, - .result = REJECT, - .errstr = "calls are not allowed", - .result_unpriv = REJECT, - .errstr_unpriv = "", - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, -}, -{ - "spin_lock: test6 missing unlock", - .insns = { - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, - 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_lock), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, 0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_unlock), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_spin_lock = { 3 }, - .result = REJECT, - .errstr = "unlock is missing", - .result_unpriv = REJECT, - .errstr_unpriv = "", - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, -}, -{ - "spin_lock: test7 unlock without lock", - .insns = { - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, - 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 1), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_lock), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_unlock), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_spin_lock = { 3 }, - .result = REJECT, - .errstr = "without taking a lock", - .result_unpriv = REJECT, - .errstr_unpriv = "", - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, -}, -{ - "spin_lock: test8 double lock", - .insns = { - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, - 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_lock), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_lock), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_unlock), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_spin_lock = { 3 }, - .result = REJECT, - .errstr = "calls are not allowed", - .result_unpriv = REJECT, - .errstr_unpriv = "", - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, -}, -{ - "spin_lock: test9 different lock", - .insns = { - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, - 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, - 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_lock), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_unlock), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_spin_lock = { 3, 11 }, - .result = REJECT, - .errstr = "unlock of different lock", - .result_unpriv = REJECT, - .errstr_unpriv = "", - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, -}, -{ - "spin_lock: test10 lock in subprog without unlock", - .insns = { - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, - 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 5), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_unlock), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_lock), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_spin_lock = { 3 }, - .result = REJECT, - .errstr = "unlock is missing", - .result_unpriv = REJECT, - .errstr_unpriv = "", - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, -}, -{ - "spin_lock: test11 ld_abs under lock", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, - 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_lock), - BPF_LD_ABS(BPF_B, 0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_unlock), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_spin_lock = { 4 }, - .result = REJECT, - .errstr = "inside bpf_spin_lock", - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "spin_lock: regsafe compare reg->id for map value", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_6, offsetof(struct __sk_buff, mark)), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_MOV64_REG(BPF_REG_9, BPF_REG_1), - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_9), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_8, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_lock), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_6, 0, 1), - BPF_JMP_IMM(BPF_JA, 0, 0, 1), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_8), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_spin_unlock), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_spin_lock = { 2 }, - .result = REJECT, - .errstr = "bpf_spin_unlock of different lock", - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .flags = BPF_F_TEST_STATE_FREQ, -}, -/* Make sure that regsafe() compares ids for spin lock records using - * check_ids(): - * 1: r9 = map_lookup_elem(...) ; r9.id == 1 - * 2: r8 = map_lookup_elem(...) ; r8.id == 2 - * 3: r7 = ktime_get_ns() - * 4: r6 = ktime_get_ns() - * 5: if r6 > r7 goto <9> - * 6: spin_lock(r8) - * 7: r9 = r8 - * 8: goto <10> - * 9: spin_lock(r9) - * 10: spin_unlock(r9) ; r9.id == 1 || r9.id == 2 and lock is active, - * ; second visit to (10) should be considered safe - * ; if check_ids() is used. - * 11: exit(0) - */ -{ - "spin_lock: regsafe() check_ids() similar id mappings", - .insns = { - BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), - /* r9 = map_lookup_elem(...) */ - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, - 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 24), - BPF_MOV64_REG(BPF_REG_9, BPF_REG_0), - /* r8 = map_lookup_elem(...) */ - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), - BPF_LD_MAP_FD(BPF_REG_1, - 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 18), - BPF_MOV64_REG(BPF_REG_8, BPF_REG_0), - /* r7 = ktime_get_ns() */ - BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - /* r6 = ktime_get_ns() */ - BPF_EMIT_CALL(BPF_FUNC_ktime_get_ns), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - /* if r6 > r7 goto +5 ; no new information about the state is derived from - * ; this check, thus produced verifier states differ - * ; only in 'insn_idx' - * spin_lock(r8) - * r9 = r8 - * goto unlock - */ - BPF_JMP_REG(BPF_JGT, BPF_REG_6, BPF_REG_7, 5), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_8), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_EMIT_CALL(BPF_FUNC_spin_lock), - BPF_MOV64_REG(BPF_REG_9, BPF_REG_8), - BPF_JMP_A(3), - /* spin_lock(r9) */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_9), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_EMIT_CALL(BPF_FUNC_spin_lock), - /* spin_unlock(r9) */ - BPF_MOV64_REG(BPF_REG_1, BPF_REG_9), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4), - BPF_EMIT_CALL(BPF_FUNC_spin_unlock), - /* exit(0) */ - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_spin_lock = { 3, 10 }, - .result = VERBOSE_ACCEPT, - .errstr = "28: safe", - .result_unpriv = REJECT, - .errstr_unpriv = "", - .prog_type = BPF_PROG_TYPE_CGROUP_SKB, - .flags = BPF_F_TEST_STATE_FREQ, -}, diff --git a/tools/testing/selftests/bpf/verifier/subreg.c b/tools/testing/selftests/bpf/verifier/subreg.c deleted file mode 100644 index 4c4133c80440..000000000000 --- a/tools/testing/selftests/bpf/verifier/subreg.c +++ /dev/null @@ -1,533 +0,0 @@ -/* This file contains sub-register zero extension checks for insns defining - * sub-registers, meaning: - * - All insns under BPF_ALU class. Their BPF_ALU32 variants or narrow width - * forms (BPF_END) could define sub-registers. - * - Narrow direct loads, BPF_B/H/W | BPF_LDX. - * - BPF_LD is not exposed to JIT back-ends, so no need for testing. - * - * "get_prandom_u32" is used to initialize low 32-bit of some registers to - * prevent potential optimizations done by verifier or JIT back-ends which could - * optimize register back into constant when range info shows one register is a - * constant. - */ -{ - "add32 reg zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_LD_IMM64(BPF_REG_0, 0x100000000ULL), - BPF_ALU32_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "add32 imm zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - /* An insn could have no effect on the low 32-bit, for example: - * a = a + 0 - * a = a | 0 - * a = a & -1 - * But, they should still zero high 32-bit. - */ - BPF_ALU32_IMM(BPF_ADD, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_ADD, BPF_REG_0, -2), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_6), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "sub32 reg zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_LD_IMM64(BPF_REG_0, 0x1ffffffffULL), - BPF_ALU32_REG(BPF_SUB, BPF_REG_0, BPF_REG_1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "sub32 imm zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_SUB, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_SUB, BPF_REG_0, 1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_6), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "mul32 reg zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_LD_IMM64(BPF_REG_0, 0x100000001ULL), - BPF_ALU32_REG(BPF_MUL, BPF_REG_0, BPF_REG_1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "mul32 imm zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_MUL, BPF_REG_0, 1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_MUL, BPF_REG_0, -1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_6), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "div32 reg zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_MOV64_IMM(BPF_REG_0, -1), - BPF_ALU32_REG(BPF_DIV, BPF_REG_0, BPF_REG_1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "div32 imm zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_DIV, BPF_REG_0, 1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_DIV, BPF_REG_0, 2), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_6), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "or32 reg zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_LD_IMM64(BPF_REG_0, 0x100000001ULL), - BPF_ALU32_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "or32 imm zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_OR, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_OR, BPF_REG_0, 1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_6), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "and32 reg zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x100000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_1, BPF_REG_0), - BPF_LD_IMM64(BPF_REG_0, 0x1ffffffffULL), - BPF_ALU32_REG(BPF_AND, BPF_REG_0, BPF_REG_1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "and32 imm zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_AND, BPF_REG_0, -1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_AND, BPF_REG_0, -2), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_6), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "lsh32 reg zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x100000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_MOV64_IMM(BPF_REG_1, 1), - BPF_ALU32_REG(BPF_LSH, BPF_REG_0, BPF_REG_1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "lsh32 imm zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_LSH, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_LSH, BPF_REG_0, 1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_6), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "rsh32 reg zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_MOV64_IMM(BPF_REG_1, 1), - BPF_ALU32_REG(BPF_RSH, BPF_REG_0, BPF_REG_1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "rsh32 imm zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_RSH, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_RSH, BPF_REG_0, 1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_6), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "neg32 reg zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_NEG, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "mod32 reg zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_MOV64_IMM(BPF_REG_0, -1), - BPF_ALU32_REG(BPF_MOD, BPF_REG_0, BPF_REG_1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "mod32 imm zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_MOD, BPF_REG_0, 1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_MOD, BPF_REG_0, 2), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_6), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "xor32 reg zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), - BPF_LD_IMM64(BPF_REG_0, 0x100000000ULL), - BPF_ALU32_REG(BPF_XOR, BPF_REG_0, BPF_REG_1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "xor32 imm zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_XOR, BPF_REG_0, 1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "mov32 reg zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x100000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_1, BPF_REG_0), - BPF_LD_IMM64(BPF_REG_0, 0x100000000ULL), - BPF_MOV32_REG(BPF_REG_0, BPF_REG_1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "mov32 imm zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_MOV32_IMM(BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_MOV32_IMM(BPF_REG_0, 1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_6), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "arsh32 reg zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_MOV64_IMM(BPF_REG_1, 1), - BPF_ALU32_REG(BPF_ARSH, BPF_REG_0, BPF_REG_1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "arsh32 imm zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_ARSH, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_ALU32_IMM(BPF_ARSH, BPF_REG_0, 1), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_6), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "end16 (to_le) reg zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_ALU64_IMM(BPF_LSH, BPF_REG_6, 32), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_6), - BPF_ENDIAN(BPF_TO_LE, BPF_REG_0, 16), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "end32 (to_le) reg zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_ALU64_IMM(BPF_LSH, BPF_REG_6, 32), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_6), - BPF_ENDIAN(BPF_TO_LE, BPF_REG_0, 32), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "end16 (to_be) reg zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_ALU64_IMM(BPF_LSH, BPF_REG_6, 32), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_6), - BPF_ENDIAN(BPF_TO_BE, BPF_REG_0, 16), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "end32 (to_be) reg zero extend check", - .insns = { - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_ALU64_IMM(BPF_LSH, BPF_REG_6, 32), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_6), - BPF_ENDIAN(BPF_TO_BE, BPF_REG_0, 32), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "ldx_b zero extend check", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -4), - BPF_ST_MEM(BPF_W, BPF_REG_6, 0, 0xfaceb00c), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_6, 0), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "ldx_h zero extend check", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -4), - BPF_ST_MEM(BPF_W, BPF_REG_6, 0, 0xfaceb00c), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_6, 0), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, -{ - "ldx_w zero extend check", - .insns = { - BPF_MOV64_REG(BPF_REG_6, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -4), - BPF_ST_MEM(BPF_W, BPF_REG_6, 0, 0xfaceb00c), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), - BPF_LD_IMM64(BPF_REG_1, 0x1000000000ULL), - BPF_ALU64_REG(BPF_OR, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, 0), - BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = 0, -}, diff --git a/tools/testing/selftests/bpf/verifier/unpriv.c b/tools/testing/selftests/bpf/verifier/unpriv.c deleted file mode 100644 index af0c0f336625..000000000000 --- a/tools/testing/selftests/bpf/verifier/unpriv.c +++ /dev/null @@ -1,562 +0,0 @@ -{ - "unpriv: return pointer", - .insns = { - BPF_MOV64_REG(BPF_REG_0, BPF_REG_10), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .result_unpriv = REJECT, - .errstr_unpriv = "R0 leaks addr", - .retval = POINTER_VALUE, -}, -{ - "unpriv: add const to pointer", - .insns = { - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 8), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, -}, -{ - "unpriv: add pointer to pointer", - .insns = { - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_10), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = REJECT, - .errstr = "R1 pointer += pointer", -}, -{ - "unpriv: neg pointer", - .insns = { - BPF_ALU64_IMM(BPF_NEG, BPF_REG_1, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .result_unpriv = REJECT, - .errstr_unpriv = "R1 pointer arithmetic", -}, -{ - "unpriv: cmp pointer with const", - .insns = { - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .result_unpriv = REJECT, - .errstr_unpriv = "R1 pointer comparison", -}, -{ - "unpriv: cmp pointer with pointer", - .insns = { - BPF_JMP_REG(BPF_JEQ, BPF_REG_1, BPF_REG_10, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .result_unpriv = REJECT, - .errstr_unpriv = "R10 pointer comparison", -}, -{ - "unpriv: check that printk is disallowed", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, -8), - BPF_MOV64_IMM(BPF_REG_2, 8), - BPF_MOV64_REG(BPF_REG_3, BPF_REG_1), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_trace_printk), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr_unpriv = "unknown func bpf_trace_printk#6", - .result_unpriv = REJECT, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_TRACEPOINT, -}, -{ - "unpriv: pass pointer to helper function", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_MOV64_REG(BPF_REG_3, BPF_REG_2), - BPF_MOV64_REG(BPF_REG_4, BPF_REG_2), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_update_elem), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr_unpriv = "R4 leaks addr", - .result_unpriv = REJECT, - .result = ACCEPT, -}, -{ - "unpriv: indirectly pass pointer on stack to helper function", - .insns = { - BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_10, -8), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr_unpriv = "invalid indirect read from stack R2 off -8+0 size 8", - .result_unpriv = REJECT, - .result = ACCEPT, -}, -{ - "unpriv: mangle pointer on stack 1", - .insns = { - BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_10, -8), - BPF_ST_MEM(BPF_W, BPF_REG_10, -8, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr_unpriv = "attempt to corrupt spilled", - .result_unpriv = REJECT, - .result = ACCEPT, -}, -{ - "unpriv: mangle pointer on stack 2", - .insns = { - BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_10, -8), - BPF_ST_MEM(BPF_B, BPF_REG_10, -1, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr_unpriv = "attempt to corrupt spilled", - .result_unpriv = REJECT, - .result = ACCEPT, -}, -{ - "unpriv: read pointer from stack in small chunks", - .insns = { - BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_10, -8), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_10, -8), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr = "invalid size", - .result = REJECT, -}, -{ - "unpriv: write pointer into ctx", - .insns = { - BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr_unpriv = "R1 leaks addr", - .result_unpriv = REJECT, - .errstr = "invalid bpf_context access", - .result = REJECT, -}, -{ - "unpriv: spill/fill of ctx", - .insns = { - BPF_ALU64_REG(BPF_MOV, BPF_REG_6, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -8), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0), - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_6, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, -}, -{ - "unpriv: spill/fill of ctx 2", - .insns = { - BPF_ALU64_REG(BPF_MOV, BPF_REG_6, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -8), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0), - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_6, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_hash_recalc), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "unpriv: spill/fill of ctx 3", - .insns = { - BPF_ALU64_REG(BPF_MOV, BPF_REG_6, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -8), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_10, 0), - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_6, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_hash_recalc), - BPF_EXIT_INSN(), - }, - .result = REJECT, - .errstr = "R1 type=fp expected=ctx", - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "unpriv: spill/fill of ctx 4", - .insns = { - BPF_ALU64_REG(BPF_MOV, BPF_REG_6, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -8), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_RAW_INSN(BPF_STX | BPF_ATOMIC | BPF_DW, - BPF_REG_10, BPF_REG_0, -8, BPF_ADD), - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_6, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_hash_recalc), - BPF_EXIT_INSN(), - }, - .result = REJECT, - .errstr = "R1 type=scalar expected=ctx", - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "unpriv: spill/fill of different pointers stx", - .insns = { - BPF_MOV64_IMM(BPF_REG_3, 42), - BPF_ALU64_REG(BPF_MOV, BPF_REG_6, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -8), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 3), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -16), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_2, 0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 1), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0), - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_6, 0), - BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_3, - offsetof(struct __sk_buff, mark)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = REJECT, - .errstr = "same insn cannot be used with different pointers", - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - /* Same as above, but use BPF_ST_MEM to save 42 - * instead of BPF_STX_MEM. - */ - "unpriv: spill/fill of different pointers st", - .insns = { - BPF_ALU64_REG(BPF_MOV, BPF_REG_6, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -8), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 3), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -16), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_2, 0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 1), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0), - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_6, 0), - BPF_ST_MEM(BPF_W, BPF_REG_1, offsetof(struct __sk_buff, mark), 42), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = REJECT, - .errstr = "same insn cannot be used with different pointers", - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "unpriv: spill/fill of different pointers stx - ctx and sock", - .insns = { - BPF_MOV64_REG(BPF_REG_8, BPF_REG_1), - /* struct bpf_sock *sock = bpf_sock_lookup(...); */ - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - /* u64 foo; */ - /* void *target = &foo; */ - BPF_ALU64_REG(BPF_MOV, BPF_REG_6, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -8), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_8), - /* if (skb == NULL) *target = sock; */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 1), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_2, 0), - /* else *target = skb; */ - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 1), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0), - /* struct __sk_buff *skb = *target; */ - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_6, 0), - /* skb->mark = 42; */ - BPF_MOV64_IMM(BPF_REG_3, 42), - BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_3, - offsetof(struct __sk_buff, mark)), - /* if (sk) bpf_sk_release(sk) */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 1), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = REJECT, - .errstr = "type=ctx expected=sock", - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "unpriv: spill/fill of different pointers stx - leak sock", - .insns = { - BPF_MOV64_REG(BPF_REG_8, BPF_REG_1), - /* struct bpf_sock *sock = bpf_sock_lookup(...); */ - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - /* u64 foo; */ - /* void *target = &foo; */ - BPF_ALU64_REG(BPF_MOV, BPF_REG_6, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -8), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_8), - /* if (skb == NULL) *target = sock; */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 1), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_2, 0), - /* else *target = skb; */ - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 1), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0), - /* struct __sk_buff *skb = *target; */ - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_6, 0), - /* skb->mark = 42; */ - BPF_MOV64_IMM(BPF_REG_3, 42), - BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_3, - offsetof(struct __sk_buff, mark)), - BPF_EXIT_INSN(), - }, - .result = REJECT, - //.errstr = "same insn cannot be used with different pointers", - .errstr = "Unreleased reference", - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "unpriv: spill/fill of different pointers stx - sock and ctx (read)", - .insns = { - BPF_MOV64_REG(BPF_REG_8, BPF_REG_1), - /* struct bpf_sock *sock = bpf_sock_lookup(...); */ - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - /* u64 foo; */ - /* void *target = &foo; */ - BPF_ALU64_REG(BPF_MOV, BPF_REG_6, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -8), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_8), - /* if (skb) *target = skb */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 1), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0), - /* else *target = sock */ - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 1), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_2, 0), - /* struct bpf_sock *sk = *target; */ - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_6, 0), - /* if (sk) u32 foo = sk->mark; bpf_sk_release(sk); */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 2), - BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, - offsetof(struct bpf_sock, mark)), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = REJECT, - .errstr = "same insn cannot be used with different pointers", - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "unpriv: spill/fill of different pointers stx - sock and ctx (write)", - .insns = { - BPF_MOV64_REG(BPF_REG_8, BPF_REG_1), - /* struct bpf_sock *sock = bpf_sock_lookup(...); */ - BPF_SK_LOOKUP(sk_lookup_tcp), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - /* u64 foo; */ - /* void *target = &foo; */ - BPF_ALU64_REG(BPF_MOV, BPF_REG_6, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -8), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_8), - /* if (skb) *target = skb */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 1), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0), - /* else *target = sock */ - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 1), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_2, 0), - /* struct bpf_sock *sk = *target; */ - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_6, 0), - /* if (sk) sk->mark = 42; bpf_sk_release(sk); */ - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 3), - BPF_MOV64_IMM(BPF_REG_3, 42), - BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_3, - offsetof(struct bpf_sock, mark)), - BPF_EMIT_CALL(BPF_FUNC_sk_release), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = REJECT, - //.errstr = "same insn cannot be used with different pointers", - .errstr = "cannot write into sock", - .prog_type = BPF_PROG_TYPE_SCHED_CLS, -}, -{ - "unpriv: spill/fill of different pointers ldx", - .insns = { - BPF_ALU64_REG(BPF_MOV, BPF_REG_6, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -8), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 3), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, - -(__s32)offsetof(struct bpf_perf_event_data, - sample_period) - 8), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_2, 0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 1), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0), - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_6, 0), - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, - offsetof(struct bpf_perf_event_data, sample_period)), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .result = REJECT, - .errstr = "same insn cannot be used with different pointers", - .prog_type = BPF_PROG_TYPE_PERF_EVENT, -}, -{ - "unpriv: write pointer into map elem value", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), - BPF_STX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 3 }, - .errstr_unpriv = "R0 leaks addr", - .result_unpriv = REJECT, - .result = ACCEPT, -}, -{ - "alu32: mov u32 const", - .insns = { - BPF_MOV32_IMM(BPF_REG_7, 0), - BPF_ALU32_IMM(BPF_AND, BPF_REG_7, 1), - BPF_MOV32_REG(BPF_REG_0, BPF_REG_7), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_7, 0), - BPF_EXIT_INSN(), - }, - .errstr_unpriv = "R7 invalid mem access 'scalar'", - .result_unpriv = REJECT, - .result = ACCEPT, - .retval = 0, -}, -{ - "unpriv: partial copy of pointer", - .insns = { - BPF_MOV32_REG(BPF_REG_1, BPF_REG_10), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr_unpriv = "R10 partial copy", - .result_unpriv = REJECT, - .result = ACCEPT, -}, -{ - "unpriv: pass pointer to tail_call", - .insns = { - BPF_MOV64_REG(BPF_REG_3, BPF_REG_1), - BPF_LD_MAP_FD(BPF_REG_2, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_tail_call), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_prog1 = { 1 }, - .errstr_unpriv = "R3 leaks addr into helper", - .result_unpriv = REJECT, - .result = ACCEPT, -}, -{ - "unpriv: cmp map pointer with zero", - .insns = { - BPF_MOV64_IMM(BPF_REG_1, 0), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_8b = { 1 }, - .errstr_unpriv = "R1 pointer comparison", - .result_unpriv = REJECT, - .result = ACCEPT, -}, -{ - "unpriv: write into frame pointer", - .insns = { - BPF_MOV64_REG(BPF_REG_10, BPF_REG_1), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr = "frame pointer is read only", - .result = REJECT, -}, -{ - "unpriv: spill/fill frame pointer", - .insns = { - BPF_ALU64_REG(BPF_MOV, BPF_REG_6, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -8), - BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_10, 0), - BPF_LDX_MEM(BPF_DW, BPF_REG_10, BPF_REG_6, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr = "frame pointer is read only", - .result = REJECT, -}, -{ - "unpriv: cmp of frame pointer", - .insns = { - BPF_JMP_IMM(BPF_JEQ, BPF_REG_10, 0, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr_unpriv = "R10 pointer comparison", - .result_unpriv = REJECT, - .result = ACCEPT, -}, -{ - "unpriv: adding of fp, reg", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_1, 0), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_10), - BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, -8), - BPF_EXIT_INSN(), - }, - .errstr_unpriv = "R1 stack pointer arithmetic goes out of range", - .result_unpriv = REJECT, - .result = ACCEPT, -}, -{ - "unpriv: adding of fp, imm", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 0), - BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, -8), - BPF_EXIT_INSN(), - }, - .errstr_unpriv = "R1 stack pointer arithmetic goes out of range", - .result_unpriv = REJECT, - .result = ACCEPT, -}, -{ - "unpriv: cmp of stack pointer", - .insns = { - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_2, 0, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .errstr_unpriv = "R2 pointer comparison", - .result_unpriv = REJECT, - .result = ACCEPT, -}, diff --git a/tools/testing/selftests/bpf/verifier/value_illegal_alu.c b/tools/testing/selftests/bpf/verifier/value_illegal_alu.c deleted file mode 100644 index d6f29eb4bd57..000000000000 --- a/tools/testing/selftests/bpf/verifier/value_illegal_alu.c +++ /dev/null @@ -1,95 +0,0 @@ -{ - "map element value illegal alu op, 1", - .insns = { - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_ST_MEM(BPF_DW, BPF_REG_2, 0, 0), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), - BPF_ALU64_IMM(BPF_AND, BPF_REG_0, 8), - BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 22), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_48b = { 3 }, - .errstr = "R0 bitwise operator &= on pointer", - .result = REJECT, -}, -{ - "map element value illegal alu op, 2", - .insns = { - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_ST_MEM(BPF_DW, BPF_REG_2, 0, 0), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), - BPF_ALU32_IMM(BPF_ADD, BPF_REG_0, 0), - BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 22), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_48b = { 3 }, - .errstr = "R0 32-bit pointer arithmetic prohibited", - .result = REJECT, -}, -{ - "map element value illegal alu op, 3", - .insns = { - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_ST_MEM(BPF_DW, BPF_REG_2, 0, 0), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), - BPF_ALU64_IMM(BPF_DIV, BPF_REG_0, 42), - BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 22), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_48b = { 3 }, - .errstr = "R0 pointer arithmetic with /= operator", - .result = REJECT, -}, -{ - "map element value illegal alu op, 4", - .insns = { - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_ST_MEM(BPF_DW, BPF_REG_2, 0, 0), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), - BPF_ENDIAN(BPF_FROM_BE, BPF_REG_0, 64), - BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 22), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_48b = { 3 }, - .errstr_unpriv = "R0 pointer arithmetic prohibited", - .errstr = "invalid mem access 'scalar'", - .result = REJECT, - .result_unpriv = REJECT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "map element value illegal alu op, 5", - .insns = { - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_ST_MEM(BPF_DW, BPF_REG_2, 0, 0), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 7), - BPF_MOV64_IMM(BPF_REG_3, 4096), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_STX_MEM(BPF_DW, BPF_REG_2, BPF_REG_0, 0), - BPF_ATOMIC_OP(BPF_DW, BPF_ADD, BPF_REG_2, BPF_REG_3, 0), - BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0), - BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 22), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_48b = { 3 }, - .errstr_unpriv = "leaking pointer from stack off -8", - .errstr = "R0 invalid mem access 'scalar'", - .result = REJECT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, diff --git a/tools/testing/selftests/bpf/verifier/value_ptr_arith.c b/tools/testing/selftests/bpf/verifier/value_ptr_arith.c deleted file mode 100644 index 249187d3c530..000000000000 --- a/tools/testing/selftests/bpf/verifier/value_ptr_arith.c +++ /dev/null @@ -1,1140 +0,0 @@ -{ - "map access: known scalar += value_ptr unknown vs const", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, len)), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 1, 3), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 1, 2), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 9), - BPF_LDX_MEM(BPF_B, BPF_REG_4, BPF_REG_0, 0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_4, 1, 4), - BPF_MOV64_IMM(BPF_REG_1, 6), - BPF_ALU64_IMM(BPF_NEG, BPF_REG_1, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 0x7), - BPF_JMP_IMM(BPF_JA, 0, 0, 1), - BPF_MOV64_IMM(BPF_REG_1, 3), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_16b = { 5 }, - .fixup_map_array_48b = { 8 }, - .result_unpriv = REJECT, - .errstr_unpriv = "R1 tried to add from different maps, paths or scalars", - .result = ACCEPT, - .retval = 1, -}, -{ - "map access: known scalar += value_ptr const vs unknown", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, len)), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 1, 3), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 1, 2), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 9), - BPF_LDX_MEM(BPF_B, BPF_REG_4, BPF_REG_0, 0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_4, 1, 2), - BPF_MOV64_IMM(BPF_REG_1, 3), - BPF_JMP_IMM(BPF_JA, 0, 0, 3), - BPF_MOV64_IMM(BPF_REG_1, 6), - BPF_ALU64_IMM(BPF_NEG, BPF_REG_1, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 0x7), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_16b = { 5 }, - .fixup_map_array_48b = { 8 }, - .result_unpriv = REJECT, - .errstr_unpriv = "R1 tried to add from different maps, paths or scalars", - .result = ACCEPT, - .retval = 1, -}, -{ - "map access: known scalar += value_ptr const vs const (ne)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, len)), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 1, 3), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 1, 2), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 7), - BPF_LDX_MEM(BPF_B, BPF_REG_4, BPF_REG_0, 0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_4, 1, 2), - BPF_MOV64_IMM(BPF_REG_1, 3), - BPF_JMP_IMM(BPF_JA, 0, 0, 1), - BPF_MOV64_IMM(BPF_REG_1, 5), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_16b = { 5 }, - .fixup_map_array_48b = { 8 }, - .result_unpriv = REJECT, - .errstr_unpriv = "R1 tried to add from different maps, paths or scalars", - .result = ACCEPT, - .retval = 1, -}, -{ - "map access: known scalar += value_ptr const vs const (eq)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, len)), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 1, 3), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 1, 2), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 7), - BPF_LDX_MEM(BPF_B, BPF_REG_4, BPF_REG_0, 0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_4, 1, 2), - BPF_MOV64_IMM(BPF_REG_1, 5), - BPF_JMP_IMM(BPF_JA, 0, 0, 1), - BPF_MOV64_IMM(BPF_REG_1, 5), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_16b = { 5 }, - .fixup_map_array_48b = { 8 }, - .result = ACCEPT, - .retval = 1, -}, -{ - "map access: known scalar += value_ptr unknown vs unknown (eq)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, len)), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 1, 3), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 1, 2), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 11), - BPF_LDX_MEM(BPF_B, BPF_REG_4, BPF_REG_0, 0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_4, 1, 4), - BPF_MOV64_IMM(BPF_REG_1, 6), - BPF_ALU64_IMM(BPF_NEG, BPF_REG_1, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 0x7), - BPF_JMP_IMM(BPF_JA, 0, 0, 3), - BPF_MOV64_IMM(BPF_REG_1, 6), - BPF_ALU64_IMM(BPF_NEG, BPF_REG_1, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 0x7), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_16b = { 5 }, - .fixup_map_array_48b = { 8 }, - .result = ACCEPT, - .retval = 1, -}, -{ - "map access: known scalar += value_ptr unknown vs unknown (lt)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, len)), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 1, 3), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 1, 2), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 11), - BPF_LDX_MEM(BPF_B, BPF_REG_4, BPF_REG_0, 0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_4, 1, 4), - BPF_MOV64_IMM(BPF_REG_1, 6), - BPF_ALU64_IMM(BPF_NEG, BPF_REG_1, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 0x3), - BPF_JMP_IMM(BPF_JA, 0, 0, 3), - BPF_MOV64_IMM(BPF_REG_1, 6), - BPF_ALU64_IMM(BPF_NEG, BPF_REG_1, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 0x7), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_16b = { 5 }, - .fixup_map_array_48b = { 8 }, - .result_unpriv = REJECT, - .errstr_unpriv = "R1 tried to add from different maps, paths or scalars", - .result = ACCEPT, - .retval = 1, -}, -{ - "map access: known scalar += value_ptr unknown vs unknown (gt)", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, len)), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 1, 3), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 1, 2), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 11), - BPF_LDX_MEM(BPF_B, BPF_REG_4, BPF_REG_0, 0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_4, 1, 4), - BPF_MOV64_IMM(BPF_REG_1, 6), - BPF_ALU64_IMM(BPF_NEG, BPF_REG_1, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 0x7), - BPF_JMP_IMM(BPF_JA, 0, 0, 3), - BPF_MOV64_IMM(BPF_REG_1, 6), - BPF_ALU64_IMM(BPF_NEG, BPF_REG_1, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 0x3), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_16b = { 5 }, - .fixup_map_array_48b = { 8 }, - .result_unpriv = REJECT, - .errstr_unpriv = "R1 tried to add from different maps, paths or scalars", - .result = ACCEPT, - .retval = 1, -}, -{ - "map access: known scalar += value_ptr from different maps", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, len)), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 1, 3), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 1, 2), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3), - BPF_MOV64_IMM(BPF_REG_1, 4), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_16b = { 5 }, - .fixup_map_array_48b = { 8 }, - .result = ACCEPT, - .retval = 1, -}, -{ - "map access: value_ptr -= known scalar from different maps", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, len)), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 1, 3), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 1, 2), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4), - BPF_MOV64_IMM(BPF_REG_1, 4), - BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_1), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_16b = { 5 }, - .fixup_map_array_48b = { 8 }, - .result = ACCEPT, - .result_unpriv = REJECT, - .errstr_unpriv = "R0 min value is outside of the allowed memory range", - .retval = 1, -}, -{ - "map access: known scalar += value_ptr from different maps, but same value properties", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, - offsetof(struct __sk_buff, len)), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 1, 3), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 1, 2), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3), - BPF_MOV64_IMM(BPF_REG_1, 4), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_hash_48b = { 5 }, - .fixup_map_array_48b = { 8 }, - .result = ACCEPT, - .retval = 1, -}, -{ - "map access: mixing value pointer and scalar, 1", - .insns = { - // load map value pointer into r0 and r2 - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_LD_MAP_FD(BPF_REG_ARG1, 0), - BPF_MOV64_REG(BPF_REG_ARG2, BPF_REG_FP), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_ARG2, -16), - BPF_ST_MEM(BPF_DW, BPF_REG_FP, -16, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - // load some number from the map into r1 - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - // depending on r1, branch: - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 3), - // branch A - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - BPF_MOV64_IMM(BPF_REG_3, 0), - BPF_JMP_A(2), - // branch B - BPF_MOV64_IMM(BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_3, 0x100000), - // common instruction - BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_3), - // depending on r1, branch: - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 1), - // branch A - BPF_JMP_A(4), - // branch B - BPF_MOV64_IMM(BPF_REG_0, 0x13371337), - // verifier follows fall-through - BPF_JMP_IMM(BPF_JNE, BPF_REG_2, 0x100000, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - // fake-dead code; targeted from branch A to - // prevent dead code sanitization - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 1 }, - .result = ACCEPT, - .result_unpriv = REJECT, - .errstr_unpriv = "R2 pointer comparison prohibited", - .retval = 0, -}, -{ - "map access: mixing value pointer and scalar, 2", - .insns = { - // load map value pointer into r0 and r2 - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_LD_MAP_FD(BPF_REG_ARG1, 0), - BPF_MOV64_REG(BPF_REG_ARG2, BPF_REG_FP), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_ARG2, -16), - BPF_ST_MEM(BPF_DW, BPF_REG_FP, -16, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - // load some number from the map into r1 - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - // depending on r1, branch: - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 3), - // branch A - BPF_MOV64_IMM(BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_3, 0x100000), - BPF_JMP_A(2), - // branch B - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - BPF_MOV64_IMM(BPF_REG_3, 0), - // common instruction - BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_3), - // depending on r1, branch: - BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 1), - // branch A - BPF_JMP_A(4), - // branch B - BPF_MOV64_IMM(BPF_REG_0, 0x13371337), - // verifier follows fall-through - BPF_JMP_IMM(BPF_JNE, BPF_REG_2, 0x100000, 2), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - // fake-dead code; targeted from branch A to - // prevent dead code sanitization, rejected - // via branch B however - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 1 }, - .result = ACCEPT, - .result_unpriv = REJECT, - .errstr_unpriv = "R0 invalid mem access 'scalar'", - .retval = 0, -}, -{ - "sanitation: alu with different scalars 1", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_LD_MAP_FD(BPF_REG_ARG1, 0), - BPF_MOV64_REG(BPF_REG_ARG2, BPF_REG_FP), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_ARG2, -16), - BPF_ST_MEM(BPF_DW, BPF_REG_FP, -16, 0), - BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 3), - BPF_MOV64_IMM(BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_3, 0x100000), - BPF_JMP_A(2), - BPF_MOV64_IMM(BPF_REG_2, 42), - BPF_MOV64_IMM(BPF_REG_3, 0x100001), - BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_3), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 1 }, - .result = ACCEPT, - .retval = 0x100000, -}, -{ - "sanitation: alu with different scalars 2", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_FP), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -16), - BPF_ST_MEM(BPF_DW, BPF_REG_FP, -16, 0), - BPF_EMIT_CALL(BPF_FUNC_map_delete_elem), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_FP), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -16), - BPF_EMIT_CALL(BPF_FUNC_map_delete_elem), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_8, BPF_REG_6), - BPF_ALU64_REG(BPF_ADD, BPF_REG_8, BPF_REG_7), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_8), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 1 }, - .result = ACCEPT, - .retval = -EINVAL * 2, -}, -{ - "sanitation: alu with different scalars 3", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, EINVAL), - BPF_ALU64_IMM(BPF_MUL, BPF_REG_0, -1), - BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), - BPF_MOV64_IMM(BPF_REG_0, EINVAL), - BPF_ALU64_IMM(BPF_MUL, BPF_REG_0, -1), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), - BPF_MOV64_REG(BPF_REG_8, BPF_REG_6), - BPF_ALU64_REG(BPF_ADD, BPF_REG_8, BPF_REG_7), - BPF_MOV64_REG(BPF_REG_0, BPF_REG_8), - BPF_EXIT_INSN(), - }, - .result = ACCEPT, - .retval = -EINVAL * 2, -}, -{ - "map access: value_ptr += known scalar, upper oob arith, test 1", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4), - BPF_MOV64_IMM(BPF_REG_1, 48), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .result_unpriv = REJECT, - .errstr_unpriv = "R0 pointer arithmetic of map value goes out of range", - .retval = 1, -}, -{ - "map access: value_ptr += known scalar, upper oob arith, test 2", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4), - BPF_MOV64_IMM(BPF_REG_1, 49), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .result_unpriv = REJECT, - .errstr_unpriv = "R0 pointer arithmetic of map value goes out of range", - .retval = 1, -}, -{ - "map access: value_ptr += known scalar, upper oob arith, test 3", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4), - BPF_MOV64_IMM(BPF_REG_1, 47), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .retval = 1, -}, -{ - "map access: value_ptr -= known scalar, lower oob arith, test 1", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 5), - BPF_MOV64_IMM(BPF_REG_1, 47), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_MOV64_IMM(BPF_REG_1, 48), - BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = REJECT, - .errstr = "R0 min value is outside of the allowed memory range", - .result_unpriv = REJECT, - .errstr_unpriv = "R0 pointer arithmetic of map value goes out of range", -}, -{ - "map access: value_ptr -= known scalar, lower oob arith, test 2", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 7), - BPF_MOV64_IMM(BPF_REG_1, 47), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_MOV64_IMM(BPF_REG_1, 48), - BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_1), - BPF_MOV64_IMM(BPF_REG_1, 1), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .result_unpriv = REJECT, - .errstr_unpriv = "R0 pointer arithmetic of map value goes out of range", - .retval = 1, -}, -{ - "map access: value_ptr -= known scalar, lower oob arith, test 3", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 5), - BPF_MOV64_IMM(BPF_REG_1, 47), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_MOV64_IMM(BPF_REG_1, 47), - BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .retval = 1, -}, -{ - "map access: known scalar += value_ptr", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3), - BPF_MOV64_IMM(BPF_REG_1, 4), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .retval = 1, -}, -{ - "map access: value_ptr += known scalar, 1", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3), - BPF_MOV64_IMM(BPF_REG_1, 4), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .retval = 1, -}, -{ - "map access: value_ptr += known scalar, 2", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3), - BPF_MOV64_IMM(BPF_REG_1, 49), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = REJECT, - .errstr = "invalid access to map value", -}, -{ - "map access: value_ptr += known scalar, 3", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3), - BPF_MOV64_IMM(BPF_REG_1, -1), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = REJECT, - .errstr = "invalid access to map value", -}, -{ - "map access: value_ptr += known scalar, 4", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 7), - BPF_MOV64_IMM(BPF_REG_1, 5), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_MOV64_IMM(BPF_REG_1, -2), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_MOV64_IMM(BPF_REG_1, -1), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .retval = 1, -}, -{ - "map access: value_ptr += known scalar, 5", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3), - BPF_MOV64_IMM(BPF_REG_1, (6 + 1) * sizeof(int)), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .retval = 0xabcdef12, -}, -{ - "map access: value_ptr += known scalar, 6", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 5), - BPF_MOV64_IMM(BPF_REG_1, (3 + 1) * sizeof(int)), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_MOV64_IMM(BPF_REG_1, 3 * sizeof(int)), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .retval = 0xabcdef12, -}, -{ - "map access: value_ptr += N, value_ptr -= N known scalar", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 6), - BPF_MOV32_IMM(BPF_REG_1, 0x12345678), - BPF_STX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 2), - BPF_MOV64_IMM(BPF_REG_1, 2), - BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .retval = 0x12345678, -}, -{ - "map access: unknown scalar += value_ptr, 1", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 0xf), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .retval = 1, -}, -{ - "map access: unknown scalar += value_ptr, 2", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4), - BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 31), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .retval = 0xabcdef12, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "map access: unknown scalar += value_ptr, 3", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 8), - BPF_MOV64_IMM(BPF_REG_1, -1), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_MOV64_IMM(BPF_REG_1, 1), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 31), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .result_unpriv = REJECT, - .errstr_unpriv = "R0 pointer arithmetic of map value goes out of range", - .retval = 0xabcdef12, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "map access: unknown scalar += value_ptr, 4", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 6), - BPF_MOV64_IMM(BPF_REG_1, 19), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 31), - BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = REJECT, - .errstr = "R1 max value is outside of the allowed memory range", - .errstr_unpriv = "R1 pointer arithmetic of map value goes out of range", - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "map access: value_ptr += unknown scalar, 1", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 0xf), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .retval = 1, -}, -{ - "map access: value_ptr += unknown scalar, 2", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4), - BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 31), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .retval = 0xabcdef12, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "map access: value_ptr += unknown scalar, 3", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 11), - BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, 0), - BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_0, 8), - BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_0, 16), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 0xf), - BPF_ALU64_IMM(BPF_AND, BPF_REG_3, 1), - BPF_ALU64_IMM(BPF_OR, BPF_REG_3, 1), - BPF_JMP_REG(BPF_JGT, BPF_REG_2, BPF_REG_3, 4), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_3), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_IMM(BPF_REG_0, 2), - BPF_JMP_IMM(BPF_JA, 0, 0, -3), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .retval = 1, -}, -{ - "map access: value_ptr += value_ptr", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_0), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = REJECT, - .errstr = "R0 pointer += pointer prohibited", -}, -{ - "map access: known scalar -= value_ptr", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3), - BPF_MOV64_IMM(BPF_REG_1, 4), - BPF_ALU64_REG(BPF_SUB, BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = REJECT, - .errstr = "R1 tried to subtract pointer from scalar", -}, -{ - "map access: value_ptr -= known scalar", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3), - BPF_MOV64_IMM(BPF_REG_1, 4), - BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = REJECT, - .errstr = "R0 min value is outside of the allowed memory range", -}, -{ - "map access: value_ptr -= known scalar, 2", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 5), - BPF_MOV64_IMM(BPF_REG_1, 6), - BPF_MOV64_IMM(BPF_REG_2, 4), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_2), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .retval = 1, -}, -{ - "map access: unknown scalar -= value_ptr", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 0xf), - BPF_ALU64_REG(BPF_SUB, BPF_REG_1, BPF_REG_0), - BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = REJECT, - .errstr = "R1 tried to subtract pointer from scalar", -}, -{ - "map access: value_ptr -= unknown scalar", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 0xf), - BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = REJECT, - .errstr = "R0 min value is negative", -}, -{ - "map access: value_ptr -= unknown scalar, 2", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 8), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 0xf), - BPF_ALU64_IMM(BPF_OR, BPF_REG_1, 0x7), - BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 0x7), - BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_1), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = ACCEPT, - .result_unpriv = REJECT, - .errstr_unpriv = "R0 pointer arithmetic of map value goes out of range", - .retval = 1, -}, -{ - "map access: value_ptr -= value_ptr", - .insns = { - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), - BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_0), - BPF_LDX_MEM(BPF_B, BPF_REG_1, BPF_REG_0, 0), - BPF_MOV64_IMM(BPF_REG_0, 1), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 3 }, - .result = REJECT, - .errstr = "R0 invalid mem access 'scalar'", - .errstr_unpriv = "R0 pointer -= pointer prohibited", -}, -{ - "map access: trying to leak tainted dst reg", - .insns = { - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), - BPF_LD_MAP_FD(BPF_REG_1, 0), - BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), - BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), - BPF_EXIT_INSN(), - BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), - BPF_MOV32_IMM(BPF_REG_1, 0xFFFFFFFF), - BPF_MOV32_REG(BPF_REG_1, BPF_REG_1), - BPF_ALU64_REG(BPF_SUB, BPF_REG_2, BPF_REG_1), - BPF_STX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .fixup_map_array_48b = { 4 }, - .result = REJECT, - .errstr = "math between map_value pointer and 4294967295 is not allowed", -}, -{ - "32bit pkt_ptr -= scalar", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_8, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_LDX_MEM(BPF_W, BPF_REG_7, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_7), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, 40), - BPF_JMP_REG(BPF_JGT, BPF_REG_6, BPF_REG_8, 2), - BPF_ALU32_REG(BPF_MOV, BPF_REG_4, BPF_REG_7), - BPF_ALU32_REG(BPF_SUB, BPF_REG_6, BPF_REG_4), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, -{ - "32bit scalar -= pkt_ptr", - .insns = { - BPF_LDX_MEM(BPF_W, BPF_REG_8, BPF_REG_1, - offsetof(struct __sk_buff, data_end)), - BPF_LDX_MEM(BPF_W, BPF_REG_7, BPF_REG_1, - offsetof(struct __sk_buff, data)), - BPF_MOV64_REG(BPF_REG_6, BPF_REG_7), - BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, 40), - BPF_JMP_REG(BPF_JGT, BPF_REG_6, BPF_REG_8, 2), - BPF_ALU32_REG(BPF_MOV, BPF_REG_4, BPF_REG_6), - BPF_ALU32_REG(BPF_SUB, BPF_REG_4, BPF_REG_7), - BPF_MOV64_IMM(BPF_REG_0, 0), - BPF_EXIT_INSN(), - }, - .prog_type = BPF_PROG_TYPE_SCHED_CLS, - .result = ACCEPT, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, -}, diff --git a/tools/testing/selftests/bpf/xskxceiver.c b/tools/testing/selftests/bpf/xskxceiver.c index 5a9691e942de..f144d0604ddf 100644 --- a/tools/testing/selftests/bpf/xskxceiver.c +++ b/tools/testing/selftests/bpf/xskxceiver.c @@ -77,6 +77,7 @@ #include <linux/if_link.h> #include <linux/if_ether.h> #include <linux/ip.h> +#include <linux/mman.h> #include <linux/udp.h> #include <arpa/inet.h> #include <net/if.h> @@ -1290,7 +1291,7 @@ static void thread_common_ops(struct test_spec *test, struct ifobject *ifobject) int ret; if (ifobject->umem->unaligned_mode) - mmap_flags |= MAP_HUGETLB; + mmap_flags |= MAP_HUGETLB | MAP_HUGE_2MB; if (ifobject->shared_umem) umem_sz *= 2; @@ -1379,6 +1380,11 @@ static void *worker_testapp_validate_rx(void *arg) pthread_exit(NULL); } +static u64 ceil_u64(u64 a, u64 b) +{ + return (a + b - 1) / b; +} + static void testapp_clean_xsk_umem(struct ifobject *ifobj) { u64 umem_sz = ifobj->umem->num_frames * ifobj->umem->frame_size; @@ -1386,6 +1392,7 @@ static void testapp_clean_xsk_umem(struct ifobject *ifobj) if (ifobj->shared_umem) umem_sz *= 2; + umem_sz = ceil_u64(umem_sz, HUGEPAGE_SIZE) * HUGEPAGE_SIZE; xsk_umem__delete(ifobj->umem->umem); munmap(ifobj->umem->buffer, umem_sz); } @@ -1619,14 +1626,15 @@ static void testapp_stats_fill_empty(struct test_spec *test) /* Simple test */ static bool hugepages_present(struct ifobject *ifobject) { - const size_t mmap_sz = 2 * ifobject->umem->num_frames * ifobject->umem->frame_size; + size_t mmap_sz = 2 * ifobject->umem->num_frames * ifobject->umem->frame_size; void *bufs; bufs = mmap(NULL, mmap_sz, PROT_READ | PROT_WRITE, - MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0); + MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB | MAP_HUGE_2MB, -1, 0); if (bufs == MAP_FAILED) return false; + mmap_sz = ceil_u64(mmap_sz, HUGEPAGE_SIZE) * HUGEPAGE_SIZE; munmap(bufs, mmap_sz); return true; } diff --git a/tools/testing/selftests/bpf/xskxceiver.h b/tools/testing/selftests/bpf/xskxceiver.h index 919327807a4e..c535aeab2ca3 100644 --- a/tools/testing/selftests/bpf/xskxceiver.h +++ b/tools/testing/selftests/bpf/xskxceiver.h @@ -56,6 +56,7 @@ #define RX_FULL_RXQSIZE 32 #define UMEM_HEADROOM_TEST_SIZE 128 #define XSK_UMEM__INVALID_FRAME_SIZE (XSK_UMEM__DEFAULT_FRAME_SIZE + 1) +#define HUGEPAGE_SIZE (2 * 1024 * 1024) #define print_verbose(x...) do { if (opt_verbose) ksft_print_msg(x); } while (0) diff --git a/tools/testing/selftests/drivers/net/mlxsw/qos_headroom.sh b/tools/testing/selftests/drivers/net/mlxsw/qos_headroom.sh index 3569ff45f7d5..88162b4027c0 100755 --- a/tools/testing/selftests/drivers/net/mlxsw/qos_headroom.sh +++ b/tools/testing/selftests/drivers/net/mlxsw/qos_headroom.sh @@ -18,7 +18,6 @@ lib_dir=$(dirname $0)/../../../net/forwarding NUM_NETIFS=0 source $lib_dir/lib.sh source $lib_dir/devlink_lib.sh -source qos_lib.sh swp=$NETIF_NO_CABLE @@ -371,7 +370,7 @@ test_tc_int_buf() tc qdisc delete dev $swp root } -bail_on_lldpad +bail_on_lldpad "configure DCB" "configure Qdiscs" trap cleanup EXIT setup_wait diff --git a/tools/testing/selftests/drivers/net/mlxsw/qos_lib.sh b/tools/testing/selftests/drivers/net/mlxsw/qos_lib.sh index faa51012cdac..5ad092b9bf10 100644 --- a/tools/testing/selftests/drivers/net/mlxsw/qos_lib.sh +++ b/tools/testing/selftests/drivers/net/mlxsw/qos_lib.sh @@ -54,31 +54,3 @@ measure_rate() echo $ir $er return $ret } - -bail_on_lldpad() -{ - if systemctl is-active --quiet lldpad; then - - cat >/dev/stderr <<-EOF - WARNING: lldpad is running - - lldpad will likely configure DCB, and this test will - configure Qdiscs. mlxsw does not support both at the - same time, one of them is arbitrarily going to overwrite - the other. That will cause spurious failures (or, - unlikely, passes) of this test. - EOF - - if [[ -z $ALLOW_LLDPAD ]]; then - cat >/dev/stderr <<-EOF - - If you want to run the test anyway, please set - an environment variable ALLOW_LLDPAD to a - non-empty string. - EOF - exit 1 - else - return - fi - fi -} diff --git a/tools/testing/selftests/drivers/net/mlxsw/qos_pfc.sh b/tools/testing/selftests/drivers/net/mlxsw/qos_pfc.sh index f9858e221996..42ce602d8d49 100755 --- a/tools/testing/selftests/drivers/net/mlxsw/qos_pfc.sh +++ b/tools/testing/selftests/drivers/net/mlxsw/qos_pfc.sh @@ -79,7 +79,6 @@ lib_dir=$(dirname $0)/../../../net/forwarding NUM_NETIFS=6 source $lib_dir/lib.sh source $lib_dir/devlink_lib.sh -source qos_lib.sh _1KB=1000 _100KB=$((100 * _1KB)) @@ -393,7 +392,7 @@ test_qos_pfc() log_test "PFC" } -bail_on_lldpad +bail_on_lldpad "configure DCB" "configure Qdiscs" trap cleanup EXIT setup_prepare diff --git a/tools/testing/selftests/drivers/net/mlxsw/sch_ets.sh b/tools/testing/selftests/drivers/net/mlxsw/sch_ets.sh index ceaa76b17a43..139175fd03e7 100755 --- a/tools/testing/selftests/drivers/net/mlxsw/sch_ets.sh +++ b/tools/testing/selftests/drivers/net/mlxsw/sch_ets.sh @@ -5,7 +5,6 @@ lib_dir=$(dirname $0)/../../../net/forwarding source $lib_dir/sch_ets_core.sh source $lib_dir/devlink_lib.sh -source qos_lib.sh ALL_TESTS=" ping_ipv4 @@ -78,5 +77,5 @@ collect_stats() done } -bail_on_lldpad +bail_on_lldpad "configure DCB" "configure Qdiscs" ets_run diff --git a/tools/testing/selftests/drivers/net/mlxsw/sch_red_core.sh b/tools/testing/selftests/drivers/net/mlxsw/sch_red_core.sh index 45b41b8f3232..299e06a5808c 100644 --- a/tools/testing/selftests/drivers/net/mlxsw/sch_red_core.sh +++ b/tools/testing/selftests/drivers/net/mlxsw/sch_red_core.sh @@ -74,7 +74,6 @@ lib_dir=$(dirname $0)/../../../net/forwarding source $lib_dir/lib.sh source $lib_dir/devlink_lib.sh source mlxsw_lib.sh -source qos_lib.sh ipaddr() { diff --git a/tools/testing/selftests/drivers/net/mlxsw/sch_red_ets.sh b/tools/testing/selftests/drivers/net/mlxsw/sch_red_ets.sh index 0d01c7cd82a1..8ecddafa79b3 100755 --- a/tools/testing/selftests/drivers/net/mlxsw/sch_red_ets.sh +++ b/tools/testing/selftests/drivers/net/mlxsw/sch_red_ets.sh @@ -166,7 +166,7 @@ ecn_mirror_test() uninstall_qdisc } -bail_on_lldpad +bail_on_lldpad "configure DCB" "configure Qdiscs" trap cleanup EXIT setup_prepare diff --git a/tools/testing/selftests/drivers/net/mlxsw/sch_red_root.sh b/tools/testing/selftests/drivers/net/mlxsw/sch_red_root.sh index 860205338e6f..159108d02895 100755 --- a/tools/testing/selftests/drivers/net/mlxsw/sch_red_root.sh +++ b/tools/testing/selftests/drivers/net/mlxsw/sch_red_root.sh @@ -73,7 +73,7 @@ red_mirror_test() uninstall_qdisc } -bail_on_lldpad +bail_on_lldpad "configure DCB" "configure Qdiscs" trap cleanup EXIT setup_prepare diff --git a/tools/testing/selftests/drivers/net/mlxsw/sch_tbf_ets.sh b/tools/testing/selftests/drivers/net/mlxsw/sch_tbf_ets.sh index c6ce0b448bf3..ecc3664376b3 100755 --- a/tools/testing/selftests/drivers/net/mlxsw/sch_tbf_ets.sh +++ b/tools/testing/selftests/drivers/net/mlxsw/sch_tbf_ets.sh @@ -1,8 +1,10 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0 -source qos_lib.sh -bail_on_lldpad +sch_tbf_pre_hook() +{ + bail_on_lldpad "configure DCB" "configure Qdiscs" +} lib_dir=$(dirname $0)/../../../net/forwarding TCFLAGS=skip_sw diff --git a/tools/testing/selftests/drivers/net/mlxsw/sch_tbf_prio.sh b/tools/testing/selftests/drivers/net/mlxsw/sch_tbf_prio.sh index 8d245f331619..2e0a4efb1703 100755 --- a/tools/testing/selftests/drivers/net/mlxsw/sch_tbf_prio.sh +++ b/tools/testing/selftests/drivers/net/mlxsw/sch_tbf_prio.sh @@ -1,8 +1,10 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0 -source qos_lib.sh -bail_on_lldpad +sch_tbf_pre_hook() +{ + bail_on_lldpad "configure DCB" "configure Qdiscs" +} lib_dir=$(dirname $0)/../../../net/forwarding TCFLAGS=skip_sw diff --git a/tools/testing/selftests/drivers/net/mlxsw/sch_tbf_root.sh b/tools/testing/selftests/drivers/net/mlxsw/sch_tbf_root.sh index 013886061f15..6679a338dfc4 100755 --- a/tools/testing/selftests/drivers/net/mlxsw/sch_tbf_root.sh +++ b/tools/testing/selftests/drivers/net/mlxsw/sch_tbf_root.sh @@ -1,8 +1,10 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0 -source qos_lib.sh -bail_on_lldpad +sch_tbf_pre_hook() +{ + bail_on_lldpad "configure DCB" "configure Qdiscs" +} lib_dir=$(dirname $0)/../../../net/forwarding TCFLAGS=skip_sw diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile index 1de34ec99290..c12df57d5539 100644 --- a/tools/testing/selftests/net/Makefile +++ b/tools/testing/selftests/net/Makefile @@ -83,6 +83,7 @@ TEST_GEN_FILES += nat6to4.o TEST_GEN_FILES += ip_local_port_range TEST_GEN_FILES += bind_wildcard TEST_PROGS += test_vxlan_mdb.sh +TEST_PROGS += test_bridge_neigh_suppress.sh TEST_FILES := settings diff --git a/tools/testing/selftests/net/forwarding/Makefile b/tools/testing/selftests/net/forwarding/Makefile index 236f6b796a52..a474c60fe348 100644 --- a/tools/testing/selftests/net/forwarding/Makefile +++ b/tools/testing/selftests/net/forwarding/Makefile @@ -15,6 +15,7 @@ TEST_PROGS = bridge_igmp.sh \ custom_multipath_hash.sh \ dual_vxlan_bridge.sh \ ethtool_extended_state.sh \ + ethtool_mm.sh \ ethtool.sh \ gre_custom_multipath_hash.sh \ gre_inner_v4_multipath.sh \ diff --git a/tools/testing/selftests/net/forwarding/ethtool_mm.sh b/tools/testing/selftests/net/forwarding/ethtool_mm.sh new file mode 100755 index 000000000000..c580ad623848 --- /dev/null +++ b/tools/testing/selftests/net/forwarding/ethtool_mm.sh @@ -0,0 +1,288 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +ALL_TESTS=" + manual_with_verification_h1_to_h2 + manual_with_verification_h2_to_h1 + manual_without_verification_h1_to_h2 + manual_without_verification_h2_to_h1 + manual_failed_verification_h1_to_h2 + manual_failed_verification_h2_to_h1 + lldp +" + +NUM_NETIFS=2 +REQUIRE_MZ=no +PREEMPTIBLE_PRIO=0 +source lib.sh + +traffic_test() +{ + local if=$1; shift + local src=$1; shift + local num_pkts=10000 + local before= + local after= + local delta= + + before=$(ethtool_std_stats_get $if "eth-mac" "FramesTransmittedOK" $src) + + $MZ $if -q -c $num_pkts -p 64 -b bcast -t ip -R $PREEMPTIBLE_PRIO + + after=$(ethtool_std_stats_get $if "eth-mac" "FramesTransmittedOK" $src) + + delta=$((after - before)) + + # Allow an extra 1% tolerance for random packets sent by the stack + [ $delta -ge $num_pkts ] && [ $delta -le $((num_pkts + 100)) ] +} + +manual_with_verification() +{ + local tx=$1; shift + local rx=$1; shift + + RET=0 + + # It isn't completely clear from IEEE 802.3-2018 Figure 99-5: Transmit + # Processing state diagram whether the "send_r" variable (send response + # to verification frame) should be taken into consideration while the + # MAC Merge TX direction is disabled. That being said, at least the + # NXP ENETC does not, and requires tx-enabled on in order to respond to + # the link partner's verification frames. + ethtool --set-mm $rx tx-enabled on + ethtool --set-mm $tx verify-enabled on tx-enabled on + + # Wait for verification to finish + sleep 1 + + ethtool --json --show-mm $tx | jq -r '.[]."verify-status"' | \ + grep -q 'SUCCEEDED' + check_err "$?" "Verification did not succeed" + + ethtool --json --show-mm $tx | jq -r '.[]."tx-active"' | grep -q 'true' + check_err "$?" "pMAC TX is not active" + + traffic_test $tx "pmac" + check_err "$?" "Traffic did not get sent through $tx's pMAC" + + ethtool --set-mm $tx verify-enabled off tx-enabled off + ethtool --set-mm $rx tx-enabled off + + log_test "Manual configuration with verification: $tx to $rx" +} + +manual_with_verification_h1_to_h2() +{ + manual_with_verification $h1 $h2 +} + +manual_with_verification_h2_to_h1() +{ + manual_with_verification $h2 $h1 +} + +manual_without_verification() +{ + local tx=$1; shift + local rx=$1; shift + + RET=0 + + ethtool --set-mm $tx verify-enabled off tx-enabled on + + ethtool --json --show-mm $tx | jq -r '.[]."verify-status"' | \ + grep -q 'DISABLED' + check_err "$?" "Verification is not disabled" + + ethtool --json --show-mm $tx | jq -r '.[]."tx-active"' | grep -q 'true' + check_err "$?" "pMAC TX is not active" + + traffic_test $tx "pmac" + check_err "$?" "Traffic did not get sent through $tx's pMAC" + + ethtool --set-mm $tx verify-enabled off tx-enabled off + + log_test "Manual configuration without verification: $tx to $rx" +} + +manual_without_verification_h1_to_h2() +{ + manual_without_verification $h1 $h2 +} + +manual_without_verification_h2_to_h1() +{ + manual_without_verification $h2 $h1 +} + +manual_failed_verification() +{ + local tx=$1; shift + local rx=$1; shift + + RET=0 + + ethtool --set-mm $rx pmac-enabled off + ethtool --set-mm $tx verify-enabled on tx-enabled on + + # Wait for verification to time out + sleep 1 + + ethtool --json --show-mm $tx | jq -r '.[]."verify-status"' | \ + grep -q 'SUCCEEDED' + check_fail "$?" "Verification succeeded when it shouldn't have" + + ethtool --json --show-mm $tx | jq -r '.[]."tx-active"' | grep -q 'true' + check_fail "$?" "pMAC TX is active when it shouldn't have" + + traffic_test $tx "emac" + check_err "$?" "Traffic did not get sent through $tx's eMAC" + + ethtool --set-mm $tx verify-enabled off tx-enabled off + ethtool --set-mm $rx pmac-enabled on + + log_test "Manual configuration with failed verification: $tx to $rx" +} + +manual_failed_verification_h1_to_h2() +{ + manual_failed_verification $h1 $h2 +} + +manual_failed_verification_h2_to_h1() +{ + manual_failed_verification $h2 $h1 +} + +lldp_change_add_frag_size() +{ + local add_frag_size=$1 + + lldptool -T -i $h1 -V addEthCaps addFragSize=$add_frag_size >/dev/null + # Wait for TLVs to be received + sleep 2 + lldptool -i $h2 -t -n -V addEthCaps | \ + grep -q "Additional fragment size: $add_frag_size" +} + +lldp() +{ + RET=0 + + systemctl start lldpad + + # Configure the interfaces to receive and transmit LLDPDUs + lldptool -L -i $h1 adminStatus=rxtx >/dev/null + lldptool -L -i $h2 adminStatus=rxtx >/dev/null + + # Enable the transmission of Additional Ethernet Capabilities TLV + lldptool -T -i $h1 -V addEthCaps enableTx=yes >/dev/null + lldptool -T -i $h2 -V addEthCaps enableTx=yes >/dev/null + + # Wait for TLVs to be received + sleep 2 + + lldptool -i $h1 -t -n -V addEthCaps | \ + grep -q "Preemption capability active" + check_err "$?" "$h1 pMAC TX is not active" + + lldptool -i $h2 -t -n -V addEthCaps | \ + grep -q "Preemption capability active" + check_err "$?" "$h2 pMAC TX is not active" + + lldp_change_add_frag_size 3 + check_err "$?" "addFragSize 3" + + lldp_change_add_frag_size 2 + check_err "$?" "addFragSize 2" + + lldp_change_add_frag_size 1 + check_err "$?" "addFragSize 1" + + lldp_change_add_frag_size 0 + check_err "$?" "addFragSize 0" + + traffic_test $h1 "pmac" + check_err "$?" "Traffic did not get sent through $h1's pMAC" + + traffic_test $h2 "pmac" + check_err "$?" "Traffic did not get sent through $h2's pMAC" + + systemctl stop lldpad + + log_test "LLDP" +} + +h1_create() +{ + ip link set dev $h1 up + + tc qdisc add dev $h1 root mqprio num_tc 4 map 0 1 2 3 \ + queues 1@0 1@1 1@2 1@3 \ + fp P E E E \ + hw 1 + + ethtool --set-mm $h1 pmac-enabled on tx-enabled off verify-enabled off +} + +h2_create() +{ + ip link set dev $h2 up + + ethtool --set-mm $h2 pmac-enabled on tx-enabled off verify-enabled off + + tc qdisc add dev $h2 root mqprio num_tc 4 map 0 1 2 3 \ + queues 1@0 1@1 1@2 1@3 \ + fp P E E E \ + hw 1 +} + +h1_destroy() +{ + ethtool --set-mm $h1 pmac-enabled off tx-enabled off verify-enabled off + + tc qdisc del dev $h1 root + + ip link set dev $h1 down +} + +h2_destroy() +{ + tc qdisc del dev $h2 root + + ethtool --set-mm $h2 pmac-enabled off tx-enabled off verify-enabled off + + ip link set dev $h2 down +} + +setup_prepare() +{ + check_ethtool_mm_support + check_tc_fp_support + require_command lldptool + bail_on_lldpad "autoconfigure the MAC Merge layer" "configure it manually" + + h1=${NETIFS[p1]} + h2=${NETIFS[p2]} + + h1_create + h2_create +} + +cleanup() +{ + pre_cleanup + + h2_destroy + h1_destroy +} + +trap cleanup EXIT + +setup_prepare +setup_wait + +tests_run + +exit $EXIT_STATUS diff --git a/tools/testing/selftests/net/forwarding/lib.sh b/tools/testing/selftests/net/forwarding/lib.sh index d47499ba81c7..057c3d0ad620 100755 --- a/tools/testing/selftests/net/forwarding/lib.sh +++ b/tools/testing/selftests/net/forwarding/lib.sh @@ -120,6 +120,15 @@ check_tc_action_hw_stats_support() fi } +check_tc_fp_support() +{ + tc qdisc add dev lo mqprio help 2>&1 | grep -q "fp " + if [[ $? -ne 0 ]]; then + echo "SKIP: iproute2 too old; tc is missing frame preemption support" + exit $ksft_skip + fi +} + check_ethtool_lanes_support() { ethtool --help 2>&1| grep lanes &> /dev/null @@ -129,6 +138,15 @@ check_ethtool_lanes_support() fi } +check_ethtool_mm_support() +{ + ethtool --help 2>&1| grep -- '--show-mm' &> /dev/null + if [[ $? -ne 0 ]]; then + echo "SKIP: ethtool too old; it is missing MAC Merge layer support" + exit $ksft_skip + fi +} + check_locked_port_support() { if ! bridge -d link show | grep -q " locked"; then @@ -787,6 +805,17 @@ ethtool_stats_get() ethtool -S $dev | grep "^ *$stat:" | head -n 1 | cut -d: -f2 } +ethtool_std_stats_get() +{ + local dev=$1; shift + local grp=$1; shift + local name=$1; shift + local src=$1; shift + + ethtool --json -S $dev --groups $grp -- --src $src | \ + jq '.[]."'"$grp"'"."'$name'"' +} + qdisc_stats_get() { local dev=$1; shift @@ -1887,3 +1916,34 @@ mldv1_done_get() payload_template_expand_checksum "$hbh$icmpv6" $checksum } + +bail_on_lldpad() +{ + local reason1="$1"; shift + local reason2="$1"; shift + + if systemctl is-active --quiet lldpad; then + + cat >/dev/stderr <<-EOF + WARNING: lldpad is running + + lldpad will likely $reason1, and this test will + $reason2. Both are not supported at the same time, + one of them is arbitrarily going to overwrite the + other. That will cause spurious failures (or, unlikely, + passes) of this test. + EOF + + if [[ -z $ALLOW_LLDPAD ]]; then + cat >/dev/stderr <<-EOF + + If you want to run the test anyway, please set + an environment variable ALLOW_LLDPAD to a + non-empty string. + EOF + exit 1 + else + return + fi + fi +} diff --git a/tools/testing/selftests/net/forwarding/sch_tbf_etsprio.sh b/tools/testing/selftests/net/forwarding/sch_tbf_etsprio.sh index 75a37c189ef3..df9bcd6a811a 100644 --- a/tools/testing/selftests/net/forwarding/sch_tbf_etsprio.sh +++ b/tools/testing/selftests/net/forwarding/sch_tbf_etsprio.sh @@ -57,6 +57,10 @@ tbf_root_test() tc qdisc del dev $swp2 root } +if type -t sch_tbf_pre_hook >/dev/null; then + sch_tbf_pre_hook +fi + trap cleanup EXIT setup_prepare diff --git a/tools/testing/selftests/net/forwarding/sch_tbf_root.sh b/tools/testing/selftests/net/forwarding/sch_tbf_root.sh index 72aa21ba88c7..96c997be0d03 100755 --- a/tools/testing/selftests/net/forwarding/sch_tbf_root.sh +++ b/tools/testing/selftests/net/forwarding/sch_tbf_root.sh @@ -23,6 +23,10 @@ tbf_test() tc qdisc del dev $swp2 root } +if type -t sch_tbf_pre_hook >/dev/null; then + sch_tbf_pre_hook +fi + trap cleanup EXIT setup_prepare diff --git a/tools/testing/selftests/net/test_bridge_neigh_suppress.sh b/tools/testing/selftests/net/test_bridge_neigh_suppress.sh new file mode 100755 index 000000000000..d80f2cd87614 --- /dev/null +++ b/tools/testing/selftests/net/test_bridge_neigh_suppress.sh @@ -0,0 +1,862 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# This test is for checking bridge neighbor suppression functionality. The +# topology consists of two bridges (VTEPs) connected using VXLAN. A single +# host is connected to each bridge over multiple VLANs. The test checks that +# ARP/NS messages from the first host are suppressed on the VXLAN port when +# should. +# +# +-----------------------+ +------------------------+ +# | h1 | | h2 | +# | | | | +# | + eth0.10 | | + eth0.10 | +# | | 192.0.2.1/28 | | | 192.0.2.2/28 | +# | | 2001:db8:1::1/64 | | | 2001:db8:1::2/64 | +# | | | | | | +# | | + eth0.20 | | | + eth0.20 | +# | \ | 192.0.2.17/28 | | \ | 192.0.2.18/28 | +# | \ | 2001:db8:2::1/64 | | \ | 2001:db8:2::2/64 | +# | \| | | \| | +# | + eth0 | | + eth0 | +# +----|------------------+ +----|-------------------+ +# | | +# | | +# +----|-------------------------------+ +----|-------------------------------+ +# | + swp1 + vx0 | | + swp1 + vx0 | +# | | | | | | | | +# | | br0 | | | | | | +# | +------------+-----------+ | | +------------+-----------+ | +# | | | | | | +# | | | | | | +# | +---+---+ | | +---+---+ | +# | | | | | | | | +# | | | | | | | | +# | + + | | + + | +# | br0.10 br0.20 | | br0.10 br0.20 | +# | | | | +# | 192.0.2.33 | | 192.0.2.34 | +# | + lo | | + lo | +# | | | | +# | | | | +# | 192.0.2.49/28 | | 192.0.2.50/28 | +# | veth0 +-------+ veth0 | +# | | | | +# | sw1 | | sw2 | +# +------------------------------------+ +------------------------------------+ + +ret=0 +# Kselftest framework requirement - SKIP code is 4. +ksft_skip=4 + +# All tests in this script. Can be overridden with -t option. +TESTS=" + neigh_suppress_arp + neigh_suppress_ns + neigh_vlan_suppress_arp + neigh_vlan_suppress_ns +" +VERBOSE=0 +PAUSE_ON_FAIL=no +PAUSE=no + +################################################################################ +# Utilities + +log_test() +{ + local rc=$1 + local expected=$2 + local msg="$3" + + if [ ${rc} -eq ${expected} ]; then + printf "TEST: %-60s [ OK ]\n" "${msg}" + nsuccess=$((nsuccess+1)) + else + ret=1 + nfail=$((nfail+1)) + printf "TEST: %-60s [FAIL]\n" "${msg}" + if [ "$VERBOSE" = "1" ]; then + echo " rc=$rc, expected $expected" + fi + + if [ "${PAUSE_ON_FAIL}" = "yes" ]; then + echo + echo "hit enter to continue, 'q' to quit" + read a + [ "$a" = "q" ] && exit 1 + fi + fi + + if [ "${PAUSE}" = "yes" ]; then + echo + echo "hit enter to continue, 'q' to quit" + read a + [ "$a" = "q" ] && exit 1 + fi + + [ "$VERBOSE" = "1" ] && echo +} + +run_cmd() +{ + local cmd="$1" + local out + local stderr="2>/dev/null" + + if [ "$VERBOSE" = "1" ]; then + printf "COMMAND: $cmd\n" + stderr= + fi + + out=$(eval $cmd $stderr) + rc=$? + if [ "$VERBOSE" = "1" -a -n "$out" ]; then + echo " $out" + fi + + return $rc +} + +tc_check_packets() +{ + local ns=$1; shift + local id=$1; shift + local handle=$1; shift + local count=$1; shift + local pkts + + sleep 0.1 + pkts=$(tc -n $ns -j -s filter show $id \ + | jq ".[] | select(.options.handle == $handle) | \ + .options.actions[0].stats.packets") + [[ $pkts == $count ]] +} + +################################################################################ +# Setup + +setup_topo_ns() +{ + local ns=$1; shift + + ip netns add $ns + ip -n $ns link set dev lo up + + ip netns exec $ns sysctl -qw net.ipv6.conf.all.keep_addr_on_down=1 + ip netns exec $ns sysctl -qw net.ipv6.conf.default.ignore_routes_with_linkdown=1 + ip netns exec $ns sysctl -qw net.ipv6.conf.all.accept_dad=0 + ip netns exec $ns sysctl -qw net.ipv6.conf.default.accept_dad=0 +} + +setup_topo() +{ + local ns + + for ns in h1 h2 sw1 sw2; do + setup_topo_ns $ns + done + + ip link add name veth0 type veth peer name veth1 + ip link set dev veth0 netns h1 name eth0 + ip link set dev veth1 netns sw1 name swp1 + + ip link add name veth0 type veth peer name veth1 + ip link set dev veth0 netns sw1 name veth0 + ip link set dev veth1 netns sw2 name veth0 + + ip link add name veth0 type veth peer name veth1 + ip link set dev veth0 netns h2 name eth0 + ip link set dev veth1 netns sw2 name swp1 +} + +setup_host_common() +{ + local ns=$1; shift + local v4addr1=$1; shift + local v4addr2=$1; shift + local v6addr1=$1; shift + local v6addr2=$1; shift + + ip -n $ns link set dev eth0 up + ip -n $ns link add link eth0 name eth0.10 up type vlan id 10 + ip -n $ns link add link eth0 name eth0.20 up type vlan id 20 + + ip -n $ns address add $v4addr1 dev eth0.10 + ip -n $ns address add $v4addr2 dev eth0.20 + ip -n $ns address add $v6addr1 dev eth0.10 + ip -n $ns address add $v6addr2 dev eth0.20 +} + +setup_h1() +{ + local ns=h1 + local v4addr1=192.0.2.1/28 + local v4addr2=192.0.2.17/28 + local v6addr1=2001:db8:1::1/64 + local v6addr2=2001:db8:2::1/64 + + setup_host_common $ns $v4addr1 $v4addr2 $v6addr1 $v6addr2 +} + +setup_h2() +{ + local ns=h2 + local v4addr1=192.0.2.2/28 + local v4addr2=192.0.2.18/28 + local v6addr1=2001:db8:1::2/64 + local v6addr2=2001:db8:2::2/64 + + setup_host_common $ns $v4addr1 $v4addr2 $v6addr1 $v6addr2 +} + +setup_sw_common() +{ + local ns=$1; shift + local local_addr=$1; shift + local remote_addr=$1; shift + local veth_addr=$1; shift + local gw_addr=$1; shift + + ip -n $ns address add $local_addr/32 dev lo + + ip -n $ns link set dev veth0 up + ip -n $ns address add $veth_addr/28 dev veth0 + ip -n $ns route add default via $gw_addr + + ip -n $ns link add name br0 up type bridge vlan_filtering 1 \ + vlan_default_pvid 0 mcast_snooping 0 + + ip -n $ns link add link br0 name br0.10 up type vlan id 10 + bridge -n $ns vlan add vid 10 dev br0 self + + ip -n $ns link add link br0 name br0.20 up type vlan id 20 + bridge -n $ns vlan add vid 20 dev br0 self + + ip -n $ns link set dev swp1 up master br0 + bridge -n $ns vlan add vid 10 dev swp1 + bridge -n $ns vlan add vid 20 dev swp1 + + ip -n $ns link add name vx0 up master br0 type vxlan \ + local $local_addr dstport 4789 nolearning external + bridge -n $ns fdb add 00:00:00:00:00:00 dev vx0 self static \ + dst $remote_addr src_vni 10010 + bridge -n $ns fdb add 00:00:00:00:00:00 dev vx0 self static \ + dst $remote_addr src_vni 10020 + bridge -n $ns link set dev vx0 vlan_tunnel on learning off + + bridge -n $ns vlan add vid 10 dev vx0 + bridge -n $ns vlan add vid 10 dev vx0 tunnel_info id 10010 + + bridge -n $ns vlan add vid 20 dev vx0 + bridge -n $ns vlan add vid 20 dev vx0 tunnel_info id 10020 +} + +setup_sw1() +{ + local ns=sw1 + local local_addr=192.0.2.33 + local remote_addr=192.0.2.34 + local veth_addr=192.0.2.49 + local gw_addr=192.0.2.50 + + setup_sw_common $ns $local_addr $remote_addr $veth_addr $gw_addr +} + +setup_sw2() +{ + local ns=sw2 + local local_addr=192.0.2.34 + local remote_addr=192.0.2.33 + local veth_addr=192.0.2.50 + local gw_addr=192.0.2.49 + + setup_sw_common $ns $local_addr $remote_addr $veth_addr $gw_addr +} + +setup() +{ + set -e + + setup_topo + setup_h1 + setup_h2 + setup_sw1 + setup_sw2 + + sleep 5 + + set +e +} + +cleanup() +{ + local ns + + for ns in h1 h2 sw1 sw2; do + ip netns del $ns &> /dev/null + done +} + +################################################################################ +# Tests + +neigh_suppress_arp_common() +{ + local vid=$1; shift + local sip=$1; shift + local tip=$1; shift + local h2_mac + + echo + echo "Per-port ARP suppression - VLAN $vid" + echo "----------------------------------" + + run_cmd "tc -n sw1 qdisc replace dev vx0 clsact" + run_cmd "tc -n sw1 filter replace dev vx0 egress pref 1 handle 101 proto 0x0806 flower indev swp1 arp_tip $tip arp_sip $sip arp_op request action pass" + + # Initial state - check that ARP requests are not suppressed and that + # ARP replies are received. + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip -I eth0.$vid $tip" + log_test $? 0 "arping" + tc_check_packets sw1 "dev vx0 egress" 101 1 + log_test $? 0 "ARP suppression" + + # Enable neighbor suppression and check that nothing changes compared + # to the initial state. + run_cmd "bridge -n sw1 link set dev vx0 neigh_suppress on" + run_cmd "bridge -n sw1 -d link show dev vx0 | grep \"neigh_suppress on\"" + log_test $? 0 "\"neigh_suppress\" is on" + + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip -I eth0.$vid $tip" + log_test $? 0 "arping" + tc_check_packets sw1 "dev vx0 egress" 101 2 + log_test $? 0 "ARP suppression" + + # Install an FDB entry for the remote host and check that nothing + # changes compared to the initial state. + h2_mac=$(ip -n h2 -j -p link show eth0.$vid | jq -r '.[]["address"]') + run_cmd "bridge -n sw1 fdb replace $h2_mac dev vx0 master static vlan $vid" + log_test $? 0 "FDB entry installation" + + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip -I eth0.$vid $tip" + log_test $? 0 "arping" + tc_check_packets sw1 "dev vx0 egress" 101 3 + log_test $? 0 "ARP suppression" + + # Install a neighbor on the matching SVI interface and check that ARP + # requests are suppressed. + run_cmd "ip -n sw1 neigh replace $tip lladdr $h2_mac nud permanent dev br0.$vid" + log_test $? 0 "Neighbor entry installation" + + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip -I eth0.$vid $tip" + log_test $? 0 "arping" + tc_check_packets sw1 "dev vx0 egress" 101 3 + log_test $? 0 "ARP suppression" + + # Take the second host down and check that ARP requests are suppressed + # and that ARP replies are received. + run_cmd "ip -n h2 link set dev eth0.$vid down" + log_test $? 0 "H2 down" + + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip -I eth0.$vid $tip" + log_test $? 0 "arping" + tc_check_packets sw1 "dev vx0 egress" 101 3 + log_test $? 0 "ARP suppression" + + run_cmd "ip -n h2 link set dev eth0.$vid up" + log_test $? 0 "H2 up" + + # Disable neighbor suppression and check that ARP requests are no + # longer suppressed. + run_cmd "bridge -n sw1 link set dev vx0 neigh_suppress off" + run_cmd "bridge -n sw1 -d link show dev vx0 | grep \"neigh_suppress off\"" + log_test $? 0 "\"neigh_suppress\" is off" + + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip -I eth0.$vid $tip" + log_test $? 0 "arping" + tc_check_packets sw1 "dev vx0 egress" 101 4 + log_test $? 0 "ARP suppression" + + # Take the second host down and check that ARP requests are not + # suppressed and that ARP replies are not received. + run_cmd "ip -n h2 link set dev eth0.$vid down" + log_test $? 0 "H2 down" + + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip -I eth0.$vid $tip" + log_test $? 1 "arping" + tc_check_packets sw1 "dev vx0 egress" 101 5 + log_test $? 0 "ARP suppression" +} + +neigh_suppress_arp() +{ + local vid=10 + local sip=192.0.2.1 + local tip=192.0.2.2 + + neigh_suppress_arp_common $vid $sip $tip + + vid=20 + sip=192.0.2.17 + tip=192.0.2.18 + neigh_suppress_arp_common $vid $sip $tip +} + +neigh_suppress_ns_common() +{ + local vid=$1; shift + local saddr=$1; shift + local daddr=$1; shift + local maddr=$1; shift + local h2_mac + + echo + echo "Per-port NS suppression - VLAN $vid" + echo "---------------------------------" + + run_cmd "tc -n sw1 qdisc replace dev vx0 clsact" + run_cmd "tc -n sw1 filter replace dev vx0 egress pref 1 handle 101 proto ipv6 flower indev swp1 ip_proto icmpv6 dst_ip $maddr src_ip $saddr type 135 code 0 action pass" + + # Initial state - check that NS messages are not suppressed and that ND + # messages are received. + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr -w 5000 $daddr eth0.$vid" + log_test $? 0 "ndisc6" + tc_check_packets sw1 "dev vx0 egress" 101 1 + log_test $? 0 "NS suppression" + + # Enable neighbor suppression and check that nothing changes compared + # to the initial state. + run_cmd "bridge -n sw1 link set dev vx0 neigh_suppress on" + run_cmd "bridge -n sw1 -d link show dev vx0 | grep \"neigh_suppress on\"" + log_test $? 0 "\"neigh_suppress\" is on" + + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr -w 5000 $daddr eth0.$vid" + log_test $? 0 "ndisc6" + tc_check_packets sw1 "dev vx0 egress" 101 2 + log_test $? 0 "NS suppression" + + # Install an FDB entry for the remote host and check that nothing + # changes compared to the initial state. + h2_mac=$(ip -n h2 -j -p link show eth0.$vid | jq -r '.[]["address"]') + run_cmd "bridge -n sw1 fdb replace $h2_mac dev vx0 master static vlan $vid" + log_test $? 0 "FDB entry installation" + + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr -w 5000 $daddr eth0.$vid" + log_test $? 0 "ndisc6" + tc_check_packets sw1 "dev vx0 egress" 101 3 + log_test $? 0 "NS suppression" + + # Install a neighbor on the matching SVI interface and check that NS + # messages are suppressed. + run_cmd "ip -n sw1 neigh replace $daddr lladdr $h2_mac nud permanent dev br0.$vid" + log_test $? 0 "Neighbor entry installation" + + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr -w 5000 $daddr eth0.$vid" + log_test $? 0 "ndisc6" + tc_check_packets sw1 "dev vx0 egress" 101 3 + log_test $? 0 "NS suppression" + + # Take the second host down and check that NS messages are suppressed + # and that ND messages are received. + run_cmd "ip -n h2 link set dev eth0.$vid down" + log_test $? 0 "H2 down" + + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr -w 5000 $daddr eth0.$vid" + log_test $? 0 "ndisc6" + tc_check_packets sw1 "dev vx0 egress" 101 3 + log_test $? 0 "NS suppression" + + run_cmd "ip -n h2 link set dev eth0.$vid up" + log_test $? 0 "H2 up" + + # Disable neighbor suppression and check that NS messages are no longer + # suppressed. + run_cmd "bridge -n sw1 link set dev vx0 neigh_suppress off" + run_cmd "bridge -n sw1 -d link show dev vx0 | grep \"neigh_suppress off\"" + log_test $? 0 "\"neigh_suppress\" is off" + + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr -w 5000 $daddr eth0.$vid" + log_test $? 0 "ndisc6" + tc_check_packets sw1 "dev vx0 egress" 101 4 + log_test $? 0 "NS suppression" + + # Take the second host down and check that NS messages are not + # suppressed and that ND messages are not received. + run_cmd "ip -n h2 link set dev eth0.$vid down" + log_test $? 0 "H2 down" + + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr -w 5000 $daddr eth0.$vid" + log_test $? 2 "ndisc6" + tc_check_packets sw1 "dev vx0 egress" 101 5 + log_test $? 0 "NS suppression" +} + +neigh_suppress_ns() +{ + local vid=10 + local saddr=2001:db8:1::1 + local daddr=2001:db8:1::2 + local maddr=ff02::1:ff00:2 + + neigh_suppress_ns_common $vid $saddr $daddr $maddr + + vid=20 + saddr=2001:db8:2::1 + daddr=2001:db8:2::2 + maddr=ff02::1:ff00:2 + + neigh_suppress_ns_common $vid $saddr $daddr $maddr +} + +neigh_vlan_suppress_arp() +{ + local vid1=10 + local vid2=20 + local sip1=192.0.2.1 + local sip2=192.0.2.17 + local tip1=192.0.2.2 + local tip2=192.0.2.18 + local h2_mac1 + local h2_mac2 + + echo + echo "Per-{Port, VLAN} ARP suppression" + echo "--------------------------------" + + run_cmd "tc -n sw1 qdisc replace dev vx0 clsact" + run_cmd "tc -n sw1 filter replace dev vx0 egress pref 1 handle 101 proto 0x0806 flower indev swp1 arp_tip $tip1 arp_sip $sip1 arp_op request action pass" + run_cmd "tc -n sw1 filter replace dev vx0 egress pref 1 handle 102 proto 0x0806 flower indev swp1 arp_tip $tip2 arp_sip $sip2 arp_op request action pass" + + h2_mac1=$(ip -n h2 -j -p link show eth0.$vid1 | jq -r '.[]["address"]') + h2_mac2=$(ip -n h2 -j -p link show eth0.$vid2 | jq -r '.[]["address"]') + run_cmd "bridge -n sw1 fdb replace $h2_mac1 dev vx0 master static vlan $vid1" + run_cmd "bridge -n sw1 fdb replace $h2_mac2 dev vx0 master static vlan $vid2" + run_cmd "ip -n sw1 neigh replace $tip1 lladdr $h2_mac1 nud permanent dev br0.$vid1" + run_cmd "ip -n sw1 neigh replace $tip2 lladdr $h2_mac2 nud permanent dev br0.$vid2" + + # Enable per-{Port, VLAN} neighbor suppression and check that ARP + # requests are not suppressed and that ARP replies are received. + run_cmd "bridge -n sw1 link set dev vx0 neigh_vlan_suppress on" + run_cmd "bridge -n sw1 -d link show dev vx0 | grep \"neigh_vlan_suppress on\"" + log_test $? 0 "\"neigh_vlan_suppress\" is on" + + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip1 -I eth0.$vid1 $tip1" + log_test $? 0 "arping (VLAN $vid1)" + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip2 -I eth0.$vid2 $tip2" + log_test $? 0 "arping (VLAN $vid2)" + + tc_check_packets sw1 "dev vx0 egress" 101 1 + log_test $? 0 "ARP suppression (VLAN $vid1)" + tc_check_packets sw1 "dev vx0 egress" 102 1 + log_test $? 0 "ARP suppression (VLAN $vid2)" + + # Enable neighbor suppression on VLAN 10 and check that only on this + # VLAN ARP requests are suppressed. + run_cmd "bridge -n sw1 vlan set vid $vid1 dev vx0 neigh_suppress on" + run_cmd "bridge -n sw1 -d vlan show dev vx0 vid $vid1 | grep \"neigh_suppress on\"" + log_test $? 0 "\"neigh_suppress\" is on (VLAN $vid1)" + run_cmd "bridge -n sw1 -d vlan show dev vx0 vid $vid2 | grep \"neigh_suppress off\"" + log_test $? 0 "\"neigh_suppress\" is off (VLAN $vid2)" + + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip1 -I eth0.$vid1 $tip1" + log_test $? 0 "arping (VLAN $vid1)" + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip2 -I eth0.$vid2 $tip2" + log_test $? 0 "arping (VLAN $vid2)" + + tc_check_packets sw1 "dev vx0 egress" 101 1 + log_test $? 0 "ARP suppression (VLAN $vid1)" + tc_check_packets sw1 "dev vx0 egress" 102 2 + log_test $? 0 "ARP suppression (VLAN $vid2)" + + # Enable neighbor suppression on the port and check that it has no + # effect compared to previous state. + run_cmd "bridge -n sw1 link set dev vx0 neigh_suppress on" + run_cmd "bridge -n sw1 -d link show dev vx0 | grep \"neigh_suppress on\"" + log_test $? 0 "\"neigh_suppress\" is on" + + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip1 -I eth0.$vid1 $tip1" + log_test $? 0 "arping (VLAN $vid1)" + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip2 -I eth0.$vid2 $tip2" + log_test $? 0 "arping (VLAN $vid2)" + + tc_check_packets sw1 "dev vx0 egress" 101 1 + log_test $? 0 "ARP suppression (VLAN $vid1)" + tc_check_packets sw1 "dev vx0 egress" 102 3 + log_test $? 0 "ARP suppression (VLAN $vid2)" + + # Disable neighbor suppression on the port and check that it has no + # effect compared to previous state. + run_cmd "bridge -n sw1 link set dev vx0 neigh_suppress off" + run_cmd "bridge -n sw1 -d link show dev vx0 | grep \"neigh_suppress off\"" + log_test $? 0 "\"neigh_suppress\" is off" + + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip1 -I eth0.$vid1 $tip1" + log_test $? 0 "arping (VLAN $vid1)" + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip2 -I eth0.$vid2 $tip2" + log_test $? 0 "arping (VLAN $vid2)" + + tc_check_packets sw1 "dev vx0 egress" 101 1 + log_test $? 0 "ARP suppression (VLAN $vid1)" + tc_check_packets sw1 "dev vx0 egress" 102 4 + log_test $? 0 "ARP suppression (VLAN $vid2)" + + # Disable neighbor suppression on VLAN 10 and check that ARP requests + # are no longer suppressed on this VLAN. + run_cmd "bridge -n sw1 vlan set vid $vid1 dev vx0 neigh_suppress off" + run_cmd "bridge -n sw1 -d vlan show dev vx0 vid $vid1 | grep \"neigh_suppress off\"" + log_test $? 0 "\"neigh_suppress\" is off (VLAN $vid1)" + + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip1 -I eth0.$vid1 $tip1" + log_test $? 0 "arping (VLAN $vid1)" + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip2 -I eth0.$vid2 $tip2" + log_test $? 0 "arping (VLAN $vid2)" + + tc_check_packets sw1 "dev vx0 egress" 101 2 + log_test $? 0 "ARP suppression (VLAN $vid1)" + tc_check_packets sw1 "dev vx0 egress" 102 5 + log_test $? 0 "ARP suppression (VLAN $vid2)" + + # Disable per-{Port, VLAN} neighbor suppression, enable neighbor + # suppression on the port and check that on both VLANs ARP requests are + # suppressed. + run_cmd "bridge -n sw1 link set dev vx0 neigh_vlan_suppress off" + run_cmd "bridge -n sw1 -d link show dev vx0 | grep \"neigh_vlan_suppress off\"" + log_test $? 0 "\"neigh_vlan_suppress\" is off" + + run_cmd "bridge -n sw1 link set dev vx0 neigh_suppress on" + run_cmd "bridge -n sw1 -d link show dev vx0 | grep \"neigh_suppress on\"" + log_test $? 0 "\"neigh_suppress\" is on" + + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip1 -I eth0.$vid1 $tip1" + log_test $? 0 "arping (VLAN $vid1)" + run_cmd "ip netns exec h1 arping -q -b -c 1 -w 5 -s $sip2 -I eth0.$vid2 $tip2" + log_test $? 0 "arping (VLAN $vid2)" + + tc_check_packets sw1 "dev vx0 egress" 101 2 + log_test $? 0 "ARP suppression (VLAN $vid1)" + tc_check_packets sw1 "dev vx0 egress" 102 5 + log_test $? 0 "ARP suppression (VLAN $vid2)" +} + +neigh_vlan_suppress_ns() +{ + local vid1=10 + local vid2=20 + local saddr1=2001:db8:1::1 + local saddr2=2001:db8:2::1 + local daddr1=2001:db8:1::2 + local daddr2=2001:db8:2::2 + local maddr=ff02::1:ff00:2 + local h2_mac1 + local h2_mac2 + + echo + echo "Per-{Port, VLAN} NS suppression" + echo "-------------------------------" + + run_cmd "tc -n sw1 qdisc replace dev vx0 clsact" + run_cmd "tc -n sw1 filter replace dev vx0 egress pref 1 handle 101 proto ipv6 flower indev swp1 ip_proto icmpv6 dst_ip $maddr src_ip $saddr1 type 135 code 0 action pass" + run_cmd "tc -n sw1 filter replace dev vx0 egress pref 1 handle 102 proto ipv6 flower indev swp1 ip_proto icmpv6 dst_ip $maddr src_ip $saddr2 type 135 code 0 action pass" + + h2_mac1=$(ip -n h2 -j -p link show eth0.$vid1 | jq -r '.[]["address"]') + h2_mac2=$(ip -n h2 -j -p link show eth0.$vid2 | jq -r '.[]["address"]') + run_cmd "bridge -n sw1 fdb replace $h2_mac1 dev vx0 master static vlan $vid1" + run_cmd "bridge -n sw1 fdb replace $h2_mac2 dev vx0 master static vlan $vid2" + run_cmd "ip -n sw1 neigh replace $daddr1 lladdr $h2_mac1 nud permanent dev br0.$vid1" + run_cmd "ip -n sw1 neigh replace $daddr2 lladdr $h2_mac2 nud permanent dev br0.$vid2" + + # Enable per-{Port, VLAN} neighbor suppression and check that NS + # messages are not suppressed and that ND messages are received. + run_cmd "bridge -n sw1 link set dev vx0 neigh_vlan_suppress on" + run_cmd "bridge -n sw1 -d link show dev vx0 | grep \"neigh_vlan_suppress on\"" + log_test $? 0 "\"neigh_vlan_suppress\" is on" + + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr1 -w 5000 $daddr1 eth0.$vid1" + log_test $? 0 "ndisc6 (VLAN $vid1)" + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr2 -w 5000 $daddr2 eth0.$vid2" + log_test $? 0 "ndisc6 (VLAN $vid2)" + + tc_check_packets sw1 "dev vx0 egress" 101 1 + log_test $? 0 "NS suppression (VLAN $vid1)" + tc_check_packets sw1 "dev vx0 egress" 102 1 + log_test $? 0 "NS suppression (VLAN $vid2)" + + # Enable neighbor suppression on VLAN 10 and check that only on this + # VLAN NS messages are suppressed. + run_cmd "bridge -n sw1 vlan set vid $vid1 dev vx0 neigh_suppress on" + run_cmd "bridge -n sw1 -d vlan show dev vx0 vid $vid1 | grep \"neigh_suppress on\"" + log_test $? 0 "\"neigh_suppress\" is on (VLAN $vid1)" + run_cmd "bridge -n sw1 -d vlan show dev vx0 vid $vid2 | grep \"neigh_suppress off\"" + log_test $? 0 "\"neigh_suppress\" is off (VLAN $vid2)" + + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr1 -w 5000 $daddr1 eth0.$vid1" + log_test $? 0 "ndisc6 (VLAN $vid1)" + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr2 -w 5000 $daddr2 eth0.$vid2" + log_test $? 0 "ndisc6 (VLAN $vid2)" + + tc_check_packets sw1 "dev vx0 egress" 101 1 + log_test $? 0 "NS suppression (VLAN $vid1)" + tc_check_packets sw1 "dev vx0 egress" 102 2 + log_test $? 0 "NS suppression (VLAN $vid2)" + + # Enable neighbor suppression on the port and check that it has no + # effect compared to previous state. + run_cmd "bridge -n sw1 link set dev vx0 neigh_suppress on" + run_cmd "bridge -n sw1 -d link show dev vx0 | grep \"neigh_suppress on\"" + log_test $? 0 "\"neigh_suppress\" is on" + + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr1 -w 5000 $daddr1 eth0.$vid1" + log_test $? 0 "ndisc6 (VLAN $vid1)" + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr2 -w 5000 $daddr2 eth0.$vid2" + log_test $? 0 "ndisc6 (VLAN $vid2)" + + tc_check_packets sw1 "dev vx0 egress" 101 1 + log_test $? 0 "NS suppression (VLAN $vid1)" + tc_check_packets sw1 "dev vx0 egress" 102 3 + log_test $? 0 "NS suppression (VLAN $vid2)" + + # Disable neighbor suppression on the port and check that it has no + # effect compared to previous state. + run_cmd "bridge -n sw1 link set dev vx0 neigh_suppress off" + run_cmd "bridge -n sw1 -d link show dev vx0 | grep \"neigh_suppress off\"" + log_test $? 0 "\"neigh_suppress\" is off" + + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr1 -w 5000 $daddr1 eth0.$vid1" + log_test $? 0 "ndisc6 (VLAN $vid1)" + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr2 -w 5000 $daddr2 eth0.$vid2" + log_test $? 0 "ndisc6 (VLAN $vid2)" + + tc_check_packets sw1 "dev vx0 egress" 101 1 + log_test $? 0 "NS suppression (VLAN $vid1)" + tc_check_packets sw1 "dev vx0 egress" 102 4 + log_test $? 0 "NS suppression (VLAN $vid2)" + + # Disable neighbor suppression on VLAN 10 and check that NS messages + # are no longer suppressed on this VLAN. + run_cmd "bridge -n sw1 vlan set vid $vid1 dev vx0 neigh_suppress off" + run_cmd "bridge -n sw1 -d vlan show dev vx0 vid $vid1 | grep \"neigh_suppress off\"" + log_test $? 0 "\"neigh_suppress\" is off (VLAN $vid1)" + + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr1 -w 5000 $daddr1 eth0.$vid1" + log_test $? 0 "ndisc6 (VLAN $vid1)" + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr2 -w 5000 $daddr2 eth0.$vid2" + log_test $? 0 "ndisc6 (VLAN $vid2)" + + tc_check_packets sw1 "dev vx0 egress" 101 2 + log_test $? 0 "NS suppression (VLAN $vid1)" + tc_check_packets sw1 "dev vx0 egress" 102 5 + log_test $? 0 "NS suppression (VLAN $vid2)" + + # Disable per-{Port, VLAN} neighbor suppression, enable neighbor + # suppression on the port and check that on both VLANs NS messages are + # suppressed. + run_cmd "bridge -n sw1 link set dev vx0 neigh_vlan_suppress off" + run_cmd "bridge -n sw1 -d link show dev vx0 | grep \"neigh_vlan_suppress off\"" + log_test $? 0 "\"neigh_vlan_suppress\" is off" + + run_cmd "bridge -n sw1 link set dev vx0 neigh_suppress on" + run_cmd "bridge -n sw1 -d link show dev vx0 | grep \"neigh_suppress on\"" + log_test $? 0 "\"neigh_suppress\" is on" + + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr1 -w 5000 $daddr1 eth0.$vid1" + log_test $? 0 "ndisc6 (VLAN $vid1)" + run_cmd "ip netns exec h1 ndisc6 -q -r 1 -s $saddr2 -w 5000 $daddr2 eth0.$vid2" + log_test $? 0 "ndisc6 (VLAN $vid2)" + + tc_check_packets sw1 "dev vx0 egress" 101 2 + log_test $? 0 "NS suppression (VLAN $vid1)" + tc_check_packets sw1 "dev vx0 egress" 102 5 + log_test $? 0 "NS suppression (VLAN $vid2)" +} + +################################################################################ +# Usage + +usage() +{ + cat <<EOF +usage: ${0##*/} OPTS + + -t <test> Test(s) to run (default: all) + (options: $TESTS) + -p Pause on fail + -P Pause after each test before cleanup + -v Verbose mode (show commands and output) +EOF +} + +################################################################################ +# Main + +trap cleanup EXIT + +while getopts ":t:pPvh" opt; do + case $opt in + t) TESTS=$OPTARG;; + p) PAUSE_ON_FAIL=yes;; + P) PAUSE=yes;; + v) VERBOSE=$(($VERBOSE + 1));; + h) usage; exit 0;; + *) usage; exit 1;; + esac +done + +# Make sure we don't pause twice. +[ "${PAUSE}" = "yes" ] && PAUSE_ON_FAIL=no + +if [ "$(id -u)" -ne 0 ];then + echo "SKIP: Need root privileges" + exit $ksft_skip; +fi + +if [ ! -x "$(command -v ip)" ]; then + echo "SKIP: Could not run test without ip tool" + exit $ksft_skip +fi + +if [ ! -x "$(command -v bridge)" ]; then + echo "SKIP: Could not run test without bridge tool" + exit $ksft_skip +fi + +if [ ! -x "$(command -v tc)" ]; then + echo "SKIP: Could not run test without tc tool" + exit $ksft_skip +fi + +if [ ! -x "$(command -v arping)" ]; then + echo "SKIP: Could not run test without arping tool" + exit $ksft_skip +fi + +if [ ! -x "$(command -v ndisc6)" ]; then + echo "SKIP: Could not run test without ndisc6 tool" + exit $ksft_skip +fi + +if [ ! -x "$(command -v jq)" ]; then + echo "SKIP: Could not run test without jq tool" + exit $ksft_skip +fi + +bridge link help 2>&1 | grep -q "neigh_vlan_suppress" +if [ $? -ne 0 ]; then + echo "SKIP: iproute2 bridge too old, missing per-VLAN neighbor suppression support" + exit $ksft_skip +fi + +# Start clean. +cleanup + +for t in $TESTS +do + setup; $t; cleanup; +done + +if [ "$TESTS" != "none" ]; then + printf "\nTests passed: %3d\n" ${nsuccess} + printf "Tests failed: %3d\n" ${nfail} +fi + +exit $ret diff --git a/usr/gen_init_cpio.c b/usr/gen_init_cpio.c index ee01e40e8bc6..61230532fef1 100644 --- a/usr/gen_init_cpio.c +++ b/usr/gen_init_cpio.c @@ -353,6 +353,12 @@ static int cpio_mkfile(const char *name, const char *location, buf.st_mtime = 0xffffffff; } + if (buf.st_mtime < 0) { + fprintf(stderr, "%s: Timestamp negative, clipping.\n", + location); + buf.st_mtime = 0; + } + if (buf.st_size > 0xffffffff) { fprintf(stderr, "%s: Size exceeds maximum cpio file size\n", location); @@ -602,10 +608,10 @@ int main (int argc, char *argv[]) /* * Timestamps after 2106-02-07 06:28:15 UTC have an ascii hex time_t * representation that exceeds 8 chars and breaks the cpio header - * specification. + * specification. Negative timestamps similarly exceed 8 chars. */ - if (default_mtime > 0xffffffff) { - fprintf(stderr, "ERROR: Timestamp too large for cpio format\n"); + if (default_mtime > 0xffffffff || default_mtime < 0) { + fprintf(stderr, "ERROR: Timestamp out of range for cpio format\n"); exit(1); } |