aboutsummaryrefslogtreecommitdiff
path: root/arch/arm64/kernel
AgeCommit message (Collapse)AuthorFilesLines
2020-03-18arm64: suspend: restore the kernel ptrauth keysAmit Daniel Kachhap1-2/+2
This patch restores the kernel keys from current task during cpu resume after the mmu is turned on and ptrauth is enabled. A flag is added in macro ptrauth_keys_install_kernel to check if isb instruction needs to be executed. Signed-off-by: Amit Daniel Kachhap <[email protected]> Reviewed-by: Vincenzo Frascino <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-18arm64: __show_regs: strip PAC from lr in printkAmit Daniel Kachhap1-1/+1
lr is printed with %pS which will try to find an entry in kallsyms. After enabling pointer authentication, this match will fail due to PAC present in the lr. Strip PAC from the lr to display the correct symbol name. Suggested-by: James Morse <[email protected]> Signed-off-by: Amit Daniel Kachhap <[email protected]> Reviewed-by: Vincenzo Frascino <[email protected]> Acked-by: Catalin Marinas <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-18arm64: unwind: strip PAC from kernel addressesMark Rutland1-1/+4
When we enable pointer authentication in the kernel, LR values saved to the stack will have a PAC which we must strip in order to retrieve the real return address. Strip PACs when unwinding the stack in order to account for this. When function graph tracer is used with patchable-function-entry then return_to_handler will also have pac bits so strip it too. Reviewed-by: Kees Cook <[email protected]> Acked-by: Catalin Marinas <[email protected]> Reviewed-by: James Morse <[email protected]> Signed-off-by: Mark Rutland <[email protected]> Signed-off-by: Kristina Martsenko <[email protected]> [Amit: Re-position ptrauth_strip_insn_pac, comment] Signed-off-by: Amit Daniel Kachhap <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-18arm64: initialize and switch ptrauth kernel keysKristina Martsenko4-0/+18
Set up keys to use pointer authentication within the kernel. The kernel will be compiled with APIAKey instructions, the other keys are currently unused. Each task is given its own APIAKey, which is initialized during fork. The key is changed during context switch and on kernel entry from EL0. The keys for idle threads need to be set before calling any C functions, because it is not possible to enter and exit a function with different keys. Reviewed-by: Kees Cook <[email protected]> Reviewed-by: Catalin Marinas <[email protected]> Reviewed-by: Vincenzo Frascino <[email protected]> Signed-off-by: Kristina Martsenko <[email protected]> [Amit: Modified secondary cores key structure, comments] Signed-off-by: Amit Daniel Kachhap <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-18arm64: enable ptrauth earlierKristina Martsenko1-10/+3
When the kernel is compiled with pointer auth instructions, the boot CPU needs to start using address auth very early, so change the cpucap to account for this. Pointer auth must be enabled before we call C functions, because it is not possible to enter a function with pointer auth disabled and exit it with pointer auth enabled. Note, mismatches between architected and IMPDEF algorithms will still be caught by the cpufeature framework (the separate *_ARCH and *_IMP_DEF cpucaps). Note the change in behavior: if the boot CPU has address auth and a late CPU does not, then the late CPU is parked by the cpufeature framework. This is possible as kernel will only have NOP space intructions for PAC so such mismatched late cpu will silently ignore those instructions in C functions. Also, if the boot CPU does not have address auth and the late CPU has then the late cpu will still boot but with ptrauth feature disabled. Leave generic authentication as a "system scope" cpucap for now, since initially the kernel will only use address authentication. Reviewed-by: Kees Cook <[email protected]> Reviewed-by: Suzuki K Poulose <[email protected]> Reviewed-by: Vincenzo Frascino <[email protected]> Signed-off-by: Kristina Martsenko <[email protected]> [Amit: Re-worked ptrauth setup logic, comments] Signed-off-by: Amit Daniel Kachhap <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-18arm64: cpufeature: handle conflicts based on capabilityKristina Martsenko1-14/+15
Each system capability can be of either boot, local, or system scope, depending on when the state of the capability is finalized. When we detect a conflict on a late CPU, we either offline the CPU or panic the system. We currently always panic if the conflict is caused by a boot scope capability, and offline the CPU if the conflict is caused by a local or system scope capability. We're going to want to add a new capability (for pointer authentication) which needs to be boot scope but doesn't need to panic the system when a conflict is detected. So add a new flag to specify whether the capability requires the system to panic or not. Current boot scope capabilities are updated to set the flag, so there should be no functional change as a result of this patch. Signed-off-by: Amit Daniel Kachhap <[email protected]> Signed-off-by: Kristina Martsenko <[email protected]> Reviewed-by: Vincenzo Frascino <[email protected]> Reviewed-by: Suzuki K Poulose <[email protected]> Reviewed-by: Kees Cook <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-18arm64: cpufeature: Move cpu capability helpers inside C fileAmit Daniel Kachhap1-0/+13
These helpers are used only by functions inside cpufeature.c and hence makes sense to be moved from cpufeature.h to cpufeature.c as they are not expected to be used globally. This change helps in reducing the header file size as well as to add future cpu capability types without confusion. Only a cpu capability type macro is sufficient to expose those capabilities globally. Signed-off-by: Amit Daniel Kachhap <[email protected]> Reviewed-by: Vincenzo Frascino <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-18arm64: ptrauth: Add bootup/runtime flags for __cpu_setupAmit Daniel Kachhap2-0/+4
This patch allows __cpu_setup to be invoked with one of these flags, ARM64_CPU_BOOT_PRIMARY, ARM64_CPU_BOOT_SECONDARY or ARM64_CPU_RUNTIME. This is required as some cpufeatures need different handling during different scenarios. The input parameter in x0 is preserved till the end to be used inside this function. There should be no functional change with this patch and is useful for the subsequent ptrauth patch which utilizes it. Some upcoming arm cpufeatures can also utilize these flags. Suggested-by: James Morse <[email protected]> Signed-off-by: Amit Daniel Kachhap <[email protected]> Reviewed-by: Vincenzo Frascino <[email protected]> Reviewed-by: James Morse <[email protected]> Reviewed-by: Suzuki K Poulose <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-18arm64: install user ptrauth keys at kernel exit timeKristina Martsenko4-4/+14
As we're going to enable pointer auth within the kernel and use a different APIAKey for the kernel itself, so move the user APIAKey switch to EL0 exception return. The other 4 keys could remain switched during task switch, but are also moved to keep things consistent. Reviewed-by: Kees Cook <[email protected]> Reviewed-by: James Morse <[email protected]> Reviewed-by: Vincenzo Frascino <[email protected]> Signed-off-by: Kristina Martsenko <[email protected]> [Amit: commit msg, re-positioned the patch, comments] Signed-off-by: Amit Daniel Kachhap <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-18arm64: rename ptrauth key structures to be user-specificKristina Martsenko2-12/+12
We currently enable ptrauth for userspace, but do not use it within the kernel. We're going to enable it for the kernel, and will need to manage a separate set of ptrauth keys for the kernel. We currently keep all 5 keys in struct ptrauth_keys. However, as the kernel will only need to use 1 key, it is a bit wasteful to allocate a whole ptrauth_keys struct for every thread. Therefore, a subsequent patch will define a separate struct, with only 1 key, for the kernel. In preparation for that, rename the existing struct (and associated macros and functions) to reflect that they are specific to userspace. Acked-by: Catalin Marinas <[email protected]> Reviewed-by: Vincenzo Frascino <[email protected]> Signed-off-by: Kristina Martsenko <[email protected]> [Amit: Re-positioned the patch to reduce the diff] Signed-off-by: Amit Daniel Kachhap <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-18arm64: cpufeature: add pointer auth meta-capabilitiesKristina Martsenko1-1/+24
To enable pointer auth for the kernel, we're going to need to check for the presence of address auth and generic auth using alternative_if. We currently have two cpucaps for each, but alternative_if needs to check a single cpucap. So define meta-capabilities that are present when either of the current two capabilities is present. Leave the existing four cpucaps in place, as they are still needed to check for mismatched systems where one CPU has the architected algorithm but another has the IMP DEF algorithm. Note, the meta-capabilities were present before but were removed in commit a56005d32105 ("arm64: cpufeature: Reduce number of pointer auth CPU caps from 6 to 4") and commit 1e013d06120c ("arm64: cpufeature: Rework ptr auth hwcaps using multi_entry_cap_matches"), as they were not needed then. Note, unlike before, the current patch checks the cpucap values directly, instead of reading the CPU ID register value. Reviewed-by: Suzuki K Poulose <[email protected]> Reviewed-by: Kees Cook <[email protected]> Reviewed-by: Vincenzo Frascino <[email protected]> Signed-off-by: Kristina Martsenko <[email protected]> [Amit: commit message and macro rebase, use __system_matches_cap] Signed-off-by: Amit Daniel Kachhap <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-18arm64: cpufeature: Fix meta-capability cpufeature checkAmit Daniel Kachhap1-1/+20
Some existing/future meta cpucaps match need the presence of individual cpucaps. Currently the individual cpucaps checks it via an array based flag and this introduces dependency on the array entry order. This limitation exists only for system scope cpufeature. This patch introduces an internal helper function (__system_matches_cap) to invoke the matching handler for system scope. This helper has to be used during a narrow window when, - The system wide safe registers are set with all the SMP CPUs and, - The SYSTEM_FEATURE cpu_hwcaps may not have been set. Normal users should use the existing cpus_have_{const_}cap() global function. Suggested-by: Suzuki K Poulose <[email protected]> Suggested-by: Catalin Marinas <[email protected]> Signed-off-by: Amit Daniel Kachhap <[email protected]> Reviewed-by: Vincenzo Frascino <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-17arm64: smp: fix crash_smp_send_stop() behaviourCristian Marussi1-2/+6
On a system configured to trigger a crash_kexec() reboot, when only one CPU is online and another CPU panics while starting-up, crash_smp_send_stop() will fail to send any STOP message to the other already online core, resulting in fail to freeze and registers not properly saved. Moreover even if the proper messages are sent (case CPUs > 2) it will similarly fail to account for the booting CPU when executing the final stop wait-loop, so potentially resulting in some CPU not been waited for shutdown before rebooting. A tangible effect of this behaviour can be observed when, after a panic with kexec enabled and loaded, on the following reboot triggered by kexec, the cpu that could not be successfully stopped fails to come back online: [ 362.291022] ------------[ cut here ]------------ [ 362.291525] kernel BUG at arch/arm64/kernel/cpufeature.c:886! [ 362.292023] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP [ 362.292400] Modules linked in: [ 362.292970] CPU: 3 PID: 0 Comm: swapper/3 Kdump: loaded Not tainted 5.6.0-rc4-00003-gc780b890948a #105 [ 362.293136] Hardware name: Foundation-v8A (DT) [ 362.293382] pstate: 200001c5 (nzCv dAIF -PAN -UAO) [ 362.294063] pc : has_cpuid_feature+0xf0/0x348 [ 362.294177] lr : verify_local_elf_hwcaps+0x84/0xe8 [ 362.294280] sp : ffff800011b1bf60 [ 362.294362] x29: ffff800011b1bf60 x28: 0000000000000000 [ 362.294534] x27: 0000000000000000 x26: 0000000000000000 [ 362.294631] x25: 0000000000000000 x24: ffff80001189a25c [ 362.294718] x23: 0000000000000000 x22: 0000000000000000 [ 362.294803] x21: ffff8000114aa018 x20: ffff800011156a00 [ 362.294897] x19: ffff800010c944a0 x18: 0000000000000004 [ 362.294987] x17: 0000000000000000 x16: 0000000000000000 [ 362.295073] x15: 00004e53b831ae3c x14: 00004e53b831ae3c [ 362.295165] x13: 0000000000000384 x12: 0000000000000000 [ 362.295251] x11: 0000000000000000 x10: 00400032b5503510 [ 362.295334] x9 : 0000000000000000 x8 : ffff800010c7e204 [ 362.295426] x7 : 00000000410fd0f0 x6 : 0000000000000001 [ 362.295508] x5 : 00000000410fd0f0 x4 : 0000000000000000 [ 362.295592] x3 : 0000000000000000 x2 : ffff8000100939d8 [ 362.295683] x1 : 0000000000180420 x0 : 0000000000180480 [ 362.296011] Call trace: [ 362.296257] has_cpuid_feature+0xf0/0x348 [ 362.296350] verify_local_elf_hwcaps+0x84/0xe8 [ 362.296424] check_local_cpu_capabilities+0x44/0x128 [ 362.296497] secondary_start_kernel+0xf4/0x188 [ 362.296998] Code: 52805001 72a00301 6b01001f 54000ec0 (d4210000) [ 362.298652] SMP: stopping secondary CPUs [ 362.300615] Starting crashdump kernel... [ 362.301168] Bye! [ 0.000000] Booting Linux on physical CPU 0x0000000003 [0x410fd0f0] [ 0.000000] Linux version 5.6.0-rc4-00003-gc780b890948a (crimar01@e120937-lin) (gcc version 8.3.0 (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36))) #105 SMP PREEMPT Fri Mar 6 17:00:42 GMT 2020 [ 0.000000] Machine model: Foundation-v8A [ 0.000000] earlycon: pl11 at MMIO 0x000000001c090000 (options '') [ 0.000000] printk: bootconsole [pl11] enabled ..... [ 0.138024] rcu: Hierarchical SRCU implementation. [ 0.153472] its@2f020000: unable to locate ITS domain [ 0.154078] its@2f020000: Unable to locate ITS domain [ 0.157541] EFI services will not be available. [ 0.175395] smp: Bringing up secondary CPUs ... [ 0.209182] psci: failed to boot CPU1 (-22) [ 0.209377] CPU1: failed to boot: -22 [ 0.274598] Detected PIPT I-cache on CPU2 [ 0.278707] GICv3: CPU2: found redistributor 1 region 0:0x000000002f120000 [ 0.285212] CPU2: Booted secondary processor 0x0000000001 [0x410fd0f0] [ 0.369053] Detected PIPT I-cache on CPU3 [ 0.372947] GICv3: CPU3: found redistributor 2 region 0:0x000000002f140000 [ 0.378664] CPU3: Booted secondary processor 0x0000000002 [0x410fd0f0] [ 0.401707] smp: Brought up 1 node, 3 CPUs [ 0.404057] SMP: Total of 3 processors activated. Make crash_smp_send_stop() account also for the online status of the calling CPU while evaluating how many CPUs are effectively online: this way the right number of STOPs is sent and all other stopped-cores's registers are properly saved. Fixes: 78fd584cdec05 ("arm64: kdump: implement machine_crash_shutdown()") Acked-by: Mark Rutland <[email protected]> Signed-off-by: Cristian Marussi <[email protected]> Signed-off-by: Will Deacon <[email protected]>
2020-03-17arm64: smp: fix smp_send_stop() behaviourCristian Marussi1-3/+14
On a system with only one CPU online, when another one CPU panics while starting-up, smp_send_stop() will fail to send any STOP message to the other already online core, resulting in a system still responsive and alive at the end of the panic procedure. [ 186.700083] CPU3: shutdown [ 187.075462] CPU2: shutdown [ 187.162869] CPU1: shutdown [ 188.689998] ------------[ cut here ]------------ [ 188.691645] kernel BUG at arch/arm64/kernel/cpufeature.c:886! [ 188.692079] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP [ 188.692444] Modules linked in: [ 188.693031] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 5.6.0-rc4-00001-g338d25c35a98 #104 [ 188.693175] Hardware name: Foundation-v8A (DT) [ 188.693492] pstate: 200001c5 (nzCv dAIF -PAN -UAO) [ 188.694183] pc : has_cpuid_feature+0xf0/0x348 [ 188.694311] lr : verify_local_elf_hwcaps+0x84/0xe8 [ 188.694410] sp : ffff800011b1bf60 [ 188.694536] x29: ffff800011b1bf60 x28: 0000000000000000 [ 188.694707] x27: 0000000000000000 x26: 0000000000000000 [ 188.694801] x25: 0000000000000000 x24: ffff80001189a25c [ 188.694905] x23: 0000000000000000 x22: 0000000000000000 [ 188.694996] x21: ffff8000114aa018 x20: ffff800011156a38 [ 188.695089] x19: ffff800010c944a0 x18: 0000000000000004 [ 188.695187] x17: 0000000000000000 x16: 0000000000000000 [ 188.695280] x15: 0000249dbde5431e x14: 0262cbe497efa1fa [ 188.695371] x13: 0000000000000002 x12: 0000000000002592 [ 188.695472] x11: 0000000000000080 x10: 00400032b5503510 [ 188.695572] x9 : 0000000000000000 x8 : ffff800010c80204 [ 188.695659] x7 : 00000000410fd0f0 x6 : 0000000000000001 [ 188.695750] x5 : 00000000410fd0f0 x4 : 0000000000000000 [ 188.695836] x3 : 0000000000000000 x2 : ffff8000100939d8 [ 188.695919] x1 : 0000000000180420 x0 : 0000000000180480 [ 188.696253] Call trace: [ 188.696410] has_cpuid_feature+0xf0/0x348 [ 188.696504] verify_local_elf_hwcaps+0x84/0xe8 [ 188.696591] check_local_cpu_capabilities+0x44/0x128 [ 188.696666] secondary_start_kernel+0xf4/0x188 [ 188.697150] Code: 52805001 72a00301 6b01001f 54000ec0 (d4210000) [ 188.698639] ---[ end trace 3f12ca47652f7b72 ]--- [ 188.699160] Kernel panic - not syncing: Attempted to kill the idle task! [ 188.699546] Kernel Offset: disabled [ 188.699828] CPU features: 0x00004,20c02008 [ 188.700012] Memory Limit: none [ 188.700538] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]--- [root@arch ~]# echo Helo Helo [root@arch ~]# cat /proc/cpuinfo | grep proce processor : 0 Make smp_send_stop() account also for the online status of the calling CPU while evaluating how many CPUs are effectively online: this way, the right number of STOPs is sent, so enforcing a proper freeze of the system at the end of panic even under the above conditions. Fixes: 08e875c16a16c ("arm64: SMP support") Reported-by: Dave Martin <[email protected]> Acked-by: Mark Rutland <[email protected]> Signed-off-by: Cristian Marussi <[email protected]> Signed-off-by: Will Deacon <[email protected]>
2020-03-17arm64: perf: Add support for ARMv8.5-PMU 64-bit countersAndrew Murray1-16/+71
At present ARMv8 event counters are limited to 32-bits, though by using the CHAIN event it's possible to combine adjacent counters to achieve 64-bits. The perf config1:0 bit can be set to use such a configuration. With the introduction of ARMv8.5-PMU support, all event counters can now be used as 64-bit counters. Let's enable 64-bit event counters where support exists. Unless the user sets config1:0 we will adjust the counter value such that it overflows upon 32-bit overflow. This follows the same behaviour as the cycle counter which has always been (and remains) 64-bits. Signed-off-by: Andrew Murray <[email protected]> Reviewed-by: Suzuki K Poulose <[email protected]> [Mark: fix ID field names, compare with 8.5 value] Signed-off-by: Mark Rutland <[email protected]> Signed-off-by: Will Deacon <[email protected]>
2020-03-17arm64: perf: Clean up enable/disable callsRobin Murphy1-52/+35
Reading this code bordered on painful, what with all the repetition and pointless return values. More fundamentally, dribbling the hardware enables and disables in one bit at a time incurs needless system register overhead for chained events and on reset. We already use bitmask values for the KVM hooks, so consolidate all the register accesses to match, and make a reasonable saving in both source and object code. Signed-off-by: Robin Murphy <[email protected]> Signed-off-by: Will Deacon <[email protected]>
2020-03-17arm64/kernel: Simplify __cpu_up() by bailing out earlyGavin Shan1-42/+37
The function __cpu_up() is invoked to bring up the target CPU through the backend, PSCI for example. The nested if statements won't be needed if we bail out early on the following two conditions where the status won't be checked. The code looks simplified in that case. * Error returned from the backend (e.g. PSCI) * The target CPU has been marked as onlined Signed-off-by: Gavin Shan <[email protected]> Signed-off-by: Catalin Marinas <[email protected]> Reviewed-by: Mark Rutland <[email protected]>
2020-03-17arm64: remove redundant blank for '=' operator韩科才1-1/+1
remove redundant blank for '=' operator, it may be more elegant. Signed-off-by: hankecai <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-17arm64: kexec_file: Fixed code style.Li Tao1-1/+1
Remove unnecessary blank. Signed-off-by: Li Tao <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-17arm64: add blank after 'if'Zheng Wei1-1/+1
add blank after 'if' for armv8_deprecated_init() to make it comply with kernel coding style. Signed-off-by: Zheng Wei <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-16arm64: BTI: Reset BTYPE when skipping emulated instructionsDave Martin1-0/+2
Since normal execution of any non-branch instruction resets the PSTATE BTYPE field to 0, so do the same thing when emulating a trapped instruction. Branches don't trap directly, so we should never need to assign a non-zero value to BTYPE here. Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Dave Martin <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-16arm64: traps: Shuffle code to eliminate forward declarationsDave Martin1-52/+55
Hoist the IT state handling code earlier in traps.c, to avoid accumulating forward declarations. No functional change. Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Dave Martin <[email protected]> Reviewed-by: Kees Cook <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-16arm64: unify native/compat instruction skippingDave Martin1-10/+8
Skipping of an instruction on AArch32 works a bit differently from AArch64, mainly due to the different CPSR/PSTATE semantics. Currently arm64_skip_faulting_instruction() is only suitable for AArch64, and arm64_compat_skip_faulting_instruction() handles the IT state machine but is local to traps.c. Since manual instruction skipping implies a trap, it's a relatively slow path. So, make arm64_skip_faulting_instruction() handle both compat and native, and get rid of the arm64_compat_skip_faulting_instruction() special case. Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Dave Martin <[email protected]> Reviewed-by: Kees Cook <[email protected]> Reviewed-by: Mark Rutland <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-16arm64: BTI: Decode BYTPE bits when printing PSTATEDave Martin1-2/+15
The current code to print PSTATE symbolically when generating backtraces etc., does not include the BYTPE field used by Branch Target Identification. So, decode BYTPE and print it too. In the interests of human-readability, print the classes of BTI matched. The symbolic notation, BYTPE (PSTATE[11:10]) and permitted classes of subsequent instruction are: -- (BTYPE=0b00): any insn jc (BTYPE=0b01): BTI jc, BTI j, BTI c, PACIxSP -c (BYTPE=0b10): BTI jc, BTI c, PACIxSP j- (BTYPE=0b11): BTI jc, BTI j Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Dave Martin <[email protected]> Reviewed-by: Kees Cook <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-16arm64: elf: Enable BTI at exec based on ELF program propertiesDave Martin1-0/+19
For BTI protection to be as comprehensive as possible, it is desirable to have BTI enabled from process startup. If this is not done, the process must use mprotect() to enable BTI for each of its executable mappings, but this is painful to do in the libc startup code. It's simpler and more sound to have the kernel do it instead. To this end, detect BTI support in the executable (or ELF interpreter, as appropriate), via the NT_GNU_PROGRAM_PROPERTY_TYPE_0 note, and tweak the initial prot flags for the process' executable pages to include PROT_BTI as appropriate. Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Dave Martin <[email protected]> Reviewed-by: Kees Cook <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-16arm64: Basic Branch Target Identification supportDave Martin7-1/+88
This patch adds the bare minimum required to expose the ARMv8.5 Branch Target Identification feature to userspace. By itself, this does _not_ automatically enable BTI for any initial executable pages mapped by execve(). This will come later, but for now it should be possible to enable BTI manually on those pages by using mprotect() from within the target process. Other arches already using the generic mman.h are already using 0x10 for arch-specific prot flags, so we use that for PROT_BTI here. For consistency, signal handler entry points in BTI guarded pages are required to be annotated as such, just like any other function. This blocks a relatively minor attack vector, but comforming userspace will have the annotations anyway, so we may as well enforce them. Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Dave Martin <[email protected]> Reviewed-by: Catalin Marinas <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-11arm64: entry: unmask IRQ in el0_sp()Mark Rutland1-1/+1
Currently, the EL0 SP alignment handler masks IRQs unnecessarily. It does so due to historic code sharing of the EL0 SP and PC alignment handlers, and branch predictor hardening applicable to the EL0 SP handler. We began masking IRQs in the EL0 SP alignment handler in commit: 5dfc6ed27710c42c ("arm64: entry: Apply BP hardening for high-priority synchronous exception") ... as this shared code with the EL0 PC alignment handler, and branch predictor hardening made it necessary to disable IRQs for early parts of the EL0 PC alignment handler. It was not necessary to mask IRQs during EL0 SP alignment exceptions, but it was not considered harmful to do so. This masking was carried forward into C code in commit: 582f95835a8fc812 ("arm64: entry: convert el0_sync to C") ... where the SP/PC cases were split into separate handlers, and the masking duplicated. Subsequently the EL0 PC alignment handler was refactored to perform branch predictor hardening before unmasking IRQs, in commit: bfe298745afc9548 ("arm64: entry-common: don't touch daif before bp-hardening") ... but the redundant masking of IRQs was not removed from the EL0 SP alignment handler. Let's do so now, and make it interruptible as with most other synchronous exception handlers. Signed-off-by: Mark Rutland <[email protected]> Cc: Will Deacon <[email protected]> Signed-off-by: Catalin Marinas <[email protected]> Reviewed-by: James Morse <[email protected]>
2020-03-11arm64: Mark call_smc_arch_workaround_1 as __maybe_unusedNathan Chancellor1-1/+1
When building allnoconfig: arch/arm64/kernel/cpu_errata.c:174:13: warning: unused function 'call_smc_arch_workaround_1' [-Wunused-function] static void call_smc_arch_workaround_1(void) ^ 1 warning generated. Follow arch/arm and mark this function as __maybe_unused. Fixes: 4db61fef16a1 ("arm64: kvm: Modernize __smccc_workaround_1_smc_start annotations") Signed-off-by: Nathan Chancellor <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-11arm64: entry-ftrace.S: Fix missing argument for CONFIG_FUNCTION_GRAPH_TRACER=yKunihiko Hayashi1-1/+1
Missing argument of another SYM_INNER_LABEL() breaks build for CONFIG_FUNCTION_GRAPH_TRACER=y. Fixes: e2d591d29d44 ("arm64: entry-ftrace.S: Convert to modern annotations for assembly functions") Signed-off-by: Kunihiko Hayashi <[email protected]> Signed-off-by: Catalin Marinas <[email protected]> Acked-by: Mark Brown <[email protected]>
2020-03-09arm64: efi: add efi-entry.o to targets instead of extra-$(CONFIG_EFI)Masahiro Yamada1-1/+1
efi-entry.o is built on demand for efi-entry.stub.o, so you do not have to repeat $(CONFIG_EFI) here. Adding it to 'targets' is enough. Signed-off-by: Masahiro Yamada <[email protected]> Signed-off-by: Catalin Marinas <[email protected]> Acked-by: Ard Biesheuvel <[email protected]> Reviewed-by: Vincenzo Frascino <[email protected]>
2020-03-09arm64: vdso32: Convert to modern assembler annotationsMark Brown1-15/+8
In an effort to clarify and simplify the annotation of assembly functions new macros have been introduced. These replace ENTRY and ENDPROC with two different annotations for normal functions and those with unusual calling conventions. Use these for the compat VDSO, allowing us to drop the custom ARM_ENTRY() and ARM_ENDPROC() macros. Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-09arm64: vdso: Convert to modern assembler annotationsMark Brown1-2/+2
In an effort to clarify and simplify the annotation of assembly functions new macros have been introduced. These replace ENTRY and ENDPROC with two different annotations for normal functions and those with unusual calling conventions. Convert the assembly function in the arm64 VDSO to the new macros. Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-09arm64: sdei: Annotate SDEI entry points using new style annotationsMark Brown1-6/+6
In an effort to clarify and simplify the annotation of assembly functions new macros have been introduced. These replace ENTRY and ENDPROC with two different annotations for normal functions and those with unusual calling conventions. The SDEI entry points are currently annotated as normal functions but are called from non-kernel contexts with non-standard calling convention and should therefore be annotated as such so do so. Signed-off-by: Mark Brown <[email protected]> Acked-by: James Morse <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-09arm64: kvm: Modernize __smccc_workaround_1_smc_start annotationsMark Brown1-8/+6
In an effort to clarify and simplify the annotation of assembly functions in the kernel new macros have been introduced. These replace ENTRY and ENDPROC with separate annotations for standard C callable functions, data and code with different calling conventions. Using these for __smccc_workaround_1_smc is more involved than for most symbols as this symbol is annotated quite unusually, rather than just have the explicit symbol we define _start and _end symbols which we then use to compute the length. This does not play at all nicely with the new style macros. Instead define a constant for the size of the function and use that in both the C code and for .org based size checks in the assembly code. Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Catalin Marinas <[email protected]> Acked-by: Marc Zyngier <[email protected]>
2020-03-09arm64: kvm: Modernize annotation for __bp_harden_hyp_vecsMark Brown1-1/+1
We have recently introduced new macros for annotating assembly symbols for things that aren't C functions, SYM_CODE_START() and SYM_CODE_END(), in an effort to clarify and simplify our annotations of assembly files. Using these for __bp_harden_hyp_vecs is more involved than for most symbols as this symbol is annotated quite unusually as rather than just have the explicit symbol we define _start and _end symbols which we then use to compute the length. This does not play at all nicely with the new style macros. Since the size of the vectors is a known constant which won't vary the simplest thing to do is simply to drop the separate _start and _end symbols and just use a #define for the size. Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Catalin Marinas <[email protected]> Acked-by: Marc Zyngier <[email protected]>
2020-03-09arm64: kernel: Convert to modern annotations for assembly dataMark Brown2-6/+10
In an effort to clarify and simplify the annotation of assembly functions in the kernel new macros have been introduced. These include specific annotations for the start and end of data, update symbols for data to use these. Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-09arm64: head: Annotate stext and preserve_boot_args as codeMark Brown1-4/+4
In an effort to clarify and simplify the annotation of assembly functions new macros have been introduced. These replace ENTRY and ENDPROC with two different annotations for normal functions and those with unusual calling conventions. Neither stext nor preserve_boot_args is called with the usual AAPCS calling conventions and they should therefore be annotated as code. Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-09arm64: head.S: Convert to modern annotations for assembly functionsMark Brown1-28/+28
In an effort to clarify and simplify the annotation of assembly functions in the kernel new macros have been introduced. These replace ENTRY and ENDPROC and also add a new annotation for static functions which previously had no ENTRY equivalent. Update the annotations in the core kernel code to the new macros. Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-09arm64: ftrace: Modernise annotation of return_to_handlerMark Brown1-2/+2
In an effort to clarify and simplify the annotation of assembly functions new macros have been introduced. These replace ENTRY and ENDPROC with two different annotations for normal functions and those with unusual calling conventions. return_to_handler does entertaining things with LR so doesn't follow the usual C conventions and should therefore be annotated as code rather than a function. Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-09arm64: ftrace: Correct annotation of ftrace_caller assemblyMark Brown1-8/+8
In an effort to clarify and simplify the annotation of assembly functions new macros have been introduced. These replace ENTRY and ENDPROC with two different annotations for normal functions and those with unusual calling conventions. The patchable function entry versions of ftrace_*_caller don't follow the usual AAPCS rules, pushing things onto the stack which they don't clean up, and therefore should be annotated as code rather than functions. Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-09arm64: entry-ftrace.S: Convert to modern annotations for assembly functionsMark Brown1-14/+14
In an effort to clarify and simplify the annotation of assembly functions in the kernel new macros have been introduced. These replace ENTRY and ENDPROC and also add a new annotation for static functions which previously had no ENTRY equivalent. Update the annotations in the core kernel code to the new macros. Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-09arm64: entry: Additional annotation conversions for entry.SMark Brown1-8/+8
In an effort to clarify and simplify the annotation of assembly functions in the kernel new macros have been introduced. These replace ENTRY and ENDPROC with separate annotations for standard C callable functions, data and code with different calling conventions. Update the remaining annotations in the entry.S code to the new macros. Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-09arm64: entry: Annotate ret_from_fork as codeMark Brown1-2/+2
In an effort to clarify and simplify the annotation of assembly functions new macros have been introduced. These replace ENTRY and ENDPROC with two different annotations for normal functions and those with unusual calling conventions. ret_from_fork is not a normal C function and should therefore be annotated as code. Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-09arm64: entry: Annotate vector table and handlers as codeMark Brown1-38/+38
In an effort to clarify and simplify the annotation of assembly functions new macros have been introduced. These replace ENTRY and ENDPROC with two different annotations for normal functions and those with unusual calling conventions. The vector table and handlers aren't normal C style code so should be annotated as CODE. Signed-off-by: Mark Brown <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-06arm64: use activity monitors for frequency invarianceIonela Voinescu2-0/+184
The Frequency Invariance Engine (FIE) is providing a frequency scaling correction factor that helps achieve more accurate load-tracking. So far, for arm and arm64 platforms, this scale factor has been obtained based on the ratio between the current frequency and the maximum supported frequency recorded by the cpufreq policy. The setting of this scale factor is triggered from cpufreq drivers by calling arch_set_freq_scale. The current frequency used in computation is the frequency requested by a governor, but it may not be the frequency that was implemented by the platform. This correction factor can also be obtained using a core counter and a constant counter to get information on the performance (frequency based only) obtained in a period of time. This will more accurately reflect the actual current frequency of the CPU, compared with the alternative implementation that reflects the request of a performance level from the OS. Therefore, implement arch_scale_freq_tick to use activity monitors, if present, for the computation of the frequency scale factor. The use of AMU counters depends on: - CONFIG_ARM64_AMU_EXTN - depents on the AMU extension being present - CONFIG_CPU_FREQ - the current frequency obtained using counter information is divided by the maximum frequency obtained from the cpufreq policy. While it is possible to have a combination of CPUs in the system with and without support for activity monitors, the use of counters for frequency invariance is only enabled for a CPU if all related CPUs (CPUs in the same frequency domain) support and have enabled the core and constant activity monitor counters. In this way, there is a clear separation between the policies for which arch_set_freq_scale (cpufreq based FIE) is used, and the policies for which arch_scale_freq_tick (counter based FIE) is used to set the frequency scale factor. For this purpose, a late_initcall_sync is registered to trigger validation work for policies that will enable or disable the use of AMU counters for frequency invariance. If CONFIG_CPU_FREQ is not defined, the use of counters is enabled on all CPUs only if all possible CPUs correctly support the necessary counters. Signed-off-by: Ionela Voinescu <[email protected]> Reviewed-by: Lukasz Luba <[email protected]> Acked-by: Sudeep Holla <[email protected]> Cc: Sudeep Holla <[email protected]> Cc: Will Deacon <[email protected]> Cc: Catalin Marinas <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-06arm64: add support for the AMU extension v1Ionela Voinescu1-0/+66
The activity monitors extension is an optional extension introduced by the ARMv8.4 CPU architecture. This implements basic support for version 1 of the activity monitors architecture, AMUv1. This support includes: - Extension detection on each CPU (boot, secondary, hotplugged) - Register interface for AMU aarch64 registers Signed-off-by: Ionela Voinescu <[email protected]> Reviewed-by: Valentin Schneider <[email protected]> Cc: Mark Rutland <[email protected]> Cc: Marc Zyngier <[email protected]> Cc: Suzuki K Poulose <[email protected]> Cc: Will Deacon <[email protected]> Cc: Catalin Marinas <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-04arm64: remove gratuitious/stray .ltorg stanzasRemi Denis-Courmont2-3/+0
There are no applicable literals above them. Acked-by: Mark Rutland <[email protected]> Signed-off-by: Remi Denis-Courmont <[email protected]> Signed-off-by: Catalin Marinas <[email protected]>
2020-03-02arm64: perf: Support new DT compatiblesRobin Murphy1-0/+56
Add support for matching the new PMUs. For now, this just wires them up as generic PMUv3 such that people writing DTs for new SoCs can do the right thing, and at least have architectural and raw events be usable. We can come back and fill in event maps for sysfs and/or perf tools at a later date. Acked-by: Mark Rutland <[email protected]> Signed-off-by: Robin Murphy <[email protected]> Signed-off-by: Will Deacon <[email protected]>
2020-03-02arm64: perf: Refactor PMU init callbacksRobin Murphy1-97/+27
The PMU init callbacks are already drowning in boilerplate, so before doubling the number of supported PMU models, give it a sensible refactor to significantly reduce the bloat, both in source and object code. Although nobody uses non-default sysfs attributes today, there's minimal impact to preserving the notion that maybe, some day, somebody might, so we may as well keep up appearances. Acked-by: Mark Rutland <[email protected]> Signed-off-by: Robin Murphy <[email protected]> Signed-off-by: Will Deacon <[email protected]>
2020-02-29efi/arm64: Clean EFI stub exit code from cache instead of avoiding itArd Biesheuvel2-15/+15
Commit 9f9223778 ("efi/libstub/arm: Make efi_entry() an ordinary PE/COFF entrypoint") modified the handover code written in assembler, and for maintainability, aligned the logic with the logic used in the 32-bit ARM version, which is to avoid cache maintenance on the remaining instructions in the subroutine that will be executed with the MMU and caches off, and instead, branch into the relocated copy of the kernel image. However, this assumes that this copy is executable, and this means we expect EFI_LOADER_DATA regions to be executable as well, which is not a reasonable assumption to make, even if this is true for most UEFI implementations today. So change this back, and add a __clean_dcache_area_poc() call to cover the remaining code in the subroutine. While at it, switch the other call site over to __clean_dcache_area_poc() as well, and clean up the terminology in comments to avoid using 'flush' in the context of cache maintenance. Also, let's switch to the new style asm annotations. Signed-off-by: Ard Biesheuvel <[email protected]> Signed-off-by: Ingo Molnar <[email protected]> Cc: [email protected] Cc: Ingo Molnar <[email protected]> Cc: Thomas Gleixner <[email protected]> Cc: David Hildenbrand <[email protected]> Cc: Heinrich Schuchardt <[email protected]> Cc: Tom Lendacky <[email protected]> Link: https://lore.kernel.org/r/[email protected]
Powered by cgit, Gruvboxed by Blaster4385.