The acpi IORT code incorrectly uses a u_int as the MSI xref type, while
INTRNG expects an intptr_t. Switch to using intptr_t to help document
this value is the xref for the MSI controller.
Reviewed by: dsl
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D60653
Before and after DMA operations we need memory barriers to ensure
stores on the CPU are observed by the device, and stores by the device
are observed by the CPU.
Previously we used a "dsb sy" instruction, however this is too strong
and can lead to slower performance.
Drew Gallatin found and tested the issue with cache-coherent devices,
reducing the barriers to dmb instructions in the outer-shareable
domain. After discussing the issue with the Arm Linux DMA maintainer I
concluded it was safe for non-coherent devices as the only dsb
instructions we need are after cache operations and these already exist.
Replace the dsb before DMA with a "dmb oshst" when cache-coherent, or
the existing dsb after invalidating the cache when not coherent. This
ensures earlier stores have completed before we signal the device.
Replace the dsb after DMA with "dmb oshld" in both cases. This ensures
later loads or stores are ordered after earlier device register loads.
Co-Authored-By: Andrew Gallatin <[email protected]>
Sponsored by: Arm Ltd
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D60407
Use the field names from the Arm Documentation for the MPIDR_EL1
register. These will later be generated from a BSD Licensed JSON file
so to reduce the diff for that rename now.
Reviewed by: kajetan.puchalski_arm.com
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D59176
Some toolchains may not understand the .arch_extension.
Reviewed by: jhb
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D59148
Like armreg.h for system registers add a header to hold macros for
system instructions.
Reviewed by: jhb
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D59147
The GICv5 ACPI code uses CpuInterfaceNumber to as the CPU ID. This a
GICv5 CPU ID and may not be the same as the appropriate FreeBSD value.
It is also possible the target CPU is disabled, e.g. when the hw.ncpu
tunable is uses to limit CPUs. If this is the case we don't want to
enable the CPU in the cpu set as it is offline so cannot handle
interrupts.
Switch to use ArmMpidr to find which pcpu to use when finding which
CPUs the IRS is attached to.
Fixes: 9556306213 ("arm64: Add ACPI support to GICv5 driver")
Differential Revision: https://reviews.freebsd.org/D59993
Sponsored by: Arm Ltd
Only use the MSI polling method when we support MSIs and the SMMU is
cache-coherent.
The SMMU writes to a memory location. If it is not cache-coherent then
the CPU may read the existing value in its cache and miss the signal
the sync operation has completed.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D59489
When the hardware doesn't support MSIs we need to poll to wait for
sync operations to complete. Add support for this to support booting
on this hardware.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D59488
In preparation for adding a non-MSI method split out waiting for sync
completions from smmu_sync.
While here fix the loop to wait for sync timeout to the worst case
Linux uses.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D59487
Handle the Arm MMU-600 erratum 1076982 where when we poll for a sync
command completion we can't use the wait-for-event instruction.
This is in preparation for later adding support for polling for sync
command completion and is not an issue on stable branches.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D59486
We need to wait for the interrupt drivers to attach before the smmu
driver. As both of these happened at BUS_PASS_INTERRUPT +
BUS_PASS_ORDER_MIDDLE there was no guarantee this would happen in the
correct order.
Fix by moving the smmu driver to BUS_PASS_ORDER_LATE.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D59485
They are architecturally defined, so should be in the common location.
Reviewed by: emaste, Kajetan Puchalski <[email protected]>
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D59181
This will be the name of the generated macro. As we only need to check
if this field is set or not use the common ICC_SGI1R_IRM_VAL name.
Reviewed by: Kajetan Puchalski <[email protected]>
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D59179
These macros are used to extract the number of breakpoints and
watchpoints from the register. Rename them to not conflict with the
common meaning of an _VAL macro in this file.
Reviewed by: Kajetan Puchalski <[email protected]>
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D59178
It's unused since 05f6f65c3b ("arm64: add CHECK_CPU_FEAT() for
checking feature support in assembly"). It's been replaces with a
commonly named ID_AA64PFR0_GIC_WIDTH macro.
Reviewed by: Kajetan Puchalski <[email protected]>
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D59177
Use the field names from the Arm Documentation for the ICC_SGI1R_EL1
register. These will later be generated from a BSD Licensed JSON file
so to reduce the diff for that rename now.
Reviewed by: Kajetan Puchalski <[email protected]>
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D59174
Use the field names from the Arm Documentation for the CTR_EL0
register. These will later be generated from a BSD Licensed JSON file
so to reduce the diff for that rename now.
Reviewed by: Kajetan Puchalski <[email protected]>
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D59173
"1 << 31" is a signed int left shift 31 which is undefined as the shift
is too large. Use an unsigned int to make the value defined.
Sponsored by: Arm Ltd
Move vgic_v3 structures in preparation for vgic interface rework for GICv5
support.
Reviewed by: Sarah Walker <[email protected]>
Sponsored by: Arm Ltd
Some ID_AA64ISAR2_EL1 fields values are incorrectly indented. Values
have an extra space before the macro to make scanning for them easier.
Add this extra space to the two fields that were missing it.
Sponsored by: Arm Ltd
When building struct arm64_bootparams to pass to initarm we store the
boot_el field as a 64-bit value, however it is defined as an int which
is 32-bits.
Switch to store using the 'w' register as this will store a correctly
sized value.
Previously this would trash the trailing padding, so is only a
correctness issue.
Reviewed by: emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D58988
Add a workaround for the Arm Cortex-A53 erratum 843419. This has been
targeted when the build is either unoptimised for any CPU/architecture
or targets the Cortex-A53 or ARMv8.0 architecture.
PR: 296240
PR: 296395
Reported by: Hal Murray <[email protected]>
Reported by: Andreas Schuh <[email protected]>
Reviewed by: cognet, mmel
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D58212
This is included via acpivar.h so needs to be in SRCS to be generated.
Reported by: bz
Fixes: bc49842769 ("acpi_einj: Support for ACPI error injection")
Sponsored by: Arm Ltd
Add a driver to support the GICv5 interrupt wire bridge (IWB). The IWB
translates the change in state of an input wire and sends a MSI to
the interrupt translation service (ITS) to be translated to an LPI.
Unlike other MSI sources each wire has a fixed Event ID value it will
write in the MSI data.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D54252
Add a driver to support the GICv5 interrupt translation service (ITS).
The ITS is responsible to handling ITS events & translating them to an
interrupt to be delivered to the interrupt routing service (IRS).
An example event is a MSI or MSI-X is delivered. The ITS will generate
an LPI depending on which device sent the interrupt and the value the
device wrote.
This is a similar concept to the GICv3 ITS, however the implementation
details are different so it needs a new driver.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D54251
Add an initial driver for the GICv5 interrupt controller.
This provides host-only support for the GICv5 interrupt controller. It
is specified in the ARM-AES-0070 document & based on version 00eac0.
In the GICv5 there are 3 interrupt spaces: PPI, SPI, and LPI. Unlike
previous interrupt controllers they don't share a single interrupt
ID range, so PPI IRQ 1 and SPI IRQ 1 are different interrupts. There
is a common irqsrc stricture that encodes this information as it is
common across the interrupt types.
Unlike previous GIC versions there are no software generated interrupts
that can target a configurable collection of CPUs. These have been
replaced with LPIs, where each CPU will have one allocated for each
IPI type.
This driver handles the CPU interface and interrupt routing service
(IRS). The CPU interface provides the interface to manage and handle
interrupts, while the IRS handles routing LPIs and SPIs to the target
CPU.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D54250
A use for it has been found in the GICv5 driver where we need to read
the Interrupt Affinity ID early in the boot on the secondary CPUs.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D54249
Add support to build system instructions from a macro. These are based
on the existing support for msr/mrs instructions with adjustments for
the different instruction format.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D57017
- ARM C1-Premium erratum 4193780
- ARM C1-Ultra erratum 4193780
- ARM Cortex-A76 erratum 4193800
- ARM Cortex-A76AE erratum 4193801
- ARM Cortex-A77 erratum 4193798
- ARM Cortex-A78 erratum 4193791
- ARM Cortex-A78AE erratum 4193793
- ARM Cortex-A78C erratum 4193794
- ARM Cortex-A710 erratum 4193788
- ARM Cortex-X1 erratum 4193791
- ARM Cortex-X1C erratum 4193792
- ARM Cortex-X2 erratum 4193788
- ARM Cortex-X3 erratum 4193786
- ARM Cortex-X4 erratum 4118414
- ARM Cortex-X925 erratum 4193781
- ARM Neoverse-N1 erratum 4193800
- ARM Neoverse-N2 erratum 4193789
- ARM Neoverse-V1 erratum 4193790
- ARM Neoverse-V2 erratum 4193787
- ARM Neoverse-V3 erratum 4193784
- ARM Neoverse-V3AE erratum 4193784
These are all variants on an erratum where TLBI+DSB instructions on
one CPU may incorrectly complete early leading to stores to an updated
address using an incorrect translation on another CPU.
In all cases the workaround is to add a second TLBI+DSB.
Approved by: so
Security: FreeBSD-SA-26:31.arm64
Security: CVE-2025-10263
Sponsored by: Arm Ltd
Some third party software expects these to not conflict. As the MTE
support isn't fully in the tree, and these values aren't in a release
we can renumber them without any backwards compatibility issues.
Sponsored by: Arm Ltd
We need to store some extra information about a page, e.g. the state of
the MTE tags. Add a MD flags field to each page.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D55953
When entering the kernel from userspace we need to check for MTE tag
failures when using asynchronous MTE. This is done by checking if either
tag fault check types that have asynchronous checks are enabled, and
if so check the register the result is stored. It then sets a flag the
kernel can later use to raise a signal.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D55952
Add the same group of functions we use to manage pointer authentication
in userspace threads.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D55951
When MTE is enabled we will use the DMAP to manage tags. To be able to
read/write them we need to change the memory attribute to
VM_MEMATTR_TAGGED.
Support changing the DMAP memory type to values known to have
equivalent cache properties.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D55949
This depended on header pollution only present when FDT is defined. As
FDT isn't check for in this file we can remove opt_platform.h and
include the correct set of vm header files.
Reported by: ivy
Fixes: 76a2904c35 ("arm64: Add RSI detection for CCA")
Sponsored by: Arm Ltd
It should be for FEAT_MTE2 as the registers aren't implemented for
FEAT_MTE.
While here fix the style of the comment explaining the check.
Reported by: markj
Fixes: 58de791536 ("arm64: mte: configure initial state for system registers")
Sponsored by: Arm Ltd
It will be removed soon & replaces with pmap_s1_invalidate_all_kernel.
This allows us to handle errata that cpu_tlb_flushID is missing
workarounds for.
Sponsored by: Arm Ltd
When using outline atomics on arm64 the compiler will create a call to
a function that performs the atomic operation. This allows us to use
the fastest operation depending on the hardware.
As these functions are implemented in libgcc create a linker script
so libraries that link against libgcc_s will include libgcc to pull
them in.
Reviewed by: imp, jhb
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45268
Support changing the property of a DMAP page that holds it's own page
table entry.
Because we need to perform a break-before-make sequence to change the
properties of pages a page that also holds it's own page table entry
will fault in the make part of the sequence.
Handle this by mapping the page with a temporary mapping as we already
do when demoting a superpage.
Reviewed by: kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D55943
When modifying mappings in pmap we may need to perform a
break-before-make sequence. This creates an invalid mapping, then
recreates it with the changes.
When modifying DMAP mappings we may be changing the mapping that
contains its own page table then after breaking the old entry we are
unable to create the new entry.
To fix this create a map that can be used & won't be affected by the
break-before-make sequence.
Reviewed by: kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D56306
Add the tests/sys/arch directory for architecture-specific tests and
use it to add arm64 SVE tests. These test the kernel is managing the
SVE state in a way we expect.
These tests require SVE hardware support to run so will skip when they
can't detect it.
Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43311
Rather than each exception calling userret use a common copy. As
syscallret already calls userret we need to skip it in that case.
Reviewed by: kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D55250
Until an issue seen on amd64 can be investigated restore two mb() calls
to virtio.
Reviewed by: andrew
Fixes: c499ad6f99 ("virtio: Use bus_dma for ring and indirect buffer allocations")
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D55766
It has been reported that the overhead of repeating all TLBI
instructions is too large [1]. The Software Developer Errata Notices
(SDEN) for the relevant Arm CPUs have been updated so a single
"tlbi vale1is, xzr" followed by "dsb ish" is sufficient to work around
the issues.
Replace the places we repeat TLBI instructions with the new sequence.
[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/
Reviewed by: kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D55646
When Top Byte Ignore (TBI) is enabled in a processor we need to pass
a canonical address to the vm subsystem.
Reviewed by: alc, kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D55619
The only new register is read-only. As the kernel just passes the
registers to the guest directly no further change should be needed.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D51764
The mdts value is in terms of the nvme page size, not the host page
size. On many architectures these are both 4k, however on arm64 it is
possible to build a system with the host page size of 16k.
Use NVME_MPS_SHIFT to get the correct nvme page shift.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D55334
The Parallels Desktop Virtio GPU implementation doesn't handle
enqueuing the VIRTIO_GPU_CMD_RESOURCE_ATTACH_BACKING request and
memory list together.
Work around this by splitting them before sending them to be enqueued.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D55147
It now panics when it's unable to find the register, so doesn't need
to return a status.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D55106
It now only returns true so this can be assumed and doesn't need to be
checked.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D55105
This will allow for callers to be simplified & not need to check the
return status.
Keep the return type for now so this can be MFCd without breaking the
KBI.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D55104
These are needed when enabling CPU features so should be processed
earlier in the boot.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D55103
If there is no ITS device so no memory resources the gicv3 driver would
exit attaching early. This caused the vgic driver to also not be
attached, even if it could.
Reported by: novel@
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D54791
Set the Fine-grained trap registers to trap any features we don't
support. These are expected to be more useful when we support nested
virtualisation, so for now just the base features and GICv3 are not
trapped.
As nested virtualisation will require VHE we only set the fine-grained
trap registers when VHE is used.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D54687
Add the macros and detection for Fine-grained traps (FEAT_FGT and
FEAT_FGT2).
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D54686
This allows us to also use the common VIRTIO_SIMPLE_PROBE and to have
devmatch load the driver when detected.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D54684
On arm64 we can use the "dc zva" instruction to zero memory. The CPU
tells software if the instruction is implemented, and if so the size
and alignment it will use.
When the size is 64-bytes the Arm Optimized Routines implementation of
memset can use dc zva to zero memory, and has a build flag to skip
checking.
Use this flag to build a version of memset that will be used when this
assumption is true.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D54776
This allows static binaries to only include the functions they
reference.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D54775
If we update Makefile.inc it may be to change the contents of these
files.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D54774
Some virtualization platforms provide broken configurations. There
is a GIC interrupt controller, however accessing the CPU interface
registers leads to an external data abort. As these are needed to
handle interrupts we are unable to boot further.
Detect this misconfiguration and panic to tell the user the issue.
Reviewed by: emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D54832
This has moved from a generated file in objdir to the source tree.
Remove the old file and any .depend files that reference it.
Reviewed by: bapt (via IRC)
Fixes: 41ccf82b29 ("libc/aarch64: Use MOPS implementations of memcpy/memmove/memset where availble")
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D54729
The SVE signal context needs to be correctly aligned. Fix this by
creating a new macro to calculate the needed size to provide this
alignment, and use it when setting and checking the saved SVE signal
context.
Reported by: cperciva
Reviewed by: cperciva, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D54396
Fix an ACPI only kernel by only building the vchiq files when FDT is
enabled.
Fixes: 745c4aa5e8 ("Make BRCM2837 port conform FreeBSD/ARM64 guidelines")
Sponsored by: Arm Ltd
This may have been called after intr_irq_shuffle. For most interrupt
controllers this appears to be safe, however for the GICv5 we need to
read a per-CPU ID register before we can assign interrupts to a given
CPU.
Fix the race by moving intr_pic_init_secondary earlier in the boot,
after devices have been enumerated and before the interrupts are moved
to their assigned CPUs.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D53685
Some non-PCI devices can send interrupts, e.g. the Arm SMMU or GICv5
Interrupt Wire Bridge. Add support for these by implementing pci_get_id
and pci_alloc_msi and the MSI/MSI-X parts of the PCIB interface.
Only the MSI parts of the PCI interface are added as that is all I am
able to test.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D53330
Use the midr value to ensure we find the correct PCPU pointer on arm64.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D53327
This allows it to be referenced by other devices.
Reviewed by: jhibbits
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D53329
Move ofw_cpu to BUS_PASS_CPU + BUS_PASS_ORDER_MIDDLE so it can be used
by other devices.
Reviewed by: jhibbits
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D53328
We only needed armreg.h for the *_SPECIALREG macros. Use the new
_armreg.h directly.
Reviewed by: emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D53325
We shouldn't need to include armreg.h just to access special registers
that are not defined in this file. Split out the parts that should be
common with arm64.h and hypervisor.h.
Reviewed by: emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D53324
It is needed for INSN_SIZE. Include it directly rather than depending
on header pollution.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D53322
fvp-base-revc.dtb works with the kernel now interrupt-maps are
supported in more cases.
Reviewed by: mhorne
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D51258
Work around an issue when HotPlug is enabled where pcib3 and pcib5
continuously report the following:
pcib3: HotPlug interrupt: 0x30
pcib3: Command Completed
pcib3: HotPlug interrupt: 0x30
pcib3: Command Completed
...
As a workaround disable HotPlug on the N1SDP as it's unlikely anyone
will depend on it.
Reviewed by: br
Fixes: 1f5c50a861 ("pci_host_generic:Add pcib_request_feature on ACPI")
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D53134
Andrew Turner