Progress Report: Linux 7.2

Linux 7.2 has been released! That was fast. Let’s dive in to yet another Asahi Linux progress report. We’ve got a lot of interesting developments for you today, so make yourself a cuppa and enjoy.

Think Different… again

The Apple Silicon platform’s power management infrastructure is complicated. Responsibilities are divided between multiple hardware blocks, including the SMC, PMGR, and PMP, all of which have featured before on this blog. While supporting these blocks is important for power use, one of the biggest obstacles to improving battery life has been the application cores themselves.

There are multiple ways to “sleep” a CPU core, and each one should be used in a specific context. The most basic way to sleep an ARM CPU core is to use a Wait For Interrupt (WFI) instruction. This tells the core to stop doing things until it is woken up by an interrupt from an interrupt source. While this does save power by virtue of stopping the core from executing any code, the core stays powered up and retains enough state for it to resume work extremely quickly. As such, WFI is typically only used for parking a core on a running system. Apple cores include a “deep” WFI mode, which shuts down more of the core at the expense of losing its state. Our downstream cpuidle driver operates by setting WFI up in this mode, saving the core’s state, then issuing a WFI loop.

Vendor-specific power management oddities like this are quite common. Thankfully for kernel maintainers, there is a standard way to deal with them: the Power State Coordination Interface. PSCI defines a standard iterface that allows an operating system to call into a defined set of CPU core power management functions implemented by system firmware, including to prepare them for sleep.

To avoid a proliferation of vendor-specific power management hacks inside the Linux kernel, the maintainers of the arm64 arch-specific code have mandated that all upstream hardware must use PSCI for power management. As such, we are not able to upstream our Apple-specific cpuidle driver. Why are we still using it then?

PSCI defines “conduits” through which calls to firmware are to be dispatched from the kernel. The two currently supported conduits in the kernel are the SMC (Secure Monitor Call) and HVC (Hypervisor Call) instructions, which are used to yield execution to a higher Exception Level. The Linux kernel expects to be running at EL2, which means that its PSCI calls must yield to firmware running in EL3. Except Apple’s cores do not implement EL3…

With the kernel already running in EL2 and no firmware running in EL3 to talk to, we are a bit stuck. Linux cannot issue an SMC or HVC instruction since there is no EL3 to yield execution to, which means we cannot make use of PSCI. Being able to properly power-manage the CPU cores is vital for battery life and efficiency, so the status quo simply will not do. One quick and dirty solution would be to have m1n1 load the kernel into EL1, and host a PSCI implementation in EL2. While this would theoretically work, it would also break a lot of architectural features, such as virtualisation. There must be something else we can do…

If you think about it, m1n1 is almost like our own firmware for Apple Silicon. mBoot (formerly iBoot) starts it in EL2, it does its job, then jumps to whatever payload is attached to it. m1n1 does not reserve any memory for itself and does not have any code that must stay resident, so its payload is free to reclaim and overwrite that memory.

On production Asahi Linux systems, m1n1 loads U-Boot rather than the kernel directly. We do this to make use of U-Boot’s UEFI implementation, allowing distros and users to utilise whichever standard UEFI bootloader (GRUB, systemd-boot, etc.) they want. UEFI also provides another feature, Runtime Services. Much as the BIOS interrupts of old did, UEFI Runtime Services provide a way for the operating system to access code originating in system firmware.

Reading the PSCI standard as published by Arm, one will notice that it deliberately defines the API without reference to any specific conduit, and only lists SMC and HVC as examples. If we take a broad interpretation of this, we could conclude that this means other conduits are allowed by the spec…

To this end, Sven has been working on implementing a UEFI Runtime Service based PSCI conduit. With m1n1’s memory region carved out like other firmware regions, this will allow the kernel to call back into it for PSCI services, even though it is running at the same Exception Level. Sven has already modified m1n1 to reserve its memory and leave behind a PSCI implementation, and the patches to the kernel enabling its use are already on the mailing list as an RFC!

Please stop Thinking Different

Given that the cpuidle situation saw no progress until very recently, one might assume that some event has catalysed work in this space. One would be correct.

The ARM specification mandates that cores in WFI loops should preserve all state. This is not the default mode on Apple Silicon. On M1 through M3 series SoCs, state retention can be configured on a per-core basis using chicken bits.

Due to a number of reasons that are not worth mentioning, Apple now sets each core’s chicken bits in mBoot and then locks down the registers controlling them starting with the M4 series. This makes our life a little easier as m1n1 now has marginally less work to do, however it also means that we cannot fine tune low level CPU behaviour. This is an issue on M4 particularly, as calling WFI causes the core to lose its state and crash whatever was running on it.

Yurkea noticed this while doing M4 bringup work, and added a kernel command line parameter to make idle loop behaviour configurable. The parameter allows us to tell the kernel how it should park cores in idle loops, including by doing a basic no-op loop. This prevents M4 machines from crashing during early kernel initialisation, before our cpuidle driver has been loaded. Once the driver takes over, it saves the the lost state before issuing WFI. The patches to enable this are already in linux-next.

They won’t stop Thinking Different

Apple takes its reputation for platform security very seriously. As such, a lot of engineering effort goes in to features that make exploiting vulnerabilities in their ecosystem infeasible for all but the most sophisticated of attackers. One such feature is the Secure Page Table Monitor, or SPTM.

Traditionally, the operating system kernel has been directly responsible for managing memory. This includes handling memory allocations, virtual mappings to physical addresses, and MMU/IOMMU management. A vulnerability in the code responsible for these operations could give an attacker access to the platform’s entire address space. In other words, you’re cooked.

Naturally this makes memory management code a very common target for attackers, and given that its job is fundamentally insecure (applications may want arbitrary allocations for just about anything) it is incredibly difficult to lock down correctly.

Many years ago, Apple introduced the Page Protection Layer to XNU. PPL uses Apple’s hardware security features to isolate pagetable management from the rest of the kernel at the hardware level. This works very well, however attackers eventually caught up and found ways inside PPL that allowed them full access to the system.

Apple has had similar issues with IOMobileFramebuffer in the past. As mentioned on previous blog posts, Apple (mostly) solved the IOMFB problem by putting it inside DCP’s firmware, behind an IOMMU. Nothing in macOS userspace nor in XNU can reach it, except via a defined set of IPC functions. Taking inspiration from this approach, PPL eventually morphed into SPTM. SPTM takes PPL and places it inside Apple’s Guarded Execution Framework (GXF), a set of Exception Levels that run parallel to the standard ARM64 EL1 and EL2 (there is no GL0). GXF also comes with SPRR, a custom pagetable permissions system used when the CPU is running code in GL1 or GL2.

When a modern Apple Silicon device starts up, iBoot or mBoot will detect if the configured boot payload is an XNU image. If it is, it will first load SPTM into GL2. There, SPTM sets up pagetables and memory management and locks down control of said features to GL2. XNU then runs, using yet another IPC protocol to talk to SPTM. If XNU cannot successfully contact SPTM, it will panic very early in init and halt the system.

This is problematic for us. If we try to run XNU under the m1n1 hypervisor, it will crash as SPTM has not been loaded. If we configure mBoot to treat m1n1 as an XNU binary, m1n1 will crash as it will be unable to manage memory in the way it needs to.

Given that SPTM is mandatory for XNU on M4 and above, the hypervisor was rendered entirely nonfunctional on these machines. We don’t know when to quit however, which is why that’s was and not is!

SPTM itself is not particularly special. It is an ARM64 Mach-O binary that lives inside the same preboot directory as the OS-specific firmware blobs for the various coprocessors. That means we could theoretically have m1n1 load it already under the hypervisor, load XNU next to it, then watch both of them! But SPTM must be run inside GL2 with SPRR enabled. If only we knew how both of them worked…

Thanks to Sven’s work reverse engineering SPRR and GXF way back when Asahi Linux was first getting started, he was recently able to teach the m1n1 hypervisor how to emulate them! This allows us to load Apple’s SPTM blob in exactly the manner XNU expects, do some quick surgery on the XNU binary, load it, and then monitor MMIO accesses just as we could from M1 to M3! The gory details of how Sven achieved this means that tracing is slower on these machines, however not unusably so. This will allow us to continue bringing up new hardware for the forseeable future!

More M3 progress!

Work has also been progressing on bringing Asahi Linux to M3 series machines.

The webcam image signal processor remained largely unchanged, except for a single skipped initialisation message on the M3 Max specifically. chaos_princess added support for that to the Linux driver, and with that enabled full webcam support on all M3 series devices with a builtin webcam.

The builtin microphones also changed slightly. New to M3 series devices is a “High Frequency” decimator requiring a new set of coefficients and a much larger initialisation message. Once again, chaos_princess managed to get this figured out in no time, bringing microphone support to all equipped M3 series devices.

ATCPHY — the hardware block responsible for negotiating USB3, DisplayPort and Thunderbolt connections over USB Type-C ports — also changed slightly, requiring a new sequence of tunables at initialisation due to the shift to TSMC’s N3 process node.

While our thesis that Apple would avoid making sweeping architectural changes just for the sake of it has mostly held true, we have of course run into a few such changes…

All M1 through base M3 series devices used an Apple-specific Texas Instruments USB port controller called CD3217 (or ACE2). This sits on the I²C bus and negotiates with USB devices when they are plugged in. Starting with M3 Pro/Max, Apple has switched to ACE3. ACE3 instead uses the SPMI bus, which required some more reverse engineering. Thanks to the combined efforts of mildsunrise and chaos_princess, we discovered that ACE3 has pretty much the same register set as CD3217, only wrapped in a SPMI interface instead of addressed over I²C. Both the SPMI interface and ACE3 itself are now working in Asahi Linux, bringing USB 3.0 and Thunderbolt support to all M3 series devices.

A massive change we were expecting was the firmware ABI for both the GPU and display controller. Since the firmware for both AGX and DCP are paired with a specific macOS version, Apple does not need to worry about keeping the interface stable across releases. This is a large reason why we “target” specific macOS versions for each generation of hardware. M3 series machines will target the ABI found in macOS 14.8.3, and support for DCP is now almost at feature parity with the existing macOS 13.5 ABI we use for M1 and M2!

With all of this progress coming on top of milestones already reached on M3, we are pleased to announce that we are almost ready to cut an official release! We will have more to say about this in the coming weeks, so stay tuned!

Don’t forget about M4 and M5 too!

Not content with helping out only with M3, Yureka has also been working on M4 and even early M5 bringup. Beyond the WFI issue on M4, these SoCs suffered from a breaking change in Apple’s NVMe controller firmware in the macOS 15.x firmware bundle. Yureka and Sven worked together on investigating the changes and implementing them in both m1n1 and Linux, so we now have working NVMe on M4 and M5! Yureka also managed to get PCIe working to a state where devices on the bus can be enumerated by Linux, and fixed an issue that caused Linux to crash shortly after boot when more than one CPU core is enabled. Not much else is working on M4 and M5 so we aren’t quite ready to enable them in the Asahi Installer just yet, but as always we will have more to say in due course.

Desktop video is hard

Last time, we announced preliminary support for the Apple Video Decoder, or AVD. This hardware block accelerates H.264 (AVC), H.265 (HEVC) and VP9 video decoding on M1 and M2, in addition to AV1 decoding on M3 machines and above. Since then, AVD support has been further refined by sofus, with AVC, HEVC and VP9 all working mostly reliably on all machines supported by Asahi Linux! We are now at the stage where we are considering desktop integration, which is where things start to get tricky…

The AVD hardware is fundamentally stateless, meaning that all it does is take an encoded frame and turn it into a video buffer. The hardware does no bitstream parsing, no decoding session tracking, or any other management of the decoding pipeline. This lends itself well to the V4L2 Stateless API, which is designed with such decoders in mind. While this API first landed in the upstream kernel over 10 years ago, userspace has been slow to adopt it outside of specialised software targeted at embedded devices. GStreamer has basic support for V4L2 Stateless, however FFmpeg (and everything that consumes it) does not without out-of-tree patches. Desktop-class software, such as web browsers, have historically focused on VA-API, NVDEC and VDPAU. More recently, effort on the desktop has begun shifting to Vulkan Video. This has left V4L2 Stateless effectively abandoned before it even got started for desktop software.

All hope is not lost, however. VA-API is now near universal in desktop-class software owing to its adoption by both AMD and Intel for their video acceleration hardware. To prevent V4L2 Stateless hardware from being unusable with such software, Bootlin authored a VA-API to V4L2 Stateless translation layer. Sadly this has been abandoned for quite some time and no longer builds without patches, however sofus has forked it and gotten it into a working state for AVD. With this translation layer installed and an environment variable set for the login session, software implementing VA-API support is now able to use AVD to accelerate video decoding! This does not yet ship by default in Fedora Asahi Remix, and will not work with Firefox’s video decoding sandbox, however we hope to have something shippable soon.

The road to direct scanout

The traditional benefit of hardware-accelerated video decoding has been the reduction in CPU load. While this alone is of course hugely important, there are other places where power and time is being wasted during video decode. Firstly, the CPU must copy the decoded frame data to GPU memory. The GPU must then composite that frame into the scene as directed by the compositor. The final rendered scene must then be copied to the display controller’s memory, and the display controller programmed to scan out the scene. This must all happen at the playing video’s framerate.

We can do better than this. Linux’s DMA subsystem supports sharing memory regions between multiple devices. In this context, it allows us to eliminate the multiple copies of the same framebuffer data from one region to another. This only works if all hardware blocks in the chain support the same framebuffer format, however. This is where things start to get tricky.

Framebuffers come in all sorts of formats and sizes. Video data for example is almost always stored and transmitted in a semiplanar Y’CbCr format, inspired by the analogue video storage and broadcast standards of the 20th century. On top of this, graphics hardware almost always uses some form of specialised addressing. Pixels are not “next” to each other in memory and are instead tiled to reduce cache misses and other performance issues. Moreover, framebuffers are often compressed too, further reducing memory footprint and pressure on the bus.

All of that to say, eliminating framebuffer copies between hardware blocks is not simple. Each hardware block must agree on the framebuffer’s pixel format, its addressing mode, and any compression applied to it. If they cannot agree on this, copies and conversions must be done in software.

One might be inclined to ask why such “shared” framebuffers can’t just be created with “raw” pixel data. Consider a 1920x1080 framebuffer containing raw 8-bit ARGB pixel data. That is just over 66 MiB of data. At 4K, this balloons to just under 214 MiB. Even without the added penalty of copies, reading and writing this much data at 30, 60, 165, or even 240 frames per second is an enormous strain on the memory bus and therefore an enormous power drain. Even if we somehow had an infinitely performant and efficient memory bus, each hardware block in the chain would still have to have enough local cache or RAM off the bus, as well as the horsepower to process such a volume of data. This is infeasible.

Luckily hardware vendors agree, and most “families” of display hardware solve this problem by implementing support for the same addressing and compression schemes. This allows their 3D engines, display controllers and video accelerators to share memory-efficient framebuffers without having their own massive local caches or constantly hammering the memory bus. On Linux, each driver declares which formats it supports and various parts of the software stack then negotiate and agree on a common format to use for shared buffers. Thankfully for us, Apple chose not to Think too Differently here. Well, mostly.

Apple hardware uses two main addressing and compression formats. AGX, named after the GPU, and Interchange. While the AGX format is used exclusively inside AGX itself, the Interchange format is also supported by DCP and AVD. On macOS, this enables the direct scanout of framebuffers from both AVD and AGX, allowing Quartz (the macOS compositor) to opportunitically choose to do almost nothing in a lot of everyday scenarios. If video is playing and nothing else is happening, the GPU can shut off completely and all the compositor has to do is make sure DCP is displaying new frames as AVD generates them. If a game or application is in fullscreen, the compositor can simply pass DCP the address of that application’s own framebuffer and disable all of its own rendering activities. The good news is that we are now well on the way to being able to replicate this in Asahi Linux!

Thanks to groundwork laid by Alyssa and Lina, the AGX driver already had basic support for Apple’s Interchange framebuffer format, it was simply not yet wired up. Oliver Bestmann and I volunteered to wire up and enable Interchange support in the DCP kernel driver, as well as both the Asahi and Honeykrisp Mesa drivers. Coupled with work already done bringing Y’CbCr and overlay plane support to DCP, this has cleared the way for direct scanout of AGX-rendered framebuffers to DCP! AVD support is on the way, with sofus currently investigating its ability to output Interchange format buffers.

While this is all fantastic news, we are not quite ready to ship this to users just yet. Kwin, the compositor for KDE Plasma, considers our setup “multi GPU” as AGX and DCP are distinct hardware blocks. As such, direct scanout using the DMA-BUF API is completely disabled for now. Kwin devs are actively working on improving this situation, and direct scanout on setups like ours could be enabled in Kwin from as soon as Plasma 6.8!

Sundry upstreaming

As always, we continue to get patches upstreamed. Progress has slowed in recent months, and it may even appear as though we’ve gone backwards. This is only because we have already upstreamed most things! Most of what is left is either enormous work that we cannot upstream yet (e.g. GPU, DCP) or new features that we have only been able to work on due to clearing most of the upstreaming backlog. That said, we still have some odds, sods, bits and bobs sitting in the tree. Recently upstreamed patches include a number of I2S peripheral changes related to enabling speakersafetyd to function, the Devicetree patches enabling the SMC-based hwmon driver, and of course fixes for all of the SMC firmware interface changes introduced by macOS 27.

Thanks again!

Frequent readers will have noticed a number of new names popping up in the blog posts recently. Thanks to the generous support of our GitHub Sponsors and Open Collective backers, we have been able to get these folks the hardware they need to work the magic that they have been. AVD support, M4 bringup, and other goodies we have in store are only possible because of you.