David Vernet

Ring 0 Linux kernel hacker, scheduler-bug-adder, BPF standardization co-chair. Currently working at Anthropic. If you want to know how it works, break it apart.

[Dario Amodei @DarioAmodei]

We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.

Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.

You can read the full post here: https://t.co/OGyPb7yaYt

[David Vernet @byte_lab]

@zyklotomic Thank you! Not sure yet, but I’d love to be able to work on sched_ext again.

[David Vernet @byte_lab]

I guess now is as good a time as any to announce that I’m starting at Anthropic tomorrow. Super grateful for the opportunity!

[David Vernet @byte_lab]

It’s interesting how one’s relationship to music evolves as you get older. Thirty Three has always been one of my favorite @SmashingPumpkin song, but now I love it primarily because of the lyrics.

“Speak to me in a language I can hear.”

Now that I’m older, I can really hear it.

[David Vernet @byte_lab]

Proud to see this go into the standard, and humbled to be a C++ standards coauthor. Please don’t blame me for the rest of the language, though.

[CPPAlliance @CPPAlliance]

The Linux kernel's most important concurrency primitive has been in production for 20+ years

Readers never block. Writers make a copy, modify the copy, then atomically swap the pointer

C++26 standardizes it as std::rcu 🧵👇

[David Vernet @byte_lab]

Woah indeed, my friend

[Phoronix @phoronix]

Woah! @AMD P-State Linux Driver Patches Can Boost 1%-Low FPS Gaming Performance By 31%

Wild improvement for the Steam Deck and other modern @AMDRyzen hardware benefits too with new proposed "epp_boost" Linux feature for amd_pstate driver.

https://t.co/XuPeFQMnGz

[David Vernet @byte_lab]

@gautshen accidentally cc’d your old AMD email address. Sorry, old habits die hard.

[David Vernet @byte_lab]

Oh yeah by the way I published the AMD pstate patches I was talking about 🤙🏼

[Ars Technica @arstechnica]

New AMD Linux patch boosts low-end gaming performance on Steam Deck https://t.co/JHOdPedkYY

[David Vernet @byte_lab]

@kudureranganath Also worth noting is that the Steam Deck has a shared power budget between the CPU and the GPU. Frequency-scaling decisions really do belong in firmware for that case, in my opinion.

[David Vernet @byte_lab]

Another area I was snooping recently is in the AMD cpufreq driver. There's a common pattern in gaming where you have a main task and a render task that are (largely) serialized on the rendering pipeline; typically with the rendering task blocking on some futex while the GPU does its thing.

You usually always want to jack the frequency up to performance levels on whatever cores they're running on if you can, because these tasks are essentially the embodiment of Amdahl's Law when it comes to frame rendering. In other words, they're serialized and on the critical path for rendering, so if either of them slow down, the entire rendering pipeline is now more likely to result in a stale frame (i.e. a frame being delivered to the compositor after the frame deadline).

The AMD cpufreq governor kind of works against you here, because these tasks are short and bursty, and the firmware cpufreq governor seems to decay its tracked util between frames to the point that, at the start of the next frame when the core is no longer idle, it starts at a low frequency and has to ramp back up to the higher frequency over the course of the frame. The solution, I think, is to have some per-core utilization tracking that writes EPP=performance to MSR_AMD_CPPC_REQ (by default on e.g. the Steam Deck it's EPP=balance_performance) on some sampling cadence if util is high enough over that period, and then drops it back to whatever it was before when the core goes idle.

I tested a local patchset and in the Civ 6 graphics benchmark (on the Steam Deck) it improved 1%-low fps by 31.8% (which is fairly massive), and p99 frame time by 4.1%. Hoping to polish them and send them upstream in the next day or two.

[David Vernet @byte_lab]

Well, it's the classic tradeoff isn't it? Some things the firmware will have a better view of (instruction mix, actual stall/retire behavior, per-CCD current, thermal headroom, etc), and others sched-ext + sched-util will have a better view of (task-placement decisions, uclamp, etc).

I personally still view sched_ext as primarily an experimentation vehicle. It's great to find optimization opportunities, and you can of course also use it for policies that either don't belong or are not yet upstreamed into the fair scheduler, but in an ideal world we would be able to integrate the policies that work best into core parts of the kernel (whether that be a cpufreq driver, the fair scheduler, or something else).

[David Vernet @byte_lab]

Random friends and family have asked me over the last few months and years what I think about immigration in the US.

I'll answer by saying this: there is a permanent exhibit on display at Ellis Island that plays a recording of an interview of my great grandmother who immigrated here from Poland to flee antisemitism. She was recalling one of the logic questions she was asked when making her way through immigration at the island:

Their question: Do you wash stairs starting at the top, or starting at the bottom?
Her answer: I didn't come to America to wash stairs

That is the core of the American spirit. You have the best people coming to the best place. Anecdotally, every single Indian and Chinese person I've met in the US (literally every single one) are highly skilled laborers who pay an exorbitant tax bill directly to the US. The second you turn off that tap, this country just becomes another gaggle of homogeneous underachievers, and I for one won't stand for it.

There is neither a cogent economic, nor moral reason to be against legal immigration. You get either skilled labor that pours fuel on American industry (thanks for making me rich, tech industry), or unskilled labor that drives down costs and raises standards of living.

[David Vernet @byte_lab]

@paulmckrcu Hmmm, how would a single-threaded benchmark help? You need multiple threads to bounce the cache line between CPUs, right?

[David Vernet @byte_lab]

Just noticed yesterday that the first 4 fds (on 64 bit) in the `fd_array` in `struct files_struct` (https://t.co/LkvYT3t0VN) share the same cacheline as the write-heavy fields like `file_lock`. This means that if multiple threads are doing write operations on fds, it can cause false sharing for fds 0-3.

Is this intentional? Looks like it's been this way since ~2005 so I assume so, but I also wonder if we should revisit this now that multi-processor systems are the norm. I tested it and if you cache-line align `fd_array`, the whole struct still fits into 11 cache lines.

[David Vernet @byte_lab]

It improved significantly on a microbenchmark I wrote that has some threads spam dup2()/close() while others spam pread() :-D. I couldn't find a more ubiquitous benchmark to showcase it because there's either not enough fd churn, or the benchmark creates more than 64 fds so the table gets reallocated anyways and fd_array becomes unused.

It's hard to say whether this is actually useful or necessary in practice, but I do think the original assumptions in 0c9e63fd38a2 ("[PATCH] Shrinks sizeof(files_struct) and better layout") are probably no longer true for most users.

[David Vernet @byte_lab]

Ah, this was previously discussed in commit 0c9e63fd38a2 ("[PATCH] Shrinks sizeof(files_struct) and better layout") https://t.co/h4Hi1C4nJD:

>UP and SMP should benefit from this patch, because most tasks will touch only one cache line when open()/close() stdin/stdout/stderr (0/1/2), (next_fd, close_on_exec_init, open_fds_init, fd_array[0 .. 2] being in the same cache line)

I kind of doubt that holds true today in general.

[David Vernet @byte_lab]

As per usual, the Intel variant is far more opaque and stateful. You can only touch the VMCS with vmread and vmwrite instructions.

AMD uses the VMCB which is just in RAM and can be accessed with normal loads and stores.

[Chrinovic Mukanya @mu_chrinovic]

interested in architectural differences between svm and intel's vmx. most of the features seem to be Intuitively similar

[David Vernet @byte_lab]

@zaydmulani Exactly. It also gives the scheduler flexibility on decisions that are tangentially related to latency. E.g. if you have an underutilized host with only low QoS work, maybe the scheduler will choose to batch everything onto to a single CPU to decrease power consumption.

[David Vernet @byte_lab]

In my opinion the core problem with EEVDF is that it uses slice length as a knob for latency. This is a mistake — like existing interfaces in the POSIX scheduling API such as niceness, it is an extremely awkward API to use correctly.

How do you determine how long a task should be scheduled for to keep its latency low, but not SO low that context switch overhead becomes a problem? It’s literally impossible to reason about this for even the world’s foremost scheduling experts.

The only usable interface for this that I’ve seen is QoS based, and the only framework that realistically enables this is sched_ext.