David Braun (dbraun.bsky.social)

PhD student @PrincetonCS audiovisual ML. @CCRMA/@Stanford, @BrownUniversity @dbraun.bsky.social

[David Braun (dbraun.bsky.social) @DoItRealTime]

🎛️Audio ML friends, I made a transpiler from Faust code to JAX! Faust is a powerful language for sound synthesis, so connecting it to JAX while supporting differentiability and learnable parameters is a huge research opportunity. 🧵

[Google Magenta Project @GoogleMagenta]

Introducing Magenta RealTime 2 (MRT2): the live music model you can play as an instrument.

MRT2 offers MIDI and prompt controls, and runs natively on a MacBook with <200ms latency.

Open weights. Open source inference engine. Suite of apps and plugins.

Hear what it can do and try it out for yourself below 🧵

[David Braun (dbraun.bsky.social) @DoItRealTime]

@jon_barron Have your agent learn argbind. It works great with dataclasses. My fork and dev branch has some minor improvements.

[David Braun (dbraun.bsky.social) @DoItRealTime]

@OfficialLoganK Could Google continue the jax-metal project? Apple seems to have abandoned it. See jax-ml/jax/pull/34485

[David Braun (dbraun.bsky.social) @DoItRealTime]

@_JonghoChoi 된장인지 ㅁㅁ인지 고민…

[David Braun (dbraun.bsky.social) @DoItRealTime]

@bigblueboo 5/4 DnB! https://t.co/OT5TBBohyN

[David Braun (dbraun.bsky.social) @DoItRealTime]

@giffmana Also missed https://t.co/5h8DMsnCez by @marco_ppasini et al.

[David Braun (dbraun.bsky.social) @DoItRealTime]

@miketuritzin Yea it was. Kudos to Derivative for supporting it early.

[David Braun (dbraun.bsky.social) @DoItRealTime]

@iquilezles Would alternating the phases of the rotors increase throughput?

[David Braun (dbraun.bsky.social) @DoItRealTime]

@nsthorat Does this have any advantage over Claude Code + PyCharm?

[arXiv Sound @ArxivSound]

``DAC-JAX: A JAX Implementation of the Descript Audio Codec,'' David Braun, https://t.co/CPEJBeaM42

[David Braun (dbraun.bsky.social) @DoItRealTime]

In other words, if I wanted to explore usage of a real-time, 8.2 ms latency, 44.1 kHz DAC model, JAX might be faster. Of course, more analysis and testing are welcome. More details in the paper linked at https://t.co/pkNDJ4lbRc

[David Braun (dbraun.bsky.social) @DoItRealTime]

Happy to release "DAC-JAX: A JAX Implementation of the Descript Audio Codec." This can reuse PyTorch weights of all model sizes, and it includes a device-parallel training script. It uses the standard JAX libraries: Flax, Optax, Orbax, and CLU. https://t.co/pkNDJ4lbRc

[David Braun (dbraun.bsky.social) @DoItRealTime]

I benchmarked the chunked compression/decompression speeds. These are the functions you would use on long files or streaming. For a hop size of 8.2 ms, JAX performs compression in 7.1 ms and decompression in 4.3 ms. PyTorch performs compression in 8.3 ms, decompression in 6.3 ms.

[David Braun (dbraun.bsky.social) @DoItRealTime]

@bigblueboo This may be a coincidence, but The Beatles also poured paint on a piano in Strawberry Fields Forever.

[David Braun (dbraun.bsky.social) @DoItRealTime]

@naotokui_en It’s possibly a reference to Strawberry Fields Forever in which The Beatles pour paint on a piano.

[David Braun (dbraun.bsky.social) @DoItRealTime]

The afternoon workshop is here https://t.co/58eFnxoftX Thanks to the Programmable Audio Workshop (@Inria @insadelyon @Grame_Lyon) for having me!

[David Braun (dbraun.bsky.social) @DoItRealTime]

New talk and workshop on Faust+JAX, this time with parameter *automation*. With simple SGD and L1 time-domain loss over the input audio and ground truth, we recover the parameter automation of a lowpass filter's cutoff frequency. https://t.co/dc4QqaxbcE

[David Braun (dbraun.bsky.social) @DoItRealTime]

Another example shows a differentiable polyphonic wavetable synth. The wavetables (2048 sample arrays) are learnable as well as the "Wavetable Position" which blends between them. Notice that the middle plot is a blend of a sine and triangle, but all the wavetables are learnable!

[David Braun (dbraun.bsky.social) @DoItRealTime]

DawDreamer now has a demo script for using multiprocessing to efficiently create one-shots from a synthesizer! Use all the cores! https://t.co/VIOmfy3ZCM