๐๏ธAudio ML friends, I made a transpiler from Faust code to JAX! Faust is a powerful language for sound synthesis, so connecting it to JAX while supporting differentiability and learnable parameters is a huge research opportunity. ๐งต
Introducing Magenta RealTime 2 (MRT2): the live music model you can play as an instrument.
MRT2 offers MIDI and prompt controls, and runs natively on a MacBook with <200ms latency.
Open weights. Open source inference engine. Suite of apps and plugins.
Hear what it can do and try it out for yourself below ๐งต
In other words, if I wanted to explore usage of a real-time, 8.2 ms latency, 44.1 kHz DAC model, JAX might be faster. Of course, more analysis and testing are welcome. More details in the paper linked at https://t.co/pkNDJ4lbRc
Happy to release "DAC-JAX: A JAX Implementation of the Descript Audio Codec." This can reuse PyTorch weights of all model sizes, and it includes a device-parallel training script. It uses the standard JAX libraries: Flax, Optax, Orbax, and CLU. https://t.co/pkNDJ4lbRc
I benchmarked the chunked compression/decompression speeds. These are the functions you would use on long files or streaming. For a hop size of 8.2 ms, JAX performs compression in 7.1 ms and decompression in 4.3 ms. PyTorch performs compression in 8.3 ms, decompression in 6.3 ms.
New talk and workshop on Faust+JAX, this time with parameter *automation*. With simple SGD and L1 time-domain loss over the input audio and ground truth, we recover the parameter automation of a lowpass filter's cutoff frequency. https://t.co/dc4QqaxbcE
Another example shows a differentiable polyphonic wavetable synth. The wavetables (2048 sample arrays) are learnable as well as the "Wavetable Position" which blends between them. Notice that the middle plot is a blend of a sine and triangle, but all the wavetables are learnable!
DawDreamer now has a demo script for using multiprocessing to efficiently create one-shots from a synthesizer! Use all the cores! https://t.co/VIOmfy3ZCM