dron

math/music/ai nerd | research @GoodfireAI | prev cambridge, bair, polaris | giving a semantics to the syntax

[dron @_dron_h]

@GoogleMagenta this is really cool work!! can't wait to play with it

[Google Magenta Project @GoogleMagenta]

Introducing Magenta RealTime 2 (MRT2): the live music model you can play as an instrument.

MRT2 offers MIDI and prompt controls, and runs natively on a MacBook with <200ms latency.

Open weights. Open source inference engine. Suite of apps and plugins.

Hear what it can do and try it out for yourself below 🧵

[dron @_dron_h]

@pranav_vd @raymondmfeng raymond spotted!!!!

[dron @_dron_h]

@zacknovack ooh super cool!!

[dron @_dron_h]

@nabla_theta @QiaochuYuan a pointer is a Thing that points to another Thing, and even here i have created an implicit binding between the two Things when they need not be related

fully abstracted, devoid of implementation or concrete ground

[dron @_dron_h]

@nabla_theta @QiaochuYuan closures and currying are also pretty emblematic to me? we're in a fuzzy space here but currying is one of those things that feels very "trivial" to me — it's a primitive!

what is a pointer? what is a cache? what is a function? none of these are constrained (much)

[dron @_dron_h]

anti sae sae club

[Goodfire @GoodfireAI]

The most popular way to interpret AI is missing the bigger picture.

Models think in curved shapes. But sparse autoencoders (SAEs) work with straight lines.

Can they still capture models’ curved neural geometry? Yes, but not how you might think! (1/7)

[Goodfire @GoodfireAI]

The most popular way to interpret AI is missing the bigger picture.

Models think in curved shapes. But sparse autoencoders (SAEs) work with straight lines.

Can they still capture models’ curved neural geometry? Yes, but not how you might think! (1/7)

[dron @_dron_h]

@nabla_theta @QiaochuYuan my CS education was extremely PL-coded though, it's possible this is more of a eurotheory thing

[dron @_dron_h]

@nabla_theta @QiaochuYuan ah. well to me haskell is (one of) the prime examples i have in mind about these abstractions

[dron @_dron_h]

@acapellascience as in most cases ai is not like other technologies! the scale/generality is very different from past forms of "cheating"

[dron @_dron_h]

@acapellascience the niche must emerge and we must make it so

but the default case is death by optimization, there are forces that make this death desirable for some

[dron @_dron_h]

@prompt_Tunes yup this work is building on top of that!
https://t.co/3DJvjD8nAQ

[Sheridan Feucht @sheridan_feucht]

But how does this addition mechanism actually work? In agreement with @NeelNanda5, @tianyi_zhou12, @thesubhashk, and others, we found that Llama calculates addition using Fourier features. Specifically, periods 2, 5, and 10 (also 20, 50, 100) stuck out, corroborating prior work.

[Goodfire @GoodfireAI]

Neural networks do math by rotating shapes.

We found a shape-rotating calculator hidden inside an LLM – and it’s used for more than just math! (1/6)

[dron @_dron_h]

@sheridan_feucht it's so good!

[dron @_dron_h]

look at those beautiful subgroups

[Sheridan Feucht @sheridan_feucht]

Next, we analyze the neurons in MLP 18 that are responsible for actually doing addition. We focus on a set of 28 MLP neurons found using DAS (Geiger et al., 2023). They form clear subgroups that fire at different frequencies and align with the Fourier probes that we trained!

[Sheridan Feucht @sheridan_feucht]

Neural networks have beautiful feature geometry, but do they have mechanisms that actually interface with those structures?

At @GoodfireAI this spring, we discovered one: a re-usable addition mechanism that reads/writes to Fourier features from prior work. 🧵

[dron @_dron_h]

@nabla_theta @QiaochuYuan this is my excuse for why i've bounced right off any math textbook i've ever opened

[dron @_dron_h]

@nabla_theta @QiaochuYuan so often explaining a CS-shaped concept often has the feeling of being obvious or trivial or horrendously abstract/devoid of (concrete) meaning — this is somewhat deliberate