Andy K

Quaere caelum | @OrnnExchange, ex MIT

[Andy K @andylawk]

My father served in the Navy. Every important lesson in my life, I've learned from him.

A big thank you to all veterans for their service. And thank you, Dad.

[Andy K @andylawk]

Gutta cavat lapidem

[Andy K @andylawk]

compute is one big campfire and I get scared of the ghosts in the shadows

[Andy K @andylawk]

Classical mechanics, quantum, the central dogma of molecular biology...we have theories underlying many elemental fields. I hope intelligence is next. It feels like a domino waiting to fall

[Andy K @andylawk]

Have been thinking a lot about fast and slow weights in neural nets recently. Anecdotally, it seems human organizations tend to converge on the same structure...there is a "frontier" contingent of workers and a stabilizing "core." You need both to operate.

[Andy K @andylawk]

The map and the territory. Sometimes I feel like my life is just about reconciling the two

[Andy K @andylawk]

@induction_labs I've always been bullish on novelty search as an learning mechanism. It feels epistemologically correct.

Would be very keen to learn more details on this reward mechanism / "discovered" set!

[Andy K @andylawk]

@wayne_nelmz the classic generator-verifier dichotomy. we've shifted quite far to one side...

[Andy K @andylawk]

@MengdiWang10 I've always though of RL as structurally a mode-seeking process because it looks to minimize a reverse-KL div between the model and the target distributions. Placing mass outside the target support blows up the loss function...

[Andy K @andylawk]

@GoodfireAI Had a chance to try this. It’s pretty damn cool

[Andy K @andylawk]

@GoogleResearch I remember proving the preservation of dot products under JL back in university...cool to see that you can quantize asymmetrically and maintain the same result.

[Andy K @andylawk]

https://t.co/sH51pYuVwz

[Andy K @andylawk]

@askalphaxiv a while ago I did some work on a modified dropout applied to the spectral components of neural nets…in general I think this sort of spectral regularization is an underexplored research direction

[Andy K @andylawk]

@ericjmichaud_ Super cool. I think that it would incredibly interesting to visualize the loss landscapes of individual "quantum" subtasks

If their distribution follows a power law, you should be able to superimpose them (with the appropriate weighting) and recover the aggregate loss surface

[Andy K @andylawk]

If you think about it, git is basically a residual stream.
Main carries the base signal, branches extract deltas and merges add them back.

[Andy K @andylawk]

a mathematical take on building startups
https://t.co/1W4viiN9qB

[Andy K @andylawk]

It's cool to think of SAEs as performing a post hoc basic rotation to extract more interpretable directions, but imo it's wishful thinking that we'd see this in training as is.

[Andy K @andylawk]

Something clicked today for me. Interpretability is at odds with scaling.

Learning polysemantic features is an efficient solution via SGD. Sparse, monosemantic features cost capacity and gradient flow — it’s foolish to expect these traits to emerge natively

[Andy K @andylawk]

@askalphaxiv The closeness of weight subspaces across domains reminds me of @phillip_isola 's platonic representation hypothesis.

My first inclination is that these shared parametric subspaces) could be the mechanism by which we converge on shared representations.

https://t.co/2JIXoMf4Px