Luan

ml enthusiast

[Luan @luan_wav]

@didier_lopes @pangram in the same direction, I think context distillation could also work with a writing guide being provided to the model https://t.co/MSA2VL7PBi

[Luan @luan_wav]

@kalomaze @Laz4rz shouldn't be that hard to adapt it, but it hurts to see how undervalued audio tasks are compared to vision ones

[Luan @luan_wav]

@kalomaze @Laz4rz also wanted to try rl'ing the new gemma models on a set of verifiable audio puzzles (morse, the bird one in the witness, etc) because I like how they handled audio tokenization, unfortunately, the main frameworks still dont really support audio multimodality

[Luan @luan_wav]

@kalomaze @Laz4rz it also bothers me how overfit these models are to human voice like frequencies, tried rl on Qwen2-Audio for morse code translation and had to record myself doing "bip-bops" for it to work

[Luan @luan_wav]

it bothers me how multimodal LLMs that handle audio are so optimized for speech-related tasks like transcription, I tested a few models on a puzzle similar to the bird one in The Witness, and they performed surprisingly badly, even though the task was just pitch identification

[Luan @luan_wav]

@leastsquared_ @VictorTaelin surpreendentemente pra mim, encontrei mt br no ICLR do Rio, mta galera tinha paper em workshop e alguns até main track

[Luan @luan_wav]

@N8Programs I've also seen some people saying that just prompting the model with a writing manual can show great results, so I guess you can do a context distillation, or train a reward model on pairs of good/bad writing pairs generated by prompting the model with the manual as guidance

[Luan @luan_wav]

@N8Programs I think this article shared a good point of view https://t.co/bYz6r65SpP. I also really dislike the idea that "undetectability" relative to human writing should be treated as a proxy for quality. It can be a good text even if it's clearly different from a human written one

[Luan @luan_wav]

@elyasbuilds really cool! would be great to see if it is also possible to probe stylometric features (gunning fog index, lexical richness, etc) from these embeddings

[Luan @luan_wav]

@SarahLacard I'm kinda skeptical about mech interp (because of the internal rashomon effect in models), but hoping it works

[Luan @luan_wav]

@willccbb we need to hopemax them more

[Luan @luan_wav]

@willccbb I guess alignment post-training really pushes models towards "conservative" views on all kinds of topics, they don't even believe they can solve erdos problems

[Luan @luan_wav]

@pleometric every time this happens I'm reminded of this paper https://t.co/JMhr57ygDv

[Luan @luan_wav]

@47fucb4r8c69323 for me the worst thing is using general human preference as a proxy signal for "good writing", when in the general case human preference can be very low taste (https://t.co/BuVHGcVhWb)

[Luan @luan_wav]

@scaling01 I was hoping that deepmind would perform better in the long run because, although they may not be the strogest regarding llms, they have historically been the best at everything else and seemed well positioned to catch up over time, but now I’m starting to lose that feeling

[Luan @luan_wav]

@benno_krojer curious about how they trained it, I've been thinking about training a small native multimodal model like that, but bootstrapping the visual pathway by doing layer-by-layer distillation from a pretrained vision model, but not sure if it would work

[Luan @luan_wav]

@teortaxesTex Amodei didn't lose the Mandate of Heaven, he's just optimizing for the audience that truly matters: the models

[Luan @luan_wav]

@47fucb4r8c69323 will try dumping a writing guidebook into a small model context and applying OPSD to see if it leads to some great results, but I don't have great hopes...

[Luan @luan_wav]

@47fucb4r8c69323 for me the worst thing is using general human preference as a proxy signal for "good writing", when in the general case human preference can be very low taste (https://t.co/BuVHGcVhWb)