Andy Mau

AI Speech Researcher @ JAIST | MATLAB Student Ambassador @ MathWorks

[Thariq @trq212]

https://t.co/hPiZr1kG7r

[Sasha Rush @srush_nlp]

New diffusion tutorial dropped https://t.co/6Mkpy5JBQB. Looks great, particularly for ML folks perspective. Would say 6-8 on this scale.

Just working your way up the diffusion tutorial ladder for 2-3 years would be a pretty strong advanced undergrad curriculum.

[Jacob Austin @jacobaustin132]

Today we're putting out an update to the JAX TPU book, this time on GPUs. How do GPUs work, especially compared to TPUs? How are they networked? And how does this affect LLM training? 1/n

[Piotr Żelasko @PiotrZelasko]

You asked for it, and we listened.
MULTILINGUAL Canary v2 and Parakeet v3!!
🌏 25 European languages
🏆 SotA on Multilingual Open ASR Leaderboard
🔥 600x and 2000x faster than real-time
🕰️ Timestamps!
🗣️ Speech translation (Canary)
🃏 Granary: all data is open, train it yourself!

[Bao Pham @baophamhq]

Diffusion models create novel images, but they can also memorize samples from the training set. How do they blend stored features to synthesize novel patterns?  Our new work shows that diffusion models behave like Dense Associative Memory: in the low training data regime (number of memories), they memorize data points by storing them as local minima of their energy function. With more data, diffusion models began to forget and produce spurious states, signaling generalization — where stored features start to blend into new patterns. Finally, when there’s a lot of data points, the basins around the stored points widen greatly and the minima flatten, maximizing the likelihood of a diffusion model generating novel patterns. This is how they create novel outputs in the high data regime. (1/N)

Here is a link to the paper: https://t.co/2O19pIyjDR

The work is done in collaboration with Gabriel Raya @gaboraya, Matteo Negri, Mohammed J. Zaki @mj_zaki, Luca Ambrogioni @LucaAmb, and Dmitry Krotov @DimaKrotov from @MITIBMLab @IBMResearch

[William Chen @chenwanch1]

What happens if you scale Whisper to billions of parameters?

Our #ICML2025 paper develops scaling laws for ASR/ST models, training models with up to 18B params and 360K hours of data, and 100+ languages

Joint work b/w @LTIatCMU and @nvidia

https://t.co/WMOWjUzglP

[Nathan Lambert @natolambert]

I hear people are pretty into GRPO and RL these days, so I wrote up a pretty comprehensive research survey of recent papers I liked. Kimi 1.5, OpenReasonerZero, DAPO and Dr. GRPO.
+ discussion on if GRPO is special and further reading. https://t.co/AAkMjVTYuK

[Jack Morris @jxmnop]

# A new type of information theory

this paper is not super well-known but has changed my opinion of how deep learning works more than almost anything else

it says that we should measure the amount of information available in some representation based on how *extractable* it is, given finite computation. for example, an encrypted text file has less V-information than the same data in plaintext, because it takes more computation to extract. note the contrast to traditional information theory, which would tell us that the two representations have the same amount of Shannon information

i’ve long wondered why certain types of basic questions didn’t have a proper theoretical answer:
> why does distillation outperform vanilla maximum likelihood training?
> why does lora work better than finetuning?
> why does self-attention work better than almost any other similar operation?
> how much “information” remains in a text embedding?
> how much “information” remains in language model weights?
> should i use fine-tuning or RAG?

the true answers to all these questions depend on some way of measuring and comparing *information content* between different representations. v-information is one step towards doing this

besides computational constraints, model architecture probably affects the “information content” in representations, along with the presence of any pretraining data used, as well as model-level statistics about the optimization in the training process – e.g. the length of time a model was trained for probably changes representations pretty drastically

this is all to say, i think there is some true notion of “information” that none of our current paradigms (Shannon information, V-information, etc.) capture. we encounter this idea every day but we dance around it and describe it in vague terms; we measure it from all sorts of angles but can’t quite characterize it theoretically

when the v-information paper came out I thought there would be a lot of follow up work developing more complex and useful notions of information for deep learning. but it hasn’t

yet I still think at the heart of these questions of what-information-lies-in-representations there’s something to be found that’s profound, elegant, and potentially extremely useful. I don’t know what it is and i’m probably not the person who will figure it out. but I really hope someone does. :)

[Sasha Rush @srush_nlp]

Neat website that has a bunch of CUDA programming problems:
https://t.co/1HQ3FJ23hy

[AlphaCephei @alphacep]

Following our recent experiments I wrote more notes on Matcha TTS design

https://t.co/KJ1YT2bjX9

[Standard Intelligence @si_pbc]

At Standard Intelligence we’ve been researching scalable cross-modality learning. We’re excited to share some early results in the form of 𝗵𝗲𝗿𝘁𝘇-𝗱𝗲𝘃, an open-source, first-of-its-kind base model for full-duplex conversational audio. 1/

[Bilawal Sidhu @bilawalsidhu]

Turns out AI generated slop can have real world impact. On Halloween, crowds gathered in Dublin for a parade that didn’t even exist.

Police released a statement confirming no parade was scheduled, urging people to “disperse safely.”

The source? A three month old SEO agency halfway around the world used AI to spin up articles for this fake event and managed to get it ranked on Google’s first page.

Nobody bothered to double check sources because “the size of the crowds caused a brief disruption to the Luas tram network, with parts of the Red and Green line services down for about half an hour.”

In other words, AI generated slop can disrupt reality.

[kyutai @kyutai_labs]

Today, we release several Moshi artifacts: a long technical report with all the details behind our model, weights for Moshi and its Mimi codec, along with streaming inference code in Pytorch, Rust and MLX. More details below 🧵 ⬇️
Paper: https://t.co/mMInmjiBIC
Repo: https://t.co/PFak47FMrm
HuggingFace: https://t.co/bqG4IS0ntg

[Andy Mau @nddmau]

@unilightwf Agree!

[Kwanghee Choi @juice500ml]

Is it accurate to call k-means clusters of self-supervised speech representations as semantic tokens? (Spoiler alert: No) In our new #INTERSPEECH2024 paper, we conclude that Self-Supervised Speech Representations are More Phonetic than Semantic. https://t.co/yfkI1ShAU6 (1/n)

[Tom Silver @tomssilver]

Six years ago I wrote a post on "Lessons from My First Two Years of AI Research." I got a lot of nice feedback and messages from people over the years saying that it helped. Now that I've finished another career chapter, I've added a few more lessons: https://t.co/qg8OzWAAvy

[hardmaru @hardmaru]

Language is primarily a tool for communication rather than thought

https://t.co/lnjn3bcx0A

“Language is a defining characteristic of our species, but the function, or functions, that it serves has been debated for centuries. Here we bring recent evidence from neuroscience and allied disciplines to argue that in modern humans, language is a tool for communication, contrary to a prominent view that we use language for thinking. We begin by introducing the brain network that supports linguistic ability in humans. We then review evidence for a double dissociation between language and thought, and discuss several properties of language that suggest that it is optimized for communication. We conclude that although the emergence of language has unquestionably transformed human culture, language does not appear to be a prerequisite for complex thought, including symbolic thought. Instead, language is a powerful tool for the transmission of cultural knowledge; it plausibly co-evolved with our thinking and reasoning capacities, and only reflects, rather than gives rise to, the signature sophistication of human cognition.”

[MATLAB Japan @MATLAB_Japan]

理工系学生必見
【業界リーダーから学ぶ製品開発の裏側】
就職までに身に着けたい 産業界で求められる知識とスキルとは
https://t.co/i7psWDwiVl

ソフトウェアの役割が拡大する自動車や航空宇宙分野の開発エンジニアが登壇！
ライブチャットで質問もできます
就活の業界研究にも
お見逃しなく！

[hardmaru @hardmaru]

RNNs are not dead yet‼️

In fact, they are coming back with a vengeance recently. Very nice paper about “The Illusion of State in State-Space Models” (https://t.co/TPhu4g8NtW) and thread 👇

[Stanley H. Chan @stanley_h_chan]

I wrote a tutorial on diffusion models for undergrad and grad students. I tried my best to give intuitive explanations for complicated equations.

Your feedback is much appreciated

Thanks to those who suggested various reading materials to me

https://t.co/fWv111nG5M