Will Monroe

Natural language processing and general language geekery. Research scientist @duolingo. @[email protected]

[Klinton Bicknell @klintonbicknell]

Duolingo AI is hiring new PhDs! Come define the future of AI-driven education! NYC or Pittsburgh. (Must get PhD by Aug 2026.)

https://t.co/mq1ESltSFV

[Will Monroe @futurulus]

@emollick Another bit of unsatisfying math that keeps me up at night: two equally valid tunings of the perfect fifth interval (C : G) in music, as a ratio of frequencies

3/2 = 1.5
2^(7/12) ≈ 1.4983
https://t.co/P69S2CvkQ8

[Klinton Bicknell @klintonbicknell]

Duolingo AI is hiring new PhDs and PhD interns! Come define the future of AI-driven education!

(New PhDs must graduate by Aug 2025, interns by Aug 2026)

https://t.co/ZmZqywIAvg

[Ethan Mollick @emollick]

Most "GPT-4o-is-bad-at-math" examples are not that surprising (it is a LLM limitation) but this is more interesting.

I gave GPT-4o the image below and even after asking it to do the math in 10 different ways, it seems stuck, until I used the same technique I used on my kids...

[Karan Dalal @karansdalal]

I’m excited to share a project I’ve been working on for over a year, which I believe will fundamentally change our approach to language models.

We’ve designed a new architecture, which replaces the hidden state of an RNN with a machine learning model. This model compresses context through actual gradient descent on input tokens. We call our method “Test-Time-Training layers.”

TTT layers directly replace attention, and unlock linear complexity architectures with expressive memory, allowing us to train LLMs with millions (someday billions) of tokens in context.

Our instantiations, TTT-Linear and TTT-MLP, both match or beat the strongest Transformers and Mamba. Arxiv: https://t.co/3eEenKB17s

[Will Monroe @futurulus]

@avzaagzonunaada @matt_boot_ Korean has /an/ for general negation and /ma/ for prohibition... 🤔

[Ethan Mollick @emollick]

What happens if you put an AI in charge of your national defense?

In war games, LLMs tend to escalate & do arms races. Base models are more aggressive & unpredictable. The authors speculate that it is because there is lots of training data on escalation, little on de-escalation.

[Ana 安慧敏 @anastudiescn]

no, but why did google show me this when I tried to google a grammar point

[Stephen Mayhew @mayhewsw]

"gpt4 will save so much time, we can all go on vacation all the time"

gpt4 saving me time:

[Klinton Bicknell @klintonbicknell]

Duolingo AI is hiring for an AI research engineer who plans to get their PhD by August for our Pittsburgh HQ! Come define the future of education and assessment. https://t.co/c76CvCwu2Y

[Stephen Mayhew @mayhewsw]

Duolingo AI Research Intern position: come to Pittsburgh, work with some of the coolest data in the world.
https://t.co/E5z20bhG52

[Will Monroe @futurulus]

@anas_ant @LukeGessler @prajdabre1 @kalikabali @Renelvon Definitely a bug, but an old one: it's been that way since 2000! https://t.co/34vOTrCLHv

Seems to me like a valiant (but not quite successful) attempt to *avoid* European bias—the relevant change to a Unicode report was only 7 months old at the time: https://t.co/zm3J6xnYze

[Will Monroe @futurulus]

@matt_boot_ ...aaand I just re-read the original post and realized I missed the point (morphemes not phones). Maybe people will find this Korean phonology tidbit interesting anyway 😅

[Will Monroe @futurulus]

@matt_boot_ Some Korean speakers swear that [t̚] is an allophone of ㅎ/h/ word-finally. Mercifully, there are no words ending in ㅎ.

...but [t̚] is *also* a word-final allophone of ㄷㅌㅈㅊㅆㅅ/d tʰ dʑ tɕʰ s sʰ/, so if you hear a [t̚] you still have no idea how to spell it

[Klinton Bicknell @klintonbicknell]

My team is hiring a staff-level AI Research Scientist at Duolingo! New York or Pittsburgh. Come define the future of education!

NYC: https://t.co/JwlXK32aok
PGH: https://t.co/75o94KKSii

[Will Monroe @futurulus]

@matt_boot_ Often the answer is "no cap" (lol): keep them all! It's what I'd do first with a linear model, unless we're talking 10M+ features.

If vec size is making things slow/mem hungry, you can drop words that only show up once or twice, or use feature hashing or truncated PCA

[Sam Bowman @sleepinyourhat]

I’m sharing a draft of a slightly-opinionated survey paper I’ve been working on for the last couple of months. It's meant for a broad audience—not just LLM researchers. (🧵)

[Janelle Shane @JanelleCShane]

chatGPT unicorn

the horn was not where i expected
9 out of 10

[Christopher Potts @ChrisGPotts]

Thank you! Quick summary: LLMs are probably inducing a semantics (mapping from linguistic forms to concepts), which may be the essence of understanding. This will make their behavior increasingly systematic. This does not imply that they will be trustworthy!

[Arvind Narayanan @random_walker]

While playing around with hooking up GPT-4 to the Internet, I asked it about myself… and had an absolute WTF moment before realizing that I wrote a very special secret message to Bing when Sydney came out and then forgot all about it. Indirect prompt injection is gonna be WILD