Phoebe Mulcaire

Machine learning at @duolingo. Multilingual structured prediction and language modeling.

[Phoebe Mulcaire @PhoebeNLP]

@alisawuffles but this is so cool, I can't wait to explore how the next-token probs differ for SuperBPE models!

[Phoebe Mulcaire @PhoebeNLP]

@alisawuffles I love this! subword tokens have some convenience advantages: for example if I have word labels that I want to map to token labels I can trust that it's a one-to-many mapping with subword tokenizers, but if there are multiple labels for a superword token I have to make a decision

[Klinton Bicknell @klintonbicknell]

Duolingo AI is hiring new PhDs and PhD interns! Come define the future of AI-driven education!

(New PhDs must graduate by Aug 2025, interns by Aug 2026)

https://t.co/ZmZqywIAvg

[Phoebe Mulcaire @PhoebeNLP]

@BlancheMinerva oh interesting. I think I see the argument for this if "highly OOD contexts" means "(almost-)exclusively unseen tokens in the context" but I'm not clear if that would apply to observed tokens in very-low-probability sequences... would be interesting even just to test that part!

[Phoebe Mulcaire @PhoebeNLP]

@mayhewsw it's like the opposite of the base rate fallacy, ignoring the conditional probability instead of the prior

[Phoebe Mulcaire @PhoebeNLP]

my influence is subtle, but inescapable
https://t.co/jp0Yx9v8hN

[meg.ai 🇨🇦 @MeganRisdal]

Not a joke. We've updated Gemma Terms. https://t.co/uMPQulo0F0

[Phoebe Mulcaire @PhoebeNLP]

"you must make reasonable efforts to use the latest version" in the DBRX and Gemma licenses makes it feel so dishonest to call them "open models." like, yes, you can see the weights, but they can just take away all your usage rights by publishing a broken "update"

[Matthew Finlayson @mattf1n]

Wanna know gpt-3.5-turbo's embed size? We find a way to extract info from LLM APIs and estimate gpt-3.5-turbo’s embed size to be 4096. With the same trick we also develop 25x faster logprob extraction, audits for LLM APIs, and more!
📄 https://t.co/NdYU8ZhuVH
Here’s how 1/🧵

[Alisa Liu @alisawuffles]

LMs are increasingly large🐘 and proprietary🔒 — what if we could “tune”🔧 them without accessing their internal weights?

Enter: proxy-tuning, which operates on only the *outputs* of LMs at decoding-time to achieve the effect of direct tuning!

📄: https://t.co/mx2SRlyTD2 1/

[Phoebe Mulcaire @PhoebeNLP]

@mayhewsw yeah I'm pretty sure one is img2img of the other—everything has the same general shape and position but is different in details. black boxes / woman in the black sweater, light fixture, floor, table etc.

[Stephen Mayhew @mayhewsw]

🚨 New Dataset Alert 🚨 I'm extremely excited to announce Universal NER v1, available now.

It is gold-standard human annotations of 18 datasets covering 12 languages, based on Universal Dependencies texts. This is the first data release of the UNER project.

1/3

[Phoebe Mulcaire @PhoebeNLP]

read the paper: https://t.co/FPGUo9AwIv

[Phoebe Mulcaire @PhoebeNLP]

come see our poster, "Automated Evaluation of Written Discourse Coherence Using GPT-4", at the BEA workshop, today at 2:30! #ACL2023NLP

[Phoebe Mulcaire @PhoebeNLP]

at #ACL2023NLP ! I'll be presenting a poster at BEA on Thursday: work with @BenNaismithELT and @JillBurstein on discourse grading with GPT-4. come say hi :)

[Phoebe Mulcaire @PhoebeNLP]

@universeinanegg @UChicagoCS @DSI_UChicago @Meta congratulations, that's so exciting! hope to see you at ACL :)

[Miles Turpin @milesaturpin]

⚡️New paper!⚡️

It’s tempting to interpret chain-of-thought explanations as the LLM's process for solving a task. In this new work, we show that CoT explanations can systematically misrepresent the true reason for model predictions.
https://t.co/ecPRDTin8h

🧵

[Phoebe Mulcaire @PhoebeNLP]

I'll be in Seattle from this Wednesday to next Tuesday—let me know if you'd like to meet up and say hi!

[Maarten Sap (he/him) @MaartenSap]

🌟Update🌟there's been a lot of debate of whether Theory of Mind (ToM) has emerged in new models (ChatGPT/GPT-3.5/GPT4), as people have reported qualitative/anecdotal evidence of good performance on these types of examples.

TLDR; still no neural ToM... 🧵
https://t.co/R6n4LgIwx5

[nostalgebraist @nostalgebraist]

wait,

what??

why do Bard AND ChatGPT *both* write an anodyne story about a young woman in idyllic "Willow Creek" at sundown???

(details: it's not deterministic, often you get a different town name, different phrasing, etc. broad strokes are similar though. gpt-4 does it too.)

[Phoebe Mulcaire @PhoebeNLP]

@nostalgebraist Bard seems to prefer one of these two for me, but Willow Creek frequently shows up as draft #3, and Sarah and an old man named John recur (I got a young woman named Sarah and an old man named Mr. Johnson in ChatGPT).

[Phoebe Mulcaire @PhoebeNLP]

@nostalgebraist I know this isn't actually it but I really want the answer to be "they couldn't get Bard working so it's a ChatGPT wrapper"