Steve Yadlowsky

no longer a good place to find me. check my website.

[Tibo @thsottiaux]

Worth clarifying that this is flatly false and the two options are just two independent ways to opt-out.

We respect our users' choice whether to use their data to “improve our models for everyone” regardless of where they express that choice. Users can opt out in the in-app settings or indeed also in our privacy portal. They do not need to opt out in both places, and we will make this clearer in our Help Center.

https://t.co/sv4uFuHC3W

[Steve Yadlowsky @graduatedescent]

Asking for a friend while meanwhile nervously checking how many unsubmitted manuscripts I have.

[Reviewer 2 @GrumpyReviewer2]

#AcademicChatter #AcademicX

[Steve Yadlowsky @graduatedescent]

Going beyond needle tasks is super important for understanding real-world long-context capabilities. Looks like a great benchmark for measuring progress at the intersection of long context and reasoning.

[Kiran Vodrahalli @kiranvodrahalli]

Happy to share Michelangelo (https://t.co/0AgJ7JmToG), a long-context reasoning benchmark which measures performance beyond needle tasks up to arbitrary context lengths and remains challenging for frontier models. Stay tuned for more Michelangelo evals to come!

[Steve Yadlowsky @graduatedescent]

@Maggiemakar @victorveitch 👀 that's terrifying

[Steve Yadlowsky @graduatedescent]

@dieworkwear Do you have suggestions for affordable places to buy pants with a better shape? Places like macys <only> have slim fitting pants these days.

[Steve Yadlowsky @graduatedescent]

Excited to be at NeurIPS this year, anyone who is around and wants to chat, please reach out! It’s always great to connect/reconnect with folks at these events!

[Steve Yadlowsky @graduatedescent]

Excited to be part of this effort! It's been a fun research experience to be working on something so new.

[Jeff Dean @JeffDean]

I’m very excited to share our work on Gemini today! Gemini is a family of multimodal models that demonstrate really strong capabilities across the image, audio, video, and text domains. Our most-capable model, Gemini Ultra, advances the state of the art in 30 of 32 benchmarks, including 10 of 12 popular text and reasoning benchmarks, 9 of 9 image understanding benchmarks, 6 of 6 video understanding benchmarks, and 5 of 5 speech recognition and speech translation benchmarks. Gemini Ultra is the first model to achieve human-expert performance on MMLU across 57 subjects with a score above 90%. It also achieves a new state-of-the-art score of 62.4% on the new MMMU multimodal reasoning benchmark, outperforming the previous best model by more than 5 percentage points.

Gemini was built by an awesome team of people from @GoogleDeepMind, @GoogleResearch, and elsewhere at @Google, and is one of the largest science and engineering efforts we’ve ever undertaken. As one of the two overall technical leads of the Gemini effort, along with my colleague @OriolVinyalsML, I am incredibly proud of the whole team, and we’re so excited to be sharing our work with you today!

There’s quite a lot of different material about Gemini available, starting with:

Main blog post: https://t.co/NzSycJl7aE

60-page technical report authored by th Gemini Team: https://t.co/CEdMRyYSLo

In this thread, I’ll walk you through some of the highlights.

[Steve Yadlowsky @graduatedescent]

@SamuelMullr Got it. Yeah in the appendix, we tried to get even closer to LLM-style architectures by using discretized bin indices as inputs and outputs, but it was really hard to pretrain with the model size and data we were working with. We didn't try only discretizing outputs.

[Steve Yadlowsky @graduatedescent]

@SamuelMullr Yeah we designed the sine + linear specifically to ensure that the superimposed function was fully out of distribution from the pretraining mixture to test what happens in this case, since we didn't have intuition for what the model would do.

[Steve Yadlowsky @graduatedescent]

@SamuelMullr - Noise (ours was trained without noisy observations; seems like this is pretty important for the results in the colab, the quality of the interpolation degrades significantly without noise)

[Steve Yadlowsky @graduatedescent]

@SamuelMullr - Sine frequencies (we pretrain on many frequencies, looks like yours is just one)
- Architecture (skim of paper suggests you don't use positional encodings, we do to better match what's seen in LLMs
- Model scale (unclear if model size affects behavior)

[Steve Yadlowsky @graduatedescent]

This paper is continuing to make the rounds. It seems like maybe our paper on in-context learning has been taken.....out of context?

[anton @abacaj]

New paper by Google provides evidence that transformers (GPT, etc) cannot generalize beyond their training data

[Steve Yadlowsky @graduatedescent]

@agstrait They are analogies for the more technical "functions" (ie., tasks) and "function classes" (ie., task families) we study. A function is like f(x) = 2x + 4 that we are trying to learn using ICL examples. Function class would be "ax + b" for any a and b, or "sin(ax)" for any a.

[Steve Yadlowsky @graduatedescent]

Might be helpful to do a thread about what our paper shows and why we think it’s interesting. First of all, we study transformer models pretrained to be in-context learners 1/

[Steve Yadlowsky @graduatedescent]

Excited about this work and the enthusiasm it has received. To clarify, we show that transformers <can> learn new tasks, but <can’t> generalize to unseen task families. Figuring out what task families a fully-trained LLM can ICL is an impactful scientific direction from here.

[Steve Yadlowsky @graduatedescent]

@AndrewLampinen Should’ve said I encourage <everyone> to take a look 🙂

[Steve Yadlowsky @graduatedescent]

Our strongest conclusion, which is already well-appreciated in a lot of the AI community, is that choosing good quality data that covers many task families is crucial to building capable in-context learners. 5/5

[Steve Yadlowsky @graduatedescent]

When they see examples from a task unlike any task family they’ve seen before, one might have many hypotheses about how they generalize or interpolate, such as finding the nearest approximation to a known task family. We designed empirical studies to test many of them. 3/

[Steve Yadlowsky @graduatedescent]

What we found is that in some limited cases (rare tasks or those <close> to a /seen task family), the model makes reasonable predictions. Other times, the model tends to behave erratically. 4/