Seungtaek Choi

Assistant Professor at HUFS (Language & AI)

[Jeff Dean @JeffDean]

Tomorrow will be my last day at Google after 27 years, and watching it grow from 25 people to 190,000+ has been an amazing journey. Below is a note I shared with many people internally at Google today. An excerpt is:

It has been an absolute pleasure to work with you and to help build some of the most widely used and impactful products of all time. As a kid, I dreamed of helping build software that would be used by many people, and Google now has thirteen products used by more than a billion people (amazing!). Our work has had a tremendous impact in the world, and I have been lucky enough to collaborate and form friendships with many colleagues that I deeply admire, respect, and enjoy. It still brings me joy every time I see people out in the world using our products to find information, handle email, translate documents, watch videos, learn new things, navigate and understand the physical world, browse the web, use their phone, run large-scale computations on our infrastructure, ride in an autonomous vehicle, or perform complex tasks with the help of our AI systems. I hope you all share this sense of joy, because it is a shared accomplishment! Thank you to all of my colleagues at Google over many years!

Now I'm excited to go start @DiscoLoopAI with my longtime friends and colleagues @Sanjay_Ghemawat, @OriolVinyalsML, and @quocleix.

(Updated post: slightly redacted to not have some personal info)

[ACL 2027 @aclmeeting]

The Winner of the 1st ACL Computational Linguistics Doctoral Dissertation Award is

🏆 Sewon Min with Rethinking Data Use in Large Language Models.

[Denny Zhou @denny_zhou]

Slides for my lecture “LLM Reasoning” at Stanford CS 25: https://t.co/eApGUHyIDo

Key points:
1. Reasoning in LLMs simply means generating a sequence of intermediate tokens before producing the final answer. Whether this resembles human reasoning is irrelevant. The crucial insight is that transformer models can become nearly arbitrarily powerful by generating many intermediate tokens, without the need of scaling the model size (https://t.co/HO2seV6vVl).

2. Pretrained models, even without any fine-tuning, are capable of reasoning. The challenge is that reasoning-based outputs often don’t appear at the top of the output distribution, so standard greedy decoding fails to surface them (https://t.co/75h2QQzT9M)

3. Prompting techniques (e.g., chain-of-thought prompting or "let’s think step by step") and supervised finetuning were commonly used to elicit reasoning. Now, RL finetuning has emerged as the most powerful method. This trick was independently discovered by several labs. At Google, credit goes to Jonathan Lai on my team. Based on our theory ( see point 1), scaling RL should focus on generating long responses rather than something else.

4. LLM reasoning can be hugely improved by generating multiple responses and then aggregating them, rather than relying on a single response (https://t.co/BA5MUzg3PR).

[alphaXiv @askalphaxiv]

"How Many Instructions Can LLMs Follow at Once?"

In this paper they found that leading LLMs can satisfy only about 68% of 500 concurrent instructions, showing a bias toward earlier instructions.

[AK @_akhaliq]

Trillion 7B Technical Report is out on Hugging Face

[elvis @omarsar0]

Why do LLMs attend to the first token?

This new paper explains why LLMs obsessively focus attention on the first token — a phenomenon known as an attention sink.

Their theory: it’s a useful trick to prevent representational collapse in deep Transformers.

• Sinks = over-mixing shields – LLMs with long contexts and deep layers tend to over-mix information, causing similar embeddings for all tokens (i.e., rank collapse or over-squashing). Attention sinks—where many heads fixate on the ⟨bos⟩ token—act as no-ops that reduce token interaction and preserve representation diversity across layers.

• Sharp experiments on Gemma & LLaMa – Perturbation tests in Gemma 7B show ⟨bos⟩ significantly slows the spread of changes through the model. Meanwhile, in LLaMa 3.1 models, over 80% of attention heads show strong sink behavior in the 405B variant, supporting the theory that larger models need stronger sinks.

• Sinks emerge naturally – Even without special pretraining, sinks tend to form at the first position, not because of the ⟨bos⟩ token itself, but due to its location. However, if ⟨bos⟩ is fixed during training and later removed, performance collapses, showing that sink formation is data-dependent.

• Theoretical grounding – The authors connect sink emergence to Jacobian norm bounds, proving that sinks reduce sensitivity to token perturbations. Their math shows that deeper models and longer contexts require stronger sinks.

• Layerwise dynamics insight – Some attention heads use ⟨bos⟩ as a “default” target, unless a special pattern (e.g., apostrophe) triggers real computation. This supports a conditional attention mechanism—attend to ⟨bos⟩ unless needed elsewhere.

[Zhiqing Sun @EdwardSun0909]

I successfully defended my PhD thesis today! 🎉

"Scalable Alignment of Large Language Models Towards Truth-Seeking, Complex Reasoning, and Human Values"

Slides (Fact-RLHF, Lean-STaR, Easy-to-Hard Generalization, Self-Align, Instructable Reward Model): https://t.co/eYrwGnjGQR

A huge thank you to my thesis committee and all attendees for their valuable feedback and support! ❤️
@wellecks @lileics @denny_zhou & Yiming

[Jason Wei @_jasonwei]

There is a nuanced but important difference between chain-of-thought before and after o1.

Before the o1 paradigm (i.e., chain-of-thought prompting), there was a mismatch between what chain of thought was and what we wanted it to be. We wanted chain of thought to reflect the thinking process of the model, but what the model was really doing was just imitating reasoning paths that it had seen in pretraining, e.g., math homework solutions. The problem with this type of data is that it is a post-hoc solution summarized after the author did all the work somewhere else, and not really a sequence of thoughts. So the solutions often had poor information density, with an egregious example being things like “The answer is 5 because…”, where the token “5” has a huge amount of new information.

With the o1 paradigm, you can see that the chain of thought looks very different from a textbook math solution (you can view examples in the blog post). These chains of thought are kinda like “inner monologue” or “stream of consciousness”. You can see the model backtracking; it says things like “alternatively, let’s try” or “wait, but”. And I have not measured directly, but I would wager a bet (my psycholinguistics friends would probably be able to confirm) that the information density is *much* more uniform in the chain of thought than average text on the internet.

[Jerry Wei @JerryWeiAI]

One of the most-crucial yet often-overlooked aspects of success in research (and life in general) is ensuring that you're optimizing for the right function. It's easy to fall into the trap of chasing some reward without taking the time to examine whether it aligns with your personal goals and values.

Consider these two researchers:
- Researcher A is primarily motivated by publishing papers and gaining recognition. They optimize for quantity of publications and citation count.
- Researcher B is driven by a desire to solve important problems and make a meaningful impact. They optimize for the potential applications of their work and its ability to advance the field.

Both of these approaches have their merits and potential drawbacks. Researcher A might build an impressive CV, publish many papers, and gain visibility in the field, which could lead to more opportunities and collaborations. Researcher B might have fewer papers, but each paper is extremely impactful in its particular area because the researcher has spent years working on a breakthrough despite not being externally rewarded while doing so.

The key insight here is that the function you choose to optimize dramatically influences not just your results, but also your day-to-day decisions and overall satisfaction. When you optimize for external metrics (like Researcher A), you might find yourself more attuned to the current trends and demands of the field. In contrast, optimizing for internal motivations (like Researcher B) might lead you to take on more challenging, potentially groundbreaking projects, even if they come with a higher risk of failure.

Choosing the right function to optimize for isn't easy. It requires honest self-reflection and often means balancing competing priorities. By being thoughtful and honest about what you're optimizing for, you can ensure that your efforts are directed towards outcomes that truly matter to you, whether those are external achievements, internal fulfillment, or a carefully chosen balance of both.

[Raphael Mansuy 🍵 @raphaelmansuy]

ARCHCODE: A new method for AI-Powered Software Development ...

As the founder of QuantaLogic, I'm always on the lookout for innovations that can revolutionize how we develop software using AI.

Today, I want to share my thoughts on a fascinating new research paper from Seoul National University: "ARCHCODE: Incorporating Software Requirements in Code Generation with Large Language Models."

👉 The Challenge of AI-Powered Code Generation

While AI has made significant strides in code generation, a persistent challenge has been ensuring that the generated code meets all necessary requirements, both functional and non-functional.

Many existing approaches focus primarily on functional requirements, often overlooking critical aspects like performance, robustness, and maintainability.

👉 Enter ARCHCODE: A Comprehensive Approach

ARCHCODE addresses this gap by introducing a novel framework that:

1. Automatically extracts both explicit and implicit requirements from textual descriptions
2. Uses these requirements to guide code and test case generation
3. Enables more thorough and targeted code evaluation

👉 Breaking Down the Innovation

✅ Functional vs. Non-Functional Requirements: A Simple Explanation

Imagine you're building a car.

Functional requirements are like specifying that the car should have four wheels and be able to move forward and backward.

Non-functional requirements, on the other hand, are like specifying the car's fuel efficiency, safety features, and ease of maintenance.

ARCHCODE considers both types of requirements, ensuring that the generated code not only works as intended but also performs well, is robust against errors, and is easy to maintain.

✅ Smarter Code Generation and Testing

By using extracted requirements to guide the process, ARCHCODE:

- Generates code tailored to meet specific requirements
- Produces test cases designed to verify each requirement
- Achieves more efficient and targeted code evaluation

This approach leads to higher quality code and more comprehensive testing, all while requiring significantly fewer test cases compared to existing methods.

👉 Impressive Benchmark Performance

ARCHCODE's effectiveness is demonstrated by its performance on key benchmarks:

- Outperforms existing methods, including GPT-4, on HumanEval and CodeContests benchmarks
- Achieves state-of-the-art results with 50x fewer generated test cases
- Introduces HumanEval-NFR, a new benchmark for evaluating non-functional requirements in code generation

👉 What This Means for QuantaLogic and the Industry

At QuantaLogic, we're excited about the potential of research as ARCHCODE to enhance our AI-powered software development platform.

This research aligns perfectly with our mission to bridge the gap between advanced AI models and their practical implementation in business processes.

https://t.co/B0r5do2ouc

[OpenAI @OpenAI]

OpenAI o1 translates a corrupted sentence.

[Seungtaek Choi @hist8233]

@Taesik_MobileAI @wewantsj Wow, huge congrats on this amazing news!

[Jan Leike @janleike]

Yesterday was my last day as head of alignment, superalignment lead, and executive @OpenAI.

[Andrej Karpathy @karpathy]

Day 24 of llm.c: we now do multi-GPU training, in bfloat16, with flash attention, directly in ~3000 lines of C/CUDA, and it is FAST! 🚀

We're running ~7% faster than PyTorch nightly, with no asterisks, i.e. this baseline includes all modern & standard bells-and-whistles: mixed precision training, torch compile and flash attention, and manually padding vocab. (Previous comparisons included asterisks like *only inference, or *only fp32 etc.) Compared to the current PyTorch stable release 2.3.0, llm.c is actually ~46% faster. My point in these comparisons is just to say "llm.c is fast", not to cast any shade on PyTorch. It's really amazing that PyTorch trains this fast in a fully generic way, with ability to cook up and run ~arbitrary neural networks and run them on a ton of platforms. I see the goals and pros and cons of these two projects as different, even complementary. Actually I started llm.c with my upcoming education videos in mind, to explain what PyTorch does for you under the hood.

How we got here over the last ~1.5 weeks - added:

✅ mixed precision training (bfloat16)
✅ many kernel optimizations, including e.g. a FusedClassifier that (unlike current torch.compile) does not materialize the normalized logits.
✅ flash attention (right now from cudnn)
✅ Packed128 data structure that forces the A100 to utilize 128-bit load (LDG.128) and store (STS.128) instructions.

It's now also possible to train multi-GPU - added:
✅ First version of multi-gpu training with MPI+NCCL
✅ Profiling the full training run for NVIDIA Nsight Compute
✅ PR for stage 1 of ZeRO (optimizer state sharding) merging imminently

We're still at "only" 3,000 lines of code of C/CUDA. It's getting a bit less simple, but still bit better than ~3 million. We also split off the fp32 code base into its own file, which will be pure CUDA kernels only (no cublas or cudnn or etc), and which I think would make a really nice endpoint of a CUDA course. You start with the gpt2.c pure CPU implementation, and see how fast you can make it by the end of the course on GPU, with kernels only and no dependencies.

Our goal now is to create a reliable, clean, tested, minimal, hardened and sufficiently optimized LLM stack that reproduces the GPT-2 miniseries of all model sizes, from 124M to 1.6B, directly in C/CUDA.

A lot more detail on: "State of the Union [May 3, 2024]"
https://t.co/eDgbngHrZ9

[IEEE ICASSP @ieeeICASSP]

Best Student Paper award: “STAR: Distilling Speech Temporal Relation for Lightweight Speech Self-Supervised Learning Models” by Kangwook Jang, Sungnyun Kim, and Hoirin Kim.
https://t.co/0SRSyagwOg

[Hyeonbin Hwang @ronalhwang]

🚨 New LLM Reasoning Paper 🚨

Q. How can LLMs self-improve their reasoning ability?

⇒ Introducing Self-Explore⛰️🧭, a training method specifically designed to help LLMs avoid reasoning pits by learning from their own outputs! [1/N]

[Seungtaek Choi @hist8233]

@ronalhwang Very interesting! Thanks for sharing such a nice work!

[Alex Warstadt @a_stadt]

LLMs are now trained >1000x as much language data as a child, so what happens when you train a "BabyLM" on just 100M words?

The proceedings of the BabyLM Challenge are now out along with our summary of key findings from 31 submissions: https://t.co/zli0jzA1XP

Some highlights 🧵

[Jason Wei @_jasonwei]

It was an honor to give a guest lecture yesterday at Stanford’s CS330 class, "Deep Multi-Task and Meta-Learning"!

I discussed a few very simple intuitions for how I personally think about large language models.

Slides: https://t.co/zKw2WGt5nG

Here are the six intuitions:

(1) First, I encouraged viewing next-word prediction as massive multi-task learning. Even though next-word prediction is very simple, because the pre-training data is so large and diverse, LMs learn a lot of tasks from next-word prediction. This can range from simple things like grammar to harder tasks like arithmetic reasoning. Anything that could be found in pre-training data could potentially be learned by a LM.

(2) Next, learning from <input, output> pairs (in-context learning) can be cast as next-word prediction. This was popularized by the GPT-3 paper (https://t.co/6GVaNVG8rY). It is very convenient to formulate tasks using <input, output> pairs since that is how we have done AI in the past decades. However, I am not sure how long that will prevail. We can do better by adding natural language instructions, showing how the reasoning works, enumerating boundary cases, giving examples of what not to do, etc.

(3) A fundamental observation is that tokens have very different information density. Some tokens are easy to predict (e.g., “large language ___” is obviously “model”). Other tokens are very hard to predict (e.g., answer to a math problem), and so LMs should spend more compute before trying to predict them. One way to do this is chain-of-thought prompting (https://t.co/NrYe656G81), which encourages LMs to give a reasoning path before giving the final answer, allowing them to do complex reasoning tasks. It is my dream that one day AI will be able to help us with extremely challenging tasks, such as writing a proposal to reduce climate change. Spending more compute on reasoning is a first step in that direction.

(4) Increasing compute for pre-training is expected to improve loss (scaling laws, https://t.co/Eiekmwbqgt). This seems trivial but the fact that loss hasn’t saturated implies that continued investment in scaling will likely produce more capable models. It is a natural question to ask why scaling improves performance; my two hand-wavy hypotheses are that (1) large LMs can memorize more knowledge about the world and (2) large LMs use more complicated heuristics to get the loss as low as possible.

(5) Although overall loss improves smoothly as you scale, individual tasks might improve suddenly (emergent abilities, https://t.co/jZryo9LkMB). Since next-word prediction is massive multi-task learning, you can view the loss as the weighted sum of many individual tasks. When you decrease the loss, it is likely that not all individual tasks improve uniformly. Loss for some tasks might be saturated (larger models no longer improving in grammar since they already have perfect grammar), and other tasks might improve in a more sudden fashion (in order to push loss lower, the larger model has to figure out how to do hard math problems).

(6) Finally, I argue that large LMs can actually learn at <input, output> relationships in context. While one paper showed that random labels in in-context examples barely hurts performance (https://t.co/DTWlKCDvld), our recent work found that language models can follow both flipped labels and semantically-unrelated labels (https://t.co/Qjj42GGRcY). The catch though, is that this ability only exists in language models that are large enough (e.g., GPT-3.5 and PaLM-1 or larger).

Some of these intuitions extendable beyond language:

- Intuition 3 (that tokens have different information density), might be generally applicable to most data. For example, in computer vision you may want to spend more compute analyzing the important parts of an image, like someone's facial expression.

- Intuition 4 (scaling laws) is applicable not just for compute, but whenever you collect finetuning data. You can plot the scaling curve with # of training examples on the x-axis and performance on the y-axis, and predict how much collecting more data will help.

- Intuition 5 (decomposing aggregate metrics into individual tasks) can be applicable whenever you’re using an aggregate metric. With finer-grained categories, you can a much better understanding of what is happening and find out which categories might need the most improvement.

View the longer-form summary blog here: https://t.co/bgX6Kef6Mz

The class is here: https://t.co/s0RxET8jzr

[Satya Nadella @satyanadella]

We remain committed to our partnership with OpenAI and have confidence in our product roadmap, our ability to continue to innovate with everything we announced at Microsoft Ignite, and in continuing to support our customers and partners. We look forward to getting to know Emmett Shear and OAI's new leadership team and working with them. And we’re extremely excited to share the news that Sam Altman and Greg Brockman, together with colleagues, will be joining Microsoft to lead a new advanced AI research team. We look forward to moving quickly to provide them with the resources needed for their success.