Hyogon Ryu

building llm @krafton_ai(@pubg) | @kaist_ai @CykorKU

[Hyogon Ryu @ugonfor]

We’ve finally released KRAFTON’s first open-weight model, trained from scratch!

We’re just getting started. Stay tuned!

[Kangwook Lee @Kangwook_Lee]

👋 KRAFTON's Raon is entering the open-weight scene!

Meet Raon-1 mini, our first model, trained from scratch.

On par with leading models in its size class, while running faster and using less memory.

Available under the Apache 2.0 license.
https://t.co/fOFoUcfDLc

[Hyogon Ryu @ugonfor]

@ming85GG @Kangwook_Lee Correction: that's plain decoding! With the drafter, it's much faster. 463 tps 😉

[Hyogon Ryu @ugonfor]

🚨 245 tps on RTX 5090/vLLM? TOO SLOW 🥱
Raon-1 mini hits 463 tps with the official drafter 😉

Correction: @Kangwook_Lee ’s MLX demo uses the drafter, but the 245 tps on RTX 5090 is without it.

[Kangwook Lee @Kangwook_Lee]

Raon-1 mini (+ official drafter) is the fastest model in its class

165 tps on M4 Max/MLX

245 tps on RTX 5090/vLLM

[Hyogon Ryu @ugonfor]

Acces Request is gone! Thank you for feedbacks!
Try it on Huggingface now! (Apache 2.0): https://t.co/Zm995s11Ae

[Hyogon Ryu @ugonfor]

We’ve finally released KRAFTON’s first open-weight model, trained from scratch!

We’re just getting started. Stay tuned!

[Kangwook Lee @Kangwook_Lee]

👋 KRAFTON's Raon is entering the open-weight scene!

Meet Raon-1 mini, our first model, trained from scratch.

On par with leading models in its size class, while running faster and using less memory.

Available under the Apache 2.0 license.
https://t.co/fOFoUcfDLc

[Kangwook Lee @Kangwook_Lee]

🫡 request access form is gone!

[Hyogon Ryu @ugonfor]

Try our model here! https://t.co/ozCQZ6hKOo

[KRAFTON AI @Krafton_AI]

Today, we’re releasing a preview of Raon-1 mini, a 30B MoE LLM with 4B active parameters, trained from scratch by KRAFTON AI.

With our efficient architecture, it matches Muse Glimmer on coding/agentic tasks while being much faster and more memory-efficient! 🔥 Plus, natural, human-like conversation.

This is just a first look. The final release (and bigger models) are coming soon! 👀

👉 Try it now, free for a limited time: https://t.co/uSrLFIGPWy
🤗 Open weights + DFlash drafter (Apache 2.0): https://t.co/XWVmxsDRei

[Hyogon Ryu @ugonfor]

I want to understand how frontier labs are managing their codebases these days.

I’ve read articles like this one(https://t.co/c74gpDQakG), and I completely agree with the concerns they raise, but they never seem to offer concrete solutions.

I feel less need to read the code myself now that I no longer write it myself. But as I spend less time reading it, the code quality keeps getting worse. That, in turn, makes the agent’s output worse and eventually makes it harder to ship code.

How can I break this cycle?

original post: https://t.co/rEQjTyJISx

[Hyogon Ryu @ugonfor]

This is what I always wondering since i started large model training. Hardware resource will be more heterogeneous, but i wanted to utilize all of the compute resource in same training.

[Nan Jiang @nanjiangwill]

Spent the past three weeks on this idea.

A sampler on a different precision or accelerator is just off-policy, like stale samples in fully async RL, and off-policy correction doesn't care why the sampler differs from the trainer. Meanwhile, inference keeps diversifying across precisions(MXFP8, MXFP4, NVFP4) and accelerators outside RL. So why not use all of that compute for RL rollout?

The sampler and the trainer are two systems built for different purposes. Hope this opens up new possibilities for RL algorithms and systems, and leads to cleaner system designs. Enjoy the blog, and stay tuned for the next one on our infra :)

[Hyogon Ryu @ugonfor]

@ipwning Wow

[Deven Pietrzak @DevenPzak]

I really enjoyed working on this project. I believe the new flop-aware paradigm of nanoGPT thinking is useful, and it seems helpful in a compute optimization context like this one. It seems like a simple and logical conclusion of optimization: figure out what’s actually needed, and use that. The process was actually fun.

One note regarding this record, as noted in the blog. Some of the techniques that scale to frontier pretraining have been redacted from this record. Most importantly, ANVIL III, the significantly improved version of the ANVIL optimizer lineage, has not been included in this record, as it is proprietary to us at @hyperstition_cc. It replicates Muon’s speed but beats it by 20-28 millinats of loss at all model sizes 124M-1.2B and training depths (up to 8x-Chinchilla), with remarkably consistent hyperparameters and extremely high SNR, as compared to Muon. You can learn more about ANVIL III and Feather-1.7B, the model we trained using internal architectural changes and ANVIL III, here:
https://t.co/PyfvnSSwVC

Using ANVIL III (which allowed me to cut steps) improved the speedrun results by another 0.7 seconds, with lower net loss. This alone proves there’s still more room to improve this record.

One final thing that I’ve had many people ask about is how much of a role autoresearch played in my work. The answer is almost none. While almost all of the implementation was done with Claude Code, I did the ideation almost fully by myself. I spoke about this extensively in a conversation with @METR_Evals.

We had a cluster of H100s while I was working on the record. At night or when I stopped working on it, I’d send out several agents to try to hill-climb the speedrun and improve the record. In total, the agents made shockingly little progress, despite spending more time in aggregate than I did. No matter what I told them, it seemed difficult to get them out of local minimums, stuck performing futile tuning of a method that saved 0.1 seconds and cost ~5 millinats (clearly a bad trade), or just tuning hyperparameters for hours on end trying to make some small change. I saw similar phenomena as those described in this METR report:
https://t.co/ZtKPp2w7nT.

However, as I also discussed with METR, I don’t think the agents lack the knowledge to do this. Instead, I think a lot of it comes from a personality issue with current LLMs. They have very little belief that large jumps like this are possible, or they approach it with the wrong paradigm. Autoresearch was not very helpful in my work, and, from personal experience, I don’t think this current generation of publicly available models, without special harnesses, will do very high-quality autoresearch. However, I think the capability and knowledge is already here, which implies that we could be significantly closer to true autoresearch than recent results would imply.

NanoGPT was a really fun challenge and it led to a number of valuable insights and processes. Finally, I’d like to give @Classiclarryd and @KellerJordan a huge thank you for all of their efforts maintaining the repo.

[Hyogon Ryu @ugonfor]

Crazy work.
It wonderfully utilize the remaining gpu resources…

[Larry Dial @classiclarryd]

New historic NanoGPT record at 39.9s (-27.7s) from @DevenPzak , obliterating the prior record of 67.6s!

This record introduces a new paradigm of thinking to NanoGPT: instead of optimizing matmuls or adding more expressive operations, optimize at the individual flop level with incredibly clever engineering and ML judgement. If a flop is low value on a particular step, skip it.
Specifically:
-(~8s) Sampled softmax. If a token doesn’t appear in a batch, skip its lm_head fwd/bwd some fraction of the time.
-Sparse values. Only run an optimizer step for ngram embeddings that occurred in the batch. Set beta1 to zero to enable this. Beta2 is applied retroactively when the row is later used.
-Sparse updates. Only update ngram and value embeddings once every 4 steps instead of once every 2.
-Sparse communication. Shard the n-gram table across GPUs, and only pass the rows receiving updates on each step.
-Sparse optimizer states. For the n-gram table, reduce from 2 floats in Adam optimizer per param, to 1 float per 768 params.
-Hand-rolled flash attention for 64 dim heads.

There are several additions that add accuracy too:
-(~4s) EMA during last 300 steps, combined with lifting final_lr to 0.3 instead of 0.15.
-(~1s) A new optimizer, Anvil2, which expands muon via a second tracked momentum buffer, improves the ortho coefficients, and modifies the cautious weight decay application.
-A couple additional dynamic skip connections in the network.

The most striking consequence of the ‘flop aware paradigm’ is you can grow parameters arbitrarily large,
only limited by the available memory, since you can selectively choose how to expend flops on those parameters on each step. NanoGPT has kept active parameters below 124M, but total is unbounded, and has grown to 640M through embedding sparsity over the last year. This PR takes that to its logical conclusion on the 8xH100, scaling up to 65B sparse embedding parameters, which accounts for 25% of the PR’s gains. At frontier scale, where one is not bounded by an 8xH100, one could imagine where this paradigm could lead.
https://t.co/Ycrzy6JFC3

As this was a very notable PR, I spoke with Deven for an hour to learn how he did it. Here’s his story on the changes: https://t.co/YEfM1VpOTi

[Larry Dial @classiclarryd]

New historic NanoGPT record at 39.9s (-27.7s) from @DevenPzak , obliterating the prior record of 67.6s!

This record introduces a new paradigm of thinking to NanoGPT: instead of optimizing matmuls or adding more expressive operations, optimize at the individual flop level with incredibly clever engineering and ML judgement. If a flop is low value on a particular step, skip it.
Specifically:
-(~8s) Sampled softmax. If a token doesn’t appear in a batch, skip its lm_head fwd/bwd some fraction of the time.
-Sparse values. Only run an optimizer step for ngram embeddings that occurred in the batch. Set beta1 to zero to enable this. Beta2 is applied retroactively when the row is later used.
-Sparse updates. Only update ngram and value embeddings once every 4 steps instead of once every 2.
-Sparse communication. Shard the n-gram table across GPUs, and only pass the rows receiving updates on each step.
-Sparse optimizer states. For the n-gram table, reduce from 2 floats in Adam optimizer per param, to 1 float per 768 params.
-Hand-rolled flash attention for 64 dim heads.

There are several additions that add accuracy too:
-(~4s) EMA during last 300 steps, combined with lifting final_lr to 0.3 instead of 0.15.
-(~1s) A new optimizer, Anvil2, which expands muon via a second tracked momentum buffer, improves the ortho coefficients, and modifies the cautious weight decay application.
-A couple additional dynamic skip connections in the network.

The most striking consequence of the ‘flop aware paradigm’ is you can grow parameters arbitrarily large,
only limited by the available memory, since you can selectively choose how to expend flops on those parameters on each step. NanoGPT has kept active parameters below 124M, but total is unbounded, and has grown to 640M through embedding sparsity over the last year. This PR takes that to its logical conclusion on the 8xH100, scaling up to 65B sparse embedding parameters, which accounts for 25% of the PR’s gains. At frontier scale, where one is not bounded by an 8xH100, one could imagine where this paradigm could lead.
https://t.co/Ycrzy6JFC3

As this was a very notable PR, I spoke with Deven for an hour to learn how he did it. Here’s his story on the changes: https://t.co/YEfM1VpOTi

[Kangwook Lee @Kangwook_Lee]

🧵 It's finally here! The full technical report on PUBG Ally is finally on Arxiv: https://t.co/C1ULXBYVSQ

Sharing a few highlights in the thread!

[Hyunseung Kim @mynsng_]

Excited to share the PUBG Ally Tech Report!
📄 https://t.co/kiVncTRppn

In June, we introduced PUBG Ally, the world’s first in-game AI agent.

Check out the unique challenges behind its architecture, data, training, evaluation, and on-device optimization!

[Hyogon Ryu @ugonfor]

@ipwning Hey

[Hyogon Ryu @ugonfor]

Okay. It’s time to solve my last remaining(since 2021) https://t.co/1r9E4zSg4x problem.

[Hex-Rays SA @HexRaysSA]

The official IDA MCP Server is here. It's free, open source, and works with any LLM.

Your agent writes IDAPython, uses ~20% fewer tokens, and can share an IDB with you in real time.

𝚞𝚟𝚡 𝚒𝚍𝚊-𝚑𝚌𝚕𝚒 𝚖𝚌𝚙 𝚒𝚗𝚜𝚝𝚊𝚕𝚕

https://t.co/jXPiF6Wyvk
(copy-paste: uvx ida-hcli mcp install)

[Hyogon Ryu @ugonfor]

@adithya_s_k No accuracy difference between shuffled & curriculum?

[elie @eliebakouch]

xiaomi MiMo oss RL environments + training code

https://t.co/ydWIbjoLdU
https://t.co/YsBVwO9Is2