kyokopom

Investigador independiente de Modelos pequeños de IA, vtuberMex, VtuberESP, Trainer LLM, actualmente estoy desarrollando mi propio modelo mini

[kyokopom @Kyokopom]

Right now my goal is to develop the first open-source model trained in FP4 using PyTorch primitives and contribute to its development. I know it’s tough, but it’s going to be a journey where I’ll need a lot of help. The first step would be training NanoGPT on an RTX 5090. The main challenge will be implementing native primitives and public APIs to make it possible. FP4 training is an area NVIDIA is currently digging into, and with their recent implementation and training of Nemotron Nano 2, they showed that FP4 training can be commercially and practically viable. Still, it’s hard for most people to get access. FP8 has been half-standardized already, now it’s FP4’s turn.

If someone could teach me more about CUDA or ROCm, that would be super helpful since I’m still kinda new to this space. And if anyone could also support me with compute, I’d really appreciate it 🙏.

I’ll be sharing all the progress of this research in a public repo, along with work on XIELU and harmonic loss. Hopefully you’ll enjoy following along!

[Dammy @OlatunjiAyokan2]

🚨Mercor is now paying people to learn.

They rejected you before.

Now they want to train you.

Complete the Academy, get certified, unlock $50–$100/hr projects.

🧵👇

[Goodfire @GoodfireAI]

Predictive data debugging lets you reveal and shape what a model learns, before you train.

We're grateful to the Ai2 team for open-sourcing their full post-training stack to enable research like this!

[Brian Zheyuan Zhang @zheyuanzhang99]

Existing continual learning methods still suffer from catastrophic forgetting. Our recent work found a simple recipe to substantially improve 𝗹𝗼𝗻𝗴-𝗵𝗼𝗿𝗶𝘇𝗼𝗻 𝗺𝗲𝗺𝗼𝗿𝗶𝘇𝗮𝘁𝗶𝗼𝗻:

Composing multiple continual learning mechanisms extends memory lifetime and raises average final retention from 1.2% to 34.9% across 3 datasets: a 28-fold improvement over naive sequential SFT.

Project page: https://t.co/LReleAxEUh
Paper: https://t.co/jLZWGcUFIK
Code + datasets: https://t.co/uwlTrYKbJ2

[Adam Shai @adamimos]

Training data for LLMs is made up of many sources. Given that, what structure should we expect in the activations?

New work from Simplex shows how the belief geometry over this type of data forms telescoping cones, and transformers represent them! 🧵👇

https://t.co/hBi2r1e32j

[Yacine Mahdid @yacinelearning]

I am big fan of "transformer in various setting is forced to learn to predict itself and build up a belief state"

[Hadi M. Dolatabadi @hmdolatabadi]

Yesterday, we've looked at how PyTorch manages GPU memory. Let's now look at what torch.compile does to the computation itself, and why a model that compiles successfully can still be slow.

1/n 🧵

[Katie Everett @_katieeverett]

Our new paper shows that optimizer memory schedules can *outscale* AdamW across the overtraining axis…in Transformers!

Joint work with @ShikaiQiu 🧵

[alphaXiv @askalphaxiv]

“Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference”

Layer dropout mostly disappeared from modern LLM pretraining because it was thought to hurt accuracy at scale.

But this Cerebras paper shows the opposite when configured correctly.

They apply layer dropout at the whole Transformer-block level, sampled per sequence, use an increasing dropout distribution across depth, and a decreasing dropout schedule over training that decays to zero.

Across 2,400 runs up to 8.2B parameters, this saves up to 25% of training FLOPs while preserving accuracy, and makes models depth-elastic enough for early exit, layer skipping, and up to 1.55x faster self-speculative decoding.

https://t.co/TwjAJYdk7z

[Radical Numerics @RadicalNumerics]

1/5 We pretrained Omnii, a multimodal language model on DNA. Then, we asked it to design a cancer vaccine

In new research, we show how Omnii learned to turn a tumor DNA sequence into a personalized mRNA vaccine design.

Read more here: https://t.co/0Uw5yLn8pd

[Hadi M. Dolatabadi @hmdolatabadi]

PyTorch can run out of memory with 60 GB reserved and only 6 GB used by tensors. I was looking into its native CUDA allocator to understand where the other 54 GB can get stuck and thought to put this together in a visual guide.

1/n 🧵

[Bingxiang He @HBX_hbx]

Introducing JustRL II 🚀

Building on JustRL (https://t.co/lwt1gYNDpG), we took a closer look at how GRPO behaves in long-CoT RL (128k). The group-mean baseline is a great fit for short traces, but over tens of thousands of tokens it gives a coarse, response-level signal.

JustRL II keeps GRPO's group structure and adds a critic for token-level credit assignment. Just as simple, keeps improving where GRPO levels off: AIME25 61→81 on a 2B model.

The same recipe powers the RL stage of MiniCPM5-2B (https://t.co/bMA9848id4), making it SOTA among models under 4B.

Data + checkpoints are open. Code lands this week. 📖 https://t.co/GlzPgZpDEC

[Zhanpeng Zhou @zhanpeng_zhou]

Recently, I came across NeXus https://t.co/x0o8K8vscK, an optimizer with promising generalization results that can be viewed as a counterpart to gradient accumulation.

Instead of the claimed "closeness" bias in original paper, we take a sharpness analysis of NeXus. Here is my result: https://t.co/pS80t3PmVE

TL;DR. We compare NeXus with plain SGD inner steps to gradient accumulation. Both the river–valley and linear stability analyses suggest a bias toward flatter minima in NeXus under suitable assumptions.

Disclosure: Parts of the proofs were inspired by discussions with GPT-6 Astra.

[Alexi Gladstone @AlexiGlad]

Energy-Based Transformers are pretty decent i guess

[Dammy @OlatunjiAyokan2]

10,000 openings. $50–$90/hr.

I just applied. Let’s see what happens.

[micro1 @micro1_ai]

We are hiring 10,000 robotics trainers over the next 7 days for one of our biggest projects yet.

We're looking for thousands of participants to help train robots by reviewing and labeling videos of them performing tasks.

- Role is open to English speaking and Western countries
- No prior AI experience required
- You'll review and label videos of robots performing tasks to help them improve
- Hourly rate: $50-90/hr
- Referral bonus: $300 per successful referral

Apply via the link in comments!

[Ali Ansari @aliansarinik]

we’re hiring 10,000 robotics trainers in the next 7 days.

$50–$90/hour, accepting applicants globally. you’ll review and label videos of robots performing tasks to help them improve. no prior AI experience required.

an entirely new category of work is emerging around teaching robots how to interact with the world.

application link in the comments below.

[Thien Tran @gaunernst]

Thank you for everyone who commented on other public MXFP8 kernels that I didn't know about. At least for the toy 8192^3 benchmark shape here, K3-generated Gluon is the best 😳
I haven't checked all the kernels closely but I'm sure they can shine under different GEMM shapes + tuning knobs. Having an assemble of kernels is useful for autotuning - it is effortless thanks to agents now.

[Gill @gurtej__gill_]

This paper points out something that should have been obvious to all of us.

We’ve spent the last 2 years throwing massive diffusion models at offline RL actors just to handle multimodal data, and just pretending that a 50ms denoising loop per step is fine for robotics.

The reality is it isn’t.

The argument of this paper is dead simple: the critic gets thrown away the second training ends.

The actor is the only thing that actually has to run live on hardware.

So why on earth are we stuffing all the compute into the actor?

People avoided deep critics before because offline value networks love to diverge into oblivion, but they managed to stabilize a deep residual critic using n-step targets and categorical cross-entropy.

That lets the critic do the heavy lifting during training, so runtime is left to a tiny deterministic policy that runs in a single forward pass.

It matches diffusion performance on OGBench while running 4x faster, no distillation hacks needed.

And i feel its genuinely a great work and definitely worth a read.

Read the full paper here: https://t.co/i0FVJuJUR2

[Hanseul Cho @hanseuljo]

We proved SOTA (until 2026.09.04) lower bounds on GD convergence rates for smooth convex minimization. Now, the gap is O(N^-1.271) vs Ω(N^-1.450) for the non-anytime rate, and O(N^-1.119) vs Ω(N^-1.184) for the non-anytime rate.

Paper: https://t.co/P39ZUz0rvE

[Alex Shtoff @AlexShtf]

I dont know how they did it, but it's not bad!

https://t.co/WUYqeU2Jx5