Nick

Machinist - Proud Grandfather - Metalhead - AI Engineering enthusiast. Training models from scratch on a 5090 in my living room. - e/acc

[Nick @xentryx]

Sorry if posted. This is great!

[Bright Mirror @_brightmirror]

Made with Claude Opus 5.5.

The fearmongering about AI always makes us forget that NOTHING WENT FOOM, as was always predicted.

Send this to your doomer friend who has a very high P(Doom).

Accelerate.

[Bright Mirror @_brightmirror]

Made with Claude Opus 5.5.

The fearmongering about AI always makes us forget that NOTHING WENT FOOM, as was always predicted.

Send this to your doomer friend who has a very high P(Doom).

Accelerate.

[Nick @xentryx]

@beffjezos Honestly? Fuck em’. You were thrown into a crazy situation and dealt with it best you knew how. Fighting for the future of humanity is no small task.

[Nick @xentryx]

Glad to be here.

To the future.

[Nick @xentryx]

@paradigmainc Can’t wait for that paper!

[fullstack @DavidFSWD]

Training a 2B model from scratch on a 2080 Ti (11GB).

84% of it is a 1.7B engram table injected into the residual stream. 312M is transformer, 20M active per token. fused MoE, val 3.812 19h in, 18d left

[Nick @xentryx]

round 1: teacher T0 generates a corpus of reasoning traces. Not just answers but full chains of thought, with the dead ends included. Student S1 trains on those traces. Standard distillation so far. Round 2: S1 becomes T1. It generates a new corpus. But here’s the beautiful catch, T1 generates traces that are cleaner than the ones it trained on, because during training it learned which parts of T0’s reasoning were strong and which were noise. The distillation acts as a filter. Signal goes up. Round 3 onward: this compounds. Each generation’s traces are cleaner than the last. The model is not learning new facts about the world. It is learning to think more efficiently about the facts it already has. Compression of cognition. Am I on the right track? 😉

[Nick @xentryx]

@percyliang Seeing those tokens per second, makes me cry a little inside. NGL.

[Larry Dial @classiclarryd]

New NanoGPT Speedrun WR at 68.0 (-5.8s) from @theonlyglitch_ , with a decrease of QK dim from 128 to 96, a kernel fusion of [QK Norm, RoPE, KeyOffset, paired head layout] into Triton, and moving MLP bwk, QKV fwd/bwk to FP8. This is a heavily involved PR with over 1k lines of triton, 4 different FP8 scaling protocols, and fancy register aware epilogue placement.

Two takeaways: 1) From a model perspective, if QK dims smaller than 128 works better at nano scale, then perhaps larger than 128 works better at Hero scale. 2) This Nvidia Engr is super legit. https://t.co/qBrhWUuvI4

[Nick @xentryx]

@iaro_e Shame it won’t work with my 5090. 🤬

[Nick @xentryx]

@TheAhmadOsman Trump has the IQ of a rotten grape but he’s not wrong about this one. We can’t stop AI development. We just can’t.

[Chris Hayduk @ChrisHayduk]

I work at OpenAI and personally think AI has been and will continue to be an extremely beneficial technology to humanity

The conversation should be around how many billions of lives it will save

[Nick @xentryx]

@elonmusk @TimSweeneyEpic @ParkerThayer Was Bernie part of it or behind it?

[Nick @xentryx]

@mark_k Well would you look at that. I wouldn't be surprised in the least if it comes out that Sanders was involved. The timing? Coincidence? I think not.

[Nick @xentryx]

@beffjezos This is truly awful. I have a feeling this guy is a huge burning Sanders supporter and I wouldn’t put it past him that he actually had a conversation with Bernie before pulling this stunt.

[Nick @xentryx]

@mark_k Still waiting to be added 😄

[Nick @xentryx]

@mobav0 @sama Incredible.

[Nick @xentryx]

@OpenBMB This is amazing. I’ve been a bit directionless post training my own model. Not any more. Thank you.

[Nick @xentryx]

@mark_k Oh yeah. Let’s do this.