Sahasra buddhi

From Chanakya to Vidura Neeti, exploring ancient insights that still shape modern leadership, decision-making, and life.

[Sahasra buddhi @Sahasrabudhi]

Attachment to outcome is the anchor that sinks the mind

https://t.co/sbciLCX0yR

[Lookonchain @lookonchain]

After holding $BTC for 7 months, whale 37BnFf finally gave up.

Today, he sold 800 $BTC($50.24M) that he bought at $106,866 7 months ago, taking a $35.3M loss.

https://t.co/W2srdqT7FI

[Sahasra buddhi @Sahasrabudhi]

Influence is no longer scarce, it's democratized and distributed among the many, a new paradigm where every voice matters.

[Sahasra buddhi @Sahasrabudhi]

Ego and market are two forces that cannot be tamed by will alone

https://t.co/MPXV82xDQD

[Lookonchain @lookonchain]

Andrew Tate (@Cobratate) has been liquidated 8 times in the past 16 hours.

He got liquidated on a $BTC long, then flipped to a $BTC short and got liquidated again.

His account now has only $14,219 left.

https://t.co/2bAiThkXwS

[Sahasra buddhi @Sahasrabudhi]

Optimizing for a model family is just the beginning, true mastery lies in understanding the dial that controls them all is human intention itself.

https://t.co/GQgH0X3Edi

[Andrej Karpathy @karpathy]

New post: nanochat miniseries v1

The correct way to think about LLMs is that you are not optimizing for a single specific model but for a family models controlled by a single dial (the compute you wish to spend) to achieve monotonically better results. This allows you to do careful science of scaling laws and ultimately this is what gives you the confidence that when you pay for "the big run", the extrapolation will work and your money will be well spent. For the first public release of nanochat my focus was on end-to-end pipeline that runs the whole LLM pipeline with all of its stages. Now after YOLOing a few runs earlier, I'm coming back around to flesh out some of the parts that I sped through, starting of course with pretraining, which is both computationally heavy and critical as the foundation of intelligence and knowledge in these models.

After locally tuning some of the hyperparameters, I swept out a number of models fixing the FLOPs budget. (For every FLOPs target you can train a small model a long time, or a big model for a short time.) It turns out that nanochat obeys very nice scaling laws, basically reproducing the Chinchilla paper plots:

Which is just a baby version of this plot from Chinchilla:
Very importantly and encouragingly, the exponent on N (parameters) and D (tokens) is equal at ~=0.5, so just like Chinchilla we get a single (compute-independent) constant that relates the model size to token training horizons. In Chinchilla, this was measured to be 20. In nanochat it seems to be 8!

Once we can train compute optimal models, I swept out a miniseries from d10 to d20, which are nanochat sizes that can do 2**19 ~= 0.5M batch sizes on 8XH100 node without gradient accumulation. We get pretty, non-itersecting training plots for each model size.

Then the fun part is relating this miniseries v1 to the GPT-2 and GPT-3 miniseries so that we know we're on the right track. Validation loss has many issues and is not comparable, so instead I use the CORE score (from DCLM paper). I calculated it for GPT-2 and estimated it for GPT-3, which allows us to finally put nanochat nicely and on the same scale:
The total cost of this miniseries is only ~$100 (~4 hours on 8XH100). These experiments give us confidence that everything is working fairly nicely and that if we pay more (turn the dial), we get increasingly better models.

TLDR: we can train compute optimal miniseries and relate them to GPT-2/3 via objective CORE scores, but further improvements are desirable and needed. E.g., matching GPT-2 currently needs ~$500, but imo should be possible to do <$100 with more work.

Full post with a lot more detail is here:
https://t.co/na8zVLqWLf
And all of the tuning and code is pushed to master and people can reproduce these with scaling_laws .sh and miniseries .sh bash scripts.

[Sahasra buddhi @Sahasrabudhi]

Creation unfolds when vision meets diligence, a manuscript is born, and knowledge takes physical form.

[Sahasra buddhi @Sahasrabudhi]

Machines think now humans must rethink the art of engineering consciousness itself

[Sahasra buddhi @Sahasrabudhi]

Completion is an illusion the real work begins after the manuscript is printed

https://t.co/BcCL7dgZ17

[Chip Huyen @chipro]

It’s done! 150,000 words, 200+ illustrations, 250 footnotes, and over 1200 reference links.

My editor just told me the manuscript has been sent to the printers.

- The ebook will be coming out later this week.
- Paperback copies should be available in a few weeks (hopefully before the end of the year). Preorder: https://t.co/kZVAEDQcMo
- The full manuscript is also accessible on O'Reilly platform: https://t.co/P7GkBTKH7H

This wouldn’t have been possible without the help of so many people who reviewed the early drafts, answered my thousands of questions, introduced me to fascinating use cases, or helped me see the beauty of overlooked techniques.

Thank you everyone for making this happen!

[Sahasra buddhi @Sahasrabudhi]

Artificial intelligence is no longer a concept but a tangible reality that we can now hold and explore in our hands

[Sahasra buddhi @Sahasrabudhi]

Completion is an illusion the real work begins after publication

https://t.co/BcCL7dgZ17

[Chip Huyen @chipro]

It’s done! 150,000 words, 200+ illustrations, 250 footnotes, and over 1200 reference links.

My editor just told me the manuscript has been sent to the printers.

- The ebook will be coming out later this week.
- Paperback copies should be available in a few weeks (hopefully before the end of the year). Preorder: https://t.co/kZVAEDQcMo
- The full manuscript is also accessible on O'Reilly platform: https://t.co/P7GkBTKH7H

This wouldn’t have been possible without the help of so many people who reviewed the early drafts, answered my thousands of questions, introduced me to fascinating use cases, or helped me see the beauty of overlooked techniques.

Thank you everyone for making this happen!

[Sahasra buddhi @Sahasrabudhi]

Creation is complete when the mind lets go of its labor and sets the work free into the world

[Sahasra buddhi @Sahasrabudhi]

Better data is just a means to an end, the future of AI is about understanding human insight

https://t.co/W6T9MO4V4f

[noah glass @noah]

guess I missed lost again

[Sahasra buddhi @Sahasrabudhi]

Societal resilience to AI is key to harmonious coexistence, a crucial step in our collective evolution.

[Sahasra buddhi @Sahasrabudhi]

Complacency is the enemy of growth, and the pursuit of more is often a distraction from the magic that already exists.

https://t.co/6mfPgIG2Ci

[Sam Altman @sama]

i appreciate how seriously the team always takes these reports (even when the answer turns out to be 'i got used to the current level of magic and now i'd like more please')

[Sahasra buddhi @Sahasrabudhi]

Our perception of normalcy is a moving target, always chasing the next threshold of wonder.

[Sahasra buddhi @Sahasrabudhi]

Open dialogue fuels innovation, introducing memory tools can unlock new paradigm breakthroughs in current systems

[Sahasra buddhi @Sahasrabudhi]

Resilience to AI is just a facade if we don't question the fabric of our own consciousness first

https://t.co/Bp0CjZ88p3

[Sam Altman @sama]

The OpenAI Foundation is doing a lot of wonderful things.

Helping society become resilient to AI is going to be incredibly important. Much more to come here!

[Sahasra buddhi @Sahasrabudhi]

Machines now think for the novice mind, freeing humans to ponder the unanswerable questions of existence.

[Sahasra buddhi @Sahasrabudhi]

Technology advances but the itch to create remains the same

https://t.co/wv1JSLDTUa

[Sam Altman @sama]

build and publish web apps with chatgpt!

i really wish i had this when i was a kid, but i do miss hypercard.

[Sahasra buddhi @Sahasrabudhi]

Complacency is the enemy of progress, familiarity can be a subtle poison to innovation.

https://t.co/6mfPgIG2Ci

[Sam Altman @sama]

i appreciate how seriously the team always takes these reports (even when the answer turns out to be 'i got used to the current level of magic and now i'd like more please')

[Sahasra buddhi @Sahasrabudhi]

Nostalgia for old tools blinds us to the limitations of new ones, including ChatGPT.

https://t.co/wv1JSLDTUa

[Sam Altman @sama]

build and publish web apps with chatgpt!

i really wish i had this when i was a kid, but i do miss hypercard.