Millennium Bismay

Applied Scientist @Amazon

[Millennium Bismay @Its_MB16]

Tired of constantly checking if you're about to hit rate limits? Or forgetting which worktree you're on mid-session? Checkout this minimalistic status line that shows most of the things at a glance! https://t.co/ICh8Sudco6

[Millennium Bismay @Its_MB16]

I built ResearchClaw, an open-source, self-hosted system that automatically discovers, classifies, and summarizes research papers based on your interests. https://t.co/Nykx5BvMvl

[Millennium Bismay @Its_MB16]

@karpathy I sometimes do the first two steps in reverse with Deep Research for literature surveys. Never thought about the third one.. Thank you @karpathy for the insight.

In the near future do you believe we will see SEO for agets and humans separately, probably every website with a .md?

[Millennium Bismay @Its_MB16]

@Pokee_AI POKEE

[Andrej Karpathy @karpathy]

Excited to release new repo: nanochat!
(it's among the most unhinged I've written).

Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single, dependency-minimal codebase. You boot up a cloud GPU box, run a single script and in as little as 4 hours later you can talk to your own LLM in a ChatGPT-like web UI.

It weighs ~8,000 lines of imo quite clean code to:

- Train the tokenizer using a new Rust implementation
- Pretrain a Transformer LLM on FineWeb, evaluate CORE score across a number of metrics
- Midtrain on user-assistant conversations from SmolTalk, multiple choice questions, tool use.
- SFT, evaluate the chat model on world knowledge multiple choice (ARC-E/C, MMLU), math (GSM8K), code (HumanEval)
- RL the model optionally on GSM8K with "GRPO"
- Efficient inference the model in an Engine with KV cache, simple prefill/decode, tool use (Python interpreter in a lightweight sandbox), talk to it over CLI or ChatGPT-like WebUI.
- Write a single markdown report card, summarizing and gamifying the whole thing.

Even for as low as ~$100 in cost (~4 hours on an 8XH100 node), you can train a little ChatGPT clone that you can kind of talk to, and which can write stories/poems, answer simple questions. About ~12 hours surpasses GPT-2 CORE metric. As you further scale up towards ~$1000 (~41.6 hours of training), it quickly becomes a lot more coherent and can solve simple math/code problems and take multiple choice tests. E.g. a depth 30 model trained for 24 hours (this is about equal to FLOPs of GPT-3 Small 125M and 1/1000th of GPT-3) gets into 40s on MMLU and 70s on ARC-Easy, 20s on GSM8K, etc.

My goal is to get the full "strong baseline" stack into one cohesive, minimal, readable, hackable, maximally forkable repo. nanochat will be the capstone project of LLM101n (which is still being developed). I think it also has potential to grow into a research harness, or a benchmark, similar to nanoGPT before it. It is by no means finished, tuned or optimized (actually I think there's likely quite a bit of low-hanging fruit), but I think it's at a place where the overall skeleton is ok enough that it can go up on GitHub where all the parts of it can be improved.

Link to repo and a detailed walkthrough of the nanochat speedrun is in the reply.

[Millennium Bismay @Its_MB16]

@miramurati @grok summarize the blog post

[Millennium Bismay @Its_MB16]

@chipro Yes. We as humans wont evolve unaided in a span of 5 years to move our brain capability notches above the current cap. Thats why Agentic AI is not another simple Tractor moment which many people compare it with.

[Millennium Bismay @Its_MB16]

@Geetashloks Jai Shri Ram

[Millennium Bismay @Its_MB16]

@Vipintiwari952 Khaleel is the replacement for Deepak. And I think CSK squad is a 8 in your scale.

[Millennium Bismay @Its_MB16]

@levelsio He thinks therefore he is!

[Millennium Bismay @Its_MB16]

@levelsio And that's how, Sam became the extension to a sofa pillow!

[Millennium Bismay @Its_MB16]

I am starting a blog series - "All You Need to Know about RAG" with a special focus on building Agentic RAG. We will dive deep into the inner workings of the most customizable components of RAG systems.
https://t.co/BpcnmYHaAG

[Millennium Bismay @Its_MB16]

Historic win for the @usacricket This win exemplifies the importance of @T20WorldCup having more teams. This will boost Cricket in many nations. Sheer excellence from Monank, Jones, Netravalkar, and Kenjige!

[Millennium Bismay @Its_MB16]

@ShreyasIyer15 @IPL @BCCI @SunRisers @hardikpandya7 @imjadeja I hope the rules be clear before the Mega Auction and may there be more retentions available, for teams have created such amazing talents and they should reap the fruits of what they have sown!
@IPL @BCCI

[Millennium Bismay @Its_MB16]

As the @IPL 2024 comes to a culmination, Thankfully, the best team throughout the tournament won... Congratulations @KKRiders
It has been a tremendous year!

But, this has been the worst of the seasons so far and we should make sure this never repeats. A thread...

[Millennium Bismay @Its_MB16]

@ShreyasIyer15 @IPL @BCCI @SunRisers 4. We would have never got a @hardikpandya7, a @imjadeja or a Axar Patel with this Impact sub rule. Please scrap it off @BCCI from @IPL or It will kill budding allrounders who are trying to improve their weaker skills!

[Millennium Bismay @Its_MB16]

I believe Tilak needs to be on that T20WC team!

Rohit
Virat
Sanju
Tilak
SKY
Dube
Pant
Hardik
Rinku
Jadeja
Axar
Kuldeep
Bumrah
Natrajan
Sandeep

@BCCI @bhogleharsha @ImRo45