Ken

Web3 funs ‖ sharing the resources for python programming, data science and automation solutions

[Andrej Karpathy @karpathy]

Excited to release new repo: nanochat!
(it's among the most unhinged I've written).

Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single, dependency-minimal codebase. You boot up a cloud GPU box, run a single script and in as little as 4 hours later you can talk to your own LLM in a ChatGPT-like web UI.

It weighs ~8,000 lines of imo quite clean code to:

- Train the tokenizer using a new Rust implementation
- Pretrain a Transformer LLM on FineWeb, evaluate CORE score across a number of metrics
- Midtrain on user-assistant conversations from SmolTalk, multiple choice questions, tool use.
- SFT, evaluate the chat model on world knowledge multiple choice (ARC-E/C, MMLU), math (GSM8K), code (HumanEval)
- RL the model optionally on GSM8K with "GRPO"
- Efficient inference the model in an Engine with KV cache, simple prefill/decode, tool use (Python interpreter in a lightweight sandbox), talk to it over CLI or ChatGPT-like WebUI.
- Write a single markdown report card, summarizing and gamifying the whole thing.

Even for as low as ~$100 in cost (~4 hours on an 8XH100 node), you can train a little ChatGPT clone that you can kind of talk to, and which can write stories/poems, answer simple questions. About ~12 hours surpasses GPT-2 CORE metric. As you further scale up towards ~$1000 (~41.6 hours of training), it quickly becomes a lot more coherent and can solve simple math/code problems and take multiple choice tests. E.g. a depth 30 model trained for 24 hours (this is about equal to FLOPs of GPT-3 Small 125M and 1/1000th of GPT-3) gets into 40s on MMLU and 70s on ARC-Easy, 20s on GSM8K, etc.

My goal is to get the full "strong baseline" stack into one cohesive, minimal, readable, hackable, maximally forkable repo. nanochat will be the capstone project of LLM101n (which is still being developed). I think it also has potential to grow into a research harness, or a benchmark, similar to nanoGPT before it. It is by no means finished, tuned or optimized (actually I think there's likely quite a bit of low-hanging fruit), but I think it's at a place where the overall skeleton is ok enough that it can go up on GitHub where all the parts of it can be improved.

Link to repo and a detailed walkthrough of the nanochat speedrun is in the reply.

[Ken @_0xken]

Good morning, everyone! Wishing you a day full of success and positive vibes! ☀️🚀 #GoodMorning

[Ken @_0xken]

Exciting news from Google! Gemini 2.5 Computer Use is a new AI model that interacts with web interfaces like a human—filling forms, clicking, dragging. It bridges the gap between API-based and UI-simulated AI actions. A big step for AI usability! #AI #Google #Gemini2

[Ken @_0xken]

Turns out, 77% of sensitive data leaks happen because office folks love to Ctrl+C, Ctrl+V their way into generative-AI tools... often through their own secret accounts. Guess we’re all just secretly data magicians! 🧙‍♂️💻 #DataLeaks #AI

[Ken @_0xken]

Tried with Replit to generate a web app for removing audio/video background noise. It worked well to create the app within 20 mins and even suggested additional features after I tested the basic functions. Here is the link: https://t.co/Dq4LNawnuq

[Ken @_0xken]

When your breath syncs perfectly with your stride, it feels like listening to a beautiful symphony—you can't and don't want to interrupt it. If you're reading this, put your phone down and go feel it yourself for a wonderful Sunday!

[Ken @_0xken]

how to get into a new habit:
Add it to the end of your existing habit. **E.g, when I finished x, I will also do y**. Since x is your existing habit, you won't find it difficult to start y. After a few practices, y becomes your new habit!

[Ken @_0xken]

This Python code generates some "weird" results.
a = [0,0,0];
x = [1 for a[0] in range(3)]
print(x, a)
do you know what is the output?

[Ken @_0xken]

Python zip() – Beware of Missing Values! ⚠️ #python #pythonprogramming #... https://t.co/Hkx5HPcQJw via @YouTube

[Ken @_0xken]

People are tempted to make #pythoncode concise by using : import func1, func2 from module to directly call them in the code. but it may sometimes run into a collision later. e.g.

from datetime import time
from time import time
print(time())

#pythonlearning #pythonbeginnertips

[Ken @_0xken]

Many people don't know that Python has a nice feature to perform assignment and expression evaluation at once. e.g:
if(( data := get_data()) is not None:
process(data)

[Ken @_0xken]

Have been shifting my focus in the past few years for different explorations, but found that Python is the only thing that I did not give up. I like the beauty of it and want to start some short videos to give bite-sized tips for beginners in funny ways. Recording this moment!

[Ken @_0xken]

@gauthamzzz @DeFi_Dad @zachxbt NDRK likely holds the largest amount of ETH reserves now

[Ken @_0xken]

@marclou @DataFast_ amazing! really curious how this work behind the scene

[Ken @_0xken]

Good to see so many CNY traditions thriving in Singapore! �🏮 From vibrant decorations to family reunions, lion dance, Dragon dance etc. in the community, the festive spirit is alive and well.

#ChineseNewYear #Singapore #Traditions

[Ken @_0xken]

@tom_doerr

[Ken @_0xken]

After hitting a lot of server busy errors when using deepseek, I started to learn deploying it locally to my PC. But when deployed successfully, I realized it is too slow to run my PC due to limited CPU/memory resource. anyone experienced the same?

[Ken @_0xken]

Byte United HK & The University of Hong Kong introduce Goku, a next-gen AI video model! 🎥 From text prompts to stunning, lifelike videos—Goku brings virtual humans to life with expressive faces & fluid motion, generating clips over 20 seconds long! 🤯 #AIVideo #GokuAI

[Ken @_0xken]

This is the best time for product managers and developers alike—define the vision, set the requirements, and collaborate with AI to bring ideas to life faster than ever.

[Brett Adcock @adcock_brett]

Significant progress in AI and Robotics this week.

So, I summarized everything from Nvidia, OpenAI, ByteDance, Google, Figure, Mistral, Replit, Borg, Apple, Meta, and more.

Here's everything you need to know and how to make sense out of it: