Amit meel

Data Scientist at @IBM | Using AI to make a positive impact and solve challenging problems. Always learning and looking for new opportunities.

[Fernando @Franc0Fernand0]

https://t.co/BTdIzmWDvr

[Tech with Mak @techNmak]

This might be one of the best Deep Learning playlists for serious learners.

It’s MiniTorch by Sasha Rush.

You build a small Torch-like framework yourself and learn what sits underneath the APIs we normally call without thinking much about them.

The playlist covers:
➡️ Numerical derivatives
➡️ Autodifferentiation
➡️ Computation graphs
➡️ Chain rule
➡️ Backpropagation
➡️ Neural networks
➡️ Tensors
➡️ Tensor functions
➡️ Shapes
➡️ Gradients
➡️ Tensor puzzles
➡️ GPU puzzles

The full MiniTorch course goes further into broadcasting, parallel tensor operations, matrix multiplication, CUDA, convolutions, pooling, softmax, dropout, CNNs, MNIST and sentiment classification.
You’re also writing the implementation, running tests and debugging it as you go. That is what makes this useful.

MiniTorch was developed for the Machine Learning Engineering course at Cornell Tech.

If you already know some Python and want to build stronger Deep Learning fundamentals, save this one.

[ Playlist and GitHub in comments ]

[Umar Jamil @hkproj]

In this (extremely long) video, I'll build a distributed training framework from first principles using PyTorch.

We'll derive everything from first principles, explaining the mathematics behind distributed training as well as every auxiliary component we build along the way, including Multi-head Latent Attention (MLA), Rotary Position Embeddings (RoPE) and YaRN.

I'll explain process groups, ranks, collective communication operations, distributed autograd, device meshes and distributed tensors, and then use these foundations to implement the major forms of parallelism used to train modern language models.

We'll derive and code pipeline parallelism, data parallelism, FSDP, HSDP, tensor parallelism and context parallelism, including pipeline schedules, block matrix multiplication, model sharding and ring attention. We'll then combine these techniques and examine how tensor layouts and gradients must be tracked as data moves between devices.

Most importantly, we won't study each parallelism technique in isolation. We'll combine pipeline, data, tensor, context and expert parallelism into a single working framework and follow the tensors, communication operations and gradients as they move across GPUs. By the end, you'll understand not only how these techniques work individually, but how modern distributed training systems compose them to train large dense and mixture-of-experts models.

We'll also build the important components of a modern transformer, including rotary embeddings, long-context extensions, multi-head latent attention and a mixture-of-experts architecture. This leads into expert parallelism, expert tensor parallelism, token dispatching, all-to-all communication and combining different parallel dimensions through dense and sparse device meshes.

We'll also build the supporting training infrastructure, including the training loop, optimizer, learning-rate schedule, checkpointing and metrics.

The goal is to give you the foundations required to understand how distributed training frameworks work internally and how to design and implement new parallelization strategies yourself. No prior knowledge of distributed training is required.

Link to the video: https://t.co/ioSYostMYS

[Sebastian Raschka @rasbt]

How can an LLM switch between low-, medium-, and high-effort reasoning? And how does an LLM learn to reason more or less?

I put together a “little” article explaining how these effort levels are implemented at inference time and during training.

[Amit meel @immeelamit]

A beginner often says, "the agent decides what to do".

It's useful but it hides the most important truth in agentic systems:

"The model purposes. The runtime decides what happens next".

This distintion is everything.

#agent #ai #agenticai #codex #claude #datascience

[Silvia Sapora @silviasapora]

After interviewing for Research Scientist roles at DeepMind, Isomorphic, Meta, Cohere and more, I wrote up everything I learned. Technical prep, logistics, negotiation, and emotional breakdowns. Check out my guide: https://t.co/eLh20ggMHW

[Avi Chawla @_avichawla]

A harnessed LLM agent, clearly explained!

Most people picture this as a model with tools bolted on. The real architecture inverts that relationship.

The model itself is deliberately thin. Intelligence gets pushed outward, and the harness composes it at runtime.

Three dimensions orbit the harness core:

- 𝗠𝗲𝗺𝗼𝗿𝘆 holds the state a model shouldn't carry in weights or context. Working context, semantic knowledge, episodic experience, and personalized memory each have their own lifecycle.

- 𝗦𝗸𝗶𝗹𝗹𝘀 hold procedural knowledge. This can cover operational procedures, decision heuristics, and normative constraints that specialize the general model per task.

- 𝗣𝗿𝗼𝘁𝗼𝗰𝗼𝗹𝘀 hold the interaction contracts. Agent-to-user, agent-to-agent, and agent-to-tools are three distinct surfaces with their own failure modes.

Between the core and these modules sit the mediators, like sandboxing, observability, compression, evaluation, approval loops, and sub-agent orchestration.

They govern how the harness reaches out and how state flows back in.

The useful question this framing unlocks is: for any new capability, where should it live?

- Stable knowledge goes to memory
- Learned playbooks go to skills
- Communication contracts go to protocols
- Loop governance goes to the mediators

Harness design becomes a question of what to externalize, and how to mediate it.

I'm building a minimal agent harness from scratch and will open-source it soon.

In the meantime, my co-founder wrote an article about the anatomy of Agent Harness, covering the orchestration loop, tools, memory, context management, and everything else that transforms a stateless LLM into a capable agent.

Read it below.

[Alex Xu @alexxubyte]

How OpenAI Built Its Data Agent

Most teams building data agents stack routers, fine-tunes, and complex retrieval pipelines on top of multiple LLMs. OpenAI didn't.

Their data agent runs on a single model and only 13 tools, across 1.5 exabytes and 90,000 tables. It's "pretty vanilla" by design.

We spoke with Emma Tang, Head of Data Platform Engineering at OpenAI, to better understand the architecture and the engineering decisions behind it.

The article covers:
- The architecture behind the data agent
- The six layers of context that make a single LLM reliable across 90,000 tables
- How OpenAI Uses Codex Internally: 3 Use Cases
- Five practical lessons for any team building a domain agent
- Where OpenAI's data platform is headed next

[elvis @omarsar0]

New research from Microsoft Research

I see a lot of AI engineers handwriting agent skill docs and hope they generalize.

Probably not optimal. This works show why.

It treats the skill doc as a trainable external state of a frozen agent instead.

It introduces SkillOpt, where an optimizer model makes validation-gated edits to the skill file. It adds, deletes, or replaces instructions, with a textual learning rate that controls how aggressively each round rewrites the doc. The agent itself never changes.

SkillOpt is best or tied on all 52 (model, benchmark, harness) cells.

On GPT-5.5 it adds 23.5 points in direct chat, 24.8 with Codex, and 19.1 with Claude Code over no skill. It beats human-written skills, TextGrad, GEPA, and EvoSkill, carries zero extra inference-time cost, and the learned skills transfer across models and harnesses.

Paper: https://t.co/mNgTmmT32U

Learn to build effective AI agents in our academy: https://t.co/1e8RZKs4uX

[Sebastian Raschka @rasbt]

It's been a while! 4 nice additions to the open-weight local-LLM-on-consumer-hardware ecosystem:

[Ahmad @TheAhmadOsman]

Step-By-Step LLM Engineering Projects Roadmap

- Build a tokenizer
- Learn embeddings
- Implement RoPE / ALiBi
- Hand-wire attention
- Build MHA
- Build a Transformer block
- Train a mini-former
- Compare objectives
- Build sampling
- Speculative decoding
- KV cache
- MQA / GQA / MLA
- Long context
- FlashAttention
- Hardware budgets
- Toy MoE
- Sparse model trade-offs
- State-space / linear attention
- Diffusion language models
- Data pipelines
- Synthetic data
- Scaling laws
- SFT / DPO / RLHF / GRPO
- Quantization
- Serving stacks
- Eval harnesses
- RAG
- Tool use / agents
- Vision-language adapters
- Interpretability
- Red-team suite
- Full capstone model system

One request:

Choose an Opensource AI lab when you make it

Opensource is where humanity gets to keep the tools

DM me when you've made it ;)

[Amit meel @immeelamit]

@theo Really True😂😂

[Amit meel @immeelamit]

@alvinsng Just because it can we shouldn't be using it everywhere. Its just plain hype😬.For complex problems and at scale it will definitely not be a solution. Latency, overall cost that comes with tokens etc, so many other things to consider 🤗

[Jia-Bin Huang @jbhuang0604]

Modern Transformer - Complete Guide

Interested in learning the recent advances in transformers?

After 13 videos, I've finally completed this series!
🥳🥳🥳

Check out the course here:
https://t.co/CsujxlWigC

[Amit meel @immeelamit]

@alexxubyte Correct retrieval and context management 😅

[Andrej Karpathy @karpathy]

Fireside chat at Sequoia Ascent 2026 from a ~week ago. Some highlights:

The first theme I tried to push on is that LLMs are about a lot more than just speeding up what existed before (e.g. coding). Three examples of new horizons:

1. menugen: an app that can be fully engulfed by LLMs, with no classical code needed: input an image, output an image and an LLM can natively do the thing.
2. install .md skills instead of install .sh scripts. Why create a complex Software 1.0 bash script for e.g. installing a piece of software if you can write the installation out in words and say "just show this to your LLM". The LLM is an advanced interpreter of English and can intelligently target installation to your setup, debug everything inline, etc.
3. LLM knowledge bases as an example of something that was *impossible* with classical code because it's computation over unstructured data (knowledge) from arbitrary sources and in arbitrary formats, including simply text articles etc.

I pushed on these because in every new paradigm change, the obvious things are always in the realm of speeding up or somehow improving what existed, but here we have examples of functionality that either suddenly perhaps shouldn't even exist (1,2), or was fundamentally not possible before (3).

The second (ongoing) theme is trying to explain the pattern of jaggedness in LLMs. How it can be true that a single artifact will simultaneously 1) coherently refactor a 100,000-line code base *and* 2) tell you to walk to the car wash to wash your car. I previously wrote about the source of this as having to do with verifiability of a domain, here I expand on this as having to also do with economics because revenue/TAM dictates what the frontier labs choose to package into training data distributions during RL. You're either in the data distribution (on the rails of the RL circuits) and flying or you're off-roading in the jungle with a machete, in relative terms. Still not 100% satisfied with this, but it's an ongoing struggle to build an accurate model of LLM capabilities if you wish to practically take advantage of their power while avoiding their pitfalls, which brings me to...

Last theme is the agent-native economy. The decomposition of products and services into sensors, actuators and logic (split up across all of 1.0/2.0/3.0 computing paradigms), how we can make information maximally legible to LLMs, some words on the quickly emerging agentic engineering and its skill set, related hiring practices, etc., possibly even hints/dreams of fully neural computing handling the vast majority of computation with some help from (classical) CPU coprocessors.

[Andrej Karpathy @karpathy]

Wow, this tweet went very viral!

I wanted share a possibly slightly improved version of the tweet in an "idea file". The idea of the idea file is that in this era of LLM agents, there is less of a point/need of sharing the specific code/app, you just share the idea, then the other person's agent customizes & builds it for your specific needs.

So here's the idea in a gist format: https://t.co/NlAfEJjtJV

You can give this to your agent and it can build you your own LLM wiki and guide you on how to use it etc. It's intentionally kept a little bit abstract/vague because there are so many directions to take this in. And ofc, people can adjust the idea or contribute their own in the Discussion which is cool.

[Sebastian Raschka @rasbt]

Components of a coding agent: a little write-up on the building blocks behind coding agents, from repo context and tool use to memory and delegation.

Link: https://t.co/iF4DsMcnhj

[Andrej Karpathy @karpathy]

Three days ago I left autoresearch tuning nanochat for ~2 days on depth=12 model. It found ~20 changes that improved the validation loss. I tested these changes yesterday and all of them were additive and transferred to larger (depth=24) models. Stacking up all of these changes, today I measured that the leaderboard's "Time to GPT-2" drops from 2.02 hours to 1.80 hours (~11% improvement), this will be the new leaderboard entry. So yes, these are real improvements and they make an actual difference. I am mildly surprised that my very first naive attempt already worked this well on top of what I thought was already a fairly manually well-tuned project.

This is a first for me because I am very used to doing the iterative optimization of neural network training manually. You come up with ideas, you implement them, you check if they work (better validation loss), you come up with new ideas based on that, you read some papers for inspiration, etc etc. This is the bread and butter of what I do daily for 2 decades. Seeing the agent do this entire workflow end-to-end and all by itself as it worked through approx. 700 changes autonomously is wild. It really looked at the sequence of results of experiments and used that to plan the next ones. It's not novel, ground-breaking "research" (yet), but all the adjustments are "real", I didn't find them manually previously, and they stack up and actually improved nanochat. Among the bigger things e.g.:

- It noticed an oversight that my parameterless QKnorm didn't have a scaler multiplier attached, so my attention was too diffuse. The agent found multipliers to sharpen it, pointing to future work.
- It found that the Value Embeddings really like regularization and I wasn't applying any (oops).
- It found that my banded attention was too conservative (i forgot to tune it).
- It found that AdamW betas were all messed up.
- It tuned the weight decay schedule.
- It tuned the network initialization.

This is on top of all the tuning I've already done over a good amount of time. The exact commit is here, from this "round 1" of autoresearch. I am going to kick off "round 2", and in parallel I am looking at how multiple agents can collaborate to unlock parallelism.
https://t.co/WAz8aIztKT

All LLM frontier labs will do this. It's the final boss battle. It's a lot more complex at scale of course - you don't just have a single train. py file to tune. But doing it is "just engineering" and it's going to work. You spin up a swarm of agents, you have them collaborate to tune smaller models, you promote the most promising ideas to increasingly larger scales, and humans (optionally) contribute on the edges.

And more generally, *any* metric you care about that is reasonably efficient to evaluate (or that has more efficient proxy metrics such as training a smaller network) can be autoresearched by an agent swarm. It's worth thinking about whether your problem falls into this bucket too.

[Andrej Karpathy @karpathy]

I packaged up the "autoresearch" project into a new self-contained minimal repo if people would like to play over the weekend. It's basically nanochat LLM training core stripped down to a single-GPU, one file version of ~630 lines of code, then:

- the human iterates on the prompt (.md)
- the AI agent iterates on the training code (.py)

The goal is to engineer your agents to make the fastest research progress indefinitely and without any of your own involvement. In the image, every dot is a complete LLM training run that lasts exactly 5 minutes. The agent works in an autonomous loop on a git feature branch and accumulates git commits to the training script as it finds better settings (of lower validation loss by the end) of the neural network architecture, the optimizer, all the hyperparameters, etc. You can imagine comparing the research progress of different prompts, different agents, etc.

https://t.co/YCvOwwjOzF
Part code, part sci-fi, and a pinch of psychosis :)