Olivia
Online
Riddick
@10Riddick10
Joined January 2014
3.3K Following
52 Followers
3.1K Posts
Bonito día 🥰
Upcoming parts for Everything About Memory Allocators are these:
Part 3: calloc and realloc
Part 4: splitting and coalescing
Part 5: first fit against best fit, measured
Part 6: big requests through mmap
Part 7: size class bins
Part 8: LD_PRELOAD, guard rails and a heap checker
C Pointers And Memory
Pointers are not just variables that hold addresses.
LowLevelCraft’s C track makes you work with sizeof, malloc, pointer arithmetic, undefined behavior, memory layout, and small allocators.
The aim is to reason about bytes and lifetime, not just make the compiler happy.
https://t.co/F9rdQFoAf4
I still don't understand why everyone is still running agents in a line. I switched to graphs three weeks ago and my fleet finished in the time my single agent used to spend on step two.
what slows every agent system I have seen is not intelligence. it is geometry. and almost nobody is talking about it.
one engineer used this to rewrite 535,000 lines of code in 11 days. a manual rewrite of that scale could take close to a year. it cost $165,000 in tokens. the graph was not cheap. it was just faster than a human year.
a node is one agent with one job. research one competitor. review one file. check one claim. the moment a node owns two independent jobs you lose the ability to parallelize them cleanly, verify them independently, and debug them in isolation. an edge is a dependency. it only exists when data actually moves across it. everything else is a fake edge. a wait you invented that costs time and produces nothing.
find the fake edges and the line collapses into something wider. jobs that can run at the same time run at the same time. what used to take the sum of forty steps now finishes in the time of the slowest layer.
the pattern behind every serious agent system looks like a diamond. fan out to gather breadth, one agent per angle, all at once. reduce with plain code, no model tokens spent. verify with a fresh skeptic on every finding. synthesize once from what survived. Claude's own research feature uses a very similar pattern in production.
the part nobody warns you about: the verifier needs clean context. give it the same conversation the worker had and it is not checking anything. it is nodding along to itself in a different window. a graph of agents sharing one context is a single loop in a costume. it breaks the same way, just later and more expensively.
one rule that holds at every scale. a worker and its verifier must never share a context.
your agents are not too slow. they are waiting in a line that did not need to exist.
full guide in the article. save it before you build your next agent from scratch.
00:15 - The right way to prompt Claude
33:21—What makes Claude act dumber in your code
01:33:39 - How Anthropic uses Claude every day
02:50:56 - A fix that makes Claude way smarter - This
The 4-hour Anthropic free course replaces about 10 paid engineering courses.
Watch it today, then bookmark it for later.
Andrew Ng just dropped the best 2-hour course on Graph Engineering: from a single agent to full automation
9:14 - your first agent
33:11 - loop engineering
1:02:46 - graph engineering
1:24:10 - agents that rewrite themselves
1:38:20 - the full graph system
free, and the best thing on graph engineering I have come across
Prompts → Agents → Loops → Graphs
most people will stop after the first two timestamps and call it a system
he spends the last forty minutes on the part that is still true next year
same model, same tokens, completely different week
watch it today
the full guide is below, save it while it is still early ↓
Google just released a free 2-hour course on Agent Harness Engineering.
How to go from one prompt to a self-improving agent harness:
0% → 10:16 — build your first AI agent
25% → 41:05 — master prompt engineering
50% → 54:45 — turn agents into graphs
75% → 1:20:10 — run loops inside an agent harness
100% → 1:43:33 — build a harness that improves itself
Most people build one agent and stop there.
Google is teaching everything that comes next:
Prompt → Agents → Graphs → Loops → Harness
Single agents are the old workflow.
Self-improving harnesses are the next one.
This free 2-hour course is worth more than most paid agent engineering programs.
Bookmark it and watch today.
Then read the full guide below to build your first self-improving agent harness ↓
this is pure f*cking treasure
7 GitHub frameworks with a combined 261,300 stars that can build a real graph-based agent stack
nodes. edges. state. checkpointing. role-based crews. durable execution. production guardrails
01 AutoGen
▸ https://t.co/9aNc7ZwqmG
60.6k stars · pioneered multi-agent orchestration, now in maintenance mode, still runs everywhere
02 CrewAI
▸ https://t.co/VTnBbTRYNB
57.4k stars · 5.2M monthly downloads, Novo Nordisk runs data science workflows on it
03 LangGraph
▸ https://t.co/vvxKtfBxcx
40.2k stars · Klarna, Uber, LinkedIn, GitLab run production agents on it
04 Agno
▸ https://t.co/tMR1ulnh1x
41.8k stars · lightweight framework built for multi-agent systems from scratch
05 Microsoft Agent Framework
▸ https://t.co/z8XWifxnEq
13k stars · unified successor to AutoGen + Semantic Kernel, GA April 2026
06 Google ADK
▸ https://t.co/ncOb2TRcxO
21.2k stars · GCP-native, opinionated runtime with built-in debugging UIs
07 Composio
▸ https://t.co/Bz9krOt70D
27.1k stars · 1000+ toolkits, sandboxed workbench, turns intent into action
the loop:
define the graph → assign the nodes → route the edges → checkpoint the state → resume where it broke → guard the dangerous calls → ship the output
save this before you build your next agent from scratch
Hands on C-Programming
Master C at the level systems programmers actually use it. Understand memory management, pointers, undefined behavior, and the C compilation model.
Link: https://t.co/F9rdQFoAf4
where does all the VRAM go during LLM inference?
(4 ways GPU memory is used)
loading the model is only the first part of the memory story.
once inference starts, GPU memory gets divided across multiple components, and some of them keep growing as context length, batch size, and concurrency increase.
the graphic breaks it into four useful buckets:
→ model weights are the mostly fixed part. once the model is loaded, their memory footprint stays roughly constant. the biggest lever here is precision. moving from FP16/BF16 to INT8 or INT4 reduces the number of bytes needed to store each parameter.
→ KV cache grows as generation continues. for every previous token, the model stores key and value tensors so attention can reuse them instead of recomputing the entire sequence. longer contexts mean a larger KV cache, and more concurrent requests mean more active caches sitting in memory.
→ activations and workspace hold temporary intermediate values needed while running attention, MLP layers, kernels, and other computations. this memory is reused across inference steps, but its size can still change with sequence length, batch size, and the kernels being executed.
→ runtime overhead comes from everything around the model itself. CUDA kernels, memory allocators, metadata, serving-engine buffers, and other runtime structures all consume some VRAM. it is usually smaller than the other buckets, but it is never zero.
this is why “the model fits on the GPU” and “the workload fits on the GPU” are two different statements.
a model may load comfortably, then run out of memory when you increase the context window, serve more users simultaneously, or increase the batch size.
it also explains why quantization can help beyond simply fitting a larger model. shrinking the weight footprint creates room that can instead be used for larger KV caches, more concurrent requests, or bigger batches.
that is the broader GPU lesson too.
performance is not just about how much arithmetic a GPU can do. it is also about what data occupies memory, how much of it moves during inference, and how often that data can be reused.
i wrote the full breakdown of how GPUs actually work and why memory movement sits at the center of LLM inference performance.
the article is quoted below.
Anyone can build an AI agent in 60 minutes.
Stanford just released a free course on building AI agents from scratch.
0% → 00:00 - build your first AI agent
75% → 48:17 - create AI agents without coding
100% → 54:39 - build $100K+/month businesses with agents
While you're scrolling, someone else is learning Anthropic's $750,000 skill.
Watch it, then use the guide below to build your first agent that prompts itself.
Transformer and Mixture of Experts, explained visually!
Mixture of Experts (MoE) is a popular architecture that uses different experts to improve Transformer models.
Transformer and MoE differ in the decoder block:
- Transformer uses a feed-forward network.
- MoE uses experts, which are feed-forward networks but smaller compared to those Transformer.
During inference, a subset of experts are selected. This makes inference faster in MoE.
Also, since the network has multiple decoder layers:
- The text passes through different experts across layers.
- The chosen experts also differ between tokens.
But how does the model decide which experts should be ideal?
The router does that.
It is a multi-class classifier that produces softmax scores over experts to select the top K experts.
The router is trained with the network, and it learns to select the best experts.
But it isn't straightforward.
There are challenges!
Challenge 1) Notice this pattern at the start of training:
- Say, the model selects "Expert 2"
- This expert gets a bit better
- It may get selected again since it's the "best"
- It learns more
- It gets selected again in the next iteration
- It learns more, and so on!
This means many experts can go under-trained due to the overselection of a few experts!
We solve this in two steps:
- Add noise to the feed-forward output of the router so that other experts can get higher logits.
- Set all but the top K logits to -infinity. After softmax, these scores become zero.
This way, other experts also get the opportunity to train.
Challenge 2) Some experts may get exposed to more tokens than others, leading to under-trained experts.
We prevent this by limiting the number of tokens an expert can process.
If an expert reaches the limit, the token is passed to the next best expert.
Overall, MoEs have more parameters to load. But a fraction of them are activated during inference. This leads to faster inference.
Mixtral 8x7B and Llama 4 are two popular MoE-based LLMs.
Have you used MoEs in production yet?
To dive deeper into how MoE inference works in production, we wrote a full article covering token dispatch, grouped expert computation, model-weight memory, multi-GPU communication, expert placement, load imbalance, and performance diagnosis.
Read it below.
A lesser-known production reality of MoE inference:
(must-know for technical LLM interviews)
MoE models reduce expert computation by routing each token to a small subset of experts.
But routing also divides a batch of token rows into several expert-specific batches.
Their sizes depend on how many tokens enter the layer and which experts those tokens select.
This creates a systems problem that is not immediately evident from architecture diagrams.
Let's walk through it.
1) Dense transformer execution
Assume a decode batch contains 32 token rows.
A dense Transformer sends all 32 rows through the same feed-forward network. For each feed-forward projection:
> 32 token rows x the same weight matrix
The accelerator performs one 32-row matrix multiplication and reuses that projection's weights across all 32 rows.
2) MoE execution
A simplified top-1 MoE router may divide those rows like this:
- Expert 1 receives 8 rows
- Expert 2 receives 3 rows
- Expert 3 receives 0 rows
- Expert 4 receives 11 rows
- All other experts collectively receive 10 rows
Each 32-row feed-forward projection has become several smaller expert projections with different row counts.
The engine must group rows by expert, run each expert, apply routing weights, and restore the original token order.
With top-k routing, each token appears in multiple expert batches.
Sparse routing reduces expert computation, but it also fragments the original batch.
3) How grouped GEMM helps
GEMM is General Matrix Multiplication.
Each active expert has its own weight matrices. Grouped GEMM schedules several expert matrix multiplications together:
- Expert 1 rows x Expert 1 weights
- Expert 2 rows x Expert 2 weights
- Expert 4 rows x Expert 4 weights
This reduces launch overhead and helps the accelerator process differently sized expert batches together.
It works best when each active expert receives several rows.
4) During low concurrency
During ordinary autoregressive decoding, each active request contributes one current token to a model step.
With 32 active requests, some experts may receive several token rows.
With only one to three requests, the selected experts may barely overlap. Most active experts can receive a single row.
Each operation now resembles GEMV, or General Matrix-Vector Multiplication.
The engine loads an expert's weights to process one token vector. That leaves little opportunity to reuse those weights across rows.
Grouped GEMM can combine the launches, but it cannot turn one-row expert batches into large matrix multiplications.
5) An example
Researchers encountered this while optimizing Qwen-3.5-397B-A17B-FP8 on Ironwood TPUs.
Each token selected ten routed experts from a pool of 512. With three concurrent requests, they found that only 5% of the selected experts were duplicated across requests.
Most routed expert batches therefore contained one row.
So they built a custom kernel that prefetched selected expert weights into the TPU's fast VMEM and processed these GEMV-like operations directly.
This resulted in a 3.6x faster MoE block at concurrency one on a TPU configuration.
This does not mean grouped GEMM is inefficient. Instead, it implies that the best execution strategy depends on the expert-batch shape:
- Several rows per expert → grouped GEMM can work well
- One row per expert → hardware-specific weight streaming may work better
The architecture stays the same, and concurrency and routing overlap change how the work is assigned to each expert.
To dive deeper, I wrote a full article covering this and the other engineering problems behind MoE inference, including memory, cross-GPU dispatch, expert placement, load imbalance, quantization, and offloading.
Read it below.
Train your own LLM from scratch!
A step-by-step repo that walks you through building and training a transformer model from scratch using PyTorch. From downloading training data all the way to generating text.
The architecture is built from the ground up following the original "Attention is All You Need" paper. MLP, single head attention, multi-head attention, transformer blocks, and the full transformer model - all coded and explained with detailed diagrams at each step.
Training data comes from The Pile - a diverse 825GB open-source dataset covering books, articles, code, websites, and more. The repo includes scripts to download it, preprocess and tokenize it using tiktoken, store it in HDF5 format, and feed it into training batches.
You can train a 13M parameter model on a single Colab T4 GPU. At 13M parameters the model starts generating proper grammar and coherent short sentences. For billion-parameter training you need at least an A100 or RTX 4090. The repo includes a full GPU compatibility table so you know exactly what's possible on your hardware.
Includes a complete SFT and RLHF guide as a separate notebook for taking your trained model further.
Key capabilities:
• End-to-end pipeline: data download → preprocessing → training → text generation
• Full transformer implementation from scratch with PyTorch
• Trains models from 13M to 2B+ parameters on a single GPU
• Training data from The Pile (825GB, 22 diverse datasets)
• Tokenization via tiktoken (r50k_base)
• SFT and RLHF guide included
100% open source.
I've shared the link in the replies!
Feliz Viernes 🇵🇪🇲🇽🔗🫶🏻😙
Google just released a free 2-hour course on Graph & Loop engineering: 1 prompt → 100 agents → loops → graphs, from 0% to 100%:
0% → 0:35 - graph engineering from scratch
30% → 31:17 - build your first agent graph
45% → 43:40 - run hundreds of agents in parallel
75% → 1:04:58 - loop engineering: route, check, repeat
100% → 1:30:09 - self-improving graphs that work while you sleep
taught by the people who run this in production
Prompts → Agents → Loops → Graphs
most people are still stuffing instructions into one agent and calling it a system
they built the thing one layer up and let it decide what runs at all
same model, same tokens, completely different week
watch it today
the full graph engineering guide is below, save it while it is still early ↓
One of the best in the business!!
Stanford: CME 295 - Transformers & Large Language Models
https://t.co/v7XOyDj7D3
Agent Evals is one of the most important topics for AI engineers and anyone who is building a real agentic system.
This course is one of the best resoruces I recently came across, and it covers evaluating agents and training them inside real environments. It consists of six workshops given by engineers and researchers from top companies and universities.
Here's what it covers:
1. Agentic Evaluations Workshop: Where agent evals actually stand, and why benchmark scores don't match what people see in use.
2. RL for Agents Workshop: Environments, rollouts, reward design, and the inference bottlenecks that appear when you move from RL for LLMs to RL for agents.
3. Training Agents 1: SFT on agent traces. Public coding-agent traces turned into prompt/completion data, a TRL + LoRA fine-tune on Hugging Face Jobs, metrics in Trackio, and an honest look at what the first eval numbers can and cannot tell you.
4. Training Agents 2: Distillation: off-policy, on-policy, and self-distillation for moving capability from a teacher into a smaller coding agent.
5. Training Agents 3: Reinforcement learning GRPO after SFT: group sampling, verifiable reward functions, reading the reward/KL/length curves, and three experiments, one of them with a deliberately gameable reward so we could watch the hacking happen.
6. Training Agents 4: From reward functions to environments. The reward stops being a function and becomes a place the agent acts in.
Course Link: https://t.co/ivTdRjIqIc
I am hosting a live workshop on building a real deep search agent from scratch. Use code deepsearch20 to get a 20% early-bird discount:
https://t.co/W6nZPXcisL
Andrew Ng just released a 1-hour course on full AI engineering: LLM → prompts ��� agent teams → graphs, from 0% to 100%:
0% → 0:17 - LLM understanding from scratch
30% → 20:02 - how to actually write prompts
55% → 29:03 - build your first team of agents
100% → 47:19 - the final stage of agents: graphs
thirty years of AI engineering compressed into one hour, by the person who lived it
Prompts → Agents → Loops → Graphs
most people will stop at the second timestamp and call it learning
he spends the last eighteen minutes on the only part that is still true next year
same model, same tokens, completely different week
watch it today
the full guide on building graphs is below, save it while it is still early ↓
Don't waste 2 years learning to become an AI agentic engineer in 2026.
Andrew Ng, the godfather of AI, gave the complete playbook to become one from scratch.
1 hour course. Free:
• 00:00 - AI agent basics
• 12:12 - AI Agentic workflows & design patterns
• 53:27 - Practical tips for building AI agents
• 1:20:30 - self-improving AI agent loops
• 1:30:19 - multi-agent AI systems
I watched it last night.
Halfway through, I realized I could get into Anthropic in weeks, not years.
Bookmark now. Watch it. Then build your own AI agent
Trends for you
Most Popular Users
Elon Musk 
@elonmusk
241.7M followers
Barack Obama
@barackobama
119M followers
Cristiano Ronaldo
@cristiano
114.3M followers
Donald J. Trump
@realdonaldtrump
111.9M followers
Narendra Modi
@narendramodi
107.2M followers
Rihanna
@rihanna
98.7M followers
NASA
@nasa
92.4M followers
Justin Bieber
@justinbieber
91.8M followers
KATY PERRY
@katyperry
90M followers
Taylor Swift
@taylorswift13
83.9M followers
Lady Gaga
@ladygaga
75.4M followers
Virat Kohli
@imvkohli
73.3M followers
Kim Kardashian
@kimkardashian
70.9M followers
YouTube
@youtube
68.8M followers
Neymar Jr
@neymarjr
66.3M followers
Bill Gates
@billgates
65.1M followers
Selena Gomez
@selenagomez
63M followers
The Ellen Show
@theellenshow
62.3M followers
CNN
@cnn
61.8M followers
X
@x
60.7M followers
✨
⭐
💫