Mayank

Data Scientist @ Perfios || ug @iiit_hyderabad || Specialist @Codeforces

[Nikhil Mourya @GonnabeNikhil]

If you're trying to get genuinely good at LLM inference / AI systems, I think one of the best projects you can do is take a small open-weight model and build the entire inference system around it.

1. Take a pretrained model and understand its inference path. Know what actually happens on the GPU for every token.

2. Build a naive inference server yourself. Don't touch vLLM yet. Make it work, then watch it fall apart.

3. Add batching. Understand why continuous batching matters and measure TTFT, TPOT and throughput.

4. Implement KV caching. Then actually understand why decode becomes memory-bandwidth bound.

5. Build a scheduler. Queues, priorities, backpressure, cancellation, timeouts.

6. Fix your KV memory management. Paged KV cache, block allocation, fragmentation, prefix caching.

7. Start optimizing the GPU path. CUDA graphs, kernel fusion, quantization, attention kernels, CPU/GPU synchronization.

8. Add speculative decoding and figure out when it actually makes inference faster.

9. Add observability. TTFT, ITL, throughput, GPU utilization, KV usage, queue time.

10. Throw real concurrent traffic at it. Find the point where throughput stops scaling and figure out why.

11. Only then go read vLLM, SGLang, TensorRT-LLM and other serving engines. Compare their design decisions with yours.

12. Keep going deeper. Multi-GPU, distributed inference, prefill/decode disaggregation, KV offloading, routing...

You don't need to build a production-ready serving engine.

The point is to get to the point where you can look at an inference system and understand why every ugly little optimization exists.

That's basically what I've been doing with my vLLM Inference Engine project, and ngl, every time I thought I understood inference, the next bottleneck humbled me lol.

Build the dumb version first. Let the bottlenecks teach you the architecture.

https://t.co/bFY39lWJz4

[Yaowei Zheng | LlamaFactory @code_hiyouga]

If you already know how to call models, connect tools, and build agents, Stanford’s new CS329Z: Engineering AI Agents is a useful place to check what you should learn next.

Especially if your system runs and keeps gaining features, but you struggle to tell whether each change actually makes it better. The course starts with RAG, tool use, MCP, and agent frameworks, then covers memory, multi-agent systems, optimization, agent data, and evaluation. It also addresses safety and the reliability of agents running over extended periods.

The assignments make this concrete: first, build a research-paper QA agent from scratch. Then design an evaluation suite for an existing agent, including benchmark tasks, code-based graders, an LLM-as-judge, and error analysis.

I think this structure is particularly useful for people learning to build agents on their own. It’s easy to work through a feature checklist: connect more tools, add memory, try multiple agents. But every new component also introduces more ways the system can fail.

Take a research assistant that gives the wrong answer. It might have failed to retrieve a key paper, misunderstood a paper it found, or dropped important evidence while composing its response. Each failure calls for a different fix. Looking only at the final answer makes it hard to know where to intervene.

That’s why I’m interested in the connection between execution traces, error analysis, and evaluation. Recording what the agent did, identifying where it failed, and testing changes against a consistent set of tasks gives you a basis for judging whether your fixes work. This also shapes how I think about the order in which to learn agent engineering. Before trying another framework, it may be worth building an evaluation suite for the project you already have. Turn the failures you’ve encountered into test cases. Track what each change fixes and what it breaks.

Read the CS329Z syllabus with those questions in mind, and its topics become concrete work you can apply to your own project.
https://t.co/ZQm0rQLxPf

[Rohit @rohit4verse]

The repo you need to master ML
https://t.co/jIeLlo70sZ

[Avinash Singh @AvinashSingh_20]

India Remote Startups Database:
https://t.co/YzehNLaeJp

Bookmark it!

[Mayank @kdb4517]

@Polymarket Are we assuming both are absolutely exclusive?

[AI @nonewthing]

Your best years should be spent doing the hardest things you can.

Youthfulness is a resource you will never have again.

[Ahmad @TheAhmadOsman]

https://t.co/uRDNr16FHA

[Andrej Karpathy @karpathy]

Personal update: I've joined Anthropic. I think the next few years at the frontier of LLMs will be especially formative. I am very excited to join the team here and get back to R&D. I remain deeply passionate about education and plan to resume my work on it in time.

[ِ @JXComps]

Kevin De Bruyne | Every knockout stage Goal/Assist in the Champions League

[Ruchi Pakhle @Ruchicodess]

AI Engineering is exploding. 🤯

But here’s what most people don’t realize:

The real bottleneck in AI today isn’t training models.
It’s running them at scale.

That field is called Inference Engineering.

And it’s becoming one of the highest-demand roles in AI.

Here’s how to break into it 🧵👇

[Siddharth @Pseudo_Sid26]

https://t.co/yVj1Q5UFkT

[Eyad @eyad_khrais]

https://t.co/i5L6UIPgH8

[g. @gxrwayne]

The SilvaSurfer

[blue @bluewmist]

the longer you grind, the more you realize people look exactly like the choices they make everyday. you can’t fake effort.

[Mayank @kdb4517]

@PriyanshuP1405 Do drop the repo link for this pls

[Mayank @kdb4517]

@bayesian_walker Looking forward to the detailed review!

[Alexandr Wang @alexandr_wang]

I’m excited to be the Chief AI Officer of @Meta, working alongside @natfriedman, and thrilled to be accompanied by an incredible group of people joining on the same day.

Towards superintelligence 🚀

[Mayank @kdb4517]

@akshad_999 Hi, I am going to buy m4 air 15' 16gb/512gb any reviews so far of how smooth it is?

[Shubhangi Parate @ShubhangiP24]

Mentally weak people are just people who made it a habit to act like a victim every time they suffered

Mentally strong people are just people who made it a habit to upgrade themselves every time they suffered