Sudheer

Aspiring GPU/Inference engineer, love making GPUs go brr

[Sudheer @Sud0_12]

Also studied benchmarking from the same series.

Eagerly waiting for future chapters 🤗

@datawhale2018 & @radixark please drop them soon

[Sudheer @Sud0_12]

Day 7 of inference engineering:

Studied the why and how of KV cache from this GOATed resource:

https://t.co/yNnWX78Os5

[Sudheer @Sud0_12]

@eliebakouch What extension are you using in classifying the tweet as human/ai?

[Sudheer @Sud0_12]

Day 6 of Inference engineering:
Ran the same aiperf benchmarks(from Day 5) on SGLang server and compared results with vLLM -
- SGLang had 2.4x higher total throughput
- SGLang had 65% lower avg request latency
- SGLang had 73% lower ITL

Started studying https://t.co/0kPNyFFENu

[🪴areen @rivereen]

can't believe tomorrow is office again

[Yichi Zhang @YichiZ03]

zero-to-sglang just hit 1,000 GitHub stars! Thanks for all your support!

Our new chapters:
📖 Chapter 2.1: mini-sglang — What Does an Inference Engine Look Like?
📖 Chapter 2.2: The Journey of a Request + Code Walkthrough

Find the course following link in comment section👇

[Sudheer @Sud0_12]

In the recent past, loop engineering and Openclaw were the buzz words everyone was after, which I didn't get on and seems like I didn't miss much.

Is Jev the next such buzzword or is it actually something you seem to be using 3-4 months down the line?

[Sudheer @Sud0_12]

@kodejeet Continuing on prev response:
There might be remote roles available but very hard to get into for Indians.

Relevant post for context: https://t.co/VfrnedI9fg

@jaga_prasanna care to provide more insights?

[prasanna @jaga_prasanna]

Bro I wish some of these guys were willing to give remote outside US

since we have US visa problems thats side also blocked

from what Ive seen india doesnt have frontier inference company yet and outside india they are not willing to provide remote and Ive been rejected for this reason alone! Mostly

If you looking for AI infra role the market is really bad for us! need to find a way soon!

[Sudheer @Sud0_12]

Day 5 of inference engineering:
Came across this super cool benchamarking repo: https://t.co/HXDApOi8gz

Ran it on T4 machine against Qwen3-0.6B. Would be diving ddep with different models and configs.

PS: Got busy with career stuff and had a break :p

[Sudheer @Sud0_12]

@kodejeet Not much from what I know. AFAIK there's only 2 ways to get into this field (in India):
1) Startups where you have to do lot of other ML work along with inference
2) Top companies like Nvidia, Sarvam etc. which require significant YOE and big names in resume (IIT, FAANG etc).

[Sudheer @Sud0_12]

@kodejeet Running it on T4 GPU, which is pretty old. You can access these GPUs for free on colab or kaggle :)

[avrl ☘ @avrldotdev]

you'll never be unemployed if you master inference engineering, but you need to be unemployed to master inference engineering.

[Sudheer @Sud0_12]

@DustinW83290153 @burkov Wasn’t AI supposed to accelerate things and produce output at fast pace?

[Sudheer @Sud0_12]

@kmeanskaran Everyone shares resources on how to learn inference engineering, but none share ways to get hired. Esp in India.

[Sudheer @Sud0_12]

@jaga_prasanna If people you like you with hands on experience are finding it difficult to land an opportunity in this field. Job market’s cooked for newbies

[Sudheer @Sud0_12]

Day 4 of inference engineering:
Sidetracked a bit and focusing on learning AMD gpu model and FLEET abstractions

[casey 🗡️ @caseysonline]

being jobless is so funn👍🏻 i sit and apply for jobs way out of my experience and hope one lands.

[Sudheer @Sud0_12]

@GonnabeNikhil Waiting for their inference engineering course

[Sudheer @Sud0_12]

Day 3 of Inference engineering:
- Deployed Llama-3.2-1B on 2xT4 using vLLM
- Wrote benchmark scripts for measuring latency(non streaming) of varied prompts
- Measured e2e p50, p90, p99 latency times
- e2e_time summary: n=50, mean ~2.92s, p50 ~1.89s, p90 ~7.64s, p99 ~8.48s

[Sudheer @Sud0_12]

@datawhale2018 For those wondering about the link. Here you go: https://t.co/VcQv6mFmh4

https://t.co/2a7MOvLuGI