Alex

@fluidinference , AI Researcher

[Alex @Alex_tra_memory]

a 1,000 followers , time to take a break

[Alex @Alex_tra_memory]

i finally got 10 PRs merged into @vllm_project 🎉

by commit count on vLLM's main, i am ranked #307–#335 of 3,756 contributors, top 8.9% (306 people have more commits, 29 of us are tied at 10)

what shipped:

🎙️ NVIDIA Canary: speech recognition + translation to and from English in 25 European languages, token-identical to the reference implementation
https://t.co/GZdFcMAxgq
https://t.co/v59kfGfPu2

🔌 Parakeet TDT + Nemotron-3-Diarization: both run in vLLM through plugins i built, now listed in its supported models
https://t.co/UPnSa5lYme
https://t.co/OCu9P4AEZd

🖼️ Gemma 4: image normalization moved from the CPU to the GPU, including the Gemma 4 Unified 12B models
https://t.co/faOzNkewyQ
https://t.co/BO94q1CHjw

🧭 CLIP, SigLIP & SigLIP2: chat-style requests now return the same embeddings as plain input. also opened the matching fix on the @huggingface Hub
https://t.co/RpyCg5uY9F
https://t.co/3UJp7c0yAO
https://t.co/ipRM2qsHm3

🛠️ + fixes: answers no longer get mislabeled as reasoning, and main's pre-commit check is green again
https://t.co/3saPRs9cEr
https://t.co/z2GF7Uf4tX

video walking through each the commits 👇

[Alex @Alex_tra_memory]

@yp0c5t doesn't seem like there is a list of core concepts to drill for RL which would make it alot more difficult to grasp. i will spin up time on friday to learn more about RL . i am not a math guy so the foundation might take me some time

[Alex @Alex_tra_memory]

trying to watch youtube videos about how transformer and these models won't don't click for me at all\. i have to spend hours just talking to claude voice mode to get them to really click . frankly even then i still feel like i don't know shit.

Agent harness
- tool: a function used by the model to write JSON requests. with usually Ask vs Act modes .
- Context management: often times the harnse just manages large files stored somewhere and not feed it to the model to prevent traffic lag. using RAG systems to find the needed info
- reasoning tokens: decoder outputs between think tags. trained to do analysis mode before answering user requests.

inference-engine:
- scheduler and kv cach managers are the key relationship questions . kv cach manager holds the blocktable, evictor and prefix tree
- Prefill vs decode . batching each token requests to prevent traffic spiking
- slow or garbage outputs.

as mentioned theres still alot i need to learn about each concept and each can into deep layers about them .

[Alex @Alex_tra_memory]

@yp0c5t my main question is if there are any specific terms related to RL i should be aware when studying.

i was using transformer and kv cache as an example of their crucial dynammics

[Alex @Alex_tra_memory]

@yp0c5t thanks i will look into it when i have time. are there any specific terms i should try to focus on like how LLM does with kv cache and multi attention heads

[Alex @Alex_tra_memory]

@yp0c5t interesting , any recommendations on concepts or terms i should study up on?

[Alex @Alex_tra_memory]

some new system design stuff i learned today.

- write heavy is often due to many clients writing to a central server
- to offload server load . we partition the database , this makes it difficult to do a global query for a lighten write load per server.
- read heavy is often servers having to send many requests to different clients
- read vs write heavy doesn't mean one can do one and not the other . write heavy can do a few query search like one individual row or multiple local rows.
- Discord is write heavy but , it's alot of local data reading. So it's not like you need to search across various Discord servers. You just need to update the nearby data
- blob data is best in cloud file / S3 . its just a simple store once and forget. unlike a ready system where its often vast amount of data fed back to the users.
- generally speaking, when it comes to blobs & write heavies, people aren't searching a massive file. It's just a simple few.

[Rishabh Agarwal @godrishuniverse]

Got my third and biggest PR merged to @vllm_project that affects all AMD GPUs, on Diwali day!
https://t.co/pFDbr2pexT

Thanks to the Nick Hill for the quick review!

P.S. Happy Diwali 🪔

#inference #vllm #amd

[Alex @Alex_tra_memory]

@thesimaximalist for deep mathematics ? i am mostly trying to construct a visualization of the transformer and inference engine architecture. number calculations for me still too difficult for me to grasp.

[Alex @Alex_tra_memory]

@Joshua_WD yeah the voice agents providing instant feedbacks and easy breadth & depth searching has been an exponential learning boost. i have learned alot more in 2 weeks using claude voice than any 2 months

[Alex @Alex_tra_memory]

@yp0c5t the ability to explain e2e how transformers and inference engines work. especially with vLLM and related to kv cache management that include eviction, prefix tree & blocktable.

[Alex @Alex_tra_memory]

decided to go on a deep rabbit hole about how to to host a big model like a (70B model at fp16 being 140 GB). 3 methods.

Pipeline Parallelism. hosting subsets of weight layers on different GPUs
Tensor Parallelism: splits every weight matrix in the model into column or row slices, one slice per GPU, so each GPU computes part of every layer and they sum their results after each one.
Expert parallelism: places different whole experts of each MoE layer on different GPUs, so a token travels only to the GPUs holding the experts the router picked for it.

[Alex @Alex_tra_memory]

@NayteThaGreat interesting thanks for the headups i will def keep that in mind for the future

[Alex @Alex_tra_memory]

i came across this tweet & went into a deep rabbithole about how speculative decoding works.

Raced 3 speculative methods against plain decoding on Qwen3-8B (4-bit) on an M5 Pro.

Best method per prompt:
Edit code 1.9× (EAGLE-3, 118 vs 61 tok/s)
Summarize 1.6× (draft model, Qwen3-0.6B)
Math 1.4× (EAGLE-3)
Explain 1.1× (EAGLE-3)

EAGLE-3 can only guess from the 32k most common tokens. Summaries copy rarer words like "kilometres" and "provincial", so those are guaranteed misses.

[Eagle @EagleCorp]

EAGLE-4 is here.

From EAGLE-1 to EAGLE-3, we’ve kept making inference faster.

EAGLE-4 is our fastest yet.

961.6 tokens/s on @Alibaba_Qwen’s Qwen3.8-27B with TokenSpeed (BF16, 4× @NVIDIA B300, TP4, one request, T=0).

Github: https://t.co/GwocZk1UTU

[Alex @Alex_tra_memory]

previously i acidentally tweeted without the original post lol

[Alex @Alex_tra_memory]

@davide_laureti theres no denying training models beat models written by LLMs . but LLMs might be able to achieve decent qualities one day.

[Alex @Alex_tra_memory]

i just gave it a shot and i am just wow by what claude was able to built so quickly. Claude wrote every weight of a legal-citation checker. Zero training, runs on the ANE.

LePhantomCite held-out: 31/31 fakes caught, 1/983 real flagged.

M5 Pro: 50µs per cite, 1.8M cites/s on ANE + GPU, 3.3 GB of case law in 2.3s.

https://t.co/zapMV5orvx

[Jacob Lauritzen @jacsebl]

Got fable to hand-write weights of a 100k param model to recognize real versus fake legal citations.

Basically one-shotted a full NN that’s better than training it

[Alex @Alex_tra_memory]

@NayteThaGreat yeah training is always assumed to be better than what the LLMs can make atm . but its gonna get to a point where LLMs can essentially create their own model weights from scratch. 1 year ? 5 years ? who knows when