srikar kovvuri

cs + math @columbia | prev @ netflix

[srikar kovvuri @srikar1651]

@theagentpod Yeah for my baseline server, TTFT went up quite a bit for longer prompts because I didn't implement any form of continuous batching like what vLLM and llama.cpp do

[srikar kovvuri @srikar1651]

Wanted to see how my baseline LLM inference server stacked up against vLLM and llama.cpp.

Benchmarked all three under simulated workloads with multithreaded concurrent clients, varying prompt lengths, and TTFT, TBT, and KV-cache measurements.

Interesting results 👇

[srikar kovvuri @srikar1651]

Built a 162K-parameter GPT in PyTorch, trained on 1.1M Shakespeare characters.

Starting with a bigram baseline, the Transformer reduced validation loss by 23%.

For inference, I sped up bigram decoding by 3.34× and implemented GPT KV caching, with logits matching within 2.4e-6.

[srikar kovvuri @srikar1651]

https://t.co/kMl3cPRtHe

[srikar kovvuri @srikar1651]

I tried 3 ways to shrink an LLM's KV cache.

Sliding-window eviction. Attention-based eviction. INT8 quantization.

On TinyLlama, INT8 used ~4× less KV memory than FP32 and preserved output quality in my tests. But CPU decode got slower.

Here's what I found:

[srikar kovvuri @srikar1651]

(4/4) Tradeoff

Less memory didn't mean faster decoding.

Importance eviction introduced scoring, sorting, and gathering overhead.

My naive INT8 implementation repeatedly dequantized and requantized the full cache during decoding.

[srikar kovvuri @srikar1651]

@david_y_xiong fire

[srikar kovvuri @srikar1651]

@bevid11 been playing around with langraph

[srikar kovvuri @srikar1651]

Built a customer service agent that can verify users, check orders, process returns, update payments/addresses, and hand off to humans when needed. Check it out here: https://t.co/heJAllllcn

[srikar kovvuri @srikar1651]

Just implemented GPT-2 from scratch -> KV caching, batch generation, greedy + nucleus sampling.

Then fine-tuned it for classification. Need to tell the difference between actual insight and someone just arguing online? Look no further. Check it out: https://t.co/WquEhOmlKv

[srikar kovvuri @srikar1651]

System design concepts hit different when actually implemented.

Added presigned URLs so the client uploads straight to S3 -> no bloated backend handling media.

Server does logic.
S3 does storage.

Next step: teaching the agent to understand what users send with vision.

[srikar kovvuri @srikar1651]

Taking a systems class & didn’t think blocking I/O mattered.
Then my WebSockets slowed down. Why?
Flask → 1 thread per connection.
WebSockets stay open → 1000 users = 1000 OS threads.
Fix: Eventlet(monkey patching) → 1 OS thread + many green threads (non-blocking). Pretty cool

[srikar kovvuri @srikar1651]

Built a little BERT model that masks words in sentences and guesses what goes in the blanks -> turns out it's actually pretty good at it. Check it out: https://t.co/nB4KXzyzcr

[srikar kovvuri @srikar1651]

@aedinpereira Ig ppl just be taking breaks for anything these days

[srikar kovvuri @srikar1651]

Word2Vec was a huge breakthrough in learning meaningful word representations from text.

Wanted to implement the neural network by playing around with pytorch. Check it out here: https://t.co/lsjF9myYny

[srikar kovvuri @srikar1651]

Been playing around with some classical ml. Lowk always took regression for granted, but it's pretty cool seeing it implemented and adding more layers. Check it out: https://t.co/I4On29bXy5

[srikar kovvuri @srikar1651]

@heyblake A Messaging platform where you can interact with agents and others users within rooms. Check it out https://t.co/Qk45r2viYP