As an AI engineer, please learn:
- Learn the roofline model and why decode is memory-bound
- Deploy vLLM and SGLang, then read their schedulers
- Understand paged attention from the code, not the blog post
- Build observability before you optimize anything
- Track TTFT, inter-token latency, throughput, queue depth
- Use Grafana + Prometheus for inference dashboards
- Turn on prefix caching and find which workloads it helps
- Learn continuous batching and chunked prefill
- Run load tests with 1000+ concurrent requests
- Report p50, p95, p99, never just the mean
- Master quantization tradeoffs (FP8, INT4, AWQ, GPTQ)
- Learn speculative decoding and where it stops helping
- Set up KV cache eviction for long contexts
- Try disaggregated prefill and decode serving
- Learn Kubernetes for AI workloads and autoscale on queue depth
- Learn how inference costs break unit economics
- Build your own model router by cost, latency, quality
- Create a token budgeting system per request
- Build one inference service and benchmark it publicly
- Read inference research instead of model release news
- Start sharing your optimization benchmarks
I put together a 10-week plan that covers every one of these at 30 minutes a day. It is 50 sessions, split between reading the theory and building on your own service, and all of them feed one artifact: an inference service you deploy, instrument, load test past 1000 concurrent requests, tune, and publish as a reproducible benchmark.
It is open on GitHub and contributions are welcome, especially newer sources worth adding. I am working through it myself and will share more content on this going forward, so stay tuned.
GitHub repo: https://t.co/UTxKAzhzJQ
(don't forget to star π)
Another insane way to use Jev:
Jev makes it dramatically cheaper to identify which corrections and trajectories from an agent run are reusable across agents.
And finally, someone open-sourced a self-improving memory layer that can put that signal to work across agent harnesses:
- Claude Code
- Codex
- Cursor
- OpenCode
- Hermes, and 20+ more
Beacon by @asymptotelabs continuously captures agent sessions across harnesses and stores their traces in a shared index.
I used it with Jev to build a live cross-agent memory workflow.
Beacon captures each run as it happens across all harnesses on my machine. Then Jev evaluates whether the trace contains a reusable engineering lesson, and the approved lesson becomes available to the next agent.
GitHub repo: https://t.co/7n0b9guyQI
(don't forget to star it β)
Preserving an agent run and learning from it are two different problems.
Most traces contain file reads, failed commands, routine exploration, and implementation details that only apply to one task.
Passing the complete transcript to another agent adds context without identifying what matters.
In this workflow, one agent never receives another agent's raw transcript. It receives only the specific engineering lesson produced from the evaluated trace.
A correction made in one agent can then prevent the same mistake when another agent encounters a similar problem.
The recording shows the complete workflow I built.
If you want to dive deeper into Jev, I also wrote a hands-on guide to building this Jev-style decision path with open models, entirely locally.
Read it below.
I have been fine-tuning LLMs for over 3 years now.
Here are 5 LLM fine-tuning techniques, explained with visuals:
Traditional fineβtuning is impractical for LLMs (billions of params; 100s of GB).
Since this kind of compute isn't accessible to everyone, parameter-efficient fine-tuning techniques (PEFT) were built.
Before we go into details of each technique, here's some background that will help you better understand these techniques:
LLM weights are matrices of numbers adjusted during fine-tuning.
Most PEFT techniques involve finding a lower-rank adaptation of these matrices, a smaller-dimensional matrix that can still represent the information stored in the original.
Now with a basic understanding of the rank of a matrix, we're in a good position to understand the different fine-tuning techniques.
(refer to the image below for a visual explanation of each technique)
1) LoRA
- Add two low-rank trainable matrices, A and B, alongside the weight matrices.
- Instead of fine-tuning W, adjust the updates in these low-rank matrices.
Even for the largest LLMs, LoRA matrices take up a few MBs of memory.
2) LoRA-FA
While LoRA significantly decreases the total trainable parameters, it requires substantial activation memory to update the low-rank weights.
LoRA-FA (FA stands for Frozen-A) freezes matrix A and only updates matrix B.
3) VeRA
- In LoRA, low-rank matrices A and B are unique for each layer.
- In VeRA, A and B are frozen, random, and shared across all layers.
- Instead, it learns layer-specific scaling VECTORS (b and d) instead.
4) Delta-LoRA
- It tunes the matrix W as well, but not in the traditional way.
- Here, the difference (or delta) between the product of matrices A and B in two consecutive training steps is added to W.
5) LoRA+
- In LoRA, both matrices A and B are updated with the same learning rate.
- Authors of LoRA+ found that setting a higher learning rate for matrix B results in better convergence.
____
Find me β @_avichawla
Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.
@Akintola_steve Good opportunity to take some time off. Spend quality time with family, rejevunete oneself.
Get back to work, either start your own business or get a job
@araseb_@lucasfancation Its better to know the reason the working behind the working of localhost. Also learning fundamentals of programming won't hurt your progress