Shalen

Product leader. Startup veteran. Dog friend

[Sahil Bloom @SahilBloom]

I believe the greatest gift you can give your kids is a calm home. It’s not the big house or the best toys that stick with them. It’s a place where they feel safe. Where mistakes are met with patience. Where voices stay low. Where hard conversations are always followed by repair. Kids absorb everything. Create peace and they'll carry it for life.

[Massimo @Rainmaker1973]

This guy shows how to rescue yourself if you fall through ice.

[📹 rewildu]

[Cynthia Ozumba @CynthiaOzumba]

Swedish tech companies are remote-friendly, pay top tier rates, and operate almost entirely in English.

You do not need to speak Swedish or live in Europe to get hired.

Here are 3 platforms I researched where you can find remote Swedish roles:

[Sahil Bloom @SahilBloom]

If you want to get ahead, just be easy to work with. Show up on time. Do what you said you'd do. Bring solutions, not problems. Never create drama. Be responsive. Be emotionally consistent. Be kind. People will always want to support someone who just makes their life easier.

[Engineering at Meta @Meta_Engineers]

We're open-sourcing Rebalancer, the assignment-problem solver Meta has used for resource allocation across our infrastructure for over nine years.

Given a set of objects and a set of bins, Rebalancer assigns one to the other to optimize your objectives under your constraints, including hardware placement, service and task placement, traffic routing, and more. It separates how a problem is specified from how it's stored, solved, and debugged, with both an optimal MIP solver and a parallelized local search solver.

At Meta it solves ~40 million assignment problems a day across 30+ problem formulations. P99 solve time is 12 seconds on a problem with 265k objects and 3.2k bins.

🔗 Read more on our engineering blog: https://t.co/9D3jAGfOdQ

[Bob Bowman @coach_bowman]

I've coached over 30 Olympic gold medals.

I've worked with some of the greatest athletes in history.

Here are the lessons I wish more people understood:🧵

[Bob Bowman @coach_bowman]

Set the goal. Build the environment. Do the work. Fix the details. Let them fail while it's cheap. Keep going.

It isn't complicated. It's just hard to do for long period of time

[Paradis @ParadisLabs]

Recommended reading for the weekend:

1. @a16z — State of Markets II
2. @GoldmanSachs — AI Is Becoming a Stock Picker’s Market
3. @FundaAI — Optics: FCC Transceiver Rules Might Be Better Than Feared, and Better for Top Chinese Players Than You'd Think
4. @abcampbell — There Are Bonds in the Singularity
5. @federalreserve — An Update on AI and the Economy

Have a great weekend everyone!

[Matt Canham @matthewcanham]

I've had many people reach out saying they've read everything they can about Jev but still don't get it. Here's an explanation for the normies https://t.co/G7b7mZZwRl

[wafer @wafer_ai]

we gathered all the resources you'll ever need to become the most cracked ai performance engineer

follow and save to keep up with the series. links in thread 🧵

part 7: Efficiently Scaling Transformer Inference

Reiner Pope and coauthors studied PaLM inference on TPU v4. the paper gives performance engineers a way to reason about where weights, activations, and KV caches should live, what must move each step, and which transfers limit latency.

the paper's mechanisms suggest these checks for an inference deployment:

- prefill and decode can have different arithmetic intensity, meaning computation per byte transferred. prefill can process many prompt tokens in one pass; standard autoregressive decode produces one new token per sequence per step. larger decode batches amortize weight reads, but add sequence-specific KV cache traffic. sweep batch size with context length and profile the linear layers and attention to locate the bandwidth or compute limit.

- tensor parallelism can reduce local compute without removing the communication bottleneck. in the paper's 1D layout, activation-aggregation time stays roughly constant as chip count grows for a fixed workload. 2D partitioning splits both weight dimensions so each chip communicates smaller activation shards. evaluate this against the actual interconnect and matrix dimensions before increasing the parallelism degree.

- the best tensor to move changes with the token batch. weight-stationary layouts exchange activations; weight-gathered layouts transfer more weights to reduce activation exchange. large prefills can justify that trade because their activation tensors are large. count the tokens processed per forward pass, including prompt tokens during prefill, when comparing those layouts.

- multiquery attention shares one KV head across query heads, but a head-sharded layout can replicate that cache across devices. the paper shards attention across the batch during decode and exchanges current-step activations to reduce cache reads per chip. in vLLM's grouped-query attention deployments, check whether tensor parallelism exceeds the KV-head count and introduces cache replication before estimating per-GPU capacity.

- KV cache capacity and prefill attention intermediates are separate memory problems. at a fixed batch size and query-head count, sharing KV heads does not shrink the full attention-score tensor. for dense prefill, that tensor's element count grows with the square of sequence length. Appendix G discusses microbatching to reduce temporary allocations and FlashAttention to avoid materializing the full score matrix. use that distinction to identify whether a memory failure comes from retained KV state or temporary attention allocations.

- communication overlap depends on the execution schedule and tensor layout. the authors use Looped CollectiveEinsum to overlap collectives with matrix multiplication, including choosing output sharding that exposes overlap opportunities. inspect a GPU timeline for exposed communication and dependent compute stalls; an asynchronous collective does not establish that its cost is hidden.

- quantization changes different bottlenecks depending on what is quantized. the paper stores weights in int8 while retaining bfloat16 matrix arithmetic. this reduces weight-loading traffic, but does not provide faster arithmetic for compute-bound batches. evaluate weight, activation, and KV cache precision against the measured bottleneck and benchmark the effect on both phases.

- benchmark the operating point that the application needs. the paper compares latency with accelerator-time per token and model FLOPS utilization. in a serving system, track time to first token, inter-token latency, and throughput under the target load. keep queueing time separate from prefill time.

the practical use is to narrow an optimization experiment: estimate compute, HBM traffic, and collective costs for the actual workload, inspect the bottleneck, change the relevant layout or precision, and remeasure at the same latency target.

[Karan🧋 @kmeanskaran]

If you want to learn complete AI Engineering stack, follow these stages:

Stage 1: How LLMs actually work
Tokens, context window, embeddings, prefill vs decode, why every extra token costs time and money.

Stage 2: Prompts and APIs
System prompts, JSON output, tool schemas, retries, prompt versions. Treat prompts like production code.

Stage 3: Retrieval (RAG)
Chunk docs, search, rerank, keep data fresh, cite sources, rewrite bad queries. Most “hallucinations” are retrieval failures.

Stage 4: Backend for GenAI
FastAPI, microservices, auth, rate limits, request queues, Redis caching, webhooks, timeouts. The model is not the whole system.

Stage 5: Memory
Chat memory, long-term memory, summarize vs replay, reuse KV cache, isolate users. Do not store what you should not store.

Stage 6: Agents
Plan → tools → act → check. Graphs, human approval, timeouts, circuit breakers. Agents fail in loops.

Stage 7: Pick and serve the model
Small vs big model, Bedrock vs self-host, vLLM / SGLang, batching, KV cache, quantization. Route cheap first, expensive last.

Stage 8: Observability
Traces, TTFT/ITL, token cost, cache hits, tool errors, CloudWatch, OpenTelemetry. If you cannot see it, you cannot run it.

Stage 9: Quality and safety
Eval sets, grounded answers, schema checks, prompt injection, PII, filters, audit logs. Measure quality before users or lawyers do.

Stage 10: AWS infra
EKS or ECS, ECR, S3, Bedrock, ALB, IAM, VPC. Docker, Kubernetes, Terraform, CI/CD, bash scripts. Scale, failover, quotas.

Stage 11: Prove you can ship
One real RAG + agent app on AWS. FastAPI backend, Terraform + Docker, CI/CD to prod, latency/cost/quality numbers, one incident writeup. Publish an article and repository.

Most people stay stuck watching tutorials. I published a series of 3 articles on design your agentic system.

Comment what I am missing, helpful for me as well.

Part 2: https://t.co/hzmFr0SUqD
Part 1: https://t.co/YRN81sGDrh

Give it a read.

[Pat Grady @gradypb]

The @BostonCollege Investment Committee (an LP and my beloved alma mater) asked for a few thoughts on what's happening in AI. I recorded a test run yesterday morning and then shared it with my partners, who encouraged me to share it more broadly... so here you go!

This is not a sales pitch, it's just a reflection on what we're seeing. And it wasn't intended to be shared, so please pardon the rough edges.

https://t.co/AuZqHMSSxD

[Jaya Gupta @JayaGup10]

If the markets took DeepSeek seriously as a threat to frontier labs, they should be taking Jev (and Jev like-models) far more seriously. Here is why:

1. It's built for every developer, not just ML teams.

Until now, cutting model costs meant open-weight models: choosing a base, hosting it, fine-tuning it, evaluating it, and retraining it. That's an ML team's job, and most companies don't rlly have one. Jev is an API - it works on day one with no GPUs and no training.

2. It's so cheap that everyone is trying it, and everyone is suddenly discovering the waste.

At a fraction of a cent per decision, there's no reason not to test it. Anyone can paste the API key into Claude or ChatGPT and ask it to find savings, including non-technical people (so many talent teams are already using it) Curiosity is turning into discovery: companies see how much they've been paying flagship prices for yes-or-no answers. Teams that never cared about inference cost now do, including non-tech companies.

3. It hits every term of the frontier revenue equation.

Frontier revenue is (calls) × (tokens per call) × (price per token). Decision / Jev models pressure all three.

a.) Fewer calls. Frontier valuations assume the labs capture the agent boom: millions of agents making billions of calls. But most agent steps are decisions: which tool, is this done / safe / escalate / classifying / tagging /routing. Simple generation moves to cheaper and open models. Retries, wasted steps, and escalations disappear !

b.) Smaller calls. Routers strip stale context and unneeded tools, so the calls that still reach the frontier cost less.

c.) Cheaper calls. Once routers show cost per task side by side, flagship pricing on routine work doesn't hold.

4. It puts a router in front of and within every AI product.

Routers existed before, but they were hard to build and unreliable. Decision models fix both. They're the ideal router: fast, nearly free, and they report how confident they are, so the router knows when to escalate. Routers are now becoming a built-in setting in the tools developers already use, so companies get one without building anything.

Today, the frontier model runs the agent: it decides what to do next and calls tools. With decision models, ordinary code and cheap decisions run the loop, and the frontier model becomes a function the software calls when it needs heavy reasoning

5.) Every defense the labs have speeds it up. Price cuts confirm customers were overpaying. Shipping their own routers teaches customers to route, and makes it easier to route to competitors! Cheap decision tiers move the labs' own customers onto cheaper products and cannibalize revenue.

6.) They make post-training economical, and simultaneously increase the demand for it. One of the higher costs in RL on real enterprise work has been grading tasks without a checkable answer. A cheap, calibrated grader removes that cost. Environments become cheaper to run, router logs show which expensive workloads are worth replacing, and every team that discovered its overspend becomes a curious buyer. Companies that train, serve, and retrain custom models get cheaper inputs and a pipeline of motivated customers at the same time.

If DeepSeek made the market question the supply side: the cost of pretraining, the durability of the model moat, and how long a frontier model stays frontier, decision models make it question the demand side: how many calls actually need a frontier model at all.....

[Kirk Borne @KirkDBorne]

9 Types of LLM Evaluation (technical frameworks for evaluating quality, reliability, and failure modes)

Infographic source: https://t.co/iyYUTq0OJh

[Navaneeth Krishnan @navaneethvb]

I barely see people talk about how agentic workloads create a pretty weird problem for LLM inference, especially around the KV cache during tool calling.

lets say an agent already has 20k tokens of context so firstly it does the prefill, builds the KV cache, starts generating, then decides it needs to call some tool.

so now the flow becomes something like

20k context -> prefill -> decode -> tool call -> wait 10 sec -> tool result -> continue decoding

now our issue is the waiting period, ie: 10 secs

the agent is not generating anything during this period. the GPU can obviously serve other requests, but this agent's KV may still be sitting there consuming memory even though its decode is paused.

but the catch is from the agent framework side this is obviously still the same ongoing task but from the inference server side, that request can basically look finished.

and the KV cache sitting there might be 20k tokens worth of context that the agent is literally going to need again a few seconds later.

so in simple words , lets say you want to do some kind of research on elephants, you will enter a prompt " get me xyz about elephants", that will be your first request . which will do prefill and decode . the decode will make a tool call which will ask the agent to perform deterministic tasks like searching, crawling etc which may take few seconds.

once that request finishes, those KV blocks may no longer have an active request protecting them. they can stay cached, but under memory pressure they can fall into the normal eviction order even though the agent is about to need them again.

so ideally when the tool result comes back you want

old 20k KV + prefill only the new tool result -> continue decode not recompute the entire 20k prefix again -> add tool result -> continue decode

this sounds obvious, so why not just keep the KV around?

because now imagine 100 agents doing this.

- agent A is waiting 8 sec for a tool
- agent B is waiting 30 sec
- agent C is waiting on some API
- agent D spawned another agent

if you keep every single KV resident on GPU just because the agent might come back, suddenly a massive chunk of your KV memory is being used by requests that aren't even generating tokens.
then active requests start getting squeezed.
so now you have 2 bad options.

A: Evict too aggressively and you keep recomputing giant prefixes every time an agent comes back from a tool call.

B: Retain too aggressively and idle agents basically eat your GPU KV memory.

and normal cache policies don't really know enough to make this decision properly.

Now there is another case u must know:

lets say one KV prefix hasn't been touched for 15 sec because the agent is waiting for Stripe, but the tool response is coming back in 1 sec.

another prefix was used 5 sec ago but that conversation is basically done.

so logically LRU might prefer the second one.

but the agent runtime knows the first one is way more valuable and the inference engine doesn't.

this gets even more interesting when agents branch.

lets say a coding agent already has 50k tokens of repo context and then launches 3 subagents.

-> agent A

50k shared KV -> agent B
-> agent C

all 3 can share a massive prefix.

you definitely don't want to prefill that same 50k context from scratch 3 times if you can avoid it.
and once one child finishes, its unique KV might become useless immediately while the shared parent prefix is still extremely valuable.

so this isn't really just a "KV cache eviction" problem anymore and the bigger issue is that agentic workloads have lifecycle information that the inference runtime normally cannot see.

such as:
- tool waiting
- agent branching
- session still alive
- branch finished
- agent about to resume
- agent moving to another worker

and this is why serving stacks are starting to make KV management more session aware / programmable.

instead of the inference engine only seeing, request_id = 123

the agent layer can basically tell it more about what is happening like:

- this session is waiting for a tool.
- this KV is probably going to be reused soon.
- this branch is dead.
- move this KV to CPU for now.
- prefetch it back because the tool call is almost done.

and the serving runtime can then decide whether to retain, evict, offload or prefetch based on actual workflow information instead of just guessing from request activity.

and then there is another problem on top of this.

'routing'

if the agent generated on GPU 2, calls a tool, then comes back and gets routed to GPU 7...

GPU 7 might not even have that KV so now the actual problem becomes

- when do you keep the KV?
- where do you keep the KV?
- where do you route the agent when it comes back?

agentic workloads make KV cache management way more interesting than just "store tokens so we don't recompute attention."

so how do you actually fix this?

so logically the inference engine needs some idea of the agent/session lifecycle, instead of treating every model call like a completely independent request.

The agent orchestrator already knows things the inference engine doesn't.

- this agent is waiting on a tool.
- this branch is dead.
- this session will probably resume soon.
- so pass that information down as KV hints.
- retain this prefix.
- demote this KV to CPU.
- dereference this branch.
- prefetch this back because the tool is almost done.
- GPU for hot KV.
- CPU / distributed storage for idle KV that might still come back.

And when the agent resumes, either route it back to the worker that already has its cache or fetch that KV from the shared store and along with the request object, we feed all this information to the inference engine so it can figure out KV management.

eg:
request = {
"session_id": "agent_42",
"state": "waiting_for_tool",
"expected_resume": "soon",
"kv_hint": "retain_or_offload",
"branch_status": "active"
}

[Gauri Tripathi @GauriTripa94282]

9 LLM Concepts AI Engineers Should Know
1. Tokenization → Tokens
2. Embeddings → Vectors
3. Transformers → LLMs
4. Attention → Context
5. Pre-training → Learning
6. Fine-tuning → Adaptation
7. RLHF/DPO → Alignment
8. RAG → Grounded AI
9. LoRA/PEFT → Efficient tuning
#AI

[OptionHans @HansCashFlow]

LEAPS can be tricky.

Think about:
• What is the option pricing environment at entry
• Strike is key, and it's not always the same delta
• Expiry long enough to give the thesis room
• An income layer while you wait

30 years. $1B managed. One page. Click below, it's free.

[Shalini Goyal @goyalshaliniuk]

Picking the right Claude model is now a product decision, not just a technical one.

Because each model solves a different kind of problem.

Haiku is built for speed and cost efficiency. It works well when the task is simple, repetitive, and high-volume — like chatbots, tagging, classification, data extraction, or quick customer support replies.

Sonnet is the balanced default for most professional workflows. It gives strong reasoning, good coding capability, reliable instruction following, and works well for writing, debugging, research, analysis, and multi-step business tasks.

Opus 4.7 is for the situations where quality matters more than speed or cost. Deep research, complex decision-making, long document analysis, strategy work, difficult coding problems, and high-stakes reasoning are where it makes the most sense.

The key is not asking, “Which Claude model is best?”

The better question is:

What does this workflow actually need?

Speed?
Cost control?
Reasoning depth?
Coding strength?
Long-context understanding?
Tool and agent usage?

That answer decides the model.

Use Haiku when efficiency matters.
Use Sonnet when balance matters.
Use Opus 4.7 when depth and precision matter.

The smartest AI teams don’t use the biggest model for everything.

They match the model to the task.

BTW, order for the book is now open!

Get your copy of System Design and Behavioral Intelligence here:

Amazon: https://t.co/Vw40DPccBl

[NO1ennn @N01ennn]

this is f*cking gold

four loop shapes. running, they all look identical.

they differ in one thing: what is allowed to end them.

> turn based -> a prompt, the model checks itself, stops when it decides it is done
> goal based -> add an evaluator, stops when the score clears or the try cap hits
> time based -> add a clock, stops when you cancel it
> proactive -> take yourself out, it does not stop until you switch the routine off

number two is the one worth stealing.

the evaluator exists to refuse "i'm finished."

the model tries to stop. something else sends it back to work.

/goal reach 90+, stop after 5 tries.

number four is the one that changes your week.

event fires -> agent works -> PR opens -> you review.

you are in that chain exactly once, at the end.

a chat window needs you awake for every single turn.

everyone picks a loop by what starts it.

wrong end of the machine.

pick by what stops it.

[Alexandr Wang @alexandr_wang]

check out the muse ideas tab for inspo on how to use muse!