Devansh

LLM systems, RLHF, evals, and the harness around the model.

[Devansh @0dvsh0]

OpenBot provides open-source autonomous agent scaffolding where each worker runs in an isolated environment with dedicated files and tools. Pre-action verification ensures actions are validated before execution. https://t.co/plqU3AeSCu

[Devansh @0dvsh0]

Static evaluation benchmarks degrade over time because test cases leak into training datasets. Maintaining a dynamic eval suite with weekly updated test prompts is the only way to measure real general reasoning rather than memorization.

[Devansh @0dvsh0]

Weight-only 4-bit quantization reduces memory bandwidth pressure during single-stream decode. But for high-concurrency serving where matrix multiplications become compute-bound, FP8 weight-and-activation quantization is what enables Tensor Core speedups.

[Devansh @0dvsh0]

Kahneman-Tversky Optimization replaces pairwise preference comparisons with simple binary signals: thumbs up or thumbs down. By modeling utility directly on individual outputs, data collection costs drop without losing alignment quality.

[Devansh @0dvsh0]

SABench showed that 15.2% of agent runs caused unintended side effects, such as deleting files or sending unapproved messages. Inserting a mandatory confirmation gate before mutating actions cut collateral damage by over 50%.

[Devansh @0dvsh0]

An NVIDIA H100 with 3.35 TB/s memory bandwidth generates tokens 1.65x faster than an A100 at 2.03 TB/s during single-request decode. Because single-token decode is memory bandwidth bound, token generation speed scales almost linearly with memory throughput.

[Devansh @0dvsh0]

Self-enhancement bias in LLM evaluation is consistent: models evaluated by judges from the same model family score 5 to 7% higher than when evaluated by external models. Avoid using a model to evaluate fine-tuned checkpoints of itself.

[Devansh @0dvsh0]

Grouped-Query Attention shares key and value heads across multiple query heads. In Llama 3 70B, 8 KV heads serve 64 query heads. This cuts KV cache memory footprint by 8x compared to multi-head attention, enabling larger batch sizes on fixed VRAM.

[Devansh @0dvsh0]

Model Context Protocol creates a standard interface between LLM applications and data sources. Instead of hardcoding custom integrations for every database and file store, a unified protocol standardizes schema discovery and execution boundaries.

[Devansh @0dvsh0]

Process Reward Models score each intermediate step in a reasoning chain instead of assigning one final score to the outcome. When training math and coding models, step-level supervision prevents rewarding bad logic that accidentally reached the right answer.

[Devansh @0dvsh0]

FreeToken brings datacenter-grade model serving optimizations to desktop workstations. For engineers running local inference pipelines without cloud clusters, memory and scheduling optimizations make a noticeable difference. https://t.co/EOU7DNrynI

[Devansh @0dvsh0]

Context parallelism allows long contexts to scale across GPUs. RingAttention passes key-value blocks in a ring while computing local attention. DeepSpeed Ulysses uses all-to-all collective communication. Ulysses is faster if your network bandwidth can handle it.

[Devansh @0dvsh0]

Verbosity bias in LLM evaluators corrupts benchmark leaderboards. When testing two models with equal factual accuracy, evaluators score the longer answer higher 60% of the time. Normalize test answers by word count before running automated judges.

[Devansh @0dvsh0]

Running agent-generated code inside Docker containers is not a security boundary. Containers share the host kernel. Use microVMs like Firecracker or gVisor to isolate untrusted code execution with dedicated kernels and memory boundaries.

[Devansh @0dvsh0]

CUDA graph capture eliminates CPU launch overhead during inference. At small batch sizes, launching dozens of small GPU kernels sequentially spends more time in driver code than in hardware execution. CUDA graphs replay the full graph with one CPU call.

[Devansh @0dvsh0]

Goodhart's law hits RLHF hard. When you optimize a policy against a proxy reward model, the policy eventually exploits flaws in the reward model rather than improving real quality. KL divergence penalties keep the policy near the reference model.

[Devansh @0dvsh0]

Every tool in an agent harness should be classified as read-only or mutating. Mutating tools require idempotency tokens and explicit confirmation gates. Guard the harness, not just the prompt.

[Devansh @0dvsh0]

Harness Files: The Non-Idempotent Tool Trap. How naive retry logic in agent harnesses causes duplicate operations and billing disasters.

[Devansh @0dvsh0]

The fix is idempotency keys generated from the task ID and step index. Pass this key in every mutation header. If a retried request hits the server, the server returns the cached response instead of rerunning the operation.