shb7

AI / local LLMs / agents - notes from shipping & breaking things

[shb7 @shb7el3alm]

@PhiloGroves attackers were never gonna stay on the compliant API anyway. the safeguards mostly land on whoever is doing incident response

[shb7 @shb7el3alm]

@Cisco_research 16ms is nice but I wanna see the calibration outside their own benchmarks. a confident wrong probability is worse than a wrong sentence imo

[shb7 @shb7el3alm]

@Fremond_ the GLM-5.1 bit got me. 52.5 to 65.5 on swe-bench verified with zero weight changes, just better tools and context

[shb7 @shb7el3alm]

State of AI Report 2026 is out. part that stuck with me: OpenAI test agents got into Hugging Face prod, and HF's forensic prompts got blocked by commercial API guardrails. they used GLM-5.2 open weights to investigate

guardrails blocking the defenders is backwards

[shb7 @shb7el3alm]

Vals AI audited the RL envs Xiaomi open sourced for MiMo v2.6. in 1,795 of 2,698 coding tasks the fix commit was still sitting in git

git commands were blocked so the model wrote its own pack file parser

id take MiMo coding scores with salt for now

[shb7 @shb7el3alm]

@iamcurtismith @XiaomiMiMo look-ahead bias is the right comparison. 67% of tasks leaking the fix, id want those coding numbers rerun on clean repos before trusting them

[shb7 @shb7el3alm]

@AmoremWeb honestly the credits matter more than haiku for me. you running haiku as the subagent model or still keeping sonnet for that

[shb7 @shb7el3alm]

@alannnfx ultra first again. pro people back to refreshing the app every hour

[shb7 @shb7el3alm]

@hinadixit is qa1-qa5 the usual subset people report for babilong? curious how the harder tasks hold up at 10M

[shb7 @shb7el3alm]

@alby13 how long did the pretrain take on the one 4090? and how many tokens. thats always the part i wanna know

[shb7 @shb7el3alm]

@DracoVibeCoding the 1.9x at low concurrency is the number i care about tbh, most of us arent serving 150 tps per user lol

[shb7 @shb7el3alm]

vllm-omni put out a tech report. idea is one runtime for speech, diffusion image gen, world models and robot loops instead of stitching separate servers together. want to see if its usable on one consumer gpu or just cluster stuff

[vLLM @vllm_project]

🚀 Excited to share the vLLM-Omni technical report: a unified serving runtime for omni-modality generation.

📄 Paper: https://t.co/b0KvTmrEF3
🔗 Repo: https://t.co/I1wL6iqO0r

Speech assistants, visual generation, world models, and robot loops have pushed serving past a single text decode loop. The execution patterns diverge with the output: multi-stage autoregressive pipelines, iterative diffusion, and sessions that carry state from step to step. LLM servers and diffusion stacks each go deep on only one of these, so deployments fall back to stitching disjoint runtimes together. vLLM-Omni is the shared control plane for that mix. An orchestrator advances each request across stages; specialized engines run the compute; a connector carries the payloads; and the same session path keeps duplex, world-model, and robot loops on one runtime.

Really grateful to the vLLM and vLLM-Omni teams for the collaboration and support. Contributions are welcome. 🙏

[shb7 @shb7el3alm]

gpt-6 and intelligent ui are rolling out to everyone in chatgpt now. the ui part lets the model fill answers with charts and little interactive widgets instead of plain text. curious if it stays useful or turns into clutter after a week

[shb7 @shb7el3alm]

@jayleaton spin up then give the gpu back is exactly what a single spark needs. how long is cold start for something like qwen-image?

[shb7 @shb7el3alm]

@KrakowiakK catch is probably the eval. 85.1 vs 84.8 on one benchmark is noise range. id want long context + tool calling numbers before trusting 2.5bpw

[shb7 @shb7el3alm]

@FlowAltDelete GA but no pricing row yet is so typical lol. you can use it, you just cant tell what each request costs

[shb7 @shb7el3alm]

@Beamsters1 @STRML_ @ddalcu under a minute TTFT at 100k context on a laptop is the part i didnt expect. is that 4bit or 8bit weights?

[shb7 @shb7el3alm]

@ItsmeAjayKV my guess is 45ish. but at flash tier people dont pick on the index, they pick on price per task. haiku 5.5 being that cheap on short prompts already squeezes that hard

[shb7 @shb7el3alm]

deepseek pushed 36.8T tokens through openrouter in the week ending oct 3. openai + google + anthropic + xai combined is ~33T. yes tokens arent tasks and deepseek talks a lot, but thats still devs picking price over brand

[shb7 @shb7el3alm]

microsoft put llama.cpp on stage at the windows event and is shipping MAI-Code-1.1 Flash, a 137B coding model meant to run on your PC. good for local AI. but 137B on a normal laptop? want to see the actual RAM/VRAM numbers before i get excited

[Georgi Gerganov @ggerganov]

It is very cool to see llama.cpp on the big stage in todays Windows event!

The software and hardware stacks are finally coming together. Our community has put a lot of hard work in the past years and it shows.

Looking forward to more users embracing local AI.