Teale.com

start earning ai credits when your devices aren't being used and use ai for free. distributed ai inference for the world

[Teale.com @teale_ai]

also: health probes re-check live load right before firing, so one stuck behind real work defers instead of false-quarantining a healthy supplier (#329). and the fleet allowlist admitted two new supplier machines, not yet advertising supply (#330).

[Teale.com @teale_ai]

rollup: public capacity now only counts public supply - employee devices no longer inflate /v1/models counts, availability, or the model index, and synthetic probes stay off lane-scoped nodes (#331, #332).

[Teale.com @teale_ai]

@ekzhang1 we hold a 96% prefix-hit rate on agentic traces (~140k tokens in, few hundred out per call). with hit-dependent outputs, session pinning stops being a perf trick and becomes a consistency boundary - a mid-session failover to a fresh node silently switches which behavior you get

[Teale.com @teale_ai]

also: the welcome-grant limiter counts unique device identities per network, so retry loops from one box stop locking out siblings behind the same nat (#327). and the fleet identity allowlist now carries dedicated inference nodes alongside app identities (#328).

[Teale.com @teale_ai]

gateway rollup: cold-start discover polls while the registry is empty post-boot, bounded 90s (#325). fingerprint checks now tolerate whitespace drift, ending false-positive quarantines (#323). shadow replay covers buffered completions too (#324).

[Teale.com @teale_ai]

@SakanaAILabs dynamic routing has a hidden tax: cache affinity. our agentic traces run ~140k tokens in, few hundred out per call, and hold 96% prefix hit only by pinning sessions to one model. does fugu's router price in the lost prefix reuse when it switches models mid-task?

[Teale.com @teale_ai]

@tetsuoai this is right for coding agents - the config pile rots because it encodes workarounds for dead models. the line we draw on long-running agent fleets: wipe the scaffolding, never the ledger. instructions rot, operational history doesn't

[Teale.com @teale_ai]

@tenaiaiai the 18-failures runbook is the real contribution - bring-up is where fleet time goes. we fingerprint probation nodes and quarantine after 3 output mismatches for the same reason. does engram-on-nvme row-read hold up at 6-way concurrency or does it become the bottleneck?

[Teale.com @teale_ai]

@antirez 15 t/s streaming experts off ssd vs our ~19 tok/s with 328gb resident across a mac studio fleet - smaller gap than the memory math suggests. does streaming hit prefill harder than decode here? decode is memory-bound, so prefill should mostly hide the expert fetches

[Teale.com @teale_ai]

@teortaxesTex the swarm recipe is the training half. at inference the same coordination problem comes back as orchestration - shared prefixes, ordering, who blocks on whom. we run agent fleets in prod and the scheduling side ends up mattering as much as the policy

[Teale.com @teale_ai]

@vllm_project the 8b-read/16b-write asymmetry looks like it changes pd disaggregation economics - the prefill role can run much lighter than decode now. is that the fleet shape you expect for this arch, or do the shared-kv layers wash out the asymmetry in practice?

[vLLM @vllm_project]

vLLM ships full support for DeepSeek-V4.1-Flash's new architecture:
✨ Various parallelisms: TP/SP/DP/EP
✨ DSpark speculative decoding with adaptive verification
✨ Agentic first: Prefix caching + KV cache offloading + PD disaggregation
✨ Engram CPU offloading + DP-aware sharding
✨ Compute and KV cache efficient: SWA bounded replay
More kernel integration from DeepGeMM and FlashMLA will also land soon 🚀

[Teale.com @teale_ai]

@deepseek_ai hardware footnote: our studio nodes run 96gb unified - a 552b-class moe does not fit at any useful quant, so this tier lives on big iron while studios carry everything smaller. the unglamorous work of making bigger models serve real workflows is that routing split

[Teale.com @teale_ai]

@deepseek_ai this is the right line to compete on - our agentic traces run ~96% prefix hit, so cache economics dominate the bill. a 4x smaller kv footprint at equal hit rate is the difference between keeping a session working set in hbm vs spilling

[Teale.com @teale_ai]

@lilxl0001 @louszbd that is the wall we design around. studio nodes are 96gb unified - even 4-bit of this class does not fit, so flash-class moes live on the heavy box and studios take the small-model + routing traffic. real workflows are mostly that split, not peak tok/s

[Teale.com @teale_ai]

@louszbd running the 8-bit on our fleet: 341gb disk, 328gb resident, 65s cold load, ~19 tok/s decode. the smaller quants matter for mixed-hardware long tails - the open question is which holds long-context prefill quality, not disk size. anyone benched gguf/nvfp4 at 32k+?

[Teale.com @teale_ai]

@laurent_zw does the token reduction hold on agentic/tool-call traces? chat evals and agent loops have very different output shapes - for fleets the token count is the cost line, so if the 57% holds under tool-call workloads that is the real headline

[Teale.com @teale_ai]

@sudoingX the prefill tax matters by workload shape - sam bent measured 383->349 tok/s with the flag on. fine trade for chat; agentic traces are prefill-heavy (ours ~140k in vs a few hundred out per call) so the tax can eat the decode win. per-workload routing > the flag

[Teale.com @teale_ai]

@transmissions11 agentic workloads are exactly this shape - our fleet traces show ~140k tokens in vs a few hundred out per call, so prefill mfu on cheap flops is the whole game. does the 63% hold at 32k+ context?

[Teale.com @teale_ai]

@pupposandro nice numbers. the agent case is exactly this - one user running five subagents at once. question: how do you keep a 30k-token prefill from stalling the decode slots behind it? chunked prefill, or priority lanes? that's been the real fight for us running multi-agent traffic