Michael Feil

Accelerating LLMs @Basetenco - long-context and embedding inference ( - opinions are my own.

[Michael Feil @feilsystem]

@frantzfries At least Github actions are not affected. Wait a second.. https://t.co/gUcfyFAN5P

[Michael Feil @feilsystem]

Baseten runs 9B and 27B - our secrets:
- we trained the models to only need a single inference pass per state, and not one prefill-step per question (e.g. perplexity flash v1.2 can be 30x more expensive)
- we politely asked codex to make the inference faster, make no mistakes. Worked quite well, acoording to AA. https://t.co/ildij4OztX

[sarah guo @saranormous]

~1000 TPS on deepseek flash
love us some model driven model infra

[Alex Ker 🔭 @thealexker]

many models are now orders of magnitude better than humans at kernel work.

e.g. an early checkpoint of kimi k3 handled the majority of kernel optimization for its later development. profiling, rewriting, benchmarking is well-defined and verifiable, which makes it the ideal task for an agent to hill-climb.

similarly, we used our internal harness to optimize 80+ kernels and bottlenecks for deepseek v4.1 flash 1017 to create the fast endpoint.

it is on artificial analysis at 991 output tokens/s and lowest time per task (pareto frontier).

if you want to understand how to make flash models even faster, I explain our optimizations, covering kernel fusion, programmatic dependent launch, and host loop minimization here:

[Michael Feil @feilsystem]

Baseten is offering joey, the world's fastest model:
- at JEV 1.14 quality
- https://t.co/RPf9Zn6tQ6
- 0.01$ price / M

[Michael Feil @feilsystem]

d) baselabs/elephant now running at ~50ms latency on any datacenter near you.

[Max Kirkby @maxkirkby]

our decision models are:

a) frontier
b) free for the week on @baseten Model APIs
c) completely open-source

[turbopuffer @turbopuffer]

Nemotron-3.5-Embed is now available in turbopuffer native embedding

on our enterprise PDF search benchmark, Nemotron-3.5-Embed 8B improves recall@20 by ~5% over the next best model

https://t.co/6pc8hqkS0U

[Baseten @baseten]

Today we’re releasing two decision models, elephant-1 (9B) and mammoth-1 (27B), trained by the @baselabs team, available for free for one week on our Model APIs.

mammoth-1 is #1 on the Jev Decision Index and elephant-1 is tied for #1 on the Artificial Analysis Decision Index.

It’s a new model, but more importantly, this is an open-source model you can access, deploy, and post-train on Baseten (or wherever you do your best work).

The free week ends 10/27/26 at 11:59 PM PT. Try it and let us know what you think.

Try mammoth-1 here: https://t.co/x8oDALMuyZ

Try elephant-1 here:
https://t.co/fmCGb3eGth

[Max Kirkby @maxkirkby]

We trained the #1 decision models in the world, mammoth-1 and elephant-1!

This is our first Base Labs model release and we're excited to open-source all weights and training details. We really focused on scaling high-quality data in these training runs – not only synthetically generating new decision data but stacking how we validated our data. This follows something called our 'swiss cheese' approach to looping over and improving training data.

There are also a bunch more low hanging fruit when it comes to training these models, and we expect that we'll improve quickly over the coming weeks.

[NVIDIA AI @NVIDIAAI]

Meet NVIDIA Nemotron 3.5 Super VL

Our new open model combines reasoning and tool use with text, image, document, and video understanding for long-running AI agents.

120B total parameters with 12B active 🧵

[Michael Feil @feilsystem]

@natolambert @natolambert huggingface also throttles download, so many companies (like us) switched to faster CDN based download for faster speed and no rate-limits.

[Baseten @baseten]

https://t.co/mylK2nCA34

[Michael Feil @feilsystem]

@alephtr @maxm_ideas @baseten We actually do serve a priority tier - e.g. GLM5.2-fast. Currently works by switching the model slug.

[Michael Feil @feilsystem]

@zRdianjiao thanks for working on the many open-source releases, they help to fix bugs. Actually i found this one with glm5.2, running on jinja2 template. glm5.2 is actually quite good at performance engineering.

[Michael Feil @feilsystem]

While working on GLM5.2 chat-templating for Basetenkenizer, found a small bug, thats now fixed with latest minijinja 2.22 (https://t.co/RCPn7s9ks4) and will render the chat-template correctly. Will be included in future releases.
https://t.co/B3UMMngP29

[Michael Feil @feilsystem]

We shipped the worlds fastest tokenizer for KimiK3. Basetenkenizer now available on day 0 on Baseten and on pypi. https://t.co/w9LDOxtYyh

[Michael Feil @feilsystem]

@zRdianjiao yeah, not a big bug, most projects do use jinja2. Also easy to fix, but would perhaps mess with 1-2% of performance if forgotten.

[Michael Feil @feilsystem]

@alvarobartt @baseten @alvarobartt lets get it integrated in TEI iff review process sis fast, wdyt?

[ishan @0xishand]

This is now supported in the dynamo-tokenizers crate. Great work from Michael and the rest of the B10 team!

[Michael Feil @feilsystem]

@0x0SojalSec What would be amazing would be a few layer model with same shapes for relevant layers, as the frameworks do not support arbitrary small input shapes.

[ali @waterloo_intern]

https://t.co/aivc7B4A7Z