RunInfra (YC F26)

The inference platform that auto-optimizes your model
by @rightnowai_co

[RunInfra (YC F26) @runinfrai]

RunInfra beta is live!

Describe your use case. Tell us what to optimize for, which models, and your latency + cost targets. We handle the rest: kernels, quantization, serverless deploy, routing, autoscaling.

And your optimized model now deploys straight through our connector integration. More coming soon.

Owning your AI isn't a someday thing anymore: https://t.co/Tk7EMOI9UC

[Jaber @Akashi203]

you can now export your optimized models in any format at @runinfrai

docker kit, GGUF, ollama bundle. OpenAI-compatible API inside, optimized kernels underneath

deploy on your own hardware

[Jaber @Akashi203]

i tried to reach LPU performance on one b200 and i outperformed it, without fusing anything or writing a single kernel

in this blog i show that the hardware is not the problem, the software is https://t.co/gBPtBFj5NH

[RunInfra (YC F26) @runinfrai]

@JensenHuang it’s now available on https://t.co/m8MQfBFAAc

[Osama @OsamaKakashi]

There's a bar missing from this chart: $0 per million tokens

We optimized DeepSeek-V4-Flash-0731 to 2.98x faster than baseline. 9248ms to 3095ms median latency, 113 to 363 tok/s. Lossless, output parity by construction

You run it on your own hardware. No per-token bill at all

Benchmark receipt and setup: https://t.co/Hu85Bl26u0

[Jaber @Akashi203]

We optimized DeepSeek-V4-Flash-0731 at @runinfrai and made it 2.98x faster than baseline. Median latency dropped from 9248 ms to 3095 ms and throughput went from 113 tokens/s to 363 tokens/s. The optimization is lossless, with output parity by construction

We don't host it. You run it on your own hardware.
It costs $60 one time and it is yours forever

The kit is tuned for vLLM 0.25.0 on 4x B200 and ships with the full benchmark receipt. Deploy it on any GPU provider or on your own machines

No deprecations, no silent model swaps, no per-token pricing. You control your intelligence!!

https://t.co/gVkn09RGdG

[Jaber @Akashi203]

I wrote about lossless inference, how to make serving faster without touching the model. exact kernels, speculative decoding, KV reuse, the math for why quantization quietly kills agent runs https://t.co/OgtW9jun7z

[Jaber @Akashi203]

what actually happens when you send 'hi' to an llm https://t.co/JqSMqDQJeH

[Jaber @Akashi203]

blog of what LLM inference actually costs

$0.09 to $290.12 per 1M output tokens. almost none of it is the model https://t.co/wPOfOdyKYE

[RunInfra (YC F26) @runinfrai]

BTW
Everyone should own their AI

[RunInfra (YC F26) @runinfrai]

Own your intelligence instead of renting it

[RightNow (YC F26) @rightnowai_co]

We signed the Open Weights and American AI Leadership letter

RightNow AI has backed open source since day zero. Our kernel work is public, and everything we build assumes the weights are public too

That is also why we built @runinfrai Take any open source model, let it get optimized at the kernel level, and deploy it on your own GPUs. No closed API in the middle and no per token bill on someone else's cloud.

Own your intelligence instead of renting it!

https://t.co/8izGNNthWw

[RightNow (YC F26) @rightnowai_co]

run Kimi on your own GPU and use it with the coding agent you already have

Claude Code, Codex, Cline, Aider, opencode. all of them expect a different API. we wrote a bridge that figures out which one is calling and translates, so you point your agent at localhost and it works with no changes

the model side: Kimi-Linear-48B is 91.5 GiB out of the box, too big for any single card. we quantized it to 28.8 GB and rewrote the decode kernels, so it runs at 113 tok/s on one 32 GB GPU. a 5090 runs it.

24 GB cards are next!

https://t.co/bTaDaSdJJC

[RunInfra (YC F26) @runinfrai]

You don't own the models behind the APIs you build on

RunInfra Catalog: pre-optimized open models. Custom kernels, quantization, serving configs already done

Run them on your GPUs, or any cloud

Buy once. Run anywhere. Keep forever

https://t.co/t6TX3Lrh1s

[Jaber @Akashi203]

kimi K3 is coming

@rightnowai_co and @runinfrai are writing optimized kernels for it. we're open sourcing all of them

own your model. serve it on your own hardware instead of renting a closed API

[Jaber @Akashi203]

open-sourced the attention kernels we built for serving @thinkymachines Inkling on vLLM

the model adds a learned position term to every attention score. the shipping path computes that with a callback, once per score. we build it as a tile instead and add it in one pass before softmax. same checkpoint, same math, just the kernel

2.66x faster attention at batch-1 64K decode on H100. 1.44x at 8K prefill

1.10x end to end on 8x H200. attention is only a slice of serving time, so a 2.66x kernel buys about 10%. that's the arithmetic, not a disappointment

it also runs on A100, which the shipping path doesn't. day-0 sends every non-Blackwell GPU down a route that then refuses to run there, so it fails at the first attention call with the weights already loaded. ours works

two bugs found upstream:

tml-fa4 takes the bias, allocates it, launches the kernel that reshapes it, then never passes it to the attention kernel on anything pre-Blackwell. no error. the output looks fine and is wrong

pack_gqa quietly redefines what a row of the score tile means. it folds 8 query heads into the sequence dimension, so 128 rows stops being 128 positions. anything that indexes by row breaks. nothing in the API mentions it. took 17 hours to find

https://t.co/Ds34bo317t

[RunInfra (YC F26) @runinfrai]

Open models only matter if you can run them fast

The weights are free. The inference is not

That is what we spend every day on

[Jensen Huang @JensenHuang]

For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.

AI will transform every industry, power every company, and be built by every country.

Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.

The world needs both frontier closed models and frontier open models.

https://t.co/AUKzoQ5Ikb

[Jaber @Akashi203]

we started by researching what comes after the transformer. what the next architecture actually has to fix

we kept landing on the same thing. a transformer has two clocks and both stop. weights freeze when pretraining ends. the context window empties when the session does. everything the model learns while you talk to it gets thrown away. every fix on the table was retrieval bolted on the side, not weights that move

so we built memoir. an architecture with weights that never freeze, and the pre-registered experiment that falsified its central claim

four sets of weights running on different clocks. fast weights update on every forward pass. episodic weights update once per session. slow weights consolidate rarely, a sleep phase. a frozen core never changes at all, which is what keeps a self-modifying model anchored

on top of that sits a pondering core. a small recurrent block that runs a variable number of steps and decides for itself when to stop. reasoning depth stops being tied to parameter count

one objective drives every tier. predict the next latent state, score it with an energy head. low energy means coherent. high energy means think longer or refuse to answer. and the update rules themselves are small networks found by meta-optimization instead of written by hand

the novel part was the join between the first two. every pondering step writes into the fast weights, so thinking reinforces the paths it uses -- a model that gets better at a problem while it is working on it. nobody had built that

so we built the falsification harness before building anything on top of it. two arms, 81,738 parameters each, bitwise identical init, same batches, same seeds. arm A writes to fast weights while pondering. arm B reads them but writes only between forward passes, the ordinary approach. kill criteria registered before the first run

12 seeds. coupled 0.5203, read-only 0.6557. the ordinary version wins by 13.5 points. paired t 3.23, read-only ahead on 10 of 12 seeds

the first run is still in the repo and it was invalid. both arms hit 100% recall and the harness printed a confident FALSIFIED anyway. a ceiling cannot separate two models. a benchmark that answers a question it cannot measure is itself part of the result

then a reviewer asked whether both arms were simply still improving at 240 steps. trained 4x longer, both reach 1.0000, the gap closes. so the claim narrowed from "the coupling is worse" to "the coupling learns slower at a fixed budget", and the docs were rewritten to the weaker version

the confound, stated up front: the coupled arm writes about 7x more often. this shows the configuration is slower, not that the timing of the writes caused it. the arm that separates those is not run yet

the most interesting thing in the architecture currently pays no rent. better to find that out after two days than after six months

all of this ran on one laptop GPU. the next experiments are already scoped -- the third arm that separates write timing from write volume, a sweep over write strength, a task that stays below ceiling at convergence, and then whether any of it holds at 1B parameters. we cannot fund that ourselves

if you want to sponsor this with compute, dm me

https://t.co/s4WT0ZtKQ9

[RunInfra (YC F26) @runinfrai]

01101011 01100101 01110010 01101110 01100101 01101100 01110011 00100000 01100001 01110010 01100101 00100000 01110100 01101000 01100101 00100000 01101101 01101111 01100001 01110100

[Jaber @Akashi203]

i open-sourced bonsai-turbo -- a batch-1 decode engine that runs @PrismML's Bonsai 27B 1.76x faster than the official llama.cpp fork. same outputs, token for token

H100, tg128, greedy: ternary 85.5 >> 151 tok/s. 1-bit 90.1 >> 159 tok/s. logit parity with the fork on 32 of 32 test prompts, gated before any speed number counts. not a lossy trick

why it's faster: at batch-1 the GPU isn't math-bound or bandwidth-bound, it's overhead-bound. the stock path executes 3703 GPU ops per token and spends ~97% of its time on that op overhead. bonsai-turbo fuses the whole per-token pass into a handful of large ops. --mega mode compiles the entire 64-layer token step -- embed >> layers >> logits >> next token -- into one cooperative kernel

the kernels were generated by our internal agent -- the same agent that powers @runinfrai

scope is deliberate: batch-1 decode only. Bonsai 27B is the ternary 27B small enough to run on a phone, and this makes the local single-user experience actually fast. it is not a batched-serving engine

roofline says ~440-490 tok/s is on the table. next: cp.async weight pipelining, then a speculative drafter -- targeting ~300

https://t.co/YtYhjpqJUg

[Jaber @Akashi203]

we built the agi compiler

it watches your llm agent work, finds the parts that are secretly deterministic, and compiles them into verified binaries that cost nothing to run

llms are just the first frontend, world models and new model types plug into the same toolchain, that's why it's a compiler

same 300 tasks, same verified answers, 6.4x less money

paper + code at the end

[Jaber @Akashi203]

NVIDIA's Director of Accelerated Computing just said RightNow is seeing up to 14x more performance and 92% lower cost from AI-optimized CUDA kernels.

the agents behind those numbers are now live on @runinfrai, and @runinfrai is now fully public >> https://t.co/o8g2Xe0t1W

bring any Hugging Face model, our agents generate optimized GPU kernels for it, deploy serverless, pay per 1M tokens. LLMs, embeddings, voice, STT/TTS. you can also export your optimized model to Modal or RunPod and run it yourself.

for startups and small businesses that want to own their AI instead of renting it.

opening a few enterprise spots for end-to-end model optimization >> https://t.co/GE2pePoVzq

all feedback is welcome!!