Fimbul

AI Wizard - inference, fine tuning, txt2img, constrained decoding and MLOps

[Fimbul @fimbulvntr]

Blog post coming soon!

[Parasail @parasail_io]

We're #1 on Kimi K3 (max) in @ArtificialAnlys's provider benchmark.

186 output tokens/sec and 12.75s to first answer token, across ten providers.

At $2.31 per 1M blended, matching the lowest price in the field. Fastest of the ten, priced with the cheapest.

Live now with the full 1.05M context.

https://t.co/0NCS0Eem5t

[Daniel Jeffries @Dan_Jeffries1]

The answers to this are simple:

1) Either you believe in freedom, and many intelligence agents (the human kind) making their own decisions about how to use power tools or you think that a closed, tightly guarded, authoritarian path is better.

We believe that most people can be trusted with kitchen knives to cut vegetables and you punish the few people who stab someone, like everything else.

AI is not magic that somehow magically transscends the entire history of civilization and technological development and economics.

2) I don't believe that their are "special people" with special knowledge who can guide and steer intelligence or the economy more widely than the distributed intelligence of mankind.

This didn't work in the Soviet era and it won't work now.

3) Better defense against risk is when everyone has equal access.

When Huggingface was under attack they had to turn to open models to defend themsleves because the closed model and their ham fisted safeguards infantalize people and block legitimate use.

We cannot have the bad guys with better tools while we have crippled, gated tools.

4) We believe in the Proactionary Principle, not the Precautionary Principle.

Precautionary means we have to prove a negative that something may or may not happen in the future. This is nonsense.

We legislate real things when they happen in the real world. Harms must be actually demonstrable not in people's heads and imaginary and most of the harms are exactly that, imaginary, kind of mass hallucination.

5) There are many smart folks on the labs but smart is meaningless.

You can be a genius in one area of your life and have no actual ability to effectively predict the future or understand how a technology plays out. Raw intelligence does not equal wisdom.

Many of the foolish predictions have continued to play our exactly the opposite of how "smart" folks predicted, like jobs. But we are just supposed to take the rest of their fantasies seriously?

6) In fact many of the folks at the labs are actively and deliberately and transparently lobbying for regulatory capture.

This is totally and completely unacceptable.

Win on merit. Win on competition.

You do not get to win with protectionism and locking out competition and delivering an inferior, gated, crippled product for "our own good" and then legislating that is how it has to be.

[Fimbul @fimbulvntr]

@saltyAom Agreed (and Elysia does a fantastic job). That said, it's usually a good idea for the framework author to tell us what to do (what you consider idiomatic) since presumably you the author know more than me-the-noob.

[Parasail @parasail_io]

Most real-time AI traffic isn't steady. It's spiky and cyclical. Size a fixed GPU endpoint for the average and spikes blow past it; size it for the peak and you're paying for idle capacity.

That's the gap we built dedicated serverless to fill: a private endpoint with consistent latency that scales with your spikes, billed per token. You only pay for what you use. https://t.co/F1KzJ9YTPj

[🐝🇬🇷 @bee_fumo]

Ok here's my wishlist:
- The ability to install unsigned apps painlessly
- The ability to access Android/data Android/obb with a basic file manager(no root required)
- The ability to disable all Google services and apps on my phone with the press of a button
- Google services becomes open source and is replaced with "Android Services"
- Root access with a basic check in the developer settings
- Blob emojis

[Devon Eriksen @Devon_Eriksen_]

I can disassemble the Lockheed SR-71 Blackbird into its component parts.

None of the these component parts is capable of supersonic flight, unless I throw them really hard, and that would be cheating.

Therefore, no part of the Lockheed SR-71 Blackbird is capable of supersonic flight.

Therefore, the Lockheed SR-71 Blackbird is incapable of supersonic flight.

This is called the "fallacy of division".

That's a fancy name for not realizing that collection of stuff might be able to do something that none of its parts can.

[Parasail @parasail_io]

Gemma 4 is live on Parasail. Zero-day Google launch partner.

26B MoE: 4B active params at inference. Competing with the GPT-OSS workhorse tier.
31B dense: 256K context, thinking mode, vision.

Audio models in testing with select partners — reach out if that's you.

Pay for what you use. Scale like crazy.

[Fimbul @fimbulvntr]

@martianwyrdlord DM me. I'm in a position where I can actually make this happen. I'm serious.

[Fimbul @fimbulvntr]

The problem is Berkson's paradox and market forces constrain her dating pool: men who are romantic AND desirable... don't date Chloé.

This becomes a utilitarian belief. For her.

And she inadvertently revealed her Pareto frontier.

Ouch.

[CHLOÉ HAPPE @bronzeageshawty]

A big revelation I've had is that Romantic men are actually losers

[tweet davidson @andyreed]

how it feels to navigate an enterprise saas codebase with claude

[prerat @prerat]

im happy to read ai-written stuff if your prompt is you rambling about your topic for 30 min and asking ai to synthesize and make it readable

but dont try to write an ai article by giving it a one line prompt and asking for 2k words of elaboration

are you compressing or not

[Penna @penadev]

@nixcraft old man yells at claude

[Fimbul @fimbulvntr]

@InnocuousShark @PhilippusArabus @CityBureaucrat Pretty much. It sells imported items, kind of like a shitty, small, low-variety Macy's. Hence the statue: "imported US stuff".

[Fimbul @fimbulvntr]

I don't doubt, but in aggregate, there isn't demand.

Chicken and egg problem: no one builds anything that can use the capability because the capability doesn't exist - no one creates the capability because no one seems to need it.

But it's not an unsolvable problem - models ARE getting faster.

[Fimbul @fimbulvntr]

@naumenko_roman @jarredsumner I went into a bit more detail about the economics here: https://t.co/RngmwAIdxh

But I suspect you were looking for the economics of throughput and training...?

[Fimbul @fimbulvntr]

What kind of answers were you expecting, though?

1. There's no demand for 100k tokens per second
2. For frontier models, it's better to get 1% extra in MMLU, or bolt on audio output, or be able to read the time in an analog clock... than 2x throughput
3. There's a ton of overhead in existing inference engines:
3.1. python interpreter
3.2. Http stuff: SSE, tls handshake, validation, parsing
3.3. Pre-inference stuff like (de)tokenization, batch scheduling, logit processing, many request parsers

People mostly don't care because the slow part is inference, and besides, most users are talking to their LLMs directly - and once you go past 50 t/s, it's above reading speed, so irrelevant.

It may seem like I'm saying "we will never need more than 64kb of RAM" but that's not what I'm going for - I'm telling you the incentives aren't aligned for 100k tokens per second.

And besides - it's not like no one is trying. Current models are about 10x faster than they were a year ago! I'm sure we'll get there anyway even if hardware advances slow down (which seems unlikely).

Finally, are you really sure it's the best use of your time to use up years of research and a ton of money making a model that is mediocre (in the classical "middling" sense of the word) but very fast? When the market says they won't pay very much for that?

In a field where each experiment costs, at minimum, thousands of dollars and tens of hours of time?

By the time you're done, everyone will have moved on to 3D models that inject knowledge directly into your frontal lobe while you sleep (hyperbole) and all you have for your efforts is a mid model that can yap nonsense incredibly quickly. Good luck selling that.

[Fimbul @fimbulvntr]

What kind of answers were you expecting, though?

1. There's no demand for 100k tokens per second
2. For frontier models, it's better to get 1% extra in MMLU, or bolt on audio output, or be able to read the time in an analog clock... than 2x throughput
3. There's a ton of overhead in existing inference engines:
3.1. python interpreter
3.2. Http stuff: SSE, tls handshake, validation, parsing
3.3. Pre-inference stuff like (de)tokenization, batch scheduling, logit processing, many request parsers

People mostly don't care because the slow part is inference, and besides, most users are talking to their LLMs directly - and once you go past 50 t/s, it's above reading speed, so irrelevant.

It may seem like I'm saying "we will never need more than 64kb of RAM" but that's not what I'm going for - I'm telling you the incentives aren't aligned for 100k tokens per second.

And besides - it's not like no one is trying. Current models are about 10x faster than they were a year ago! I'm sure we'll get there anyway even if hardware advances slow down (which seems unlikely).

Finally, are you really sure it's the best use of your time to use up years of research and a ton of money making a model that is mediocre (in the classical "middling" sense of the word) but very fast? When the market says they won't pay very much for that?

In a field where each experiment costs, at minimum, thousands of dollars and tens of hours of time?

By the time you're done, everyone will have moved on to 3D models that inject knowledge directly into your frontal lobe while you sleep (hyperbole) and all you have for your efforts is a mid model that can yap nonsense incredibly quickly. Good luck selling that.

[Fimbul @fimbulvntr]

Egalitarianism isn't a prerequisite for libertarianism (or the NAP).

Even worse: blank-slatism. A notion so anathema to evidence that it is indefensible without invoking mysticism.

But it's a "mind virus" that has infected billions, including many libertarians. The cure is to just... abandon it.

[Fimbul @fimbulvntr]

Hi! Actual person who works on inference and model training here.

We know about a lot of tricks. There are so, so many tricks to get inference faster.

The problem is that unlike game design, whenever you change the architecture of a model to do one thing faster, you often have to retrain the entire model from scratch.

But yes, it's moving incredibly fast - a year ago we didn't have nearly as many context length and attention tricks as we have today, and models were much bigger and slower to inference and train (MoE took over after deepseek R1).

Now the industry is moving towards multimodal and reasoning, while simultaneously incorporating speculative decoding, mixed quantization and hybrid linear attention mechanisms.

It's not 100,000 tokens per second, though.

There's also varying levels of different forces pulling frontier labs this way and that:
- Smarter models
- Faster
- More context
- More capabilities
- Cheaper to train
- Cheaper to inference

So speed isn't the only problem - people are trying to optimize for 5~10 different things at any one time. And still, progress is moving at breakneck pacing at any one vertical.

Finally, there's also a psychological component:
- People are unwilling to pay much extra for individual request speeds once you break the 50~100 t/s barrier. There just isn't that much demand for 100k tokens per second
- Fast models are seen as "dumber" as slower models, which are perceived to be "thinking harder"

I can go into details about any of these things, but there's way too much to cram into a single post.