Superlinked

Open-source inference for agents.

[Superlinked @superlinked]

Small models are getting good enough that the bottleneck is shifting.

The question is becoming less "can an open model do this?" and more "how do I serve 10, 20 or 50 specialised models without turning my infrastructure into a mess?"

An agent might use one model for OCR, another for embeddings, another for reranking, another for extraction, another for guardrails and another for generation.

That looks very different from serving one giant LLM.

Traditional inference infrastructure is generally built around routing requests to a small number of large model workers. With lots of fast, small-model requests, that approach can leave GPUs sitting idle while the router tries to keep up.

In our latest write-up, @Svonava breaks down the architecture we ended up with at Superlinked:

Shared queues instead of top-down routing.

Workers forming their own batches.

Multiple models packed onto the same GPU.

Models loaded and evicted as demand changes.

Different runtimes hidden behind one inference layer.

In our tested setup, moving to centralised queueing and worker-side batching roughly doubled cluster throughput.

The models are getting smaller. The serving problem isn't.

Read the full article and watch the talk below:

https://t.co/30rrBLUlYe

[Superlinked @superlinked]

Open source AI models are useless if you need a warehouse of GPUs to run them. Real accessibility means smaller footprints and smarter infrastructure handling. Stop over-engineering. Check the SIE repo: https://t.co/1EsK5MDKCB #AI #OpenSource #Tech

[Superlinked @superlinked]

Teams should not have to rebuild the production inference layer around every model, runtime and deployment.

That is why we are building SIE: one production path for small, specialized models, whether you run them yourself or use a managed service.

[Superlinked @superlinked]

Tired of endless thinking tokens? Stop paying for compute you don't need. Fine-tune your models to jump straight to the answer using QLoRA. We crushed the benchmarks for cheap.

Check the SIE repo here: https://t.co/1EsK5MDKCB

#AI #MachineLearning #LLM

[Superlinked @superlinked]

We analyzed dozens of Reddit discussions and found 40 recurring problems with running open models in real products. The biggest lesson?

The model was rarely the hardest part.

Keeping it fast, stable, measurable and private under real traffic was.

[Superlinked @superlinked]

Switching models like Gemma? Don't settle for default speeds. Proper parameter tuning for speculative decoders bumped performance from 30 to 119 tokens/sec on an A100. Stop guessing configurations. Optimize your inference stack here: https://t.co/1EsK5MDKCB

#LLM #AI #MachineL...

[Superlinked @superlinked]

What happens when your model starts reading values from the past?

@f_makraduli ran into exactly that while implementing FlashNorm.

The idea behind FlashNorm is surprisingly simple: RMSNorm doesn't do much arithmetic, but it can still take meaningful wall-clock time because of kernel launches, memory movement and sequential dependencies.

So instead of treating normalization and matrix multiplication as a fixed sequence, FlashNorm rearranges the work.

- Fold the normalization weights into the following projection.
- Defer the scalar normalization.
- Run the RMS calculation and matrix multiplication in parallel using CUDA streams.
- The maths works. The first implementation appeared to work too.
- Unit tests passed. Perplexity looked normal.
- Then longer generations started repeating tokens with a one-step lag.

The problem was a race condition. One CUDA stream hadn't finished writing before the next operation read the buffer, so the model was occasionally consuming stale values.

Fixing it required explicit synchronization between the streams.

We wrote up the full story, including how FlashNorm works, where the speedup actually comes from, and why the reported 33 to 35% improvement applies to the norm-plus-projection operation rather than the entire LLM.

Read the article and watch Filip's talk below.
https://t.co/dt32mMWwYl

[Superlinked @superlinked]

Stop focusing on faster token generation. Smart specialization is the real bottleneck breaker. Loading multiple LoRAs lets you adapt models instantly instead of swapping everything. Generate fewer tokens by being precise.

Check out the SIE repo: https://t.co/1EsK5MDKCB

#AI #...

[Superlinked @superlinked]

Your GPU can be busy processing nothing.

Batch a short query with a long document and pad both to the same length: some compute goes to empty tokens.

Variable length attention skips that padding. "Nothing" is a surprisingly practical optimization target.

[Superlinked @superlinked]

"The query ran successfully" is a surprisingly low bar for text-to-SQL.

That column used to contain dollars, now it contains euros. The SQL still works.

Give the agent that business context. Put the failure case in your eval.

[Superlinked @superlinked]

Before you give your agent a bigger context window, check what you're asking it to carry.

Small models can extract entities and rerank passages before reasoning starts. There's a lot of context engineering to do before you write another prompt.

[Superlinked @superlinked]

Most teams guess their GPU parameters. This team spent $70,000 on a single month of sweeps to actually find the right ones. They built SIE to run better inference on PyTorch, Candle, and SGLang. Check the repo: https://t.co/1EsK5MDKCB #AI #LLM #GPU

[Superlinked @superlinked]

A financial AI answer can be beautifully written and still pull its number from the wrong filing.

"Where did you get that?" is a pretty good eval.

[Superlinked @superlinked]

"The model runs" is a milestone. It's also where a second project often begins.

Routing. Queues. Autoscaling. Monitoring. GPU provisioning.

SIE open-sources the model serving layer and the cluster around it. 🎉

https://t.co/1EsK5MDcN3

[Superlinked @superlinked]

"The model runs" is a milestone. It's also where a second project often begins.

Routing. Queues. Autoscaling. Monitoring. GPU provisioning.

SIE open-sources the model serving layer and the cluster around it. 🎉

https://t.co/1EsK5MDcN3

[Superlinked @superlinked]

An agent rarely needs just one model.

It needs an entire ecosystem of specialized models that can actually work together.

[Superlinked @superlinked]

Benchmarks showing 400k tokens/sec are often misleading. They assume massive, perfectly batched inputs. Real traffic involves tiny, varied requests where performance tanks. Don't trust the marketing numbers.

Check the SIE repo for a better approach: https://t.co/1EsK5MDKCB

#...

[Superlinked @superlinked]

"Small" compared to a frontier model can still be enormous compared to your GPU. Start with the hardware you can actually run and the task (!) you need done.

Then benchmark the candidates. Parameter count only tells you part of the story.

[Superlinked @superlinked]

Stop pooling embeddings. Keep them separate for better search results. This late interaction trend is finally yielding good models. Even better, just index screenshots as images instead of OCR. See it in action at https://t.co/1EsK5MDKCB

#AI #VectorSearch #MachineLearning

[Superlinked @superlinked]

"We just need embeddings."

Then came the PDFs, the reranker, the hybrid search, and the scanned table nobody mentioned in the kickoff.