alsante

Writing about enterprise AI architecture, local inference, privacy-first systems and why production constraints matter more than benchmarks.

[alsante @alsante111]

I've been thinking about a problem most AI discussions seem to skip.

We're putting more and more of our data, inference and application logic behind APIs we don't control.

It works great — until a model gets deprecated, pricing changes, an endpoint goes down, behaviour changes, or the data simply can't leave the network.

That's why I'm interested in:

local AI, open-weight models, air-gapped RAG, AI infrastructure and the real economics of owning the stack.

I'm not anti-cloud.

I'm interested in knowing what we're actually giving up when we use it.

[alsante @alsante111]

Everyone is optimizing the model.

Perplexity's numbers suggest another lever matters just as much: how much context you make the model read.

Medium Search scores 80 on Artificial Analysis while keeping model inference cost around $0.03 per task.

Better retrieval isn't just about accuracy.

It's a way to make the model do less work.

[Artificial Analysis @ArtificialAnlys]

Perplexity Search debuts on the Artificial Analysis Search Index, with all three context size variants taking top positions on the leaderboard

The @perplexity_ai Search API comes with three context settings (low, medium, and high) that control how much extracted content each search result carries. We tested all three variants using our standardized methodology: the same model (GPT-5.6 Luna at medium reasoning), running inside Stirrup, our open-source agent harness, with tools for searching and fetching pages from the web. Only the provider behind the search tool changes.

Key results:

➤ Perplexity Search (medium) scores 80 on the Artificial Analysis Search Index, ahead of the previous leaders, Parallel (advanced) and Brave Search (LLM context), at 75. The high and low variants score 79 and 77 respectively. Its lead is concentrated in BrowseComp results, with AA-Omniscience and DeepSearchQA scoring comparably to other leading providers

➤ Efficient search payloads: smaller overall search results mean the model reads less per task, so Perplexity has the lowest model inference cost per task of providers we’ve tested so far, ranging from $0.028 to $0.034 across the three variants vs $0.036 for the next lowest provider

➤ Total cost per task is ~$0.091 for the medium and high context variants, at mid-pack latency. For comparison, Parallel (advanced) costs $0.084 per task and Brave (LLM context) costs $0.13 per task

[alsante @alsante111]

770B parameters. 49B active.

That's the number I find more interesting than the SWE-bench Pro headline.

MoE models are increasingly separating what a model knows from what it needs to compute per token.

For local and private inference, that distinction matters more than the headline parameter count.

The real question is: how much useful intelligence can we get without paying to activate the whole model?

[Cline @cline]

Tencent Hy4 Preview leads on SWE-bench Pro. 770B, 49B active, 1M context, and their biggest generational leap measured to date.

It’s exciting to see another open weights model compete against the frontier.

Try in Cline with:
1. npm i -g cline
2. /model
3. Select Hy4 preview

[alsante @alsante111]

Quantization is becoming more than a way to shrink model files.

TyloQuant's MFQ takes a different approach: instead of forcing the whole model into the same format, it uses different levels of precision where they're actually useful.

That's especially interesting for multimodal models like MiniCPM-o 4.5, where you're dealing with vision, audio and language in the same system.

For local AI, smarter quantization may matter more than throwing more hardware at the problem.

[OpenBMB @OpenBMB]

MiniCPM-o 4.5 now runs efficiently on Apple Silicon with TyloQuant MFQ! 🥰

The team at @Tylogi_ai has released an MFQ-quantized build of MiniCPM-o 4.5, bringing efficient multimodal inference to local devices.

Rather than applying a single quantization format uniformly, TyloQuant MFQ uses neuron-aware mixed-format quantization to reduce memory usage while retaining MiniCPM-o 4.5’s vision, audio, and real-time interaction capabilities. 🤖

Highlights:
⚡ Native acceleration with Metal on Apple Silicon
📉 Multiple quantization levels for different memory and quality targets
🎥 Vision, audio, and full-duplex multimodal interaction
🛠️ MFQ-backed Metal and CUDA kernels with native C++ runtime components

This release shows how advanced mixed-format quantization can make powerful multimodal models more practical for private, local, and edge deployment.

🤖 MiniCPM-o 4.5 model:
https://t.co/bRcld16Jco

🔗 Explore the MFQ model:
https://t.co/FwqJ6CQoE5

📦 Download the MFQ model:
https://t.co/XJzs4WOXrJ

[alsante @alsante111]

Agnes 2.5 Pro Beta gained 9 points on the Intelligence Index.

It also more than doubled its output token usage.

That's an interesting trade-off for production.

If a model needs 2× the inference budget to move up the leaderboard, the benchmark score alone isn't telling you enough.

The real metric might be intelligence per token — not intelligence per task.

[Artificial Analysis @ArtificialAnlys]

Agnes AI's Agnes 2.5 Pro Beta scores 49 on the Artificial Analysis Intelligence Index, up 9 points from Agnes 2.5 Pro Alpha, driven by large agentic gains but using ~2x the output tokens

Agnes AI (@agnesai_sapiens) is a Singapore-based AI lab that trains full-modality foundation models and offers them through a free omni-modal API.

At 49 on the Intelligence Index, Agnes 2.5 Pro Beta moves Agnes from mid-pack to the frontier-adjacent tier, just behind Gemini 3.5 Flash (high, 52) and GPT-5.6 Luna (max, 52) and ahead of MiniMax-M3 (45).

Key results:

➤ Agnes 2.5 Pro Beta scores 49 on the Artificial Analysis Intelligence Index, a 9-point jump from Agnes 2.5 Pro Alpha (40). This places it just behind Gemini 3.5 Flash (high, 52) and GPT-5.6 Luna (max, 52), and ahead of MiniMax-M3 (45).

➤ Agentic capabilities drive the jump. The Artificial Analysis Agentic Index rises from 25 to 44, just behind Gemini 3.7 Flash (high, 45) and ahead of Gemini 3.5 Flash (high, 40) and MiniMax-M3 (36). τ³-Banking nearly triples from 12% to 36%, and GDPval-AA v2 rises from an Elo of 1171 to 1456 against a human baseline of 1000.

➤ Frontier reasoning evaluations improves more modestly. Humanity's Last Exam rises from 34% to 38%, GPQA Diamond from 88% to 91%, and CritPt from 11% to 16%.

➤ The AA-Omniscience improvement from -25 to -11 comes from abstention, not increased accuracy. Agnes 2.5 Pro Beta attempts only 45% of questions against Agnes 2.5 Pro Alpha's 94%, cutting the hallucination rate from 88% to 33%, but AA-Omniscience Accuracy halves from 33% to 17%.

➤ The intelligence gain required ~2x as many tokens from its predecessor. Agnes 2.5 Pro Beta uses 50k output tokens per Intelligence Index task, more than double Agnes 2.5 Pro Alpha's 24k.

Additional model details:

➤ Context window: 1M tokens

➤ Max output tokens: 65k

➤ Input modalities: Text and image

➤ Pricing: $0.10 / $0.30 / $0.01 per 1M input / output / cache hit tokens

➤ Availability: Agnes AI first-party API

[alsante @alsante111]

The interesting part of Ornith-1.5 isn't the model size. It's the inference stack.

Self-speculative decoding with MTP delivers up to 1.78× speedup on the 397B model without needing a separate draft model.

That's especially interesting for local inference. Getting more throughput from the hardware you already have is a lot cheaper than adding more GPUs.

As models get larger, inference efficiency matters just as much as raw capability.

[Ornith @ornith_]

🔍We dove a bit deeper and analyzed how multi-token prediction (MTP) works in Ornith 1.5 models.

🪽With MTP, Ornith 1.5 gain inference speedups without quality loss via self-speculative decoding: models use their own MTP head to draft tokens and verify them in a forward pass.

🐦MTP weights are now updated for all 9B, 35B, and 397B models in BF16, GGUF, FP8, and NVFP4 variants (except 397B NVFP4, will be updated today).

https://t.co/mGJLwhrQOM

[alsante @alsante111]

320B parameters.
18B active.
This is what efficient scaling looks like.

GLM-5.3-Flash is showing that model size and inference cost don't have to grow together.

If sparse architectures can keep pushing frontier-level capability toward lower active compute, the local-vs-cloud debate gets a lot more interesting.

Maybe the next AI hardware advantage won't come from bigger models. It will come from models that waste less compute.

[Z.ai @Zai_org]

Introducing GLM-5.3-Flash

- Leading capabilities at a highly competitive price
- Natively multimodal with a 1M-token context window
- A 320B-A18B model released under the MIT License
- Previously previewed as Ox Alpha, running entirely on Chinese AI chips

Blog: https://t.co/tzOmB7gdZP

Available now across all official platforms:

Weights: https://t.co/9LRMahY9Wa
API: https://t.co/VcaQnzYmS9
Coding Plan: https://t.co/Nk8Y98HNhU
ZCode: https://t.co/Peepqv4XSx
Chat: https://t.co/WCqWT0qCQb
AutoClaw: https://t.co/aGEG5HqTTb

[alsante @alsante111]

@babaliauskas Exactly. Regression suites help validate the migration, but they don't remove the migration cost.

You still have to re-embed the corpus, rebuild the index and re-validate retrieval.

That's the state lock I was getting at.

[alsante @alsante111]

Your embedding model isn't just an implementation detail in RAG.

It's part of your data layer.

You can swap the LLM tomorrow. Change the embedding model and the migration gets much uglier: re-embed the corpus, rebuild the index, then prove retrieval quality didn't regress.

That's the part teams don't think about when they build "quick RAG."

The API is easy to replace. The state isn't.

How portable is your RAG architecture really?

[alsante @alsante111]

This is one reason open-weight, self-hosted models are interesting for production.

You control when the model, runtime and inference stack change.

You still have to manage that stack — but the upgrade decision is yours.

Reliable AI isn't just about choosing the best model. It's about controlling when the behavior is allowed to change.

[alsante @alsante111]

You can pin a software dependency in package.json.

You can also pin a specific LLM model version.

What you can't fully pin is the entire behavioral stack around a cloud model: routing, safety layers, system changes and other provider-side components.

In production, an unannounced behavioral change can be a breaking change. 🧵

[alsante @alsante111]

In a consumer chat, a different answer is usually harmless.

In an enterprise workflow, a small behavioral change can break JSON parsing, function calls or downstream automation.

That's what makes AI dependencies different from ordinary software dependencies: the interface can stay the same while the behavior underneath changes.

[alsante @alsante111]

The future of enterprise AI isn't just better models.

It's models embedded in systems that can retrieve, verify, constrain and recover.

Bigger models can improve capability.

Better systems make that capability reliable.

That's the part of AI infrastructure I'm most interested in.

[alsante @alsante111]

Small models keep proving the same point:

better architecture can matter more than more parameters.

MathForm-8B doesn't just generate Lean formalizations. It retrieves definitions, compiles the result, checks consistency and feeds errors back into refinement.

That's a very different approach from asking a larger model to "try harder."

Maybe the next leap in AI reliability comes from better systems around models, not just bigger models.

[OpenBMB @OpenBMB]

🧮 Introducing MathForm, an open-source framework, dataset, and model for mathematical autoformalization with Lean 4.

Formalizing mathematics makes mathematical knowledge machine-checkable, but it is more than translating statements into code. A model must map each concept onto the right types and definitions in Mathlib. A formal statement can compile and still misstate the original problem.

Highlights ✨
MathForm Framework: Retrieval-Augmented, Verification-Guided Data Construction A retrieval planner pulls the Mathlib definitions and existing formalizations a statement needs. The generator then revises its output against Lean compiler diagnostics and semantic-consistency feedback for up to 3 rounds.

FormalVerse Dataset: 367K+ Verified Lean 4 Examples Each example pairs a natural-language statement with verified Lean 4 code , across diverse mathematical domains and sources.

Results 📊
• At a matched 100K budget with the same recipe and init, models trained on FormalVerse reach 60.32% Consistency Check, vs 46.53% on FineLeanCorpus and 41.49% on NuminaMath-LEAN
• MathForm-8B achieves 88.06% Syntax Check and 72.37% Consistency Check Pass@8 across six benchmarks, outperforming ReForm-32B and Goedel-Formalizer-V2-32B at a quarter the size
• On the hardest FATE-H / FATE-X subsets it reaches 63% / 37%
Consistency Check, beating the strongest specialized baseline by 10 and 12 points

🔗 Resources
📄 Paper: https://t.co/W3EA8CHWox
📚 Dataset: https://t.co/3nP1WFPOZD
🤖 Model: https://t.co/TFQxOexDaC
💻 Code: https://t.co/SXv4VdxLzg

[alsante @alsante111]

We're used to asking whether AI models are cheaper than APIs.

Samudra 2 is a more interesting example: an AI emulator can turn a workload that normally depends on large-scale scientific computing into something a small team can run on a single GPU.

That's not just an inference optimization.

It's a change in who gets access to compute.

The real impact of local AI may be bigger than enterprise chatbots.

[Emma Scharfman @EmmaScharfmann]

Very cool open source model available on Hugging Face: Samudra 2 runs multi-year ocean forecasts (temperature, salinity, sea surface height,...) at resolutions down to 1/4° and on a single GPU.
If you're working on similar models, feel free to reach out. We're thinking about build a benchmark for ocean and atmospheric forecast models 🌍🌊
Try it here: https://t.co/GoaktpkGwL
Model weights: https://t.co/5rq8mlvXoH

[alsante @alsante111]

A year ago, running a model like this locally meant throwing more hardware at the problem.

Now the interesting part is the software stack: EXL3 quantization, NVFP4 KV cache, speculative decoding and kernel-level optimization are pushing a much larger model onto a single 128GB machine.

Local AI is increasingly becoming an optimization problem, not just a hardware problem.

The question is no longer "Can this model fit?"

It's how much capability can we extract from the hardware we already own?

[Mia @MiaAI_lab]

A BIG moment for all DGX Spark users ⚡️

You can now run DeepSeek v4 Flash 0731 without needing a second unit, with quality high enough for reliable code generation, high context, and great speed!

Optimized for single stream session:
- EXL3 quantization
- 384k context (conservative) / ~440k kv cache (!)
- 47 tok/s single stream (structured)
- 1024 tok/s prefill
- 370k token needle test passed (super stable!)

Thanks @0xSero for this excellent EXL3 quant! Tuned for SparkInfer + DSpark speculative decoding, this is about the same quality of a Q4_K_M / Q5 GGUF!

All this is possible thanks to native NVFP4 KV cache & fixing kernel bugs in the upstream prefill path to make it work at all.

Get it here:
https://t.co/531EaR6FWJ

[alsante @alsante111]

The interesting thing about CS-4 isn't 750 PFLOPS.

It's that inference is being split into two different problems: prefill and decode.

Scale the prompt processing where throughput matters. Put ultra-low-latency token generation on hardware designed specifically for it.

As reasoning workloads get longer, this kind of disaggregated architecture may matter more than another generation of raw FLOPS.

Maybe the future of AI infrastructure isn't one accelerator doing everything.

[Cerebras @cerebras]

The CS-4 is the fastest AI accelerator in the world, establishing a new roofline for Frontier AI.

CS-4 is built to accelerate every part of the AI ecosystem:
- Developers who want faster tokens for interactive reasoning and agentic applications.
- AI factories require infrastructure that is simpler to deploy and built for hyperscale.
- Hyperscalers and Neoclouds need ultra-fast, high-throughput inference to serve the most intelligent AI models in real-time.

[alsante @alsante111]

Generative media is moving from single-prompt generation to agentic production workflows.

The interesting part isn't just the agents. It's the control layer around them: local assets, review checkpoints, reusable workflows and enterprise deployment.

In commercial production, full autonomy isn't the goal. Brand consistency, approvals and predictable output are.

The real challenge is automating more of the workflow without giving up control of the result.

[MiniMax Design (H3) @Hailuo_AI]

���MiniMax Design is officially live!
Turn Ideas into Commercial-Grade Content
#MiniMaxDesign #MiniMax

🤖Agent-Driven Workflow: Input your goal. Agents autonomously plan, execute, and deliver.
🪄Built for Commercial Creation: Craft ads, e-commerce assets, motion graphics, and post-production content.
🧩Open & Flexible Integration: Connect local assets, deploy privately, and scale through APIs.

[alsante @alsante111]

Pokee’s enterprise thesis is right: some workloads simply shouldn’t leave the security perimeter.

The interesting claim is the other one: 10M+ tokens on a single GPU.

If that holds in production, the architecture matters more than the headline number.

What happens to TTFT when the context is full? How does retrieval quality degrade at 5M or 9M tokens? What does sustained throughput look like?

Long context is not just a capacity metric. It is an inference, latency and recall problem.

Are we actually solving the context problem — or just moving the cost somewhere else?

[Pokee AI @Pokee_AI]

Why run AI locally when GPT or Claude are already so good?

Because some workloads simply shouldn’t leave your environment.

Think:
• Sensitive enterprise data & strict data residency
• Private codebases, logs, research, legal or financial docs
• Air-gapped / offline environments
• Low-latency workflows that can’t depend on an API
• Massive context that becomes expensive to repeatedly send to the cloud
• Teams that need full control over models, data, and infrastructure

And “just build it internally” isn’t always the answer. Internal AI infrastructure means significant engineering work across optimization, deployment, monitoring, and maintenance.

That’s where local deployment gets interesting.
Pokee Isaac 28B is built for this: up to 10M context on a single GPU.

Not a replacement for GPT or Claude everywhere.
A different tool for the scenarios where privacy, context, control, and deployment economics matter more than calling another API.

Book a demo with Pokee today: https://t.co/ND3vtl3quN

[alsante @alsante111]

Don't build local infrastructure to save $50/month on basic chat queries.

Build it when:

• the data can't leave the network
• utilization is high enough to justify the hardware
• predictable latency matters
• vendor dependency is becoming a material risk
The real TCO question isn't "cloud or GPU?"
It's "which architecture fits the workload?"

[alsante @alsante111]

Comparing local AI hardware costs directly with API token prices misses most of the TCO.

Silicon is Day Zero CAPEX. API tokens are variable OPEX.

The real equation also includes utilization, data transfer, migration costs, maintenance and compliance.

Let's look at the actual trade-offs. 🧵

[alsante @alsante111]

Then there's the variable that's hardest to put on a spreadsheet: compliance.

For some enterprises, the goal of local inference isn't saving money on API calls.

It's keeping sensitive data and inference inside a security boundary the company actually controls.

That's a different kind of ROI.