We launched Jarvislabs Token Factory two months ago, running on GPUs in India, and have been heads down improving it ever since.
We benchmarked the same three open-weight models on Sarvam's Token Factory and ours.
We're on par with or faster than Sarvam across speed and latency metrics, while offering lower pricing.
- GLM 5.3 Flash is almost 2x faster.
- Gemma 4 31B and DeepSeek V4.1 Flash are slightly faster, with significantly lower pricing for input, cached, and output tokens.
And we're running on H200s, not B200s btw.
Qwen3.8-27B Goes From 82 to 381 Tokens/s on a Single RTX 3090
Qwen3.8-27B reportedly reached 381 tokens/s on a single RTX 3090 with 24GB VRAM.
That’s roughly 4.6 times its original throughput, achieved without upgrading the GPU or changing the model.
Developer TeksEdge achieved this through a series of inference optimisations involving Multi-Token Prediction (MTP), DFlash2, context lookup, prefix caching and faster verification.
The interesting part is that the speedup didn’t come from one magical optimisation. It came from addressing several bottlenecks in the generation pipeline.
How the Speed Improved
TeksEdge reports the following progression:
Standard decoding: 82 tokens/s
Optimised MTP: 114 tokens/s
DFlash2 + context lookup: 138 tokens/s
Optimised lookup verification: 381 tokens/s
The final jump is particularly interesting. It suggests that a significant amount of inference time can be recovered when the system reuses information and verifies multiple proposed tokens efficiently.
Here’s how the individual techniques work.
1. Speculative Decoding Reduces Sequential Generation
Most language models generate text autoregressively. They predict one token, feed it back into the model and then predict the next.
This creates a sequential bottleneck because each generation step depends on the previous one.
Speculative decoding attempts to reduce that overhead.
A smaller drafting process proposes several future tokens, while the main model checks whether those predictions are correct.
When multiple proposed tokens are accepted, the target model can advance several positions through a verification cycle rather than performing a separate generation step for every token.
In this experiment, optimised MTP increased throughput from 82 to 114 tokens/s. Combining DFlash2 with context lookup pushed it further to 138 tokens/s.
The acceptance rate is important here. More proposed tokens don’t guarantee more generated tokens. The target model must still verify them, and rejected proposals reduce the speed advantage.
2. Context Lookup Reuses Text Already Available
This is one of the most interesting parts of the experiment.
When a model is asked to reproduce or transform information already contained in its prompt, generating every token from scratch can be wasteful.
Context lookup searches the existing input for matching sequences and proposes those tokens for verification.
Instead of predicting an entire passage again, the inference engine can reuse matching text and ask the target model to verify it.
TeksEdge reports lookup blocks of up to 16 tokens, with approximately 15 accepted tokens per block in its workflow.
In a 25,000-token document reproduction test, throughput reportedly increased from around 260 to 382 tokens/s
This approach is particularly useful for:
Reproducing long documents
Restructuring existing text
Copying or transforming code
Reusing information already present in the prompt
However, it works best when the output overlaps significantly with the input. An open-ended conversation or a task requiring entirely new information offers fewer opportunities for context reuse
3. Prefix Caching Eliminates Repeated Computation
Coding agents often send repeated instructions, tool definitions and repository context across multiple requests
Without caching, the inference engine may repeatedly process large sections of an unchanged prompt
Prefix caching addresses this by storing computations associated with previously processed prompt prefixes
When a new request contains the same prefix, the engine can reuse the cached computation instead of starting from scratch
TeksEdge reports reducing time to first token from 22.4 seconds to just 0.56 seconds
That’s approximately a 40-fold reduction in initial waiting time
For coding agents that repeatedly inspect files, call tools and submit follow-up requests, this can make a substantial difference
381 Tokens/s Isn’t Ordinary Chat Speed
We've raised $1.9m in preseed to scale sovereign and local AI at JohnAIC! We intend to use the funding to secure GPU and supply chain and to build amazing applications to run out of box on JohnAIC.
Really greateful to @smdcmc (@crane_vc) and @siddharthaa7 (@neon_fund) for backing us!
📢 Meet Qwen4-Max, your new AI teammate for work, creativity, and discovery!
Qwen4-Max is built on a new MoE architecture with native vision and a 1M-token context window.
Highlights:
- Outperforms Qwen3.8-Max with up to ~50% fewer tokens, while average cost per task drops over 60% in our internal evaluation.
- Advances real-world productivity through greater proficiency with professional tools, stronger long-horizon execution, and higher-quality deliverables.
- Handles real tasks, including 3D modeling, game development, manufacturing, finance, law, healthcare, and frontier scientific discovery.
- $1.5/M tokens for input, $4.5/M tokens for output, and $0.075/M tokens for implicit caching — all lower than Qwen3.8-Max, with implicit caching down by 70%.
The API is live on QwenCloud. You can also try Qwen4-Max on Qwen Studio, QwenWork and Qoder.
Start building today! 🚀
- Blog: https://t.co/AuiWxugjE6
- API: https://t.co/aS1QEkS8nu
- Token Plan: https://t.co/YRvcGdB9Bv
More to come! The open weights of smaller Qwen4 models will be released. Stay tuned. 👀
We're open-sourcing our latest versions of Decision Models: Decider v1.3 and Decider Flash v1.3, and making them free on @OpenRouter for a week.
The new capability in v1.3 models is "locate". This lets you describe what to find, and the model returns the matching text with character offsets.
This is useful for batch-mode decision + evidence/entity extraction. On 17 out-of-domain NER, PII and extractive QA tasks, Decider models significantly outperform the best general extraction model.
And, on 63 of our own production classification tasks, v1.3 is 6.54 points more accurate than the classifiers we run today.
Weights are open on Hugging Face: https://t.co/6olEOrOq4y
I was curious about the current state of India's AI inference ecosystem, so I decided to look into what local providers are offering.
Compared pricing across Indian inference providers for four models: DeepSeek V4.1 Flash, GLM 5.3 Flash, Kimi K3 and GLM 5.3.
The same model can have quite different pricing depending on the provider.
Also built a directory where you can discover Indian inference providers, explore their model offerings and compare pricing.
site: https://t.co/Y7ps6xdjIW
After Google acquired my AR startup, I started thinking about the next interface, this time for AI. Text and voice work fine, just like DOS did before the Mac. But nothing’s more intuitive than face to face.
After two years of research, today we’re launching Atmee v2: a real-time avatar model that makes live video agents feel truly life-like.
• Any character from 1 image + 10s of audio
• Contextual expressions, gestures, even interacting with the environment
• Live video calls, voice only, or video generation
At its core, Atmee v2 is a video foundation model, it can render anything, but it’s built to excel at characters, which is what video models like Seedance 2.5 and Wan 3.0 are mostly used for. In comparison:
Atmee v2
• Up to 50x faster
• 10-100x cheaper
Build tutors, sales coaches, support agents, companions, game characters, even whole living worlds. The interface after text and voice starts today.
Crazy!!
A senior official at a leading beverage company in India, one of the nearly 30 affected investors, told Moneycontrol that 10 of his major US stock holdings — including Apple, Tesla, Amazon, Alphabet, Nvidia and AMD — were liquidated without his authorisation for a total of $335,768.
The entire sale proceeds were then redirected into a single unauthorised order to buy 300,000 shares of Chinese stock Zhibao Technology Inc. (ZBAO). The purchase was executed at an average price of $1.05 per share. Over the following days, ZBAO's stock price fell nearly 60%, closing at $0.42 on October 23.
Important break by Priyansh and @shwwetapunj
A ‘hello world’ milestone for Project Suncatcher 🚀
After launching our prototype satellite a few weeks ago, we’ve now successfully sent a command up to orbit, booted up the TPU, ran the prompt, and received our first complete response back on earth - a haiku!
So one of our ambassadors @divagr1925 has been working on something cool
He found a way to use Qwen's Gated DeltaNet state as memory, and built a 32-bit RISC-V CPU directly into the model's weights. So instead of relying on predicting the answer to a calculation, the model can execute instructions internally. It can also run C and MicroPython programs.
He calls it Qwenputer, love the name btw. The results so far are pretty impressive. On one arithmetic benchmark, accuracy jumped from 48.6% to 100%.
Pretty cool to see developers experimenting with Qwen's architecture. Great work, my friend!
I did say we'd be releasing our models periodically...introducing Gemini 4 Carbon, our next frontier model.
Following Argon's release just two weeks ago, Carbon shows the pace at which we’re pushing the frontier, it ranks #1 on the recently updated v5.0 of @ArtificialAnlys Intelligence Index at 59. It also leads across key cybersecurity evaluations scoring 88.2% on SEC-bench Pro for finding vulnerabilities, 54.8% on ExploitGym for exploitation, and 71.0% on CWE-bench for patching.
We’re seeing significant improvements in long-horizon tasks even compared to Argon, and we’ve strengthened frontier safeguards.
As we did with Argon, we’re going to roll it out gradually + responsibly. Some trusted cyber defenders will get access starting this week. We'll be expanding it to more partners through our Fairwind Program, ahead of wider availability soon, so hang tight!
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: https://t.co/TZSugXi2ZE
My quick takeaways on the Anthropic S-1:
1. This is a historical moment
2. $6.6M in revenue per employee!! Hard to even comprehend how good that is but Anthropic is AI native as it gets so makes sense.
3. I always thought OpenAI would drop their S-1 first
4. Customers spend about 9x more after a year. Net Revenue Retention is approximately 900% as of Sep 30, 2026. Good SaaS companies brag about 120%.
5. Cowork is growing faster than Claude Code did. Goes to show you the demand for consumer products. Duo will probably rip.
6. 3 years ago this company did $41M in revenue. In the first six months of 2026, it did $16.3 billion. I don't think anyone has a mental model for that curve yet, I sure don’t.
7. Nearly half of revenue comes through cloud marketplaces. About 48% comes from customers buying through them. Big companies buy where their budget already lives, and the lesson for founders selling to enterprises is to get listed where your buyers already spend.
8. Their run rate formula is published: the last 28 days of usage times 13, plus subscriptions times 12, which i guess is the new ARR math for founders?
9. It looks like They're merging Cowork into Claude dot au which tells you the chat box era is ending and the "go do the work for me" era is here.
10. Thousands of Anthropic employees are about to become liquid at once, so expect the biggest wave of new AI startups and angel checks San Francisco has ever seen. Reminds me when Facebook went public but this is way bigger.
Let’s go!
11. Buried in the filing, Anthropic says the gap between what Claude can do and how widely it's used is one of its biggest growth opportunities, so the company itself is telling you the models are ahead of the market and someone has to close that gap for dentists, landlords, and plumbers.
12. Whatever happens, exciting times to be building in AI.
1) For closed models, these would be super difficult to price because of limited history(claim severity, and claim occurence). Inference providers should explicitly limit their liability to n times client-specific monthly billing, subject to a cap. 2) In case of open source model inference, the inference provider can easily put a disclaimer that any losses due to use of the open-source model are not their responsibility. In such cases, the availability SLAs can be priced and sold. -- Raj Bhatt
There is an absolute barrage of inference providers popping up that get to $1M/mo extremely quickly.
This is the rough economics of inference providers:
Even if you rent on-demand B300s at $8/GPU-hr, many open source models (GLM 5.3, Kimi K3, DeepSeek 4.1 Flash / Pro) can get you >$10-20/GPU-hr at reasonable throughputs (100tok/s). DeepInfer from DeepSeek directly shows they can get $38/GPU-hr on off-peak hours. Every chip / model has a different throughput / interactivity curve, but there is a lot of arbitrage between even high rental spot prices and token prices. New chips (Vera Rubins) also change this meaningfully. Your ability to be making gross margin relies on your utilization and how good your stack is.
Gateways like OpenRouter, Vercel and more help solve that problem by aggregating demand to you. May inference providers discount their tokens to get longer term commits too. I've even heard stories of selling back Chinese model tokens to Chinese labs!
One thorn in the economics of these players is that many Chinese open source model labs are now asking for rev share if you're at a certain scale, espp for pre-release access to their models for day 1 support. That further pushes the threshold to be in the money.
The barrier to entry seems low and falling. There are even stories of smaller ones who can "vibe code" a way to stand up OSS models on a cluster they rent. And as a result, we're seeing 10s of such companies pop up almost every week and immediately skyrocketing revenue at low-ish gross margin and raising rounds on the back of that.
Unsurprisingly, because the actual service provided seems commodity, this eventually drives up compute prices and drives down token prices. In theory, it would seem like neoclouds who have the lowest cost of compute and themselves have started inference businesses would win the race to the bottom (Crusoe Coreweave Nebius and GMI all have inference businesses), with larger players (Modal Fireworks Baseten) being second because they are well capitalized to make bigger compute commits cheaply and have demand (high utilization), but the reality is the pie is growing so quickly that even upstarts are absolutely crushing it and growing aggressively (inference dot net, DeepInfra, Wafer, Relace, Morph).
Strategies vary across the board: neoclouds want to serve inference, inference players want to be neoclouds and do higher margin reserved instances / post training, and some want to go heterogeneous hardware or creatively flexible compute constructs.
The risks are: do open source models continue to be competitive? Will their creators rent seek more? Will their shape change compute needs? How do different chips change economics? Who gets compute in a supply crunch?
Below is a aggressive sketch of how gross margin scales with utilization for various models / chip sets.
For now, there are many inference providers and they’re all ripping!