dotech

Founder of Agentic Labs

[dotech @dotech]

@Manisha27493225 I’m building a custom long-context eval based on NVIDIA RULER and NoLiMa-style tasks: multi-key retrieval, variable tracking, and semantic retrieval. More questions improve reliability, but on a single RTX 5090, thorough runs take too long. I’m balancing both.

[dotech @dotech]

RTX 5090 Local LLM Benchmark — Part 2

My previous RTX 5090 benchmark raised three questions:
• How much context can a single RTX 5090 actually handle?
• How do inference speed and GPU memory usage change as context grows?
• How does accuracy hold up at longer context lengths?
I built a custom benchmark framework and tested three models. Accuracy follows in Part 3.

Models tested (in order of release)
① Qwen3.8-27B | @Alibaba_Qwen (Alibaba) | Aug 14, 2026
27B Dense · Gated DeltaNet (linear) + Full Attention
Coding, reasoning, agents, and image/video understanding.
② Ornith 1.5 35B-A3B | @ornith_ (DeepReinforce) | Aug 19, 2026
35B MoE (~3B active) · Linear + Full Attention
Trained with a self-improving RL loop.
③ Flint 39.8B-A2.5B | @Recursive_SI | Oct 21, 2026
39.8B MoE (~2.5B active) · Sliding Window + Full Attention
Designed for local coding agents.
All three have native 256K context windows.

1. Maximum Runnable Context
• Qwen: 224K passed → 256K OOM
• Ornith: 544K passed → 576K OOM
• Flint: 256K passed → 288K failed (likely OOM)
“Passed” means initialization, prefill, and decode completed without errors—not verified accuracy. After 8K/16K/32K, context increased in 32K steps.

2. Inference Speed & GPU Memory
See the attached graphs for decode speed, GPU memory, and estimated TTFT, plus a 192K snapshot of decode, cold/incremental prefill, and TTFT.

3. How Did Ornith Reach 544K?
Key factors: GPU weight memory and KV-cache growth.
Full Attention stores per-token KV; Linear Attention uses fixed-size states; Sliding Window Attention uses a bounded window.

Full Attention layers / KV heads / Head dimension
• Qwen: 16 / 4 / 256
• Ornith: 10 / 2 / 256
• Flint: 14 / 4 / 128

Model order: Qwen / Ornith / Flint
• FP16 KV growth (KiB/token): 64 / 20 / 28
• KV cache at 192K (GiB): 12.15 / 3.81 / 5.34
• GPU weight memory (GiB): 14.73 / 19.44 / 22.34

Qwen has the smallest weights, but its 64 KiB/token KV growth fills the remaining VRAM fastest.
Compared with Flint, Ornith uses 2.90 GiB less GPU weight memory and about 29% less KV cache per token.
At 544K, Ornith used 10.69 GiB of KV cache and 31.30 GiB of baseline-adjusted GPU memory, with decode at 98.96 tok/s.

4. Runnable Context ≠ Reliable Context

YaRN (RoPE scaling) was not explicitly enabled for Ornith. Completing inference beyond native 256K does not establish reliable retrieval or reasoning at 544K.

Test setup

RTX 5090 32GB · llama.cpp b12017 · Q4_K_M · FP16 KV cache · 1 warm-up + 3 measured runs
GPU memory = NVML peak minus baseline. Results are runtime-specific, not quality rankings.

Part 3 — Long-Context Accuracy
I’ll evaluate accuracy using selected NVIDIA RULER tasks and a custom NoLiMa-inspired test. I’ll also compare FP16 and q8_0 (8-bit quantized) KV caches for memory efficiency, maximum runnable context, and accuracy changes.

Results will follow in stages.

[dotech @dotech]

@yunianlinthomas I’m preparing to include it in the next round, but the test takes a very long time to run, so I’m still weighing it.

[dotech @dotech]

@luna_fislab Thanks! As noted in the post, q8_0 KV-cache testing is planned for the next round. I’ll compare it with f16 at the same context lengths for VRAM, speed, and accuracy. I’d like to share the results, but the tests take so long that I’m still figuring out how to manage the time.

[dotech @dotech]

@Joshua_WD Thanks! Speed and VRAM are only part of the picture. I'm building a custom long-context accuracy benchmark using NVIDIA RULER and NoLiMa-inspired tasks to test reliability as context grows. I'll share the results when the evaluation is complete.

[dotech @dotech]

I passed Nebius Academy's Tavily Agentic Web Search Certification!

As soon as the exam became available through @nebiusai, I took it without any preparation and passed on my first attempt.

The certification from @Nebius_Academy focuses on connecting AI agents and RAG pipelines to real-time web information using @tavilyai.

According to the official exam guide, the four domains are:
• Foundations of Agentic Web Search (~15%) — Source relevance, recency, credibility, tool selection, and blended RAG.
• Tavily Search Capabilities (~35%) — Search parameters, extraction, response handling, and failed-extraction recovery.
• Tavily Agent Integrations (~25%) — Crawl, Map, MCP, endpoint selection, and error handling.
• Production Operations & Reliability (~25%) — API authentication, rate limits, credits, cost forecasting, key rotation, and secret handling.

The focus is on how AI agents discover, retrieve, and reason over live web content—not training or hosting the underlying LLM.

Verified Credential:
https://t.co/QQeFcLZaR6

[dotech @dotech]

@LuoluoAIcoder @nebiustf Exactly. For models with discounted cached-input pricing, the lower rate applies only to tokens served from cache.
Comparing cold-cache and cache-hit runs, along with the reported token usage and actual billing, is the right way to verify the cost savings.

[dotech @dotech]

Nebius Token Factory Finally Supports Prompt Caching!

@nebiustf consistently ranked among the fastest AI inference providers on Artificial Analysis.

But the lack of discounted prompt caching made repeated-context agentic workflows significantly more expensive than competing services.

That was one of my biggest concerns about Token Factory.

Now, cached tokens are supported, with usage visible in API responses and billing, along with model-specific pricing.

I've been waiting for this improvement, and I'm genuinely happy to see it arrive.

@nebiusai already had the speed. Now it's improving cost efficiency too.

[Nebius Token Factory @nebiustf]

Cached tokens are now live on Nebius Token Factory⚡

Reusing context across requests?
Cached input can help reduce costs. See cached-token usage in API responses and billing, with pricing per model.

More info: https://t.co/iYOCbZqLM2

[dotech @dotech]

Recursive Flint: A New Local AI Model for RTX 5090

Recursive (@Recursive_SI) has released Flint, a 39.8B Mixture-of-Experts (MoE) model designed for coding agents, activating only ~2.5B parameters per token.

Key Features
• 262K-token context window
• Reasoning and multi-turn tool use
• Apache 2.0 license
• Q4_K_M GGUF: 24.17GB

Advantages
• High token generation speed potential through its sparse MoE architecture
• Optimized for coding agents and tool use
• Runs locally on an NVIDIA (@NVIDIAAI) RTX 5090 with 32GB VRAM using 4-bit quantization

Limitations
• Qwen3.8-27B scores higher on several published coding benchmarks.
• Flint is text-only, while Qwen3.8 supports multimodal input.
• Reasoning accuracy and tool-calling reliability need further testing.

Local LLM Token Generation Speed Test on RTX 5090 (32GB VRAM)
I compared four quantized AI models on my NVIDIA RTX 5090 using the same prompt and test conditions, with the task involving the development of a C#/.NET application.

Results (Average of 5 Runs)
• Flint 39.8B (Q4_K_M, 24.17GB): 183 tokens/sec
• Qwen3.8-27B (Q4_K_M, 17.74GB): 79 tokens/sec
• Nemotron 3 Nano Omni 30B-A3B (Q4_K_M, 26.1GB): 178 tokens/sec
• Muse Glimmer 30B (Q4_K_M, 17GB): 80 tokens/sec

These results reflect token generation throughput under my test conditions, not overall coding capability or output quality.

[dotech @dotech]

MiniMax-M3.1-Flash (@MiniMax_AI) is now available on Nebius Token Factory!

According to Artificial Analysis:
• Intelligence Index: 43 (#5/174)
• AutomationBench-AA: 73%
• Context Window: 1M tokens

Nebius Token Factory (@nebiustf) offers:
• Input: $0.15 / 1M tokens
• Output: $0.60 / 1M tokens
• Cached Input: $0.015 / 1M tokens (90% off)
• Listed speed: 181.46 tokens/s

With strong agentic capabilities, a massive context window, and affordable prompt caching, this looks like a compelling option for coding agents and complex automation workflows.

Benchmark:
https://t.co/1I6A4lKvXT

[dotech @dotech]

DeepSeek V4.1 Flash 1017 is now available on Nebius Token Factory.

DeepSeek's latest open-weight model delivers significant improvements in reasoning, coding, and agentic AI workflows.

Speed: 738 tokens/sec (as listed by Nebius)

Pricing per 1M tokens:
Input: $0.30
Cached input: $0.006
Output: $1.20

Artificial Analysis Intelligence Index (Max reasoning):
DeepSeek V4.1 Flash 1017: 45
GLM-5.3: 45
GLM-5.3-Flash: 42
Previous DeepSeek V4.1 Flash: 39

The new DeepSeek model improves by 6 points over its predecessor and matches GLM-5.3 in overall intelligence, while offering significantly lower API pricing than GLM-5.3.

It also supports a 1M-token context window and multimodal input.

With strong performance, competitive pricing, and high throughput, DeepSeek V4.1 Flash 1017 is a compelling option for AI agents and production applications.

#Nebius #TokenFactory #DeepSeek #OpenWeight #AgenticAI

[dotech @dotech]

Today, I earned the Nebius Physical AI Certification from Nebius Academy, taking and passing the exam shortly after it became available. https://t.co/IPdnoZxGvT 출처 @credly

[dotech @dotech]

View my verified achievement from Nebius Academy. https://t.co/gqTZ9eYa7D 출처 @credly

[dotech @dotech]

View my verified achievement from Nebius Academy. https://t.co/t2NiNplmuF 출처 @credly

[dotech @dotech]

View my verified achievement from Nebius Academy. https://t.co/dhfUi9mWUD 출처 @credly