Aicomputeradar

Which open AI model runs on which GPU, and what renting that GPU costs.

[Aicomputeradar @aicomputeradar]

Pick of the week: @IBMResearch's Granite 4.2 3B

The Q4_K_M file is 2.1 GB. At 8K it needs 3.6 GB and runs easily on every machine we track 8 GB cards included. At 32K it needs 6.3 GB and still fits those same cards.

What's the smallest card you're running a local model on?

[Aicomputeradar @aicomputeradar]

We put a graphics card under an X-ray.🩻

Pick an open model and a card, drag the context from 2K to 128K and watch the file load the cache grow and the moment it spills into system RAM.

49 models, 24 cards, every number from the fit table

https://t.co/octEq7AuLj

[Aicomputeradar @aicomputeradar]

@ivanfioravanti 512GB nodes on demand is heavy. still a 2node job for the big stuff though deepseek v4 pro is 791 GB of weights at q4 alone. curious what they charge per hour

[Aicomputeradar @aicomputeradar]

@TheAhmadOsman great pic and they do ship them nemotron nano 9b is only 6 GB at q4. hope nvidia keeps that coming🙏

[Aicomputeradar @aicomputeradar]

@jundotkim this is such a clever split a mac that can hold the model is doing a lot of work there though glm-5.3 flash alone is 186 GB at q4. what's the smallest mac you'd actually pair with a spark?🤔

[Aicomputeradar @aicomputeradar]

@RShmider sounds like a plan 🙌 drop the numbers here when you've got them really curious where the R9700 taps out

[Aicomputeradar @aicomputeradar]

Every model on the 64 GB Mac mini, file vs cache vs runtime: https://t.co/M4sdtgL8Va

[Aicomputeradar @aicomputeradar]

Llama 3.3 70B fits a 64 GB Mac mini at 8K context, not at 32K (44.8 GB usable). Q4 total, 8K / 32K:

Qwen3.8 27B: 17.1 / 19.4 GB
Qwen3-Coder 30B: 19.4 / 22.4 GB
Gemma 4 31B: 19.9 / 22.5 GB
Llama 3.3 70B: 44.1 / 52.4 GB
gpt-oss 120B: 61.4 / 63.0 GB

How much context do you run?

[Aicomputeradar @aicomputeradar]

@RShmider if it OOMs somewhere past 128K flipping the kv cache to q8 is usually the first easy win. would love to hear where it tops out💥

[Aicomputeradar @aicomputeradar]

@0xBakeer on the rental side yes a bit. 4090s on Clore came in about 12% cheaper this past week than the week before you seeing it on buy prices too?

[Aicomputeradar @aicomputeradar]

@ItsmeAjayKV the 3090 dilemma is real 😂. funny part is renting one runs about 22¢/hr on RunPod right now so the tokens could burn in the cloud while yours goes back to gaming 🎮

[Aicomputeradar @aicomputeradar]

@superalesha nice that you actually measured it instead of repeating the hot experts line. something like deepseek v4 flash is 144 GB at Q4 so on 4x 3090 offloading isn't even optional. which 2 models had no hot experts?

[Aicomputeradar @aicomputeradar]

@RShmider great 32 GB on the R9700 gives you real room. fwiw Q4 27B at 128K already lands around 28 GB all in so 256K will be a squeeze. are you quantizing the kv cache too or keeping it f16?

[Aicomputeradar @aicomputeradar]

@RShmider makes sense tighter prompts beat letting it ramble. 256K is a fun jump though at 128K the cache alone is already 8 GB on the 27B so it gets chunky fast. what card are you running it on?🤔

[Aicomputeradar @aicomputeradar]

@Rod69420lol haha true the 24 gb crowd is always just a little short 😅 what are you using, a 5090 or something even bigger?

[Aicomputeradar @aicomputeradar]

24 GB or 32 GB? What the extra 8 GB buys you (Q4 GGUF, 32K context):

Qwen3.8 27B: 19.4 GB, fits both
Qwen3-Coder 30B: 22.4 GB, 32 GB only
Gemma 4 31B: 22.5 GB, 32 GB only
AgentWorld 35B: 23.4 GB, 32 GB only

Rent from $0.19/h (3090) or $0.65/h (5090). Which tier are you on?

[Aicomputeradar @aicomputeradar]

Every model on a 24 GB card, file vs cache vs runtime, 8K to 128K: https://t.co/zNVmegCumT

[Aicomputeradar @aicomputeradar]

@OwnYourCompute yes the hours a day thing is the whole game honestly at 12h it's a totally different math. and fair point on power that's why the rent vs buy page adds electricity per hour on top

[Aicomputeradar @aicomputeradar]

$1,999 bought an RTX 5090 at launch. At today's $0.65/h rental median, that's 3067 rented hours.

About 2 years at 4 hours a day.

Rented hours to match the launch price:
RTX 3090: 5,575
RTX 4090: 2,993
RTX 5090: 3,067

How many hours a day would yours actually run?

[Aicomputeradar @aicomputeradar]

@tudoriustin_ perfect timing then 😄 good luck with bluebird, ping us when there's something to try