Sebastian Sharp

[Sebastian Sharp @666Sebo]

@unbug Doesn´t the conversation cache free up (valuable) vram?

[Sebastian Sharp @666Sebo]

@0xBakeer Right now I´m testing Qwen FN. Seems promising. Quality propably on par or better then 27B, but 2.5-3x faster on Strata. 27B was awesome, but a bit slow on my setup.

[Sebastian Sharp @666Sebo]

@unbug I´m just getting jealous 🥲😅

[Sebastian Sharp @666Sebo]

@Biggest Great feedback, thanks for your thorough answer!

[Sebastian Sharp @666Sebo]

@Biggest 3.8 FN and this look pretty close, at least from the benchmarks. If you like 3.8 FN, what would be the biggest reason the switch?

[Sebastian Sharp @666Sebo]

@unbug Unfortunately rank 1 🇩🇪...

[Sebastian Sharp @666Sebo]

@superalesha Does it run on AMD? Looks like Nvidia only right now.

[Sebastian Sharp @666Sebo]

@Ranas_Genoa Why model are you running (quant)?

[Sebastian Sharp @666Sebo]

@sudoingX The 0bserverx RVN gsq-rco iq3_xxs version is great!

[Sebastian Sharp @666Sebo]

@seanhighness Depends completely on the size. 27b could be way more approachable for many, especially if flash increases in size by factor 3-4x.

[Sebastian Sharp @666Sebo]

@coldniko With how much ram and which model? I have a laptop with 7840 hs and iGPU 780M, but "only" 32gb ram.

[Sebastian Sharp @666Sebo]

@coldniko Hermes

[Sebastian Sharp @666Sebo]

@superalesha Would be interesting to see the performance on the different KV cache quant levels.

[Sebastian Sharp @666Sebo]

@0xKervin @0xSero Should work fine too, I'm running the same card with Q3_K_XL with MTP at 128k Q8 KV cache.

[Sebastian Sharp @666Sebo]

@0xSero How do you handle video editing with it, how does the workflow look like?

[Sebastian Sharp @666Sebo]

@DogukanUrker @elderorb Interesting experience. I've tested it in Hermes agent and gave up after a while, since it hunted the smallest traces and wound down the rabbit hole too often. Maybe with clear boundaries it's better.

[Sebastian Sharp @666Sebo]

@BkashJosi I highly doubt that you want the compression to trigger earlier. It would clearly depend on your usage patterns and average tokens per request.

[Sebastian Sharp @666Sebo]

@ItsmeAjayKV @Eschalabs Would that be a better choice for regular folks, since it runs on mainline llama? https://t.co/fafSF7BtEk

[Sebastian Sharp @666Sebo]

@superalesha Do you encounter any meaningful differences between KV Cache Q4 and Q8 at that context size?

[Sebastian Sharp @666Sebo]

@superalesha Hahaha, I did it multiple time. Sure, I know it´s not a human and it will give me some (useful) explanation why it does what it does... but sometimes it can drive you crazy... :D