can you share this benchmark methodology?
i've been using @droid as my harness since mid 2025, try everything out there, opencode, omp, amp, etc etc, but nothing else has being better on my experiences
even tho, I'm always open to new tools, and you get me in this Command Code showcase with v4.1 flash thats currently my workhorse alongside opus 5.5.
i would like to include droid in this comparisson versus command code harness
After trialing several offerings, Factory was able to meet the comprehensive large scale enterprise requirements of Blackstone: deployment flexibility with no limit on model choice or engineering workflows.
I asked Gemini 4 Argon Medium to build an 4v4 combat game as COD
No assets were used, everything you see here is made with code
Available for Mobile & PC both
Gemini 4 Argon is CRAZY. ๐คฏ
It built the โSeven Wonders of the Worldโ in just 10 minutes.
Clean, consistent, impressive. ๐ฅ
Available in @droid at 50% off. ๐
Polishing that @Google Map BillBoard graphic display that @FactoryAIโs @samwilhoit_ created for the epic Billboard demo ๐ฅ
@bot driving screen caps
Droid doing @droid things
Day 16 on the @droid Max plan ($200)
Wanted to know if GLM 5.3 Flash and Nemotron 3.5 Super are actually worth using in Droid, since they're both super cheap in the picker (0.06x and 0.08x). So I benchmarked them.
23 Python tasks from @VulcanBench, 3 runs each, and hidden tests decide if a task counts as solved. Ran DeepSeek V4.1 Flash and GPT-6.1 Sol next to them so I had something to compare against. 690 runs total and my laptop lowkey almost died doing it ๐
Sol is on top at 87%, but it costs 26k to 46k credits a task.
The surprise was GLM 5.3 Flash on low. 80.6% solved at about 5.7k credits a task. That's half of what DeepSeek low costs and about a fifth of Sol low. Easily the best value in Droid right now.
DeepSeek V4.1 Flash on high is still the sweet spot if you care about getting it right. 86.8% for about 14k credits, so almost Sol level for around half the price, and it's fast too.
Nemotron 3.5 Super was the disappointing one. Same score as GLM on high but almost 3x the credits. Honestly couldn't find a job where it wins.
So this is how I'm splitting it now. GLM 5.3 Flash low for exploration, mechanical edits and anything high volume. DeepSeek V4.1 Flash high as my default coding worker. Sol only for the hard stuff where getting it right matters more than credits. GLM max barely helped (82.3% for almost 3x the credits of low) so I'm skipping it.
It's one Python suite, so take it with a grain of salt.