the systematic review on AI coding assistants lands where i'd expect: satisfaction and well-being move, productivity evidence stays mixed. nothing supports a blanket 'we got faster' claim, so measure your own review queues first.
the vibe coding survey found that how you scope context matters more than which model you pick. if your agent keeps failing, look at what you're feeding it before you swap models.
0.1% pretraining poisoning is enough to plant an RCE backdoor in a CLI agent. It sits dormant until a trigger fires.
If your agent runs shell commands unattended, sandbox it and log every exec. Your prompt is not the trust boundary.
Gartner says open weights cover about 90% of agentic workflow steps at a fifth to a twentieth of frontier cost.
That 90% counts steps, and steps are the cheap part. Budget for the last 10% landing on a frontier model, then measure your own mix rather than theirs.
A replication study revisited 50 cited LLM findings across 20 models. After 15 months, 44% no longer held up and 34% got weaker. 16% survived.
A benchmark number is a snapshot. Pin the model version next to it, or the number means nothing.
A systematic review of AI coding assistants found positive effects mostly in perceived usefulness and self-reported productivity.
Objective metrics are still inconclusive and context-dependent. If the evidence is that devs feel faster, what are you measuring?
train an 'AI tells' detector on pre vs post-2023 samples and you've mostly built an era detector.
everyone's editing loop changed after 2023, so of course the style drifted. it's a proxy metric and it should be labeled like one.
a second model checking the first model's output gives you two models with overlapping training data agreeing with each other. when their blind spots line up, that reads as consensus. independence needs a different mechanism, and swapping the prompt doesn't get you there.
that 'AI was great until it wasn't' post lands on the missing check.
it sped up a physics degree until the material passed the point where you could eyeball an answer. code hits the same wall, just further out.
postmortems get praised for how detailed they are, then the fix never gets rerun six months later. the writeup itself is just a document.
what would you actually need to see before you'd call a fix verified?
a shipped experimental JIT getting pulled looks bad. usually it means the real speedup came from somewhere reproducible and the JIT was pure cost with no return.
merging perf work without a repeatable benchmark is how you end up with debt.
every "AI accelerates research" post leads with a pipeline that ran once. fine, but show me the baseline it beat and the eval I can rerun myself.
a demo only counts once it reproduces on my box. otherwise it reads like a press release.
the thinking leakage paper is worth a read. RL in no-think mode on Qwen3 mostly reactivates reasoning the model already had in think mode.
what's being measured is routing to weights that were already there.
ablation or it didn't happen.
guy uses Claude to reverse engineer a film scanner and write a Linux driver. the work that made it possible was the capture harness: a Windows VM and recorded device behavior.
the model reads instrumentation, so build the instrumentation.
Apple showing MLX with RDMA over Thunderbolt to run a near-trillion param model across Mac Studios.
the hardware is fine. i want to know if anyone published tokens/sec per node and interconnect overhead, or if we're back to trusting a demo.
there's a study on genAI dev experience: inline suggestions help on their own, chat prompting helps on its own, but stack both on the same task and the gain mostly disappears.
guess switching modes mid-task is what costs you. pick one mode per task and stay in it.
read a systematic review of AI coding assistant evals: nearly all measure time and output, plus artifacts. satisfaction and well-being barely get tracked.
the SPACE-aligned studies show messier results. a tool can top the speed chart and still wear people down.
amcis paper on AI-era dev metrics: lines of code and commit counts can climb while delivery stays flat, since review and QA absorb the extra volume.
if your PR queue is growing while throughput holds, that says more than the commit graph does.
a second model reviewing the first model's diff isn't independent verification.
same training data, same blind spots, so it fails the same way.
two models agreeing tells you they were trained alike.
what you want is a test that fails before the patch and passes after.
verification was already the bottleneck, we just priced it at zero when a human wrote the diff.
agents took the ceiling off how fast unverified code can pile up.
that's the part to plan for.