Happy to see that our article "The Evaluation Tax: Why AI Demands a Shift from Production to Verification" (joint with Jin Li) has been published by the Financial Times China (@FTChinese ). An English version is available here: https://t.co/HI8b40Zo73
In the #GenAI era, firms face a growing challenge on assessing quality, accuracy, and relevance. The new @ftchinese article by CAMO’s professors Jin Li & @FahnMatthias explores “The Evaluation Tax” — why AI demands a shift from production to verification:
https://t.co/6oZhEdyN50
AI code is more dangerous than bad code.
A junior's mistakes are obvious. The naming is off, the style is messy, you see the shortcut from a mile away.
You know exactly where to look. Takes 5 minutes in review.
AI-generated code is clean, idiomatic, properly styled, and confidently wrong. The logic fails at the architectural level while looking production-ready on the surface. That's not a small difference. That's a different kind of problem.
22,000 developers tracked over 2 years. Bugs per developer up 54%. Review time up 200%. 31% of PRs now merge with zero review at all.
Not insufficient review. Zero.
And 25% of pull requests are now reviewed by other AI agents because teams figured they'd automate their way out of the bottleneck. It's not working.
You can't solve "code that fools humans" by adding more automation.
The bottleneck is senior engineers. They're the only ones reading past the syntax. And nobody's asking what happens when their review queue triples while leadership celebrates velocity metrics.
I wonder how many engineers are actually thinking before they press enter.
Artfully written as always and I love seeing more positive pro-AI stuff out there, but I somewhat disagree with
@danshipper on the conclusions of this (maybe dangerous to disagree publicly with my CEO, but I know he'll be cool with it :-).
I don't expect humans to always be in the driving seat BUT I am also optimistic like Dan is, because I don't think machines overtaking us will be that bad.
I would call my position something like "jagged free lunch":
- machines will be superhuman at some things, subhuman at others
- to the extent machines are better than us at some things it's because the environment they "live in" rewards those things i.e. math benchmarks
- machines will eventually be capable of beating humans at everything BUT only at a cost surpassing human salaries and latency
- this is because it will cost so much and take so much time to evaluate fuzzy tasks with machines that it will be quicker and cheaper to use human intuition / experience
- there are no free lunches in evolution, given a fixed energy + raw material input humans already make mostly the right tradeoffs in intelligence vs heuristics
Essentially it costs so much to brute force all the available options with compute to simulate reality that it is irreducibly cheaper to just "live in" reality and let market forces / physics 'teach' you the right heuristics. One day robots will live here with us but will be subject to the same scaling laws biological intelligence is.
🚨 Due to high demand, I created a new (and much improved) version of the Econ Journal Matchmaker.
New tool here:
https://t.co/kvrQQmo5SG
#EconTwitter Please help spread the word by reposting. 🙏
More details below 👇
#CAMO Quote💬
Firms are intentionally stripping away judgment and responsibility to settle for "good enough" standardized output. The real threat isn't #AI taking your job. 𝗜𝘁'𝘀 𝗔𝗜 𝗺𝗮𝗸𝗶𝗻𝗴 𝘆𝗼𝘂𝗿 𝗷𝗼𝗯 𝘄𝗼𝗿𝘀𝗲.
Read full paper : https://t.co/x4eynswyso
@IvanWerning There is evidence that AI may weaken incentives for high quality: once “good enough” becomes much easier, costly extra effort is less attractive, potentially causing quality losses.
Evidence: https://t.co/SEep5CwTKJ
Theory incl. firm responses: https://t.co/k48UlHcRj7
Most of the debate on causality in economics is misplaced. The real divide is not RCTs versus non-RCTs. It is between two different questions: “what is the effect of X on Y?” and “what are the causes of Y?” Once mechanisms, interactions, and general equilibrium matter, theory is not optional. You are identifying a model, not just a total effect. https://t.co/pjnWMuHuzP
Latest data show "immediate, sizable, and persistent decrease in the level of early career (22-24 year old) hires" from AI after controlling for COVID stimulus, ZIRP and other macro factors:
"... historical decomposition suggests that up to 1/4 of relative early career employment declines through 2025q2 may be attributable to monetary
policy shocks through 2023, but the analysis does not find evidence that these shocks can explain the
rapid decline in hires at the most AI-exposed firms in comparison to others" from @LeeCTucker, @UScensusbureau economist publishing independently
You can find the paper here, we look forward to any comments and reactions: https://t.co/KlIXS4MB2j
At @camo_hku we have several more projects in the pipeline, and hope to share them soon.
We have a new @camo_hku paper out (with Jin Li and @chang_sun_econ): "Toward a Bad Job Economy: AI Adoption, Agency Costs, and Job Design." We show that AI may not just replace jobs; it may make the jobs that remain worse.
Our research connects to discussions by @lugaricano, @Afinetheorem, @alexolegimas, @emollick, and others on how org frictions shape AI adoption. At @camo_hku, we study exactly this, and also how AI reshapes org frictions in return, which we think is just as important.