Mahan Tafreshipour

🎓 PhD @UCIrvine | 🤖 LLM Research | 🔍 Empirical SE

[Mahan Tafreshipour @mahan_tp]

The system around the model is where reliability lives.

[Mahan Tafreshipour @mahan_tp]

@ayesha_fatiima Flash memory

[Mahan Tafreshipour @mahan_tp]

@solopribuilds I think the term is self-explanatory

[Mahan Tafreshipour @mahan_tp]

@gneubig I really needed this.

[Mahan Tafreshipour @mahan_tp]

@willcb How is it to train harness?

[Mahan Tafreshipour @mahan_tp]

@GarrettCAllen @_ueaj

[Mahan Tafreshipour @mahan_tp]

Models will internalize the harness but it won’t disappear.

Compilers ate register allocation. We still write code.
What stays external: your repo state, incidents, and guarantees the model can’t self-enforce. A model can’t be both the actor and the check on the actor.

[Bojie Li @bojie_li]

Exactly aligned with an excerpt from my open-source book "AI Agents in Depth": https://t.co/6VBSv9Mete

"Will models eventually eat the harness? Yes.

Each time a model reliably internalizes a capability, the corresponding layer of the harness can be removed. Look back at all the fallback logic piled into today’s harnesses: multi-stage context compression, retry loops, and permission checks that pessimistically default to “unsafe.” Every seemingly ugly pile of technical debt marks something the model still cannot do reliably.

When the next generation of models internalizes those constraints, the corresponding code can be deleted. And the reason models can internalize them is precisely that agents have already encountered these failure modes in real-world applications, turning those experiences into training signals for the next iteration.

Users bring real, difficult problems. The application layer uses a harness to compensate for what the model cannot yet do. And those compensations, in turn, become training signals for the model’s next generation."

[Mahan Tafreshipour @mahan_tp]

Claude.md:
@ass.md

[Cole Gottdank @coleywoleyyy]

the most detailed spec I've ever written

[Mahan Tafreshipour @mahan_tp]

@GorryTan I like your sense of humor but you should work on your naming😂

[Mahan Tafreshipour @mahan_tp]

@veebrandt Yes but less efficient.

[Mahan Tafreshipour @mahan_tp]

@JulianGill2 @ashebytes Interesting. You're describing a human-AI interaction problem rather than a model capability problem. AI should be able to infer intent from minimal signals and guide users through the cold-start phase.

[Mahan Tafreshipour @mahan_tp]

What AI can NOT do?

[Mahan Tafreshipour @mahan_tp]

@dggoldst Linear Algebra

[Mahan Tafreshipour @mahan_tp]

@a1zhang @lateinteraction cannot be absorbed by training because they
reflect human or organizational priorities the model has no
way to know."

[Mahan Tafreshipour @mahan_tp]

@a1zhang @lateinteraction "As base models grow stronger through RL training, rules[harnesses] describing how
to code (style, formatting, generic process) are increasingly
internalized into model capabilities, while rules encoding
what not to do (project-specific scope constraints, safety
prohibitions)

[Mahan Tafreshipour @mahan_tp]

@soran0312_ 16

[Mahan Tafreshipour @mahan_tp]

@1owroller This was dope!

[giatt @1owroller]

girlfriend overheard me and started yelling at me asking me “who the fuck is kimmy” “why the fuck are you talking about chinese models”