Alireza Bashiri

harness engineering @

[Alireza Bashiri @al3rez]

No, I am not

ex-Meta
ex-Netflix
ex-Google

or any other ex's.

I was born in Iran, Tehran.
I had no friends because we had to travel every 3 months due to my dad's work.

My unfair advantage?

Sheer will!

I once told my uncle, "I'm going to make $6,000/month" one day, and he laughed at me.

He said, "It's good to have dreams, but they're just dreams."

Now I'm making between $12-$15k/month every month.

I bought a house and an SUV for my mom, helped my brother get into Webflow boot camp, and invested in real estate in multiple countries. I've launched more than 10 businesses and generated over $10m+ dollars for my clients over the past 10 years.

If I can do it, you can do it.

If you believe, you, like me, have 2 hands.

You can do it!

[Alireza Bashiri @al3rez]

@robj3d3 Yeah I literally rented a aeron herman miller to try it out for a few months then I bought Sayl. it's so convinent

https://t.co/uC7PUySu2G

[Alireza Bashiri @al3rez]

Read the original paper 👇
https://t.co/y7f4qYCMNY

[Alireza Bashiri @al3rez]

Long-running coding agents need project state, not just longer conversation history.

A paper called StateHarness explores this problem.

The researchers built a runtime that tracks what the agent is doing across long development tasks.

It records:

• Current problems and hypotheses
• Important findings
• Repository dependencies
• Previous decisions
• Unfinished work

The harness also checks which callers, tests, and documentation may be affected when the agent changes code.

When enough execution evidence accumulates, a separate diagnostic session reviews the history for missed information.

The reported results on RefactorBench:

Full-chain completion: 29.4% → 35.3%

Cleanup accuracy: 87.0% → 95.4%

On LHTB, mean reward improved from 0.427 to 0.576.

The researchers also reported improvements on ChainSWE, although some regressions increased.

I think this is worth studying.

Most coding agents rely heavily on conversation history to remember what happened.

But a long-running task needs more structured information.

What changed?
Why did it change?
What still needs work?
What other files depend on it?

I'd like to see more harnesses maintain this state explicitly instead of repeatedly summarizing the conversation.

[Alireza Bashiri @al3rez]

Read the original paper 👇
https://t.co/L9RYLaLDoy

[Alireza Bashiri @al3rez]

An AI agent benchmark reported 94.3% success.

After researchers fixed its evaluation logic, that dropped to 65.5%.

A paper called "When Agent Evaluation Becomes Unreliable" audited the software responsible for scoring agent tasks.

The researchers discovered a reward-function bug.

The evaluation code changed the environment state before measuring the final result.

That made some failed executions appear successful.

Across 4,320 episodes, correcting the scoring logic reduced measured success from 94.3% to 65.5%.

They also tested how configuration changes affected results.

Tool descriptions, policy documents, and error messages changed observed trajectory reliability.

This is something I think agent benchmark discussions often miss.

The benchmark harness is software too.

It can have bugs.
It can change the environment.
It can reward the wrong behavior.

Before trusting an agent score, I'd want to know:

• Does the evaluator modify task state?
• Can known failures pass?
• Can known successes fail?
• Can the results be reproduced?
• Does the same configuration produce consistent labels?

A benchmark score is only as reliable as the system that measures it.

[Alireza Bashiri @al3rez]

@steveruizok Make it so that it fixes my handwriting and I will use the s*** out of it

[Alireza Bashiri @al3rez]

@imbabybrooklyn Peak

[Alireza Bashiri @al3rez]

@doodlestein why every time you take a screenshot it looks like curved

[Alireza Bashiri @al3rez]

Read the original paper 👇
https://t.co/7sY1bxaNCE

[Alireza Bashiri @al3rez]

Adding more coding agents can make a task 3–7x more expensive.

Researchers studied this in "Scaling Inference-Time Compute with Teams of Coding Agents."

They built a minimal multi-agent harness and tested different team sizes, model combinations, and delegation strategies.

The results were interesting.

Multi-agent teams improved task performance, but cost 3–7x more than a solo agent.

Then the researchers changed where the extra compute went.

Delegating only planning matched the reward of delegating planning, implementation, and verification.

At roughly half the cost.

They also found that cheap models worked well as planners, but using them for implementation reduced reward.

A mixed team of 17 agents came within 1.4 points of a strong five-agent team while costing 70% less.

I think the important lesson is about allocation.

Before adding another coding agent, ask what work it will perform.

Planning?
Implementation?
Verification?

More agents create more coordination and duplicated work.

I'd rather measure the value of each delegation stage than assume every task needs a team.

[Alireza Bashiri @al3rez]

@ky__zo Grok bot bro

[Alireza Bashiri @al3rez]

Read the original paper 👇
https://t.co/2sO7iGlOSh

[Alireza Bashiri @al3rez]

We might be optimizing the wrong thing in coding-agent harnesses.

A paper called TurnCost analyzed 600 real coding-agent trajectories.

The researchers wanted to understand what actually drives inference cost.

Their findings:

98.6% of input tokens were served from prefix cache.

Turn count explained 84% of the variance in log trajectory cost.

Average tool-observation size explained only 4%.

And delegating work to another model increased cost in half the evaluated settings.

This is interesting because a lot of context engineering focuses on compressing tool output.

Shorter logs.
Smaller observations.
Fewer tokens per response.

But the researchers found that most observations were already small.

The expensive trajectories often had more turns, even when their observations were shorter.

I think this changes how we should evaluate coding-agent efficiency.

I'd measure:

• Total model turns
• Repeated tool calls
• Failed repair loops
• Cache hit rate
• Cost per verified solution

The paper doesn't prove that removing turns preserves task success.

But it gives us a good reason to investigate unnecessary agent loops before building another context-compression system.

[Alireza Bashiri @al3rez]

@chrissev @bot @Muse let me in let me in

[Alireza Bashiri @al3rez]

Read the original paper 👇
https://t.co/fbSOfHAQTq

[Alireza Bashiri @al3rez]

Installing more skills can actually make your coding agent worse.

A new paper called "One Skill Too Many" studied what happens when agents have similar or overlapping skills installed.

The researchers analyzed 20,947 repositories and ran 6,368 agent executions across three models.

They found something interesting.

Nearly 1 in 4 installed skills had another skill that performed a similar job.

When a competing skill was available, it displaced the intended skill in about 20% of runs.

And when the agent opened the competing skill first, it lost more than a third of the original skill's exclusive functions.

The task could still pass.

That's the problem.

Imagine installing a skill that prohibits Git operations.

Another skill handles the same task but doesn't enforce that restriction.

The agent completes the work, yet ignores the original constraint.

The researchers also found that installation location influenced which skill ran, while listing order had little effect.

I think skill management needs to become part of harness engineering.

I'd want the harness to:

• Detect overlapping skills
• Resolve conflicts before execution
• Record which skill the agent selected
• Test whether required behaviors survived

More skills don't automatically mean more capability.

[Alireza Bashiri @al3rez]

Read the original paper 👇
https://t.co/hDl7tzMPcS

[Alireza Bashiri @al3rez]

Training a coding model on more agent trajectories doesn't always make it better.

A paper called CoTrace studies what happens when training data comes from different agent harnesses.

The researchers found that combining trajectories from different harnesses can damage agent behavior during SFT and RL.

The problem is that a trajectory contains more than reasoning.

It also contains assumptions about:

• Available tools
• Tool-call formats
• Execution order
• Error handling
• Context management
• Runtime behavior

A model can learn an action sequence that works in one harness but fails in another.

CoTrace investigates matching training trajectories to the harness used during deployment.

The researchers evaluated the approach on Terminal-Bench 2.1 and SWE-bench Lite.

I think this adds an important detail to the multi-harness RL discussion.

Training across different harnesses may improve generalization, but the training system needs to account for differences between their execution rules.

I'd want every training trajectory to record its harness version, tool schemas, permissions, and environment configuration.

Without that information, a successful rollout can become misleading training data.

[Alireza Bashiri @al3rez]

Read the original paper 👇
https://t.co/HCQut35NFG

[Alireza Bashiri @al3rez]

What if a coding agent could prove that its tools are safe before running them?

A paper called L3M explores a formally verified harness built with Lean 4.

The researchers replace ordinary conversation history with a Git-backed state trajectory.

That gives agents operations such as:

• Resume
• Rewind
• Fork
• Compare
• Merge

The security model is the more interesting part.

Agents can extend their tooling by writing Lean programs.

The system uses types to represent permission requirements.

It also uses machine-checked proofs to decide whether an operation is reversible and can run without human review.

The researchers report that an untuned L3M system achieved roughly two-thirds of the task success of an unsafe shell baseline on Terminal-Bench.

That's a meaningful trade-off to study.

The agent loses some flexibility, but its allowed operations have stronger safety guarantees.

I think this is a useful direction for unattended coding agents.

Instead of asking the model whether a command is safe, the runtime can require evidence before allowing it.

The challenge is making enough real development operations available through that restricted interface.