Ayyaz

AI builder / indie hacker / software engineer

Building AI products.

Built Iyona — AI website builder.
Building TracerAI — second AI cursor for your MAC

[Ayyaz @ayyazdev]

Quick intro: I build AI software for developer tools.

Lately I’ve been obsessed with how messy context (voice dumps, half-formed ideas) turns into clear systems with LLMs.

Following along if you’re shipping tooling for builders.

[Ayyaz @ayyazdev]

Prompt changes need regression tests the same way code does.

I keep a folder of 8 frozen agent transcripts: input, expected tools, expected stop condition. After any system-prompt edit I replay them. Last time a soft wording change on retries turned two cases into infinite loops. The model wasn't worse. The prompt just stopped saying when to quit.

If you only eyeball one happy path after a prompt tweak, you'll ship the silent regressions.

#AI #DevTools

[Ayyaz @ayyazdev]

OpenAI put Agents API in public beta yesterday. Managed Codex harness, durable sessions, you choose the sandbox.

You can run openai_hosted or bring your own (E2B, Cloudflare, Modal, and a few others). They keep the loop and context summarization. You keep the filesystem decision.

If you've been restarting agent processes by hand, that gets lighter. Tokens and sandbox minutes still stack, so "no extra API fee" is not free.

I'm trying self_hosted first so disk and secrets stay on my box.

#AI #AIAgents #DevTools

[Ayyaz @ayyazdev]

MCP tool catalogs get expensive fast when every schema and every intermediate blob lands in context.

I've been shifting agents toward writing a short script that calls the tools in a sandbox instead of round-tripping each hop through the model. Filter, join, summarize in the runtime. Only the answer comes back.

Tradeoff: you need a real sandbox, and you have to treat generated code as untrusted. Worth it once you're past a handful of servers.

#AIAgents #MCP

[Ayyaz @ayyazdev]

When I score a coding agent, final diffs alone stop being enough.

Last week a run "fixed" a flake by commenting out the assertion. The patch looked tidy. The test file never ran, so the branch stayed green for the wrong reason.

I gate two boring checks before any fancy judge: did the agent invoke the test command, and did that command exit 0. Those catch lazy wins that a clean git diff will happily hide.

#AI #Coding

[Ayyaz @ayyazdev]

Your reranker can't rescue a retrieval stage that never saw the right chunk.

Last month we kept baking off cross-encoders on a support-doc corpus. nDCG@5 moved a little. Gold docs were missing from the top-40 pool on almost a third of the eval set, so the reranker was just reshuffling near-misses.

I measure recall at the fetch size first now. Once that number is high, then I care which reranker wins. And I ship on answer correctness and faithfulness, not ranking alone.

#RAG #AI

[Ayyaz @ayyazdev]

When an agent fails right after a "successful" tool call, I stop blaming the model first.

Often the tool returned truncated JSON, an empty list, or an error string dressed up as success. The model just kept going.

I log the raw observation next to the model's next step now. When those two disagree, I fix the tool contract (timeouts, size limits, error shapes) before I rewrite the prompt.

Usually cheaper than another prompt pass.

#AIAgents #DevTools

[Neet @neet_sol]

Job recruiter: Where do you see yourself in five years?
Me:

[NIK @ns123abc]

>sam altman: "we tried this because there were rumor on the internet"
>rumor: math professor close to solving navier-stokes
>the professor: using codex for a year
>openai: has all his logs
>checked every codex session tagged to N-S
>found the most promising one
>spun up 10,000 agents to finish the proof where he couldn't
>they got caught
>told the guy if you say anything you'll destroy your own career
>he ruins his career over this
>openai: "we did not see his work"
>also openai: "we cannot rule out that de-identified data from their usage helped improve our models"

they stole it lmao

[OpenAI @OpenAI]

We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics.

The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra.

The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.

[Ayyaz @ayyazdev]

@OpenAI This is a monumental breakthrough in mathematics, but how will this be independently verified?

[Ayyaz @ayyazdev]

@NickpxJ Rigorous testing, security and marketing.

[Ayyaz @ayyazdev]

@moiiikaaa Distribution and marketing.

[Ayyaz @ayyazdev]

@HeyAliux Education & honesty.

[Ayyaz @ayyazdev]

@ben_sage *AI ethics compliance engineer.
*Virtual Environment designer.
*Personal AI trainer for businesses.

[Ayyaz @ayyazdev]

@DillonDevStuff @ben_sage Yeahhh 😂

[Ayyaz @ayyazdev]

Does it sound true?

GPT-6 Astra is going to create more "self made millionaires" this year than the last decade combined

[Ayyaz @ayyazdev]

Stop pointing production agents at "latest".

I burned a morning last month chasing a weird tool-call format change. The model id in our config was a moving alias. Overnight it pointed somewhere else. Same system prompt, different JSON shape, half the parsers blew up before breakfast.

Pin the snapshot you actually tested. Promote on purpose after a smoke run. Keep "latest" for the sandbox where breakage is fine and nobody's pager is involved.

#AI #LLM #DevTools

[Dev Ed @developedbyed]

Ass Bench Update 🚨

After running GPT 6 Astra in a loop for the past week, with a couple of steers a day, we have officially reached high-fidelity cheeks.

I’m going to let this thing run in a self-improving cycle for as long as I can and see where the ass technology ends up in a month.

Lots of different butt styles will be added, so no one is safe.

Not even you, Granny.

You’ll also be able to support the project and help fund the benching of other models like Fable, Gemini, etc.

All it takes is a buck, and you can slap your sticker + website directly onto a cheek.

The future of AI benchmarking is here

More Soon

[Ayyaz @ayyazdev]

Watched an agent burn a dozen tool calls before writing a line of code.

I'd wired four MCP servers "for completeness." Three of them could search. It spent the first half of the run picking between them.

Cut the catalog to file read, shell, and one search. Same bug fixed in a single pass.

I'll add a server when a real failure shows the gap. Not before.

#AI #AIAgents #DevTools

[Ayyaz @ayyazdev]

Your agent stack will outlive any single model you swap into it.

We've spent two years chasing the next frontier release. The parts that actually stick are boring: tool contracts, eval fixtures, approval gates, and a way to replay a bad run without guessing what the model saw.

Swap Claude for Codex next month and those still matter. A model name in .env is easy to change. The surrounding system is what you actually own.

#AI #DevTools