Richard Tang

building ax-eval: eval your product surfaces (api/cli/mcp/sdk) for codex and claude code

ai & data science @ asana

[Richard Tang @richardt830]

agents are becoming first-class users of software, but most teams are not running evals against how discoverable and usable their product surfaces are for agents.

so I built ax-eval: the first open-source cli tool to test whether agents can discover and use your product, across api/cli/mcp/sdk and between codex and claude code.

I ran sample tests for four products @stripe , @NotionHQ , @linear , @ExaAILabs and here is what I found:

[Richard Tang @richardt830]

🔥🔥 https://t.co/trqP76xN8H

[Richard Tang @richardt830]

@emollick Agreed. I think much of it is also about mental anchor. names like Claude Cowork has much lower psychological barrier for knowledge workers to adopt than technical “agent mode”

[Mark Zuckerberg @finkd]

(1) Today we're releasing Muse Spark 1.1 -- a strong agentic and coding model at a very low price. It's available through our new Meta Model API and in Meta AI.

[Richard Tang @richardt830]

really appreciate your article, zara.

one challenge i find is blocking designated time slots during the day to do all of the following: read, post, interact. i feel like every hour there’s new stuff coming out that i want to research on/engage with/write a sincere response to, but it would really distract me from my deep focused work. i know you created a tool to block 30 mins on your calendar to read bookmarked articles but i still feel it’s not enough?

wondering if you’ve encountered this issue and how you’re dealing with it?

[Paul Graham @paulg]

A Brown professor gave his students a take-home midterm exam. After suspecting many cheated using AI, he made the final in-person. The orange dots are the midterm scores and the gray dots are the final scores. Looks like all but 3 cheated on the midterm.

[Richard Tang @richardt830]

@TheLoneWulf_WA @emollick So he is implying that Fable is more independent, whereas GPT-5.6 Sol works with you in steps more. what I’m saying is this seems to be a reverse from previous Anthropic models, which are more user-centric, and previous GPT models, which are more independent.

[Richard Tang @richardt830]

@TheLoneWulf_WA @emollick Yeah, that's what I mean.

[Richard Tang @richardt830]

thing about SF weather is I have to prepare for six sets of clothing:

outdoors/work
gym
indoors

and 2x for cold and warm days each

sometimes I rotate all six of them in a single day.

iykyk

[Richard Tang @richardt830]

@geoffreylitt softest engineers are holding their phones like construction site manager holding walkie-talkies 🤔

[Geoffrey Litt @geoffreylitt]

A workflow I'm enjoying: "Walk-driven development"

> go on a nice walk outside 🚶
> record a long audio note: ideas, goals, things to build 🎙️
> agent auto-creates docs/tasks, and kicks off cloud coding agents for me 🤖

[Richard Tang @richardt830]

do you proactively compact context or let codex automatically do it for you? wondering if there's any difference in terms of quality/cost?

[Richard Tang @richardt830]

@HamelHusain customers’ internal IT team and vendors’ CSM team?

[Richard Tang @richardt830]

@RhysSullivan much needed!!!

[Cole Bemis @colebemis]

Twitter Bookmarks 🤝 Notion

[Richard Tang @richardt830]

@snowmaker agent experience (AX) is the future user experience (UX)!

[Richard Tang @richardt830]

@snowmaker interesting 🤔 and good investors and advisors are like a good learning rate that helps the teams get out of local optima.

[Richard Tang @richardt830]

so well said - “Taste isn’t valuable because it’s impossible to copy. Taste is valuable exactly because it defines what everyone else chooses to copy.”

[Mitchell Hashimoto @mitchellh]

https://t.co/hXE5lXLpFT

[Andrew Ng @AndrewYNg]

“Loop engineering” is a hot buzzphrase after mentions of it by Boris Cherny (Claude Code’s creator) and Peter Steinberger (OpenClaw's creator) went viral on social media. Loops are now a key part of how we get AI agents to iterate at length to build software. In this letter, I’d like to share my 3 key loops, shown in the image below, for building 0-to-1 products. These loops guide not just how I build software, but also how I decide what software to build.

Agentic coding loop: Given a product specification and optionally a set of evals (that is, a dataset against which to measure performance), we can have an AI agent write code, test its work, and keep iterating until the code is bug-free and meets its specification. This idea of closing the loop took off around the end of last year, and it has been a game changer in enabling coding agents to work longer productively without human intervention. For example, over the weekend, I was building an app for my daughter to practice typing, and my coding agent could easily work for around an hour, using a web browser to check what it had built multiple times before getting back to me, without needing my intervention.

The engineering loop executes quickly. Every few minutes, the coding agent might build and test a new version of the software. I hear frequently from developers who are finding new ways to engineer more effective engineering loops. This is an active area of invention!

Developer feedback loop: In this loop, a developer examines the current product and steers the coding agent to improve it. Last year, a lot of developers (including me) were acting as the QA (quality assurance) function for our coding agents, manually finding bugs and then asking the agent to fix them. But with coding agents much more able to test their own code, the amount of time we need to spend on this function has decreased significantly. This allows us to make higher-level product decisions, such as what key features to offer, where the UI needs improvement, and so on.

The developer-feedback loop operates over time intervals between tens of minutes and hours — that's how frequently a developer might review a product and give feedback. In the case of the typing app, I changed my mind a few times about the visual design, what cat costumes she can unlock as she learns (she loves cats), and the user flow for a grown-up to log in and steer the child's learning experience.

When a developer has a clear vision for what to build, it is still a lot of work to translate that vision into a specification for a coding agent to implement. Further, after the developer has seen an implementation, they might update (or perhaps clarify) the spec to steer it toward what they want. If you find that the system repeatedly runs into certain problems, building a set of evals for the agent becomes useful.

AI-native teams are increasingly using AI to help shape product direction, for example, automating the gathering and analysis of usage data, summarizing written and verbal customer feedback, or carrying out competitive analysis. However, for pretty much all the products I’m involved in, I see humans as having a significant context advantage over current AI systems — we know a lot more than the AI system about the users and the context the product has to operate in — and thus humans play a critical role. Many people describe this human contribution as “taste,” but I prefer to think of it as humans having a context advantage, since that gives us a clearer path to helping AI systems get better. This also speaks to why this step can’t be automated: So long as the human knows something the AI does not, human-in-the-loop is needed to to inject that knowledge into the system.

External feedback loop: This includes a wide range of tactics like asking a few friends for feedback, launching to alpha testers, or putting the code into production with A/B testing. These tactics are usually slow, rarely taking less than hours and sometimes taking days or even weeks. This data informs the developer vision, which in turn continues to drive the detailed product spec, which in turn drives the coding agent.

With coding agents speeding up software development, more engineers are starting to play a partial product management role. For many engineers who are growing into this role, the hardest part is shaping the product vision and striking a balance between building (bridging the gap between vision and spec) and getting user feedback to evolve the vision. It is important to do both!

I will write more about how to do this in future posts, but for now, I find it encouraging that engineers are playing an expanded role (just as product managers and designers now do more engineering).

[Original text: The Batch]

[Sarah Tavel @sarahtavel]

For the past 25 years, application software startups have had a singular focus: increasing company and employee productivity. AI enables a paradigm shift: Rather than sell software to improve an end-user's productivity, consider what it would look like to sell *the work itself*.

When you sell the work, instead of selling a 15% productivity improvement, you sell a 95% improvement. Incumbents are not only advantaged in selling software, they are stuck there. You don't need to play their game. Post: https://t.co/29BkdGIMC3