Dan Patrascu-Baba

CTO @ Atherio

Join my community:

[Dan Patrascu-Baba @danpdc]

After some more heavy use of Fable 5.2 my conclusion is very clear:

I use it as part of the sub for planning and as the orchestrator of my implementation loop. But when I need to switch to API pricing I revert back to Opus 5.5

The difference in quality doesn't justify the huge difference in costs

[Dan Patrascu-Baba @danpdc]

And contrary to popular belief, doing evals is not easy. It's a science on itself. In real projects I found you need a person almoast full-time on evals as the eval suite needs to be continously adjusted and improved

[Dan Patrascu-Baba @danpdc]

That's something I was thinking/saying for many months now. Coding is probably one of the best taks for LLMs because code is per se deterministic, has a defined structure and can be verified with automatic tools (long before AI).

[Gergely Orosz @GergelyOrosz]

Feels like inside most companies / tech companies, the vast majority of AI spend is coding-related (and spent by devs… in a few hedge funds, traders/quants also spend big time)

I wonder if this will change? Or is coding uniquely suited to being a long-running task thanks to all the validation loops it can do, and prove that whatever is produced is correct in ways we can programmatically test?

[Dan Patrascu-Baba @danpdc]

Then there are the evals. For coding, you basically have all these tools that to the evaluation of the output. Wit non-coding agents you need to take care of the evals yourself.

[Dan Patrascu-Baba @danpdc]

@GergelyOrosz That's something I was thinking/saying for many months now. Coding is probably one of the best taks for LLMs because code is per se deterministic, has a defined structure and can be verified with automatic tools (long before AI).

[Dan Patrascu-Baba @danpdc]

@Dave_DotNet @dotnet I'm old, do you think I remember things like this 🤣 You're probably right

[Dan Patrascu-Baba @danpdc]

Time to go to sleep then. Thanks GitHub for helping out :)

[Dan Patrascu-Baba @danpdc]

So, I guess GitHub Actions are down again?

[Dan Patrascu-Baba @danpdc]

After some more deep planning work with Fable 5.2 my conclusion is that it's just a slightly better version of Opus 5 (not even Opus 5.5). It constantly disobeys instructions, doesn't call skills even when asked to and even bluntly goes against decision agreed just one turn previously.

[Dan Patrascu-Baba @danpdc]

At Atherio we're building a whole new area of functionality which in many ways is quite different compared to what we have so far.

This is packaged with a lot of architectural and technical decisions that need to be made. While working on the planning phase (AI-Assisted of course) I realized once more how important technical understanding is.

There are a lot of areas where even Fable 5.2 struggled a lot.

For instance, I want to event source this entire new area of functionality and Fable 5.2 struggled to map event sourcing theory to our use cases and made a lot of weird choices that I needed to correct when it comes to mapping events, what and how we hydrate and so on.

It also struggled to map event sourcing concepts to our data access properly, creating a lot of unwanted coupling between the "old" area of the system and the new one.

At a point the agent went into proposing some complicated logic to circumvent a possible problem with how we handle session management although these two areas don't have any touchpoint at all and the problem was a clear hallucination.

This is a practical example on why I think 100% automated software factories are not feasible right now, even with the latest and shiniest models.

And if anyone is interested I have an extensive technical write up on this in my community, that I strongly encourage you to join.

[Dan Patrascu-Baba @danpdc]

@mjovanovictech I think the type of SaaS usually tackled by indie hackers definitely is. Even Marc acknowledged this indirectly in a post from yesterday where he mentioned TrustMRR is the one in his portofolio that's the least susceptible to AI irrelevance. And I think he's right on that point

[Dan Patrascu-Baba @danpdc]

We know how wild the tech industry has become by the fact that now we start to charge $99 for participating in a hackathon that has $100k in sponsorships.

It's the wild west and people try to do anything to secure their financial future. There's no moral left!

[Dan Patrascu-Baba @danpdc]

@mjovanovictech There are so many startup incubation initiatives even around Europe. Most of them are also free to enter, though there might be a selection process.

Idk, to me this initiative sounds horribly wrong on so many levels.

[Dan Patrascu-Baba @danpdc]

Btw, I used this approach 7 months ago when implementing AI-Assisted Engineering practices in an engineering org working mostly on poli-repo microservices. It's indeed effective.

[Matt Pocock @mattpocockuk]

A lot of folks ask me "how do you work with agents in multiple repos at once?"

I have a very simple solution - the Home Repo

I have a single repo (the home) where all of my skills, steering files, wiki etc live.

I run my agent in the home repo, and then inside AGENTS.md I have pointers to the other repos:

```
For skills, clone mattpocock/skills to ~/repos/skills.
For Sandcastle, clone mattpocock/sandcastle to ~/repos/sandcastle
```

This is extremely simple and incredibly effective. The home repo lets you:

1. Put all your steering files and tooling in one place
2. Have a single spot to run all your agents in
3. Make sweeping changes over multiple repos

Talked about this in the course yesterday and folks are loving it (see below).

I'm sure you will too. Enjoy!

[Dan Patrascu-Baba @danpdc]

@mattpocockuk Used this approach 7 months ago when implementing AI-Assisted Engineering practices in an engineering org working mostly on poli-repo microservices. It's indeed effective.

[Dan Patrascu-Baba @danpdc]

As a technical decision maker one of the most challenging things is to distinguish hype from real value and what's actually feasible.

One of the most effective way to do this is by "following the money"

And let's take the software factories example.

When it comes to people hyping software factories, if you look closely, there are 3 categories:

1️⃣Those who sell you $1000+ courses on how to run software factories
2️⃣Those that work at companies where they have the luxury of unlimited tokens
3️⃣People who don't have a clue about it, but are talking a lot because of people in the first two categories

Nobody that's hyping software factories have done this on a real product in the average company.

Unfortunately, there are technical decision makers that lack the ability to distinguish or balance between hype and value.

The result often is a broken engineering culture, burnt down engineers and an endless search for scapegoats.

Proton is a very good example for this.