Fernando__

Trying/building things and logging them on:

[Fernando__ @fmeiraf]

Programming used to be a walled garden.

Harness engineering is the new programming.

The wall isn't gone — it just moved.

For now.

[Fernando__ @fmeiraf]

@bentlegen @badlogicgames @mitsuhiko @newsystems_ @sergical 🙏

[Fernando__ @fmeiraf]

@HamelHusain curious: is the increased token usage happening on a unique task/project you were previously spending less tokens on or this just opened up exploring more tasks/projects at the same time?

[Fernando__ @fmeiraf]

@dexhorthy @trq212 what is "deep llm intuition"?

[Fernando__ @fmeiraf]

I think in the end it makes sense: TUI makes it look fast and “close” to the action. In the end all the bash the agent executes always happened there in terminal. It’s natural.

Moving to the app makes it feel bland and lifeless. It’s not where we expect the agent to work using CLIs.

I believe the app move will make more sense when the app feels like the natural place to be in.

[Fernando__ @fmeiraf]

After reading @robzolkos write-up I got very interested in what was defined as "anemic" code in some of the commits, and decided to generate a specific gist to visualize that. Love how codebase like this can turn into a very interesting learning exercise.

https://t.co/2ZZThjCIgu

[Fernando__ @fmeiraf]

@robzolkos @37signals Thank you for doing this! You actually inpired me to dive further into the "anemic" code concept and I generated this to read during the weekend:
https://t.co/LGFcgX9NUe

[Fernando__ @fmeiraf]

i.e slop hedge

[Fernando__ @fmeiraf]

One of this year’s best learning decisions has been:

Using my 30 min lunch breaks to read a (full or partial) chapter of @ramalhoorg Fluent Python book

Suddenly, with all AI available, it makes more sense to know what is really good use of the language

[Fernando__ @fmeiraf]

@willmcgugan Does this mean we could have custom Textual components being triggered based on custom agent rules in toad? That would be very nice

[Fernando__ @fmeiraf]

Spent the last few days converting all the agents I’m working on to be ACP-compatible, 𝗮𝗻𝗱 𝗶𝘁’𝘀 𝗯𝗲𝗲𝗻 𝘀𝘂𝗽𝗲𝗿 ��𝗼𝗿𝘁𝗵 𝗶𝘁! Definitely an interesting tool to test and get a live feel for agent workflows locally - going to use this more.

[Fernando__ @fmeiraf]

When software looks so beautiful that it makes you want to work more on it 🎨

[Fernando__ @fmeiraf]

Having a blast using Toad @willmcgugan ! This looks so good + the markdown rendering is so well done.

[Fernando__ @fmeiraf]

If you want to run/try it yourself: https://t.co/0CAK4V7u8z

[Fernando__ @fmeiraf]

Last week's experiment: 𝗮𝘀𝘆𝗻𝗰 𝗲𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁 𝘁𝗼 𝘀𝗶𝗺𝘂𝗹𝗮𝘁𝗲 𝗮 𝗦𝘂𝗽𝗲𝗿𝘃𝗶𝘀𝗼𝗿 → 𝗪𝗼𝗿𝗸𝗲𝗿𝘀 𝘀𝗲𝘁𝘂𝗽 𝘄𝗶𝘁𝗵 𝗿𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 𝗿𝗲𝘃𝗶𝗲𝘄 𝗮𝗻𝗱 𝗳𝗼𝗹𝗹𝗼𝘄-𝘂𝗽 𝗮𝘀𝘀𝗶𝗴𝗻𝗺𝗲𝗻𝘁𝘀.

[Fernando__ @fmeiraf]

The main motivators for me to try this approach arise from production challenges like:

- Increase inference efficiency for multi-step tasks

- Create context windows that allow the agent to focus on a very specific part of the process

- Enable flexibility to craft specific workflows for isolated parts of a multi-step task

- Leverage async environments to create more dynamic exchanges between agents

[Fernando__ @fmeiraf]

I really like the idea of the REPL more than anything. There are some challenges with this approach though. I have tested the code base you shared last week and wrote some of my observations here :

https://t.co/bPT2mvgrkf

Looking forward to your further improvements and releases on this idea

[Fernando__ @fmeiraf]

In the midst of so much information—new models, post-training strategies (read: RL), and others—I found myself very excited when I first read the blog post about the RLM (https://t.co/I0LCAwwaTE) approach. The reason is simple: inference strategies are mostly overlooked and non-sexy in terms of discussion, yet very present in AI engineering day-to-day (Chain of Thought, ReAct, etc.).

For most practitioners out there, at this point context engineering is not just jargon anymore, and everyone is taking context rot (https://t.co/28p8j3CjGw) somewhat seriously (if you are not, you should!). Keeping the context controlled and relevant is definitely one of the main engineering tasks when building performant agents. For this reason, strategies at inference time are something I really look forward to whenever something different shows up in the wild—and that’s the case for RLM.

I’ll be very straight to the point here: from the many things discussed in the RLM blog post and idea, for me the one that is most interesting is having the REPL as the main environment for agent actions.

I tried it myself over the last week, and I want to share below some of the things I found.

> The good, and the not-so-good

The thoughts logged here came from interacting with the repo published by the authors, to which I only added LiteLLM and enabled some async operations for my own use cases (https://t.co/NYSTbRlrHD).

>> The good

Quoting from the authors:

“The natural solution is something along the lines of, ‘well maybe if I split the context into two model calls, then combine them in a third model call, I’d avoid this degradation issue’. We take this intuition as the basis for a recursive language model.”

From this, there are two obvious things the RLM approach enables:

- Dealing with context rot (by outsourcing a lot of the info to the REPL).

- Helping with very large context (letting the LLM use native programming-language resources to parse and break down the understanding as variables).

There are a few things that are not so obvious from the above, but are enabled by the REPL environment idea:

- It enables the model to think and spend tokens mostly on coding—which we already know these models do somewhat well (or well enough for the tasks we assign them).

- (The most interesting to me) it enables the model to create its own tools on the fly.

From the blog post, and from my own interactions, it’s clear to see an interesting pattern:

LLMs try to get ahead in the REPL, running many commands at once, which to me seemed like an interesting way to compress information gathering, making it more efficient and rich.

As I said, the REPL idea is something I really found interesting, but from the tests, the positives above can kind of backfire…

>> The bad

This is only my opinion after a week of a few interactions with this idea, mostly for a coding task: generating data visualizations.

- LLMs doing more coding: with the REPL, most of the LLM interactions involve coding responses that will be parsed and executed. If you’ve used LLMs for coding tasks, you know errors happen quite often—which isn’t different here, which leads to…

- REPL management: as you start getting errors, dealing with the REPL starts getting somewhat complex. Depending on the error location, you could have some variables from the proposed code added to the REPL, but further lines that didn’t execute might not be, and I’ve seen this confuse LLMs quite a bit. I believe you could prompt/RL this out and/or improve context management for REPL information, but instead of feeling like this makes context gathering go fast, when it goes wrong, it actually gets very slow—and annoying.

- Tool-creation automation: this one sounds very attractive, but I have the feeling it might not play out well in production agents. The reason is simple: for tasks that are well-defined/scoped, you tend to see consolidation in how the task is performed, which means you’ll have at least one well-defined tool that helps the LLM achieve good results most of the time. If you’re not adding these functions to the REPL globals—and, as a consequence, to the prompts—you’ll be relying on the randomness of the model to create these tools from scratch every time.

> Conclusion

The verdict is that I find the REPL idea very interesting, and I’ll definitely follow more closely other people doing work with this going forward. For now, it doesn’t seem like a clear replacement for the normal ways (read: for-loops and context management in templated prompts).

From my tests so far, the frustrations with code execution and handling the REPL are not trivial to overcome. On the same note, dealing with code suggests you might be better off with handcrafted functions—which puts you back into the traditional tool loop all over again.