Sure!
Post-training focus on a handful of contemporary languages and solution domains doesn’t erode the vast knowledge in the training corpus.
For example, there are nearly 7 decades of literature on lisp. 5 of SQL. Lisp isn’t popular today, but your model can “play that guitar just like he’s ringing a bell”.
Similarly, if you’ve spent your life on the command line in Linux, you’ll find that modern SOTA models are fluent in dominant paradigms from decades past.
My point is twofold: first, models are better versed in a wider variety of solution domains than this article suggests, and second the “elegance” of an approach is orthogonal to their fluency.
In other words, there are old roads that might not be en vogue on which models will very happily tread.
Canny operators are well served by building deep and expansive knowledge of yestertech.
@Oklahoma_Brave@nmlinguaphile I don't know what cosmic alignment of the stars and/or algorithm brought you into my feed, but you're amazing, and I'm sure your sister is as well.
There's an unnamed and fundamental constraint being widely ignored, and it lies near the heart of Bob's approach.
Everyone understands and accepts that context is limited. It's a given.
Human attention is orders of magnitude more limited than model context. Spend it wisely.
@firestormodk Not error -- failure. There's a difference. When I have to read the code it is not an error to read the code; but it is a failure of my plan to not read the code.
@vicentematuss My agents write the unit tests. I don't read them. The agents write the gherkin, and I read or spot check them. The agents write the QA procedures and I either read or spot check them.
Next week, will be closer to the autumn equinox than the summer solstice. Your days are leaving the heady long days of summer and rapidly approaching the balanced autumn cusp.
If you started from the graph, and treated everything (skills, prompts, loops) as composable graphs from the very beginning, where would you be today?
Looking forward to posts on ontologies, topologies, projections, and intensional/extensional divides.
From prompt → context → harness → loop → graph engineering:
The list keeps growing, and every new term gets treated as a replacement for the last one.
In reality, however, each layer wraps the one before it, and the cleanest way to tell them apart is to ask what a single unit of work looks like.
> Prompt engineering is the message:
The model remembers nothing before this call, so the prompt has to carry the full universe of what it needs. a role, the background, the instructions, a few examples, and a format.
When the output falls short, the skill is working out which ingredient lets you down, not rewriting the instructions every time.
The unit of work is one input.
> Context engineering is the memory:
Across many steps, the window is finite, and the available information is not, which forces a curation step. A curator keeps what matters, compresses what is useful but bulky, and drops the rest.
Good curation is mostly about knowing what to throw away, not packing more in.
The unit of work is what stays in the window.
> Harness engineering is the machine:
On its own, a model just generates text. The harness gathers what it needs, runs it, calls tools or sub-agents, and verifies the result with tests or a judge.
That verify step is the entire difference between calling an api and running an agent.
The unit of work is one pass through the machine.
> Loop engineering is the run:
One pass rarely finishes the job, so something has to decide whether to run the machine again. That decision needs a goal defined upfront, brakes like max iterations and budget caps, and a completion check that is automated rather than felt.
An agent that stops asking for tools has ended its turn, which is not the same as finishing the task.
The unit of work is the whole run.
> Graph engineering is the coordination:
Once several loops have to work together, you need to say what runs when, what runs in parallel, and who checks whom. Nodes do the work, edges decide what runs next, and shared state flows between them.
A single loop is just a one-node graph with an edge pointing back at itself, which is why graphs govern loops instead of replacing them.
The unit of work is the whole job.
Here is the part that ties it together.
Prompt and context both live inside the harness gather step. The harness is one pass, the loop decides whether to run that pass again, and the graph decides which loops run at all.
Zoom out, and the unit of work gets bigger. Zoom in, and you are back at the prompt.
That also tells you where to debug. Find the layer whose unit of work broke, then fix that layer.
The prompt is the easiest layer to edit, which is why it keeps taking the blame for failures that live three layers up.
My co-founder published a deep dive on graph engineering, covering the core idea, how to get started, shared state, routing you can trust, and when a graph is genuinely overkill.
Read it below.
@DanielMiessler An ISA is necessary but not sufficient.
Finding the correct route between here and an ISA requires terrain knowledge that LLMs lack.
Private org data and latent expert knowledge are not published.
We must provide them via context.
You ARE early, but not wrong.
"It turns prompt engineering into intent engineering"
Agreed. All roads appear to be leading us back to the core intensional / extensional divide from Datalog.
I've been saying this for something like eight months now, with varying approaches and volume levels, and no one is paying attention yet.
This is either because I am way early or my idea isn't one third as good as I think it is. :)
But here it goes again:
I think we will soon figure out that the entire game for AI is articulation of ideal state.
And that all of our conversations about specs, and plans, and loops, and PRDs and so many other things that have been talked about in software engineering and harness engineering for the last six months...all converge on this same idea.
I think the way it will be articulated is in the form of a single artifact that captures, enhances, iterates on, climbs toward, builds, and tests the ideal state.
One document that replaces all these loops and specs and PRDs and plan files and design files, et cetera.
My current implementation of this is the ideal state articulation (ISA) system.
It turns prompt engineering into intent engineering, in the sense that it abandons telling the AI how to do things and replaces that with telling it exactly what you want the output to be.
@karpathy had a recent post where you talked about going on a walk and just talking freely through a bunch of ideas related to what you're trying to accomplish. to me, that is 100% on point and 100% part of this ideal state articulation.
We also had this recent open AI hack where a goal was given and a goal was presumably achieved. Similar to a paperclip maximizer.
And in both cases, the problem is that we conveyed a goal without conveying ideal state. Which, if properly articulated, would have included lots of paperclips but humanity still existing or getting the top grade on the test, but without building new exploits and hacking companies.
The problem is always in the guessing. It's in the gap between stated and implicit. Or stated and difficult to articulate.
That's why I see the whole entire game as this articulation of what we want.
This is what prompt engineering always has been. But it got muddled up with earlier models that weren't very smart, so we had to combine the intent engineering with step-by-step instructions.
I think it's time to cut through that now and isolate this central idea.
An extraordinary amount of our problems, really across anything but especially when working with others and building things with AI, is assuming that the receiver has the same idea in their mind as the one in ours.
And that's why I'm proposing this approach for addressing that issue directly.
Let's get extremely good at articulating the ideal state for what we have asked for.
Really nailing this does so many things for us simultaneously. It not only helps the receiving system build what we actually want with a lot less gap between our mind and its mind, but after it's built, it also becomes the testing harness as well as the heart of the documentation.
It's a single system for general hill-climbing, toward anything.
https://t.co/Jmv777pwnB