Noah Bilgrien

AI, creativity, intelligence augmentation. More interested in jetpacks than robots.

[Noah Bilgrien @nobillygreen]

@michael_nielsen "Zen and the art of motorcycle maintenance" has beauty in its contrast and intrigue.

[Noah Bilgrien @nobillygreen]

@DrJimFan "if the inference is correct, then the entire trace becomes a mini data set" sounds a lot more like reinforcement learning!

[fabian @fabianstelzer]

introducing: ComfyAGI 🧙‍♂️😉

we've taught Claude to generate ComfyUI workflows, so you can now build comfy workflows just with prompts...

We're open sourcing the entire prompt chain for this (which is a glif too), so u can remix & adapt it

join our Discord & come hack with us:

[Noah Bilgrien @nobillygreen]

@AndrewYNg Is there a "product" oriented term for "falsifiability"? This seems highly, highly similar to falsifiability.

[Andrew Ng @AndrewYNg]

Planning is a key agentic AI design pattern in which we use a large language model (LLM) to autonomously decide on what sequence of steps to execute to accomplish a larger task. For example, if we ask an agent to do online research on a given topic, we might use an LLM to break down the objective into smaller subtasks, such as researching specific subtopics, synthesizing findings, and compiling a report.

Many people had a “ChatGPT moment” shortly after ChatGPT was released, when they played with it and were surprised that it significantly exceeded their expectation of what AI can do. If you have not yet had a similar “AI Agentic moment,” I hope you will soon. I had one several months ago, when I presented a live demo of a research agent I had implemented that had access to various online search tools.

I had tested this agent multiple times privately, during which it consistently used a web search tool to gather information and wrote up a summary. During the live demo, though, the web search API unexpectedly returned with a rate limiting error. I thought my demo was about to fail publicly, and I dreaded what was to come next. To my surprise, the agent pivoted deftly to a Wikipedia search tool — which I had forgotten I’d given it — and completed the task using Wikipedia instead of web search.

This was an AI Agentic moment of surprise for me. I think many people who haven’t experienced such a moment yet will do so in the coming months. It’s a beautiful thing when you see an agent autonomously decide to do things in ways that you had not anticipated, and succeed as a result!

Many tasks can’t be done in a single step or with a single tool invocation, but an agent can decide what steps to take. For example, to simplify an example from the HuggingGPT paper (cited below), if you want an agent to consider a picture of a boy and draw a picture of a girl in the same pose, the task might be decomposed into two distinct steps: (i) detect the pose in the picture of the boy and (ii) render a picture of a girl in the detected pose. An LLM might be fine-tuned or prompted (with few-shot prompting) to specify a plan by outputting a string like "{tool: pose-detection, input: image.jpg, output: temp1 } {tool: pose-to-image, input: temp1, output: final.jpg}".

This structured output, which specifies two steps to take, then triggers software to invoke a pose detection tool followed by a pose-to-image tool to complete the task. (This example is for illustrative purposes only; HuggingGPT uses a different format.)

Admittedly, many agentic workflows do not need planning. For example, you might have an agent reflect on, and improve, its output a fixed number of times. In this case, the sequence of steps the agent takes is fixed and deterministic. But for complex tasks in which you aren’t able to specify a decomposition of the task into a set of steps ahead of time, Planning allows the agent to decide dynamically what steps to take.

On one hand, Planning is a very powerful capability; on the other, it leads to less predictable results. In my experience, while I can get the agentic design patterns of Reflection and Tool use to work reliably and improve my applications’ performance, Planning is a less mature technology, and I find it hard to predict in advance what it will do. But the field continues to evolve rapidly, and I'm confident that Planning abilities will improve quickly.

If you’re interested in learning more about Planning with LLMs, I recommend:
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, Wei et al. (2022)
- HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face, Shen et al. (2023)
- Understanding the planning of LLM agents: A survey, by Huang et al. (2024)

[Original text: https://t.co/pWmIR9wEki ]

[Noah Bilgrien @nobillygreen]

@karpathy @alexinch_ai @karpathy a question I've been thinking about a lot lately: what do you think a programming language that was optimized and designed to be written by LLM's in collaboration with humans would look like?

[Noah Bilgrien @nobillygreen]

@MartinNebelong @emmanuel_2m So Englebart's theory of "augmentation over automation", so to be somewhat reductive?

[Noah Bilgrien @nobillygreen]

@icreatelife

[Noah Bilgrien @nobillygreen]

@deliprao And by true pdf I mean the modeled pdf

[Noah Bilgrien @nobillygreen]

@deliprao I'm not sure why you would choose to show this rather than just showing the true probability density function

[Noah Bilgrien @nobillygreen]

@deliprao ...overall volume is very confusing. The color gradient also seems completely unnecessary and seems to imply information that isn't actually there.

[Noah Bilgrien @nobillygreen]

@deliprao If I'm being honest, the paper doesn't give a good simple definition, but my TLDR would be that it's roughly a discretized probability density function?

While these plots make a lot of sense for a single figure, the fact that two figures next to each other don't have the same...

[Replicate @replicate]

The Stable Diffusion innovation just doesn't stop!

Here's a new open-source model from the @monaverse that produces seamless tiling images: https://t.co/xEFyyY8W3I

[Noah Bilgrien @nobillygreen]

@TomerUllman This is a fascinating syllabus and such a great collection. I'm a tad surprised I don't see anything from John Pollock though on defeasible reasoning?

[Noah Bilgrien @nobillygreen]

@ASpittel Missing answer: [ ] the smallest version of each that can produce a prototype. None of them exist on their own.

[Chris McComb (he/him) @ccmccomb]

This semester I've been teaching a new class on designing AI/ML products in @CMU_Mech, and as part of that class I have learned SO MUCH about great tools for ML rapid prototyping. Here are a few of my favorites:

[@[email protected]🌎 @terrymorse]

What are the one-year risks of COVID-19 for the general population?

Possible outcomes are:
– didn't catch it
– infected, fully recovered
– infected, long COVID
– infected, death

Here's a simple risk analysis for vaccinated and unvaccinated, based on published data.

[Noah Bilgrien @nobillygreen]

@waitbutwhy @JesseAbraham

[Noah Bilgrien @nobillygreen]

@michael_nielsen I'd say it's a collection of "schools of thought" aimed at solving the same or similar problems. And I think of schools of thought as "a collection of theories whose relative goodness and complexity can be compared."

Not sure if that's it, but I those two ideas are valuable.