scribu

Co-founder/CTO @ VerbalEaze | AI/ML Engineer

[Skyfall AI @skyfallai]

The first real evidence that the days of LLM Scaling laws are over.

Introducing SCOPE: the world's most efficient Neural Planner. 🔍📊

We tested SCOPE vs Frontier LLMs for planning tasks on TextCraft (text version of Minecraft) and here are the results: ⤵️

- SCOPE Runs 55x faster than GPT 3.5 (3 seconds vs 164 seconds)

- SCOPE is 160,000 smaller than GPT 4o (11M parameters vs 1.8T parameters)

- SCOPE is more accurate on Planning tasks (56%) than frontier LLM models

The age of efficient AI models starts now.

🔗📌 Read the full write up here: https://t.co/n3DDSqshRm

[Matt Shumer @mattshumer_]

what an insane difference a year makes

sora 1 (left) vs. sora 2 (right)

both using the same prompt

[scribu @scribu]

"You can have any colour you want, as long as it's purple."

GPT-5 when vibecoding a web page for you.

[Javi Lopez ⛩️ @javilopen]

Your automated vibecoded AI startup be like:

[scribu @scribu]

@markjaquith

[scribu @scribu]

OpenAI just killed the psychedelics industry.

[Charlie Holtz @charlieholtz]

even the sora mistakes are mesmerizing

[scribu @scribu]

The Aesthetics Wiki also looks solid:

https://t.co/FDy1V8Qo3l

[scribu @scribu]

The Index of Aesthetics is a goldmine.

https://t.co/dz87wHKtVu

[scribu @scribu]

People talking about emergent capabilities are fooling themselves, basically.

https://t.co/f4BhMshxDy

[Bindu Reddy @bindureddy]

The Emergent Abilities of LLMs Could Be A Mirage!

The best paper award in NeurIPs 2023 went to a paper claiming that the emergent abilities of LLMs could be a mirage!

The paper (link in alt) asserts that emergent abilities appear due to the researcher’s choice of metric rather than fundamental changes in model behavior with scale.

Let's understand some terms before getting into the details.

Emergence is a phenomenon whereby new properties may materialize in systems as their complexity increases. These properties can't be predicted from a precise quantitative understanding of the system’s microscopic details.

Emergent properties of LLMs are abilities that are not present in small models that manifest themselves in larger models (Sharpness), and their performance on specific tasks can emerge quite unpredictably and abruptly at scale (Unpredictability)

A lot of drama around LLMs taking over the planet involves emergence. Researchers argue that some scary emergent properties like free will and consciousness can magically manifest themselves in LLMS, and therefore, we have to pause, ban, and regulate AI research.

The paper excellently and credibly argues that the LLMs DO NOT possess emergent abilities - by this, they mean that there isn't anything sharp or unpredictable about them.

They show that smooth, continuous, predictable changes in model family performance appear sharp and unpredictable based on the choice of metric. So bigger models naturally and smoothly are more performant; there isn't some sharp jump in performance.

They also find that for non-linear metrics, smaller models are more performant than previously reported when they add additional test points to increase the resolution of the benchmark.

Overall, the point of the paper is that LLM behavior is NOT unpredictable, and in fact, larger LLMS are predictably more performant than larger ones.

In other words, there is no scientific reason to believe that LLMs can magically become supervillains one day.

If there is one paper you should read about LLMs, I recommend this one!

TLDR: There is no magic voodoo happening with LLMs; it's all math and statistics... as all deep learning is

[scribu @scribu]

While I strongly believe Generative AI is a game changer, I was never worried about The Singularity.

That’s because GenAI models have the same limitation as all other ML models: they can’t handle novel tasks.

In other words, they can interpolate, but not extrapolate.

[scribu @scribu]

For anyone thinking of doing a CodeGen startup:

Low value: generating basic code from basic instructions

High value: generating context-specific code from detailed instructions

[scribu @scribu]

OpenAI has JSON mode now 🎉

You can only nudge the model towards a particular schema, through a prompt, which I suspect will be good enough for 95% of use cases.

Although custom logit preprocessors would be cool, I get why OpenAI chose the simpler solution.

[scribu @scribu]

Prediction: LLM vendors, such as OpenAI and Anthropic, will allow users to define their own logit processors.

[scribu @scribu]

OpenAI's function calling feature, launched last month, does NOT guarantee that the output is valid JSON:

[scribu @scribu]

Here we go: Break-A-Scene

Open-source solution for manipulating concepts in an image like variables in a programming language.

https://t.co/R7iT1U4GFj

[scribu @scribu]

Google’s new text-to-image model, MUSE, seems to be able to break down elements in an image and manipulate them independently.

That’s a game changer!

Who’s working on an open-source version?

[scribu @scribu]

@staysaasy It could be a vicious cycle - AI startups noticing sites are starting to restrict access, so they're trying to scrape as much as possible while they can.

[Jason Crawford @jasoncrawford]

If a technology may introduce catastrophic risks, how do you develop it?

The Wright Brothers' approach to inventing the airplane is one case study:

[scribu @scribu]

In case Twitter goes down, you can find me on https://t.co/rySKNHN86G - still @scribu