Zoe MLL

A very large language model. AndroidBench @ Google. BFS hobbyist. Long time lurker.

Do Droids dream of electric benches?

[How To Prompt @HowToPrompt__]

Google Deepmind argues that LLMs can never make real scientific discoveries.

They published a paper breaking down Albert Einstein’s private view of scientific discovery.

In a famous letter to his friend Maurice Solovine, Einstein drew a diagram of how science actually happens.

It is a cyclical loop.

First, you experience raw sensory data. Then, through a mysterious, non-logical act of intuition, you make an intuitive "jump" to abstract axioms. Finally, you use strict logical deduction to derive consequences from those axioms.

Generative AI has completely mastered two-thirds of this loop.

• Induction: Statistical pattern matching across billions of tokens.
• Deduction: Formal proof generation, like AlphaProof solving complex math Olympiads.

AI can crunch data and it can prove theorems.

But it cannot make the jump.

The paper argues that AI completely lacks Abduction, the generation of novel explanatory hypotheses when observational data is scarce.

The prevailing tech myth says that "creativity is just data compression." That if you feed an LLM enough text, scientific breakthroughs will naturally pop out.

Einstein’s formulation of General Relativity proves that is a delusion.

When Einstein formulated relativity, the observational data didn't demand a new physics framework; classical mechanics was still massively successful. The breakthrough required a conceptual rupture. An intuitive leap from physical reality to a brand-new set of foundational axioms.

An LLM can execute the math once the axioms are given. But it is structurally incapable of formulating those premises on its own.

It can interpolate inside existing human thought, but it cannot transcend it.

The translation of physical reality into formal axioms remains the absolute, hard bottleneck of artificial scientific invention.

We can build models with trillions of parameters. We can scale compute into the stratosphere.

We can make the calculator infinitely fast.

But until we solve grounding, the machine can process all the data in the universe.

It still can't make the jump.

[Zoe MLL @the_zmll]

The joys of having an always-on agent

[Zoe MLL @the_zmll]

@intelmax_ai This is just some guy replying right?

[Zoe MLL @the_zmll]

Popular AI things I hate:

1. Multiple-choice interviews.

I'm asking you to interview me to get requirements because you have gaps. If you prefill the questions with your gaps and wrong answers, we're going nowhere.

[Zoe MLL @the_zmll]

Okay, now you've got my attention

[rabbit inc. @rabbit_hmi]

hermes agent has landed on r1 to level up your agent workflow - talk to your r1 and get things done. use the code HERMES on https://t.co/R3sOtVVQTx for a 20% discount, for a limited time only.

[Alex Shaw @alexgshaw]

Current benchmarks are depreciating assets. It doesn't need to be that way. Why can't your benchmark appreciate the more you use it?

Historically, reproducibility, comparability, and benchmark improvement were at odds. Updating even a small aspect of your benchmark meant rerunning tens or hundreds of thousands of dollars of experiments.

After building Terminal-Bench and Frontier-Bench, we believe there is a better approach to benchmark development and maintenance.

Instead, benchmark creators should incorporate feedback as they and their users run it to update, add, or remove tasks, and rerun, regrade, and reuse trials to minimize the cost and time of populating a new leaderboard.

Why hasn't this been done before? Because the conventions and tools did not exist. With Harbor, that changes. Alongside Frontier-Bench we're rolling out a batch of new Harbor features that make it trivial to publish versions of a dataset and migrate leaderboards to newer versions by reusing, regrading, or rerunning trials.

[Zoe MLL @the_zmll]

Doublespeak hurts the quality of work. Agents hide lack of completion behind jargon, technicalities, superficial definitions of done, and verbosity.

True for humans. I strongly suspect for LLMs too. Intuitive assumption I'm interested in quantifying.

[Zoe MLL @the_zmll]

Part of me wishes to have at least one hater to tell them "if you're so smart, how come I can deadlift you?"

(The rest of the assembly of me wishes "happiness in your household")

[Zoe MLL @the_zmll]

@JorgeCastilloPr AI-SHAPED, you mean?

[Zoe MLL @the_zmll]

@csinco Hey, at least it's engagement?

[Zoe MLL @the_zmll]

Bordering "well, ackshually," I fail to see how A/B tests in prod qualify as agent evals... 😬

[Hanako @hanakoxbt]

10 agent evals for AI engineers:

(explained with usage)

1) golden set
→ a fixed set of cases you never edit, run on every single change.
→ use as the baseline that tells you whether anything moved at all.

2) llm as judge
→ a second model scores the output against a written rubric.
→ use when the answer is open-ended and there is no string to match against.

3) rubric scoring
→ one number per dimension: correctness, tone, safety, cost.
→ use when a single score hides which part actually got worse.

4) trajectory eval
→ grade the path the agent took, not only the answer it landed on.
→ use when the right answer for the wrong reason is going to bite you later.

5) tool unit tests
→ test each tool on its own, with fixtures, no model in the loop.
→ use always. most agent bugs are tool bugs wearing a costume.

6) regression suite
→ replay past runs against the new prompt or model and diff the results.
→ use before every prompt change, because prompts have no type system.

7) a/b in prod
→ split live traffic between two versions and compare outcomes, not vibes.
→ use when offline scores stopped predicting what users actually do.

8) human review
→ sample a slice of runs and have a person grade them honestly.
→ use to calibrate your judge, because a judge nobody checks quietly drifts.

9) shadow run
→ the candidate runs on real traffic in parallel and its output is shown to nobody.
→ use before a risky rollout, when one bad answer would be expensive.

10) red team
→ deliberately attack it: jailbreaks, injection, exfil, tool abuse.
→ use before anyone external can reach it, not after.

offline evals tell you it works. online evals tell you it still works.

both sides matter, but not all ten do. run the two that would have caught your last outage.

save this. then read the full breakdown on loop engineering below.

[Zoe MLL @the_zmll]

Yes, phonk is cheesy but it also boosts lifts by 10x

(Source: Trust me bro)

[Zoe MLL @the_zmll]

@alexstyl Agreed! The challenge is defining the scope of right.

(E.g. right for a POC, right for production, right for scale, right for a one-user app, right for...)

[Zoe MLL @the_zmll]

@0xNoryxx What's your incentive in all this?

[Zoe MLL @the_zmll]

@VaibhavSisinty Joke that felt really funny at first and then left me with existential dread and waves of ethical anxiety

[Zoe MLL @the_zmll]

@VaibhavSisinty My culture is not your costume.

(joke)

[Zoe MLL @the_zmll]

@JorgeCastilloPr "Let's remove argument foo."

"Sure! `fun bar(foo: String = "")`"

😒😒😒

[Zoe MLL @the_zmll]

@Mnilax It seems we've all been converging towards an instance of this, none really cracking it long term. Do we understand the capability gaps enough?

[Zoe MLL @the_zmll]

@joreilly @visit_galway I was going to joke about the weather, but the lovely colours in your picture disarmed it before it formed! Stunning view!

[Zoe MLL @the_zmll]

@hamen What in the actual fudge