Hunter Gerlach

Senior Principal Architect @RedHat | Exploring the many facets of modern software engineering (...and every once in a while: sports). Build better software.

[Hunter Gerlach @HunterGerlach]

Everyone's talking about Mythos for security. Once that's sorted, I'm looking forward to the inevitable focus on performance. Addressing wasted time, energy, and money on unoptimized work will unlock huge amounts of compute.

[Ethan Mollick @emollick]

So Mythos was, indeed, not marketing hype.

Remember this is a general purpose model that just happens to be good at finding exploits because good models are good at lots of things. Expect similar from OpenAI & Google. And from open models in 8 months. https://t.co/KbhalQYX8R

[Hunter Gerlach @HunterGerlach]

https://t.co/k89Mtzwnjs

[Hunter Gerlach @HunterGerlach]

"The evidence we've presented here points to a clear conclusion: discovery-grade AI cybersecurity capabilities are broadly accessible with current models, including cheap open-weights alternatives."

https://t.co/LJS48XIA23

[Hunter Gerlach @HunterGerlach]

"The arrival of Mythos doesn't mean the foundation of open source is crumbling; it means the standard for platform maintenance has been raised. [...] Open source is the baseline for innovation, and we intend to keep this foundation strong."

https://t.co/YvAjOs2kru

[Hunter Gerlach @HunterGerlach]

"Organizations that contribute back to the projects they depend on realize between 2x and 5x return on their open source investment…"

vs.

"Technical debt and duplicated engineering effort adding up to millions of dollars in avoidable spending"

https://t.co/ron8mus1fM

[Hunter Gerlach @HunterGerlach]

Talking to multiple agents is the new "get out of the building".

[Hunter Gerlach @HunterGerlach]

Just a friendly PSA that this toggle exists and you don't have to get sucked into short-form content on @X...

[Hunter Gerlach @HunterGerlach]

Slowly

[Greg Brockman @gdb]

how did we ever write all that code by hand

[Hunter Gerlach @HunterGerlach]

New post...

https://t.co/ETMamh3KWu

[Hunter Gerlach @HunterGerlach]

TIL works for per-second throughput (e.g. TIL-3/sec), annual totals (e.g. TIL-15/year), or any standalone token count.

And it's also a nice bonus that the acronym was already being used to convey human inference and reasoning (TIL == today I learned).

[Hunter Gerlach @HunterGerlach]

Inference is exploding...
millions → billions → trillions... → quadrillions of tokens.

Here���s a proposal to measure that scale:

Token Inference Level (TIL).

[Hunter Gerlach @HunterGerlach]

TIL is a simple logarithmic measure of token volume:

TIL = log₁₀(tokens).

Examples:
TIL-9 ≈ billions
TIL-12 ≈ trillions
TIL-15 ≈ quadrillions

[Hunter Gerlach @HunterGerlach]

The beginning...

[U.S. Department of Energy @ENERGY]

https://t.co/0RwhGSduDn

[Hunter Gerlach @HunterGerlach]

For anyone out there vibe coding: remember that git is your friend.

Iterate in small steps.
Commit working changes.
Repeat.
Reset when models go off the rails.

[Hunter Gerlach @HunterGerlach]

[dns haiku]

[Hunter Gerlach @HunterGerlach]

Autocorrect and AI can result in some funny responses

[Hunter Gerlach @HunterGerlach]

I really liked this definition because it's technology agnostic:

"An agent is an entity capable of perceiving its environment, making decisions, and taking actions to achieve specific goals."

However, @simonw's definition is admittedly easier to communicate.

[Simon Willison @simonw]

I'm ready to accept a definition of "agent" that I think is widely-enough agreed upon to be useful:

An LLM agent runs tools in a loop to achieve a goal

This is a big piece of personal character development for me! I've been dismissing the term as hopelessly ambiguous for years

[Hunter Gerlach @HunterGerlach]

Anyway, these are just my observations and opinions. If you've had experiences to the contrary, I'd love to hear about them so I can improve the framework I outlined above.

[Hunter Gerlach @HunterGerlach]

Great observations...

As an eval pilled ai engineer, I think the necessity for evals depends primarily on two things:

1. how large your user base is
2. how deterministic your probabilistic system needs to be

[swyx 🇸🇬 @swyx]

Claude Code: no evals

[well known code agent company]: no evals

[well known code agent company 2]: kinda halfassed evals

[leading vibe coding company]: no evals

[ceo of company selling you evals]: mmmmm yess all my top customers do evals, you should do evals

[vc's in love with ceo of evals company]: mmmmm yes all my top founders do evals, must do evals

(NOTE: i -do- also think that evals are impt, but the eval pilled ai engineers have also noticed that it is not a strict requirement for success and, at least for 0-to-1 stage, may even be anticorrelated, think thru why)

[Hunter Gerlach @HunterGerlach]

(As an aside...
Solving complex systems problems like these almost always go back to Goldratt's Theory of Constraints.)