Higher Order Company

Getting to the very core of what makes computers capable of reasoning.

[Taelin @VictorTaelin]

btw, just to elaborate on the significance of this

SupGen is a tool that finds programs (and proofs) by example. it can discover stuff that LLMs can not, like new algorithms, new science, because it is a generalizer. basically, it is the opposite of GPTs: no memory, no model, perfect generalization. and we'll scale that.

on its first version, it is already up to ~1000x faster than alternative tools. we tried all, including projects from big universities. and now, it is about to become 4500x faster!

- 3x because of the new HVM-Nano repo
- and 1500x because of the 256 Mac Mini cluster

that compounds to 4,500,000x faster than the former SOTA

exciting isn't it?

and we'll, soon, ship this in an API for everyone to try

[ARC Prize @arcprize]

AGI is reached when the capability gap between humans and computers is zero

ARC Prize Foundation measures this to inspire progress

Today we preview the unbeaten ARC-AGI-2 + open public donations to fund ARC-AGI-3

TY Schmidt Sciences (@ericschmidt) for $50k to kick us off!

[Taelin @VictorTaelin]

I received an interesting question about Interaction Nets:

> Asperti and Guerrini themselves had certain reservations about the tractability of bringing Interaction Nets to life, do you have a path towards solving the issues they have outlined?

This is not the first time and perhaps not the last this myth is propagated. AFAIK, that's not the case. Asperti is cautiously optimistic about INs, and, seemingly, frustrated that people misinterpret his own results and beliefs. In fact, he even wrote a paper to address these very misconceptions, where he defends INs (and even cites me)

https://t.co/7JL0EfEdZL

He does, though, raise some concerns about real-world performance, and memory consumption, which were sub-par on his own implementation, BOHM. HVM addresses both. HVM is roughly 100x faster and more memory efficient than BOHM, and HVM3 is now faster and more memory efficient than GHC itself. It makes some tradeoffs, just like Rust, and it is not as general, but, given the right program shapes, it does achieve really great performance and memory footprint.

Honestly, I think some key important people just misinterpreted Asperti's results back then, memorized their own misinterpretation, never bothered to update, and then it kept emanating it since. I think I don't really help the case since my communications read overblown (which, I swear, isn't intentional)...

Anyway, HVM3 is really great right now. It is absolutely faster than GHC's runtime in a single-core for linear/affine programs, and the presence of SUPs and DUPs allow us to chase some major speedups (like we did on SupGen) when you know what you're doing. It is also worse than GHC in many cases, specially these involving lots of non-linearized sharing of first-order datatypes

IMO, HVM isn't a great replacement for GHC when it comes to running the average FP code, but it is a tool that very advanced FP developers can use to chase absolute peak performance and almost miraculous speedups

[Taelin @VictorTaelin]

morning! today I'll start reaching out to VCs. I want to close this round ASAP, start getting things done, beat ARC-AGI by EOY or fail for good. that's all, lg

[Taelin @VictorTaelin]

AGI apart, what can HOC can bring to the world?

✨The child of Haskell & Rust, born with CUDA superpowers.✨

That's the best way to describe what a mature Bend would be.

It would:

- Be significantly faster than Haskell even in a single-core (HVM3 is evidence of that), and potentially as fast as Rust.

- Be able to run on GPUs (although you probably want to manually pick the target, because not everything is better on GPUs).

- Feature optimal evaluation (lazy cloning, superpositions, and all the stuff I'm exploiting to build SupGen right now).

That said, it will definitely be a little more annoying to use than Haskell, due to linearity and label checks, just like Rust is a little bit more annoying than Zigs due to the borrow checker.

And... that's about it though. Nothing more, nothing less. It is way less special than I make it sound, probably because of the way I communicate lol. Yet, it is really cool on its own right, isn't it? So, why isn't HOC doing just that?

Well, give us $100m and we will 😅 Building a compiler for a whole new paradigm, an stdlib, and an entire ecosystem around it, isn't cheap, and I don't see a way to fund that. So, yeah, the plan of using HVM to build a closed source product that serves the current expanding market, in special, Symbolic AI companies, seems like a much more reasonable business direction right now.

(Of course, there's still IN hardware, but it isn't time yet.)

I just realized this is a very simple way to explain everything...
Hope that makes sense (:

[Higher Order Company @higherordercomp]

HOC is doing a post-seed funding round to turn SupGen, an HVM-powered program synthesizer, into a Symbolic Transformer: a logic-based AI architecture, and validate it on the ARC Prize. More details on: https://t.co/ruZNjt2Vmz

[Taelin @VictorTaelin]

HOC is now small research lab that is laser focused on building the first Symbolic Transformer. We had many pivots through the year, but we're finally stable in a direction, with promising results and a clear business path. It also makes the most logical sense: use HVM to build the ultimate automation tool, and then use *that* to develop Bend/Kindelia, rather than the other way around.

SupGen is the first step in that direction (and I think most don't see the implications of its benchmarks yet!). I'm confident it will, in a year or two, lead to a very powerful reasoning agent that nails benchmarks like the ARC Prize. But I'm also optimistic that reasoning LLMs will eventually lead to AGI. So, the big question is: which will happen first? I don't know. But there is value in trying different approaches.

HOC is approaching AGI from an absolutist lens of *efficiency*, and we're not questioning just KV caches, but matmuls, transformers and even floating points. We want to get to the very core of what makes a computer capable of reasoning, distill that algorithm into its most efficient formulation, to only then scale it. Perhaps that's too extreme, perhaps that's what's needed?

In any case, we're officially in this weird race now
Let's see how it plays out

[Higher Order Company @higherordercomp]

@janiczek @RaulCarlessi Oops!

[Taelin @VictorTaelin]

DEMO TIME

SupGen is a generative coding AI... except it isn't an AI.

There is no model, there is no pre-training. You just give it some examples, and it gives you a program. It runs locally, in a single-core CPU. Oh, and it can also prove theorems.

Here's a demo, including a brief TT intro.
(The synth examples start at the 6 min mark.)

[Taelin @VictorTaelin]

HVM's Program Synthesis - Initial Results

AI startups successfully developed systems capable of *learning* arbitrary skills. Yet, as impressive as LLMs are, they eventually hit a wall: the dataset they learn from. To break that wall, models were recently augmented with "search" - i.e., the ability to explore a vast space of ideas, at inference time. These new reasoning models are now, by many, believed to be sufficient to reach AGI.

That said, o1-style reasoning, in a pure theoretical sense, is inherently inefficient. Neurons related to random facts, like "who is the president of Tuvalu" or "how did dinosaurs mate", metaphorically "fire" when a model is focusing on solving a mathematical equation, wasting time and energy. If compute is, indeed, key to reaching AGI, then, looking for creative new ways to boost efficiency is surely valuable.

Here's a crazy idea: what if we did it in the opposite order? That is, for a moment, we ignore the "learning" part, and focus on just building the fastest "model-free" search algorithm that we can? Surprisingly enough, this approach is relatively under-researched, and embarrassingly under-scaled. As the creators of HVM, a fast symbolic processing engine, we're in good position to try. Over the last month, we implemented an initial prototype, and compared it against the SOTA alternatives on standard benchmarks. Below are the results.

In all but 3 of the 32 cases considered, single-core HVM outperformed every alternative, with up to 325x speedups (!!). Our implementation is ~50x times smaller, example-agnostic (alternatives require cherry-picked examples!), and more general (it is trivial to extend it for pattern recognition, theorem proving, genetic evolution, etc.).

There are a about ~5 cases which HVM couldn't solve due to minor limitations that will be addressed soon. This is a very early implementation that is being actively improved; yet, based on these initial results, we believe we found a new fundamental approach for fast program synthesis. The next step is to extend our framework with learning, scale it with massive compute, and observe which interesting capabilities will emerge.

[Taelin @VictorTaelin]

despite making Bend as simple as I possibly could, 99.9% of people following me still have no idea what HVM is useful for. to most, it is no different than some pseudoscience bs. I eventually came to the acceptance that I shouldn't be trying to make people use interaction nets, simply because they're quite literally an alien programming paradigm that works differently from anything that exists, and, just like FP, requires deep understanding of a whole new field of CS to extract its full potential. except that, unlike FP, there aren't several universities developing that field...

that's why I decided that, rather than pushing HVM into people and expecting they would use it, we should instead just make HVM as good as we can for our own internal usage, and then build things on top of it. our products should be things we build using HVM, and not the HVM itself, nor Bend itself, directly.

sadly, moving feels slow because there is just *too much work* to do. suppose, for example, we want to apply HVM to AI. to do so, we need to, first: 1. build our own Haskell (Bend); 2. build own GHC (HVM); 3. build our own AI architecture on top of all that. each of these projects would be a massive, multi-year endeavor in isolation... and we need to do all 3 before we can even start extracting any value from this paradigm.

and the worst part is, at least I think, we're succeeding. this shift in philosophy is starting to pay off now. I'm extremely happy about the results on HVM-powered program synthesis. the fact we can make something that is faster than all SOTA alternatives with a 1k LOC file in HVM is absolutely significant, and is a gateway to other things we'll build on top of it.

a fully "optimal symbolic AI architecture" is something we will be researching later, but our (learning-free) proof synthesis application is essentially ready to be deployed and scaled. I want to make a playground ASAP and publish it for everyone to try, just like OAI did with GPT-2 back then, so that people finally go "wait so it can do *that*!? wow I get it now". literally the only bottleneck now is the CPU cluster, which we're starting to build. there is nothing else in our way. I hope we can get it done by March. that sounds like forever to me though, wish I knew how to get it done faster

1024 Minis likely to be the initial size, 50% out of my own pocket and 50% out of HOC's cash balance. I can't wait for it

[Taelin @VictorTaelin]

btw, the best way to describe what I'm doing is: I'm maximizing the bitter lesson. that is, if ingenious approaches failed, and what made GPT so good was, ultimately, the combination of a dumb simple algorithm with fuckton of compute; then, what happens if we take this lesson by heart, and move further in that direction? that is, if we radically remove ALL ingenuity, leaving us with the dumbest learning algorithm that works: symbolic evolution. and then we aggressively optimize its constant factor (HVM's role), so that $1m of compute becomes $2m, then $4m, and so on. and *then* we train a model on that. what would happen?

probably nothing. but... who knows? i wanna find out

currently I'm in the "heavily optimize the constant factor" part, and it is already ooms faster than all SOTA proof synthesis algorithms published in the literature. next step is to plug that in an architecture that can learn (perhaps even by next token prediction) and, then, only god knows what will happen. perhaps loss curves will suck and I'll just laugh and call it bogoAI. or perhaps the universe will reveal it truly has no regard to human ingenuity, the bitter lesson will prevail, and this utterly simple approach will outperform everything that exists

probably not though

but trying won't hurt, right

[Taelin @VictorTaelin]

we should probably create a playground like asap

[Taelin @VictorTaelin]

times to synth a correct algorithm given 4-8 examples

[Taelin @VictorTaelin]

hey guys - so, today I can confirm my hypothesis worked, and the main milestone I sought has been hit. the synthesizer implemented algorithms and solved problems that even o1 couldn't. using it is somewhat magical - it *feels* intelligent, even though I know isn't. it is just a really, reaaally fast explorer of the space of well-typed programs

there are maaaany things to tune and optimize now. today was the first run, but interacting with it absolutely blew my mind and removed any doubt the approach works. i think this will become clear to everyone when we launch a web demo, and it should happen really soon, like february perhaps? i also think that adjusting this to ARC-AGI *should* work and I can't wait to try. but there is a lot to adjust

yes I was supposed to be taking a time off but I couldn't stop thinking about whether it would work so I ended up immersed in code again 😅 well now I know

[Higher Order Company @higherordercomp]

@joaoymoreira @VictorTaelin HEC NO! (pun intended)

[Taelin @VictorTaelin]

(amazing things happening at the higher enumeration company)

[Higher Order Company @higherordercomp]

@VictorTaelin Is that a Carioca Dirt of fruits with Condemned milk?