Scott Clennell

[Scott Clennell @scottclennell]

Theres a new version of claude code coming out today but regrettably this does not appear to be Fable 5.5.

It does appear to be releasing an ability to chart in the terminal, looks like its called "easel".

[Scott Clennell @scottclennell]

Ok this is cool as hell.

[Cheng Lou @_chenglou]

My dear friends, I’d like to show you my latest toy project

Introducing Handwired: turn your regular JavaScript code into a neural net!
- no training
- exact outputs vs the JS program
- fully differentiable

All runnable in a browser, without GPU

[Scott Clennell @scottclennell]

https://t.co/5B3phsdvR7

[Scott Clennell @scottclennell]

Ok this is wild. OpenAI just published mathematical proofs, hundreds of them, that are the equivalent of decades of mathematical academic work. Performed in several hours of (internal) model use.

Implications are staggering.

[Scott Clennell @scottclennell]

@MingtianZhang @typesafeai That's a really smart idea...however I would say that there is an ordering bias AND a name bias (especially in significant uncertainty).

For example, if you ask whether a hidden coin is heads or tails...it follows heads regardless of position.

[Scott Clennell @scottclennell]

@burkov Truth is somewhere in between. Been testing it intensely for 3 days.

The probabilities aren't in real world remotely calibrated.

But it *orders* stuff incredibly well, which is itself incredibly useful, and at a frontier equivalent level.

[Scott Clennell @scottclennell]

@neural_avb In other words it not hallucination about the answer, it's genuine uncertainty as to the *best* response.

Try it with two nouls instead - does this contain general information and does this contain a bug report - I'd suspect both would come back with high confidence.

[Scott Clennell @scottclennell]

@neural_avb (your example incidentally contains both information and a bug report, which is likely the reason for the indecision. Jev is not good at this, understandably)

[Scott Clennell @scottclennell]

@code_kartik It basically is that - but fast (so fast it opens other use cases), cheap (similarly, that opens other use cases), and structured in response (which brings big benefit to code).

It can't *do* anything other models cant, its how it does it that changes things massively.

[Scott Clennell @scottclennell]

Reposition maths in a non mathematical sense and it can do some reasonable progress. And this might be suitable for reasonably small/trivial transforms in buckets in code (e.g. "last measurement top 2% out of last 100"). But don't force the wrong use cases.

[Scott Clennell @scottclennell]

The number of people I'm seeing trying to use Jev for games/Tetris/images is astounding.

It's no good at maths people, that's not the point of it.

And if it's no good at maths, it's no good at spatial problems.

Square peg round hole.

[Scott Clennell @scottclennell]

@narphorium Agreed. Expected it to fail completely past 1 or 2 stage problems but it didn't. Suggests a lot of the reasoning we see from LLMs across the board happens at first pass (so not just a Jev thing). Hoping to share proper results soon!

[Scott Clennell @scottclennell]

Continuing with work on @typesafeai Jev. Plenty of impressive visual demos from people which are great. Reality is it's going to be game changing in a far more invisible way - both in code (by itself) and stacked with frontier models.

Remain very impressed.

[Scott Clennell @scottclennell]

@narphorium (incidentally on your scenario and without reframing the question, it gets 5 out of 6!)

[Scott Clennell @scottclennell]

@narphorium It actually scores very highly on multi step benchmarks, way higher than you'd expect.

Across 8 step benchmark tests on BBH I've got it scoring 79% correct (against n=24).

I also synthesised a ladder in case of training and it scored the same level.

[Scott Clennell @scottclennell]

My benchmarking for @typesafeai #Jev model continues to astound me, to the point it starts to change what we think about LLMs full stop.

It's performance on benchmarks, including those that basically *require* reasoning and multiple passes, is way way higher than it should be.