zeljko

AI @ UCL / King's

[zeljko @w_is_h]

GPT Sol models were especially bad, the elo they achive was negative (i did not have a player bad enough for them to play against). Opus is a bit better, but still a complete beginner at Go could easily beat it.

Why are the models so bad at Go, and why are they unable to learn anything?

[zeljko @w_is_h]

kifu-bench, a self improvement benchmark for ai agents. agents play the game of Go, write things down, devise strategies and improve. the first full run is finished, results on github - in short, no agent managed to learn anything, and is in general really bad at the game of Go

[zeljko @w_is_h]

results with all the traces and games played -- https://t.co/wbV7au9rOO

[zeljko @w_is_h]

Sol has weird memory files for every game that are completely useless and keeps adding more garbage. but, more results tomorrow, after 200-300 runs.

[zeljko @w_is_h]

if you let an agent play the game of GO for 100 or 1000 matches, will it improve? after ~50 games, both Sol 6.1 and Opus 5.5 get worse the more they play.

my idea was to test in-context learning, self improvement and memory. so now i'm questioning the skill level of the models.

the setup is very simple:
- memory/skills enabled (a filesystem where agents can write strategies or whatever else)
- code execution enabled (no brute forcing or mc stuff)
- no internet and no subagents
- native claude code and codex harness

every agent picks an opponent and plays a game of GO, during it, it can write strategies, leave messages, execute code and all the rest. at the end, the game engine gives it a result, the agent can again decide to save some learnings/skills/code - full freedom, nothing is forced. the session resets, memory stays and another game is started.

the elo, for the 9x9 board i had to calculate my own elo, so it does not translate to human players but it is relative.

i wanted to submit this to a conference, but @petergostev this morning published a very similar idea with chess (i did 'borrow' a bit of frontend and the idea for the model to pick its level). So no publication, will keep the experiments running and opensource stuff once done.

i picked this over chess because there is more headroom - the training data is much more sparse than chess...

[zeljko @w_is_h]

Opus never removes anything, just keeps adding more stuff to the memory, has a couple of scripts for now and a 1k+ lines of claude.md

[zeljko @w_is_h]

@ThomasWolgast i would be more for pay the reviewers, and require a small payment for submissions. but i'm also questioning the usefulness of papers in the llm era - at least in the ai field

[zeljko @w_is_h]

finished reviewing a couple of papers for a neurips workshop, all were pure ai slop. and it feels the prompt was 'make it scientific, and professional' so it has a million of coined terms, and ultra dense sentences. is everyone seeing this or am i unlucky

[zeljko @w_is_h]

@pidotdev @mitsuhiko not sure how important it will be in the future that the code is 'nice' from a human perspective

[zeljko @w_is_h]

@DhravyaShah very nice, always thought that the caveman approach is the best one, similar as in agentslate

[zeljko @w_is_h]

@ekzhang1 one fun thing is that luna achieves the exact same accuracy as Jev, at around the same processing time per question

[zeljko @w_is_h]

@ekzhang1 this depends heavily on the dataset, on the 2,450 questions from MedXpertQA, the accuracy is 31% with a mean probability of 61%, significant overconfidence

[zeljko @w_is_h]

@hxiao do not think jev has image input, people that use it for computer control and similar feed the dom/html/whatever

[zeljko @w_is_h]

@danhergir a simple - how many r's in 'strawberry' and the output options were: 2, 3, 4. same for parallelogram. not really a trap, just a straightforward counting question

[zeljko @w_is_h]

my understanding of Jev (typesafe ai) is a calibrated universal classifier, so here are some results. first classic LLM failures (strawberry and others).

[zeljko @w_is_h]

after a bit more time with Jev, i do not see it, did not find a usecase where i would use this over deepseak v4.1 flash, or a tiny ultra fast model running locally

[zeljko @w_is_h]

very nice model, ultra fast and cheap, while achieving great results - probably useful for simpler things, if reasoning is needed, or cot, not the best.

[zeljko @w_is_h]

and lastly, some polarising questions (from serious to silly) - testing calibration