Rusty Roboz

Currently exploring Artificial Intelligence, Quantum Computing and VR Development

[Rusty Roboz @RustyRoboz]

Acabo de publicar un articulo en Medium sobre como estoy publicando un prototipo de juego de cartas para parejas en Android, puedes leerlo aquí: ‘MVP cogiendo polvo’ a Google Play: la resurrección de Cartastrofe https://t.co/5Mk0DISWZI

[Rusty Roboz @RustyRoboz]

Trailer de TopoKarts!

[Rusty Roboz @RustyRoboz]

TopoKarts — un kart racer 3D donde conduces sobre superficies topológicas: esferas, cintas de Möbius, toros y planos hiperbólicos.

Hecho con Fable 5 antes de que lo bloquearan

pruébalo aquí: https://t.co/n9ZU4B86ox

https://t.co/EPWQFCBu7E

[Rusty Roboz @RustyRoboz]

Pequeño logro personal: ya está Cartastrofe en Google Play!

Empezó como un proyecto pequeño, sacado en ratos y aprendido a base de prueba y error, así que verla ya publicada hace bastante ilusión.

Si os apetece echarle un ojo:
https://t.co/FO06Evnjvi

[Matt Shumer @mattshumer_]

https://t.co/ivXRKXJvQg

[Dagan Shani @daganshani1]

How OpenAI pushed the AI field into a dangerous race. The whole segment about AI from my new short doc: ֿ "Moloch - AI & The Deadly Force Driving Us To The Brink"

[Carlos Santana @DotCSV]

La gente está tomando vídeos generados por Sora y aprovechando su consistencia para generar un entorno 3D interactivo.

De texto > vídeo > 3D 🔥

[Carlos E. Perez @IntuitMachine]

1/n No Search, No Problem: Achieving Grandmaster Level Using Only a Transformer

A new research paper presents a groundbreaking advancement in chess-playing artificial intelligence, demonstrating for the first time that it is possible to train a neural network to play chess at a grandmaster level without relying on explicit search techniques. This finding challenges the long-held belief that sophisticated search algorithms are indispensable for mastering complex games like chess.

Historically, chess AIs such as Deep Blue and AlphaZero have depended on robust evaluation functions, extensive opening books, and advanced search techniques like alpha-beta pruning and Monte Carlo tree search to anticipate future moves. The question of whether neural networks could achieve expert-level play through supervised learning alone, without the computational overhead of search algorithms, remained open until now.

The breakthrough came by harnessing the power of modern transformers, scaled up to 270 million parameters, and training them on a dataset of 10 million human chess games annotated with strategic evaluations by the Stockfish 16 chess engine. This approach allowed the neural network to predict Stockfish's evaluations of new board positions accurately.

The performance of this neural network is exceptional, surpassing AlphaZero's value and policy networks, solving 93.5% of a wide range of chess puzzles, and achieving a blitz rating of 2895 on Lichess, a score higher than that of most grandmasters. Remarkably, this was achieved without employing any search strategies beyond evaluating all potential next moves.

This significant finding reveals that with enough model capacity and a substantial training dataset, it is possible to distill the complex search and evaluation algorithms of advanced chess engines like Stockfish into the parameters of a neural network. This represents a paradigm shift, suggesting that capable chess AIs can be developed without the need for manually designed heuristics or search algorithms.

The success of this approach underscores the potential of using transformers and self-supervised learning to approximate complex algorithms, opening new avenues for research into how far this technique can eliminate the need for search in strategic reasoning and its applicability to other domains. This work not only marks a milestone in AI chess but also signals a broader implication for the future of artificial intelligence in strategic reasoning tasks.

[Stephen Zerfas @stephen_zerfas]

We figured if cut the guesswork, we'd make jhanas at least 10x faster to learn.

So we set out to collect a biosensor dataset big enough to train ML models that would make jhanas easy-to-verify. Imagine the music or instruction you're listening changing in realtime based on your brainwaves.

Great shot here of @RogerThisdell

[Rusty Roboz @RustyRoboz]

https://t.co/I99YeU1OwN

[Jose Luis Calvo @joselcs]

1/ Me he puesto a ordenar una lista de libros sobre Inteligencia Artificial. Sobre todo divulgativos, aunque también hay alguna novela, de estudio o de referencia.

Abro hilo 👇

[François Chollet @fchollet]

Unfortunately , too few people understand the distinction between memorization and understanding. It's not some lofty question like "does the system have an internal world model?", it's a very pragmatic behavior distinction: "is the system capable of broad generalization, or is it limited to local generalization?"

[François Chollet @fchollet]

To understand X means you have the ability to act appropriately in response to situations related to X -- for instance, you understand how to make coffee in a kitchen if you can walk into a random kitchen and make coffee.

[Andrej Karpathy @karpathy]

# On the "hallucination problem"

I always struggle a bit with I'm asked about the "hallucination problem" in LLMs. Because, in some sense, hallucination is all LLMs do. They are dream machines.

We direct their dreams with prompts. The prompts start the dream, and based on the LLM's hazy recollection of its training documents, most of the time the result goes someplace useful.

It's only when the dreams go into deemed factually incorrect territory that we label it a "hallucination". It looks like a bug, but it's just the LLM doing what it always does.

At the other end of the extreme consider a search engine. It takes the prompt and just returns one of the most similar "training documents" it has in its database, verbatim. You could say that this search engine has a "creativity problem" - it will never respond with something new. An LLM is 100% dreaming and has the hallucination problem. A search engine is 0% dreaming and has the creativity problem.

All that said, I realize that what people *actually* mean is they don't want an LLM Assistant (a product like ChatGPT etc.) to hallucinate. An LLM Assistant is a lot more complex system than just the LLM itself, even if one is at the heart of it. There are many ways to mitigate hallcuinations in these systems - using Retrieval Augmented Generation (RAG) to more strongly anchor the dreams in real data through in-context learning is maybe the most common one. Disagreements between multiple samples, reflection, verification chains. Decoding uncertainty from activations. Tool use. All an active and very interesting areas of research.

TLDR I know I'm being super pedantic but the LLM has no "hallucination problem". Hallucination is not a bug, it is LLM's greatest feature. The LLM Assistant has a hallucination problem, and we should fix it.

</rant> Okay I feel much better now :)

[Carlos Santana @DotCSV]

🔴 ¡NUEVO PIKALABS 1.0! 🤯

Watdef*** ?!? ¿Es este el DALL·E de vídeos?

PikaLabs acaba de anunciar la salida de su modelo de generación de vídeo a la versión 1.0 y WOW! Por el trailer este modelo podría ser el más avanzado hasta la fecha! Toca probar 😁
https://t.co/NS2cyVMQWO

[Jim Fan @DrJimFan]

In my decade spent on AI, I've never seen an algorithm that so many people fantasize about. Just from a name, no paper, no stats, no product. So let's reverse engineer the Q* fantasy. VERY LONG READ:

To understand the powerful marriage between Search and Learning, we need to go back to 2016 and revisit AlphaGo, a glorious moment in the AI history.
It's got 4 key ingredients:

1. Policy NN (Learning): responsible for selecting good moves. It estimates the probability of each move leading to a win.

2. Value NN (Learning): evaluates the board and predicts the winner from any given legal position in Go.

3. MCTS (Search): stands for "Monte Carlo Tree Search". It simulates many possible sequences of moves from the current position using the policy NN, and then aggregates the results of these simulations to decide on the most promising move. This is the "slow thinking" component that contrasts with the fast token sampling of LLMs.

4. A groundtruth signal to drive the whole system. In Go, it's as simple as the binary label "who wins", which is decided by an established set of game rules. You can think of it as a source of energy that *sustains* the learning progress.

How do the components above work together?

AlphaGo does self-play, i.e. playing against its own older checkpoints. As self-play continues, both Policy NN and Value NN are improved iteratively: as the policy gets better at selecting moves, the value NN obtains better data to learn from, and in turn it provides better feedback to the policy. A stronger policy also helps MCTS explore better strategies.

That completes an ingenious "perpetual motion machine". In this way, AlphaGo was able to bootstrap its own capabilities and beat the human world champion, Lee Sedol, 4-1 in 2016. An AI can never become super-human just by imitating human data alone.

-----
Now let's talk about Q*. What are the corresponding 4 components?

1. Policy NN: this will be OAI's most powerful internal GPT, responsible for actually implementing the thought traces that solve a math problem.

2. Value NN: another GPT that scores how likely each intermediate reasoning step is correct.
OAI published a paper in May 2023 called "Let's Verify Step by Step", coauthored by big names like @ilyasut @johnschulman2 @janleike: https://t.co/iAvXNjjhcK
It's much lesser known than DALL-E or Whipser, but gives us quite a lot of hints.

This paper proposes "Process-supervised Reward Models", or PRMs, that gives feedback for each step in the chain-of-thought. In contrast, "Outcome-supervised reward models", or ORMs, only judge the entire output at the end.

ORMs are the original reward model formulation for RLHF, but it's too coarse-grained to properly judge the sub-parts of a long response. In other words, ORMs are not great for credit assignment. In RL literature, we call ORMs "sparse reward" (only given once at the end), and PRMs "dense reward" that smoothly shapes the LLM to our desired behavior.

3. Search: unlike AlphaGo's discrete states and actions, LLMs operate on a much more sophisticated space of "all reasonable strings". So we need new search procedures.

Expanding on Chain of Thought (CoT), the research community has developed a few nonlinear CoTs:
- Tree of Thought: literally combining CoT and tree search: https://t.co/KM1P2ZJrjG @ShunyuYao12
- Graph of Thought: yeah you guessed it already. Turn the tree into a graph and Voilà! You get an even more sophisticated search operator: https://t.co/5ncT5tuTOY

4. Groundtruth signal: a few possibilities:
(a) Each math problem comes with a known answer. OAI may have collected a huge corpus from existing math exams or competitions.
(b) The ORM itself can be used as a groundtruth signal, but then it could be exploited and "loses energy" to sustain learning.
(c) A formal verification system, such as Lean Theorem Prover, can turn math into a coding problem and provide compiler feedbacks: https://t.co/vpOBOI2FR5

And just like AlphaGo, the Policy LLM and Value LLM can improve each other iteratively, as well as learn from human expert annotations whenever available. A better Policy LLM will help the Tree of Thought Search explore better strategies, which in turn collect better data for the next round.

@demishassabis said a while back that DeepMind Gemini will use "AlphaGo-style algorithms" to boost reasoning. Even if Q* is not what we think, Google will certainly catch up with their own. If I can think of the above, they surely can.

Note that what I described is just about reasoning. Nothing says Q* will be more creative in writing poetry, telling jokes @grok, or role playing. Improving creativity is a fundamentally human thing, so I believe natural data will still outperform synthetic ones.

I welcome any thoughts or feedback!!

[Rusty Roboz @RustyRoboz]

@gfodor I have tried a cyberpunk adventure in which I was a hacker who had to plan a data heist on a mega corporation called XCorp, cool GPT!

[Rusty Roboz @RustyRoboz]

I have created a custom GPT about a Zen master that you can talk to about your problems, things that worry you, or ask for advice to lead a more fulfilling life. You can try it here: https://t.co/43LerLtMJo

[Rusty Roboz @RustyRoboz]

@ProperPrompter I can definitely relate to your skepticism about Mixed Reality. It's amazing how our perspectives can shift when we're exposed to compelling examples. I'd love to hear more about the 10 examples that changed your mind. It might help others who are still on the fence. #Quest3

[Rusty Roboz @RustyRoboz]

@lexfridman More of this kind of algorithis should be made public. It would be cool to see more companies follow suit and share their recommendation algorithms with the community