@badlogicgames@bentlegen Prompting gpt 5.5 to “push back” works ok-ish. opus otoh can’t help but find your idea interesting no matter how much you tell it to push back
Can language models learn useful priors without ever seeing language?
We pre-pre-train transformers on neural cellular automata — fully synthetic, zero language. This improves language modeling by up to 6%, speeds up convergence by 40%, and strengthens downstream reasoning.
Surprisingly, it even beats pre-pre-training on natural text!
Blog: https://t.co/Pni0RsIcxL
(1/n)
@thsottiaux 1. Make the session id available to the model so that it can reference it in commits.
2. A capable Hook system, eg in plan mode I would like the model to narrate (TTS) context while presenting the multiple choice questions.
3. Automatic forking in chats to preserve tokens.
@badlogicgames Yes, from a technical standpoint this is straightforward. But I wonder what would need to happen for us to trust them more so that they can be this autonomous. At which point is the model behavior reliable enough that we can tell them some safeguards and let them roam?
@giffmana@crystalsssup I always thought that was a bit simplistic. At least it hinges on the scope of understanding. Just because someone can’t explain the inner workings of something doesn’t imply not understanding its function. But that functionality might be hard to convey verbally.
@giffmana@RichardSSutton@eigenrobot I think this breaks down to the definition of state and action. Either the state is described by the language or can be inferred from it (no pure imitation learning) or not (imitation of actions).
sharing this to remind both you and myself, put your work out there openly, put it on your own website so you control the process. Share knowledge in all forms, from making short-form videos that capture a few second in someone's attention span, to marble sculptures that last centuries. Knowledge only lives by being shared.
@aryehazan@bremen79 The way I did it was progressively asking it to breakdown the problem into self-contained sub-questions that could be used to make a determination. I'd feed each one to a fresh context gemini and report back.
This particular subquestion produced the counter example in response.
@francoisfleuret @freddiekarlbom I think RL subsumes bandits and denotes learning by trial and error, while reinforcing successful (i.e. high reward) actions in one or more states.
Unpopular opinion: benchmarks like these are moving the field in the wrong direction
No I don't want an AI to be able to memorize (useless?) questions like "How many paired tendons are supported by a sesamoid bone?" in its weights
I want the "intern", as @karpathy is suggesting
IT WORKS! Demo time 🥳
(next 10x productivity bump is here?)
Suppose you must refactor a large codebase, e.g.:
> "use I32 instead of U32 as the native number type"
That task, by itself, is simple enough to be automated by Sonnet alone, right? Problem is: what if the codebase is too large to fit on the AI's context, and RAG solutions / editors like Cursor can't precisely isolate which parts require edit (because it needs some semantic reasoning)? What now? Do we fallback to human coding, as did the old Mayans?
Of course not! We just split the codebase in chunks, and send each chunk to ~500 DeepSeek instances in parallel (!). Each instance then decides if given block should be edited or not. This is viable because DeepSeek is smart, fast and really cheap. Finally, we send the blocks that need edit to Sonnet, in a single prompt, and let it work normally. The entire process takes less than a minute, is cheap enough, and very effective. We conclude with a 'git diff' for manual review.
In short, AI has came to grep search & replace. Since this script feels like dropping a bomb in your codebase, I'm calling it AoE, and it is available on VictorTaelin/AI-Scripts. Enjoy!
Here's the demo:
@ymahlau We also evaluated fdtdx against other popular FDTD solvers and found it to decrease the gap in performance to proprietary solutions such as tidy3d.
See the whitepaper for more information: https://t.co/t7i23qrq47
📢 Introducing fdtdx: A #jax based FDTD solver that combines blazing-fast performance with automatic differentiation for electromagnetic simulations and inverse design!
Have a look at our preprint https://t.co/UvzHyN1iay for more results or watch out for the talk of my colleague @ymahlau at the Photonics West conference in January 2025 https://t.co/Km4so0ErIg