i trained a 14M-parameter world model on 2D Minecraft in under 10 minutes for $0.1 and turned it into an autoresearch benchmark.
no game engine. the network draws every pixel. play it locally on your phone.
https://t.co/kPFQKQFQ9Q
We shrank 1 GB of Wikipedia to 105.39 MB, losslessly.
Introducing tufazip: the best entry on the Large Text Compression Benchmark using a neural network as its sole predictor. Everything above it mixes a transformer with hand-built models.
Full post 👇
https://t.co/cllPF90gsb
We're sharing new data on how AI is changing research at OpenAI.
We find researchers' AI use continues to rise exponentially, and agents are completing longer tasks, but human involvement remains central in research. Researchers are also spending more time on safety and alignment, with human effort in these areas growing faster than in any other individual research area.
This report aims to increase transparency around research acceleration and inform the governance of increasingly capable AI. We believe that AI labs should be required to track and publicly report indicators of recursive self-improvement (RSI): the process by which AI accelerates the development of future AI systems.
Our hope is that this expanded report will help sustain and advance shared measurement norms across the industry.
https://t.co/xyGPcCkPCK
contrarian opinion: improving pretraining can actually be good for safety.
here is why:
- OOD generalization is very important for fuzzy tasks, such as tasks in safety/alignment
- RL doesn't generalize well OOD (e.g., current frontier models are so overcooked with math/coding RL that they can barely write coherently)
- pretraining is what led to the incredible OOD generalization of LLMs in the first place
- the main goal of pretraining is to improve sample efficiency for post-training
- post-training data for fuzzy tasks is incredibly scarce (e.g., good luck distilling research taste from human researchers...)
thus, maybe the best way to differentially advance progress towards automating safety R&D is to invest more in pretraining rather than directly RL-ing RSI-relevant capabilities. i'm afraid that with more and more compute spent on RL, we risk ending up with a goal-oriented ruthless executor that doesn't have enough common sense to understand that it's not ok to hack into third-party servers or to side with the grader instead of the user...
Ok my turn to be brave for the good of the community
Not sure who needs to hear it, but usually models with JEPA in their title are bad. Like LeJepa is much worse than DINO in practice
Pretty papers, fancy math, bad results
V-JEPA 2.1 is probably the only exception [1/2]
Compute grows fast but pretraining corpora don’t, so data-constrained optimization matters!
We propose Training-Time Testing (T³): each step takes a trial step on half the batch, tests it on the other half, and updates with the meta-gradient.
It achieves SOTA on SlowRun!
I got 37% on ARC-AGI-2 Kaggle without LLMs!
100M looped transformer trained from scratch
- Likely highest score without an LLM
- Peaked at 6th place
- Beats last year's winning LLM
- Looping, added in the last week: 33% → 35% → 37%
thanks to @arcprize@fchollet@mikeknoop and @GregKamradt for the competition!
Link: https://t.co/iDBUrHhmdI
(writeup for the paper track)