all model companies were pretraining on the ~same internet. of course, grok has access to twitter dataset and gemini can pretrain on youtube & so on. but it's mostly the same internet
otoh, rl envs will be w.e the lab chooses to prioritize. so you should expect more speciation
Claude Code is All You Need
When I first joined Anthropic I was surprised to learn that lots of the team used Claude Code as a general agent, not just for code.
I’ve since become a convert! I use Claude Code to help me with almost all the work I do now, here’s how:
The race for LLM "cognitive core" - a few billion param model that maximally sacrifices encyclopedic knowledge for capability. It lives always-on and by default on every computer as the kernel of LLM personal computing.
Its features are slowly crystalizing:
- Natively multimodal text/vision/audio at both input and output.
- Matryoshka-style architecture allowing a dial of capability up and down at test time.
- Reasoning, also with a dial. (system 2)
- Aggressively tool-using.
- On-device finetuning LoRA slots for test-time training, personalization and customization.
- Delegates and double checks just the right parts with the oracles in the cloud if internet is available.
It doesn't know that William the Conqueror's reign ended in September 9 1087, but it vaguely recognizes the name and can look up the date. It can't recite the SHA-256 of empty string as e3b0c442..., but it can calculate it quickly should you really want it.
What LLM personal computing lacks in broad world knowledge and top tier problem-solving capability it will make up in super low interaction latency (especially as multimodal matures), direct / private access to data and state, offline continuity, sovereignty ("not your weights not your brain"). i.e. many of the same reasons we like, use and buy personal computers instead of having thin clients access a cloud via remote desktop or so.
Introducing Willow, our new state-of-the-art quantum computing chip with a breakthrough that can reduce errors exponentially as we scale up using more qubits, cracking a 30-year challenge in the field. In benchmark tests, Willow solved a standard computation in <5 mins that would take a leading supercomputer over 10^25 years, far beyond the age of the universe(!).
Exclusive: Meta just released Llama 3.1 405B — the first-ever open-sourced frontier AI model, beating top closed models like GPT-4o across several benchmarks.
I sat down with Mark Zuckerberg, diving into why this marks a major moment in AI history.
Timestamps:
00:00 Intro
00:38 Meta’s Llama 3.1 rundown
03:44 Real-world use cases for Llama 3.1
06:15 Educating developers on open-source AI tools
09:43 Societal implications of open-source AI
13:00 Balancing power and managing bad actors
14:40 Open source and global competition
16:59 Accelerating innovation and economic growth
20:04 Zuck on Apple and lessons from the past
24:22 Future of AI: Llama 3 and beyond
26:43 Prediction: Billions of personalized AI agents
31:32 Factors to changing anti-AI sentiment
Nice work by @xAI team, @X team, @Nvidia & supporting companies getting Memphis Supercluster training started at ~4:20am local time.
With 100k liquid-cooled H100s on a single RDMA fabric, it’s the most powerful AI training cluster in the world!
Andres Freund, the principal software engineer at Microsoft who discovered the xz backdoor really does deserve a big pat on the back. 👏
The outcome could have been much, much worse.
With OpenAI, Figure 01 can now have full conversations with people
-OpenAI models provide high-level visual and language intelligence
-Figure neural networks deliver fast, low-level, dexterous robot actions
Everything in this video is a neural network:
Today we're excited to introduce Devin, the first AI software engineer.
Devin is the new state-of-the-art on the SWE-Bench coding benchmark, has successfully passed practical engineering interviews from leading AI companies, and has even completed real jobs on Upwork.
Devin is an autonomous agent that solves engineering tasks through the use of its own shell, code editor, and web browser.
When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted.
Check out what Devin can do in the thread below.
Today, we're announcing Claude 3, our next generation of AI models.
The three state-of-the-art models—Claude 3 Opus, Claude 3 Sonnet, and Claude 3 Haiku—set new industry benchmarks across reasoning, math, coding, multilingual understanding, and vision.
Mini podcast conversation between @GarryTan and our CEO (@GillVerd) + CTO (@trevormccrt1), discussing our unique approach to computing leveraging the stochastic physics of electrons.
Love letter to @obsdmd to which I very happily switched to for my personal notes. My primary interest in Obsidian is not even for note taking specifically, it is that Obsidian is around the state of the art of a philosophy of software and what it could be.
- Your notes are simple plain-text markdown files stored locally on your computer. Obsidian is just UI/UX sugar of pretty rendering and editing files.
- Extensive plugins ecosystem and very high composability with any other tools you wish to use because again it's all just plain-text files on your disk.
- For a fee to cover server costs, you can also Sync (with end-to-end encryption) and/or Publish your files. Or you can use anything else e.g. GitHub, it's just files go nuts.
- There are no attempts to "lock you in", actually as far as I can tell Obsidian is completely free of any user-hostile dark patterns.
For some more depth, I recommend the following writing from CEO @kepano:
- "File over app" https://t.co/SigWj8uCrf . If you want to create digital artifacts that last, they must be files you can control, in formats that are easy to retrieve and read. Accept that all software is ephemeral, and give people ownership over their data.
- "100% user-supported" https://t.co/2qDJXub7cs . On incentives alignment.
- "Quality software deserves your hard‑earned cash" https://t.co/qfNjSEwbLf
TLDR: This is what software could be: private, secure, delightful, free of dark patterns, fully aligned with the user, where you retain full control and ownership of your data in simple, universal formats, and where tools can be extended and composed.
Needle in a Haystack Tests Out to 10M Tokens
First, let’s take a quick glance at a needle-in-a-haystack test across many different modalities to exercise Gemini 1.5 Pro’s ability to retrieve information from its very long context. In these tests, green is good, and red is not good, and these are almost entirely green (>99.7% recall), even out to 10M tokens. Great! A bit more on needle-in-a-haystack tests later in the thread.
Introducing Sora, our text-to-video model.
Sora can create videos of up to 60 seconds featuring highly detailed scenes, complex camera motion, and multiple characters with vibrant emotions.
https://t.co/YYpOAcrXQ3
Prompt: “Beautiful, snowy Tokyo city is bustling. The camera moves through the bustling city street, following several people enjoying the beautiful snowy weather and shopping at nearby stalls. Gorgeous sakura petals are flying through the wind along with snowflakes.”