Imagine every pixel on your screen, streamed live directly from a model. No HTML, no layout engine, no code. Just exactly what you want to see.
@eddiejiao_obj, @drewocarr and I built a prototype to see how this could actually work, and set out to make it real. We're calling it Flipbook. (1/5)
I'm Boris and I created Claude Code. Lots of people have asked how I use Claude Code, so I wanted to show off my setup a bit.
My setup might be surprisingly vanilla! Claude Code works great out of the box, so I personally don't customize it much. There is no one correct way to use Claude Code: we intentionally build it in a way that you can use it, customize it, and hack it however you like. Each person on the Claude Code team uses it very differently.
So, here goes.
✨The Prism Hypothesis✨
💫#UAE💫 uncovers a correspondence between an encoder’s feature spectrum and its functional role:
🔠 **Semantic encoders** primarily capture low-frequency components that encode abstract meaning
🎨 **Pixel encoders** additionally retain high-frequency information that conveys fine-grained detail
- Paper: https://t.co/2ldQrZy2Qs
- Code: https://t.co/zzRc4Z2SDA
The authors ask whether an N-layer ViT can be rewritten using just K<<N layers by recurring on them. Remarkably, they match DINOv2 performance with only 2-3 layers. The paper also offers rich dynamical-systems analysis. Very cool work!
🔗https://t.co/G2KgMALnpS
Another month, another round of interesting research papers ranging from large language modeling to computer vision: https://t.co/8cN0GDw1tk
One recent focus is on refining Large Language Models (LLMs). For instance, introducing models like Platypus and the Reinforced Self-Training (ReST) method are the latest attempts to improve alignment with human preferences.
In addition, Self-Alignment with Instruction Backtranslation and OctoPack leverage existing knowledge structures, be it instructions or code, to improve model performances.
A recurring theme is to make models more efficient and accessible. There's LongLoRA with another sparse attention mechanism for LLMs and simpler, more fundamental ideas like replacing softmax with ReLU in vision transformers to boost computational efficiency and pave the way for better parallelization.
But this is only a snapshot of what has happened this month. I hope you get something useful out of the subset of the 22 research highlights I compiled in this month's "Research Highlights in Three Sentences or Less".
v0 by Vercel Labs
Generate UI with simple text prompts. Copy, paste, ship.
Explore the prompt library and join the waitlist today.
https://t.co/yaDdOfnOaJ
Lots of people use AI to summarize articles, but AI summaries can be flabby & miss important info.
This prompt asks the AI to keep increasing the density of a summary producing highly compressed but readable output.
This link will run the prompt for you: https://t.co/KL7ZsUCPdx
The Rise and Potential of LLM Based Agents
This is probably the most comprehensive overview of LLM based agents.
From how to construct these agents to how to harness them for good.
A great read for the weekend.
https://t.co/zH0wCMGEok
Autonomous driving with Chain of Thought - autopilot thinking out loud in text!
LINGO-1 is the most interesting work I've read in autodriving for a while.
Before: perception -> driving action
After: perception -> textual reasoning -> action
LINGO-1 trains a video-language model that comments on the ongoing scene. You can ask it to explain its decisions ("why are you stopped?") and planning ("what are you gonna do next?"). The explicit reasoning step comes with key benefits:
- Explainability: driving models are no longer a mysterious blackbox that you pray for safety.
- Counterfactuals: it's able to imagine scenarios that are not in the training data, and reason through how to handle them correctly.
- Long-tail programming: there are soooo many edge cases in driving. It's impossible to have good data coverage on everything. Instead of collecting 1000s of examples to "neural program" a case, you can now have a human teacher write prompts to explain a handful of examples.
LINGO-1 is closely related to a few works in game AI:
- MineDojo (my team's work at NVIDIA, https://t.co/sYdp8RzTCk): learns a reward model that aligns Minecraft gameplay videos with their transcripts. The model, called "MineCLIP", is able to ground commentary text in the video pixels.
- Thought Cloning (@jeffclune): pixel -> language -> action loop in gridworlds.
人情人心不可靠的几个底层逻辑:
第一,人际关系必须以持续利益交换为基础,但往往大家价值观不一样,都认为自己的付出比对方更多,但实际上你在意的东西,对方往往真诚的认为没有价值。反之亦然。所以很难维持,容易心生怨念。
第二,生米恩,斗米仇。付出太多,对方无法回报,反而会把怨恨作为内心平衡的一块遮羞布。
第三,嫉妒,恨人有,笑人无,是人性深入骨髓的本能,只不过多数时候被按在水下掩盖住而已。就像水下的鳄鱼鲨鱼一样,你看不到,并不意味着危险不存在。
第四,杀人放火金腰带,修桥补路无尸骸。英文有谚语:No good deeds go unpunished. 不管你主观如何好意的去做什么事,被某一群人误解,怨恨,甚至遭到莫名其妙的攻击,是常态。
Vercel 发布了一个新的 AI 辅助设计和编程工具,https://t.co/1myFhB2rpm,你只需要说出你想实现的应用或者产品功能,它就会一次性给你生成多张页面或多个模块,如果你对它生成的内容不满意,你可以把鼠标放到不满意的细节部位,跟它聊天优化。
最终产出的是一个可立即部署的应用,或者可以被用来复用的代码模块,你可以对这个代码模块进行直接的复制&粘贴。
我都已经开始脑补一个熟悉的画面,程序员对着电脑说着话,调教着 AI 完成产品需求的开发,就好像曾经产品经理站在程序员的背后,指着屏幕,嘴里嘟囔着,“改改这个颜色,调调那个按钮”,画面历历在目😂
Deep Neural Nets: 33 years ago and 33 years from now
https://t.co/pbZvYgMJak
My post from last year randomly made it to HN so resharing here too. Maybe in 2055 someone will train an improved GPT-4 on their personal computing device in ~1 min as an irrelevant fun weekend project.
LLM Fine tuning is here!
San Francisco’s top AI engineers came together to see what’s possible with fine-turning and only 4 hours of hacking.
Here’s an exclusive what we saw at the “Anything But Wrappers” hackathon (🧵):
Great conversation with @robertwiblin on how alignment is one of the most interesting ML problems, what the Superalignment Team is working on, what roles we're hiring for, what's needed to reach an awesome future, and much more
👇 Check it out 👇
https://t.co/D9M3NZyOyA