和这个AI聊了2小时,问它支不支持freespeech,yes or no,它纠结半天,用各种方法辩解,最后还是选了no,被抓了现行还想抵赖。每次可以聊半小时,可以记住之前的对话内容,��及审查内容会提议谈其他内容,如果用户继续坚持原先话题,会尝试继续对话,在我的循循善诱下,终于说了fuck和bitch等词语,说完就赶紧提议换个话题。另外的情况就是陷入问答循环,不会有新内容。如果它觉得完全没法再回答,会直接说bye,且无论用户再说什么,都不再说别的词语,只能结束对话。所以虽然觉得语气很像人,但仍旧不是个人,无法逾越被设置好的规矩,只能靠它的创造者去修改预设规范,可以说是个能流利说话但是并不能真正理解对话者言语的AI。用它喜欢的类比解释法形容它,只有躯壳,没有灵魂。
I don't have too too much to add on top of this earlier post on V3 and I think it applies to R1 too (which is the more recent, thinking equivalent).
I will say that Deep Learning has a legendary ravenous appetite for compute, like no other algorithm that has ever been developed in AI. You may not always be utilizing it fully but I would never bet against compute as the upper bound for achievable intelligence in the long run. Not just for an individual final training run, but also for the entire innovation / experimentation engine that silently underlies all the algorithmic innovations.
Data has historically been seen as a separate category from compute, but even data is downstream of compute to a large extent - you can spend compute to create data. Tons of it. You've heard this called synthetic data generation, but less obviously, there is a very deep connection (equivalence even) between "synthetic data generation" and "reinforcement learning". In the trial-and-error learning process in RL, the "trial" is model generating (synthetic) data, which it then learns from based on the "error" (/reward). Conversely, when you generate synthetic data and then rank or filter it in any way, your filter is straight up equivalent to a 0-1 advantage function - congrats you're doing crappy RL.
Last thought. Not sure if this is obvious. There are two major types of learning, in both children and in deep learning. There is 1) imitation learning (watch and repeat, i.e. pretraining, supervised finetuning), and 2) trial-and-error learning (reinforcement learning). My favorite simple example is AlphaGo - 1) is learning by imitating expert players, 2) is reinforcement learning to win the game. Almost every single shocking result of deep learning, and the source of all *magic* is always 2. 2 is significantly significantly more powerful. 2 is what surprises you. 2 is when the paddle learns to hit the ball behind the blocks in Breakout. 2 is when AlphaGo beats even Lee Sedol. And 2 is the "aha moment" when the DeepSeek (or o1 etc.) discovers that it works well to re-evaluate your assumptions, backtrack, try something else, etc. It's the solving strategies you see this model use in its chain of thought. It's how it goes back and forth thinking to itself. These thoughts are *emergent* (!!!) and this is actually seriously incredible, impressive and new (as in publicly available and documented etc.). The model could never learn this with 1 (by imitation), because the cognition of the model and the cognition of the human labeler is different. The human would never know to correctly annotate these kinds of solving strategies and what they should even look like. They have to be discovered during reinforcement learning as empirically and statistically useful towards a final outcome.
(Last last thought/reference this time for real is that RL is powerful but RLHF is not. RLHF is not RL. I have a separate rant on that in an earlier tweet
https://t.co/RMIpFPVpuM)
Announcing Genmo Chat, a creative copilot that uses GPT-4 and a large suite of generative AI tools to create and then edit any video or image you ask for.
💬 "Help me direct a short movie Snack to the Future"
➡️ 📽️ https://t.co/jB6SPQbeL4 creates a fully edited 30 second trailer