if anthropic keeps doing this cheap thing of nerfing their models before a new release, people will eventually lose trust. and in the meantime some other model, maybe gemini, will catch up and anthropic might never recover from that. i already see a wave of people unhappy with anthropic models and how they communicate
idk for some reason claude models are so difficult to talk to, like their language is from some other dimension and sometimes very difficult to understand. it almost feels like sometimes it doesnt even understand itself so uses some twisted language to pretend it understands. while gemini models have such clarity in their words.
the robot clip isnt the interesting part to me, whats downstream of it is. look at the timecodes in this cube demo the video is labelled as sped up (up to 60x, pauses cut. a trillion-parameter model driving an arm is painfully slow and nobody is going to deploy that to cook a dish or open a bottle cap. thats not the point, and for training it doesnt matter. a rollout is just a sequence of actions and it costs the same whether it took a minute or an hour to record.
so claude can drive an arm through a long task it wasnt explicitly trained on. but the post doesnt say how many attempts it took or how many runs failed. my guess is its not one prompt and one clean pass rather it is several tries and the clip is the one that worked.
thats actually the useful part. a model that succeeds some of the time on open ended tasks is a data generator. run it 5 times keep the 1 that worked and you have a demonstration you didnt need a human teleoperator for. do that across thousands of tasks and collecting robot data stops being the bottleneck.
so the hard problem moves. how do you know which of the 5 rollouts was the good one? "cube is solved" is easy to check. "drew a robot" or "fit the gear and tested the mesh" is not. you need a verifier, and for open ended tasks building the verifier is most of the problem. a rollout that succeeds by luck or by a weird path is poisonous training data.
so i dont think this kills VLAs it changes their job. big general models become the teachers, filtered rollouts become the dataset and a small policy gets trained on it. the winner is whoever builds the best filter not whoever owns the most robots.
Fable 5.2 shows substantially improved skills in controlling physical robots, even on tasks it has never explicitly been trained to do.
On RoboDojo, a benchmark with a simulated two-arm robot, it outperformed other models and most AI systems built to control robots.
my guess is humans mostly dont do it in speech rn as im writing this im not thinking in text per se neither in speech. its something i would call human latent and words come after.
if we really thought in speech explaining things would be easy but people find it difficult time and again to put what they think into words. and people who are born deaf think just fine without ever hearing speech.
thats why i wrote speech isnt really the medium we think in.
my guess is humans mostly dont do it in speech rn as im writing this im not thinking in text per se neither in speech. its something i would call human latent and words come after.
if we really thought in speech explaining things would be easy but people find it difficult time and again to put what they think into words. and people who are born deaf think just fine without ever hearing speech.
thats why i wrote speech isnt really the medium we think in.
the main problem with current full duplex models is that almost everyone is iterating on the same moshi architecture which imo has some real bottlenecks.
in moshi every text token is locked 1:1 to an audio frame that gets spoken right away (apart from padding and special tokens). because of this rigid lockstep the native architecture cant generate a burst of text without advancing audio time. if the model generates 20 text tokens time has to move forward 20 frames so exactly 1.6 seconds of audio gets synthesized alongside. you cant pause the audio clock to let the LLM write out a block of text.
within base moshi you could in theory train it to output reasoning text or JSON tool calls while forcing the audio stream to output silence tokens. but since the clock cant be decoupled that "silent thinking" still runs at 12.5 tokens per second. a 50 token thought chain or tool call means the user sits through 4 solid seconds of awkward dead air.
the other option is synthetic data where it keeps talking while producing the JSON or the thinking. but there is no data in the wild for that so you generate a lot of it and now your data is the bottleneck. the voice will only ever be as natural as the most natural TTS you can use it will always carry the quirks of the harness you built to create it and if whoever is making that data cant think through the scenarios of dyadic conversations well, you are cooked.
so people came up with delegation system which is a pain in itself. you train the frontend model to be smart enough to hold a decent conversation and to hand off tool calling or thinking to a much larger, more capable model. this is engineering and its the only thing that has made the 2 or 3 full duplex models we have like openai gpt live possible at all.
thinking cant be done in speech. the signal to noise ratio is low and there isnt enough speech data to give the model a good enough world model in its weights. how much data is there of people doing maths out loud? even we humans dont think in speech. yes there is "thinking out loud" but speech is not the medium humans think in. thinking or reasoning needs a dense representation like text or maybe some latent but pure speech is a fuck all dream.
i saw dessa a few days ago. it seems good and sounds good but god knows how much data they trained it on and even it needs text, or some good engineering like ASR over its thinking speech, to actually interact with the real world or be even remotely useful. train on all the call centre data u want then ask it a basic question.
this also reminds me of a question from @juliarturc something like: if you converted every text available into speech and trained on that, what would happen? idk it might be smart or it might not. it depends on a lot of things like whether the speech encoder you use preserves everything needed for speech understanding. codecs are good enough these days but i believe they are still a bottleneck.
even if you solve that,context window is a problem and so is context degradation in speech compared to text LLMs current speech models have the memory of a goldfish.
there are just too many bottlenecks but at the rate we are going we might get there next year only if a lot of people come out in the open and open source the data they are sitting on and stop giving fake demos by training moshi on their privately acquired data.
@anders_thuesen@duplexio_ai yeah separate clocks seems like the right fix rubato showed the thinking stream can run at hardware speed without messing up the audio timing. are you guys planning to do a tech report on duo? eager to read
@dimn42 this is pretty close to duplexSLA they give planning and tool calls their own action channel next to the speech up to 10 action tokens per 160ms chunk so it thinks and calls tools without stopping the audio.
Thought experiment:
1. Pass the entire Internet through a TTS model.
2. Make a speech corpus that is a strict superset of whatever SoTA LLMs are trained on today.
3. Train an e2e speech-to-speech voice assistant.
Would that be better than current LLM-based voice assistants?
@posedscaredcity caramel is just moshi with the inner monologue removed just a user stream and an agent stream of speech tokens. so where does the text in yours come from? did you add back a text stream aligned with its own speech like moshis inner monologue?