Is the speaker token really what keeps a voice consistent throughout an utterance?
The answer turned out to be far more interesting than a speaker tag. https://t.co/45enZPzmp8
you can get (somewhat) rid of the causal dependency between the codebook estimation for the nth codebook on the prev 1,2 ... n-1 codebooks in your standard RVQ setting by having a sort of global-local-global design while doing feature extraction from the incoming signal (1/n)
@_Abhinavgpt_@charanxcodes Yep it happened again two times last week. Idk what is there problem, whole reddit is flooded with these kind of reports that meta is banning account for no reasons... Ig some new intern has joined there team🤡
the main problem with current full duplex models is that almost everyone is iterating on the same moshi architecture which imo has some real bottlenecks.
in moshi every text token is locked 1:1 to an audio frame that gets spoken right away (apart from padding and special tokens). because of this rigid lockstep the native architecture cant generate a burst of text without advancing audio time. if the model generates 20 text tokens time has to move forward 20 frames so exactly 1.6 seconds of audio gets synthesized alongside. you cant pause the audio clock to let the LLM write out a block of text.
within base moshi you could in theory train it to output reasoning text or JSON tool calls while forcing the audio stream to output silence tokens. but since the clock cant be decoupled that "silent thinking" still runs at 12.5 tokens per second. a 50 token thought chain or tool call means the user sits through 4 solid seconds of awkward dead air.
the other option is synthetic data where it keeps talking while producing the JSON or the thinking. but there is no data in the wild for that so you generate a lot of it and now your data is the bottleneck. the voice will only ever be as natural as the most natural TTS you can use it will always carry the quirks of the harness you built to create it and if whoever is making that data cant think through the scenarios of dyadic conversations well, you are cooked.
so people came up with delegation system which is a pain in itself. you train the frontend model to be smart enough to hold a decent conversation and to hand off tool calling or thinking to a much larger, more capable model. this is engineering and its the only thing that has made the 2 or 3 full duplex models we have like openai gpt live possible at all.
thinking cant be done in speech. the signal to noise ratio is low and there isnt enough speech data to give the model a good enough world model in its weights. how much data is there of people doing maths out loud? even we humans dont think in speech. yes there is "thinking out loud" but speech is not the medium humans think in. thinking or reasoning needs a dense representation like text or maybe some latent but pure speech is a fuck all dream.
i saw dessa a few days ago. it seems good and sounds good but god knows how much data they trained it on and even it needs text, or some good engineering like ASR over its thinking speech, to actually interact with the real world or be even remotely useful. train on all the call centre data u want then ask it a basic question.
this also reminds me of a question from @juliarturc something like: if you converted every text available into speech and trained on that, what would happen? idk it might be smart or it might not. it depends on a lot of things like whether the speech encoder you use preserves everything needed for speech understanding. codecs are good enough these days but i believe they are still a bottleneck.
even if you solve that,context window is a problem and so is context degradation in speech compared to text LLMs current speech models have the memory of a goldfish.
there are just too many bottlenecks but at the rate we are going we might get there next year only if a lot of people come out in the open and open source the data they are sitting on and stop giving fake demos by training moshi on their privately acquired data.
People have yet to realise just how powerful this is and the possibilities it unlocks for solving real consumer problems. We’ve invested deeply in voice technology, building our own speech models in-house, so this use case is a natural fit for us. @iradotso ftw
trust me guys, this one is the coolest thing i have ever tested. as a self proclaimed introvert, who doesn't likes to talk to people on call. she called the PG owners and brokers on my behalf and talked to them and negotiated the deal.... and hunted PG for me end to end (which seems like a nightmare task for me).
it reminds me my tasks lined up for the day, or any important mails or msges for me that i need to see etc. just as a personal assitent. we had LLMs and agents before this but nothing felt so real.
obv there are alternates in market rn, i have tested them too, but this somehow feels soo comfortable to use. and does the tasks properly.
@iradotso FTW
i was one of the early members and can i just say @iradotso is awesome . she helps me with numerous things like every morning she goes to arxiv and find papers for me and gives me a good read .
one time my furlenco carpenter didnt show up and she chased him to make sure he came the next day all by herself. she can call people on behalf of u and email and handle everything.
(1/n) i'm putting out sneak peeks of a new neural audio codec that i've been working on recently.
it matches @kyutai_labs mimi on ~1200x less data.
5,770 hours of english audio. no chinese, hindi, or japanese in training. yet it matches mimi's STOI on all four languages.
I built a port of Microsoft’s X++ windows only compiler for Apple Silicon
While building at https://t.co/7tG0Yat5uc. I couldn’t find an open-source X++ compiler port for macOS, so I built one of my own🔥
Still expanding support. Contributions welcome: https://t.co/0NkuGy9u40