“There is a very current view in our world that philosophy should be left to the philosophers, sociology to the sociologists, and death to the dead. I believe this is one of the great heresies – and tyrannies – of our time.”
The Aristos. A Self-Portrait in Ideas
John Fowles
@Neuralwarlock With a clever design and a good drafter one can easily scale it up to hundreds of tps. So in one 80ms tick model can spit 10-20 tool call or reasoning tokens and fill other channels with special token to sync all channels.
@Neuralwarlock I’m trying to solve this problem.
The idea is to create separate channel and lm head for tool call and reasoning and generate multiple tokens per tick and prefill other layers with special token.
Model itself needs to be trained to continue the conversation on those prefills
@redrodeo03 Use icefalls guides on trt-llm. It builds trt engines with some optimizations. Afaik it borrows some stuff from distill-whisper repo.
With my measurements it was under 100ms wall time for all 8 on 5090. I might have measured medium and large v3, it was a year ago to be fair.
@bojie_li Most of the takes are not very accurate. Duplex architecture like moshi/personaplex vs more traditional duplex design like interleaved TDM approach. Higgsfield is most likely is trained interleaving user/agent audio+reasoning and tool calls.
@bojie_li I’m most excited about the temporal transformer architecture like persoplex. Nvidia recently released their version: Nemotron voicechat. I think pretrained backbones fit pretty well with this approach. The bigger problem is good and large duplex audio datasets.
@bojie_li Interleaving approach is indeed expensive and context will fill up pretty fast. Moshi has very cool approach-it uses parallel streams at fixed step and you can run up to 400 parallel streams in single A100. It runs it at 80ms step it takes only 45k token positions per hr.
@nemumusitocha Turns out, it sends large time-frequency grid thru 30 blocks and 120 mamba scans. Those are sequential work that can’t be parralelized like matrix multiplications.
@nemumusitocha It’s little slow for such a tiny model. For example sidon is a 200m or 350m (don’t remember the exact param size) can reach to 350x realtime on 5090 and this small ~10m model is 55x and best I could do is 154x realtime with some kernel fusions and casting some layers to bf16.
@itzhaydar What exactly you don’t understand? How are you gonna run these open weight models? You need massive compute unit to use them in the same level you use Claude or codex. It’s not cheaper for sure and they still can’t compete with astra/opus
@Spitamenes001@yarbatman What did Tajiks do in those cities? They didn’t build shit. Timurids built these beautiful places and Tajiks as always claim what’s not theirs lol!
@Bluehopium@Asilbek_787@SSharifava@SecRubio Most of the architecture was built during the ruling of Timurids. How is it exclusively Persian? This style is consistent in other cities like Bukhara and ancient Kesh.
@SSharifava@SecRubio As usual tajik complex kicks in lol. Samarkand had 2700+ years of history. It was especially flourishing and most of the current architecture was built by Timurids, direct ancestors of Uzbeks. So you can go and fuck yourself you little tajik retard.