backchannel

The next wave of AI is real-time—voice, video & the physical world. Follow the builders shaping it, what they’re building & how it works. Presented by @agoraio.

[backchannel @heybackchannel]

Backchannel reporting on backchannels—this is the kind of meta-story we chose our name for.
Thanks, @freemanjiangg, for giving us the opportunity. Couldn’t have scripted it better.

[backchannel @heybackchannel]

@trentct https://t.co/tyruJh9b1u
we also curated some

[backchannel @heybackchannel]

Curious where Jev fits in real-time conversational agents.

Could turn-taking be a good use case? What other real-time decisions do you think Jev is a good fit for?

[backchannel @heybackchannel]

Real-time 3D scene generation:

@op7418 built a real-time 3D scene generator with Jev.

From a library of dozens to hundreds of prefab 3D assets, Jev makes hundreds of parallel decisions based on the prompt, then helps determine object selection, placement, lighting, color, and state — enough to assemble an indoor scene in about a second.

https://t.co/gbHnczo6Mq

[歸藏(guizang.ai) @op7418]

我用 @typesafeai JEV 模型做了一个实时 3D 场景生成器。

这个速度实在太夸张了！

从预制的几十上百种 3D 模型里，根据输入的文案，进行并发几百次的判断。

同时对这些素材的着色、光照、位置以及状态进行处理，一秒完成符合要求的室内场景搭建。

[backchannel @heybackchannel]

Curious where Jev fits in real-time conversational agents.

Could turn-taking be a good use case? What other real-time decisions do you think Jev is a good fit for?

[backchannel @heybackchannel]

Voice agents acting on partial speech:

@BhosalePratim replaced the LLM-based tool-calling decision with Jev in a voice agent experiment.

The idea: make decisions on partial transcripts instead of waiting for the user to finish speaking, potentially enabling earlier actions at lower cost.

https://t.co/NILR5OS1ck

[Pratim Bhosale @BhosalePratim]

Played around with @typesafeai Jev today, mostly to understand what it does for tool calling.

Instead of an LLM deciding what to do, I substituted that part with Jev.

My learning is that we will be able to make the agent act before the user finishes the sentence. So far, we've used different LLM combinations (non-thinking + thinking ) plus state machine setups to get the right experience for the end user.

TLDR is to understand more about how voice agents can run decisions on partials instead of waiting for the end of the turn, with lower costs.

[backchannel @heybackchannel]

Lokutor, built by Daniel Varela and team (@lokutorAI), is taking a CPU-native approach to voice agents.

Its noise suppression, turn detection, STT, and TTS can run locally on ordinary CPUs; the LLM is bring-your-own.

The stack targets cloud, on-prem, and on-device deployment.

Lokutor reports ≈1.2–1.3 s to first audio for a full turn, including turn detection, and ≈120 ms for streaming TTS alone.

https://t.co/8nckBPDULb

#BackchannelSignals

[backchannel @heybackchannel]

UNMUTE// is trying to make voice agents portable across runtimes.

Instead of defining an agent inside one specific framework, you describe it in YAML + Markdown: its instructions, models, tools, state, tasks, and turn-taking behavior.

UNMUTE then compiles that spec into a runnable agent project.

The useful part is the separation: the agent definition no longer has to be the same thing as the runtime executing it. You can keep the behavior and workflow in one spec, then change where and how the agent runs separately.

https://t.co/qTuy17cTzH

#BackchannelSignals

[slng @slng_ai]

UNMUTE// is out. An open standard for voice agents.

No amount of routing or caching helps if the agent was never separable from the framework running it.

So we wrote it down instead, and gave it away.

@unmute_ai is open and MIT licensed, with three compile targets - @pipecat_ai, @livekit agents and SLNG.

[backchannel @heybackchannel]

Most speech APIs tell an agent what you said. @nrol_ling and @OrukLabs are building Oruk to preserve more of how you said it.

Send in speech and Oruk can return the transcript alongside vocal context such as frustration, hesitation, warmth, sarcasm, or excitement.

That makes distinctions a transcript alone can flatten:

“I’m not sure what to do next.”

The words may be identical, while the delivery can sound hesitant, frustrated, or calm—and an agent may want to respond differently to each.

Oruk’s file API combines transcription with emotion and speaking-style labels. Its Realtime API is currently in preview, streaming transcript tokens alongside phrase-level emotion scores.

Importantly, Oruk frames these as descriptions of vocal delivery, not claims about someone’s actual feelings or internal state.

You can try it with your own voice: https://t.co/baHMwww4aU

#BackchannelSignals

[Nathan Roll @nrol_ling]

it's time your speech models actually understood you

voice is the only interface fast enough to keep up with your agents, robots, and claude code, but they need more context.

@OrukLabs transcribes text, tone-of-voice, emphasis, intent, and emotion

[backchannel @heybackchannel]

Camera + mic → ambient control

@reczko_konrad is feeding live camera and microphone context into Jev to control lighting and bloom effects in real time.

Instead of waiting for an explicit command, the system reacts to the semantics of what’s happening around it.

https://t.co/PIf2iGgOdw

[Konrad Reczko @reczko_konrad]

TypeGPU + ruNNtime + Jev @typesafeai is a very fun combo :D

ruNNtime gives me efficient local inference, TypeGPU lets inference and rendering share GPU resources directly with zero copy. That’s 3 separate NN inferences plus rendering, all happening in realtime

Since we control the pipeline, Jev can just sit in the middle and add the semantic bit.

camera + mic → Moonshine + YOLO26 + DepthART → Jev → lights, shadows and bloom

[backchannel @heybackchannel]

@jackcheng is using Jev to resolve multimodal intent from speech + gesture.

Instead of treating “put that there” as a voice command alone, Jev uses the live gesture context to understand what “that” and “there” refer to.

https://t.co/bw58osede9

[Jack Cheng @jackcheng]

Jev is the future

[backchannel @heybackchannel]

@eandualem is using Jev to drive real-time body responses for a 3D robot avatar during voice conversations.

Reported latency drops from roughly 3–6s to ~0.5s, making the character’s movement feel much more connected to the conversation.

https://t.co/4yCWScXhKA

[Elias Andualem @eandualem]

Decision speed in agentic systems, shown on a robot.

In agentic systems, a lot of the work is turning semantic understanding into decisions: which pipeline, act now or wait, stop or continue. Doing that with an LLM works, but each decision costs seconds, and those seconds land in front of everything else.

Jev, the new System One model from TypeSafe AI, is built for that layer. State and typed questions in, calibrated decisions out in well under a second. No text to parse.

Charlie is the demo: a 3D robot in a browser voice call. GPT-Live carries the conversation and only talks. A separate model decides what the body does, and that model was the latency: 3 to 6 seconds between the words and the movement, never in sync.

With Jev as the decision layer, it answers eight typed questions on every fragment of speech in one call, while I am still talking. The body starts about half a second after the words, often before the sentence ends.

Jev is early access.
Avatar Studio: https://t.co/NxNkUyVLgB
Jev: https://t.co/HopK5hqS0O

#Jev #TypeSafeAI #AgenticAI #AIAgents #DecisionModels #OpenSource @typesafeai

[backchannel @heybackchannel]

@uezochan is testing Jev as a turn-end gate for conversations with an AI character.

After an utterance ends, Jev makes a judgment roughly 0.5 seconds later and adjusts the additional hold time based on its score. Jev itself is returning the decision in around 200ms.

The experiment is still early, but he says the judgments are already close to what he expects — potentially even better than a dedicated turn-end model.

https://t.co/J0FLqHlDn5

[うえぞう@うな技研代表 @uezochan]

Jevで音声対話のターンエンド判定やってみた。発話終了後0.5秒後にJevでターンエンドしたかどうかを判定して、スコアに応じて追加ホールド。
timeout=3.0とか出てるのが追加ホールド時間。Noneは追加なし。elapsed=0.224とか出てるのがJevの処理時間。

まだまだ追い込みがいるけど可能性を感じる

[backchannel @heybackchannel]

@tsuyoshi_osiire is experimenting with Jev as a real-time behavioral signal layer.

Instead of waiting for an explicit click or request, the system looks at interaction signals such as mouse movement and hesitation, then uses Jev to infer what the user may need and react proactively.

The interesting part is less the classification itself than putting semantic judgment directly inside the UI interaction loop.

https://t.co/GOlG9OefMT

[かねこつよし @tsuyoshi_osiire]

Jevによるリアルタイム接客の試作📝

クリックする前のカーソルの往復や滞在から、迷いや関心をAIで推定。
お客さんの様子をつぶさに見て、状態を判断。
説明を補ったり、比較を提案したり、そっと見守ったりする。

パラメーターを頑張れば、百貨店の店員さんを再現できそう。

[backchannel @heybackchannel]

What happens when a ~300ms decision model sits inside a real-time voice loop?

@moritzkremb connected voice input to Jev: speak → transcript → Jev decides → browser acts.

He reports ~$0.0002 per decision — and at one point the browser finished “go back” before he finished saying it.

A nice glimpse of where @typesafeai’s Jev could fit in real-time AI.

https://t.co/MnX8g06EA8

[Moritz Kremb @moritzkremb]

whoa this actually worked! Jev lets me control my browser in real time with my voice now

> i talk
> transcript sent to Jev
> jev returns probabilities in ~300ms
> browser clicks

costs: $0.0002 per decision

i'm stunned how fast this is. when i asked it to "go back", it even finished the request before i finished my sentence 😂

[backchannel @heybackchannel]

Most insurance claim intake still feels like filling out a form—even when there’s a voice agent on top.

@Saboo_Shubham_ built a different kind of demo with Gemini 3.8 Live: the conversation builds a field notebook in real time.

You can describe what happened by voice, turn on the camera to show the damage, and watch the agent pin evidence photos, record what it sees in the camera feed, and even sketch the incident for you to confirm.

At the same time, policy lookup, missing-information checks, routing, and the adjuster packet can run in the background without stopping the live conversation. Safety or injury signals can also trigger escalation to a human.

It’s a good example of voice agents moving beyond a “talking UI” into a workflow that can listen, look, document, and move the claim forward at the same time.

Open-source code: https://t.co/2NIgQNN6u4

#BackchannelSignals

[Shubham Saboo @Saboo_Shubham_]

My mind is f*** blown.

I built this live insurance claim agent that can see, talk, think and draw in real-time using the new Gemini 3.8 LIVE.

Even switched my language to Hindi mid-call and it still worked.

VOICE AI can't be more real. Made it 100% open-source.

[backchannel @heybackchannel]

Streaming TTS can start speaking before the sentence is finished—but once the audio starts playing, how does the system know exactly which part of the text has actually been spoken?

@XSquareRobot’s X2-NativeCursor tracks that progress directly from the TTS model’s own codec tokens, before waveform decoding, instead of running a second acoustic aligner over the generated audio.

A lightweight 2M-parameter observer emits a monotonic cursor back into the source text, so applications can keep text highlighting, interruption accounting, subtitles, and dialogue history aligned with what the user has actually heard.

That’s a useful missing piece for streaming voice systems: generating speech in realtime is one problem; knowing exactly where playback is in the text is another.

https://t.co/H7f0BJv1fz

#BackchannelSignals

[X Square Robot @XSquareRobot]

Most "streaming" TTS systems still wait for a complete sentence before speaking.

X2Streaming-TTS removes that wait. It consumes text tokens as they arrive and generates speech with strict zero lookahead.

The challenge: once speech is played, it cannot be revised.

Take “He finished 3…”—should “3” become “three” in “3 laps,” or “third” in “3rd place”? Speaking too early risks an error; waiting defeats the purpose of streaming.

X2Streaming-TTS addresses this with:

1️⃣ Causal commitment
Ambiguous numbers, units, and symbols are held until their pronunciation becomes clear. Segments are closed using both punctuation and acoustic capacity.

2️⃣ Speech-state inheritance
Waveform-decoder state and acoustic history are carried across segments, preserving pitch, timbre, and continuity.

Results:

⚡ 15.8 ms median TTFT for one request
⚡ Under 120 ms at 64 concurrent requests
🎯 Lowest recognition error in 6/8 evaluated streaming conditions
🔢 0% CER on numeric and streaming-ambiguity tests
🗣️ Quality comparable to evaluated offline baselines

Low-latency voice AI is not just about speaking faster—it is about knowing when it is safe to speak.

Paper: https://t.co/J9ilrTz0Jw
Code: https://t.co/MfkFB8a1nc

#TTS #VoiceAI

[backchannel @heybackchannel]

@nischal @egma_ai @theBhulawat Saw your Show HN and couldn’t resist! Excited that Egma will support Agora soon—DM us anytime if we can help.

[backchannel @heybackchannel]

Voice agents need CI—not humans repeatedly redialing the same script.

@egma_ai, built by brothers @nischal and @theBhulawat, is an open-source platform for testing voice agents and monitoring them in production.

Teams keep scenarios as Markdown beside the agent code, run simulated voice or text callers, mock tool results, and grade expected behavior from the CLI or a coding agent.

• BYOK for STT, LLM, and TTS, with no markup on model usage
• Production monitoring across conversations, tool calls, and metrics
• Managed cloud or self-hosting via egma self-host up

Voice-agent QA is becoming a repo-native engineering loop—versioned, repeatable, and runnable before every release.

https://t.co/xDGsQsG0kO

#BackchannelSignals

[backchannel @heybackchannel]

Physical AI doesn’t have to be a robot: @WarnerTeddy’s AvianVisitors turns local bird-call classification into a quiet e-ink object for the home.

A Raspberry Pi, USB microphone, and BirdNET-Pi identify visiting birds by sound, then render the detections as both a live web collage and a 13.3-inch @pimoroni color e-ink frame.

The wooden-framed display shows the birds heard over the past 24 hours. It checks for updates every 15 minutes and refreshes only when a new detection changes the collage. Without a local microphone, it can also follow nearby BirdWeather stations.

The broader platform includes:

- 666 Gemini-generated kachō-e illustrations covering 333 species
- Bird-call recordings and detection statistics
- A stamp-style species atlas that fills as new birds appear
- Regional filtering through eBird
- An educator mode for organized listening sessions
- Home Assistant and MQTT integrations

Instead of producing another notification stream or glowing dashboard, AvianVisitors makes the invisible environment quietly visible.

Code: https://t.co/ZHZuLl3iGo
Live garden: https://t.co/1CIjDRn5SK
#BackchannelSignals

[backchannel @heybackchannel]

Most TTS benchmarks ask whether a voice sounds natural. Voice Acting Arena asks whether it can act.

@realmrfakename and @laion_ai have launched Voice Acting Arena: two anonymous models perform the same scene with the same acting brief, and listeners pick which performance works better.

It’s not just about pronunciation or voice quality. Can the model follow the direction? Sell the scene? Laugh, gasp, or sob at the right moment?

It pushes TTS evaluation beyond “does this sound human?” toward “does this performance fit the scene?”

It’s live now—pick a scene, listen to both takes, and see which model you’d cast. https://t.co/92LWNmgijI
#BackchannelSignals

[mrfakename @realmrfakename]

We’ve been working with LAION on a voice acting arena. You listen to two models doing the same scene and compare how well they pull it off.

It’s ready to try now - would love to hear what you think 🙂