Justin Uberti

Head of Realtime AI @OpenAI. Created WebRTC. Past: Distinguished Engineer @google, AIM. Amateur mathematician/musician.

[Justin Uberti @juberti]

Just posted our technical deep dive on the GPT-Live system, by @zahanm and yours truly.

[OpenAI @OpenAI]

GPT-Live can listen while it speaks.

To make that feel natural at ChatGPT scale, we rebuilt the voice stack from client to model.

This new architecture keeps audio flowing continuously, so deeper reasoning and tool use don't interrupt the conversation.

[Justin Uberti @juberti]

@Capslock404 for 2), CF is dismissed wholesale; claiming that digital neurons could not possibly do the job of biological neurons is quite a leap
for 3), there are many other theories (eg GWT) that don't have these physical requirements

[Justin Uberti @juberti]

It's good to be cautious here, but these arguments are surprisingly hand-wavy:
1) intelligence doesn't imply consciousness
2) we think computational functionalism is "fatally flawed"
3) some theories require specific physical properties that LLMs lack

[nature @Nature]

LLMs can mimic thought and feeling with uncanny skill, but intelligence alone is not enough for subjective experience

https://t.co/oJkyfCmqjZ

[Justin Uberti @juberti]

@lewingtonpitsos Tools definitely help! The model needs some way to access the outside world, but l think the trend will continue to be toward thinner harnesses.

[Justin Uberti @juberti]

Exactly. Orchestration is just a crutch to help the model scale up. Don’t get too attached to it.

[Pavan Katta @PavanKatta1998]

The model eats the harness, aka bitter lesson

[Justin Uberti @juberti]

@ai_sentience See the author’s followup at

[Doris Tsao @doristsao]

The outpouring of interest in our paper has been … a bit insane.

May I make four points?

1. We have not shown that “consciousness has a frame rate.”

I can see why people draw this conclusion: we report that inferotemporal and prefrontal cortex alternate between conscious and suppressed states roughly every 60 ms during binocular rivalry.

Consciousness depends on many brain areas, and our paper reports results from only a few select nodes.

We still don’t know how conscious access works--the actual recipe for generating conscious experience. What we have shown is that deep in the brain there is an extremely strong signal, rich enough to reconstruct a high-dimensional face in detail, that is inaccessible to consciousness. As I argued in the original tweet, this makes the problem of conscious access a lot more experimentally tractable. But to claim “consciousness has a frame rate,” one would need to have actually solved the mechanism for conscious access and shown that this mechanism operates at a frame rate.

While we currently don’t know conclusively one way or the other (see VanRullen & Koch, 2003 for a review of positive evidence), I personally am skeptical that consciousness has a frame rate. I believe the brain, within posterior parietal cortex, has continuous tracking mechanisms that actively represent the persistence of matter, including ourselves as physical bodies. The coupling between these persistent tracking processes and feature representations across the brain (like those in face patches) is, I think, the essential mechanism of conscious access.

This coupling need not happen at the rate of inferotemporal alternations (and inferotemporal cortex itself may not even alternate outside of the artificial situation of binocular rivalry). As an analogy: imagine the area crucial for conscious access holding a steady drone note, while inferotemporal cortex alternates between two notes, one in harmony and one dissonant. Inferotemporal content might couple to the drone only at moments of harmony, yet experience would still be continuous, because the drone itself (the tracking process that I believe is at the core of conscious perception) never stops. And areas without conflicting content, like auditory cortex during visual rivalry, could stay in continuous harmony with the drone.

In sum, there is no reason to conclude from our study that consciousness is discrete, with a "frame rate."

2. Our results resemble Jezek et al. 2011 (as @jjsakon notes).

Jezek et al. trained rats in two boxes that were identical except for their lighting cues, so that hippocampal place cells formed two distinct maps, one per box. They then “teleported” the rat by abruptly switching the lights. For a few seconds afterwards, CA3 place-cell activity flickered between the two maps, synced to the theta cycle (~125 ms).

@StanDehaene first pointed this paper out to me, and reading it blew me away. To my mind it is the strongest evidence for a bistable attractor in the brain, because the competing maps are stored within the hippocampus itself.

I think the phenomenon we see may be very similar at the circuit level: a bistable attractor network within inferotemporal cortex that alternately converges to one of two patterns, as cells in face patches work together to build a coherent representation of the input. In our case, both images are present in the input, so the competition could arise within IT or be inherited from earlier areas such as V1--recording there is an important next step. (It also still puzzles me that we didn’t see strong coherence between face patches ML and AM.)

More broadly, what’s different about our paper is the brain area (inferotemporal cortex, often modeled as a feedforward DNN) and the ability to relate the two competing patterns to consciousness, with one granted conscious access and the other not.

3. Please comment on bioRxiv!

The Twitter attention has been cool, but I really hope colleagues will be generous enough to post detailed commentary on bioRxiv, where there are currently 0 comments. As @seemaychou points out, peer review doesn’t have to come from just three anonymous reviewers. We would be truly grateful for your critical feedback.

4. Stay tuned

This work was done at Caltech, Berkeley, in collaboration with @EngelTatiana at Princeton, funded by NIH and HHMI.

I hope to share our vision soon for how we can solve consciousness @AsteraInstitute

[Justin Uberti @juberti]

@ai_sentience Yes, but that’s not really the news from this paper, it’s that multiple competing percepts are interleaved at this rate. No claims are made for steady state perception.

[Gavin Nelson @Gavmn]

Dots, now on your Home Screen

[Justin Uberti @juberti]

Nonlinear activation functions can do some amazing things!

[Cheng Lou @_chenglou]

My dear friends, I’d like to show you my latest toy project

Introducing Handwired: turn your regular JavaScript code into a neural net!
- no training
- exact outputs vs the JS program
- fully differentiable

All runnable in a browser, without GPU

[Justin Uberti @juberti]

Absolutely insane live demo. So many things that could have gone wrong. But they nailed it.

[Tesla @Tesla]

Real life, baby

[Justin Uberti @juberti]

If your Live API app typically selects from a few specific tools, you can now use the new Decisions API to speed up delegation, while retaining the ability to use a full LLM for long-tail queries.

[jiayu @daidaijiayu]

you can now use native decision delegation in your voice app.

a week ago i wrote this voice x decision doc. today i update the guide because now it is miles better

you can trigger a decision on every input or output frame (roughly every 200ms).

you can have a realtime decision engine judging what i’m saying while i’m still saying it!!
try realtime action × realtime voice 👇

https://t.co/UNOrQyvoJ0

[Justin Uberti @juberti]

@elmaelma01_ Sorry to hear that. Could you give a short example?

[Justin Uberti @juberti]

GPT-Live-1.1 offers much better multilingual support (including several new voices), more natural delegation, and improved steerability. Rolling out now to ChatGPT Voice, Desktop, and Dots.

[OpenAI @OpenAI]

New voices. Better accents. And less "checking..."

GPT-Live-1.1 brings these updates to Voice in ChatGPT, Codex, and dots, with new voices in French, German, Hindi, Mexican Spanish, Korean, and Japanese.

Rolling out today to all users. Coming soon to the API.

[Justin Uberti @juberti]

@altryne @petergostev https://t.co/gRT5JC7WPV

[Justin Uberti @juberti]

@ahatzz11 @thsottiaux @Ali_Shobeiri Dots is a new system with many new capabilities, but like all v1s there are some bugs. We have worked hard to make ChatGPT Voice insanely fast and reliable and are doing the same for Dots.

[Justin Uberti @juberti]

@_josh_meyer_ @ai_coustics @pipecat_ai @veris_ai Would it be possible to run the same task/persona permutations against all noise profiles? That would help illustrate the impact of each noise profile and the relative lift from smarter models or DSP.

[Justin Uberti @juberti]

This is very good news, we need interpretability now more than ever

[Nick @nickcammarata]

interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had

with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works

https://t.co/zHij5zwKGT

[Justin Uberti @juberti]

@_josh_meyer_ @ai_coustics @pipecat_ai @veris_ai IMO this is confusing and makes it hard to understand the amount of degradation in each particular scenario.

[Justin Uberti @juberti]

@_josh_meyer_ @ai_coustics @pipecat_ai @veris_ai But street is at 28% somehow?

[Justin Uberti @juberti]

@_josh_meyer_ @pHequals7 AEC3’s main weakness is in doubletalk scenarios where it tends to bias heavily toward suppression, see https://t.co/k349h8SoEt

[Justin Uberti @juberti]

Echo cancellation is a complicated situation. While gpt-live is quite robust to echo, AEC and AGC are very intertwined and accordingly local AEC is still useful to ensure the model receives a usable signal.

[Sidharth Raja @sidharth_raja]

@juberti do you think that in the long term, a separate echo cancellation step (neural/linear) still makes sense, or just send raws(/+loopbacks) and let the model figure that out too?

[Justin Uberti @juberti]

@YogSho I think it's a hard problem since you need a lot of context and intelligence to figure out which audio source you want to attend to. This is the classic "cocktail party" problem.

[Justin Uberti @juberti]

Why turn detectors (or any other low-parameter endpointing mechanism) are a dead end

[Josh Meyer @_josh_meyer_]

How to kill a voice agent? Just turn on the TV ☠️

We benchmarked 14 voice agents with a TV playing in the background.

Task completion fell from 38.7% to 8.6%

🧵/