Sonal Kumar

Research @ Cartesia | CS PhD @ UMD

[SALMA Workshop Chairs @SALMAworkshop26]

🎙️ Meet the #SALMA2026 keynote speakers!

Join us at #EMNLP2026 in Budapest on Wed, Oct 28 for talks by Philipp Koehn, Berrak Sisman, Gabriel Skantze, Sebastian Möller, Piotr Żelasko & Julia Hirschberg.

📅 Program: https://t.co/WnKoLmX5tA

[arXiv Sound @ArxivSound]

Sonal Kumar, Sinan Hersek, Artem Dementyev, Mengzhen Pan, Ishan Chatterjee, Anurag Kumar, Ramani Duraiswami, Dinesh Manocha, Andrea Colaco, "SEA-LM: Egocentric Spatial Audio Understanding for Wearable Microphone Arrays" https://t.co/180wZac8GO

[Sonal Kumar @realsonalkumar]

Scope: simulated, stationary scenes and two smart-glasses designs. Real recordings and moving sources are next.

Joint work between @gammaumd @umdcs @Google and @GoogleDeepMind with Sinan, @artem_dementyev @AcouIntel @dmanocha @ramani_d and others.

Paper: https://t.co/V3bWWaJHi7

[Sonal Kumar @realsonalkumar]

Audio LLMs are good at what they hear, but mostly not where it's coming from.

From my @Google internship: SEA-LM, egocentric spatial audio understanding for smart glasses.

One model → what, where, when, who (you vs. bystanders), and what was said.

https://t.co/polowfIMw4

[Sonal Kumar @realsonalkumar]

How it works: any 4–9 mic glasses layout is beamformed into First-Order Ambisonics, encoded by FOACoder (pretrained on sound localization + wearer voice activity), and fed into Gemma-4-E4B alongside its native mono audio pathway.

Token dilution: long transcripts drown out the few coordinate and timestamp tokens in the loss. We upweight those tokens 5× with a spatio-temporal weighted cross-entropy.

On our held-out simulated eval set, vs. a SELDNet + Gemma-4 baseline (same LLM, data, curriculum):
• azimuth error 80.6° → 14.9°
• elevation error 27.4° → 12.8°
• fewer hallucinated sources
One checkpoint across 1,211 array configs (4–9 mics).

[Sonal Kumar @realsonalkumar]

@utkarsh4430 Awesome work. Congratulations @utkarsh4430

[Utkarsh Tyagi @utkarsh4430]

🎉 Excited that “Not Every Rubric Teaches Equally” is accepted to NeurIPS 2026! 🎉

POW3R turns static rubric rewards into policy-aware training signals, emphasizing the criteria that actually separate good and bad rollouts for the current state of the model.

Link - https://t.co/YriZ0YivNX

Our team at @ScaleAILabs has been cooking on some really exciting RL/post-training research lately. Check it out 👀

RGSD - https://t.co/cl1wMqdIQE
Reward Hacking with Rubrics - https://t.co/zKvEI1hINh

[Sonal Kumar @realsonalkumar]

Excited to share that TAC has been accepted at NeurIPS 2026.

TAC enabled timestamped audio captioning and speaker diarization in overlapping settings. TAC captions when combined with text-only reasoner achieves SOTA scores on most of the audio(-visual) benchmarks.

Work done at @Adobe with @pseetharaman, @urinieto, @justin_salamon, @ZeyuJin and others. Super exciting work.

More here: https://t.co/Zonr3rKnJv

[Sonal Kumar @realsonalkumar]

Introducing SteerDuplex + SteerBench! We present a model whose speaking style can be steered mid-conversation, alongside the first benchmark designed to measure steerability in full-duplex dialogue.

[Utkarsh Tyagi @utkarsh4430]

Full-duplex voice models can listen and speak at the same time, but can they change how they speak when you ask them to?

Introducing SteerDuplex + SteerBench: the first benchmark that measures steerable full-duplex dialogue.

65.1% audio-steering APR vs 20.6% for the best open baseline. 🧵

[DAIR.AI @dair_ai]

NEW paper from Google

(bookmark it)

It's on advancing automated scientific review.

Just pay attention to the focus on agentic verification which is something I've been writing about recently.

AI is accelerating paper generation, but review is still mostly human throughput. The paper frames this as a systems problem and proposes four levels of AI-human collaboration for scientific evaluation.

It also discusses Google Paper Assistant as an early tool for scaling parts of the review process.

The important point is the direction of travel. AI-scientist workflows create verification debt. The review stack now needs its own agents, taxonomies, and human oversight patterns.

Useful read if you track AI research agents, automated peer review, or verification infrastructure for scientific discovery.

Paper: https://t.co/KXSJD5orkp

Learn to build effective AI agents in our academy: https://t.co/LRnpZN7L4c

[Sonal Kumar @realsonalkumar]

We researchers from US universities (UNC, UMD, Purdue, UW-Madison) are planning to host an audio-centric multimodal workshop at NeurIPS 2026, Atlanta. We would love to hear the community's opinion about this workshop through this form: https://t.co/SDxUbiGuuK. Thank you!

[Sonal Kumar @realsonalkumar]

As speech and audio language models rapidly evolve, we need stronger conversations around architectures, data, evaluation, and safety for audio-first AI. SALMA brings together researchers working across audio LLMs.

Consider submitting or sharing with anyone in this space!

[SALMA Workshop Chairs @SALMAworkshop26]

📢 Call for Papers is out - https://t.co/CipfCmhng9

We invite submissions to SALMA 2026: Speech and Audio Language Models Workshop, co-located with EMNLP 2026.🎙️

🗓️ Direct Submission: July 27, 2026
ARR Commitment: August 26, 2026

Please spread the word and consider submitting!

[Utkarsh Tyagi @utkarsh4430]

1/ New from @ScaleAILabs: Rubrics (a.k.a. checklists) have become the default reward interface for RL on open-ended tasks without final verifiable answers.

But most rubric RL still relies on static aggregation: fixed human weights over criteria, summed into one scalar reward.

We show that this conflates what should matter in the final answer with what can actually teach the current policy.

https://t.co/H5wTQ27ulb

[GAMMA UMD @gammaumd]

Congratulations to all GAMMA members graduating this year! 🎓

We were happy to celebrate together at the GAMMA graduation gathering and see so many lab members, alumni, and friends come back for this special week. Wishing everyone the very best for the upcoming hooding ceremonies and their next chapter! 🎉

[Bryan Catanzaro @ctnzr]

Today we're releasing Nemotron 3 Nano Omni.

Audio, Video, Image, Text ➡️ Text

Ask questions about all your data.

Amazing efficiency powered by the Nemotron Hybrid SSM MoE architecture.

State of the art multimodal intelligence.

[DailyPapers @HuggingPapers]

Audio Flamingo Next

A next-generation open audio-language model by NVIDIA with 30-minute audio support and time-grounded reasoning. Trained on 1M+ hours, it outperforms larger models on speech, sound, and music understanding.

[Jonah Casebeer @CasebeerJonah]

GenAE: An audio autoencoder engineered for generative modeling.
To appear at ICASSP 2026.
w/ @__gzhu__ @zhepeiw03 @NicholasJBryan
arXiv: https://t.co/akXYENgCZG
Video: https://t.co/udshifZ7tY

[Justin Salamon @justin_salamon]

This is big. SOTA audio reasoning. SOTA video reasoning. SOTA audio captioning. SOTA sound event detection. Better than Gemini. Better than Qwen.

TAC: Timestamped Audio Captioning

📑 paper: https://t.co/3IVuI6qn4S
🌐 website with more demos: https://t.co/FhoFqqTYQo

[Sonal Kumar @realsonalkumar]

More Demos: https://t.co/kQ6CrKmOZh
Paper: https://t.co/PQb0Ci4fMf

Special thanks to my mentors at Adobe - @pseetharaman @urinieto @justin_salamon and others who made this possible.

[Sonal Kumar @realsonalkumar]

Excited to announce our most recent work TAC: Timestamped Audio Captioning, which captures every detail in the audio with precise timestamps.

🛠️ We propose a strategy to generate complex, polyphonic audio mixtures which enables precise temporal grounding for overlapping events.

[Sonal Kumar @realsonalkumar]

🧠 Describe-then-reason: A TAC(-V) and LLM (text-only reasoner) cascade that leverages these grounded captions to reason over video and audio. This approach beats native multimodal models on benchmarks like MMAR, MMAU-Pro, VideoHolmes, and Daily-Omni.