Yin-Jyun Luo

Research Scientist @SpotifyResearch | PhD in Disentangled Audio Representations @c4dm | Prev. Intern @StabilityAI @SonyAI_global @AIST_JP

[Yin-Jyun Luo @jun_luolo]

@liyzhen2 Now I am even more excited 🫨

[Yin-Jyun Luo @jun_luolo]

Diffusion autoencoders have been theoretically backed in terms of learning informative representations, and exploring the modellability of the learnt repre makes a lot sense! The tweak on leveraging frozen SSL model is cool.

[jo.schb @jo_schb]

💡 The idea

We start from a frozen self-supervised encoder (DINOv2, MAE, or CLIP) and combine it with a generative decoder.

Then we fine-tune only the [CLS] token embedding - injecting low-level info while keeping the rest frozen.

[Peter Sobot @psobot]

This is my team. I’m super excited to work on this stuff. If you are too, let’s talk: we’re hiring.

https://t.co/D1apaw3Nqx

[Kwang Moo Yi @kwangmoo_yi]

Preprint of today: Vavilala et al., "Generative Blocks World: Moving Things Around in Pictures" -- https://t.co/2UV2B2qzXL

I have a soft spot for reviving old ideas in modern methods -- block world via primitives now with Diffusion models for generating/editing images.

[Yin-Jyun Luo @jun_luolo]

@tkipf Yes - AudioSlots is yet another motivator behind our work. Though they are not directly comparable due to the problem domain, they share a common high level idea to encode constituent sources as separate latent entries which I can really appreciate

[Yin-Jyun Luo @jun_luolo]

@zacknovack The acronym that sticks in my head like the song does 😂 Good job!

[Yin-Jyun Luo @jun_luolo]

@keunwoochoi Oh wow 🤩 congrats 🥳

[Yin-Jyun Luo @jun_luolo]

@unilightwf fair enough 😅🫡

[Yin-Jyun Luo @jun_luolo]

Pumped to see a comeback of GMVAE among a sea of VQ!

https://t.co/ZqPqIWRkTj

Speaking of, Wei-Ning's https://t.co/W9q7jHCzeP on TTS has a substantial impact to my research on style transfer via (unsupervised) disentanglement. But it seems overshadowed by his own work HuBERT😅

[Yin-Jyun Luo @jun_luolo]

@92HsChoi The two-stage paradigm relies on 1st's reconstruction and 2nd's distribution modelling. Not noising posterior defo gives better 1st reconstruction. Not sure about the effect for 2nd but I think having a proper prior during 1st training is more impactful.🤔

[Keitaro Tanaka @Kakanat1105]

主著論文がAPSIPA Trans.にアクセプトされました🙌
Our paper has been accepted for publication in APSIPA Transactions!!🚀
A big thanks to the co-authors (professors!), reviewers, and everyone who supported this work.
Special mention to @jun_luolo, @_tai_shi, and @yoshipon0520🙏

[Yin-Jyun Luo @jun_luolo]

At the NeurIPS workshop of Audio Imagination, we present a supervised method as a preliminary step towards answering these questions.
https://t.co/tCnStND5Fw

[Yin-Jyun Luo @jun_luolo]

How to train a model to extract separate entities of rep. associated to individual sources of a music mixture?
How to also divide each entity into subspaces of pitch and timbre?
How to then have the model take arbitrary comb. of these building blocks to sample novel mixtures?

[Yin-Jyun Luo @jun_luolo]

By feeding to the decoder different combinations of pitch and timbre latents, we achieve applications such as:
- instrument swapping between two mixtures or within a mixture
- stem exchange between two mixtures

[Yin-Jyun Luo @jun_luolo]

There goes the one proof to my ISMIR presence.

It was very nice to catch up w/ the Taiwanese Gang, and it's my honour to be confronted by "why are you still doing disentanglement?"

That's right, I will also be presenting DisMix https://t.co/tCnStND5Fw at the NeurIPS Workshop!

[Hao-Wen (Herman) Dong 董皓文 @hermanhwdong]

@yoyolicoris @affige_yang @jun_luolo 😂

[Yin-Jyun Luo @jun_luolo]

@ArxivSound A recent work has extended the proof to time series
https://t.co/kDIneAYgdP

[Yin-Jyun Luo @jun_luolo]

@ArxivSound Great to more interests in pitch-timbre disentanglement. Using paired data with shared attributes has shown good results in speech
https://t.co/yXfF1uLmaI
https://t.co/V3cCUNe6DX
They are also proven identifiable under assumptions
https://t.co/J0PRYyK2fy
https://t.co/IvBF3gGCPp

[arXiv Sound @ArxivSound]

``Self-Supervised Multi-View Learning for Disentangled Music Audio Representations,'' Julia Wilkins, Sivan Ding, Magdalena Fuentes, Juan Pablo Bello, https://t.co/vWc7Ulfz2q

[Yin-Jyun Luo @jun_luolo]

@unilightwf well, at the very least they do things more efficiently and I can't complain when disentanglement is proven useful in yet another task :)

[Yin-Jyun Luo @jun_luolo]

Audio codecs are turning into VQ-based voice conversion models with an extra focus on compression. Disentanglement seems to be the sauce.

[arXiv Sound @ArxivSound]

``LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec,'' Yiwei Guo, Zhihan Li, Chenpeng Du, Hankun Wang, Xie Chen, Kai Yu, https://t.co/1yMzq15hbK