Jongheon Jeong

Assistant Professor @ Korea University | PhD @KAISTCoE | Interned @Amazon | Working on trustworthy machine learning

[Jihoon Tack @jihoontack]

Excited to share my first work after joining @MSFTResearch!

LLMs have entered the agentic era, and we now collaborate with agents on complex tasks over many turns of interaction.
But does your agent actually follow what you intended?

We show where agents get lost: LLMs Get Lost in Evolving User Intent.
📄 https://t.co/XpberQ2BL7 🧵 1/N

[Jinwoo Shin @JinwooShin0417]

RLDX-1 took the first place "officially" in the RoboCasa365 benchmark!

https://t.co/CvkX3XUuHT

Tech report: https://t.co/AKJxLy8qhr

[Yewon Kim @haiyewon]

What role can AI play in unlocking musical creativity?

At #CHI2025 (🥇 Best Paper Award 🥇), we present Amuse: a songwriting tool for musicians that turns photos, sounds, and stories into chords 🎶

📄 https://t.co/ScJX1AGccv
🎧 https://t.co/FtkINAXdEK

[Turing Post (Ksenia Se) @TheTuringPost]

A very interesting study from Korean @kaist_ai and DeepAuto AI introduced InfiniteHiP.

It's an inference framework that handles long context up to 3M tokens on a single GPU with ~19x speed boost.

But how? The key improvements are:

• Offloading memory — it moves less-used data from the GPU to regular memory.

• Smarter hierarchical context pruning

• Dynamically adjusted RoPE to handle longer texts without retraining.

Here are the details:

[Jihoon Tack @jihoontack]

At #NeurIPS to present papers! Details in 🧵.

I’ll be in Vancouver (Dec 10-15) and would love to connect - love to discuss topics like efficient LLMs, agents, and LLM safety.

Also looking for PostDoc and RS positions. Happy to connect if you’re hiring or know of relevant roles!

[Saining Xie @sainingxie]

Nothing ever happens… until it does!

Just saw JanusFlow by @deepseek_ai uses REPA for training and shows some solid improvements:

https://t.co/7VeZmQPHme

[Saining Xie @sainingxie]

Representation matters.
Representation matters.
Representation matters, even for generative models.

We might've been training our diffusion models the wrong way this whole time. Meet REPA: Training Diffusion Transformers is easier than you think! https://t.co/lyWwiTYjEt(🧵1/n)

[Sihyun Yu @sihyun_yu]

Introducing REPA! We show that learning high-quality representations in diffusion transformers is crucial for boosting generation performance. With REPA, we speed up SiT training by 17.5x (without CFG) and achieve state-of-the-art FID = 1.42 using CFG with the guidance interval. 🧵[1/7]

https://t.co/MoN9vjZ93U

[Yann LeCun @ylecun]

When training a visual encoder with self-supervised learning, we know for a fact that using a decoder with a reconstruction loss doesn't work nearly as well as using a joint embedding architecture with feature prediction loss and a collapse prevention mechanism.

This paper from @sainingxie's shop at NYU shows that *even* if you are interested in generating pixels (e.g. to produce pretty pictures with a diffusion transformer), it pays to include a feature prediction loss so that the internal representation of the decoder can predict features from a pre-trained visual encoder such as DINOv2.

[batuhan the fal guy @isidentical]

imgsys becoming mainstream!!! love to see we can help people by providing a venue for open-sourced binary preference labeling :)

[AK @_akhaliq]

Margin-aware Preference Optimization for Aligning Diffusion Models without Reference

Modern alignment techniques based on human preferences, such as RLHF and DPO, typically employ divergence regularization relative to the reference model to ensure training stability. However, this often limits the flexibility of models during alignment, especially when there is a clear distributional discrepancy between the preference data and the reference model. In this paper, we focus on the alignment of recent text-to-image diffusion models, such as Stable Diffusion XL (SDXL), and find that this "reference mismatch" is indeed a significant problem in aligning these models due to the unstructured nature of visual modalities: e.g., a preference for a particular stylistic aspect can easily induce such a discrepancy. Motivated by this observation, we propose a novel and memory-friendly preference alignment method for diffusion models that does not depend on any reference model, coined margin-aware preference optimization (MaPO). MaPO jointly maximizes the likelihood margin between the preferred and dispreferred image sets and the likelihood of the preferred sets, simultaneously learning general stylistic features and preferences. For evaluation, we introduce two new pairwise preference datasets, which comprise self-generated image pairs from SDXL, Pick-Style and Pick-Safety, simulating diverse scenarios of reference mismatch. Our experiments validate that MaPO can significantly improve alignment on Pick-Style and Pick-Safety and general preference alignment when used with Pick-a-Pic v2, surpassing the base SDXL and other existing methods. Our code, models, and datasets are publicly available via

[Jiwoo Hong @jiwoohong98]

Aligning T2I diffusion model directly to ANY preference with 💡MaPO💡

We introduce MaPO, aligning diffusion models to preference datasets with 🌟ANY🌟 disparate styles or topics, such as safety or animations:

👉Safety: https://t.co/8oDmort1RM
👉Cartoon: https://t.co/we4WnZxSnk

[Sayak Paul @RisingSayak]

Introducing MaPO, a memory-efficient technique for aligning T2I diffusion models on preference data 🔥

We eliminate the need to have a reference model when performing alignment fine-tuning.

Code, models, datasets, and paper are up at:
https://t.co/7DjqFJTA6N

1/7

[Abhinav Shrivastava @abhi2610]

Call for Papers: #INRV2024 Workshop on Implicit Neural Representation for Vision @ #CVPR2024! Topics: Compression, Representation using INR’s for images, audio, video & more! Ddl: 3/31. Submit now! @CVPR
Website: https://t.co/380CMq5Ouc
Submission Link: https://t.co/KwkGqyKjhy

[Jihoon Tack @jihoontack]

🙌 At #NeurIPS to present two papers
Wed. 10AM: Efficient meta-learning / 5PM: Improving MAE with meta-learning.

I am working on continual learning/efficiency/AutoML for LLMs! DM me if you are interested in chatting
👀 Also looking for an internship, welcome any recommendations

[KAIST AI @KAIST_AI]

Multi-scale Diffusion Denoised Smoothing
Status: Accept (poster)
Lab name / External affiliate companies: ALIN
Authors: Jongheon Jeong (@jongheon_j), Jinwoo Shin (@jinwoos0417)
Url: https://t.co/aQvyesI8Yv

[Jon Barron @jon_barron]

I just pushed my color-coded latex table script to github: https://t.co/zkGXzyKSdO

Every table I've made since grad school has been procedurally generated. I *highly* recommend doing this --- if you copy-paste and manually-bold numbers, you will eventually make a costly mistake.

[Olivier Hénaff @olivierhenaff]

With the advent of AI assistants, humans depend on their outputs at an unprecedented rate. It is vital that these models be aligned with human abilities, judgements, and preferences.

Self-supervised video pretraining yields a big step towards human-aligned visual representations

[AK @_akhaliq]

Collaborative Score Distillation for Consistent Visual Synthesis

paper page: https://t.co/uMZnQUmgO2

Generative priors of large-scale text-to-image diffusion models enable a wide range of new generation and editing applications on diverse visual modalities. However, when adapting these priors to complex visual modalities, often represented as multiple images (e.g., video), achieving consistency across a set of images is challenging. In this paper, we address this challenge with a novel method, Collaborative Score Distillation (CSD). CSD is based on the Stein Variational Gradient Descent (SVGD). Specifically, we propose to consider multiple samples as "particles" in the SVGD update and combine their score functions to distill generative priors over a set of images synchronously. Thus, CSD facilitates seamless integration of information across 2D images, leading to a consistent visual synthesis across multiple samples. We show the effectiveness of CSD in a variety of tasks, encompassing the visual editing of panorama images, videos, and 3D scenes. Our results underline the competency of CSD as a versatile method for enhancing inter-sample consistency, thereby broadening the applicability of text-to-image diffusion models.

[Jonathan Richard Schwarz @schwarzjn_]

Very happy to announce that our latest paper on Neural data compression with INRs, Meta Learning & Sparse Subnetwork selection has been accepted to #ICML2023 (Scores 7, 7, 7). 1/N

Paper: https://t.co/qYt6LPpYD6