Liam Schoneveld

Computer vision researcher @ Woven by Toyota.

[Matthias Niessner @MattNiessner]

📢Pix2NPHM: Learning to Regress NPHM Reconstructions From a Single Image📢

We directly regress neural parametric head models (NPHMs) from a single image — fast, stable, and significantly more expressive than classical 3DMMs such as FLAME.

Face tracking & 3D reconstruction are often limited by the representational capacity of PCA-based face models. By lifting NPHMs to a first-class reconstruction primitive, we enable more accurate geometry, richer expressions, and finer animation control.

Pix2NPHM obtains fast and reliable NPHM reconstructions on real-world data. Inference-time optimization against surface normals and canonical point maps can further increase fidelity.

Key to successful and generalized training of our ViT-based network are:
(1) large-scale registration of existing 3D head datasets, and
(2) self-supervised training on vast in-the-wild 2D video datasets using pseudo ground-truth surface normals.

Finally, we show that geometry-aware pretraining on pixel-aligned reconstruction tasks significantly outperforms generic visual pretraining (e.g., DINO-style features) in terms of generalization.

🌍https://t.co/89IXGnDl4O
🎥https://t.co/7AZIcnD3Mq

Great work by @SGiebenhain, @TobiasKirschst1, @liamschoneveld, Davide Davoli, Zhe Chen

[Liam Schoneveld @liamschoneveld]

🐑🐑 SHeaP inference code is out ! 🐑🐑For all your real-time head pose and expression tracking desires! Check it out at: 🤗 HuggingFace spaces: https://t.co/PhrPUc64WA 📟 Github: https://t.co/qYLHXaYsPV

[Liam Schoneveld @liamschoneveld]

@Michael_J_Black @MattNiessner 😍 This approach can still be improved a lot, I think

[Liam Schoneveld @liamschoneveld]

📢 Our new paper - SHeaP - is out! 📢

TLDR: self-supervised head tracking and geometry (FLAME) prediction, learned via photometric loss with a 2D gaussian splatting renderer.

See more:
🌍 https://t.co/UZFzynT7sG
🎥 https://t.co/uvJ8KgqMcX

[Liam Schoneveld @liamschoneveld]

@minchoi Are its predictions in a local or world coordinate system?

[Liam Schoneveld @liamschoneveld]

Great work by @jiapeng_tang and co ! Gaussian Avatars from just a handful of input images, by leveraging a multi-view diffusion prior !

[Matthias Niessner @MattNiessner]

📢📢𝐆𝐀𝐅: 𝐆𝐚𝐮𝐬𝐬𝐢𝐚𝐧 𝐀𝐯𝐚𝐭𝐚𝐫 𝐑𝐞𝐜𝐨𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 𝐟𝐫𝐨𝐦 𝐌𝐨𝐧𝐨𝐜𝐮𝐥𝐚𝐫 𝐕𝐢𝐝𝐞𝐨𝐬 𝐯𝐢𝐚 𝐌𝐮𝐥𝐭𝐢-𝐯𝐢𝐞𝐰 𝐃𝐢𝐟𝐟𝐮𝐬𝐢𝐨𝐧📢📢

We reconstruct animatable Gaussian head avatars from monocular videos captured by commodity devices such as smartphones.

Key idea: distill reconstruction constraints from a multi-view head diffusion model to complete unobserved regions.

https://t.co/prz5HnGoWq
https://t.co/XkWBKScwb2

Great work by @jiapeng_tang @davidedavoli @TobiasKirschst1 @liamschoneveld

[Liam Schoneveld @liamschoneveld]

@dome_271 Classifier-free guidance always seemed like some weird hack to me. There must be a more mathematically elegant solution out there, waiting to be found.

[Liam Schoneveld @liamschoneveld]

@camo2572 @LabAgainstWar @AlboMP @RichardMarlesMP @SenatorWong I don’t think it would be that hard to come up with a better plan than spending $360b for offensive nuclear subs we are not even contractually guaranteed to receive?

I feel like literally any plan is better than that one.

[Liam Schoneveld @liamschoneveld]

@1111nonbeliever @janusch_patas They could release the code but you wouldn’t get very far without their data and compute 😅

[Liam Schoneveld @liamschoneveld]

@MartinGTobias Most of these measures make sense to me? As someone working in tech, I think it’s well worth spending a little money to encourage more women into the field.

[Liam Schoneveld @liamschoneveld]

@finbarrtimbers Perhaps limiting the representation space via the limited size of the codebook forces the network to better compress what's really important in the images.

[Liam Schoneveld @liamschoneveld]

@lvminzhang Is there a paper accompanying this code somewhere?

[Liam Schoneveld @liamschoneveld]

@janusch_patas Thanks!

[Liam Schoneveld @liamschoneveld]

@AlboMP Wow that was easy! Now could you quickly do gambling ads and mining companies paying no royalties!?

[Liam Schoneveld @liamschoneveld]

@senbmckenzie Are you suggesting we should raise teacher salaries? Good on you!

[Liam Schoneveld @liamschoneveld]

@levelsio @whittomd Wow LEDs! What futuristic technology

[Liam Schoneveld @liamschoneveld]

@techchildrights @laion_ai Coming from a human rights organization, I am sure you appreciate the importance of transparency. Without @laion_ai's ongoing AI transparency efforts, we would know very little about the data going into these models.

[Liam Schoneveld @liamschoneveld]

@Saboo_Shubham_ @laion_ai This definitely wouldn’t work as well on papers that haven’t had 1000s of blog and Reddit etc posts written about it though

[Liam Schoneveld @liamschoneveld]

@dome_271 I actually had this problem a long time ago when trying to use ConvNets to generate audio. Perhaps looking at audio generative model literature may help as high frequency details are perhaps even more important in that domain.