Paul Streli

PhD student at SIPLAB, @ETH Zürich; previously @meta @imperialcollege @tiktok_uk
@Meta PhD Research Fellow

[Paul Streli @paulstreli]

Introducing TouchInsight for detecting touch input from all ten fingers on any surface, purely based on vision-based hand tracking.

w/ Mark Richardson, Fadi Botros, Shugao Ma, Robert Wang, and @cholz.

paper: https://t.co/twuvU7RAtk

Join us for a demo at #UIST2024 next week!

[Paul Streli @paulstreli]

@CSS_ETHZurich Really enjoyed the conference and the chance to share our work! Thanks to everyone who stopped by!

[Paul Streli @paulstreli]

EgoPressure at @CVPR 2025 (Highlight)!

w/ @_yimzhao_ , @taeinkwon1 , @mapo1 , @cholz

🌍 Project page: https://t.co/Ndm0Mo8Bvx
📰 Paper: https://t.co/w8C669rnny

📍Come visit us during our poster session in ExHall D Poster 149, 15 Jun, 16:00 CDT — 18:00 CDT!

[Paul Streli @paulstreli]

This project started as Yiming's master’s thesis with #SIPLAB and #CVG at @CSS_ETHZurich! Huge thanks to all collaborators. Excited to see how the community builds on this to advance hand-object understanding and take another step toward mobile, situated interaction in #XR.

[Yi-Chi Liao @yichiliao]

#CHI2025 Human-in-the-loop optimization (HiLO) often starts from scratch — slow and inefficient. How can we leverage prior experience to boost HiLO? We introduce Continual HiLO (CHiLO): a framework where computational optimizers learn across users and improve over a lifetime.

[Christian Holz @cholz]

Releasing EgoSim, a simulator for body-worn cameras. #NeurIPS2024 D&B

EgoSim takes real mocap data (e.g., AMASS) and synthesizes multi-modal egocentric videos

Plus: MultiEgoView, a real dataset from 6 GoPro cameras and ground-truth 3D poses during several activities (13 people)

[Karan Ahuja @realkaranahuja]

🔬 My lab at @NorthwesternU has a new website! Visit https://t.co/J5zto5MALI to see our latest research from CHI, ECCV & UIST 2024.

🎓 We're recruiting 2 PhD students (fully funded) for Fall 2025! Interested candidates, apply by Dec 1.

Please RT to help spread the word!

[Paul Streli @paulstreli]

@Marc_Y_Wang @NElmqvist @luileito Congrats 🥳

[Christian Holz @cholz]

new: MANIKIN reveals & overcomes SMPL-based limitations for full-body IK tasks via
– a biomechanically-inspired neural-analytical formulation
– a neural IK method for predicting body poses from end-effectors
@eccvconf #ECCV2024

@cs_jiaxi_jiang @paulstreli @Xuejingluo
@CSatETH

[Paul Streli @paulstreli]

@theadrianm_ Our publication really focuses on touch modeling and probabilistic input decoding. A lot of work has gone into the underlying technology stack, thanks to incredible efforts from colleagues at Meta. :)

[Paul Streli @paulstreli]

Paul Streli, Mark Richardson, Fadi Botros, Shugao Ma, Robert Wang, and Christian Holz. TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric Vision. In Proceedings of ACM UIST 2024.

full video: https://t.co/FMnwO9IdTK

[Paul Streli @paulstreli]

Implementing our framework, we present a purely vision-based ten-finger text entry system on a surface-aligned virtual keyboard that runs on a standalone mobile MR headset (Quest 3).

[Paul Streli @paulstreli]

Started this as intern at @Meta🚀Huge thanks to Mark and Fadi for their mentorship, and to Shugao, Rob, and Christian for all the advice and support. Wouldn't have been possible without the many talented colleagues at Reality Labs whose work laid the foundation for this project!

[Paul Streli @paulstreli]

TouchInsight effectively integrates sensing uncertainties with user uncertainties in a closed-form expression that allows us to reason about user intentions in a probabilistic framework. We refine text input predictions through additional priors from a language model.

[Paul Streli @paulstreli]

It locates input events with a mean error of 6.3 mm, and accurately detects touch events (F1 = 0.99) and identifies the finger used (F1 = 0.96). We demonstrate the effectiveness of our approach for a core application of dexterous touch input: two-handed text entry.

[Paul Streli @paulstreli]

Introducing TouchInsight for detecting touch input from all ten fingers on any surface, purely based on vision-based hand tracking.

w/ Mark Richardson, Fadi Botros, Shugao Ma, Robert Wang, and @cholz.

paper: https://t.co/twuvU7RAtk

Join us for a demo at #UIST2024 next week!

[Paul Streli @paulstreli]

TouchInsight comprises a neural network to predict the moment of a touch event, the finger making contact, and the touch location. It represents locations through a bivariate Gaussian distribution to account for uncertainties due to sensing inaccuracies.