Simon Stepputtis

Assistant Professor at @virginia_tech

Creating intelligent robots that reason about their environment and the humans within them through hybrid AI 🚀

[Simon Stepputtis @SimonStepputtis]

Thrilled to join @virginia_tech as an assistant professor in @VirginiaTech_ME this fall!

At the TEA lab (https://t.co/VH4anRDWRI), we’ll explore hybrid AI systems for efficient and adaptive agents and robots 🤖

Thank you to everyone who has supported me along the way!

[Simon Stepputtis @SimonStepputtis]

If you’re interested in robot learning, hybrid AI systems, or AI/ML for autonomous systems, I’ll be recruiting students soon and would love to connect!

[Simon Stepputtis @SimonStepputtis]

Thrilled to join @virginia_tech as an assistant professor in @VirginiaTech_ME this fall!

At the TEA lab (https://t.co/VH4anRDWRI), we’ll explore hybrid AI systems for efficient and adaptive agents and robots 🤖

Thank you to everyone who has supported me along the way!

[Simon Stepputtis @SimonStepputtis]

I am excited to announce that we are accepting applications for the Pathways@RSS 2025 Fellowship!

📌 Fellowship for BSc, MSc & early PhD students
📌 Supports first-time RSS / Conference attendees
📌 Attend & thrive at #RSS2025

🗓️ Find out more and apply by March 21st: https://t.co/NcB0N0muln

[Simon Stepputtis @SimonStepputtis]

@joecampb Thank you for your kind words! It is great working with you too!

[Simon Stepputtis @SimonStepputtis]

📣 Thrilled to announce that I'm on the job market for Fall 2025 faculty positions! I am currently a postdoc @CarnegieMellon @CMU_Robotics.

🔍 My research is dedicated to developing robots that can intelligently reason about their environments and the humans within them. By considering the affordances, attributes, and relationships of objects in the environment, my work enables robots to efficiently learn behaviors and generalize them to novel settings 🚀

⭐️ My focus is on developing neurosymbolic models that combine the expressivity of deep neural networks with the reasoning abilities of symbolic AI.

🤖 I have previously applied my work to applications such as in-home assistance, manufacturing, and beyond.

🌐 Learn more about my work at https://t.co/nxt2vKcGRz

🔭 If you think I might be a good fit for a position at your institution, please don't hesitate to reach out!

[Muhan Lin @MuhanLin424433]

🚀 Excited to share our latest work: extending RLAIF to work well with small language models which may produce incorrect rankings! 🚀 Catch our poster session at #EMNLP in Miami next Thursday!

📄 Paper: https://t.co/eGXXKfVE3p
🎥 Presentation: https://t.co/6amDzfvRFb

[Simon Stepputtis @SimonStepputtis]

🚀 Excited to share our paper ShapeGrasp: Zero-Shot Object Manipulation with LLMs through Geometric Decomposition at #IROS2024 (Session ThCT3.3)!

🎉 Amazing work by @SamuelLi826114 on leveraging geometric decomposition and LLMs for zero-shot task-oriented grasping. The LLM dynamically generates hypotheses on what each geometry is for and selects the best one for the task! 🦾🤖

Explore more at: https://t.co/uV61G0AHOz

[Simon Stepputtis @SimonStepputtis]

Have you ever wondered how robots can proactively support a human’s task? 🤖🧑‍🍳 A key aspect of proactive user support is fast action anticipation. ⭐ Our method uses external domain knowledge to reduce response time and selects the most helpful, non-disruptive task. ⏱️🥗 Below, our system quickly predicts that the user wants to make a dressing and adds further ingredients for the dressing. 🤖🧂

In our new paper, we present an approach for action anticipation from short video contexts in a collaborative kitchen task, preparing salads. 🥗 Through our neuro-symbolic approach, our method quickly and accurately selects helpful actions!

📜 Paper: https://t.co/sAU87JwPvs
🌎 Website: https://t.co/91CcyuWYEA

[Sarthak Bhagat @sarthak__bhagat]

Want a robot to assist you in the kitchen **without any instructions** simply by watching you?🤖🏠

🚀 Presenting our recent paper on action anticipation from short video context for human-robot collaboration, accepted at Robotics and Automation Letters (RA-L).

[Simon Stepputtis @SimonStepputtis]

@liuziyi93 Really nice paper! We found similar things when we were exploring how well LLMs were able to identify player roles in Avalon

Paper and Dataset: https://t.co/GejGPUaKmz

[Simon Stepputtis @SimonStepputtis]

@joecampb @PurdueCS Congratulations! Exiting news to see you move to a professor position!

[Simon Stepputtis @SimonStepputtis]

I'm excited to share our lates work, Sigma, providing an efficient approach to multimodal segmentation that can overcome RGB-only shortcomings like low-light or overexposure issues by reasoning over thermal and depth information. At the core of our multimodal method, sigma achieves global receptive field coverage with linear complexity by utilizing recent advances in Mamba!

Please take a look at our paper and code!
Paper: https://t.co/8XaEj7yDuC
Code: https://t.co/42XApwGtGY

[AK @_akhaliq]

Sigma

Siamese Mamba Network for Multi-Modal Semantic Segmentation

Multi-modal semantic segmentation significantly enhances AI agents' perception and scene understanding, especially under adverse conditions like low-light or overexposed environments. Leveraging additional

[Joseph Campbell @joecampb]

Ever wanted to know how out-of-distribution data will affect a deep neural network's accuracy?

@yuzhe_lu is at #NeurIPS2023 this week presenting our paper which shows how to predict model accuracy given only unlabeled data from the target dist.

Paper: https://t.co/Rv5LbKXxUn

[Simon Stepputtis @SimonStepputtis]

In our latest EMNLP 2023 Findings paper, we delve into the intricate realm of Avalon: The Resistance, a complex social deduction game that requires a nuanced understanding of long-horizon player interactions, their strategic discussions, as well as deception and persuasion amongst them.

When predicting hidden player roles, we find that LLMs struggle identifying and tracking deceptive user behavior. To address this challenging setting, we introduce a new NLU dataset of 20 human-played games grounded in the game's state information. With labeled utterances for deception and persuasion, we hope that our dataset opens up future development in understanding deceptive behavior, as well as many other research avenues!

Our paper will be presented at the Novel Ideas in Learning-to-Learn through Interaction Workshop at @emnlpmeeting today (December 7th)!

Full Paper: https://t.co/irrYaCXRzr
Website: https://t.co/jQpqeYvVyc

[Simon Stepputtis @SimonStepputtis]

Please check out our ICCV CVEU workshop paper Knowledge-Guided Short-Context Action Anticipation in Human-Centric Videos in which we utilize symbolic domain knowledge to alter a transformer's attention mechanism to improved future action prediction.

Link: https://t.co/JNzSUXM2jS

[Simon Stepputtis @SimonStepputtis]

Please check out our new paper at the IROS-HmRI workshop!

[Joseph Campbell @joecampb]

We have a really cool paper at the Human Multi-Robot Interaction workshop today at IROS in which we show how to use LLMs to produce natural language explanations for agent policies.

Paper: https://t.co/Pu8zb9Drg4

[Simon Stepputtis @SimonStepputtis]

Today, we are presenting our work "Sample-Efficient Learning of Novel Visual Concepts" at @CoLLAs_Conf 2023.

Check out our poster this afternoon at poster 12!

Project Website: https://t.co/CsCmzjRLAU

[Joseph Campbell @joecampb]

What if transfer learning in RL could selectively transfer only beneficial knowledge? What if it was interpretable and we knew exactly what was transferred?

I have a poster today on exactly this topic at CoLLAs.

Check it out: https://t.co/K7QwcLu6sH

#CoLLAs2023

[Simon Stepputtis @SimonStepputtis]

I am excited to announce our talk at #CoLLAs2023 for our paper titled "Sample-Efficient Learning of Novel Visual Concepts"!

With our neuro-symbolic architecture, we quickly learn about novel objects, as well as their attributes and affordances.

Website: https://t.co/CsCmzjRLAU

[Simon Stepputtis @SimonStepputtis]

Our workshop on Articulate Robots is happening right now at #RSS2023 in Room 306A and online!

More information: https://t.co/yJZVQ7E3rL