Dong Won (Don) Lee

PhD at Personal Robots Group @MIT advised by @cynthiabreazeal, @lpmorency working on human robot interaction, e14 fellow, amazon fellow, msml cmu, ex-microsoft

[AI at Meta @AIatMeta]

🚀New from Meta FAIR: today we’re introducing Seamless Interaction, a research project dedicated to modeling interpersonal dynamics.

The project features a family of audiovisual behavioral models, developed in collaboration with Meta’s Codec Avatars lab + Core AI lab, that render speech between two individuals into diverse, expressive full-body gestures and active listening behaviors, allowing the creation of fully embodied avatars in 2D and 3D.

These models have potential to create more natural, interactive virtual agents that can engage in human-like social interactions across a variety of settings.

Learn more: https://t.co/vl2jmfE7wX

[Sherjil Ozair @sherjilozair]

Very happy to hear that GANs are getting the test of time award at NeurIPS 2024.

The NeurIPS test of time awards are given to papers which have stood the test of the time for a decade.

I took some time to reminisce how GANs came about and how AI has evolve in the last decade.

[Dong Won (Don) Lee @_dongwonlee]

Can we use a single session level score for alignment align an LLM? Can we use multimodal signals to align a LLM?

Yes, when you use them together!

At #EMNLP2024, we present a new approach for aligning LLMs, called GELI 🍮, which can utilize extralinguistic, multimodal signals such as facial expression and only a single explicit global annotation at the end of the conversation for alignment.

If you’re interested, join us at Dialogue and Interactive Systems on November 12, 14:00-15:30 at BRICKELL.

For more information:
Paper: https://t.co/7ei09qdC51
Code: https://t.co/CPg2oDf0G9
HF: https://t.co/2ywWqmx0cT

[Paul Liang @pliang279]

heading to #emnlp2024! would love to chat with those interested in joining our Multisensory Intelligence research group at MIT @medialab @MITEECS

https://t.co/DzrATcTvET

Our group studies the foundations of multisensory AI to create human-AI symbiosis across scales and sensory mediums.

my collaborators will also be presenting 2 exciting papers on multimodal mixtures of experts and social intelligence in AI agents - you don't want to miss them!

🧵

[Dan Hendrycks @hendrycks]

As an alternative to RLHF and adversarial training, we released short-circuiting.
It makes models ~100x more robust. It works for LLMs, multimodal models, and agents.

Unlike before, I now think robustly stopping models from generating harmful outputs may be highly tractable and not hopeless.

https://t.co/zv0Jk3Twi9

[Phillip Isola @phillip_isola]

New paper: The Platonic Representation Hypothesis

In which we posit that _different_ foundation models are converging to the _same_ representation of reality.

paper: https://t.co/z0rh2gPeCc
website: https://t.co/Qo3BLTo25I
code: https://t.co/T0YUxG77mg

1/8

[Chief AI Officer @chiefaioffice]

AI phone agents are here.

Here are 8 startups enabling this you should know about + funding:

1. Bland AI - undisclosed

- available as an API
- $0.09/minute

[Yi Tay @YiTayML]

New paper from @RekaAILabs 🔥 (yes an actual paper).

This time we're releasing part of our internal evals which we call Vibe-Eval 😃 This comprises of a hard set which imo is pretty challenging for frontier models today.

The fun part here is that we constructed it by trying to wreck our own model (e.g., Reka Core). 😈 However, it turns out on >50% of them, none of the frontier models can do them anyways.

Making hard prompts has a lot of challenges that people often don't talk too much about. We discuss some of them in the paper.

Notably, because our benchmark is hard, we are able to also induce further separability across frontier models.

We benchmark 13 representative multimodal models and share some findings and insights.
🔹 Gemini Pro 1.5 and GPT-4V are roughly equally capable. Gemini 1.5 is really really strong and surpassed our expectations from Gemini 1.0.
🔹 Claude-3 and Reka models are a slight tier behind but substantially much better than any OSS model. Reka Core and Flash are roughly better than Claude-3 on normal tasks. However, on hard tasks they are nerf-ed due to our data collection process.

More details in the paper (link below). 👇

[Abhinav Rao @AetherSuRa]

New paper on LLMs+culture! 🎊🎉

Thrilled to share our work on NormAd, a dataset evaluating whether LLMs can adapt to the diversity of cultural norms worldwide! (Spoiler: they can't!)

ArXiv: https://t.co/vZUSsHC34u

w/ @akhila_yerukola @vishwayvs @_doctor_kat @MaartenSap [1/n]

[Yubin Kim @ybkim95_ai]

I'm excited to share my recent publication in CHIL 2024, "Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Data".

Our study reveals the potential of LLMs as personal health learners with wearable sensors.

Arxiv: https://t.co/G8aZiB2Y9M

[Carlos E. Perez @IntuitMachine]

1/n A Ontology for Agentic AI

Single Agent Architectures
planning, self-correction, and suitability for straightforward tasks.

Examples:
* ReAct (Reason + Act): Iterative process of thought, action, and observation.
* RAISE: ReAct with memory mechanism (short-term and long-term).
* Reflexion: Self-reflection through linguistic feedback for improved success rate and reduced hallucination.
* AutoGPT + P (Planning): Combines object detection, OAM, and LLM-driven planning for robot control.
* LATS (Language-Action Tooling System): Focus on tool augmentation and affordance learning for improved task completion.

Multi-Agent Architectures
Collaboration, communication, leadership, and suitability for complex tasks.

Examples:
* HuggingGPT: Leverages Hugging Face model hub for diverse tool access and task execution.
* CAMEL (Collaborative Agents for Multimodal Execution of Language): Combines visual and linguistic information for real-world task execution.
* Toolformer: Agents learn to use tools through instruction tuning and reinforcement learning.
* Multi-Agent Collaboration via Conditional Delegation and Role Playing: Agents with specialized roles collaborate through delegation and role-playing.

[Carlos E. Perez @IntuitMachine]

AlphaLLM: An LLM that Learns and Improves Itself

Large Language Models (LLMs) have revolutionized the field of Natural Language Processing, demonstrating remarkable capabilities in various tasks. However, they still struggle with complex reasoning and planning, often requiring extensive fine-tuning with high-quality data to improve performance. This reliance on external data presents a significant limitation, hindering the scalability and generalizability of LLMs. To address this challenge, researchers have explored self-improvement techniques, where LLMs learn from their own outputs and feedback. However, the effectiveness of these techniques in complex tasks remains questionable.

The paper "Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing" introduces ALPHALLM, a novel framework that tackles the limitations of current LLMs and paves the way for effective self-improvement. ALPHALLM addresses the key pain points of LLMs by integrating Monte Carlo Tree Search (MCTS) with data synthesis and critic models. This combination enables LLMs to explore a vast space of potential solutions, learn from self-generated data, and receive reliable feedback for continuous improvement.

One of the primary challenges in LLM self-improvement is the scarcity of high-quality data. ALPHALLM tackles this issue with its "imagination" component, which synthesizes new prompts and learning examples. This reduces the reliance on external datasets and allows the LLM to explore a wider range of scenarios. Furthermore, by utilizing MCTS, ALPHALLM efficiently navigates the vast space of possible solutions, considering various strategies and identifying the most promising ones. This approach is particularly effective for complex reasoning and planning tasks, where exploring diverse options is crucial.

Another critical aspect of self-improvement is the ability to evaluate and learn from one's own outputs. However, LLMs may struggle to accurately assess the quality of their responses, especially in complex tasks. ALPHALLM addresses this challenge by incorporating a trio of "critic models" that provide reliable feedback during the search process. These critic models guide the MCTS search towards promising solutions and assign rewards based on the quality of the generated text. This feedback loop enables the LLM to learn from its mistakes and continuously improve its performance.

Prior research in LLM self-improvement has explored various techniques, such as heuristic-based data filtering and principle-based evaluation. However, these methods often suffer from limitations, such as the need for manual rule design or the reliance on the LLM's own reasoning abilities for self-evaluation. ALPHALLM overcomes these limitations by utilizing MCTS for efficient exploration, critic models for reliable feedback, and data synthesis to reduce data dependency.

While the details of the experiments and results are not provided in the available text excerpt, the paper suggests that ALPHALLM significantly improves the performance of LLMs on mathematical reasoning tasks. Notably, the results indicate that ALPHALLM, when applied to LLaMA-2 70B, achieves performance comparable to GPT-4 on benchmark datasets. This highlights the potential of ALPHALLM for achieving state-of-the-art results without relying on extensive external data.

In conclusion, ALPHALLM presents a promising approach for LLM self-improvement, addressing the limitations of current methods and paving the way for more efficient and scalable learning. By integrating MCTS with data synthesis and critic models, ALPHALLM enables LLMs to explore a vast space of solutions, learn from self-generated data, and receive reliable feedback for continuous improvement. This framework holds the potential to unlock the full capabilities of LLMs and revolutionize the way we interact with and utilize these powerful language models.

[andrew gao @itsandrewgao]

Someone just dropped a dataset of 15 trillion tokens (as many as were used to train Llama 3)!!!

Download this now before it gets taken down for “copyright reasons”

Breakdown in thread 🧵 👇👇

[Leena Mathur @lmathur_]

Curious about socially-intelligent AI? Check out our paper on underlying technical challenges, open questions, and opportunities to advance social intelligence in AI agents:

Work w/ @lpmorency, @pliang279

📰Paper: https://t.co/n2ydEbQkvM
💻Repo: https://t.co/kgNb7ZZv5g

🧵1/9

[Maria Khalusova @mariaKhalusova]

I put together a quick colab notebook using Llama-3-8B-Instruct for chatting with a PDF.
Unstructured API for partitioning and chunking a large PDF file, FAISS for vector storage, @langchain for RAG, and quantized Llama-3-8B-Instruct (so that it fits on free Colab's GPU).
This was fun!
https://t.co/y3oTSGpW5G

[Rafael Rafailov @ NeurIPS @rm_rafailov]

We have a new preprint out - your language model is not a reward, it’s a Q function!
1. The likelihood of the preferred answer must go down - it’s a policy divergence
2. MCTS guided decoding on language is equivalent to likelihood search on DPO
3. DPO learns credit assignment

[Xingyu Fu @XingyuFu2]

Can GPT-4V and Gemini-Pro perceive the world the way humans do? 🤔

Can they solve the vision tasks that humans can in the blink of an eye? 😉

tldr; NO, they are far worse than us 💁🏻‍♀️

Introducing BLINK👁 https://t.co/EGDh0bMnyJ, a novel benchmark that studies visual perception abilities NOT yet “emerged” in Multimodal LLMs 🔥🔥

Paper: https://t.co/teFGLiXU12

(1/n)

[Aran Komatsuzaki @arankomatsuzaki]

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

The first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating systems such as Ubuntu, Windows, and macOS

proj: https://t.co/v26QfbZ3Lp
abs: https://t.co/0MA1XrbaIQ

[Diyi Yang @Diyi_Yang]

Learning social skills is out of reach for most people🙁

How can we make social skill training more accessible?

We introduce 🌟APAM🌟 (AI Partner and AI Mentor) that leverages LLMs for social skill training via realistic practice and tailored feedback!

[HAI @HAIPurdue]

We are super proud of Denison Guvenoz and Avery Kruppe, who presented their work, "Evaluating Robots' Social Intelligence", at the Purdue Fall Undergrad Research Expo! This marks the first public presentation by the HAI Group; keep up the great work team!