Jean Y. Song

Designing Intelligence Augmentation Group / 지능 증강 설계 연구 그룹 /

[Jean Y. Song @jeanysong]

Some of the bus stations in Seoul are equipped with smart facilities.

A digital screen giving a live view outside the station (and showing ads below), smartphone charging station, air conditioner, air purifier, and a voice guide announcing the next arriving bus right on time.

[Elisa Nguyen @_elinguyen]

✨New preprint alert!

Happy to share our latest research: "Towards User-Focused Research in Training Data Attribution for Human-Centered Explainable AI"

📎https://t.co/7l7Qjqlrms

Work with Johannes Bertram, @EKortukov, @jeanysong, @coallaoh

1/9

[Jean Y. Song @jeanysong]

@yanchen1191 @LearningAtScale @gezhang001 @XiaohangTang Congrats!

[Yan Chen @HCI_Prof_YC]

Honored to receive the Best Paper Award 🏆
@LearningAtScale !

Go team! @gezhang001 @XiaohangTang Prof. Steve Oney.

The structure and flow in students’ code are key factors in deciphering their learning. CFlow enables instructors to discern and analyze these patterns at scale.

[Seong Joon Oh @coallaoh]

For industrial AI engineers: Join our 45-minute user study to win 25€ ($27.05).

Here's the link:
https://t.co/84RjI9mPVh

We're seeking novel directions for XAI, particularly in data-centric XAI, where we try to find the cause of wrong ML behaviours in training data. We're motivated by the observation that practitioners don't really use XAI techniques. We aim to develop usable XAI interfaces that actually create value for AI engineers.

@_elinguyen is leading the study. @jeanysong is a great collaborator of ours :)

[Jean Y. Song @jeanysong]

ML개발자 및 연구자분들께:
독일의 University of Tübingen과 한국의 DGIST가 공동으로 진행 중인 연구에서 설문 참여자를 모집합니다. 본 설문의 목적은 "모델 개발에서 데이터 속성(data attribution)의 역할"을 이해하는데 있습니다. 설문 참여에 총 45분 정도가 소요됩니다. 설문링크:

[Jean Y. Song @jeanysong]

Hi, ML ppl around the world 🌏
We (with @_elinguyen & @coallaoh) are working on an exciting study to understand "data attribution in model development" and would love to hear your input.
You can participate through the following survey link that takes ~45mins to complete.

[Jean Y. Song @jeanysong]

https://t.co/i8OTiX0Zk0

설문과 답변이 모두 영어로 이루어져야하는 점 양해부탁드립니다.

저희가 개발한 간단한 시스템도 사용해보실 수 있으니, 관심있으신 전문가분들의 많은 참여 부탁드립니다.

선착순 40명께는 25€ 바우처를 이메일로 보내드립니다.

RT 부탁드려요 :-) 감사합니다!

[Jean Y. Song @jeanysong]

ML개발자 및 연구자분들께:
독일의 University of Tübingen과 한국의 DGIST가 공동으로 진행 중인 연구에서 설문 참여자를 모집합니다. 본 설문의 목적은 "모델 개발에서 데이터 속성(data attribution)의 역할"을 이해하는데 있습니다. 설문 참여에 총 45분 정도가 소요됩니다. 설문링크:

[Jean Y. Song @jeanysong]

Link to the survey: https://t.co/i8OTiX0Zk0

The first 40 ppl to complete the survey will receive a 25€ voucher to use.

We'd appreciate your valuable and sincere participation on the study✨️
Please feel free to RT this post to your circle :-)

[Jean Y. Song @jeanysong]

Hi, ML ppl around the world 🌏
We (with @_elinguyen & @coallaoh) are working on an exciting study to understand "data attribution in model development" and would love to hear your input.
You can participate through the following survey link that takes ~45mins to complete.

[Jeongeon Park @jeongeonp_]

[Please RT] We are looking for {grad students, researchers} who will be giving a {conference, defense, or any academic} talk soon, and planning a rehearsal!

Participate in our study and receive a $20 Gift Card. Sign up here: https://t.co/L2YzM9Vy5z

[Jeongeon Park @jeongeonp_]

Are you presenting at #FAccT2024, #NACCL2024, or #DIS2024?🎙️
Planning to conduct a rehearsal before your speech?
Want to try using an LLM to receive feedback?

Participate in our survey + interview and receive a $20 reward! 💸
Sign up through this link ➡️ https://t.co/L2YzM9V0g1

[Jean Y. Song @jeanysong]

DGIST 학부의 열정 넘치는 학생들이 참신한 주제로 연구를 하고자하는데, 사전 설문조사가 필요하다고 합니다. 연구 경험이 있으신 분들의 많은 참여 부탁드립니다!

[Jeongeon Park @jeongeonp_]

저희 연구실에서 학부생 인턴 분들이 [연구자들이 겪는 어려움과 '생각의 구조화' 관련 설문조사] 를 진행하고 있습니다.
연구 경험이 있으신 분들의 많은 참여 부탁드립니다! (예상 소요 시간 5분 이내)

설문 링크: https://t.co/XU7ReMAQV3
상세 내용 (스레드 참고): 👇

[Jean Y. Song @jeanysong]

많은 RT 부탁드릴게요!

[Jean Y. Song @jeanysong]

안녕하세요, 2년 째 프로그래밍기초 과목의 조교를 담당하고 있는 학생들과 함께, 새로운 연구를 준비하고 있습니다.
"공부와 학습에 ChatGPT를 사용하는 것"에 대한 우려나 의견이 있으신 교수자 및 다양한 교육자 분들의 고견을 듣고자 합니다. 설문 참여 부탁드립니다🙏

https://t.co/9gHya9akDg

[Jean Y. Song @jeanysong]

ChatGPT와 같은 대규모언어모델의 사용을 긍정적으로 바라보시는 시선, 부정적으로 바라보시는 시선 모두모두 환영합니다! 더욱 다양한 의견이 모일수록 더 양질의 연구결과가 나올 것이라고 기대합니다...❤️

[Andrew Ng @AndrewYNg]

Tool use, in which an LLM is given functions it can request to call for gathering information, taking action, or manipulating data, is a key design pattern of AI agentic workflows. You may be familiar with LLM-based systems that can perform a web search or execute code. Some of the large, consumer-facing LLMs already incorporate these features. But tool use goes well beyond these examples.

If you prompt an online LLM-based chat system, “What is the best coffee maker according to reviewers?”, it might decide to carry out a web search and download one or more web pages to gain context. Early on, LLM developers realized that relying only on a pre-trained transformer to generate output tokens is limiting, and that giving an LLM a tool for web search lets it do much more. With such a tool, an LLM is either fine-tuned or prompted (perhaps with few-shot prompting) to generate a special string like {tool: web-search, query: "coffee maker reviews"} to request calling a search engine. (The exact format of the string depends on the implementation.) A post-processing step then looks for strings like these, calls the web search function with the relevant parameters when it finds one, and passes the result back to the LLM as additional input context for further processing.

Similarly, if you ask, “If I invest $100 at compound 7% interest for 12 years, what do I have at the end?”, rather than trying to generate the answer directly using a transformer network — which is unlikely to result in the right answer — the LLM might use a code execution tool to run a Python command to compute 100 * (1+0.07)**12 to get the right answer. The LLM might generate a string like this: {tool: python-interpreter, code: "100 * (1+0.07)**12"}.

But tool use in agentic workflows now goes much further. Developers are using functions to search different sources (web, Wikipedia, arXiv, etc.), to interface with productivity tools (send email, read/write calendar entries, etc.), generate or interpret images, and much more. We can prompt an LLM using context that gives detailed descriptions of many functions. These descriptions might include a text description of what the function does plus details of what arguments the function expects. And we’d expect the LLM to automatically choose the right function to call to do a job.

Further, systems are being built in which the LLM has access to hundreds of tools. In such settings, there might be too many functions at your disposal to put all of them into the LLM context, so you might use heuristics to pick the most relevant subset to include in the LLM context at the current step of processing. This technique, which is described in the Gorilla paper cited below, is reminiscent of how, if there is too much text to include as context, retrieval augmented generation (RAG) systems offer heuristics for picking a subset of the text to include.

Early in the history of LLMs, before widespread availability of large multimodal models (LMMs) like LLaVa, GPT-4V, and Gemini, LLMs could not process images directly, so a lot of work on tool use was carried out by the computer vision community. At that time, the only way for an LLM-based system to manipulate an image was by calling a function to, say, carry out object recognition or some other function on it. Since then, practices for tool use have exploded. GPT-4’s function calling capability, released in the middle of last year, was a significant step toward general-purpose tool use. Since then, more and more LLMs are being developed to similarly be facile with tool use.

If you’re interested in learning more about tool use, I recommend:
- Gorilla: Large Language Model Connected with Massive APIs, Patil et al. (2023)
- MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action, Yang et al. (2023)
- Efficient Tool Use with Chain-of-Abstraction Reasoning, Gao et al. (2024)

Both Tool Use and Reflection, which I posted about last week, are design patterns that I can get to work fairly reliably on my applications — both are capabilities well worth learning about. In the future, I’ll describe the Planning and Multi-agent collaboration design patterns. They allow AI agents to do much more but are less mature, less predictable — albeit very exciting — technologies.

[Original text: https://t.co/gHCOYSsKQO ]

[Christopher Manning @chrmanning]

LLMs and other generative AI are enormously powerful, because they soak up, abstract, and can mashup the work of millions of humans. But they are only a bit more intelligent than an encyclopedia. Central to intelligence is the ability to learn, adapt, and act in novel situations.

[Jeff Dean @JeffDean]

I’m very excited to share our work on Gemini today! Gemini is a family of multimodal models that demonstrate really strong capabilities across the image, audio, video, and text domains. Our most-capable model, Gemini Ultra, advances the state of the art in 30 of 32 benchmarks, including 10 of 12 popular text and reasoning benchmarks, 9 of 9 image understanding benchmarks, 6 of 6 video understanding benchmarks, and 5 of 5 speech recognition and speech translation benchmarks. Gemini Ultra is the first model to achieve human-expert performance on MMLU across 57 subjects with a score above 90%. It also achieves a new state-of-the-art score of 62.4% on the new MMMU multimodal reasoning benchmark, outperforming the previous best model by more than 5 percentage points.

Gemini was built by an awesome team of people from @GoogleDeepMind, @GoogleResearch, and elsewhere at @Google, and is one of the largest science and engineering efforts we’ve ever undertaken. As one of the two overall technical leads of the Gemini effort, along with my colleague @OriolVinyalsML, I am incredibly proud of the whole team, and we’re so excited to be sharing our work with you today!

There’s quite a lot of different material about Gemini available, starting with:

Main blog post: https://t.co/NzSycJl7aE

60-page technical report authored by th Gemini Team: https://t.co/CEdMRyYSLo

In this thread, I’ll walk you through some of the highlights.