Clarence Lee

[Clarence Lee @ClSheng34]

@ZhijingJin @aclmentorship @aclmeeting Which part of the grand hall foyer is this at?

[Clarence Lee @ClSheng34]

@aclmentorship @aclmeeting @Lianhuiq @ManlingLi_ @zhaofeng_wu @SihangZhao @shannonzshen @ziqiao_ma Which part of the grand foyer is this at? How may we recognise the group

[Clarence Lee @ClSheng34]

@ShayneRedford @seanmcgregor @kevin_klyman @sayashk @RishiBommasani @PeterSlattery1 @random_walker @percyliang This is nice work, congrats!

[Clarence Lee @ClSheng34]

@margs_li @LukeZettlemoyer Congratulations @margs_li, really cool work

[Clarence Lee @ClSheng34]

I am excited about the ideas of this paper: Synthetic Data for any Differentiable Target https://t.co/B3VhYJcwAV. State of the art synthetic data generation is surprisingly heuristical. Meta Learning and targeted synthetic data generation can bring us closer to understanding what data quality really means for training models.

[Clarence Lee @ClSheng34]

Human interaction with AI is becoming the new dataset. In my experience of pretraining LLMs, I’ve always found data curation to be rather surgical, find a target task, optimize for it and do data selection. Real human problems are far more diverse and long-tail than any benchmark captures. Yet, agentic systems represent an exciting new scaling axis: human decision making in the wild. The process of prompting and collaborating with an agent are now structured records of how humans think through problems. The open question is how to efficiently collect and distill these traces into usable training signals.

[Clarence Lee @ClSheng34]

Most people misunderstand the data scarcity issue in pretraining LLMs.

In 2024 NeurIPs, Ilya Sutskever stated that the age of scaling is ending due to us hitting a data wall. However, data scarcity is present not just in the lack of internet crawled data, (which is a rather diverse source), but rather the lack of tacit knowledge (knowledge that people have but find hard to express or write dow). These sources can be expressed in the form of human reasoning or our judgements in day to day tasks.

Autonomous AI Agents and building on them will continue to expose the gaps in how we express or structure knowledge. AI agents are able to quickly exploit information that is well structured. In our quest to develop AI agents, we will inadvertently improve the way we communicate our day to day tasks, which is an overall positive process. Looking forward to seeing how AI agents help to operationalize the day to day gut feelings and intuitions that people have.