Ellen Tan

Researcher at @MetaAI (FAIR)

[Kyle Chan @kyleichan]

6. The Influence of Kaiming He
Xie deeply admires Kaiming He, with whom he worked closely at FAIR. He learned that Kaiming's superpower is his extreme focus and ability to build an impenetrable baseline ("scaffold") before aiming for breakthroughs [02:34:30]. Kaiming taught him that the real research "signals" come from the failures and surprises during the experimental process, not from armchair theorizing [02:08:12].

7. "Impact" vs. "Understanding"
Xie dislikes the word "impact," finding it too ego-driven and aggressive. Instead, he views research through the lens of philosopher Hannah Arendt: the goal of research is "understanding." Publishing a paper is about sharing a profound realization with the world to find resonance and intellectual kinship, rather than just changing the world by force [01:31:27].

8. Intelligence is More Than Human Language
Xie cautions against human arrogance in defining intelligence solely through language. Drawing on evolutionary biology, he argues that building an AI with the survival and physical reasoning skills of a squirrel is actually a much harder problem than building an AI that can write code or pass exams [06:13:05].

9. The Power of "Research Taste"
Xie equates research to filmmaking, where "taste" dictates everything from the problem you choose to solve down to the formatting of the paper. Influenced by Kaiming He (who even gifted him the Buddhist Diamond Sutra to teach him to look past superficial "forms" to find true substance), Xie believes good taste means avoiding crowded, hyped areas in favor of fundamental, eternal problems like representation learning [02:45:03].

10. The Degradation of Open Academic AI Research
Xie expresses concern over how major industrial AI labs have become increasingly closed off, shifting from open academic exploration to secretive commercial competition. This shift strips researchers of their autonomy and turns them into easily replaceable cogs in a massive engineering machine, heavily motivating his decision to start a new, more open research-driven company [05:21:26].

[Jason Weston @jaseweston]

Self-Improving Pretraining
We've updated our results given feedback:
- larger 8B baseline to match reward model size
- cross-task evals given different RM objectives
Overall, we see clear wins

[Jason Weston @jaseweston]

📈Self-Improving Pretraining 📈
✍️: https://t.co/GsvYMuMT4b
Reinvents pretraining: no more next token prediction!
- Uses existing LM from last self-improvement iteration to give rewards to pretrain new model on *sequences*
- Large gains in factuality, safety & quality
🧵1/5

[Ellen Tan @ellenxtan0]

@ShangwenLi1 Congratulations and great work!

[Shang-Wen Li @ShangwenLi1]

It's a great pleasure to work with you and present the community an OSS representation natively trained for dense CV tasks. As LLM going so fast, we believe dense perception will also catch up!

[Hu Xu (back on robotics) @Hu_Hsu]

🎉 MetaCLIP accepted as #ICLR2024 spotlight. Big thxs/cong. to @sainingxie @ellenxtan0 @berniebear_1220 @RussellHowes1 @sharmavasu55 @ShangwenLi1 @gargighosh @LukeZettlemoyer @cfeichtenhofer and @_akhaliq post. Code https://t.co/v4FHn1uOjJ w/ ViT-G(IN 82.1)/demo/metadata builder.

[Eric Michael Smith @ericsmithnyc]

Our paper on robust bias evaluation of generative LLMs has been accepted to EMNLP 2023! https://t.co/9KLr86yR6b Thanks to co-authors @davides_io @ellenxtan0 @sagharh @megnung @zycalice @_juderoque Jane Dwivedi-Yu @elepre17 @adinamwilliams

1/2

[Ellen Tan @ellenxtan0]

So proud to share the release of our ROBBIE paper (EMNLP’23) on FAIR's 10-year anniversary. Immensely grateful for my amazing collaborators, without whom this accomplishment would not have been possible.

[AI at Meta @AIatMeta]

Today we're also sharing updates on three important categories of socially responsible AI research. This includes ROBBIE, a new tool to help provide a fuller picture of fairness in LLMs across 12 different demographic axes and five LLMs.

Details ➡️ https://t.co/6kz2CC9agX

[AI at Meta @AIatMeta]

As part of our continued belief in the value of an open approach to today's AI, we've published a research paper with more information on Code Llama training, evaluation results, safety and more.

Code Llama: Open Foundation Models for Code ➡️ https://t.co/u7iXXE08Bd

[AI at Meta @AIatMeta]

Today we’re releasing Code Llama, a large language model built on top of Llama 2, fine-tuned for coding & state-of-the-art for publicly available coding tools.

Keeping with our open approach, Code Llama is publicly-available now for both research & commercial use.

More ⬇️

[AK @_akhaliq]

Demystifying CLIP Data

paper page: https://t.co/tmlFEGtb6F

Contrastive Language-Image Pre-training (CLIP) is an approach that has advanced research and applications in computer vision, fueling modern recognition systems and generative models. We believe that the main ingredient to the success of CLIP is its data and not the model architecture or pre-training objective. However, CLIP only provides very limited information about its data and how it has been collected, leading to works that aim to reproduce CLIP's data by filtering with its model parameters. In this work, we intend to reveal CLIP's data curation approach and in our pursuit of making it open to the community introduce Metadata-Curated Language-Image Pre-training (MetaCLIP). MetaCLIP takes a raw data pool and metadata (derived from CLIP's concepts) and yields a balanced subset over the metadata distribution. Our experimental study rigorously isolates the model and training settings, concentrating solely on data. MetaCLIP applied to CommonCrawl with 400M image-text data pairs outperforms CLIP's data on multiple standard benchmarks. In zero-shot ImageNet classification, MetaCLIP achieves 70.8% accuracy, surpassing CLIP's 68.3% on ViT-B models. Scaling to 1B data, while maintaining the same training budget, attains 72.4%. Our observations hold across various model sizes, exemplified by ViT-H achieving 80.5%, without any bells-and-whistles.

[Ping (Iris) Yu @ping_iris_yu]

Check our Shepherd model that can critique LLM generations! Training data is available on Github: https://t.co/3fSOnFn0AD

https://t.co/WQRtTGSBa1

[Ellen Tan @ellenxtan0]

🥳

[AI at Meta @AIatMeta]

📚 New research from Meta AI — Shepherd is a language model specifically tuned to critique model responses & suggest refinements. It goes beyond the capabilities of untuned models to identify diverse errors & suggest improvements.

Read the paper ➡️ https://t.co/BtrTV0YEun

[Ellen Tan @ellenxtan0]

So proud to be a part of the team 🎉

[Yann LeCun @ylecun]

This is huge: Llama-v2 is open source, with a license that authorizes commercial use!

This is going to change the landscape of the LLM market.
Llama-v2 is available on Microsoft Azure and will be available on AWS, Hugging Face and other providers

Pretrained and fine-tuned models are available with 7B, 13B and 70B parameters.

Llama-2 website: https://t.co/PKrrXgHdem
Llama-2 paper: https://t.co/aINNrXNhMb

A number of personalities from industry and academia have endorsed our open source approach: https://t.co/N7HwgW9Suh

[Lu Tang @huluwa44]

Excited to share our recent paper that got accepted to #NeurIPS2022. We proposed a safety- and fairness-enabling decision-making tool that is causality-driven and robust to the uncertainty due to given variables.
@ellenxtan0

https://t.co/CG0p1UChvo

[INTERPOLATE @ NeurIPS 2022 @interpolate2022]

‼️ ** 3 days left **

Submissions are open for INTERPOLATE: First Workshop on Interpolation Regularizers and Beyond at
@NeurIPSConf
2022.

Applications, theory, and extensions are welcome! 🥳

Paper submission deadline: Sep. 22, 2022
https://t.co/fS9y4MscUZ.

[Lu Tang @huluwa44]

Happy to share our recent paper that got accepted to #ICML2022. We proposed an interpretable ensemble that leverages data across distributed research networks to quantify causal treatment effects.

https://t.co/zUWuPDToSk

[Beth Ann Griffin @BethAnnGriffin1]

We are so excited to share the presentation schedule for the RAND Center for #CausalInference 2022 Symposium on August 18. Please join us for any of our four amazing sessions, or the whole day! https://t.co/beqwegJyGr

[Harvard Data Science Initiative @harvard_data]

🎓 Portobello the rabbit graduates with a Master's in #DataScience from @Harvard. Congratulations to the Classes of 2020, 2021, and 2022! View details for this year's Commencement ceremonies here: https://t.co/zQer7jAZKb #Harvard22 #Graduation