Ilia Kulikov

ramming llms @MetaAI

[Jason Weston @jaseweston]

🥥🌪️ Introducing CoCoMix - a LLM pretraining framework that predicts concepts and mixes them into its hidden state to improve next token prediction.

📈 More sample-efficient and outperforms next token prediction, knowledge distillation, and inserting pause tokens.

🔬Boosts interpretability and steerability by analyzing and modifying the predicted concepts.

📝: https://t.co/VO34CSuDOG

[Ilia Kulikov @uralik1]

We are using fairseq2 for llm post-training research in our team. This release comes with a decent documentation (https://t.co/SUga1vz9P6) 😅 My favorite feature of the lib is the runtime extension support: one can develop research code without forking out the entire lib repo!

[fairseq2 @fairseq2]

👋 Hello world! We’re thrilled to announce the v0.4 release of fairseq2 — an open-source library from FAIR powering many projects at Meta. pip install fairseq2 and explore our trainer API, instruction & preference finetuning (up to 70B), and native vLLM integration.

[Ilia Kulikov @uralik1]

Interested in LLM inference algorithms? Please come and watch our tutorial next week!

https://t.co/SGa8SuXS7L

[Sean Welleck @wellecks]

Curious about inference-time scaling, the #1 trending topic in LLMs?

Come to our NeurIPS tutorial: Beyond Decoding: Meta-Generation Algorithms for LLMs (Tue. @ 1:30)!

https://t.co/SRqIeKUDpl

[Ilia Kulikov @uralik1]

@kchonyc @jaseweston thanks for being such amazing advisors

[Ilia Kulikov @uralik1]

literally me now:

[Ilia Kulikov @uralik1]

@dustinvtran @kchonyc you may find this one interesting too!
https://t.co/17hssNDjx4

[Ilia Kulikov @uralik1]

paper: https://t.co/jImg2bS2DW
code: https://t.co/XWgzRWT8fq

happy holidays🎄

[Ilia Kulikov @uralik1]

🚨new research!
Probability of short sequences tend to be too high with autoregressive NMT (and beyond). We quantified this tendency and defined the oversmoothing rate. We minimize its upper bound, oversmoothing loss, and present our findings!
w/ Maksim Eremeev and @kchonyc !
1/n

[Ilia Kulikov @uralik1]

@kchonyc The proposed regularization expands the dynamic range of the prob and rank of eos when it is not supposed to be. We see improvements in translation quality when large beam sizes (up to 1000) are used. But, the gap between performance w/ smaller and larger beam is still there!
2/n

[Ilia Kulikov @uralik1]

@Tantir @toubabokooomi Да.

[Ilia Kulikov @uralik1]

Apparently ancestral sampling yields high quality translations if we sample enough number of times, but how to choose one of them in the end? @BryanEikema shows how to scale utility computations over large hypotheses spaces efficiently! very cool

[Bryan Eikema @BryanEikema]

Check out our latest work on minimum Bayes risk decoding for NMT! We show that MBR is a robust decision rule and sampling-based approximations scale well with more computation. Unlike MAP, more computation always improves translation quality. paper: https://t.co/KV9a60v9Au 1/4

[Ilia Kulikov @uralik1]

it is over now. Thanks everyone, I thought we will break the gathertown infra!!

[Ilia Kulikov @uralik1]

Poster #90 in gather town, I am there *now*!

[Ilia Kulikov @uralik1]

🚨Our new research at SPNLP21 workshop!🚨
Done with @wellecks & @kchonyc

We worked out a way to measure the mode mismatch on the sequence level between distributions along the sequence modeling pipeline. But it is not easy to use it with big models in real world tasks. (1/6)

[Ilia Kulikov @uralik1]

phew, the room doesn't look overcrowded yet

[Sean Welleck @wellecks]

"Mode recovery in neural autoregressive sequence modeling"

We study mismatches between the most probable sequences of each stage of the "learning chain": ground-truth, data-collection, learning, decoding

Led by @uralik1 w/ @kchonyc, SPNLP talk on Friday

https://t.co/aud3hXF1Zq

[Ilia Kulikov @uralik1]

We have both live QA and poster on Friday afternoon EST timezone:
talk & poster: https://t.co/cDMG2hLTSe (underline link)
arxiv: https://t.co/RVHSnbDpEf

We will be happy to discuss our work with you, please join the session! (6/6)

[Ilia Kulikov @uralik1]

🚨Our new research at SPNLP21 workshop!🚨
Done with @wellecks & @kchonyc

We worked out a way to measure the mode mismatch on the sequence level between distributions along the sequence modeling pipeline. But it is not easy to use it with big models in real world tasks. (1/6)

[Ilia Kulikov @uralik1]

Finally we have studied mode recovery under different learning chain settings and figured that (1) mode recovery cost is non-trivial at every step of the chain and (2) the pattern of mode recovery heavily depends on the property of the ground-truth distribution. (5/6)