gonzo_ML

Writing summaries of AI/ML papers that we found interesting.

Authors: @altsoph, @che_shr_cat

[gonzo_ML @gonzo_ml]

Longformer: The Long-Document Transformer

Authors: Iz Beltagy, Matthew E. Peters, Arman Cohan
Paper: https://t.co/6LGjJUF4G2
Code: https://t.co/6gnTtSDwpR

A recent transformer from Allen Institute for AI.

[Lena Voita @lena_voita]

NLP Course | For You - interactive lectures-blogs, research thinking exercises and related papers with summaries. And fun!

These are not just course materials - this is something developed specially for you (yes, you!) out there.
https://t.co/HU7Txe2HMa

#NLProc #NLPCourseForYou

[gonzo_ML @gonzo_ml]

Great thread on ML educators on twitter!

[Amit Chaudhary @amitness]

My favorite ML educators: [1/n]

- @JayAlammar: visual guides on NLP
- @fishnets88: intuition & real-world anecdotes
- @CShorten30, @ecsquendor, @ykilcher : netflix of research papers
- @letiepi: bite-size animated videos on ML
- @nlpguy_: experience on NLP in industry

[gonzo_ML @gonzo_ml]

The experiments were carried out on WSJ and LibriSpeech datasets; CER and WER were used as metrics.

[gonzo_ML @gonzo_ml]

Imputer: Sequence Modelling via Imputation and Dynamic Programming
William Chan, Chitwan Saharia (@Chitwan_Saharia), Geoffrey Hinton (@geoffreyhinton), Mohammad Norouzi (@mo_norouzi), Navdeep Jaitly
Google Research

Paper: https://t.co/kUw6XMiWFy

#nlp #nlg #nonautoregressive

[gonzo_ML @gonzo_ml]

Separately, the authors were able to show that the first option (imitation learning) is the lower bound estimation for the second option (dynamic programming), and the second option, in turn, is the lower bound estimation for the real log-likelihood.

[gonzo_ML @gonzo_ml]

Experiments show the best strategy during the inference stage is a greedy iterative sampling. The stop of sampling is done by timeout or by cycling detection.

[gonzo_ML @gonzo_ml]

Levenshtein Transformer
Jiatao Gu (@thoma_gu), Changhan Wang (@ChanghanWang), Jake Zhao
Facebook AI, 2019

Paper: https://t.co/4gh2gWO5pw
Code: https://t.co/r73beHlzpo

#nlp #nlg #nonautoregressive #transformer

[gonzo_ML @gonzo_ml]

Tests were done on NMT, summarization, and "automatic post-editing for machine translation" tasks, everywhere the model was very effective with almost no quality loss comparing to the baselines.