Kaustubh

non compos mentis!

[Iceland Cricket @icelandcricket]

Nerves of ice and the land of spice are into the final!

What a great SF match and near miracle chase by England.

[François Chollet @fchollet]

In the era of algorithmic distraction, the ability to maintain a single thread of thought for four hours is a superpower. It is the only way to solve hard problems.

[Kaustubh @kaustubh_w]

@IndiGo6E @airindia Got the bag now. Thanks.

[Kaustubh @kaustubh_w]

@IndiGo6E lost my luggage and is sending me on a weird loop between call and WhatsApp with one medium pointing to the other. Worse part - none of the numbers shared by ground staff are responsive. #indigolostmyluggage #indigosucks. @airindia is better than this!

[Kaustubh @kaustubh_w]

@IndiGo6E @airindia @IndiGo6E no response on the DMs. Why?

[Andrej Karpathy @karpathy]

The (true) story of development and inspiration behind the "attention" operator, the one in "Attention is All you Need" that introduced the Transformer. From personal email correspondence with the author @DBahdanau ~2 years ago, published here and now (with permission) following some fake news about how it was developed that circulated here over the last few days.

Attention is a brilliant (data-dependent) weighted average operation. It is a form of global pooling, a reduction, communication. It is a way to aggregate relevant information from multiple nodes (tokens, image patches, or etc.). It is expressive, powerful, has plenty of parallelism, and is efficiently optimizable. Even the Multilayer Perceptron (MLP) can actually be almost re-written as Attention over data-indepedent weights (1st layer weights are the queries, 2nd layer weights are the values, the keys are just input, and softmax becomes elementwise, deleting the normalization). TLDR Attention is awesome and a *major* unlock in neural network architecture design.

It's always been a little surprising to me that the paper "Attention is All You Need" gets ~100X more err ... attention... than the paper that actually introduced Attention ~3 years earlier, by Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio: "Neural Machine Translation by Jointly Learning to Align and Translate". As the name suggests, the core contribution of the Attention is All You Need paper that introduced the Transformer neural net is deleting everything *except* Attention, and basically just stacking it in a ResNet with MLPs (which can also be seen as ~attention per the above). But I do think the Transformer paper stands on its own because it adds many additional amazing ideas bundled up all together at once - positional encodings, scaled attention, multi-headed attention, the isotropic simple design, etc. And the Transformer has imo stuck around basically in its 2017 form to this day ~7 years later, with relatively few and minor modifications, maybe with the exception better positional encoding schemes (RoPE and friends).

Anyway, pasting the full email below, which also hints at why this operation is called "attention" in the first place - it comes from attending to words of a source sentence while emitting the words of the translation in a sequential manner, and was introduced as a term late in the process by Yoshua Bengio in place of RNNSearch (thank god? :D). It's also interesting that the design was inspired by a human cognitive process/strategy, of attending back and forth over some data sequentially. Lastly the story is quite interesting from the perspective of nature of progress, with similar ideas and formulations "in the air", with a particular mentions to the work of Alex Graves (NMT) and Jason Weston (Memory Networks) around that time.

Thank you for the story @DBahdanau !

[Kaustubh @kaustubh_w]

Have been waiting (not so patiently) on this number for the 35 minutes so far. @IndiGo6E

[Kaustubh @kaustubh_w]

@tataplayfiber @reliancejio @tataplayfiber Sharad.

[Kaustubh @kaustubh_w]

@tataplayfiber network outage in my area (Bangalore) for more than 17 hours and counting. And ZERO comms about it. @reliancejio let's talk. Tata can't keep up with its uptime and I am fed up.

[Kaustubh @kaustubh_w]

@tataplayfiber @reliancejio The AUDACITY. The complaint I raised yesterday is closed WITHOUT resolving the issue. Brand Tata has never been this low. Guess 1st time for everything.

[Kaustubh @kaustubh_w]

@tataplayfiber @reliancejio Thanks for responding and showing I truly don't matter. Not a single call or a notification from your side. Way to lose customers. Kudos.

[Kaustubh @kaustubh_w]

@RailMinIndia @PiyushGoyal waiting rooms (all of them) at Vasco station shut close and passengers squatting in the platform. Some things will never improve. Shame.

[Kaustubh @kaustubh_w]

@HDFC_Bank Your support ops are beyond tragic. It's impossible to speak to a rep on your support #, the website (online) keeps failing and there's no other way to raise a complaint. @ICICIBank I see your value now. @anjanirathor

[www.sidin.co @sidin]

If Ravi Shastri can catch Covid then what what is point of using alcohol based sanitizers and wipes?

[Anusha Ravi Sood @anusharavi10]

New guidelines issued for lockdown that will come into force from Monday in Karnataka. Food deliveries/takeaways allowed but citizens can't take their vehicles but walk, take away from standalone bars allowed.

[Rakesh K Mishra @3RakeshMishra]

Very preliminary but encouraging result: #Covishield protects against #B1617. Early results using in vitro neutralization assay show that both convalescent (prior infection) sera and Covishield vaccinated sera offer protection against the B.1.617 variant, aka #DoubleMutant.

[Kaustubh @kaustubh_w]

@sleepmonk Was one of the nicest arrival experience at TXL. Dedicated baggage carousels for each gate, easy immigration. Out in less than 5 minutes.

[God @TheTweetOfGod]

You would think people from Nevada would be good at sitting at a table and counting.

[Kaustubh @kaustubh_w]

@TheTweetOfGod @sleepmonk