David Low

Director of Advanced Analytics @ Singtel

[Andrej Karpathy @karpathy]

Promising. Everyone should hope that we can throw away tokenization in LLMs. Doing so naively creates (byte-level) sequences that are too long, so the devil is in the details.

Tokenization means that LLMs are not actually fully end-to-end. There is a whole separate stage with its own training and inference, and additional libraries. It complicates the ingest of additional modalities. Tokenization also has many subtle sharp edges. Few examples:

That "trailing whitespace" error you've potentially seen in Playground? If you end your (text completion API) prompt with space you are surprisingly creating a big domain gap, a likely source of many bugs:
https://t.co/f2PBaw2iA8

Tokenization is why GPTs are bad at a number of very simple spelling / character manipulation tasks, e.g.:
https://t.co/XR3d5g4uwp

Tokenization creates attack surfaces, e.g. SolidGoldMagikarp, where some tokens are much more common during the training of tokenizer than they are during the training of the GPT, feeding unoptimized activations into processing at test time:
https://t.co/y72eaIeRrP

The list goes on, TLDR everyone should hope that tokenization could be thrown away. Maybe even more importantly, we may find general-purpose strategies for multi-scale training in the process.

[Jim Fan @DrJimFan]

Why does ChatGPT work so well? Is it “just scaling up GPT-3” under the hood? In this 🧵, let’s discuss the “Instruct” paradigm, its deep technical insights, and a big implication: “prompt engineering” as we know it may likely disappear soon:👇

[Xavier Bresson @xbresson]

I recently spoke with a journalist about applications of Graph Neural Networks.

I made a few slides to facilitate the discussion.

I share them here -- hopefully they can be useful :)

https://t.co/8pFyIlnrhg

[hardmaru @hardmaru]

How many can you recognize? 👶🏻

(From @TheInsaneApp)

[Xavier Bresson @xbresson]

I have a postdoctoral opening for a project GraphNNs+Ophthalmology. The position is available asap. The duration is one year, renewable another year on mutual agreement. Given the current restrictions, priority will be given to Singaporeans/PRs.

[Natasha Jaques @natashajaques]

I couldn't resist. With contributions from @maxhkw

[Xavier Bresson @xbresson]

My main talks on Graph Neural Networks in 2020

1. Introduction to GNNs
https://t.co/TmkJwFF3ds

2. Recent developments in GNNs
https://t.co/mter34Qutg

3. Benchmarking GNNs
https://t.co/fXz2zOllAC

Hope they can be useful.
Happy new year to everyone !

[Xavier Bresson @xbresson]

Sharing my lecture slides on "Recent Developments of Graph Network Architectures" from my deep learning course. It is a review of some exciting works on GNNs published in 2019-2020. #feelthelearn
https://t.co/nS2OAgHFPo

[Xavier Bresson @xbresson]

Hi #Singapore! Do you want to learn #AI and #DeepLearning with an #expert? I will #teach a 3-day #course (lectures and coding exercises) on Dec 7-9 at One-North.

Register here: https://t.co/hLoX28wFh3

@SkillsFutureSG @SmartNationSG @SGInnovate

[Chaitanya K. Joshi @chaitjo]

"Multi-Head Attention: Collaborate Instead of Concatenate" -@jb_cordonnier @loukasa_tweet M. Jaggi (EPFL).

Multi-head attention is critical to train Transformers, but most attention heads can later be pruned/are redundant. This paper asks why! 🧐

Paper: https://t.co/Y46Enfidbt

[Xavier Bresson @xbresson]

.@ylecun @OriolVinyalsML @wellingmax @PeterWBattaglia @69alodi S. Jegelka S. Osher and I are organizing an exciting workshop on the intersection of deep learning and combinatorial optimization at @ipam_ucla on Feb 22-26 2021.

https://t.co/FZCfkB30hQ

[hardmaru @hardmaru]

Your state-of-the-art method is only state-of-the-art if other people can actually use it.

[Denny Britz @dennybritz]

I built a library and API to analyze and visualize the AI/ML citation graph. It comes with notebooks that you can run locally or in Google Colab:

https://t.co/LdrcB0cTVY

What is this useful for? Here's an example (1/3)

[Jake VanderPlas @jakevdp]

The frequency of random seeds between 0 and 1000 on github (data from https://t.co/xwutMzNI2N)

[Richard Socher @RichardSocher]

Introducing ProGen, a large language model trained on 280 million protein sequences that can generate viable proteins based on user specifications. A step towards AI & #nlpproc helping cure disease and clean our planet.
Paper: https://t.co/kFmTC6RpYl
Blog: https://t.co/UCBxHqlTPl

[Xavier Bresson @xbresson]

New paper on benchmarking graph neural networks w/ @vijaypradwi @chaitjo T. Laurent and Y. Bengio

https://t.co/OXq9uJejVm

Our goal was to identify trends and good building blocks for GNNs.

[Chaitanya K. Joshi @chaitjo]

Excited to share a blog post on the connection between #Transformers for NLP and #GraphNeuralNetworks (GNNs or GCNs).

https://t.co/kb00tG8D9B

[Olivier Grisel @ogrisel]

Plotly's interactive parallel coordinates plot is really nice to build some insights on the impact of hyper-parameters on the generalization performance.

[evolvingstuff @evolvingstuff]

Transformers as Soft Reasoners over Language

"we explore whether transformers can similarly learn to reason (or emulate reasoning), but using rules expressed in language, thus bypassing a formal representation."

https://t.co/5uib4bRPud
Datasets and demo: https://t.co/wKxgdCpHiE

[OpenAI @OpenAI]

We've trained an AI system to solve the Rubik's Cube with a human-like robot hand.

This is an unprecedented level of dexterity for a robot, and is hard even for humans to do.

The system trains in an imperfect simulation and quickly adapts to reality: https://t.co/O04izt3KvO