Anshuman Sahoo

Principal Machine Learning Engineer, Architecture, Engineering & Construction at Autodesk; University of Toronto

[Anshuman Sahoo @anshuML264]

How does Megatron tensor parallelism actually split an MLP across GPUs?

I rebuilt a SwiGLU block in plain PyTorch and verified it against a dense reference - including backward pass and optimizer step.

https://t.co/CooCmAQ9EC

[Yann LeCun @ylecun]

Trump is just the trigger.
There are deep historical forces:
The dismantling of the FCC fairness doctrine by Reagan, the resulting emergence of Fox News, conservative talk radio, and other propaganda outlets, the removal of guardrails for the influence of money in politics (e.g. Citizen United), the continuous reduction of taxes on high incomes, the ever-increasing wealth and income inequalities enabled by the above, the stagnation of working class income despite continued increase in productivity (and corporate profits), the total lack of social protection and benefits compared to other developed countries.
People in the middle class and below have been screwed for 40 years and are angry.
They should be angry at Republicans.
But the Republican party has managed to deflect their anger towards immigrants, non-Christians, non-whites, liberals, LGBT, China, academia, intellectual "elites", scientists, journalists, and non-existent scarecrows like Antifa.

[Anshuman Sahoo @anshuML264]

@itsCblast Trade

[Anshuman Sahoo @anshuML264]

@KevinNaughtonJr zero sum game simulation

[Aston Zhang @astonzhangAZ]

"Imagine learning a textbook with no figures."

Multimodal chain-of-thought (Multimodal-CoT) in Language Models

- Outperform GPT-3.5 by 16% (75%->91%) and surpass human performance on ScienceQA
- Less than 1B params (so you can train more easily)
- Code & model released

[1/6]

[BenchSci @BenchSci]

Introducing #ASCENDbyBenchSci, an intuitive #AI #SaaS platform with the power to save pharma researchers years and millions in their preclinical research. https://t.co/zQBhbujNZM

[Andrej Karpathy @karpathy]

More on cramming: CIFAR10 hyperlightspeedbench.
Train CIFAR10 to 94% in under 10 seconds on a single A100. With a single readable 600-line https://t.co/gVf4g3bzPN, bunch of nice tricks implemented within.
https://t.co/koGgN4CUKU

[amy @a_e_roberts]

Processing images in transformers just became 100x better.

Since 4.25, we've been working on a new class: ImageProcessor.

Here's everything you need to know 👇 🧵 1/

[Andrej Karpathy @karpathy]

Random quick note on Transformer block unification. People are usually a bit surprised that the MLP and Attention blocks that repeat in a Transformer can be re-formated to look very similar, likely unifiable. The MLP block just attends over data-independent {key: value} nodes: