Bernardo O

Computer Science & Mathematics | Ignorance is not bliss

[Gary Marcus @GaryMarcus]

BREAKING: Explosive new paper from MIT/Harvard/UChicago.

Things just got worse — a lot worse — for LLM’s and the myth that they can understand and reason.

The paper documents a pattern they called Potemkins, a kind of reasoning inconsistency (see figure below). They show that LLMs - even models like o3 — make these errors frequently.

You can’t possibly create AGI based on machines that cannot keep consistent with their own assertions. You just can’t.

“success on benchmarks only demonstrates potemkin understanding: the illusion of understanding driven by answers irreconcilable with how any human would interpret
a concept … these failures reflect not just incorrect understanding, but deeper internal incoherence in concept representations”

Game over for any hopes of building AGI on a pure LLM substrate. cc @geoffreyhinton, checkmate.

[Yann LeCun @ylecun]

New paper: turns out you can train deep nets without normalization layers by replacing them with a parameterized tanh()

[Tenobrus @tenobrus]

extremely real. this is also why asking what someone wants to eat or watch or do is.... not as nice as u think it is.

[Eric Fan @ericfan_journo]

Every year, a lottery determines who gets an H-1B visa to work in the US. The game, it turns out, is rigged.

Outsourcing and staffing firms are exploiting loopholes, crowding out US employers and immigrants who play it fair.

We have exclusive data to show it for the 1st time 🧵

[Rohan Paul @rohanpaul_ai]

This is really a 'WOW' paper. 🤯

Claims that MatMul operations can be completely eliminated from LLMs while maintaining strong performance at billion-parameter scales and by utilizing an optimized kernel during inference, their model’s memory consumption can be reduced by more than 10× compared to unoptimized models. 🤯

'Scalable MatMul-free Language Modeling'

Concludes that it is possible to create the first scalable MatMul-free LLM that achieves performance on par with state-of-the-art Transformers at billion-parameter scales.

📌 The proposed MatMul-free LLM replaces MatMul operations in dense layers with ternary accumulations using weights constrained to {-1, 0, +1}. This reduces computational cost and memory utilization while preserving network expressiveness.

📌 To remove MatMul from self-attention, the Gated Recurrent Unit (GRU) is optimized to rely solely on element-wise products, creating the MatMul-free Linear GRU (MLGRU) token mixer. The MLGRU simplifies the GRU by removing hidden-state related weights, enabling parallel computation, and replacing remaining weights with ternary matrices.

📌 For MatMul-free channel mixing, the Gated Linear Unit (GLU) is adapted to use BitLinear layers with ternary weights, eliminating expensive MatMuls while maintaining effectiveness in mixing information across channels.

📌 The paper introduces a hardware-efficient fused BitLinear layer that optimizes RMSNorm and BitLinear operations. By fusing these operations and utilizing shared memory, training speed improves by 25.6% and memory consumption reduces by 61% over an unoptimized baseline.

📌 Experimental results show that the MatMul-free LLM achieves competitive performance compared to Transformer++ baselines on downstream tasks, with the performance gap narrowing as model size increases. The scaling law projections suggest MatMul-free LLM can outperform Transformer++ in efficiency and potentially in loss when scaled up.

📌 A custom FPGA accelerator is built to exploit the lightweight operations of the MatMul-free LLM. The accelerator processes billion-parameter scale models at 13W beyond human-readable throughput, demonstrating the potential for brain-like efficiency in future lightweight LLMs.

[Elon Musk @elonmusk]

All those moments,
will be lost in time
like tears in rain …

[Bernardo O @bernovi]

@GergelyOrosz Got it in 2018 while doing research for the university. My search terms were related to python and python frameworks

[Naval @naval]

NY: What do you have?

LA: Who do you know?

SF: What do you know?

[AI at Meta @AIatMeta]

Meta AI presents CICERO — the first AI to achieve human-level performance in Diplomacy, a strategy game which requires building trust, negotiating and cooperating with multiple players.

Learn more about #CICERObyMetaAI: https://t.co/hG2R0T2HZx

[Bernardo O @bernovi]

My first paper as a co-author! 🎉 I am grateful to have worked w/ @jessyjli @BareaSinno @katwellll @malihealikhani on this #NAACL2022 paper. We used news articles ranging from 1942 - 1974 from various media outlets and NLP models to analyze polarization in political ideology.
👇🏻

[Kate @katwellll]

I am very excited to present the camera-ready version of our new #NAACL2022 paper, Political Ideology and Polarization of Policy Positions: A Multi-dimensional Approach https://t.co/1xySSG8VCo
🎉 (highlights ⬇️)

[Elon Musk @elonmusk]

So much of AI is about compressing reality to a small vector space, like a video game in reverse

[Hugging Face @huggingface]

🚨Transformers is expanding to Speech!🚨

🤗Transformers v4.3.0 is out and we are excited to welcome @facebookai's Wav2Vec2 as the first Automatic Speech Recognition model to our library!

👉Now, you can transcribe your audio files directly on the hub: https://t.co/ABKmyNnz58

[Linda Xie @lindaxie]

This Robinhood / $GME situation is the perfect opportunity to learn about decentralized finance (DeFi) if you haven't already. Here's an explainer I wrote on it last year https://t.co/x7CuyRP9CW

[DHH @dhh]

The HEY stack:
- Vanilla Ruby on Rails on the backend, running on edge
- Stimulus, Turbolinks, Trix + NEW MAGIC on the front end
- MySQL for DB (Vitess for sharding)
- Redis for short-lived data + caching
- ElasticSearch for indexing
- AWS/K8S

[Elon Musk @elonmusk]

Tesla Goes Bankrupt
Palo Alto, California, April 1, 2018 -- Despite intense efforts to raise money, including a last-ditch mass sale of Easter Eggs, we are sad to report that Tesla has gone completely and totally bankrupt. So bankrupt, you can't believe it.

[Bernardo O @bernovi]

@MostFamousQuote Occam's Razor

[MIT CSAIL @MIT_CSAIL]

2014: MIT professor Regina Barzilay is diagnosed with breast cancer.

2017: she develops an ML model that could have diagnosed her cancer years in advance.

Today: she is announced as the first winner of a $1m award for her work.

More: https://t.co/6MFDVeu3AI

[MIT CSAIL @MIT_CSAIL]

#otd in 1955 the term “artificial intelligence” was coined in a conference proposal written by a team organized by John McCarthy: https://t.co/WHHsULnjgJ

[Computer Science, The University of Texas at Austi @UTCompSci]

The @NSF has selected @UTAustin to lead NSF AI Institute for Foundations of Machine Learning, bolstering the university’s existing strengths in this emerging field with a $20 million investment based on the Forty Acres.

[Sharif Shameem @sharifshameem]

This is mind blowing.

With GPT-3, I built a layout generator where you just describe any layout you want, and it generates the JSX code for you.

W H A T