Yingshan Chang

[Yingshan Chang @_Yingshan]

Can a Transformer count inductively? ▶️ Yes, but different schema for positional embeddings are required for different forms of counting.

Can we treat counting as a primitive operation of Transformer computation? ▶️ No, because it requires a non-trivial computation budget and particular input-output formats.

Ok, a Transformer's struggle at inductive counting is possibly due to the lack of recursion along the sequence dimension. Then how about recursive LM architectures? ▶️ RNN (Elman) and LSTM easily achieve all forms of counting we’ve asked. Modern architectures (S4, Mamba, RWKV) have degraded performance.

1/n

Check out our paper at https://t.co/WKOl8hfhos 😀

[Anil Seth @anilkseth]

1/3 @geoffreyhinton once said that the future depends on some graduate student being suspicious of everything he says (via @lexfridman). He also said was that it was impossible to find biologically plausible approaches to backprop that scale well: https://t.co/pbMiB8Qgis.

[Gravitational Lens @cation_motion]

OMG, this is killing me 😂😂😂

[Jimin Sun @jimin__sun]

Just landed in Miami to attend #EMNLP2024 🐊

I’ll be presenting the poster of our “Tools fail” paper on Wednesday Nov 13th, 16:00-17:30 at Jasmine — come check out our poster for a chat!

[Shaurya Dewan @SRDewan18]

Our method breaks down the Mutual Information into the Redundancy (R), Synergy (S), and Uniqueness (U) of the conditioning tokens. R is the redundant info from multiple tokens, S is the info from token interactions, and U is the unique information from each token.
4/n

[Shaurya Dewan @SRDewan18]

Diffusion models have advanced significantly, but how well do we understand their workings? How do textual tokens impact output, and where do biases and failures occur? In our @NeurIPS 2024 paper, we introduce DiffusionPID to answer these questions and more. #neurips2024
1/n

[So Yeon (Tiffany) Min @SoYeonTiffMin]

We introduce Situated Instruction Following (SIF), to appear in ECCV 2024! There is inherent underspecification in instructions when humans act as they speak. SIF is addresses these dynamic, temporally evolving intent of instructions in the context of physical human actions.(1/7)

[Aran Nayebi @aran_nayebi]

I’m thrilled to be joining @CarnegieMellon’s Machine Learning Department (@mldcmu) as an Assistant Professor this Fall!

My lab will work at the intersection of neuroscience & AI to reverse-engineer animal intelligence and build the next generation of autonomous agents.
Learn more here: https://t.co/ihlVjbNS37

Feel free to email me if you’re interested or want to collaborate! I’m able to advise PhD students who are either in any department in SCS (@SCSatCMU) or are in Neural Computation (@cmuneurosci).

[Xiaochuang Han @XiaochuangHan]

❓Are there any unique advantages of diffusion-based LMs over autoregressive LMs?
❓Can we scale and instruction-tune diffusion LMs?
​
Presenting "David helps Goliath: Inference-Time Collaboration Between
Small Specialized and Large General Diffusion LMs" at #NAACL2024!
​
📖 https://t.co/D000UUxGVR
⏰ June 17, 11:00am-12:30pm
📌 Oral Presentation, Generation Session; DON ALBERTO 2

[Filip Piekniewski🌻 🐘:@[email protected] @filippie509]

Something to read for the weekend: https://t.co/WON24Rqaux

[The Struggling Scientists @TheStrugglingS4]

The average overworked PhD student

[Pouya Pezeshkpour @PPezeshkpour]

LLMs excel in math. Introducing a new benchmark, we observe: They struggle with creative and many-step questions (even with CoT), their performance varies widely even on similar topics, and they engage in genuine reasoning only in about half of cases. 1/n
https://t.co/nC1BiBQTLZ

[Percy Liang @percyliang]

We should call models like Llama 3, Mixtral, etc. “open-weight models”, not “open-source models”. For a model to be open-source, the code and training data need to be public (good examples: GPT-J, OLMo, RedPajama, StarCoder, K2, etc.). Weights are like an exe file, which would be ridiculous to call open-source.

[Naihao(Neo) Deng @NaihaoDeng]

So excited to see this fascinating work by my labmate Artem🤩. This is an inspiration for everyone who loves animal 🤩.
https://t.co/3GnWI98SaA

[Pratyush Maini @pratyushmaini]

1/We've nailed a framework to reliably detect if an LLM was trained on your dataset: LLM Dataset Inference.

After over a year of thinking of writing about how hard this is, we had a breakthrough that made me quite literally jump from my seat!

📝: https://t.co/B9jvMsoNGW Long🧵

[fly51fly @fly51fly]

[LG] How Far Can Transformers Reason? The Locality Barrier and Inductive Scratchpad
E Abbe, S Bengio, A Lotfi, C Sandon, O Saremi [Apple & EPFL] (2024)
https://t.co/wo6uFNrMOf

- Transformers can be Turing-complete in expressivity, but this does not address learnability. This paper introduces the notion of distribution locality to capture when weak learning is efficiently achievable by regular Transformers.

- The locality measures the least number of tokens required in addition to the token histogram to correlate non-trivially with the target. High locality distributions cannot be learned efficiently, e.g. syllogisms on long chains.

- Agnostic scratchpads still suffer from the locality barrier. Educated scratchpads can help by breaking the locality at each step.

- Notion of 'inductive scratchpad' can both break locality and improve out-of-distribution generalization, e.g. generalizing arithmetic tasks to almost double input size.

- Key results: conjecture backed by experiments relating locality to weak learning; theorem proving locality barrier under assumptions; experimental results on scratchpads.

[Yan Holtz @R_Graph_Gallery]

Looking for the best color palette? 😔

Check the tool we just created with @joseph_barbier:

🎨 2500+ palettes
🐍 Python Library to get them
🔍 Easy-to-use application to find your perfect match

https://t.co/fRyxx7KKqw

Feedback welcome, we're working hard on this right now!

[Ece Takmaz @ecekt2]

We are looking for more reviewers for the Cognitive Modeling and Computational Linguistics Workshop (CMCL @ ACL 2024). The deadline for reviews is June 25. Please contact me or [email protected] if you would like to be a reviewer! https://t.co/RbN8R64Fo6 #nlproc

[Zico Kolter @zicokolter]

I'm thrilled to share that I will become the next Director of the Machine Learning Department at Carnegie Mellon. MLD is a true gem, a department dedicated entirely to ML. Faculty and past directors have been personal role models in my career.
https://t.co/zW6QUWEsVk

[Rohan Paul @rohanpaul_ai]

📌 This paper investigates the dramatic breakdown of state-of-the-art LLMs' reasoning capabilities when confronted with a simple common sense problem called the "Alice In Wonderland (AIW) problem".

This is despite their strong performance on standardized reasoning benchmarks. The key conclusion is that current LLMs lack basic reasoning skills, and existing benchmarks fail to properly detect these deficiencies.

'Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models'

📌 The AIW problem is a concise natural language task that asks: "Alice has N brothers and she also has M sisters. How many sisters does Alice's brother have?" While easily solvable by humans using common sense reasoning (the correct answer is M+1), most tested LLMs, including GPT-3.5/4, Claude, Gemini, LLaMA, Mistral, and others, show a severe collapse in performance, often providing nonsensical answers and reasoning.

📌 Notably, even when LLMs occasionally provide correct answers, they often express strong overconfidence in their wrong solutions and generate confabulations (persuasive but nonsensical explanations) to justify their incorrect responses. Standard interventions like enhanced prompting or asking models to re-evaluate their answers fail to improve performance.

📌 The authors introduce a harder variation called AIW+, which causes an even stronger performance collapse across all tested models, including GPT-4 and Claude 3 Opus, which performed relatively better on the original AIW problem.

📌 This study highlights a striking discrepancy between LLMs' high scores on standardized reasoning benchmarks (e.g., MMLU, ARC, Hellaswag) and their poor performance on the AIW problem, suggesting that current benchmarks do not adequately reflect models' true reasoning capabilities and weaknesses.

📌 The authors emphasize the need for the ML community to develop new reasoning benchmarks that can properly detect such deficits and guide the improvement of LLMs' reasoning skills. They also stress the importance of fully open and reproducible training pipelines, including dataset composition, to enable proper analysis and progress in this area.

[Naihao(Neo) Deng @NaihaoDeng]

Excited to share that our paper "Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMs" has been accepted to #acl2024 findings. We just released an updated version: https://t.co/xJQsCRGczq