Letian Peng

ボクは旅に出よう！

[Shibo Hao @Ber18791531]

🍫 CocoaBench v1.0 is out!

CocoaBench is a benchmark for unified digital agents, built around open-world tasks that require composing 💻 coding, 👀 vision, 🌐 search.

Since our first research preview last December, we have expanded the benchmark substantially with community contributed tasks, and spent months testing and refining the tasks, evaluations, and agent runs.

Some takeaways:
• Even the best agent system reaches only 45.1% on CocoaBench v1.0.
• Coding agents like Codex are already surprisingly strong on general tasks beyond software engineering.
• Stronger agents tend to push more of the work into code.
• Open source models still lag behind leading frontier models on these general tasks.

👇More on the website and in the paper

#AI #Agents #LLM #Benchmark #CocoaBench

[Letian Peng @LetianPeng]

@Ber18791531 素晴らしい！

[Shibo Hao @Ber18791531]

🍫 CocoaBench is calling for contributions from the community! Join us and help shape how next-generation agents are evaluated and built🚀✨

#LLM #AI #Agent #CocoaBench

More details in the threads 👇

[Albert Ge @albert_ge_95]

🔭 Towards Extending Open dLLMs to 131k Tokens
dLLMs behave differently from AutoRegressive models—they lack attention sinks, making long-context extension tricky.
A few simple tweaks go a long way!!
✍️blog https://t.co/Epf2y2Lnsk
💻code https://t.co/c04Cj5iT1y

[Letian Peng @LetianPeng]

🚀 New Preprint Alert: LLMs can be radically efficient when their pre-training objectives are tailored for specific paradigms! 🌱💡

Introducing Cuckoo 🐦: a 0.3B LM specialized for Information Extraction (IE), pre-trained solely on extractive patterns of language modeling.

🔓 Cuckoo unlocks the scaling potential of IE models by shifting from vocabulary prediction to token tagging - enabling lightweight, task-specific performance that scales effortlessly with trillions of LLM's pre-training data.

🔍 Key Insights:
🪶 Tiny Yet Mighty: 96% fewer parameters and 20X faster than standard 8B LLMs.
🎯 Paradigm-Tuned: Tags tokens instead of predicting them, aligning perfectly with IE tasks.
🚀 Free-Riding Evolution: Cuckoo leverages LLM advancements without extra cost.

🔥 Performance:
- Outperforms an 8B LLaMA on few-shot IE tasks (entities, relations, machine reading comprehension (MRC), & instruction-following).
- Beats IE pre-training on human-labeled web links, GPT-4-generated data, & MRC datasets.

📄 Paper: https://t.co/HGpw9YoC82
🛠️ Code: https://t.co/L3oUFyql5s
Time to let your Cuckoo fly! 🐦💥 #LLM #InformationExtraction #NLP

[Rohan Paul @rohanpaul_ai]

The paper addresses the challenge of understanding how LLMs generalize knowledge, especially in compositional tasks.

LLMs struggle with basic knowledge composition, like reverse or transition inference.

This paper proposes that linear correlations exist between related knowledge within LLMs. It suggests that a linear transformation can map next token prediction logits from one knowledge prompt to another, mirroring human knowledge composition.

-----

📌 LLMs implicitly use linear transformations for knowledge composition. This linearity, observed in logit space, explains both generalization and hallucination as a function of transformation precision.

📌 The resilient linear correlation, even after fine-tuning, suggests a fundamental architectural bias in LLMs. This bias, rooted in vocabulary representations, predetermines how knowledge is composed and generalized.

📌 The linear transformation matrix W acts as a knowledge composition operator. Its weights directly reflect real-world correlations and its precision governs the accuracy of knowledge transfer across related prompts.

----------

Methods Explored in this Paper 🔧:

→ The authors fit a linear transformation (W, b) between the next token prediction logits of related prompts. For example, they examined the relationship between "X lives in the city of" and "X lives in the country of".

→ They sampled numerous output logits from prompts with various inputs to fit this transformation.

→ Pearson correlation coefficients were used to evaluate the linear relationship between different knowledge types.

→ The precision of the linear transformation W was analyzed by checking if its weights align with real-world knowledge pairs, using Hit@Top-N metrics.

-----

Key Insights 💡:

→ A linear transformation can approximate the relationship between next token prediction logits for related knowledge.

→ This linear correlation is resilient to large-scale fine-tuning of LLMs.

→ The linear transformation's weights often mirror real-world knowledge relationships. High weights are assigned to correct knowledge pairs like (Paris, France) in the City→Country example.

→ When the linear transformation is precise, it can enable compositional generalization.

→ However, an imprecise linear transformation can lead to compositional hallucination, where learning one knowledge incorrectly generalizes to related knowledge.

→ Vocabulary representations are crucial for forming these linear correlations and enabling knowledge composition.

-----

Results 📊:

→ High Pearson correlation coefficients were observed for related knowledge pairs like City→Country (0.89) and Math operations (0.93 for X+1→X+2).

→ Lower correlation was found for less related pairs like CEO→Company (0.55) and cross-language pairs.

→ In City→Country knowledge composition, W precision reached 42% for top-1 influenced cities and 67% for top-1 influencing countries using Hit@Top-1 metric.

→ Generalization success was significant (53.70% for City→Country) only when both correlation intensity and W precision were high.

[Letian Peng @LetianPeng]

✨My New ICLR2025 pub: Some knowledge is memorized by LLMs🤖, but cannot be sampled by even prompting 1,000,000 times😵‍💫 - we provide a simple way to find it. Here's why and how.

Why a prompt cannot hold all potential answers? Because answer (next token) representations are grouped in clusters. One example is a character cluster (A, B, C, ...). Whenever the LLM wants to predict one of those characters (e.g., What is a white and black animal?➡️P➡️Panda🐼), it cannot avoid predicting other characters (e.g., What is a white and black animal?➡️Q, V, X, ...➡️❌) with high probability (rank). On the other hand, some correct answers in other clusters are lost because a cluster dominates the probability.

We find that LLMs memorize out-of-cluster knowledge even if it's almost impossible to sample it from the initial prompt. We get such knowledge by prompting "What is a white and black animal other than Panda?" This eliminates the probability of "P" (and the character cluster) and navigates the LLM to search in another cluster for the answer. The navigation successfully finds the correct knowledge that is missed by the initial prompt.

This phenomenon also reminds us of recent efforts to sample reasoning paths for reinforcement learning. Sometimes, we fail to sample any correct thinking path from the initial prompt. Maybe we need a smarter navigation strategy to explore all potential thoughts hidden by the spurious correlation between vocabularies.

➡️Dive deeper into our work, which includes the application of navigation to math reasoning and data synthesis: https://t.co/GP8DSlqKPy

➡️View the spurious correlation: https://t.co/cP8jP1fgie

➡️Code: https://t.co/ZtBPESY7l1

[Letian Peng @LetianPeng]

New preprint: Do you know LLMs 🤖 learn knowledge composition in their parameters? We attribute a certain kind of compositional generalization to the linear correlation between answers from LLMs for certain prompts.

That is to say, if LLM believes X lives in the city of Paris🗼, then LLM will also believe X lives in the country of France 🇫🇷. The correlation is also resilient to gradients, thus updating X’s residential city to NYC 🗽 will also update X’s residential country to USA 🇺🇸.

However, some correlations learned by LLM are imprecise. For example, Indianapolis 🏙️ is correlated with India 🇮🇳 rather than the USA 🇺🇸 (Ground-truth). We find that learning that X lives in Indianapolis 🏙️ will cause hallucinations as if X lived in India 🇮🇳.

➡️ Dive deeper into the connections of correlation, generalization, and hallucination: https://t.co/SKIFczOJQn

➡️ Code: https://t.co/zpqC7vBfUV

[Shang Data Lab @UCSD @shangdatalab]

We're thrilled to announce that 4 groundbreaking papers by our brilliant students have been accepted to #ICLR2025! 🎉 Focused on core representations in #LLM, our research covers both their limitations and applications. Join us in Singapore 🇸🇬 for these exciting insights! 🚀

[Shang Data Lab @UCSD @shangdatalab]

Thrilled to share that our brilliant team at #SDLab has published 6 papers (4 Main, 2 Findings) at #EMNLP2024! From incubating small models by LLMs to addressing risks in LLM benchmarking, our research pushes the boundaries of #LLM and #NLP. Huge congrats to the team! 🚀💡🌴☀🐊

[Letian Peng @LetianPeng]

🎇With great pleasure, we announce that 🛌InBedder has been accepted by the main conference of #ACL2024!🎇

🛌InBedder is a great choice to customize the semantics in your text embeddings!

Code: https://t.co/bgJR0uglh5

Paper: https://t.co/IMNdc2z764

[Letian Peng @LetianPeng]

As a boss, you want to cluster topics that people care about but your embedder does it on opinions?

To analyze your texts smartly, check our new instruction-following embedder: InBedder!

⭐Star our repo to support us: https://t.co/zo2fH1e8fn
📄Paper: https://t.co/ix4J2rQ2L9

[Letian Peng @LetianPeng]

Great thanks to the talent authors that contribute their intelligence to InBedder! @shangjingbo @YuweiZh49446108 @zlwang_cs @zihanwangki

[Letian Peng @LetianPeng]

As a boss, you want to cluster topics that people care about but your embedder does it on opinions?

To analyze your texts smartly, check our new instruction-following embedder: InBedder!

⭐Star our repo to support us: https://t.co/zo2fH1e8fn
📄Paper: https://t.co/ix4J2rQ2L9