Chaojun Xiao

Postdoc @TsinghuaNLP @OpenBMB, LLM

[OpenBMB @OpenBMB]

LLMs lie to please you? The physical source of hallucinations has been found. 🤯
Today, we highlight a breakthrough from the OpenBMB and Tsinghua University: the first-ever identification of H-Neurons —— a sparse class of neurons encoding hallucinations inside LLMs.
📄 Paper: https://t.co/MBycfDWknQ

Why it matters:
1️⃣ Pinpoint Detection: A remarkably sparse subset of neurons (<0.1%) can reliably predict hallucinations across diverse domains—from general knowledge to biomedical topics.
2️⃣ The Root Cause: Hallucinations are actually "Over-Compliance." Interventions prove these neurons force the model to prioritize satisfying your prompt (even with false premises) over telling the truth.
3️⃣ Origin Traced: H-Neurons emerge during Pre-training, not post-training. This means the tendency to hallucinate is baked into the base model's "next-token prediction" objective.

It bridges the gap between macroscopic behavior and microscopic mechanisms.
#AI #THUNLP #LLM #Interpretability #HNeurons #MachineLearning

[Chaojun Xiao @xcjthu1]

We have released the paper of InfLLM-V2, an efficient trainable sparse attention mechanism!

Using InfLLM-V2, we can train a sparse-native model with only 5B long-context tokens !!

https://t.co/y1ihrJUKOC

[OpenBMB @OpenBMB]

🤔Loving the new DeepSeek-V3.2?
🔥Remember the first sparse-native models!

✨ InfLLM‑V2: Seamless Long‑Context Adaptation
Paper: https://t.co/rkfLsxupxO

1️⃣ Ultra‑fast adaptation: only 5B long‑text tokens to train sparse attention (vs. ~1T in DSA from DeepSeek-V3.2)
2️⃣ End‑to‑end speedups: 2.1× prefill, 2.3× decode; up to 4–9× kernel speedup at 128K

Top results on long‑context benchmarks and strong deep‑thinking performance—keeping 98.1%/99.7% of dense accuracy while being much faster

Try the first open sparse‑native model:
MiniCPM4.1‑8B｜https://t.co/xCBOryD7V6

[Chaojun Xiao @xcjthu1]

Our reasoning models with trainable sparse attention!!!

[OpenBMB @OpenBMB]

Introducing MiniCPM 4.1-8B: First Open-Source Reasoning LLM with Trainable Sparse Attention

✅ Strong Reasoning Capability: Surpasses similar-sized models on 15 tasks!
✅ Fast Generation: 3x decoding speedup for reasoning
✅ Efficient Architecture: Trainable sparse attention, frequency-ranked speculative decoding

Download Models:
Huggingface: https://t.co/xCBOryD7V6
Github: https://t.co/zaffsLsx5m
Technical Report: https://t.co/9dvi6uD4DX

#AI #MiniCPM #LLM #OpenBMB #ArtificialIntelligence #MachineLearning

[Chaojun Xiao @xcjthu1]

We release Ultra-FineWeb, a high-quality pre-training corpus with 1.1 T tokens !!

[OpenBMB @OpenBMB]

🚀 Introducing Ultra-FineWeb 🔥
~1T English and 120B Chinese tokens!
~Training fuel of MiniCPM4!

🎯 Highlights
~Efficient Verification Strategy: Reduces data verification cost by 90%
~High-Efficiency Filtering Pipeline: Optimizes selection of both positive and negative samples
~Massive High-Quality Pretraining Data: Built on FineWeb and Chinese FineWeb, with 1T English and 120B Chinese tokens
📊 Performance Gains
~+3.613 & +1.331 on English benchmarks vs. FineWeb & FineWeb-edu
~+1.98 & +0.61 on Chinese benchmarks vs. Chinese FineWeb & Chinese FineWeb-edu-v2
🔍 More Info
📄 Paper: https://t.co/T46eBmYhEN
🤗 Dataset: https://t.co/KluL5t2kUn
#UltraFineWeb #AI #LLM

[Chaojun Xiao @xcjthu1]

Our technical report is available for reading on arXiv. We look forward to your thoughts and recommendations!

[OpenBMB @OpenBMB]

🔥MiniCPM4: Ultra-Efficient LLMs on End Devices
🚀Technical Report: https://t.co/wpZLo4dxMe
💡Paper on HuggingFace: https://t.co/Ve69MQdTiL
📥Download Models: https://t.co/fVJ7d5SNNR

[Chaojun Xiao @xcjthu1]

Efficiently scaling the context length !!!!

[OpenBMB @OpenBMB]

🚀 MiniCPM4 is here! 5x faster on end devices 🔥
✨ What's new:
🏗️ Efficient Model Architecture
- InfLLM v2 -- Trainable Sparse Attention Mechanism
🧠 Efficient Learning Algorithms
- Model Wind Tunnel 2.0 -- Efficient Predictable Scaling
- BitCPM -- Ultimate Ternary Quantization
📚 High-Quality Training Data
- UltraClean -- High-quality Pre-training Data Filtering and Generation
- UltraChat v2 -- High-quality Supervised Fine-tuning Data Generation
⚡ Efficient Inference System:
- https://t.co/OQA29fxLVm -- Light Lightweight and Efficient CUDA Inference Framework
- ArkInfer -- Cross-Platform Deployment Framework

📖 Technical Report: https://t.co/LbnE6Es1DP
🤗 Models: https://t.co/8YuCqKbk1q
⭐ GitHub: https://t.co/zaffsLsx5m
#AI #LLM

[OpenBMB @OpenBMB]

🚀 MiniCPM4 is here! 5x faster on end devices 🔥
✨ What's new:
🏗️ Efficient Model Architecture
- InfLLM v2 -- Trainable Sparse Attention Mechanism
🧠 Efficient Learning Algorithms
- Model Wind Tunnel 2.0 -- Efficient Predictable Scaling
- BitCPM -- Ultimate Ternary Quantization
📚 High-Quality Training Data
- UltraClean -- High-quality Pre-training Data Filtering and Generation
- UltraChat v2 -- High-quality Supervised Fine-tuning Data Generation
⚡ Efficient Inference System:
- https://t.co/OQA29fxLVm -- Light Lightweight and Efficient CUDA Inference Framework
- ArkInfer -- Cross-Platform Deployment Framework

📖 Technical Report: https://t.co/LbnE6Es1DP
🤗 Models: https://t.co/8YuCqKbk1q
⭐ GitHub: https://t.co/zaffsLsx5m
#AI #LLM

[Zhiyuan Zeng @ZhiyuanZeng_]

Is a single accuracy number all we can get from model evals?🤔
🚨Does NOT tell where the model fails
🚨Does NOT tell how to improve it

Introducing EvalTree🌳
🔍identifying LM weaknesses in natural language
🚀weaknesses serve as actionable guidance

(paper&amp;demo 🔗in🧵)

[1/n]

[Chaojun Xiao @xcjthu1]

@vinayakchronicl Based on the fitted curve, maybe in 2025-12, we can have a GPT-4-level LLM with only 8B parameters

[Chaojun Xiao @xcjthu1]

1/4 🚀 Densing Law of LLMs 🚀

OpenAI's Scaling Law showed how model capabilities scale with size. But what about the trend toward efficient models? 🤔

We introduce "capacity density" and found an exciting empirical law: LLMs' capacity density grows EXPONENTIALLY over time!

[Chaojun Xiao @xcjthu1]

4/4 Read the full paper: https://t.co/Fb7QK6Yyle

[Chaojun Xiao @xcjthu1]

3/4
Key Corollary:
- Inference costs dropping exponentially💰
- Edge AI gaining importance (Moore's Law × Density Law)📱
- ChatGPT accelerated density growth significantly 🚀
- Model compression ≠ Density improvements 🔄
- Each model has a short "optimal cost-effective period"⚡️

[Chenyang Song @nlp_rainy_sunny]

(Repost) We are thrilled to introduce our new work 🔥#SparsingLaw🔥, a comprehensive study on the quantitative scaling properties and influential factors of the activation sparsity within LLMs.💪

📎Arxiv: https://t.co/DKg3FIiqH5
📎Codes: https://t.co/gj6bfz8lDc

🧵1

[Chaojun Xiao @xcjthu1]

5/5 Our paper aims to offer a fresh perspective on LLM research and inspire more efficient, scalable foundation models. We also discuss open issues and future research directions in this emerging field.

Read the full paper: https://t.co/tTBXrSbAYU

[Chaojun Xiao @xcjthu1]

1/5 🚀 Excited to share our latest paper on Configurable Foundation Models! 🧠

Inspired by the human brain's functional specialization, we propose a concept: Configurable Foundation Model, a modular approach to LLMs.

[Chaojun Xiao @xcjthu1]

4/5 We conducted empirical analyses on Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.3, revealing:
- Sparse activation patterns
- Functional specialization of neurons
- Functional partitions within the models

[Chaojun Xiao @xcjthu1]

3/5 Benefits of our approach:
✅ Efficient inference on resource-limited devices
✅ Dynamic assembly of modules for complex tasks
✅ Scalable capabilities through modular design
✅ Potential for continuous model updates and improvements

[Chaojun Xiao @xcjthu1]

2/5 Key Concepts:
- Emergent bricks: Functional neuron partitions that emerge during pre-training
- Customized bricks: Post-training modules to enhance LLM capabilities
- Brick operations: Retrieval, routing, merging, updating, and growing

[Chaojun Xiao @xcjthu1]

@NllpWorkshop

[TsinghuaNLP @TsinghuaNLP]

How does NLP benefit legal systems? Legal AI focuses on applying AI techniques to legal tasks. In this work (https://t.co/jvn7SskkWi), we summarize the history, the current state and future directions of researches in legal AI. #NLProc #AI #legalAI #TsinghuaNLP