Here is the technical report on SubQ 1.1 Small.
https://t.co/bu8AEc4lsk
This is the second iteration on our Subquadratic Sparse Attention (SSA) model, and the first to be deployed with design partners in the coming weeks.
The results are compelling and verified by @AppenResearch.
- Near-perfect long-context retrieval up to 12M tokens on the needle-in-a-haystack test, with up to nearly 1,000x attention compute reduction.
- A balance of long-context optimization and general reasoning ability, with strong performance retained across knowledge, coding, and non-coding enterprise agent benchmarks.
- At 1M tokens, SubQ 1.1 Small requires 64.5x less compute than dense attention and runs 56x faster than FlashAttention-2.
These results highlight a significant scaling advantage thanks to the efficiency gains from the SSA architecture.
We included some details and learnings from the development process which may be helpful to the community.
Comment with questions, I’ll try to respond!
Introducing SubQ - a major breakthrough in LLM intelligence.
It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA),
And the first frontier model with a 12 million token context window which is:
- 52x faster than FlashAttention at 1MM tokens
- Less than 5% the cost of Opus
Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention).
Only a small fraction actually matter.
@subquadratic finds and focuses only on the ones that do.
That's nearly 1,000x less compute and a new way for LLMs to scale.
Our Geo Regional Asia group is excited to host @AshmalVayani for a session "Seeing the World as It Speaks: Multilingual, Culturally Aware Multimodal AI" on October 15th.
Thanks to program leads @AhmadMustafaAn1 and @KanwalMehreen2 for organizing this session! 🔥
Learn more: https://t.co/iyYqhpOoMp
[Thinking Beyond Tokens: From Brain-Inspired Intelligence to Cognitive Foundations for Artificial General Intelligence and its Societal Impact]
To get us to AGI, new cognitive foundations are needed.
This paper argues that token prediction alone (of those top LLMs) is not enough for achieving AGI.
We need systems that combine brain-inspired memory, modular reasoning, and other components, along with built-in alignment and governance.
What's emerging Today?
- Modular architectures (e.g., MoEs, neuro-symbolic hybrids)
- Large Reasoning Models (LRMs) and Concept Models (LCMs)
- Multi-agent societies and tool-using agents
What’s missing in today’s LLMs:
- No long-term memory or World model
- Weak causal reasoning and planning
- Shallow alignment (just fine-tuned for alignment)
Key components for future AGI:
- Memory: Lifelong learning, RAG, external memory
- Reasoning: Chain-of-Thought, agent collaboration
- World models: Predictive simulators for long-term goals
- Embodiment: Real-world interaction, vision-action loops
- Social cognition: Emotional awareness, value alignment
Paper link: https://t.co/ou0IXjexza
🚀 Heading to #CVPR2025 in Nashville? Don’t miss the VideoLLMs Workshop on June 11 (Grand A1)!
• 🔑 6 keynotes
• 🗣️ Panel “VideoLLMs vs Expert Models”
• 🏆 $6 K challenge-prize reveal
• 📑 Paper talks and Poster session
See you there!
#AI#ComputerVision#GenerativeAI
Travel grants sponsored by Amazon and Apple are available for participants at CVPR-2025 VidLLMs workshop. Fill out this form if you are an attendee looking for support. Limited grants of $1,000 per person are available. #CVPR@CVPR
https://t.co/6wzIX2rRvQ
🚀 Join the Multilingual Video Reasoning Challenge! 🌍
🎥 879 real videos, 8,025 QA, 15 domains, 14 languages
🧠 Prove your VLM’s cultural IQ in open‑ended & MCQ tracks.
💰 Prizes + global glory!
Register & rules: https://t.co/xlIOoWR0st
#AI#MultilingualAI#CVPR2025
@CathyChenZhao@vidllms@CVPR https://t.co/gpYg0oRzNf
As mentioned in the workshop website above, the submissions are non-archival; they will not be published in the proceedings.
🚀 Join the Multilingual Video Reasoning Challenge! 🌍
🎥 879 real videos, 8,025 QA, 15 domains, 14 languages (Arabic→Urdu).
🧠 Prove your VLM’s cultural IQ in open‑ended & MCQ tracks.
💰 Prizes + global glory!
Register & rules: https://t.co/xlIOoWRyi1
#AI#MultilingualAI#VLM
🚨 Submission deadline for papers to 1st Video LLMs Workshop at #CVPR2025 has been extended by 5 days to April 20 !
📌 Non-archival | 📷 Both Novel and published work welcome 📷
🎯CFP: https://t.co/VNI4cCUJaZ
Submit at: https://t.co/c8ogzwIRr0
📊 We hope SB-bench serves as a critical tool in building fairer, more responsible AI systems. We welcome the community to explore, critique, and expand on this work!
🙏 Thanks to the entire core team! Vishal Narnaware, Rohit Gupta, Sirnam Swetha, and Mubarak Shah.
🚀 Introducing SB-Bench, a comprehensive framework for evaluating stereotype biases in LMMs. With over 7.5K real-world images across 9 domains, it rigorously tests LMMs’ fairness in ambiguous scenarios. ⚖️
🔗 Website: https://t.co/rlxtxHM04I
📖 Paper: https://t.co/rX2Ep892ST
🚀 Introducing SB-Bench, a comprehensive framework for evaluating stereotype biases in LMMs. With over 7.5K real-world images across 9 domains, it rigorously tests LMMs’ fairness in ambiguous scenarios. ⚖️
🔗 Website: https://t.co/rlxtxHM04I
📖 Paper: https://t.co/rX2Ep892ST
🚨 Exciting news! 🚨
We’re thrilled to announce the First Workshop on Video Large Language Models at @CVPR ! 🎥🤖 Join us for an exciting lineup of expert speakers and engaging challenges for participants, with cash prizes for the winners! 💸
#CVPR2025#VidLLMs#VideoLLMs