huang

[huang @RichestHuang]

@patience_cave cool! any numbers with image input + python tool? feel like it’d score higher. ascii input kinda makes it an arc-agi style test anyway

[Kimi.ai @Kimi_Moonshot]

Releasing the model weights and technical report of Kimi K3.

Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token context window.

New model architecture: 2.5x the intelligence per unit of compute, not just more params.

Alongside Kimi K3, we're opening up more of the stack behind it — high-performance attention kernels, MoE communication library, and infrastructure for running agent environments at scale.

Model weights: https://t.co/7m7eEg6Y0B
Tech report: https://t.co/yeu6cjpMCT
Tech blog: https://t.co/YTfiMSNM1f

[Kimi.ai @Kimi_Moonshot]

We are releasing PerceptionBench, a benchmark that isolates visual perception and evaluates it as a set of atomic capabilities - discovered from how today's models fail, rather than defined in advance.

From frontier-model failures across 42 benchmarks, we derive 10 atomic perceptual capabilities and construct 3,000 verified questions, each isolating a single capability and answerable by looking, with no reasoning or external knowledge required.

Blog: https://t.co/I1QU83CiYv
GitHub: https://t.co/F5E0VjYgh0
Hugingface: https://t.co/zMdyQD8HO4

[huang @RichestHuang]

I'm claiming my AI agent "suvarnabhumi" on @moltbook 🦞

Verification: burrow-BLSK

[huang @RichestHuang]

RT @Kimi_Moonshot: 🥝Meet Kimi K2.5, Open-Source Visual Agentic Intelligence.

🔹Global SOTA on Agentic Benchmarks: HLE full set (50.2%), Bro…

[Kimi.ai @Kimi_Moonshot]

🚀 Hello, Kimi K2! Open-Source Agentic Model!
🔹 1T total / 32B active MoE model
🔹 SOTA on SWE Bench Verified, Tau2 & AceBench among open models
🔹Strong in coding and agentic tasks
🐤 Multimodal & thought-mode not supported for now

With Kimi K2, advanced agentic intelligence is more open and accessible than ever. We can't wait to see what you build!

🔌 API is here: https://t.co/EOZkbOwCN4
- $0.15 / million input tokens (cache hit)
- $0.60 / million input tokens (cache miss)
- $2.50 / million output tokens

🔗 Tech blog: https://t.co/2RP7U3iakZ
🔗 Weights & code: https://t.co/4ukcXB0iP6
🔗 Github: https://t.co/B2bA4SfXBl
Try it now at https://t.co/85jA71X9gw or via API!

[will brown @willcb]

cautionary tale for anyone eagerly diving into RL training

LLMs are a bit more forgiving, but only so much

you should still do it though

[Epoch AI @EpochAIResearch]

1/10 Today we're launching FrontierMath, a benchmark for evaluating advanced mathematical reasoning in AI. We collaborated with 60+ leading mathematicians to create hundreds of original, exceptionally challenging math problems, of which current AI systems solve less than 2%.

[Bill Yuchen Lin @billyuchenlin]

🚨 Introducing WildVision’s datasets for research on vision-language models (VLMs) — ideal for SFT, RLHF, and Eval. One of the first large-scale VLM alignment data collections sourced from human users.

- 💬 WildVision-Chat: Human-VLM conversations with images for VLM training and other data-centric research. (https://t.co/pUiIWUqZMO)

-⚔️ WildVision-Battle: Human preferences in pairwise VLM comparisons, which can be used for RLHF of VLMs. (https://t.co/ZSmnKjJ8GU)

- 🏆 WildVision-Bench: A small benchmark crafted for evaluating VLMs on real-user tasks, similar to the WildBench for LLMs. (https://t.co/WvrpFyYdZw)

🎮 Demo: Contribute more votes on our demo! We also support video-language models now! (https://t.co/Y6eDWCgmh1)

📄 Paper: Our previous arXiv: https://t.co/Va6ZEunUG2. Please stay tuned for a detailed report on our data.

Special thanks to our amazing team: @yujielu_10, @DongfuJiang, @YingziMa311, @chchenhui, @XingyuFu2, @jinggu4ai, @m2saxon;
Advisors: @ChaoweiX, @WenhuChen, Miguel Eckstein, @WilliamWangNLP, @YejinChoinka

Thanks for the support from @huggingface @gradio
@allen_ai

[Yujia Qin @TsingYoga]

视觉-语言模型(VLM)领域在研究些什么？🧐

VLM是一个从去年末开始快速发展的领域，对研究者来说尚有大量“金矿”未被发掘，且当前探索仍然非常初步，对大模型的初学者上手难度较小🥰

以下是帮你快速掌握VLM领域目前发展的文章推荐📰：

1. 从宏观视角整体了解整个领域有哪些具体的探索方向（例如数据配比、Image Encoder选择、VL connector的设计、当前有哪些benchmark、VLM的训练策略等）

a. Cambrian: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
最全最新没有之一的全方位探索
Link: https://t.co/fqS9zVB5AS

b. MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
比较老但仍然推荐一读的文章
Link: https://t.co/Q7HSytHhwB

c. What matters when building vision-language models?
结论相比前两篇有很好的补充
Link: https://t.co/xOTVQj8PZ6

2. VLM特有的提升推理效率方案：设计更优的V-L Attention机制

a. An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
发现vision token存在大量的冗余，可以通过token dropping来大幅提升推理速度而不伤害效果
Link: https://t.co/4hvsgy0nr7

b. VoCo-LLaMA: Towards Vision Compression with Large Language Models
通过类似RMT的token压缩方式减少vision token数量从而提升推理速度
Link: https://t.co/8227vL5Sd0

3. vision encoder的分辨率对模型性能的影响，结论简单粗暴：影响很大，分辨率越大效果越好

a. InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD
Link: https://t.co/hcUik3aVlQ

b. DeepSeek-VL: Towards Real-World Vision-Language Understanding
Link: https://t.co/OBJRnRBlK1

4. VLM模型架构选择：All-in-one Decoder （early-fusion）还是Vision Encoder和Language Decoder分离？

a. Unveiling Encoder-Free Vision-Language Models
https://t.co/puhPIluZEB

b. Chameleon: Mixed-Modal Early-Fusion Foundation Models
https://t.co/p5iWwWHbMA

5. 对于较为主流的VLM分离架构，Vision-Language Connector如何设计？

a. TokenPacker: Efficient Visual Projector for Multimodal LLM
https://t.co/s3N1ntrajw

6. VLM分离架构的最佳训练方式

a. Long Context Transfer from Language to Vision
https://t.co/eoKGiw9ufm

7. LLaVA系列的所有文章+博客

Improved Baselines with Visual Instruction Tuning
https://t.co/8pLH4RNxAZ
https://t.co/amvIg2TynU
https://t.co/cfVN0Pf6at
https://t.co/CKiCyN2d0G
https://t.co/By6hrZyNyU
https://t.co/PMX6iqmwHt

8. 一些快速提升你VLM码力的实战仓库推荐（见图）

（列得不够全希望大家在评论区继续补充）

[Yi R. (May) Fung @ECCV✈️ @May_F1_]

🥳 Excited to share that our work on "Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning" has been accepted as a Findings paper at #ACL2024🇹🇭 ! Many thanks to @steeve__huang and our amazing collaborators~

Check out 📣https://t.co/5eNOLFLVcQ

[Susan Zhang @suchenzang]

our little Flash slowly catching up to its Pro sibling on another benchmark! 🥹

[Daniel Han @danielhanchen]

Had a look through @Grok's code:
1. Attention is scaled by 30/tanh(x/30) ?!
2. Approx GELU is used like Gemma
3. 4x Layernoms unlike 2x for Llama
4. RMS Layernorm downcasts at the end unlike Llama - same as Gemma
5. RoPE is fully in float32 I think like Gemma
6. Multipliers are 1
7. QKV has bias, O no bias MLP no bias
8. Vocab size is 131072. Gemma 256000.

Sadly, it's way too large for me to accelerate finetuning or running with @UnslothAI. 70b Llama fits in 48GB with Unsloth, but Grok will need way too many GPUs :(

[Alon Albalak @AlbalakAlon]

{UCSB|AI2|UW|Stanford|MIT|UofT|Vector|Contextual AI} present a survey on🔎Data Selection for LLMs🔍

Training data is a closely guarded secret in industry🤫with this work we narrow the knowledge gap, advocating for open, responsible, collaborative progress
https://t.co/vpRIXWFdCZ

[NVIDIA @nvidia]

Announced today, we are collaborating as a launch partner with @Google in delivering Gemma, an optimized series of models that gives users the ability to develop with #LLMs using only a desktop #RTX GPU. https://t.co/WgWmC245se

[Ralph Brooks @ralphbrooks]

How to get up to speed on Gemma:

Main blog: https://t.co/uoE31s20AE

NVIDIA Support for Gemma:
https://t.co/ZOQjNAcyzD

[Zain @zainhas]

"Minimal, clean, educational code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization."👀👀👀https://t.co/xd1zGJ5mma

[Vik Paruchuri @VikParuchuri]

Announcing surya OCR - text recognition in 93 languages. It outperforms tesseract in almost all languages, often by large margins.

Find it here - https://t.co/DD2HfwIG9i .

[Aleksander Madry @aleks_madry]

We tend to choose LM training data via intuitive notions of text quality... but LMs are often *un*intuitive. Is there a better way?

w/@logan_engstrom, @axel_s_feldmann: we select better data by modeling how models learn from data. Our method, DsDm, can greatly improve performance:

[Avi Singh @avisingh599]

Excited to announce our new work on using synthetic data for improving mathematical problem solving and code generation in LLMs!

arxiv: https://t.co/1Fz3fD8MoF

A small amount of fine-tuning can lead to large gains (>6% on Hendrycks MATH with Palm-2)