1/13. Rufus-Air (https://t.co/NgbCbDx12t) π§΅
Sep 2025βApr 2026, minus a 2-month production pause: a small research team at Amazon Search built an open post-training recipe from scratch on GLM-4.5-Air-Base, using public data and open-source components, rarely over 400 H200s.
π Happy New Year everyone!
π€ New preprint: TinyHelen's First Curriculum: Training and Evaluating Tiny Language Models in a Simpler Language Environment
π€ We train and evaluate tiny language models (LMs) using a novel text dataset with systematically simplified vocabularies and linguistic structures, mimicking how children learn language through simplified environments as part of their initial curriculum.
π€ Motivation: Training LMs and their application agents has become costly due to the large datasets and models required. Simplified language environments serve as a basic training and testing ground for LMs, presenting commonsense knowledge and communication skills in a plainer way. This can enhance learning efficiency, potentially reducing the model size and data volume needed for effective training. Strategies developed in these environments for smaller models, agents, and datasets may apply to larger, more complex language environments.
π€ What Has Been Done:
π We introduce a "no noise, low complexity" principle, supporting our proposed text simplification pipeline that transforms the original training data of LMs into a version with reduced noise and complexity, which have been proven to enhance the learning efficiency of LMs.
π We implement this pipeline to create a leaner dataset suite, the first to retain the traditional LM training dataset composition and evaluation benchmark intentions while being significantly linguistically simpler. It consists of a 71M dataset for pre-training, a 7M dataset for instruction-tuning, a benchmark that evaluates general linguistic proficiency, and a benchmark for measuring instruction-following ability.
π Despite their infancy and limited initial performance, our instruction-following models mark one of the initial steps toward developing purely text-based self-evolving agents, an initiative we name TinyHelen.
π The leaner datasets offer a testing ground for low-resource scenarios, enabling us to preliminarily evaluate how model architectures and curriculum learning strategies at the pre-training stage affect language modeling performance. Experiments show that transformer decoder llama performs better on leaner-glue than state space model decoder mamba, controlling the model size (14M parameters) and pre-training data (100M tokens scale). Moreover, we find iteratively introducing training instances based on the LM's perplexity can reduce both pre-training steps and data requirements.
I am at #EMNLP2024!
I will present our work "Why Does New Knowledge Create Messy Ripple Effects in LLMs? " on Web 10:30am.
Thanks to all the collaborators @hengjinlp@zhangzxUIUC@Glaciohound@ManlingLi_
Looking forward to have a chat!
Paper Link: https://t.co/AZz83MIp4J
πΎ Introducing AgentOccam: Automating Web Tasks with LLMs! π AgentOccam showcases the impressive power of Large Language Models (LLMs) on web tasks, without any in-context examples, new agent roles, online feedback, or search strategies. πππ
π§ Link: https://t.co/je87xFElc2
π§ By refining the observation and action spaces, AgentOccam achieves a groundbreaking zero-shot performance, outperforming previous methods on the WebArena benchmark. This simple yet effective approach underlines the importance of aligning these spaces closely with LLM capabilities for enhanced efficiency. π
β¨ Highlights:
- AgentOccam leads with a 29.4% improvement over state-of-the-art methods SteP, and a 161% boost in success rate compared to the vanilla agent. π€
- Achievements made possible without complicating the process with additional examples or strategies. π«
- All our replication work, prompts, and evaluator error rectifications are transparently shared in the appendix. π
π Special thanks to my super brilliant and considerate mentor Yao and Rasool, our supportive manager Huzefa, and the invaluable suggestions and contributions from Sapana, Pratik, and George. Your guidance and support have been pivotal in this journey!
#AgentOccam #LLM #WebAutomation #AI
πExcited to share our new work at NAACL 2024! https://t.co/uCTIRvcijI
We study a critical generalization issue of retrieval-augmented generation (RAG) systems - how to maintain the model performance as global knowledge shifts and the background corpus evolves. πβ¨
πOur key findings reveal that by mitigating the context over-memorization effect in reader models, we can notably make RAG systems generalize better across various corpuses and domains - all while preserving their initial performance in established domains! #NAACL2024 #Machine_Learning #NLP
This is a brand new framework for LLM knowledge editing (KE) that solves the critical problems of ambiguity in previous KE methods. A very exciting work!
Seeking practical edits for language models that clarify uncertainties and suit a broad range of users? π€ Our latest theory and solutions for knowledge editing are all you need! π
Dive into our latest research here:
https://t.co/75CIP4oPnZ