Kushal Arora

Research Scientist at Toyota Research Institute.

Ph.D. student at @rllabmcgill, @MILAMontreal.
Prev: FAIR, MSR, BorealisAI, Amazon, UF.

[Jean Mercat @MercatJean]

Our white paper just came out on Arxiv https://t.co/HcSCepvXfM. We open-sourced it all https://t.co/lfySRERfeQ. Our project website also has links to the white paper, the weights, and more videos https://t.co/DRnViyXRWz

[Jean Mercat @MercatJean]

Releasing VLA Foundry: an open-source framework that unifies LLM, VLM, and VLA training in a single codebase. End-to-end control from language pretraining to action-expert fine-tuning — no more stitching together incompatible repos.

[Katherine Liu @robo_kat]

ReFiNe expands on a neat idea we first presented at CoRL with Recursive Octree Auto-Decoders: that recursion can enable very high compression rates of 3D data. In ReFiNe, we use this property to represent continuous fields and can decode multiple NeRFs/SDFs with a single network.

[Sergey Zakharov @ZakharovSergeyN]

Excited to introduce our paper, ReFiNe, at #SIGGRAPH2024 this Thursday! Learn how we encode multiple assets as continuous neural fields with high precision & low memory usage by exploiting object self-similarity.

@RaresAmbrus @robo_kat @adnothing
Webpage: https://t.co/JoSqVTIDBr

[Kushal Arora @karora4u]

@Swarooprm7 Congrats Swaroop!

[Kushal Arora @karora4u]

@ke_huang275 @achalddave Though it is difficult to say why benchmarks got better with IT, my speculation is this is due to the DCLM-IT data, as it contains datasets such as Nectar, no_robots, StarCoder2-Self-OSS-Instruct, which have math, code, QA data that might help improve the benchmarks performance.

[Kushal Arora @karora4u]

@ke_huang275 @achalddave @ke_huang275 We trained for 10 epochs as we saw AlpacaEval score improving beyond first few epochs. So, we decided to keep fine-tuning.

Here is how the AlpacaEval looked for each epoch:

[Sachin Grover @sachingrover]

I am looking for positions in LLM based agents, and combining planning and learning techniques/systems.

I have around 2.5 years of industry research including two years at @PARCinc as a research scientist and multiple summer intern positions @amazon @alexa99. 1/6

[Thomas Kollar @tkollar]

Building language models is difficult and requires high quality preprocessing, modeling, evaluation and large scale training.

As significant collaborators in this project at TRI, the resulting 7B model DCLM-7B is a significant achievement. It is a competitor to Mistral 7B and LLaMA-7B, even though trained on less data. And it’s fully open. And that’s just the start of the competition.

Excited to see how others leverage these results to build even more capable language models and improve dataset quality.

[Kushal Arora @karora4u]

For more details, see the paper https://t.co/YMfc46es7n, and the website: https://t.co/OuJPQLfTYk.

[Kushal Arora @karora4u]

One thing I have come to greatly appreciate over the last year is the role of data filtering in building SOTA language models.
DCLM introduces a filtered 240T dataset, a 7B open-source model that is competitive w/ Llama3 with 2-6x fewer tokens & a pipeline to build new datasets.

[Vaishaal Shankar @Vaishaal]

I am really excited to introduce DataComp for Language Models (DCLM), our new testbed for controlled dataset experiments aimed at improving language models. 1/x

[Aran Komatsuzaki @arankomatsuzaki]

DataComp-LM: In search of the next generation of training sets for language models

- Provides a corpus of 240T tokens from Common Crawl
- Trains a LM using their filtered dataset, which performs similarly on NLU tasks w/ 6.6x less compute than Llama 3 8B

proj: https://t.co/soBq1ZnAwT
abs: https://t.co/r8nWIHwq1t

[Luca Soldaini 🎀 @soldni]

Really impressed by the work DCLM folks did!!

[Achal Dave @achalddave]

Check out DataComp for language models! Open data, open code, open training recipe, and close to Llama3-8B performance. This has been a labor of love over the last year, a huge thanks to all the collaborators for helping make this happen!

[Vaishaal Shankar @Vaishaal]

I am really excited to introduce DataComp for Language Models (DCLM), our new testbed for controlled dataset experiments aimed at improving language models. 1/x

[Kushal Arora @karora4u]

Sedrick is an amazing researcher and has done amazing work on pre-training, scaling, evaluation, Japanese LMs, code models, VLMs, and more in the last year. If you are at NAACL, do get a coffee with him!

[Sedrick Keh @sedrickkeh2]

I'm attending #NAACL2024 at Mexico City this week! Excited to chat about pre-training, evaluation, and multimodality!

(also excited for🌮🌯🫔)

[Sedrick Keh @sedrickkeh2]

Recurrent models like RWKV and Mamba have gained attention recently, but these can be costly to train and iterate on.

What if we could simply...
turn Mistral/Llama/Gemma into an RNN? 🎩🪄

Presenting our work, Linearizing Large Language Models!

https://t.co/hbaSUWk8uc

[Sedrick Keh @sedrickkeh2]

2024 has seen tons of cool work on RNNs (cc @RWKV_AI, @BlinkDL_AI, @GoogleDeepMind Griffin, @AI21Labs). We hope our work helps further research into linear models!

Work done at @ToyotaResearch with @MercatJean @vslevic @sedrickkeh2 @karora4u @achalddave @adnothing @tkollar

[Kushal Arora @karora4u]

A really large in-the-wild robotics dataset from TRI colleagues And university partners, a major step in the direction of building Robotics Foundation Model.

[Alexander Khazatsky @SashaKhazatsky]

After two years, it is my pleasure to introduce “DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset”

DROID is the most diverse robotic interaction dataset ever released, including 385 hours of data collected across 564 diverse scenes in real-world households and offices