Ujjwal Karn

Software Engineer

[AI at Meta @AIatMeta]

Introducing Muse Glimmer, an open-weight 30B-parameter model optimized for local, always-on agent workflows.

Muse Glimmer delivers strong performance on key agentic use cases and benchmarks compared with leading models in its size category, and is designed to run entirely on consumer hardware like a Mac or PCs with performant GPUs.

In keeping with our long tradition of sharing fundamental AI research, we’re releasing model weights under a permissive Apache 2.0 license.

🧵👇

[Xianjun Yang @xianjun_agi]

📢My New Paper: Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencoder

TLDR: We proposed to use features from SAEs as a measure for data diversity&complexity and proved it's effectiveness on data selection for LLM tuning.

https://t.co/4yD89OYHf4

[AI at Meta @AIatMeta]

As we continue to explore new post-training techniques, today we're releasing Llama 3.3 — a new open source model that delivers leading performance and quality across text-based use cases such as synthetic data generation at a fraction of the inference cost.

[Ujjwal Karn @ujwlkarn]

We recently released the Llama Guard 3 Vision research paper. In the paper, we delve into the details of how we trained the model and discuss its performance and adversarial robustness. We hope you find the insights useful!
https://t.co/ZU09c3jwM3

[AI at Meta @AIatMeta]

Following the release of our latest system level safeguards, today we're sharing new research papers outlining work on Llama Guard 3 1B & Llama Guard 3 Vision that support input/output safety on the edge and in multimodal prompts.

[AI at Meta @AIatMeta]

As part of our support of open science, we've published the full Llama 3 research paper covering a range of topics.
• Model training
• Model architecture
• Results of our ongoing work on integrating image/video/speech capabilities
• Much more

Paper ➡️ https://t.co/qlTVsBjjGz

[Ahmad Al-Dahle @Ahmad_Al_Dahle]

It’s here! Meet Llama 3, our latest generation of models that is setting a new standard for state-of-the art performance and efficiency for openly available LLMs.

Key highlights

• 8B and 70B parameter openly available pre-trained and fine-tuned models.
• Trained on more than 15T tokens, 7x+ larger than Llama 2's dataset!
• Improved tokenizer with vocabulary of 128K tokens for better performance.
• State-of-the-art performance across industry benchmarks.
• New capabilities, including enhanced reasoning and coding.
• 3x more efficient training than Llama 2.
• New trust and safety tools with Llama Guard 2, Code Shield, and CyberSec Eval 2.
• Integrated into Meta AI, and available in more countries across our apps.
• And, just the beginning with more models and new capabilities coming soon!

Visit the Llama 3 website to read more and download the models. https://t.co/1JTQEtBfcf

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

Generative large language models (LLMs) are based upon the decoder-only transformer architecture. Currently, these types of generative LLMs are incredibly popular. However, I use encoder-only architectures for 90% of use cases as a practitioner. Here’s why…

History of encoder-only models. The encoder-only transformer architecture was popularized by the proposal of BERT in 2018. At the time of its proposal, BERT set a new state-of-the-art performance on every natural language task that was considered in its publication. For this reason, BERT revolutionized research in natural language processing, replacing many domain-specific techniques with a single model that can solve nearly all tasks!

Encoder-only architecture. Although the original transformer architecture contains both an encoder and a decoder, BERT leverages an encoder-only architecture. The encoder-only architecture just contains several repeated layers of bidirectional self-attention and a feed-forward transformation, both followed by a residual connection and layer normalization. The original encoder-only BERT models that were proposed have the following sizes:

- BERT Base: 12 layers, 768-dimensional hidden representations, 12 attention heads in each self-attention module, and 110M parameters.
- BERT Large: 24 layers, 1024-dimensional hidden representations, 16 attention heads in each self-attention module, and 340M parameters.

Notably, BERT Base is the same size as the original GPT model. In other words, these models are significantly smaller (and therefore easier to manage/deploy!) compared to the generative LLMs that are popular today.

BERT pretraining. Similar to generative LLMs, BERT has an extensive pretraining process. Instead of next token prediction, however, we pretrain BERT using a Cloze objective, which randomly masks out words/tokens from the input and tries to predict them. Because BERT uses bidirectional self-attention (instead of masked self-attention, which is used by decoder-only models), the model can look at the entire sequence both before and after the masked token to make a prediction.

Using BERT in practice. To use BERT to solve a practical task, we simply finetune the model over task-specific data. In particular, BERT is very good at solving sentence and token-level classification tasks. Additionally, extensions of BERT (e.g., sBERT) can be used for semantic search, making BERT applicable to retrieval tasks as well. In general, finetuning BERT is easy/efficient and yields high performance even with small amounts of training data.

What can’t we do? Encoder-only (BERT) models are small, use bidirectional self-attention, and can be easily fine-tuned to impressive performance. As such, finetuning BERT to solve classification tasks is oftentimes preferable to performing few-shot prompting via an LLM, assuming we have the ability to train models and a little bit of training data. However, encoder-only models cannot generate text, so we can only use them for solving discriminative tasks.

[Mark Tenenholtz @marktenenholtz]

@tunguz Using attention, which goes something like this:

1. Look at a word’s representation
2. Look at the context around that word (i.e. the sentence it’s in)
3. Grab the most relevant part of the sentence and dot product it with the word’s vector

[MIT Intro to Deep Learning @MITDeepLearning]

Online learning is in full swing! 🚀🚀 Tune into @MITDeepLearning, #FREE for everyone! 💻🏡

All 2020 lectures on #NeuralNets, #RNNs, #CNNs, #GANs, #RL, and more #DeepLearning by @xanamini + @apsoleimany now online!

⭐️⭐️ Watch all lectures + labs here 👉 https://t.co/CKBjCfbHJu

[BhuvanaKA @KA_Bhuvana]

This blog post on Convolutional Neural Networks is amazing.
https://t.co/9XcceOsqB9
Thanks @ujwlkarn for this awesome post.
#deeplearning #AI #ComputerVision

[Soy0uz 🛰️ @soy0uz]

This is probably the clearest explanation of #Convolutional #Neural #Networks I've ever read and a great starting point for everyone interested in #AI and #ComputerVision. Also, #bullshitFree, which is awesome !
Thanks @ujwlkarn ! :)
https://t.co/1qdudUsXXJ

[Guido van Rossum @gvanrossum]

Cool sites about the Python 3 transition: https://t.co/R0HIGfO0WN, https://t.co/00WDCOtral, https://t.co/sDmPBs2sty, https://t.co/giiS9CNn8V

[Rachel Thomas @math_rachel]

my new post: What do machine learning practitioners actually do? https://t.co/fNOKVJGISr

[Andrej Karpathy @karpathy]

most common neural net mistakes: 1) you didn't try to overfit a single batch first. 2) you forgot to toggle train/eval mode for the net. 3) you forgot to .zero_grad() (in pytorch) before .backward(). 4) you passed softmaxed outputs to a loss that expects raw logits. ; others? :)

[hardmaru @hardmaru]

PythonRobotics: Excellent collection of robotics algorithms (mapping, SLAM, path planning and tracking, control) by @Atsushi_twi 🤖 https://t.co/Ou5O4lpTDM

[Sebastian Ruder @seb_ruder]

New NLP Newsletter w/ even more cool stuff this time! Transfer learning, Chris Olah, Software 2.0, NMT with attention notebook, gradient boosting in-depth, Defense Against the Dark Arts, interpretability and bias, RL, scene understanding https://t.co/qo0y677c0F

[Nick Johnson - 二クジョンソン @HarveyNick]

New blog post: Kaggle’s Yelp Restaurant Photo Classification Competition, https://t.co/px5xDfQire Style: Part 1 https://t.co/Wz6sth1Dpd

[hardmaru @hardmaru]

Our work on "World Models" is out！🌐

Can neural network agents dream, and learn inside of their own dream worlds?

Read more to find out:

Full Interactive Article: https://t.co/diRMWBqpys
PDF: https://t.co/YZ9Wehzyfv

[hardmaru @hardmaru]

Papers with Code: A searchable site that links machine learning papers on ArXiv with code on GitHub. They also tag any framework libraries used, along with other info like GitHub stars. I think such a feature would be a nice addition to ArXiv-Sanity. https://t.co/Q5kUhsqIED

[PyTorch @PyTorch]

FairSeq Toolkit - Major Update
- Distributed Training
- Transformer models (big Transformer on WMT Eng-German in < 5 hours on DGX-1)
- Fast Inference: translations @ 92 sent/sec for big Transformer
- Story Generation
Read more at Michael Auli's post: https://t.co/eptKDuh0WI