Gabriel Fortià

[Sam Altman @sama]

GPT-5.6 sol launches thursday!

happy building

[Addy Osmani @addyosmani]

https://t.co/0DyFIXJueI

[DHH @dhh]

@ajanerom Spain blocks all of Cloudflare during those matches in an insane, medieval attempt to counter piracy. Totally batshit. (Omarchy is hosted with Cloudflare).

[Spatial Sports @spatialsports]

@sundarpichai it’s so over

[Guillermo Rauch @rauchg]

@javierguzmandev @cramforce Sadly we had to exit Spain, insanely difficult to employ and grow our company there. We tried!

[Gabriel Fortià @gabrixdev]

@LinasLekavicius Cool project! Congrats

[@levelsio @levelsio]

People who say this never built businesses themselves

They're right that the technical part might not be the hardest in AI because models are trained already for you to use

But the difficulty lies in distribution (getting users), user experience/interface (do ppl like to use it?), and then churn (will they keep paying?)

None of those are easy at all!

[Gabriel Fortià @gabrixdev]

@omarsar0 Aren’t AnyTool and Self Reflective Agents the same paper? I found “AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls”

[Gabriel Fortià @gabrixdev]

@milan_milanovic Interesting

[Gabriel Fortià @gabrixdev]

Interesting point. I think that architecture is also important (not just data quantity).

[Yann LeCun @ylecun]

I've made that point before:
- LLM: 1E13 tokens x 0.75 word/token x 2 bytes/token = 1E13 bytes.
- 4 year old child: 16k wake hours x 3600 s/hour x 1E6 optical nerve fibers x 2 eyes x 10 bytes/s = 1E15 bytes.

In 4 years, a child has seen 50 times more data than the biggest LLMs.

1E13 tokens is pretty much all the quality text publicly available on the Internet. It would take 170k years for a human to read (8 h/day, 250 word/minute).

Text is simply too low bandwidth and too scarce a modality to learn how the world works.

Video is more redundant, but redundancy is precisely what you need for Self-Supervised Learning to work well.

Incidentally, 16k hours of video is about 30 minutes of YouTube uploads.

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

Generative large language models (LLMs) are based upon the decoder-only transformer architecture. Currently, these types of generative LLMs are incredibly popular. However, I use encoder-only architectures for 90% of use cases as a practitioner. Here’s why…

History of encoder-only models. The encoder-only transformer architecture was popularized by the proposal of BERT in 2018. At the time of its proposal, BERT set a new state-of-the-art performance on every natural language task that was considered in its publication. For this reason, BERT revolutionized research in natural language processing, replacing many domain-specific techniques with a single model that can solve nearly all tasks!

Encoder-only architecture. Although the original transformer architecture contains both an encoder and a decoder, BERT leverages an encoder-only architecture. The encoder-only architecture just contains several repeated layers of bidirectional self-attention and a feed-forward transformation, both followed by a residual connection and layer normalization. The original encoder-only BERT models that were proposed have the following sizes:

- BERT Base: 12 layers, 768-dimensional hidden representations, 12 attention heads in each self-attention module, and 110M parameters.
- BERT Large: 24 layers, 1024-dimensional hidden representations, 16 attention heads in each self-attention module, and 340M parameters.

Notably, BERT Base is the same size as the original GPT model. In other words, these models are significantly smaller (and therefore easier to manage/deploy!) compared to the generative LLMs that are popular today.

BERT pretraining. Similar to generative LLMs, BERT has an extensive pretraining process. Instead of next token prediction, however, we pretrain BERT using a Cloze objective, which randomly masks out words/tokens from the input and tries to predict them. Because BERT uses bidirectional self-attention (instead of masked self-attention, which is used by decoder-only models), the model can look at the entire sequence both before and after the masked token to make a prediction.

Using BERT in practice. To use BERT to solve a practical task, we simply finetune the model over task-specific data. In particular, BERT is very good at solving sentence and token-level classification tasks. Additionally, extensions of BERT (e.g., sBERT) can be used for semantic search, making BERT applicable to retrieval tasks as well. In general, finetuning BERT is easy/efficient and yields high performance even with small amounts of training data.

What can’t we do? Encoder-only (BERT) models are small, use bidirectional self-attention, and can be easily fine-tuned to impressive performance. As such, finetuning BERT to solve classification tasks is oftentimes preferable to performing few-shot prompting via an LLM, assuming we have the ability to train models and a little bit of training data. However, encoder-only models cannot generate text, so we can only use them for solving discriminative tasks.

[Santiago @svpino]

I had an amazing machine learning professor.

The first thing I learned from him was how to interpret learning curves. (Probably one of the best skills I built and refined over the years.)

Let me show you 4 pictures and you'll see how this process flows:

1/5

[elvis @omarsar0]

RAG for LLMs

Great to see an overview of all the retrieval augmented generation (RAG) research that has been happening.

Should be a great read for the end of the year.

https://t.co/GxqlqRFokF

[Gabriel Fortià @gabrixdev]

@gneubig Great work! I wonder if GPT-4 will also surpass Gemini Ultra. It would be interesting to see how Gemini performs with a generalistic tool usage dataset like ToolBench or API Bank.