MF FOOM

masked attention head

[MF FOOM @MF_FOOM]

mf trained a simple model to translate ada-002 embeddings back to text and found something interesting:

sentence embeddings have arithmetic properties

the classic "king - man + woman = queen" still works just like w/ token embeddings — but can go a few steps beyond that as well

[MF FOOM @MF_FOOM]

he can't keep getting away with it

[Tri Dao @tri_dao]

FlashAttention is widely used to accelerate Transformers, already making attention 4-8x faster, but has yet to take advantage of modern GPUs. We’re releasing FlashAttention-3: 1.5-2x faster on FP16, up to 740 TFLOPS on H100 (75% util), and FP8 gets close to 1.2 PFLOPS!
1/

[MF FOOM @MF_FOOM]

@CFGeek bingo. ty!

[MF FOOM @MF_FOOM]

there was a good thread recently estimating the current upper bound of tokens in the world (including Gmail and other private repositories), but I can't find it

can anyone point me to it?

[MF FOOM @MF_FOOM]

@main_horse @OpenAI

[MF FOOM @MF_FOOM]

@atroyn ah i see i'm behind on the sota, i was under the impression folks were still training embedding models purely w/ unsupervised contrastive learning — but yup looks like BGE, Gecko, etc all mix in supervised Q-A pairs along with their unsupervised contrast pairs

[MF FOOM @MF_FOOM]

Why not just train your embedding model to embed queries directly into answer space? Feels like an inefficient use of flops.

[sankalp @dejavucoder]

i find HyDE (hypothetical document embeddings) to be such an elegant and intuitive idea - like to improve your semantic search, you ask an LLM to generate a hypothetical answer to your query and then you use this query for embedding search. idea is the (embedding of) hypothetical query is going to be closer in latent/embedding space than your actual natural language query. it's funny how you are literally using a hallucinated answer to get better results.

[MF FOOM @MF_FOOM]

@atroyn cheers for responding btw

[MF FOOM @MF_FOOM]

@atroyn by 'they’re already trained this way,' you're saying contrastive pre-training already gives embedding models this property, or folks are already finetuning their embedding models on query-answer pairs in industry?

[MF FOOM @MF_FOOM]

@milkman_moo yup, exactly.

[Charles Foster @CFGeek]

Contrast pairs are overpowered. Once you have them, you can use them to generate control vectors, and to initialize classifiers, and to do RL/DPO, and probably more

[MF FOOM @MF_FOOM]

@andriy_mulyar @TheSeaMouse still a very big improvement and does not rule out instruction annealing playing a role in it

[MF FOOM @MF_FOOM]

@andriy_mulyar @TheSeaMouse those scores are from the instruct tuned models though, the base model scores are markedly lower

https://t.co/8kBOQ8aSkc

[Hassan Hayat 🔥 @TheSeaMouse]

@lateinteraction Baseline mmlu is still impressive. Although notably Reka Edge is also a similarly strong model

[MF FOOM @MF_FOOM]

@xlr8harder my guess is because it's easier to have an animal known only for that one property (camouflage)

octopi are known for a whole bunch of things (intelligence, 8 legs, etc) — so could confuse kids to call someone an octopus expecting them to understand it's about camouflage

[MF FOOM @MF_FOOM]

@EgeErdil2 @NathanpmYoung 600B*

[MF FOOM @MF_FOOM]

This is the most in-depth technical report we’ve gotten about a frontier-class model in a while!

[Reka @RekaAILabs]

Along with Core, we have published a technical report detailing the training, architecture, data, and evaluation for the Reka models.

https://t.co/ROrakRAcPu

[MF FOOM @MF_FOOM]

@PandaAshwinee congrats that’s amazing!

[MF FOOM @MF_FOOM]

Yes, and get SOTA by doing so.

https://t.co/p74bOL2Pnx

[MF FOOM @MF_FOOM]

could you turn a pre-trained decoder into an encoder by just removing the attention mask and finetuning on encoder-like tasks?

[MF FOOM @MF_FOOM]

@haizelabs p.s. i may have another improvement to push the speed frontier even further, will reach out when i verify :)

[MF FOOM @MF_FOOM]

@haizelabs awesome work!

some questions about the 'Reducing Batch Size' section:

a) is batch size not essentially free to scale? each batch can be computed in parallel, even across multiple cards

b) what is the pareto optimal B you reccomend? or is this a Haize Labs secret hehe

[MF FOOM @MF_FOOM]

@haizelabs and meant each sequence in each batch, not each batch :p

[MF FOOM @MF_FOOM]

@haizelabs sorry to be clear i'm wondering about starting B & rate at which you scale it, not some good average fixed B