Toby Farmer

Co/founder - Cowboy Applied Intelligence Labs.

[Aleksa Gordić (水平问题) @gordic_aleksa]

Ok - here is a full technical breakdown of what we know about Gemini:

* There are 3 model sizes: Ultra, Pro, Nano. The only disclosed size is for Nano and it's 1.8 & 3.25B. The info is not particularly useful because we could have bound the size given it runs on Pixel.

* Ultra follows Chinchilla scaling laws - as the idea is to get the best possible perf for a given compute budget. Inference was not a concern, PR was, you want bold numbers. Smaller models are all heavily in the data saturation regime.

* Gemini is natively multimodal - i.e. trained from scratch on different modalities.

Compare that with Flamingo whose recipe was:
1) train an LLM - Chinchilla
2) train a vision encoder using contrastive pre-training
3) freeze the backbones and train the system e2e

modality support:
input: text, audio, image, video
output: text, image (a big advantage compared to GPT-4 V)

Multimodal demos:
* https://t.co/eyDyyfqO1O
* https://t.co/VxvjRUeauN

* Needless to say the model is massively multilingual as well.

---COMPUTE---

I'll just leave you with this excerpt from the paper:

"but at Gemini Ultra scale, we combine SuperPods in multiple datacenters using Google’s intra-cluster and inter-cluster network.

Google’s network latencies and bandwidths are sufficient to support the commonly used synchronous training paradigm, exploiting model parallelism within superpods and data-parallelism across superpods."

parallelism across GPUs - sweet - how about parallelism across data centers :)

(see below for TPU update)

---EVALS---

* Gemini Ultra’s performance exceeds current SOTA results on 30 of the 32 academic benchmarks BUT IMPORTANT NOTE: not clear how this translates into actual perf given data contamination & in general eval issues with LLMs.

* They report 90% on MMLU which is better than human experts and better than GPT-4 but again it seems the eval methodology was not the same (number of shots + CoT look to be different).

* They report better results than GPT-4V on the new MMMU (multimodal) benchmark.

In general I wouldn't give eval numbers too much attention bc it's not clear whether the comparison is fair between GPT-4 & Gemini - i just want to play with the model. :)

(It's also better than Whisper which is impressive, but:
1) We won't have the weights
2) We have no access to the model
3) We have no idea how much bigger these models are than Whisper in case we ever got the weigths (ok distillation could fix this))

---MISC UPDATES---
* They also share AlphaCode 2 that is estimated to perform better than 85% of codeforces competitive programming competition participants (compared to 50% from original AlphaCode). It leveraged Gemini Pro to get to these results.

* They introduce TPU v5p with 2x more FLOPS and 3x more HBM. A single pod composes of 8960 chips!

* Gemini is already powering many google products (a fine-tuned version of Pro is already in Bard, Nano is running on Pixel)

* On december 13th Pro will be accessible through an API!

Each one of these could be an update in its own right - I simply dislike Google's shipping strategy. It's a firehose method, and if we're talking about safe deployment it's much better to gradually give people access to these systems - a la OpenAI. Just my 2 cents.

And finally: I'm really happy for my ex-colleagues for finally being allowed to show the world what they've been working on. So many familiar names in that long long long contributors section. :)

Tech report: https://t.co/9Vda5lFs30

website: https://t.co/qsQWv7eVxF

[Toby Farmer @TobyFarmer]

Love the opportunity to make the world a better place.

[FICCI-ILIA @ficci_ilia]

“Many of the world’s most needy people are isolated by not only economics, government, geography, connectivity, but the lack of understanding of language can be one of the most devastating realities & we need to work on that” Mr. Toby Farmer, @Facebook AI #Bhashantara2020

[Toby Farmer @TobyFarmer]

Amazing effort by our team on this! Every person - every language matters!

[Mike Schroepfer @schrep]

1/n M2M-100 can translate between any one of 100 lanagues to any other of 100 languages without going to English in between: https://t.co/KehdYeHYZL via @engadget