Mohammed Fahsi

Front-end dev, traveler and creative code enthousiast.

[jhey ʕ•ᴥ•ʔ @jh3yy]

Make your tooltips and menus stay in view with 3 lines of CSS 🤙

.menu {
top: anchor(top);
left: anchor(right);
position-try-fallbacks: flip-block, flip-inline;
}

Property updated 📝
Responds to scroll, layout change, etc. Watch 👇
Anchor Positioning is coming 🚀

[Mohammed Fahsi @followmohammed]

Lovin the visuals and slick UI! 😍

[Colin @cvaneenige]

Here's https://t.co/G1W7cXtrXU – Delivering satisfying visuals through patterns. Take a look and let me know what you think! ✨ #webgl #threejs

[Yam Peleg @Yampeleg]

The Missing Pieces of GPT-4
Some overlooked tricks & The future of open source LLMs.

> TL;DR: Reinforcement Learning from Inhuman (!!) Feedback.

The following text details the current state of open-source LLMs highlighting some details that seperate large scale commercial LLMs apart from smaller scale LLMs.

I gave an interesting talk yesterday with the same name ("The missing parts of GPT-4"): while we wait for the video & slides to be up, I wanted to share some of the most interesting details with all fo you guys.

*** Before I Start ***
I HAVE NO "SPECIAL"/"SECRET" KNOWLEDGE ABOUT GPT-4 WHATSOEVER.
I want this to be clear: All assumptions in this text can be wrong.
All info in this text comes from public sources or my own experimental results: I would be happy to be called out about any inaccurate info.
I just want to know the truth.
***

----------
Part 1: Where all of this is going?
(My Opinion. Feel free to skip for the technical details)
----------

During OpenAI's visit to israel last week, one (very good) question stood out: [1]

"Could the open-source LLMs potentially match GPT-4's abilities without additional technical advances?
Or
Is there a secret souce in GPT-4 unknown to the world that set it apart from the other models?" - @ishaygreen

This is an amazing question!!

If Yes: It is very hard to admit publicly that GPT-4 use such a powerful secret that we should not even try to compete with it.
If Not: They have a BIG problem.

[BTW, The rest of the question: "Or am I wasting my time installing Stable-Wizard-Vicuna-30B?" 🤣🤣]

***

Sutskever's answer: "..There will always be a gap between the open source models and the private models and this gap may even be increasing with time.."

***

Right.
Here's What's Gonna Happen:

This will end up like Stable diffusion vs DALL-E 2. [2]
-
You would always be able to scale LLMs to infinity if you really want to do so.
But smaller LLMs are about to become so powerful that it will make no sense.
-
Reason:
They keep improving in an incredible rate every single day.
Yes, they are not perfect yet.
Yes, some measurements are world class p-hacking.
But look at the trend.
[And the speed of the trend]
---
I have nothing against OpenAI: I just want access to capable LLMs because these are useful tools.
So if OpenAI would want to keep chasing huge LLMs beyond the point of "capable open-source models".. Go for it.. I guess.. 🤷
All I want is for the thousands of people working on this (myself included) to have their own capable models that are free to use, train, do RLHF and so on.
Because these are useful tools.
That's it.
.

***
[1] - https://t.co/H5awjUFkz2 [26:30]
[2] - See an interesting discussion about this on @abacaj's tweet.

----------
Part 2: LLaMA's Impact
----------

You already know about LLaMA.
Meta's powerful LLM, released open-source on february.
Ever since then, the model is at the center of a global effort to replicate the capabilities of the comercial LLMs openly.
Thereby providing millions of people all over the world an access to this technology.
There are tens of thousands [1] of people from start-ups, government organizations, academic institutions and even private individuals contributing to this effort.

Thank you to the good people of Meta!! ❤️
This was an incredible gift to the world: Giving us all a sit at the table for the first time.

[1] - Number of users on reddit, discord, twitter..

***

Where are we now?

There are 3 steps for training "ChatGPT"like models:

- Pretraining.
- Fine-Tuning.
- Reinforcement Learning.

* - ! DO IT YOURSELF ! - *

The "common knowledge" nowadays is that all of this is complex.. intimidating.. expensive..
.
It is not.
.
Good News! There is a fire sale right now!
Starting from a base-model, you can train LoRA adapters for all 3 stages (Oftentimes even for free! (colab))

- Try it yourself! 20B params. Free. Here: https://t.co/pLfQTs2ODa
- Preatraining: Check Huggingface's run_clm: https://t.co/5kCaExbdmE
- Fine-Tuning: Check Vicuna's code: https://t.co/X8yai0ltaZ
- Reinforcement Learning: Check TRLx: https://t.co/B6XwGXToTt

TRLx by @carperai: In my opinion this is the best open-source RL package nowadays.
I use it myself and it is VERY effective (and also fast).

* - ! DO IT YOURSELF ! - *

---------
Part 3: Pretraining
---------

This is where 99% of the compute (and cost) go. When you hear things like "4 million dollars to train..", it usually refers to this step.
Unfortunatly, this is usually out-of-reach for most people for obvious reasons.

What do we know about this step?

- "Capabilities on exams appear to stem primarily from the pre-training" - GPT-4's paper tells us that the pretraining step is where most capabilities come from.
- "Deduplicating Training Data Makes LLMs Better" - Literally a name of a paper by Google. Listen to it, it is 200% true.
- "Repeating only a small fraction of the data.. ..can damage performance significantly" - Scaling Laws and Interpretability of Learning From Repeated Data, telling us the same thing.

***
FalconLM
By UAE's Technology Innovation Institute.

The technology innovation institute stunned the world two weeks ago when they unveiled one of the the most powerful base models ever: FalconLM.
The model held the first place on Huggingface's leaderboard by a considerable margin for some days.
And that's not all: It was released under a completely open license: commercial use is also allowed.
-
🔥 Another bomb was dropped days ago 🔥
The UAE have started working on a monstrous base model: FalconLM with 180 billion parameters, Using the massive 5-trillion tokens dataset they collected and cleaned.
We don't have confirmed details about any of this yet but.. wow..
-
Important!
"Who will win? GPT-4 or Falcon 180B?".
There is no reason to compare them at all - we are exposed to GPT-4 only after the next steps of the training.
This model (if true at all) is at an earlier stage: Base model.
.
BUT!
If you look at GPT-4's paper: GPT-4's base model's performance metrics are known.
And they are not "that far off" from GPT-3's base model. Base on these numbers: The main difference between GPT-3 & GPT-4 is in the RL step.
Noted.
.
Apart of LLaMA we are starting to see another trend:
Organisations are Funding & Releasing powerful base models.
This trend is absolutely a threat to GPT-4.

---------
Part 4: Fine-Tuning
---------

TL;DR: No magic. You train the same as pretraining but on "solved tasks": Task followed by the Solution. [Question > Answer, for example]

One of the most important discoveries by the open-source community was that this also works with synthetic data.
Meaning: Which reduces the cost of collecting high quality datasets significantly.

From this point forward the speed of improvements had been incredible:

2 Weeks after LLaMA: Alpaca - TL;DR: "We can train poweful models on synthetic data"
3 Weeks after LLaMA: Vicuna - TL;DR: "We can measure our LLMs by using other LLMs"
5 Weeks after LLaMA: Wizard - TL;DR: "We can ask LLMs to make data samples harder to get MUCH better LLMs"
9 Weeks after LLaMA: Guancano - TL;DR: "We can all now train huge models."
14 Weeks after LLaMA: Orca - TL;DR: "GPT-4 prefers an open model over GPT-3.5" [This means nothing on it's own but this happening for the first time is a milestone]

-
Days Ago: An open model defeated GPT-4 on a logic puzzle written by.. GPT-4.
Yes. on ONE specific example. CALM DOWN.
This is the first anecdotal example I saw, I thought I'd share.
*
Because this is not going to be the last.
*

---------
The best model: Nous-Hermes.
---------

Nous-Hermes | Among the top LLMs to date.
A particularly powerful model was released last week.

- Model: https://t.co/bmj0Xp0oIm

- Authors: @teknium, @karan4d
- Data Contributors: @teknium, @karan4d, @NousResearch, @huemin_art, @winglian, @erhartford, @main_horse
- Funding: @RedmondIT

I Highly recommend to try it, I use it myself instead of ChatGPT for anything other than code or math (we will get to why code is "different" in a moment).
It is so good, that some people on reddit are already starting to say that on certain prompts it is "at the level of GPT-4".
Even @elonmusk commented about it.

And just to let you know: As far as I know, the authors are already gathering feedback from the users about specific topics to improve for the next iteration.

The secret is the data:
The dataset used for training this model is completely synthetic and specially generated and filtered to yield the most powerful model possible using several other models.

> What did that paper say again? Fine-tuning is never going to work?

This model beats nearly the entire GPT-4-ALL leaderboard on the first place.

- Try It: https://t.co/qso98Qcuog
- Model: https://t.co/bmj0Xp0oIm
- GGML Version: https://t.co/1hGtChbo0q
- GPTQ Version: https://t.co/PG03UqH4PZ

> Also: God bless the @TheBlokeAI

***
This was my own opinion.
There is a matter of "taste" answering this question.
***
---------
> > > ! Continuing in a Comment ! < < <
---------

[STYLIZED STATION @StylizedStation]

Lets talk about a single texture that's been a huge game changer in the world of gaming.

And the crazy part? You probably haven't even noticed it. This texture is literally EVERYWHERE. And there's something pretty special about it that's made it so popular.

Another big🧵

[Barsee 🐶 @heyBarsee]

AI is moving too fast.

This is the most significant month for AI that might change everything🧵👇

1. Wonderdynamics
It's like having a whole ass studio in your pocket.

This tool animates, adds characters, lighting into a live scene.

[Nick St. Pierre @nickfloats]

Here's a compilation of my Midjourney threads so far, including tutorials on how I use my Additive Prompting framework to generate characters/models, design sets, &amp; more.

Taking today off Twitter to work on animations &amp; visuals for my YouTube series (link in bio)

🧵Thread

[☄︎ @0xca0a]

i've been solving other peoples problems again 🫠 in a thread of some boxes that have accumulated.

how to make a chess board look good https://t.co/tcldK362sM
odd question but ok.

[harley turan @hturan]

here’s GPT-4 writing midi-controllable GLSL shaders

[kynd.info @kyndinfo]

Interpolation and Animation
https://t.co/KKQvRZZeG1

[Prateek Jakhar @prateektwts]

Awesome JavaScript one-liners ⚡

[Mohammed Fahsi @followmohammed]

@HamillHimself Is this a rock fact? 🤔

[Mohammed Fahsi @followmohammed]

@midaswireless @helium This would be hella sweet!!! 😍 @SkenkyMenky @modsign @MrDikkens

[midaswireless.hnt 🎈 @midaswireless]

Happy Chinese New Year, 恭喜发财, earn more #HNT 💎👐 5️⃣winners, 1️⃣Midas 926 gateway each
1.follow us @midaswireless and @helium
2.retweet and Like
3.tag 3 friends
🕰Ends on 8th Feb.
Midas will touch you #MIDASTOUCH

[Mohammed Fahsi @followmohammed]

@TradesTrey And here i thought we were just going for $25 shhmeckkles! But we just keep going!

[Mohammed Fahsi @followmohammed]

@cvaneenige @tailwindcss @vercel Looking hella sweet bro! 🔥🔥🔥

[Mohammed Fahsi @followmohammed]

This is so awesome! 😍

[☄︎ @0xca0a]

threejs has a market, today we release https://t.co/OYkohaMx5q 🎉

🔎 free minified/ready-to-use CC0 models/textures/hdris
🚥 starters for threejs and react-three-fiber
👤 accounts
📬 request assets
🥇 favorites
🔌 api endpoint
☁️ CDN with caching
📦 npm install assets soon

[Mohammed Fahsi @followmohammed]

Wow! Can't wait to check itnout!!! 😍😍😍

[Bruno Simon @bruno_simon]

Three.js Journey is live! 🎉

Learn how to create stunning 3D WebGL websites with the #threeJs library through more than 30 hours of video for $95 only (VAT incl.)

https://t.co/MpOZvj5LfL

#threejsJourney #webGL

[Mohammed Fahsi @followmohammed]

@ZiggoZakelijk Maar het meest absurde is dat alleen de actie geannuleerd is, maar ik zit wel vast aan een abonnement van een jaar. Te bizar voor woorden.

[Mohammed Fahsi @followmohammed]

@ZiggoZakelijk Wat een ongelooflijk slechte service leveren jullie. Heb een abonnement afgesloten, krijg nu (4 maanden later) te horen dat ik niet in aanmerking kom voor de actie van dat abonnement, maar annuleren gaat niet want ik zit er aan vast voor een jaar...what!?

[Mohammed Fahsi @followmohammed]

@ZiggoZakelijk Blijkbaar mag je als bestaande klant geen gebruikt maken van acties, want die zijn alleen voor nieuwe klanten. Maar zou ik mijn verlopen abonnement eerst annuleren en erna pas een nieuw abonnement nemen, dan kan het wel. Hoe krom is dit?