oryan

Software Engineer & AI nerd

[Amit Sheen @amit_sheen]

📢 Excited to announce that starting today I’m #OpenForWork and providing services in anything front-end related, from hands-on development to consulting, DevRel, and more.

Please reach out for details, DMs are open.

* Barbenheimer is so last week, here's the new blockbuster:

[Yam Peleg @Yampeleg]

Now this is the right way to give criticism!

Now let's close some dataset holes and crush them.

[Yam Peleg @Yampeleg]

Adam8bit
+ model.half()
+ gradient_checkpointing_enable()
+ batch_size = 1
+ flash attention
+ zero_optimization: stage: 3
+ offload_optimizer
+ offload_param
+ eps = 1e-4
+ weight_decay = 0
+ dropout = 0

Going to make some coffee
No way the loss is not 𝚗𝚊𝚗 when I come back.

[oryan @oryanmoshe]

The struggle is real

[oryan @oryanmoshe]

@Yampeleg So essentially if I paid for the rights to watch a movie, I'm not allowed to review it afterwards because it's copyrighted?
It's not like you uploaded the document, you wrote *about* it.
Also, $1,000 is INSANE imho.

[oryan @oryanmoshe]

@curious_vii @Yampeleg @GitHubNext Thanks for sharing! Didn't read that one

[WizardLM @WizardLM_AI]

Introduce the newest WizardLM models trained with only 1k high-quality evolved data!

WizardLM-13B-V1.1 achieves:

(1). 6.74 on MT-Bench
(2). 86.32% on Alpaca Eval (ChatGPT is 86.09%)
(3). 99.3% on WizardLM Eval

Github: https://t.co/4OuXolwS1P

Weights: https://t.co/k5OgURzxTz

[oryan @oryanmoshe]

@Yampeleg Works for me and it feels like magic

[oryan @oryanmoshe]

This is absolutely insane, go try it out!

[Yam Peleg @Yampeleg]

You can get ChatGPT Code Interpreter for right now.

1. Log off -> Log in to ChatGPT.
2. Go to settings -> Beta features
3. Enable Code interpreter.

Breaking!

[oryan @oryanmoshe]

https://t.co/3BKgR4Tgu0

[Uri Eliabayev @urieli17]

בשקט בשקט ובאופן עצמאי @Yampeleg בונה לנו מודל שפה גדול בעברית ראשון מסוגו. הוא משחרר בקהילה טיזרים קטנים ומראה לנו על מה הוא עובד וזה פשוט מדהים.

אני לא זוכר מתי פעם האחרונה שבאמת חיכיתי שמשהו כזה יצא לאוויר. זה באמת יכול לשנות את חוקי המשחק עבור כל התחום בישראל.

[Yam Peleg @Yampeleg]

One. Billion. Tokens. Context Window. 🤯

I didn't want to get sucked into twitter but I couldn't help it.

Here is how they do it:

Dilated Attention
They use an approach similar to dilated convolutions: "convolutions with holes" that are expanding as the depth of the network increases.

Allowing the layer to "see" parts of the input that are farther apart from each other.

Illustration:

The filter is the red part (moving through the input seq)

Standard Convolution:
🟩🟩🟥🟥🟥🟩🟩🟩

Dilated Convolution:
🟩🟥🟩🟥🟩🟥🟩🟩

Note: It is the same filter size but the dilated can "see far".
In order not to lose information: We usually increase the dilation rate as the depth of the layer increases (Wavenet, Google 2016).

So they apply the same idea but to the 2D attention maps.

1. Input Segmentation: Dilated attention takes and splits it into equally sized segments.

2. Sparsification: Each of the segments is then sparsified based on a given interval 'r'.
(Meaning: Every 'r'th row is selected, others are discarded)

3. Outputs: The outputs for the sparsified segments are then gathered and concatenated to form the final output of the attention layer.

4. Mixing with dense attentions: To capture both short-range and long-range, the final network use a mix of dilated & dense attention layers with different segment sizes and dilation rates.

Note about the code:

I didn't go into it too much but the codebase has references to many architecture tricks (with papers).

Looks like a cool reading list, I didn't know about some of the papers there.

[Yam Peleg @Yampeleg]

BatGPT from Wuhan university. (Yes.)

- Paper: https://t.co/dVlKcded72

Jokes aside, pay attention to the following tricks:

- Trick 1: Reversing the sequence during training. [learn to predict the next token & learn to predict the previous token]

- Trick 2: Reinforcement learning from inhuman. Using feedback from models to train and RL model.

- Trick 3: Formalizing the RLHF data collection to simple pairs comparison [see picture]

Results: They do not measure themselves on known benchmarks but rather on Chinese benchmarks.

They are SOTA but this might be hard to estimate what it means.

Anyway, the tricks are important to know about.

- For more on trick 2: See my post "the missing pieces of GPT-4.
- Apologist about the previous tweet about this paper - I was misled.

[oryan @oryanmoshe]

🚀 "The Secret Behind My Latest Blog Post: ChatGPT Did the Heavy Lifting" 🚀

Learn how I collaborated with ChatGPT to create high-quality content and the tips & tricks that made our teamwork a success. Check it out! ➡️ https://t.co/cvJ2O41pZo #ChatGPT

[oryan @oryanmoshe]

🚀 Just published a new blog post: "Mock Data & Stubs: Fake It Till You Make It"! 🚀

Learn how to reduce iteration times by using mock data, stubs, and fakers.

Check it out and let me know your thoughts! 👉 https://t.co/LhyPhCfMfA

#programming #development #productivity

[oryan @oryanmoshe]

Wordle 213 4/6

⬛🟩⬛⬛⬛
⬛🟩⬛⬛⬛
⬛🟩🟩⬛⬛
🟩🟩🟩🟩🟩

מדויקת 18.1.2022 - 6/6

⬜🟨⬜⬜⬜
⬜⬜🟨🟨🟨
⬜🟨⬜🟨🟨
⬜🟩⬜🟩🟨
🟨🟩⬜🟩🟩
🟩🟩🟩🟩🟩

[oryan @oryanmoshe]

Wordle 211 4/6

⬛🟨🟨⬛⬛
🟨🟨⬛⬛🟨
🟩⬛🟨🟨⬛
🟩🟩🟩🟩🟩

מדויקת 16.1.2022 - 5/6

🟨⬜⬜⬜⬜
⬜⬜🟨⬜⬜
⬜🟨⬜⬜🟩
🟩⬜⬜🟨🟩
🟩🟩🟩🟩🟩

[oryan @oryanmoshe]

@joeltelling Wordle 198 2/6

🟩🟩⬛⬛⬛
🟩🟩🟩🟩🟩

[oryan @oryanmoshe]

Wordle 198 2/6

🟩🟩⬛⬛⬛
🟩🟩🟩🟩🟩