Matan Kalman

[Matan Kalman @matankal]

@karpathy @N8Programs Another really awesome resource for neural implementations of simple algorithms is "The Art of Transformer Programming" (https://t.co/EgRKFGautv). It has a couple of really cool examples manually setting weights for addition, search and sort.

[Matan Kalman @matankal]

There are many more cool examples in our project page: https://t.co/epugdqYxai

[Matan Kalman @matankal]

Our generative UI project is finally launching today! Here is a Japanese visual novel it built from my image:

https://t.co/JIJBCjRFFK

It should roll out within the next few days as a feature in Gemini called "dynamic view"

[Matan Kalman @matankal]

@actualrealyorth @waxhn @fly51fly It shouldn't matter as it is all symmetrical. Whichever head you choose, during training the transformer would learn to use for deciding which tokens to mask.

[Matan Kalman @matankal]

@waxhn @fly51fly It also has the advantage of adding no new parameters to the transformer (since a head is re-used).

You don't have to assume that the relationship is the same across heads. During training with selective attention the transformer learns to use head0 to decide what tokens to mask

[Matan Kalman @matankal]

@waxhn @fly51fly We need to have some score matrix to use for masking. We tried several different variants (including averaging across heads, and using a completely separate set of parameters for generating this matrix - see appendix A.2), but eventually selecting an arbitrary head worked best.

[Matan Kalman @matankal]

@danielbigham @karpathy We call this "alpha" in our paper (https://t.co/aKbCqEJCqu). It depends of course on the specific tasks and models, you can see various values in tables 2 and 3. For many cases (e.g. t5xxl vs t5s) we saw values around 60%-75%. This can be much higher for some easier tasks.

[Matan Kalman @matankal]

@dvirsky אנחנו מבקרים כל שנה, וכן יש לנו תקציב שאנחנו שמים בצד כל שנה. עד עכשיו הצלחנו קצת לחסוך כי הנסיעות האלה היו נסיעות עבודה מבחינתי, אבל עדיין היינו צריכים 4 כרטיסי טיסה לשאר המשפחה. את השהות עשינו אצל ההורים.

[Matan Kalman @matankal]

A set of retro posters, create with #Imagen

[Matan Kalman @matankal]

Animal oil paintings, created by #Imagen

[Matan Kalman @matankal]

Using #imagen, Google's new text-to-image tool, to create a set of Father's Day cards.

#HappyFathersDay

[Matan Kalman @matankal]

@DMullinsGames The best game I played in this Ludum Dare. Scores are well deserved!