Luis Rogelio Román Rivera

PhD, Neural Networks, Computer Vision and AGI interests. Working with storage, backups and virtualization.

[The Lunduke Journal @LundukeJournal]

Linux market share is up to 15.65% in North America, 8.76% Worldwide.

The North America numbers are absolutely skyrocketing, month to month: 3.7% -> 6.14% -> 10.61% -> 15.65%.

Questions: How many of these are AI bots? And how high could this go?

[Massimo @Rainmaker1973]

Bring a tiny black hole into your room.

[📹 vaytrove]

[Charly Wargnier ☄️ @DataChaz]

Best use of AI I've seen this year

[Tom Yeh @ProfTomYeh]

Transformer by hand ✍️ ~ 6 steps walkthrough below

Studying the Transformer architecture is like opening up the engine hood of your car. So many unknown parts: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking.

But which of those actually make the car run?

I think at the core, there are two most essential components:
Attention weighting and the feed-forward network.

Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model.

So I drew and calculated those two parts entirely by hand.

Goal: push five features through one transformer block, filling in every cell yourself.

1. Given

Five positions of input features, arriving from the previous block.

2. Attention matrix

Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own.

3. Attention weighting

We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features *across positions*, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on.

4. First layer

Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens *across feature dimensions*, vertically, and each feature grows from 3 numbers to 4.

Note that every position goes through the same weight matrix. That is what "position-wise" means.

5. ReLU

We cross out the negatives. They become zeros.

6. Second layer

Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again.

You have just calculated a transformer block by hand. ✍️

The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes across positions, so a feature can see its neighbours. The FFN mixes across feature dimensions, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run.

💾 Save this post!

#AIbyHand #Transformers #DeepLearning

[DAN KOE @thedankoe]

https://t.co/kNfAIZcqin

[Mirza Beig @TheMirzaBeig]

#GLSL: Black Hole, Distortion,
🌀 Gravitational Lens (Shader):

#define R iResolution.xy

void mainImage(out vec4 c, in vec2 p)
{
vec2 u = p / R, m = iMouse.xy / R,
o = m - u, l, i; o.x *= R.x / R.y;

float d = length(o);

c = mix(vec4(0),

texture(iChannel0, u + o / d * .1 / (d * d)),
step(.3, d));
}

Try it INTERACTIVE:
👉 https://t.co/O8m36yHoEP

#gamedev #indiedev

[Understanding Linux: The Kernel Perspective @unix_byte]

Ken Thompson is best known for his work on Unix. However, shortly before Unix, Ken implemented one of the first practical regular expression engines (Communications of the ACM, 1968). Since then, regular expressions have become integral to many Unix tools, and today nearly every modern programming language supports them.

[tetsuo @tetsuoai]

Linux From Scratch (LFS) is a project that provides you with step-by-step instructions for building your own custom Linux system, entirely from source code.

[Tom Yeh @ProfTomYeh]

Transformer by hand ✍️ Given input X, this exercise goes through the key equations behind the 𝗦𝗲𝗹𝗳 𝗔𝘁𝘁𝗲𝗻𝘁𝗶𝗼𝗻 layer and the 𝗙𝗲𝗲𝗱 𝗙𝗼𝗿𝘄𝗮𝗿𝗱 layer, showing you how X is transformed into Y.

[Luis Rogelio Román Rivera @lrogelio]

@PaniniMangaMx Tomo 1 Solo Leveling

[ℏεsam @Hesamation]

Andrej Karpathy calls large language models the new computing paradigm:

CPU -> LLM
bytes -> tokens
RAM -> context window

this is the large language model OS (LMOS)

[Mohit Mishra @chessMan786]

Exploring The Depths Of Linux Input Output: A Developer's Guide To Storage Management
Understanding the I/O stack is crucial for system programmers, kernel developers, and anyone working on Linux system optimisation or device driver development.

[Tom Yeh @ProfTomYeh]

Autoencoder by hand✍️Excel~ I designed this exercise to show how an Encoder-Decoder network convert input to code and reconstruct input from code. It is annotated with equations, PyTorch, and graphs. 👇Join the 'AI Math' community. Download xlsx.

[ℏεsam @Hesamation]

Andrej Karpathy explains Tensor Cores and TF32 precision

[Victor @victor_explore]

A dedicated course on Transformers and their uses,
Transformers are applied across Deep Learning, including computer vision (CV), reinforcement learning (RL), Generative Adversarial Networks (GANs), speech, and biology.
(link in comment)

[Modular @Modular]

How can an open community simplify AI? What does @clattner_llvm think about Zig, recruiting top talent, and the Stargate Project? The recent expert panel on compute at the Democratize Intelligence Summit tackled these questions and more: https://t.co/p0HcjCfNn3

[Neo Kim @systemdesignone]

If you want to become good at system design (I'm not kidding), learn these case studies:

[Vivek Galatage @vivekgalatage]

A website about memory management - quite literally!

https://t.co/9lRNBhsVeu