rlattt @robertlatt - Twitter Profile

Almost all generative language models use a decoder-only transformer architecture, making the decoder-only transformer one of the most influential architectures in modern AI. Let’s take a deeper look at an implementation to understand exactly how it works… Implementation credits: The implementation of a decoder-only transformer we will examine in this post is taken from the NanoGPT repository, created by Andrej Karpathy. This repository implements pretraining of a (large) language model—GPT-2 in particular—in only ~600 lines of code. It is an awesome learning resource. Block definition: Each block of the decoder-only transformer contains causal self-attention and a feed-forward transformation. Given an input, we apply layer normalization and causal self-attention, followed by a residual connection. Then, we apply layer normalization again, followed by a feed-forward layer and another residual connection. The decoder-only architecture has several blocks; e.g., GPT/GPT-2 has 12 total blocks. Embedding layers: The decoder-only model has two different embedding layers that it defines. The first embedding layer stores token embeddings for all tokens within the tokenizer’s vocabulary. The other embedding layer stores all positional embeddings used in creating the model’s input. Forward pass: Assuming we receive a tokenized sequence as input, we first retrieve all token embeddings and add positional embeddings to these token embeddings. After (optionally) applying dropout, we pass these token embeddings through the transformer blocks, then apply a final layer normalization. From here, we can either predict next tokens over all tokens in the sequence (pretraining) or on the last token (inference). Next tokens are predicted using a linear projection that outputs a vector with the same size as the vocabulary. Generating text: If we want to generate a textual sequence, we just perform the forward pass defined above. However, we keep iteratively predicting the next token for the last token within the input sequence, adding it to our input sequence and repeating. This autoregressive process allows us to generate an entire sequence of text by just iteratively extending our input sequence by a single token. Configuration: Notably, all of the settings within the architecture described above depend upon configured hyperparameters. For example, we can define the size of the token embeddings, the total number of blocks, the size of the model’s context window, and more by just tweaking the model’s configuration. To create a much larger language model, we typically just increase the embedding dimension and total number of layers. This approach is followed by most modern LLMs to increase model’s total number of parameters.

cwolferesearch's tweet photo. Almost all generative language models use a decoder-only transformer architecture, making the decoder-only transformer one of the most influential architectures in modern AI. Let’s take a deeper look at an implementation to understand exactly how it works…

Implementation credits: The implementation of a decoder-only transformer we will examine in this post is taken from the NanoGPT repository, created by Andrej Karpathy. This repository implements pretraining of a (large) language model—GPT-2 in particular—in only ~600 lines of code. It is an awesome learning resource.

Block definition: Each block of the decoder-only transformer contains causal self-attention and a feed-forward transformation. Given an input, we apply layer normalization and causal self-attention, followed by a residual connection. Then, we apply layer normalization again, followed by a feed-forward layer and another residual connection. The decoder-only architecture has several blocks; e.g., GPT/GPT-2 has 12 total blocks.

Embedding layers: The decoder-only model has two different embedding layers that it defines. The first embedding layer stores token embeddings for all tokens within the tokenizer’s vocabulary. The other embedding layer stores all positional embeddings used in creating the model’s input.

Forward pass: Assuming we receive a tokenized sequence as input, we first retrieve all token embeddings and add positional embeddings to these token embeddings. After (optionally) applying dropout, we pass these token embeddings through the transformer blocks, then apply a final layer normalization. From here, we can either predict next tokens over all tokens in the sequence (pretraining) or on the last token (inference). Next tokens are predicted using a linear projection that outputs a vector with the same size as the vocabulary.

Generating text: If we want to generate a textual sequence, we just perform the forward pass defined above. However, we keep iteratively predicting the next token for the last token within the input sequence, adding it to our input sequence and repeating. This autoregressive process allows us to generate an entire sequence of text by just iteratively extending our input sequence by a single token.

Configuration: Notably, all of the settings within the architecture described above depend upon configured hyperparameters. For example, we can define the size of the token embeddings, the total number of blocks, the size of the model’s context window, and more by just tweaking the model’s configuration. To create a much larger language model, we typically just increase the embedding dimension and total number of layers. This approach is followed by most modern LLMs to increase model’s total number of parameters.

6

417

52

439

98K

Who to follow

jackson rodriguez

@0xjackson

founder in residence @1kxnetwork | building something for digital relationships, networks, and capital | by way of @3rm_co, @asksurf, and others

Simon Berens

@sberens

bright lighting enthusiast, founder @brighterlamp

Tara Viswanathan

@TaraViswanathan

Building @unltdindustries 🏗️ prev: Founder & CEO Rupa Health (exit: 2024), @stanford, TX raised, CA living 🤠🌊 sharing stuff I learn & want to remember 🙌

rlattt @robertlatt

almost 3 years ago

@nicholastao_ 🤦🏽‍♂️

0

193

rlattt @robertlatt

almost 3 years ago

@_jasonwei We give the same emotional connection experience focusing on virtual YouTuber and anime fandom at @YoDayo_Home you can make images and chat with your favorite anime characters there

0

377

robertlatt retweeted

Shayne Longpre

@ShayneRedford

about 3 years ago

We're releasing our 2nd lecture playlist for Evaluating Generative AI/LLMs, taught at @MIT. Intro to 3 challenge areas: 1⃣ Bias & Toxicity 2⃣ Hallucination & Factuality 3⃣ Robustness & Consistency 📽️: https://t.co/lOpDECHhpR 1/🧵

2

363

84

308

65K

rlattt @robertlatt

about 3 years ago

@Denise_TEW2013 @YoDayo_Home Sorry about it, we are in good shape now!!

0

1

0

16

rlattt @robertlatt

about 3 years ago

@manlikeNachoes Yes did you email me?

1

0

70

robertlatt retweeted

Sully

@SullyOmarr

over 3 years ago

Vector databases & embeddings are the current hot thing in AI. Pinecone, a vector DB company, just raised $100M at ~1b valuation. Shopify, Brex, Hubspot and others use them for their AI apps But what are they, how do they work and why are they SO crucial in AI? Let's find out

SullyOmarr's tweet photo. Vector databases & embeddings are the current hot thing in AI.

Pinecone, a vector DB company, just raised $100M at ~1b valuation.

Shopify, Brex, Hubspot and others use them for their AI apps

But what are they, how do they work and why are they SO crucial in AI? Let's find out https://t.co/0o4zdIssDd

112

3K

393

4K

2M

rlattt @robertlatt

over 3 years ago

@kirbyman01 @Snap @a16z So for social game apps that give away free credits to new users and daily credits etc. How do these apps solve the retention when their users are making multiple accounts to avoid paying for in-app credits?

0

88

rlattt @robertlatt

over 3 years ago

@NAI7677 love it! can you reshare it on @YoDayo_Home as well?

1

0

98

robertlatt retweeted

Moescape AI

@MoescapeAi

over 3 years ago

We launched a new feature to make your own waifus with our AI!! We gives everyone 150 free tries so enjoy! This generative AI feature is available for both web and iOS users!!