Lutece_48

Most of my RT are RT Co-Founder

[Andrej Karpathy @karpathy]

Fun LLM challenge that I'm thinking about: take my 2h13m tokenizer video and translate the video into the format of a book chapter (or a blog post) on tokenization. Something like:

1. Whisper the video
2. Chop up into segments of aligned images and text
3. Prompt engineer an LLM to translate piece by piece
4. Export as a page, with links citing parts of original video

More generally, a workflow like this could be applied to any input video and auto-generate "companion guides" for various tutorials in a more readable, skimmable, searchable format. Feels tractable but non-trivial.

[Andrej Karpathy @karpathy]

Seeing as I published my Tokenizer video yesterday, I thought it could be fun to take a deepdive into the Gemma tokenizer.

First, the Gemma technical report [pdf]:
https://t.co/iPVo3iLXQC
says: "We use a subset of the SentencePiece tokenizer (Kudo and Richardson, 2018) of Gemini for com- patibility. It splits digits, does not remove extra whitespace, and relies on byte-level encodings for unknown tokens, following the techniques used for both (Chowdhery et al., 2022) and (Gemini Team, 2023). The vocabulary size is 256k tokens."

The tokenizer.model file is with this code release:
https://t.co/SwcVU2nkkS

I decoded this model protobuf in Python and here is the diff with the Llama 2 tokenizer:
https://t.co/4HoAeYJsZz

Notes:
- vocab size is quite large: 32K -> 256K
- add_dummy_prefix is False. Different from Llama but consistent with GPT. This is a bit more consistent w.r.t. "leave the data alone", as there is no preprocessing step that adds a space to the encoding text.
- the model_prefix is the path of the training dataset, which is amusing to look at: "/cns/mf-d/home/gemini-data-access/tokenizers/final_v1_51GB_run1/bpe_coverage_0_999995_v5/255969". Seems to indicate the tokenizer training corpus was ~51GB (?).
- a lot of user_defined symbols (i.e. special tokens) are present, e.g. "hardcoding" a sequence of up to 31 newlines as tokens, and a large number of other unclear tokens. I tried decoding the octal representations but it's not clear what's happening here. Also a lot of more special tokens for what look like html elements, e.g. <table>, <tr>, <td>, <i>, <b>, etc. Not 100% sure what the unused tokens are for, maybe this is pre-allocated space to make easier future finetunes that try to add more special tokens, as there is no need to resize vocabularies and perform model surgeries (?).

TLDR this is basically the Llama 2 tokenizer, except bigger (32K -> 256K), with a lot more special tokens, and the only functional departure is that add_dummy_prefix is turned off to False. So e.g. tokenizing:

"hello world" becomes:
[17534, 2134]
['hello', '▁world']

which otherwise would have been preprocessed to " hello world" (note leading space) and tokenized as:
[25612, 2134]
['▁hello', '▁world']

cool

[EDDIE CHENG @ecomEddie]

I’m 33.

When I was young, I wasted years drinking, smoking, and being a degenerate.

Then I discovered Naval Ravikant and he changed my life forever.

Here are 20 teachings from the wisest person of this century (that will change your life too):

[Andrej Karpathy @karpathy]

"My benchmark for large language models"
https://t.co/YZBuwpL0tl

Nice post but even more than the 100 tests specifically, the Github code looks excellent - full-featured test evaluation framework, easy to extend with further tests and run against many LLMs.
https://t.co/KnmDD1AJci

E.g. for the 100 current tests on 7 models:
- GPT-4: 49% passed
- GPT-3.5: 30% passed
- Claude 2.1: 31% passed
- Claude Instant 1.2: 23% passed
- Mistral Medium: 25% passed
- Mistral Small 21% passed
- Gemini Pro: 21% passed

Also a huge fan of the idea of mining tests from actual use cases in the chat history. I think people would be surprised how odd and artificial many "standard" LLM eval benchmarks can be. Now... how can a community collaborate on more of these benchmarks... 🤔

[Alex & Books 📚 @AlexAndBooks_]

Every industry has 1 book that will teach you 90% of what you need to know about it.

Here are the 20 best books in 20 different industries:

(ranging from automotive to venture capital)

[Joe Perkins @joeprkns]

Last night I used GPT-4 to write code for 5 micro services for a new product.

A (very good) dev quoted £5k and 2 weeks.

GPT-4 delivered the same in 3 hours, for $0.11

Genuinely mind boggling

[Balaji @balajis]

HISTORY IS RUNNING IN REVERSE

The Bitcoin ETF is the spiritual reversal of Executive Order 6102. Back in 1935, they seized the gold. But now, digital gold is back.

Ninety years ago, FDR and his fellow travelers rode the 20th century arc of centralization. The chokepoints of then-new technologies for mass media and mass production allowed them to gain control over the population, recruit top talent for their "Brain Trust", and seize the gold after a series of epic legal battles[1].

Those gold clause cases are forgotten today, but received as much contemporary coverage as 9/11 or the Moon Landing. They were the most important issue in the country, receiving far more coverage than seemingly comparable Supreme Court decisions like Roe vs Wade. Why?[2]

The reason is that the transition from a gold-backed to fiat-backed system was comparable to a soft communist revolution, as the *visible* seizure of gold laid the groundwork for the *invisible* seizure of wealth via money printing.

And the classically trained judges at that time fully understood this. Justice McReynolds' then-famous dissent denounced the ruling in the harshest terms, noting that the "Constitution is gone" and the "dollar...may be 30c tomorrow, 10c the next day, and 1c the day following".[3]

McReynolds was right.

While the court was forced into a grudging institutional surrender by FDR's threat of court-packing[2], the gold clause case affected every economic decision-maker in the country, as it amounted to the US government explicitly defaulting on its bonds by seizing the assets of its citizens, laying the groundwork for the century of monetary debasement to come.[4]

Now all of that is unwinding. FDR's team could ride the wave of centralizing technology that built giant states around the world. But today, technology today favors *decentralization* — personal computers, end-to-end encryption, mobile phones, and of course cryptocurrency [5].

Thus, top talent isn't being pulled into a government Brain Trust. It's being brain drained *out* of the US establishment. And as a consequence the epic legal battles are, on balance, going our way.

It's not just the DC Circuit case.[6] The ideological conflict between decentralization and centralization is reflected in the 3-2 vote for the Bitcoin ETF approval itself. Read Peirce's brilliant pro-liberty approval[7], Crenshaw's dour denial[8], and Gensler's reluctant approval[9].

You'll see echos of the gold clause case, but in reverse. This time, it is the centralized state that is being forced into a grudging institutional surrender. And a surrender it is, as Crenshaw's dissent[8] makes clear:

"...there is no primary regulator for the bitcoin spot markets. Spot bitcoin ETPs will be participating in an unregulated, fragmented, continuously traded, global free-for-all. Even if there were a primary regulator for this market, much of it could be beyond the reach of U.S. regulation..."

Let that sink in! This is what the US establishment truly fears: not Bitcoin as "fraud", but Bitcoin as freedom. They want to rule not just you but the world, so they're scared of the prospect of "a global free-for-all..beyond the reach of US regulation". And they know that any spot ETF will bid up the price of self-custodied Bitcoin outside their control[10], as Satoshi intended.

So: since FDR's seizure of gold, our lives have revolved around the centralized state rather than the decentralized market. The state has had control for so long we've forgotten what freedom is like. But now gold is slipping out of their hands, and back into yours.

History is running in reverse.

[1]: https://t.co/yl0MCTRrlg
[2]: https://t.co/92ahAV1AeY
[3]: https://t.co/MknSZv5Q6g
[4]: https://t.co/dbYcaOXyff
[5]: https://t.co/1DWoz5f6nW
[6]: https://t.co/COJmRAwO3K
[7]: https://t.co/pipMkosz5k
[8]: https://t.co/4tSlPrST9c
[9]: https://t.co/ZWjdGrRMXJ
[10]: https://t.co/XMCEWu2PK9

[Tim Ferriss @tferriss]

The psychological difference between zero acts of creation and one act of creation, no matter how small, is impossible to overstate. If you’re lucky, sometimes that one idea, one sentence, or one shitty first draft can turn into something bigger.

[Nicolas Cole 🚢👻 @Nicolascole77]

I asked 170,000 people for the best book they read in 2023.

These 16 turned up the most (so they will make for great reading over the holidays):

[Roope Rainisto @rainisto]

I am probably the last person to do this tutorial - everyone did this a while ago - but no matter;

Consistent characters in MidJourney and SD

It's quite useful! 🧵

[elvis @omarsar0]

How Transformers Work

This is probably one of the most beautiful visualizations of how today's LLMs work.

https://t.co/C28PDIxnA6

[Wes Gurnee @wesg52]

Do language models have an internal world model? A sense of time? At multiple spatiotemporal scales?

In a new paper with @tegmark we provide evidence that they do by finding a literal map of the world inside the activations of Llama-2!

[Max Tegmark @tegmark]

No, LLM's aren't mere stochastic parrots: Llama-2 contains a detailed model of the world, quite literally! We even discover a "longitude neuron".

[Linus ✦ Ekenstam @LinusEkenstam]

Here are 19 quick terms to get better AI images

Borrow from photography to maximize the output in Midjourney, SDXL, Runway etc.

Free cheat sheet 🙌🏻

[Aadit Sheth @aaditsh]

I wish more people stopped using ChatGPT on beginner mode.

Here are....

10 ChatGPT prompts to supercharge your prompting skills 👇

[Aadit Sheth @aaditsh]

ChatGPT prompts that will 10x your productivity (ranked in order):

[Naval @naval]

How to Get Rich (without getting lucky):

[Global Investigative Journalism Network @gijn]

NEW! 🧵 Internet research expert @henkvaness offers his tips for navigating LinkedIn from his forthcoming open source guide.

📌 Stay tuned for subsequent posts on social search techniques using Twitter, TikTok, Telegram, &amp; Instagram later this week.

https://t.co/T0sHu7VH8e