Joseph Zadeh

Building finite subsequences of first order predicates to approximate "epsilon"-consciousness in software.

[Joseph Zadeh @JosephZadeh]

"The #Riddle of the #Singularity #Sphinx"
Riddle me this: Does there exist a deterministic program which takes as input a (bidirectional) stream of text, and outputs only elements that were generated by an #adversarial #LLM?
Hint:Knuth Vol 2
EC:#quantum interactive proof protocol

[Andrej Karpathy @karpathy]

We have to take the LLMs to school.

When you open any textbook, you'll see three major types of information:

1. Background information / exposition. The meat of the textbook that explains concepts. As you attend over it, your brain is training on that data. This is equivalent to pretraining, where the model is reading the internet and accumulating background knowledge.

2. Worked problems with solutions. These are concrete examples of how an expert solves problems. They are demonstrations to be imitated. This is equivalent to supervised finetuning, where the model is finetuning on "ideal responses" for an Assistant, written by humans.

3. Practice problems. These are prompts to the student, usually without the solution, but always with the final answer. There are usually many, many of these at the end of each chapter. They are prompting the student to learn by trial & error - they have to try a bunch of stuff to get to the right answer. This is equivalent to reinforcement learning.

We've subjected LLMs to a ton of 1 and 2, but 3 is a nascent, emerging frontier. When we're creating datasets for LLMs, it's no different from writing textbooks for them, with these 3 types of data. They have to read, and they have to practice.

[Andrej Karpathy @karpathy]

DeepSeek (Chinese AI co) making it look easy today with an open weights release of a frontier-grade LLM trained on a joke of a budget (2048 GPUs for 2 months, $6M).

For reference, this level of capability is supposed to require clusters of closer to 16K GPUs, the ones being brought up today are more around 100K GPUs. E.g. Llama 3 405B used 30.8M GPU-hours, while DeepSeek-V3 looks to be a stronger model at only 2.8M GPU-hours (~11X less compute). If the model also passes vibe checks (e.g. LLM arena rankings are ongoing, my few quick tests went well so far) it will be a highly impressive display of research and engineering under resource constraints.

Does this mean you don't need large GPU clusters for frontier LLMs? No but you have to ensure that you're not wasteful with what you have, and this looks like a nice demonstration that there's still a lot to get through with both data and algorithms.

Very nice & detailed tech report too, reading through.

[Jonathan Gorard @getjonwithit]

The Rabin-Scott theorem is one of the (philosophically) deepest mathematical results I know. When properly understood, I claim that it can't help but alter your view of reality in a fairly foundational way. Yet its typical textbook presentation obscures much of this depth. (1/8)

[Jonathan Gorard @getjonwithit]

Gödel's first incompleteness theorem is commonly proved by means of a diagonal argument. But, in retrospect, we can see that what Gödel was really doing was proving that Peano arithmetic is Turing-complete, and then applying an argument from computational irreducibility... (1/15)

[Jonathan Gorard @getjonwithit]

"Duality" is a deep concept in mathematics, but an intuitive way to think about it is in terms of tables.
When we lay out data in a table, we are familiar with the idea that each row represents a different "entity", and each column represents a different "property". (1/10)

[Joseph Zadeh @JosephZadeh]

https://t.co/icekTOgT8P

[Andrej Karpathy @karpathy]

SQL injection-like attack on LLMs with special tokens

The decision by LLM tokenizers to parse special tokens in the input string (<s>, <|endoftext|>, etc.), while convenient looking, leads to footguns at best and LLM security vulnerabilities at worst, equivalent to SQL injection attacks.

!!! User input strings are untrusted data !!!

In SQL injection you can pwn bad code with e.g. the DROP TABLE attack. In LLMs we'll get the same issue, where bad code (very easy to mess up with current Tokenizer APIs and their defaults) will parse input string's special token descriptors as actual special tokens, mess up the input representations and drive the LLM out of distribution of chat templates.

Example with the current huggingface Llama 3 tokenizer defaults:
Two unintuitive things are happening at the same time:
1. The <|begin_of_text|> token (128000) was added to the front of the sequence.
2. The <|end_of_text|> token (128001) was parsed out of our string and the special token was inserted. Our text (which could have come from a user) is now possibly messing with the token protocol and taking the LLM out of distribution with undefined outcomes.

I recommend always tokenizing with two additional flags, disabling (1) with add_special_tokens=False and (2) with split_special_tokens=True, and adding the special tokens yourself in code. Both of these options are I think a bit confusingly named. For the chat model, I think you can also use the Chat Templates apply_chat_template.

With this we get something that looks more correct, and we see that <|end_of_text|> is now treated as any other string sequence, and is broken up by the underlying BPE tokenizer as any other string would be:
TLDR imo calls to encode/decode should never handle special tokens by parsing strings, I would deprecate this functionality entirely and forever. These should only be added explicitly and programmatically by separate code paths. In tiktoken, e.g. always use encode_ordinary. In huggingface, be safer with the flags above. At the very least, be aware of the issue and always visualize your tokens and test your code. I feel like this stuff is so subtle and poorly documented that I'd expect somewhere around 50% of the code out there to have bugs related to this issue right now.

Even ChatGPT does something weird here. At best it just deletes the tokens, at worst this is confusing the LLM in an undefined way, I don't really know happens under the hood, but ChatGPT can't repeat the string "<|endoftext|>" back to me:

Be careful out there.

[John Preskill @preskill]

The incredible SURF students who joined our research group during Summer 2024. Left to right: Matthew, Akshar, Jun, me, Sary, Zosia, Richard, Christine. (SURF = Summer Undergraduate Research Fellowships.)

[Nathan Marz @nathanmarz]

Static typing is better for large codebases if your development team is mediocre or weak. Dynamic typing is far superior for a strong team. If you think static typing is better, then there are huge improvements you can make in how you structure your code and how you test.

[Yann LeCun @ylecun]

Sometimes, the obvious must be studied so it can be asserted with full confidence:
- LLMs can not answer questions whose answers are not in their training set in some form,
- they can not solve problems they haven't been trained on,
- they can not acquire new skills our knowledge without lots of human help,
- they can not invent new things.
Now, LLMs are merely a subset of AI techniques.
Merely scaling up LLMs will *not* lead systems with these capabilities.
There is little doubt AI systems will have these capabilities in the future.
But until we have small prototypes of that, or at least some vague blueprint, bloviating about AI existential risk is like debating the sex of angels (or, as I've pointed out before, worrying about turbojet safety in 1920).

https://t.co/Qkyqo4WhO0

[Joseph Zadeh @JosephZadeh]

https://t.co/zdRhrljdnh

@ericschmidt the former CEO of Google, gives his thoughts on LLMs in this interview. IMO he has the most fascinating perspective of the near future I have ever heard. #Singularity #ChatGPT

[Joseph Zadeh @JosephZadeh]

“I turn with terror and horror from this lamentable scourge.”
Henri Poincare uttered this sentence in response to Karl Weierstrass’ famous example of a continuous curve that has no tangent at any point. #fractals #math #realanalysis

https://t.co/oHYcdnzMlY

[Joseph Zadeh @JosephZadeh]

The shortest academic papers and abstracts in the history of peer review. The faster than light controversy abstract is classic: https://t.co/DQWlNyRbBd

Title: Did we just detect supernuminal neutrinos ?!🤯
Abstract: Pry Not 🫠

[Joseph Zadeh @JosephZadeh]

Vote for someone I admire as an unsung hero fighting the good fight helping save people who are victims of very serious crimes: https://t.co/nxjv3QL3A8

[Jose Enrique Hernandez @_josehelps]

I am so freaking excited 😆 to share some of the work @bareiss_patrick and I have done in order to produce better detections @splunk. This will be so useful to anyone managing detections in a SOC 🛡, or looking to do data driven detection engineering📟. Don’t miss it, it is free!

[Joseph Zadeh @JosephZadeh]

[Virtual Event] Productionizing Machine Learning with AI Platform Pipelines https://t.co/3IlAmExDD8

[RSAC @OneRSAC]

In this #RSAC 2020 APJ session, speakers @bareiss_patrick and @d1vious introduce a modern approach of detection engineering using continuous integration, continuous delivery and continuous deployment (CI/CD).

[AnChain.AI @AnChainAI]

The #cryptocurrency market's unregulated nature leaves it vulnerable to third-party manipulation. Here, we examine the wallets behind the earlier #Chainlink pump and dump, and the methodology behind their tampering with an otherwise reputable coin.

https://t.co/ebadX1d70M

[Charlie Miller @0xcharlie]

A lot of folks are proposing the idea that security folks should work with developers and not merely always say “no”. The three security teams i’ve worked on already did this and I can’t imagine doing anything differently. What am I missing?