Abi

your friendly neighbourhood pseudo-intellectual™️ I don’t know what I’m talking about.

[Abi @quirkdom]

I wonder how watermarks are affected when the model is asked to conform to STE. Does everything detect as watermarked (even when it isn’t; because word choice sample size is so small) or watermarking becomes painfully obvious to the reader?

[Alex Cui @alexcdot]

Claude's watermark probably doesn't work how you think. As the CTO of GPTZero, I'll explain how Anthropic, Google and OpenAI are building text watermarking in this brief explainer and whether it can be defeated.

Almost all forms of watermarking that are fast and cheap enough for a frontier lab have the same formula, following the KGW method:

In generation:

1. Let's say you've generated n tokens so far. Take those n tokens + a secret key to generate a random hash
2. Use that hash to randomly reweight the probabilities for the n+1 token, and then sample from that new distribution. In the simple case, you could split 50% of all English words into a green or red set based on your hash, and boost the probability of words in the green set.

For watermark detection:

1. For each token, see if it was in the green or red set.
2. To do this, recreate the hash based on the secret key and the text preceding the current token. Then, recreate the green and red set of words.
3. Once you've checked all the words in the text, if the next token is selected disproportionally from the green set more than 50% of the time, you claim the text has the watermark.

I can tell you want to ask the following:

1) Isn't it easy to mess up the hash if you paraphrase the text? The answer is mostly yes, however, you can use a statistical model to get your hash instead of a deterministic function (SIR, Adaptive Watermark). Since the entire watermark is probabilistic, this is fine.

2) Doesn't this make the text much worse? The answer is yes, it does - Yes, it does – but for most people, it's imperceptible (Google claims in human feedback study with 20,000 texts), since there are exponentially many ways to write the same paragraph. DiPmark does something more sophisticated to avoid shifting the text distribution on average. Of course, watermarks fail on short text or highly predictable texts like "2+2=4".

3) Shouldn't it be easy to figure out the green and red sets? The answer is no. You would need an exponentially large number of samples from the watermarker to reconstruct those sets exactly, but it's a risk if the detector is open to the wild (Watermark Stealing)

Still, there are couple challenges that a frontier lab needs to overcome:
1. Their watermark needs to work token-by-token because they are streaming their text to users. Many watermark methods plan sentences or paragraphs at a time, or change the text after its entirely written, in order to make their watermark robust to paraphrasers, and a frontier lab cannot afford to do this yet (SemStamp, PostMark)
2. If the secret key leaks, the watermark is busted. To avoid a large blast damage from this, you need to have a couple secret keys in rotation.
3. There are some texts, like code, that cannot be arbitrarily changed, otherwise the code will break. In those cases, the watermark needs to selectively change words in parts of the text that can tolerate synonyms (i.e. like variable naming) - see SWEET, EWD, Invisible Entropy.
4. They will need to educate their users on how to deal with false positives and false negatives of a detector, which is a big challenge (one we put a lot of effort into)

So, how do I see this playing out in the next 6 months?
1. If Anthropic releases the watermark detector publically, I think they defeat their own watermark. People find reliable watermark removal strategies by testing against Anthropic (AI detectors like GPTZero have an advantage here because they can train against these adversaries once they become popular).
2. If they keep the detector private to the government, like Google has done, it's "safer". However, there are some papers showing trained approaches that work robustly to zero-shot break watermarks without any data, simply because they try to write the text just like a human (Zhang et al. 2024, Watermarks in the Sand). Also, making your detector makes it battle-tested and stronger long-term (my experience).
3. In my testing, the watermarks don't survive intense paraphrasing (especially if you combine word choice and syntax attacks), or human text substitution (rewrite your AI text by plagiarizing human authors). The free paraphrasers I've tried have quickly bypassed Google Deepmind's SynthId for what it's worth.
4. All-in-all, frontier labs are likely okay with this because they expect most users to not attack the watermark, and also because they + European regulators likely don't care past a certain point - its good enough.
5. Overall, I think users of frontier LLMs will not really care about this, because 1) they don't realize watermarks are there, 2) EU will force everyone to conform, 3) this seems more like regulatory hoop-jumping than an earnest effort from frontier labs to expose LLM use

Lastly, people's first concern shouldn't be watermarking, it should be AI detectors!

If you're posting, "its not X, its Y!!", I don't think the watermark is going to make a difference :)

[Kenton Varda @KentonVarda]

@harryh As the original creator of Jeff Dean Facts I hereby designate this an official Jeff Dean Fact. However, I must point out it is also a Sanjay Ghemawat fact.

[Anushka @Kulfei]

do devs get promoted and become Mahadev?

[Beff (e/acc) @beffjezos]

when you say "life is a stochastic path integral" instead of "shit happens"

[Abi @quirkdom]

That’s the point buddy.

[Christoph Nakazawa @cnakazawa]

Maybe I'm stupid but I really don't get passkeys. They are convenient, sure, but they effectively reduce 2fa into 0fa.

[patagucci perf papi @kenwheeler]

if you’ve ever worked in open floorplan, this is basically the fuckin four seasons

[Doug @DouglasAll]

acabei de ver Odisseia em uma sala comum e digo

[unhingedchipmunk @amanyayayayayay]

kya jain log underground metro use krte hain

[Wellu @Wellutwt]

Watching nolan masterpiece
#TheOdyssey

[Duhlorean @Duhlorean98]

The nuke moment is easily the weakest scene in Oppenheimer where it looks like a tiny gasoline explosion.

I get that Nolan has an obsession with practicality but I wish the actual scene looked more like this clip. Should’ve used a CGI + archival footage approach.

[Brian Lovin @brian_lovin]

Someone should make an app to record meetings with automatic AI transcripts. Underexplored space rn.

[Battle Programmer Yuu @netspooky]

Vendor: Our advanced hardware platform-

Me: raspberry pi?

Vendor, nervous: uses next gen computer vision sensors-

Me: raspberry pi camera?

Vendor, frustrated: to send real time insights to our cloud platform-

Me: is this your storage bucket?

Vendor: ...yeah, wait, shit...

[P @_ptonik]

discovered dolby atmos remasters recently and I regret to inform you that music apparently has been in black and white this whole time.

even on my budget, sub $1k setup (2x sonos era 300s) albums i've heard 500 times suddenly grew a third dimension, almost making me cry. WHY IS NOBODY TALKING ABOUT THIS.

switched from spotify to apple music purely for this and I'd do it again in front of daniel ek's family. its insanely good - do yourself a favor and go hear it.

[​𝐥𝐲𝐫𝐚 @sunnkssdseraph]

Overlooked fact: every significant relationship in your life began with one moment of someone being slightly braver than the situation required. One person said the thing. Asked the question. Made the first move. Relationships don't start with mutual courage. They start with one person's excess courage.

[mar ❁ @russellist63]

this was cinema

[Owen Carey @owenthcarey]

Practicing LeetCode at 32 years old.

[Abi @quirkdom]

@paraschopra What are your thoughts on e-ink displays?

[Abi @quirkdom]

@flornkm Accessibility has left the chat.

[Paras Chopra @paraschopra]

I tried ditching typing for speaking using AI tools but something always felt off. Typing (or writing) gives you a buffer in the head to think, revise and explore thoughts in a way that speaking doesn't.

My bet is that if someone does a careful controlled test on quality of work output while typing v/s dictating, we'll see dictation having much worse outcomes.

[Abi @quirkdom]

@philipyoungg If you’re even remotely thinking about running small local models (for whatever reason), you’ll want the 48GB. I have the 24GB and found out the hard way when Gemma 4 caused OOM.