Yair Halberstadt

[Yair Halberstadt @YairHalberstad1]

@GaryMarcus @JeffLadish @sapinker He says, debating someone's personality...

[Yair Halberstadt @YairHalberstad1]

@GaryMarcus @JeffLadish @sapinker But you said you wouldn't debate him?

https://t.co/B4TEilTjP2

[Gary Marcus @GaryMarcus]

@slatestarcodex @sapinker @clairlemon i won’t debate your personality but am happy to join Steve in arguing that extinction is very unlikely.

[Yair Halberstadt @YairHalberstad1]

@AlanMCole You've strawmanned his argument. His point isn't that Magnus can win because he's more clever, but that Magnus can react to what you do. It's a response to people who say "but we'll just X", by saying X wouldn't work on a clever human adversary nevermind AI.

[Yair Halberstadt @YairHalberstad1]

@lukebeehewitt During training, runs are too short for manipulating people to work, so they're biased against thinking in that direction.

[Yair Halberstadt @YairHalberstad1]

@phl43 @treetowntree @NevinClimenhaga @grok Ok, a variant of this worked, biggest problem is that google has internal safeguards to stop you giving instructions that manipulate COT, but you can work around them.

[Yair Halberstadt @YairHalberstad1]

@phl43 @treetowntree @NevinClimenhaga @grok Hmmm I might play around with seeing if I can get a model to break it when I get the chance

[Yair Halberstadt @YairHalberstad1]

@phl43 The AI doesn't need to kill everybody with the virus. Instead it would be part of a general campaign to make coordinated response against it difficult. That could include nukes, diseases, attacking power grid and communications, etc.

[Yair Halberstadt @YairHalberstad1]

@phl43 E.g. what already exists in state sponsored labs (At the very least smallpox)? How difficult would it be to blackmail workers into stealing it? What about stealing the DNA sequences and tricking a lab into synthesising it?

[Yair Halberstadt @YairHalberstad1]

@TheZvi My statement is: AI is clearly the greatest threat and most important challenge humanity has ever faced, and yet we're running towards it as far as we possibly can. Slowing down is vital to maximise the chances it ends up going well.

[Yair Halberstadt @YairHalberstad1]

@TheZvi FWIW: my team was recently transferred to working on cloud TPUs, and I told my manager I wouldn't work on them due to existential risk. I am currently in limbo, looking for another role at Google.

[Yair Halberstadt @YairHalberstad1]

@NevinClimenhaga reasoning without COT

[Yair Halberstadt @YairHalberstad1]

@NevinClimenhaga This requires the step where entropy decreases to be out of order. So need to make sure it doesn't generate in order, then scramble, then reorder, has to go straight to scrambled, which might be tricky for small models but prob easily doable with astra (with more powerful 0 shot)

[Yair Halberstadt @YairHalberstad1]

@phl43 @treetowntree @NevinClimenhaga @grok You tell it to write to the thinking scratchpad the tokens in order 2nd,1st,4th,3rd... Then in the final output reorder them. Entropy in final output insufficient for synthID

[Yair Halberstadt @YairHalberstad1]

@phl43 @treetowntree @NevinClimenhaga @grok I wonder if it could avoid it just by generating the final tokens out of order and then reordering.

[Yair Halberstadt @YairHalberstad1]

@phl43 @treetowntree @NevinClimenhaga @grok I think the whole confusion started because you called the new program a resampler not a rewrite, and resampling is impossible.

[Yair Halberstadt @YairHalberstad1]

@phl43 But basically think one crux is ability to one-shot viruses with complex properties (e.g. delayed activation till some signal). Not saying they'll have that but will get more evidence as we see if one shotting NS counterexamples extends to one shooting materials science and bio

[Yair Halberstadt @YairHalberstad1]

@phl43 Got it! Either way if you do come it would be great to meet up!

[Yair Halberstadt @YairHalberstad1]

@phl43 Could also do email