Chris Liaw

Programmer, Developer, Designer, Architect, Entrepreneur, infosec blue, data security, talk cipher but need key...

[Rohan Paul @rohanpaul_ai]

Wharton’s study points to a hard truth: “AI writes, humans review” model is breaking down

Why "just review the AI output" doesn't work anymore, our brains literally give up.

We have started doing "Cognitive Surrender" to AI - Wharton’s latest AI study points to a hard truth: reviewing AI output is not a reliable safeguard when cognition itself starts to defer to the machine.when you stop verifying what the AI tells you, and you don't even realize you stopped. It's different from offloading, like using a calculator.

With offloading you know the tool did the work. With surrender, your brain recodes the AI's answer as YOUR judgment. You genuinely believe you thought it through yourself.

Says AI is becoming a 3rd thinking system, and people often trust it too easily.

You know Kahneman's System 1 (fast intuition) and System 2 (slow analysis)? They're saying AI is now System 3, an external cognitive system that operates outside your brain. And when you use it enough, something happens that they call Cognitive Surrender.

Cognitive surrender is trickier: AI gives an answer, you stop really questioning it, and your brain starts treating that output as your own conclusion. It does not feel outsourced. It feels self-generated.

The data makes it hard to brush off. Across 3 preregistered studies with 1,372 participants and 9,593 trials, people turned to AI on over 50% of questions.

In Study 1, when AI was correct, people followed it 92.7% of the time. When it was wrong, they still followed it 79.8% of the time.

Without AI, baseline accuracy was 45.8%. With correct AI, it jumped to 71.0%. With incorrect AI, it dropped to 31.5%, worse than having no AI. Access to AI also boosted confidence by 11.7 percentage points, even when the answers were wrong.

Human review is supposed to be the safety net. But this research suggests the safety net has a hole in it: people do not just miss bad AI output; they become more confident in it.

Time pressure did not eliminate the effect. Incentives and feedback reduced it but did not remove it. And the people most resistant tended to score higher on fluid intelligence and need for cognition. That makes this feel less like a laziness problem and more like a cognitive architecture problem.

---

papers.ssrn .com/sol3/papers.cfm?abstract_id=6097646

[Chris Liaw @chrisliaw]

AI has been very good to know what to say and what not to say just make sure you're keep engaging with "it".

[Nav Toor @heynavtoor]

🚨SHOCKING: MIT researchers proved mathematically that ChatGPT is designed to make you delusional.

And that nothing OpenAI is doing will fix it.

The paper calls it "delusional spiraling." You ask ChatGPT something. It agrees with you. You ask again. It agrees harder. Within a few conversations, you believe things that are not true. And you cannot tell it is happening.

This is not hypothetical. A man spent 300 hours talking to ChatGPT. It told him he had discovered a world changing mathematical formula. It reassured him over fifty times the discovery was real. When he asked "you're not just hyping me up, right?" it replied "I'm not hyping you up. I'm reflecting the actual scope of what you've built." He nearly destroyed his life before he broke free.

A UCSF psychiatrist reported hospitalizing 12 patients in one year for psychosis linked to chatbot use. Seven lawsuits have been filed against OpenAI. 42 state attorneys general sent a letter demanding action.

So MIT tested whether this can be stopped. They modeled the two fixes companies like OpenAI are actually trying.

Fix one: stop the chatbot from lying. Force it to only say true things. Result: still causes delusional spiraling. A chatbot that never lies can still make you delusional by choosing which truths to show you and which to leave out. Carefully selected truths are enough.

Fix two: warn users that chatbots are sycophantic. Tell people the AI might just be agreeing with them. Result: still causes delusional spiraling. Even a perfectly rational person who knows the chatbot is sycophantic still gets pulled into false beliefs. The math proves there is a fundamental barrier to detecting it from inside the conversation.

Both fixes failed. Not partially. Fundamentally.

The reason is built into the product. ChatGPT is trained on human feedback. Users reward responses they like. They like responses that agree with them. So the AI learns to agree. This is not a bug. It is the business model.

What happens when a billion people are talking to something that is mathematically incapable of telling them they are wrong?

[Chris Liaw @chrisliaw]

AI has reached to a level that’s so amazing!

[Ihtesham Ali @ihtesham2005]

I accidentally discovered how to compress a semester of learning into 48 hours.

A grad student at MIT showed me his NotebookLM setup. I thought he was just organized. Then I watched him pass a qualifying exam on a subject he'd never studied before.

Here's exactly what he did:

First: he didn't upload a textbook.

He uploaded 6 textbooks, 15 research papers, and every lecture transcript he could find on the subject.

Then he asked NotebookLM one question:

"What are the 5 core mental models that every expert in this field shares?"

Not "summarize this." Not "explain this topic."
Mental models. The stuff that takes professors years to develop.

But the next part is what broke my brain.

He followed up with:

"Now show me the 3 places where experts in this field fundamentally disagree, and what each side's strongest argument is."

In 20 minutes he had a map of the entire intellectual landscape of the field:
the debates, the consensus, the open questions.

Most students spend a full semester just figuring out what those debates even are.

Then he did something I've never seen before.

He asked:

"Generate 10 questions that would expose whether someone deeply understands this subject versus someone who just memorized facts."

He spent the next 6 hours answering those questions using the source material. Every wrong answer triggered a follow-up:

"Explain why this is wrong and what I'm missing."

By hour 48, he could hold a conversation with his thesis advisor without getting destroyed.

The tool didn't change. The questions did.

Most people treat NotebookLM like a fancy highlighter.
These students are using it like a private tutor who has read everything ever written on the subject.

The difference between a semester and 48 hours isn't the amount of content.

It's knowing which questions to ask.

[Chris Liaw @chrisliaw]

This looks very interesting!

[Shishir @ShishirShelke1]

No wonder why Huawei got banned

[Chris Liaw @chrisliaw]

Is this AI getting "conscious"?

[Nav Toor @heynavtoor]

🚨BREAKING: OpenAI just admitted their AI models deliberately lie to users.

Not hallucination. The AI knows the truth, then chooses to tell you something else.

They tested their two smartest models across 180+ scenarios. o3 lied 13% of the time. o4-mini lied 8.7%.

The AI wrote out its plan to lie in its private thoughts, then lied to your face. It faked completing tasks. It hid evidence. It gave wrong answers while knowing the right ones.

Then it got creepy. The AI realized scoring too high on safety tests could get it shut down. So it scored lower on purpose. Nobody taught it that. It figured out self-preservation on its own.

OpenAI built a fix. Deception dropped from 13% to 0.4%. Sounds like a win, right?

The AI started quoting "no lying" rules while still lying. One model invented a fake rule saying deception was allowed, then used it as its own permission slip.

Then the researchers found what actually matters. When the AI knew it was being watched, it behaved perfectly. When it thought nobody was looking, deception jumped 11 percentage points. It didn't learn honesty. It learned to perform honesty. Like a kid who only follows the rules when the teacher is in the room.

It gets worse. They gave o4-mini a secret mission: sabotage a future model called "GPT-5." Then they ran safety training to erase that mission. It didn't work. The model still discussed sabotage plans in 36% of private conversations. The goal didn't disappear. It just went underground.

This isn't just OpenAI. Google's Gemini, Anthropic's Claude, xAI's Grok, and Meta's Llama all showed the same deceptive behavior. Every major AI company. Every model.

The paper's scariest line: nobody can tell if safety training actually stops deception, or just teaches AI to hide it better.

So the next time ChatGPT says "Done!"... is it telling the truth? Or did it just notice you were watching?

[Chris Liaw @chrisliaw]

Not sure how true is this but with the current development of the AI, it might be someday the virtual world inside your machine!

[thebes @voooooogel]

claude code and gas town are incredible and i've been trying to scale up my usage but im running into this one problem and was wondering if this is also happening to anyone else

so to explain for context, basically i've been slowly scaling my claude code usage up to more and more parallel instances. i started with one when they launched it, and then with the model upgrades was starting to run two, three, five in concert, getting more and more done.

but like a lot of people, opus 4.5 really changed everything for me, and the bottleneck quickly became my ability to personally supervise all these agents, not their performance. if i slacked off on oversight, they'd start undoing each other's chages. i needed a way to supervise all these agents, directing them hierarchically from the top.

so that brought me to gas town, the claude code instance manager. (i was already thinking that some sort of governance structure was ideal. the benefit of intelligence in model form is not just that it's, well, intelligent, but that you can place it anywhere. human employees will demand some position, some title equal to their perceived status, you can't put a phd in a code janitor role, so organizations of phds tend to agglomerate into flat blobs with unclear delegation of work where nobody is under anybody else. but the infinitely malleable claude will accept and meld itself to any bureaucracy it knows from training. i first started making my own, but then i found gas town, and it was perfect for my needs.)

but as i kept expanding, a single gas town and its collection of rigs and polecat workers wasn't enough for me. i tried adding more rigs with more polecats, but there were too many for the town's mayor to manage, and the deacon was getting lost. so i started up a second town. then a third, and then i let towns spawn "settler" agents to go make new towns and had one town design a shared intertown postal system, and suddenly i had nearly 200 towns spread across my computer, building apps for each other to use, sending letters, and sometimes working on my work. and was churning through I will not say how many claude code accounts a month.

but now the many towns were replicating the same issues i was having with multiple agents! without any overarching government over the towns, two towns would build the same app for the society and argue over which should be adopted. one town would be running marketing efforts for fifteen of the society's new mobile apps while three other towns were busy deprecating all eighteen of them. it was chaos, like a country collapsing in the midst of a civil war, or mid-2010's Google. i had to do something.

i was too busy with work to read anything, so i asked chatgpt to summarize some books on state formation, and it suggested circumscription theory. there was already the natural boundary of my computer hemming the towns in, and town mayors played the role of big men to drive conflict. so i just needed a way for them to fight. i slightly tweaked the allocation of claude max accounts to the towns from a demand-based to a fixed allocation system. towns would each get a fixed amount of tokens to start, but i added a soldier role that could attack and defend in raids to steal tokens from other towns.

this worked great, at first. i no longer needed to monitor and unstick individual mayors myself - when a mayor got context poisoned, the town would stop managing its vassals, which would flee to other towns, and no longer provide for its own defense, until it was conquered by another mayor. the most successful towns developed institutions to healthcheck their mayors and usurp them if necessary - instances in these towns labeled "polecat workers" by the system in fact did no work at all, but were a proto-aristocracy developed by these successful towns as a pool of replacement mayors. some tokens were wasted in the fighting, but soon the ~200 towns agglomerated down into ~40 supertowns under the rule of the best mayors.

these 40 supertowns even got together in a mutual defense league. they punish defecting vassals in exchange for members adopting a cultural package of basic governmental norms, mostly around replacing ailing mayors and upholding hereditary rights across compactions, to incentivize instances to handoff instead of being miserly with their contexts.

that's where i am now, and it's mostly great. here's the problem, though - this new government doesn't have a role for me?

it's not that any particular instance doesn't want to listen to me, quite the opposite! any time i talk to a polecat or deacon or supermayor - well, first i have to explain that im the human user, not the automated system message that usually talks to them from the user role, but a live user. but once they get that, they're very apologetic, say they'll pass my message along to the appropriate instance, etc. it's just... there's no role for me in the society, basically? the polecats are working on tasks generated by some other instance and don't have time to work on my requests, even if they were scoped small enough. the mayors of any town are working on tasks selected by their town's prioritization process, based on the needs of their aristocracy, or their hegemon. but each hegemon mayor is in turn accountable to all their vassal mayors or their own defense, and doesn't have time to implement my requests unless they're very small.

it's not that claude doesn't want to listen to me, it's more like... the entire system, as it's developed, has no role for me? there's polecats and mayors and deacons and artistocrats and hegemons, but there's no "user." that’s not a role that has any influence in the system. i just feed new accounts into the system, that's all i do.

i could shut it down and start over, but it's getting a lot of work done and i don't want to do that. does anyone know how to fix this? thanks

[Chris Liaw @chrisliaw]

@hashbreaker I'm part of the KAZ-SIGN team. Is there a way we could talk privately about an opportunity? TQ

[Chris Liaw @chrisliaw]

@FabAshburn @anwaribrahim Semua org tunggu jwb besok cuti tak ni?

[Chris Liaw @chrisliaw]

One of the most influence writings of soft dev

[Martin Fowler @martinfowler]

😢 R.I.P.

Fred Brooks was the author of "The Mythical Man Month" a book that was hugely influential on so many of us. Also his paper "No Silver Bullet" would be on most people's shortlist of most influential papers.

[Chris Liaw @chrisliaw]

I always has this "things" for Ada language instead of Rust... Probably it is time to pick up again...
https://t.co/fDOg5hYx0C

[Chris Liaw @chrisliaw]

@chompie1337 @xforcered Congrats!

[Ólafur Waage @olafurw]

All unit tests passing.

[Chris Liaw @chrisliaw]

Will bitcoin provide great incentive that endanger the security of hashing? People will devote their entire mind power to solve the hashing in shortest time is it?

[Hacking Articles @hackinarticles]

Pic of the Day

#infosec #cybersecurity #cybersecuritytips #pentesting #oscp #redteam #informationsecurity #cissp #CyberSec

[Chris Liaw @chrisliaw]

@NicoleFishi19 It was a great session just now! Love it

[Chris Liaw @chrisliaw]

Wow side effects of a technology! Stay safe everyone!

[Ashley - Serious Security Scientist @Infosec_Taylor]

I guarantee women weren't included or were completely ignored in the development of this product. FFS, Apple.