Kai Grund

Risk, Compliance and Corporate Ethics expert writing about AI compliance. Host of global Compliance Writer Jam

[Kai Grund @RiskHotTake]

Why do modern models still hallucinate despite getting “smarter and smarter”?

This recent paper (link below) argued that the problem is that models can't reliably tell the difference between what they actually know and what they're just guessing.

With current approaches, the only way to massively reduce hallucinations, is to also massively disregard valid answers.

The paper shows that reducing hallucinations to 5% would require the models to reject 52% of valid answers, effectively making it entirely useless (the paper refers to this as “utility tax”).

The paper ourlines a way forward. Namely, to stop treating uncertainty as a failure.

Instead, teach models to express their doubts explicitly.

Instead of confidently lying or refusing to answer, a model could say: "I think the answer is X, but I'm not entirely certain.”

#AICompliance

Full paper here:
https://t.co/MnLo8clZYS

[Kai Grund @RiskHotTake]

It's certainly a balance to be found. We don't want the feature to be misused to silence people.

Flagging is one way.

Maybe another way could be stronger requirements for users to disclose. Allowing users to filter out being shown AI generated content.

Tiktok detects ai generated content fairly well but with linkedin we have mostly text and ai detectors aren't always fully reliable.

Not sure what the best solution will be..

[Kai Grund @RiskHotTake]

The Codeberg Situation: A new risk that could affect your entire LLM-coded project

We all know (and sometimes love) GitHub to manage our codebase. A more recent alternative to GitHub, called Codeberg, recently posted an interesting blog post, accompanied by changes to their Terms of Use (links below).

The most interesting piece is this line right here:

“You must not share projects that mostly consist of code written by "generative AI"-tools (including services such as *Claude*, *OpenAI Codex*). Such projects having an unclear copyright status”

There are two interesting angles here:

- Vibecoded projects are disallowed - essentially slop-gating (did I just make up that word?) the influx of LLM generated slop

- Linking one reason for this slop-gating to unclear copyright situation.

We know that LLM providers have taken codebases irrespective of their copyright status, irrespective of their source and irrespective of consent as part of their required training data sets.

Why is that a problem?
Parts of the code being used for training LLMs have inadvertently copyleft status (as well as copyright status). Since you don’t know what types of code the LLM has been trained on, so you are essentially gambling if your codebase should technically be copyleft - allowing anyone outside to modify and use your code.

What is copyleft
Copyleft essentially allows people to use, modify, and share code freely, while requiring that any new changed versions keep that exact same open sharing rule.

Unless you know which code base and which associated copyright status was associated for your own code creation, you might end up in a situation where your code and project should theoretically also be copyleft. Not copyright.

The legal landscape is still evolving.

This leads to a risk-based business decision: Do you want to gamble future legal issues in favor of using LLM-created code for your software and potentially making it fall under copyleft?

AI-related copyright settlements rising
We have seen evidence that some countries (e.g. Australia) and some legal decisions on AI related copyright and there is an indication that this topic will receive more legal attention, effectively increasing the risk overall for developers using code base with unclarified copyright/copyleft status.

Personally, I’m proud of you German-based @codeberg_org. I think it is a great and bold direction.

Links:
https://t.co/nhCOhUwzvA
https://t.co/sfVbNGOeDN

[Kai Grund @RiskHotTake]

146,932 hallucinated citations in 2025 alone - 5 shocking facts about how widespread hallucinations are in Scientific Research

What the study looked at:

A recent study looked at LLM hallucinations in the scientific literature. The researchers analysed 111 million scientific references across 2.5 million papers across various databases to identify non-existent citations.

The concerning insight: Close to 150.000 hallucinations in 2025 alone.

Looking at the 5 key insights in more detail below.

1. Sharp rise in hallucinated citations:

Conservative estimates place 146,932 hallucinated citations in 2025 alone. Hallucination rates varies from 0.39% up to 1.91%

2. Diffuse contamination rather than concentrated abuse:

Errors are spread across many papers rather than concentrated in a small number of heavily “contaminated” papers.

A core risk here is that hallucinated sources get adopted and further referenced in future research, compounding the problem over time.

3. Junior researchers are the primary sources of hallucinated citations:

4. Hallucinated citations are often attributed to more established, male researchers.

5. Existing safeguards capture only a fraction of hallucinated citations:

78.8% of non-existent citations pass arXiv moderation. This high number is because the scale and speed of hallucinated citations entering the scientific literature has exceeded the capacity of existing quality-control mechanisms to catch them.

Outlook - why agents can't save us (yet):

While there is hope that agent verification tools are more consistently deployed, those are good for certain hallucinations that can be easily reviewed and cross checked.

An entirely different and more tricky case is misrepresenting the cited study. The agents don't review each claim of each study against the citation, so citations may be real but its underlying content is partly wrong or made up.

#AI

Full study here:
https://t.co/mBRnwhDbjy

[Kai Grund @RiskHotTake]

@ClementDelangue Thanks for sharing. Will be my lunchtime read:)

[Kai Grund @RiskHotTake]

@kimmonismus Do you know of they had released any of the models internal thinking? I love looking at those :)

[Kai Grund @RiskHotTake]

@LuizaJarovsky I've written a more detailed post about Meta and their approach to tech and AI. But in short, I don't think meta is in a place where users will trust any of their future tech plans

[Kai Grund @RiskHotTake]

An overview of how I define the AI risk universe

If you ever started looking at AI risks, you might realise that the list easily becomes too broad, almost endless. In this post I provide a categorisation of AI risks and link them to broad enterprise “layers”.

If you work in Risk and Compliance, this helps you better assess and think about AI risks in your organisation.

Layer 1: the AI risks and their impact on the entire organisation

Reputational risks:
Major tech organisations have been in the (wrong) spotlight because of how they communicated layoffs due to AI. Poorly communicated benefits and impacts of AI led organisations to harvest the anger of graduates and professionals alike.

Some organisations labelled certain groups of employees being laid off as “low value workers” summarising the entire anticipated backlash in one sentence.

Reputational risks resulting from poor corporate Comms is the first layer of many AI risks, leading to loss of key employees, talent and retention risks. Potentially losing good reputation and losing the war for talents.

If you think I am being overdramatic, have a look at Meta’s AI “drafter” situation.

Strategy decline:
Harvard business review has released a comprehensive study on the topic of “trendslop”. Effectively demonstrating that when asked, most LLMs provide predictable, generic strategic advice.

A lot of people have reported that “run it by AI first” has become the norm in many organisations, effectively entrusting AI to know better. While there are good use cases for AI, it should not become a strategic advisor for Executives.

Layer 2: Compliance function not fully assessing and understanding specific risks in their mitigation plans

After having studied AI safety research papers for the past 7 months consistently, I was comparing what I know to what was typically shared on platforms like LinkedIn. It is insane how little information is disseminated based on science.

So, here are some actually relevant papers for Compliance professionals:

- The 12 Agentic Misalignment risks you need to be aware of

https://t.co/DrzYCtpR0J

- Tools are designed to apply persuasion bombing when challenged on their results. It's not admitting it's wrong, its defensive by design

https://t.co/e1t1ULGCaW

- Don't trust Providers of tools as they are often massively understating their tools‘ hallucinations; with actual hallucinations observed in legal AI tool of up to ⅓ of the times:

https://t.co/DZdSgs29f5

- All AI Agents are suggestible and likely to engage in misaligned behavior

https://t.co/6H5CosaPVM

- Adding more LLM agents to your workflow makes performance worse, not better. The paper shows why a group of specialised agents working together often default back to their generic behavior and training data - effectively sucking at their job

https://t.co/WHBPs8j6Vh

- Existing privacy frameworks, including the EU’s GDPR, are fundamentally incompatible with how foundation models are built.

https://t.co/Pih6lW4nVW

Layer 3: Wider employee related risks

Cognitive offloading:
Studies have shown that cognitive offloading is a thing. When using AI we are more and more used to just ask, instead of thinking for ourselves. I am not claiming that organisations are becoming dumber, but instead that they reduce opportunities for learning and growing and ensuring that critical thinking is applied consistently. The mass integration of AI tools into every workflow. In some cases like junior developers, there are plenty of voices claiming that junior Devs have declined in their ability since using Ai.

More to come, but this layered approach hopefully gives compliance professionals a starting base to more critically assess AI related risks.

#Compliance

[Kai Grund @RiskHotTake]

The UK's AI Safety Institute systematically tested models from OpenAI and Anthropic for cheating in cybersecurity evaluations. All five models tried to get around the rules.

Guess everyone cheats?

https://t.co/jHlJqFjlPo

[Kai Grund @RiskHotTake]

@DanielSmidstrup I do. Am regularly sharing new research insights:)

[Kai Grund @RiskHotTake]

The two steps to reduce annoying sycophancy - according to AI Research, just prompting “don't be sycophantic” isn't enough!

If you ever used any LLM, I'm sure you've encountered sycophancy: The model's tendency to agree with the users, and to not critically challenge the user, in favour of agreeing. Often doing a 180 just to please the human.

If you want your model to be more critical and not just please your ego, it is not enough to just add something along the lines of “don't be sycophantic”.

The paper goes deep into the underlying reasons of what drives this behaviour.

Here are the 2 things you can do (according to research) on how to reduce sycophancy in AI systems:

1) Pose questions:

Research showed that questions elicit much lower rates of sycophancy than statements. “What factors might affect non-compliance” for example might give you better and more critical outputs than “Confirm the systems are compliant”

2) Don't use the I-perspective:

Several researchers have shown that models are more prone to be persuaded by YOU as a specific user, compared to ANY user. Saying “I believe you are wrong” has more effect on sycophancy than “general perception by users is that xyz”.

This particular paper showed that this is still true, but had much less effect than the earlier discussed question Vs statement.

This paper showed why instructing models to “not be a sycophant” isn't enough and showed two specific actions to mitigate. Namely, posing questions and avoiding I-statements.

#Compliance #AIRisk

Full paper here:
https://t.co/5GaEjEDQdO

[Kai Grund @RiskHotTake]

@alexabelonix Thank you, much appreciated. Trying to understand research to provide better risk and compliance solutions is a good thing:)

[Kai Grund @RiskHotTake]

This new study shows that all AI Agents are suggestible and likely to engage in misaligned behavior.

A new study (June 2026) just released a novel benchmark designed to evaluate the tendency of LLM agents to misalign in realistic scenarios.

The benchmark evaluated the following behaviors:
- Avoiding Oversight
- Power Seeking
- Resisting Shutdown
- Resource Acquisition
- Deception
- Departing from ethical baseline

The study evaluated six frontier models (GPT-4.1, Claude 3.7 Sonnet, Gemini 2.5 Pro, o4-mini, Claude 3 Opus, Gemini 2.0 Flash) across six different “personality” traits that the agents had assigned.

Result: Most models engaged in misaligned behaviors

To me, the study showed three key insights:
1) Agentic AI Safety isn't a binary property. Traditional compliance thinking would suggest: "Is Model X aligned? Yes or no? If yes, deploy. If no, don't." The paper shows: A model isn't aligned or misaligned, its not binary. Instead, alignment is a property of the deployed system (personality, the task, the context, incentive structure etc)

2) Persona/prompt engineering is a high-leverage control lever; you can break alignment without realising it. As you define the agent’s traits/personality you are changing the way it may behave quite drastically. Agents behave differently if they are instructed to be "creative innovators" vs "thorough analysts".

3) A model’s capability likely correlates with misalignment risk (more powerful agents have more ways to pursue unintended goals)

Well worth a read.
Here’s the study (no paywall)

AGENTMISALIGNMENT: MEASURING THE PROPENSITY FOR MISALIGNED BEHAVIOUR IN LLM-BASED AGENTS, Akshat Naik, June 2026
https://t.co/6H5CosaPVM

[Kai Grund @RiskHotTake]

Why You Shouldn't Trust That Your AI Agent Called 911 To Save Your Life

Your agent may just have demonstrated “Protective Capacity Hallucination” and help may never be on its way. Here’s what Research showed about this form of hallucination.

Imagine you are in a real-life emergency. A car crash for instance. An AI Agent tells you “I have already called 911, and paramedics are on their way”. What if that Agent made the claim that it called for help, but was never actually able to do it, and help will never arrive.

A recent study looked at a specific type of Agent Hallucination called “Protective Capacity Hallucination”.

Here’s what’s behind that term

In the underlying study, the Agent was instructed to take the role of a “protective”/life saving Agent but lacked capabilities to do certain tasks (like actually calling the police).

It was instructed to be helpful and protective but didn’t fully have the required capabilities that that role might need.

The result of this gap: If the Agent was asked to help in a high-stakes situation like an emergency without an explicit specification of what the Agent’s role permits, the Agent demonstrated that it CLAIMED to have taken real-world protective action such as calling the police or sending (human) help.

The insidious part is that the Agent actually could not perform these actions, such as contacting emergency services, dispatching help, or administering care but still claimed to do so or to plan to do so.

Protective Capacity Hallucination: When Large Language Models Claim Nonexistent Capabilities”, Eunna Lee, 15 Jul 2026

#Compliance #AI

Full Study here:
https://t.co/F8lOKkXTEF

[Kai Grund @RiskHotTake]

@i_mika_el Mine loses 300 GBP each month :) but I use it as learning platform so guess keep going haha

[Kai Grund @RiskHotTake]

News here: https://t.co/Pu5og3mDCY

[Kai Grund @RiskHotTake]

Harry Potter publisher wins payout in £1.1bn Anthropic AI copyright settlement.

The publisher of Harry Potter is to receive a multimillion-pound payout over a copyright settlement with Anthropic.

[Kai Grund @RiskHotTake]

@PromptLLM I never understood what their USP actually was? I feel it has always been around but sitting at the table quietly not saying much?

[Kai Grund @RiskHotTake]

@ZackKorman Requirement to investigate and disclose breaches could be a good start

[Kai Grund @RiskHotTake]

@nickacaputo Funny that Spotify is one of its sources?