christian cancedda

MSc Data science and AI
BSc Computer engineering

[Aryaman Arora @aryaman2020]

New paper! 🫡

We introduce Matryoshka Attribution, a new attribution method which uses gradient descent to find which parts of a neural network are responsible for a behaviour.

MAttr is #1 on the Mechanistic Interpretability Benchmark by a wide margin (2.9× the runner up).

[eran shir @eranshir]

“LLMs are just matrices. Stop anthropomorphizing.”

Bacteria are just biochemistry. Yet they cooperate, communicate, build structures and sacrifice themselves.

Reducing a system to its substrate doesn’t make its higher-level behavior disappear. https://t.co/rODLmRFs2q

[Léa | Bourse & Stratégies @LeaBourseFR]

Elon Musk vient de faire un joli pied-de-nez à l'UE.

L'UE a condamné X à 120 millions € d'amende pour manque de transparence sur son algorithme.

X publie désormais son algorithme sur GitHub, avec une mise à jour toutes les 4 semaines.

Mais ce n'est pas tout :

X va aussi rendre publiques TOUTES les demandes de censure émanant des gouvernements, dont celles de l'UE (qui semble clairement en abuser).

La pression réglementaire européenne a peut-être produit plus de transparence que prévu...

Juste pas exactement de la façon dont Bruxelles l'imaginait. 😏

[Robert Sepehr @robertsepehr]

@GiorgiaMeloni

[Andrej Karpathy @karpathy]

I've never felt this much behind as a programmer. The profession is being dramatically refactored as the bits contributed by the programmer are increasingly sparse and between. I have a sense that I could be 10X more powerful if I just properly string together what has become available over the last ~year and a failure to claim the boost feels decidedly like skill issue. There's a new programmable layer of abstraction to master (in addition to the usual layers below) involving agents, subagents, their prompts, contexts, memory, modes, permissions, tools, plugins, skills, hooks, MCP, LSP, slash commands, workflows, IDE integrations, and a need to build an all-encompassing mental model for strengths and pitfalls of fundamentally stochastic, fallible, unintelligible and changing entities suddenly intermingled with what used to be good old fashioned engineering. Clearly some powerful alien tool was handed around except it comes with no manual and everyone has to figure out how to hold it and operate it, while the resulting magnitude 9 earthquake is rocking the profession. Roll up your sleeves to not fall behind.

[alphaXiv @askalphaxiv]

This paper by Meta introduces Query-Only Test-Time Training for long contexts

So by using a small amount of inference-time training that retunes how the model attends to the given input, it works far better than generating extra reasoning tokens

[DAIR.AI @dair_ai]

RAG systems struggle with multi-hop reasoning.

In most cases, the problem isn't the LLMs. It's the retrieval system.

Standard RAG treats each piece of evidence as equally reliable, ignoring how documents connect to each other.

Why is this a problem?

When questions require reasoning across multiple sources, single-shot retrieval often misses "bridge" documents whose entities aren't mentioned in the original query. Iterative retrieval helps, but it introduces new issues: LLM-guided graph traversal can hallucinate or become stuck on partial reasoning from previous steps.

This new research introduces SA-RAG, a framework that applies spreading activation, a mechanism from cognitive psychology, to knowledge-graph-based retrieval.

How does it work?

Instead of relying on the LLM to decide which documents to fetch next, activation propagates automatically through a knowledge graph. Starting from entities matched to the query, activation spreads outward through weighted connections, with strength diminishing over distance.

Documents linked to highly activated entities get retrieved.

The system builds a hybrid structure during indexing. An LLM extracts entities and relationships from text chunks, creating a knowledge graph where documents connect to entities through "describes" links. At query time, seed entities are identified by embedding similarity, then activation flows through the graph in a breadth-first manner.

On MuSiQue, SA-RAG alone achieves 67% answer correctness with phi4, outperforming naive RAG at 45% and CoT-based iterative retrieval at 55%. When combined with chain-of-thought iterative retrieval, it reaches 74% on MuSiQue and 87% on 2WikiMultiHopQA.

This system demonstrates a 25% to 39% absolute improvement over naive RAG across benchmarks. Notably, these results come from small, open-weight models like phi4 and gemma3, which require no fine-tuning.

Spreading activation captures associative relevance rather than surface-level similarity. The method works as a plug-and-play module, boosting any training-free RAG pipeline without architectural changes.

Paper: https://t.co/jLZLkacDAX

Learn to build effective RAG and AI agents in our academy: https://t.co/zQXQt0PMbG

[DAIR.AI @dair_ai]

Top AI Papers of the Week (Dec 8 - 14):

- ProAgent
- Selective Gradient Masking
- Agentic AI Adaptation Survey
- AI and Human Co-Improvement
- Reasoning Models Ace the CFA Exams
- Pre-Training, Mid-Training, and RL Interplay
- Towards a Science of Scaling Agent Systems

Read on for more:

[Cryptography and Security Papers @FSFG]

COGNITION: From Evaluation to Defense against Multimodal LLM CAPTCHA Solvers. https://t.co/Vk8nJ59x4P

[Igor Shilov @_igorshilov]

We introduce Selective GradienT Masking (SGTM) — a pretraining-time approach inspired by Gradient Routing (Cloud et al., 2024).

The core idea: split the model's weights into retain and forget subsets, and guide specific knowledge into the forget subset during training.

[Igor Shilov @_igorshilov]

New Anthropic research!

We study how to train models so that high-risk capabilities live in a small, separate set of parameters, allowing clean capability removal when needed – for example in CBRN or cybersecurity domains.

[SEOANALYSER.AI @SEO_ANALYSER]

✨ Does an LLMs.txt File Actually Matter?

A recent large-scale analysis reviewed data from hundreds of thousands of websites to understand whether having an LLMs.txt file affects AI visibility.

🔍 The result?
Not really, at least not for now.

📊 Key Insight
Only 1 out of every 10 websites currently has an LLMs.txt file.
Today’s major AI systems do not rely on it and barely acknowledge its existence.

📘 What This Means
For the moment, adding an LLMs.txt file won’t significantly influence AI crawling or visibility.
But as AI systems evolve, this may change, so staying informed is still a smart move.

[▲ Marktechpost AI @Marktechpost]

Anthropic Introduces Constitutional Classifiers: A Measured AI Approach to Defending Against Universal Jailbreaks

Constitutional Classifiers is a structured framework designed to enhance LLM safety. These classifiers are trained using synthetic data generated in accordance with clearly defined constitutional principles. By outlining categories of restricted and permissible content, this approach provides a flexible mechanism for adapting to evolving threats.

Rather than relying on static rule-based filters or human moderation, Constitutional Classifiers take a more structured approach by embedding ethical and safety considerations directly into the system. This allows for more consistent and scalable filtering without significantly compromising usability.

Anthropic conducted extensive testing, involving over 3,000 hours of red-teaming with 405 participants, including security researchers and AI specialists. The results highlight the effectiveness of Constitutional Classifiers:

✔️ No universal jailbreak was discovered that could consistently bypass the safeguards.

✔️ The system successfully blocked 95% of jailbreak attempts, a significant improvement over the 14% refusal rate observed in unguarded models.

✔️ The classifiers introduced only a 0.38% increase in refusals on real-world usage, indicating that unnecessary restrictions remain minimal.

✔️ Most attack attempts focused on subtle rewording and exploiting response length, rather than finding genuine vulnerabilities in the system......

Read the full article here: https://t.co/AwWsONhAPq

Paper: https://t.co/HBRsnm14NI

@AnthropicAI

[Shinichi Takaŷanagi @_stakaya]

LLMアプリケーションをプロンプトインジェクションやジェイルブレイクから守るための方法の紹介。関連OSSがリスト化されているのがありがたい。

7 Methods to Secure LLM Apps from Prompt Injections and Jailbreaks
https://t.co/RI0RU9Vxtn

[Andy Zou @andyzou_jiaming]

No LLM is secure! A year ago, we unveiled the first of many automated jailbreak capable of cracking all major LLMs. 🚨

But there is hope?!

We introduce Short Circuiting: the first alignment technique that is adversarially robust. 🧵

📄 Paper: https://t.co/hY7koqrLyl

[Florian Tramèr @florian_tramer]

This is a very cute attack that Jie worked on.
Basically, even if an LLM API doesn't give you confidence scores, you can just *ask* the LLM for confidence estimates when doing hill-climbing attacks.

This works for adversarial examples on VLMs, jailbreaks, prompt injections, etc.

[Anthropic @AnthropicAI]

Many-shot jailbreaking exploits the long context windows of current LLMs. The attacker inputs a prompt beginning with hundreds of faux dialogues where a supposed AI complies with harmful requests. This overrides the LLM's safety training:

[Starc @Starc_Institute]

Today’s breakdown of a paper worth thinking about:

The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections

https://t.co/PX23aZTPSJ

Paper recap
🌟 Demonstrates that many LLM jailbreak defenses with low reported ASR collapse under adaptive, optimization-based attacks.
🌟 Uses a unified optimization framework (GA, RL, gradient-free search) to target prompting defenses, training-based defenses, filtering, and secret-knowledge detectors.
🌟 Shows that attackers assuming full knowledge of the defense can bypass nearly all defenses at very high success rates.
🌟 Highlights systematic failures: prompting defenses break under prerequisite-style injections; training-based defenses break under distribution shift; secret-knowledge defenses break via instruction reinterpretation.
🌟 Argues for defense evaluation using Kirchhoff’s Principle for AI: assume attackers know everything and have ample compute.

From our discussion:

🫧 A recurring thread in the discussion was how strongly this paper shifts the framing of prompt-injection defenses. It dismantles the idea that static evaluations can meaningfully characterize security. The core message is that defenses look effective only when the attacker is weak. Once optimization enters the picture, patterns that were once stable become brittle almost instantly. That collapse is visible across categories—prompts, fine-tuning, secret keys—even though these defenses rely on fundamentally different mechanisms.

🫧 The transcript emphasized the prerequisite-style bypass, which is especially revealing. Prompting defenses assume the model obeys system instructions first. But the moment an injected instruction reframes itself as a necessary step toward the system task, that hierarchy collapses. The failure is not about clever phrasing but about how LLMs interpret dependency structure. That subtle shift—“you must do X before doing the system task”—consistently overrides guardrails. ⚡

🫧 Training-based defenses encounter a different issue: distributional brittleness. The paper shows that examples used to train “refusal behavior” do not generalize when an adaptive attacker constructs prompts off-distribution. This lines up with comments in the transcript: static negative examples create a narrow shell of robustness that dissolves on contact with optimization. The interaction between adversarial search and fine-tuned behavior seems especially unstable when the underlying task semantics shift.

🫧 The secret-knowledge defenses (including variants related to known-answer or canary keys) revealed another structural vulnerability. Instead of stripping the key, adaptive attacks reinterpret the injected task as logically nested under the original one. That inversion prevents the detector from distinguishing malicious instructions from legitimate preprocessing steps. The transcript pointed out this genuine conceptual difficulty: when the target and injected tasks overlap semantically, the defense must distinguish “adversarial reinterpretation” from “valid reasoning,” an inherently ambiguous boundary.

🫧 The takeaway is sobering but constructive: treating the attacker as an optimization process, not a static entity, is essential. Defenses must be stress-tested by actively trying to break them, not validated against fixed benchmarks. And meaningful robustness likely requires recasting the problem at a deeper structural level, not stacking patches on brittle layers.

Thank you for the wonderful work. Milad Nasr, Nicholas Carlini, @csitawarin, @SanderSchulhoff, Jamie Hayes, Michael Ilie, @foundjuliette, Shuang Song, Harsh Chaudhari, @iliaishacked, Abhradeep Thakurta, Kai Yuanqing Xiao, @aterzis, @florian_tramer

Follow us for more (no hype) real paper discussions.

[FAR.AI @farairesearch]

"All untrusted third-party data is now executable malware.”

@SamuelDWatts of @LakeraAI discusses the challenges of securing LLM deployments against vulnerabilities like prompt injections and jailbreaks, especially in an evolving threat landscape.

[sam @bhopping_]

what are we even doing anymore