Surdas Mohit

Scientist, public servant working on digital governance at GoC. Master of the dad joke

[Gathoni Ireri @gathoni_ireri]

We have a new paper out! We assess the case for Africa-centric AI safety evaluations. I'm pretty excited about it, particularly because it tries to handle questions I've also had about the practicalities of conducting evaluations.
https://t.co/QEFQqKy43L

[Andrej Karpathy @karpathy]

Don't think of LLMs as entities but as simulators. For example, when exploring a topic, don't ask:

"What do you think about xyz"?

There is no "you". Next time try:

"What would be a good group of people to explore xyz? What would they say?"

The LLM can channel/simulate many perspectives but it hasn't "thought about" xyz for a while and over time and formed its own opinions in the way we're used to. If you force it via the use of "you", it will give you something by adopting a personality embedding vector implied by the statistics of its finetuning data and then simulate that. It's fine to do, but there is a lot less mystique to it than I find people naively attribute to "asking an AI".

[Alouettes de Montréal @MTLAlouettes]

Fiers de vous. On sera de retour. We will be back.

#Alouettes #CoupeGrey #GreyCup @BFLCANADA

[Nora Ammann @AmmannNora]

Very excited to see this come out, and to be able to support!

Beyond the funding itself, the RfP itself is a valuable resource & great effort by the @AISecurityInst team! It shows there is a lot of valuable, scientifically rigorous work to be done.

[Lisa Soder @lisa_soder_]

2 days to go! Abstracts for @taig_icml due 7 May, 23:59 AOE! 🚀
Already have a Technical AI Governance paper accepted at #ICML2025, #ICLR2025, or a similar venue?  Request a fast‑track review for quicker turnaround.

[Subbarao Kambhampati (కంభంపాటి సుబ్బారావు) @rao2z]

No, Virginia, those Rorschach musings by LRMs may not be "reasoning traces" in any meaningful way.. #SundayHarangue

I have been saying for sometime now that the intermediate tokens/"mumblings" that LRMs tell themselves are not to be seen as "reasoning traces" (c.f. https://t.co/jl0LyWJUys).

But they look like they are, no? The fact that sometimes these intermediate tokens look like reasoning traces--with human like "hmm..", "aha..", "let me think step by step..", "let me consider a different way.." mumbles--is mostly an anthropomorphization trap/tendency. After all the LLM prompt completions are able to imitate the style of everything under the sky--why would you think they won't also imitate the style of any human reasoning musings that are present in the pre-training data?

How can we be sure they aren't sound? One reason the reasoning trace anothropomorphization continues unabated is because it is hard to either prove or disprove that the so-called reasoning traces are correct/incorrect. These "traces" can get upwards for 30 pages long of free form self-chatter even for small problems--and there is no clear method to check if they are correct. Approaches like Process Reward Models try to make the reasoning traces a bit more locally consistent--but they seem to have taken a back seat anyways especially given the success of DeepSeek R1. (To be clear, it is also hard to formally evaluate people mumbling to themselves while they are solving a problem).

It is no wonder then that none of the benchmark evals on LRMs actually evaluate the reasoning traces--and focus on only evaluating the correctness of the final answers (c.f. https://t.co/PQqS95qNRp).

..formal derivational traces come to the rescue! Thankfully, there do exist cases where the LLMs are trained to mutter to themselves in specific formats that are supposed to correspond to a formal solver solving that procedure. Two examples are "SearchFormer" (https://t.co/Bnj2NyD71A) and "Stream of Search" (https://t.co/fahALSNsi2). SearchFormer, for example, gets the derivational traces for solving various search problems from the open/closed list manipulations of an A* search algorithm. These are then used to train (SFT) the base LLM--so that at the inference stage, the LLM will imitate open/closed list manipulations before actually giving its (guess of) the solution.

Although SearchFormer paper only considers the solution accuracy in the evaluation phase, we can actually check if the traces output actually correspond to valid and sound A* search traces (essentially by having A* simulate that trace..).

Some concrete results: So we (okay, really, @PalodVardh12428, @kayastechly and @karthikv792) did this. The graphic below shows the summary of the results for their grid world problems. Let us ignore the many cases (shown in red) where the trace turns out to be syntactically wrong. Even when the traces are syntactically valid (shown in blue), we note that in 12% of the cases the traces are semantically invalid (as simulated by A* search) even though the solution guess happens to be correct. The semantic invalidities include things such as trying to remove nodes from open when they are not actually on the open list. (Here is a version of the graph with clickable stats: https://t.co/C3JLKpJ5xm thanks to Atharva).

While we have suspected this would be the case die. while now, (see: https://t.co/vCTsp5StE3), these stats give direct evidence.

If the mumbling makes people happy, why care that aren't quite reasoning traces? The usual argument in favor of intermediate tokens that sorta/kinda look like human mumblings is that it makes people happy! Indeed, even though R1-Zero was doing pretty good with RL post-training, DeepSeek folks felt the compulsion to make it chatter to itself in a sorta/kinda human way by getting additional human annotated derivational trace data so R1 can imitate those human-like traces (never mind that there was no evaluation of the correctness of the traces beyond the "look, it said aha to itself!!").

I think making reasoning traces that sorta/kinda look human-ish, knowing that they have no semantic import and we don't plan to evaluate them anyways--is deliberately leveraging the cognitive flaws of the end users (c.f. https://t.co/SjPmgpprW5).

Given that the supposed use of LRMs is solving problems that the end users themselves don't know (or even may be able to verify), increasing the false confidence with these types of ersatz reasoning traces seems like a trap!

There are some indications that this trap can catch the researchers too. In the SearchFormer paper, there are claims about how LLM trained on A* derivational traces is "more optimal than A*" because the imitative derivational trace it outputs is sometimes shorter than that output by A* for the same problem (nevermind that these traces may not actually be sound in any A* sense); https://t.co/erOf16pVGj.

Once we agree that it maybe questionable to go with anthropomorphized mumbling to make them be "more acceptable" for humans, we might also note that sticking to the anthropomorphic format may be an unnecessary albatross from the "post-hoc solution accuracy" point of view. The RL can train LLMs to emit any old intermediate tokens as long as the bottom line improves (viz. https://t.co/JeZ8L0ryia)

[African School of Governance (ASG) @ASG_Africa]

The Grand Launch of the African School of Governance is live now!

Based in Kigali, Rwanda, the #ASG4Africa is a solution to Africa’s governance issues through ethical leadership development, evidence-based policymaking & more.

Follow live: https://t.co/j8Gu8epe68!

#ASGLaunch

[Blake Arsenault @blakesarsenault]

Ottawa right now

[Jiaqi Ma @Jiaqi_Ma_]

@Yoshua_Bengio @PeterHndrsn @dawnsongtweets @RishiBommasani @NIST Finally, panel discussion on AI Regulation and Governance -- @ghadfield, Wan Sie Lee (@IMDAsg), @MohitSurdas, Sella Nevo (@RANDCorporation), @robreich, and @BasicScienceSav

[Cohere Labs @Cohere_Labs]

Last night was one to remember at the @neurIPSconf @Cohere @CohereForAI party 2024! Thank you everyone who came out to celebrate and connect with us! 🪩

[CoreWeave @CoreWeave]

We're excited to partner with @Cohere to build a new AI data center in Canada. Congratulations to the team at Cohere and to the Canadian government for bringing this exciting project to life! Read more details in the @Bloomberg article below:
https://t.co/Pm1WFCG1J5

[Sasha Luccioni, PhD 🦋🌎✨🤗 @SashaMTL]

Honestly, this is on par with a lot of the recent reviews I've been getting 😅

[Amarnath Amarasingam @AmarAmarasingam]

If we’re buying donuts and you say “just whatever makes sense”, we having a donut ordering debrief when we get back in the car.

[Noah Giansiracusa @ProfNoahGian]

How am I just now reading this article?! IT IS BRILLIANT!! So insightful and articulate—cannot recommend it highly enough! @ShannonVallor

[Marta Cantero @MartaCantero]

Who is regulating AI?

Our paper (w/@ChrisTMarsden) on AI co-regulation has just been published: https://t.co/Pb9hrIjDgK

We explore the role of standards in the AI Act, democratic legitimacy, and the need for inclusive approaches to AI regulation

[Jordan Howlett @J_the_stallion8]

Welp @donaldglover

[Daniel Litt @littmath]

gpt-6 will have the intelligence of someone who decided not to get a PhD

[Suresh Venkatasubramanian (mostly in the sky now) @geomblog]

Catastrophic risk.

[Dave Jorgenson @davejorgenson]

the surgeon general wants to put a warning on social media