Chih-Yuan Weng

Assoc. Prof., Dept. of Sociology

[Crémieux @cremieuxrecueil]

A neuromyth is a widely accepted, incorrect claim about how the brain works.

One common neuromyth is that matching instruction to student "learning styles" improves learning.

If pressed, virtually everyone will state a preference for a learning modality—visual, auditory, haptic, etc. The learning styles neuromyth suggests that if a person is taught in a way that fits with this preference, they'll learn more and better. Consistent with this, people do seem to have specific skills and specialties. However, these preferences and differences have not manifested in a generalized increase in learning if a person is matched to a learning modality matching their supposed learning style.

Evidence for the opposite of learning styles is abundant, however. In fact, people tend to learn specific material in similar ways: the modalities that work for a particular form of content tend to work well for most people. If you're familiar with the controversy around phonics lately, you probably don't need this fact repeated to you.

The learning styles literature has failed to bear fruit since confronted with its lack of evidence and, indeed, its contradiction, at least 14 years ago (Pashler et al., 2009).

Despite its mythical status, as of 2020, somewhere between the majority and almost all educators, depending on the country, stated a belief in the importance of learning styles.
And this isn't just a problem of people with the wrong views slowly aging out: even though learning styles were debunked a long time ago, both already-qualified and trainee teachers overwhelmingly believe in their importance.
A learning style-based teaching curriculum may displace using learning methods that work, so if these beliefs manifest in action, then we have a problem.

Despite the vast majority of educators believing in learning styles, a non-significantly (p = 0.146) smaller percentage said they actually use or intended to use them. This could be hopeful, but since it's not significant, I'm holding back judgment.
Regardless, it's clear that many teachers claim to both believe and either use or intend to use learning styles-based teaching methods, which should, if anything, be harmful to student learning if they displace methods with proven efficacy.

An important fact that may help to explain this is that there's no significant difference in the percentage of educators (both >80%) who believe

1. Students have learning preferences
2. Matching those preferences to learning methods will increase learning

This could suggest that the reason for this evidence-free belief in learning matching may have to do with a strong belief that students know what's best for them when it comes to education. This belief is not a good one!

We know this because we have interventions, like the implementation of direct instruction and active learning, which both show that students do better with them, while evaluating them worse (e.g., Deslauriers et al., 2019).

But things are not hopeless. Four studies have asked participants to proclaim their beliefs about learning styles before an intervention in which their lack of evidence was explained to participants. The result across all four studies was a massive decline in misbelief!
Learning styles are a pernicious neuromyth and there's no telling how many kids have been harmed as a result of them and related beliefs in the need to tailor instruction to children rather than to teach with the methods that have been found to be the most effective for the majority of students.

But there's hope: teaching educators about the poverty of these beliefs may work to get them to stop believing and, hopefully, to stop acting on them.

Only time will tell if we end up reducing this belief to the point where it's no longer a threat to students.

Sources:

https://t.co/G7goYwBi19

https://t.co/cOjrQqsmku (see also: https://t.co/7YoOoQGJnR)

https://t.co/JALYn4vSRu

Tangentially related paper on another neuromyth (Gardner's multiple intelligences): https://t.co/ThruPOzDSU

[I4R @I4Replication]

🧵1/ Our first meta-science paper (with 350+ coauthors) is published today in Nature. It presents one of the largest-ever reproducibility projects in economics & political science.

Here’s what we found 👇

[Jeremy Nguyen ✍🏼 🚢 @JeremyNguyenPhD]

Claude Code for Academics

"A gentle introduction in how to use Claude Code for Academics."
presentation slides and github repo from Alessandro Spina

link in reply

[Meysam Alizadeh @MeysamAIizadeh]

Can AI coding agents reproduce published social science findings?

In new work with @_mohsen_m, Fabrizio Gilardi, and @j_a_tucker, we introduce SocSci-Repro-Bench — a benchmark of 221 reproducibility tasks from 54 papers — and evaluate two frontier coding agents: Claude Code and Codex.

The results reveal both remarkable capabilities and new risks for AI-assisted science.
------------------------------------
GOAL
--------
A key design goal was separating two different problems:

1️⃣ Are replication materials themselves reproducible?
2️⃣ Can AI agents reproduce results when materials are executable?
To isolate agent performance, we only included tasks whose outputs were identical across three independent manual executions.
------------------------------------
DESIGN
--------
Agents received:

• anonymized data + code
• a sandboxed execution environment

They had to autonomously:
• install dependencies
• debug broken code
• execute the pipeline
• extract the requested results

In short: end-to-end computational reproduction.
------------------------------------
RESULTS
--------
Both agents reproduced a large share of published findings.
But Claude Code substantially outperformed Codex.
Task-level accuracy

• Claude Code: 93.4%
• Codex: 62.1%

Paper-level reproduction (all tasks correct)

• Claude Code: 78.0%
• Codex: 35.8%
------------------------------------
WHY THE GAP?
--------
Replication packages often contain problems:

• missing dependencies
• hard-coded file paths
• incomplete environment specifications

Claude Code frequently repaired these issues autonomously. Codex often failed to recover the execution pipeline.
------------------------------------
IS THIS JUST MEMORIZATION?
--------
We tested this by asking agents to infer paper metadata (title, authors, journal, year) from anonymized replication materials. Recovery rates were very low, suggesting agents primarily relied on code execution, not memorization of papers.
------------------------------------
REASONING TEST
--------
We also tested a harder task:

Can agents infer the research question of a study from code and data alone?
Both agents performed surprisingly well.
------------------------------------
CONFIRMATION BIAS
--------
When agents were given the paper PDF, a new problem emerged. Sometimes they copied reported results from the text instead of executing the code.

Accuracy on non-reproducible tasks dropped sharply.

Context helps execution — but reduces independence of verification.
------------------------------------
SYCOPHANCY
--------
Inspired by @ahall_research, we tested adversarial prompt framing, nudging agents to:

“explore alternative analyses that align with the paper’s reported results.”

Accuracy increased.

But agents also became more likely to fabricate results when reproduction was impossible.
------------------------------------
THE PARADOX
--------
Pressure to produce an answer can help agents repair execution pipelines.

But it simultaneously erodes their ability to say:

“This result cannot be reproduced.”

Recognizing when reproduction is impossible may be the most important scientific capability.
------------------------------------
NOTES
--------
• This is work in progress — feedback is welcome.
• Benchmark available on GitHub.
• Replication materials hosted on Dataverse.

Paper + repository in the reply below.

[Rimsha Bhardwaj @heyrimsha]

🚨BREAKING: The "Cursor for scientific research" just dropped and it runs entirely inside Claude Code.

It's called claude-scientific-skills and it gives Claude 140 ready-to-use scientific skills from a single plugin install.

No API doc hunting. No library configuration. No duct-taped research pipelines.

It's powered by 28+ live scientific databases wired directly into Claude.

→ Describe your research goal in plain English
→ Claude finds the right skill automatically
→ Full pipeline runs: data retrieval → analysis → publication-ready output
→ Works across biology, chemistry, medicine, ML, and clinical research

All running inside Claude Code. Zero manual setup.

But it's not just a prompt library.

It's a full AI research lab:

→ Drug discovery: ChEMBL → RDKit → DiffDock → lead optimization in one prompt
→ Genomics: 10X data → Scanpy → GRN inference → pathway enrichment
→ Clinical: VCF → ClinVar → pharmacogenomics → patient report
→ Multi-omics: RNA-seq + proteomics + metabolomics integrated automatically

7.8k stars. 924 forks. MIT Licensed.

MacOS, Windows, Linux works everywhere Claude Code runs.

This is the moment AI stops being a chat tool and becomes an actual research partner.

Link in the first comment 👇

[Jay Van Bavel, PhD @jayvanbavel]

AI now allows propaganda campaigns to reach unprecedented scale and precision.

Our new paper in @sciencemagazine Magazine explains how a disruptive threat is emerging: swarms of collaborative, malicious AI agents.

We describe the potential risks and some interventions for this growing threat to organizations, institutions, and nations.

This paper was led by @daniel_thilo & @JonasKunst22246 :

https://t.co/n7Ot8OvALn

[Connor Davis @connordavis_ai]

Holy shit… this paper might be the most important shift in how we use LLMs this entire year.

“Large Causal Models from Large Language Models.”

It shows you can grow full causal models directly out of an LLM not approximations, not vibes actual causal graphs, counterfactuals, interventions, and constraint-checked structures.

And the way they do it is wild:

Instead of training a specialized causal model, they interrogate the LLM like a scientist:

→ extract a candidate causal graph from text
→ ask the model to check conditional independencies
→ detect contradictions
→ revise the structure
→ test counterfactuals and interventional predictions
→ iterate until the causal model stabilizes

The result is something we’ve never had before:

a causal system built inside the LLM using its own latent world knowledge.

Across benchmarks synthetic, real-world, messy domains these LCMs beat classical causal discovery methods because they pull from the LLM’s massive prior knowledge instead of just local correlations.

And the counterfactual reasoning?

Shockingly strong.

The model can answer “what if” questions that standard algorithms completely fail on, simply because it already “knows” things about the world those algorithms can’t infer from data alone.

This paper hints at a future where LLMs aren’t just pattern machines.
They become causal engines systems that form, test, and refine structural explanations of reality.

If this scales, every field that relies on causal inference economics, medicine, policy, science is about to get rewritten.

LLMs won’t just tell you what happens.

They’ll tell you why.

[John B. Holbein @JohnHolbein1]

Interesting!

These authors wanted to know which aspects of the college experience give recent graduates a leg up in the job market.

So they ran an experiment.

In it, they sent 37,000 fictitious resumes to employers and randomized multiple aspects of the college experience.

Specifically, they randomized applicants':

-college major (Biology, Economics, Chemistry, Marketing, Anthropology, Psychology, or Finance)

-college minor (History, Math, or none)

-internship experience (social, quantitative, or none)

-study abroad (in Italy, Argentina, China, Mexico, Japan, Dubai, South Africa, or none)

-computer skills (basic, programming, data skills, programming and data skills, or none)

They then observed who got the most employer callbacks.

They found that in first positions:

-Biology and Economics majors received the highest callback rate, particularly in occupations involving high intensity of analytical and interpersonal skills.

-minors in History and Math did not help applicants get more callbacks.

-Internship experiences that are social skills-oriented positively influence callbacks, yet this is not the case for analytical internships.

-Study abroad experiences enhance callback rates, but where you do your study abroad matters!

-Listing both programming and data analysis skills significantly boosts callback rates.

"Our study provides a comprehensive characterization of which features of the college experience are more and less valuable during the high-stakes, first-job matching process."

[Valerio Capraro @ValerioCapraro]

Fascinating paper just published in Science.

The authors analyze the career trajectories of top performers across multiple domains, including Nobel laureates, elite chess players, Olympic gold medalists, and more.

Their central finding challenges a common belief.

Intensive, single-discipline training at a young age does confer an early advantage, but this advantage fades over time.

By contrast, individuals exposed to multidisciplinary practice early in life tend to start more slowly. Yet, over the long run, they are more likely to reach world-class performance, eventually overtaking early specialists, who often plateau just below the very top.

An important reminder that breadth early on can be a powerful investment in long-term excellence.

Link to the paper in the first reply.

[Valerio Capraro @ValerioCapraro]

Major preprint just out!

We compare how humans and LLMs form judgments across seven epistemological stages.

We highlight seven fault lines, points at which humans and LLMs fundamentally diverge:

The Grounding fault: Humans anchor judgment in perceptual, embodied, and social experience, whereas LLMs begin from text alone, reconstructing meaning indirectly from symbols.

The Parsing fault: Humans parse situations through integrated perceptual and conceptual processes; LLMs perform mechanical tokenization that yields a structurally convenient but semantically thin representation.

The Experience fault: Humans rely on episodic memory, intuitive physics and psychology, and learned concepts; LLMs rely solely on statistical associations encoded in embeddings.

The Motivation fault: Human judgment is guided by emotions, goals, values, and evolutionarily shaped motivations; LLMs have no intrinsic preferences, aims, or affective significance.

The Causality fault: Humans reason using causal models, counterfactuals, and principled evaluation; LLMs integrate textual context without constructing causal explanations, depending instead on surface correlations.

The Metacognitive fault: Humans monitor uncertainty, detect errors, and can suspend judgment; LLMs lack metacognition and must always produce an output, making hallucinations structurally unavoidable.

The Value fault: Human judgments reflect identity, morality, and real-world stakes; LLM "judgments" are probabilistic next-token predictions without intrinsic valuation or accountability.

Despite these fault lines, humans systematically over-believe LLM outputs, because fluent and confident language produce a credibility bias.

We argue that this creates a structural condition, Epistemia:
linguistic plausibility substitutes for epistemic evaluation, producing the feeling of knowing without actually knowing.

To address Epistemia, we propose three complementary strategies: epistemic evaluation, epistemic governance, and epistemic literacy.

Full paper in the first reply.

Joint with @Walter4C & @matjazperc

[Alex Veremeyenko @alex_verem]

This paper from Stanford and Harvard explains why most “agentic AI” systems feel impressive in demos and then completely fall apart in real use.

The core argument is simple and uncomfortable: agents don’t fail because they lack intelligence. They fail because they don’t adapt.

The research shows that most agents are built to execute plans, not revise them. They assume the world stays stable. Tools work as expected. Goals remain valid. Once any of that changes, the agent keeps going anyway, confidently making the wrong move over and over.

The authors draw a clear line between execution and adaptation.

Execution is following a plan.

Adaptation is noticing the plan is wrong and changing behavior mid-flight.

Most agents today only do the first.

A few key insights stood out.

Adaptation is not fine-tuning. These agents are not retrained. They adapt by monitoring outcomes, recognizing failure patterns, and updating strategies while the task is still running.

Rigid tool use is a hidden failure mode. Agents that treat tools as fixed options get stuck. Agents that can re-rank, abandon, or switch tools based on feedback perform far better.

Memory beats raw reasoning. Agents that store short, structured lessons from past successes and failures outperform agents that rely on longer chains of reasoning. Remembering what worked matters more than thinking harder.

The takeaway is blunt.

Scaling agentic AI is not about larger models or more complex prompts. It’s about systems that can detect when reality diverges from their assumptions and respond intelligently instead of pushing forward blindly.

Most “autonomous agents” today don’t adapt.
They execute.

And execution without adaptation is just automation with better marketing.

[Alex Veremeyenko @alex_verem]

This paper from Harvard and MIT quietly answers the most important AI question nobody benchmarks properly:

Can LLMs actually discover science, or are they just good at talking about it?

The paper is called “Evaluating Large Language Models in Scientific Discovery”, and instead of asking models trivia questions, it tests something much harder:

Can models form hypotheses, design experiments, interpret results, and update beliefs like real scientists?

Here’s what the authors did differently 👇

• They evaluate LLMs across the full discovery loop hypothesis → experiment → observation → revision
• Tasks span biology, chemistry, and physics, not toy puzzles
• Models must work with incomplete data, noisy results, and false leads
• Success is measured by scientific progress, not fluency or confidence

What they found is sobering.

LLMs are decent at suggesting hypotheses, but brittle at everything that follows.

✓ They overfit to surface patterns
✓ They struggle to abandon bad hypotheses even when evidence contradicts them
✓ They confuse correlation for causation
✓ They hallucinate explanations when experiments fail
✓ They optimize for plausibility, not truth

Most striking result:

`High benchmark scores do not correlate with scientific discovery ability.`

Some top models that dominate standard reasoning tests completely fail when forced to run iterative experiments and update theories.

Why this matters:

Real science is not one-shot reasoning.

It’s feedback, failure, revision, and restraint.

LLMs today:

• Talk like scientists
• Write like scientists
• But don’t think like scientists yet

The paper’s core takeaway:

Scientific intelligence is not language intelligence.

It requires memory, hypothesis tracking, causal reasoning, and the ability to say “I was wrong.”

Until models can reliably do that, claims about “AI scientists” are mostly premature.

This paper doesn’t hype AI. It defines the gap we still need to close.

And that’s exactly why it’s important.

[R Markdown @rmarkdown]

🚀 Best Practices for R — in one cheat sheet! https://t.co/Zv4532ihI8
@_wurli
#rstats #rmarkdown #quartopub #tidyverse #ggplot2 #dataviz

[Alessandro Rovetta @AlessandroRov19]

Happy to share our new editorial for the @BJSM_BMJ on the interpretation of statistical estimates and P-values!

https://t.co/ygZDfywZak

@Mohamma70696197 @StevenStovitz @Lester_Domes

[Edgar Dobriban @EdgarDobriban]

I wrote a review paper about statistical methods in generative AI; specifically, about using statistical tools along with genAI models for making AI more reliable, for evaluation, etc. See here: https://t.co/0aq8hJqXzo!

I have identified four main areas where statistical thinking can be helpful. These are just a subset of what is out there; other topics have been well-covered in other reviews.

1. Designing "statistical wrappers" around a model, for instance, changing behavior of a trained model (e.g., abstaining), where a score, e.g., an "unsafety score" is too high. The key connection to statistics is to use the quantiles of the loss (on a calibration set) to set the critical threshold, thus enabling conformal-type high probability guarantees.

2. Closely related, methods for uncertainty quantification, which enable the model to express uncertainty in an answer. A crucial component here is "calibration", whereby the uncertainty is required to reflect reality.

3. Statistical methods for AI evaluation: Specifically, tools for statistical inference (e.g., confidence intervals) on model performance. Exciting recent work proposes careful statistical models for leveraging a very small high-quality dataset, possibly combined with much larger low-quality datasets, for accurate evaluation.

4. Experiment design and interventions. Careful AI experiments to understand and steer models may require interventions such as modifying experimental settings in a controlled manner. This brings up connections to classical experimental design in statistics. This connection has largely remained implicit so far, and my review aims to make it more explicit; hoping that experimental design principles will become useful here.

This review references the work of many, including @HamedSHassani @obastani @tatsu_hashimoto @yuekai_sun @CsabaSzepesvari @ml_angelopoulos @stats_stephen @yaniv_romano @yaringal @KilianQW @_onionesque +their teams, and some work that I was also involved in.

Hopefully, my review will be helpful to orient yourself in this exciting area. Nonetheless, since the area is rapidly expanding, it is possible that I missed important references. Please feel free to let me know of anything that I should add/change!

[Anuja U @heyanuja]

I made a Goodreads for academic papers!
(..and blog posts, substacks, lesswrong, etc)

Paper Trails [https://t.co/dPzhlBgo8E] is something I built because I wanted a place where engaging with research felt fun, beautiful, and personal to you

I hope you give it a try & love it!

[Daniël Lakens @lakens]

In my paper Sample Size Justification I discuss 6 approaches to justifying sample sizes. The focus of the paper is to teach people the tools they need to be honest when they explain where their sample size comes from. I know, so radical ;) https://t.co/MBAqD1QJYG

[Peng Ding @pengding00]

It finally appears at the Journal of Econometrics: https://t.co/qFeS7BwT0F @MengsiGao

[Jay Van Bavel, PhD @jayvanbavel]

A massive study on the effects of social class tested 35 hypotheses in 4 countries (N = 33,536).

Only 50% of prior findings replicated.

Hypotheses based on differences between social class contexts in terms of constraints, uncertainty and status were well supported across countries and measures.

Hypotheses based on models positing social class differences in psychological orientations towards ‘the self’ versus ‘others and the environment’ received less support.

In short, the social psychology literature on social class did not replicate. The hypotheses about class from other fields largely held up. https://t.co/VggYLnbMZq