Genevieve Conaty

@genbynight

Music for mavericks. San Francisco Bay Area - going out worldwide! My debut album, Your Own Brand of Candy, is just out.

Joined April 2011

1.7K Following

502 Followers

1.1K Posts

genbynight retweeted

Reza Jamei @RezaJamei46330

4 months ago

How do we teach LLMs to generate original scientific thought? One missing ingredient may be simulating the internal state of a scientist who doesn't yet know the answer. A toy version of the idea is studied in detail in the paper linked in the first comment. Here is a high-level summary: When humans learn something — a skill, a fact, a concept — they generate an internal thinking corpus that leads them to accept it, along with preference data for which thought patterns were more or less helpful. Benjamin Franklin famously practiced this deliberately: he'd read an essay, set it aside until he forgot the exact wording, reconstruct it from the ideas alone, then compare the gap. In contrast, we typically don't train LLMs this way. The vast majority of LLMs' training data is language meant for communication, not internal thinking. Research shows (o1, STaR) massive benefits from limited versions of adding preference data over reasoning to the training pipeline of models after training (doable for mechanically verifiable answers). Still, LLMs have a huge repository of knowledge and even problem-solving skills that they're trained on without training on the process that discovered those methods and facts. It's of course not easy to generate genuine reasoning data that could lead to discovery using a model that already knows the answer. Prompting a model to pretend it doesn't know something leaves its world model intact — it's role-play, not genuine uncertainty. I've experimented with creating lesions — deliberately inducing a temporary misconception — in an LLM's weights so that it answers questions about one issue poorly, then letting it generate many thinking paths around the issue and grading those traces based on whether they make progress toward recovering the un-lesioned knowledge. For example: fine-tune a model to believe organisms can deliberately rewrite their own DNA to adapt (Lamarckian evolution), then prompt it with contradictory evidence like the Luria-Delbrück experiment, and grade whether it notices the tension. The goal is to simulate the kinds of situations we care about: cases where a model does not know a method or holds inconsistent beliefs. Early results in a small-scale study are promising: 1. training a clean model on thinking preference data generated from 55 training lesions increased preference toward recovery on held-out pairs from 56 different lesions in 76-80% of the test cases. 2. when we lesion both the base model and the preference-trained model on held-out misconceptions and compare their greedy-decoded responses, the trained model scores about 1.2-1.4 points higher on a 1-10 GPT-4 judge scale. The primary metric here is transfer on held-out lesion identities, while the generation result is supplementary evidence under lesion stress. The cleaner test — whether the clean trained model produces better reasoning without being lesioned — remains future work. Neither metric is ideal and may not translate directly to realistic generative tasks, but I think this is about as much as an experiment at this scale can show. Given the scalability of the experiment, I think this method is a good candidate to produce a meaningful improvement in the quality of thinking in LLMs, especially in the area of generating original scientific thought.