Dan Wilhelm

[Dan Wilhelm @danwilhelm]

@_sanjoydas In smaller models where the residual stream contains the solution early (eg training a logistic regression on it gives perfect accuracy), the final MLPs often are combating noise introduced via the skip connections. So I'd suggest removing these if your final MLPs have them.

[Dan Wilhelm @danwilhelm]

This is okay since everyone knows the analogy is flawed (e.g. saying computers are "thinking"). The issue with anthropomorphizing LLMs is that it confuses non-technical people who may falsely believe we've created machines that "feel" or are "conscious" in a way similar to humans.

[Dan Wilhelm @danwilhelm]

@gavinrbrown1 This seems similar to "flag planting". Instead of properly understanding something new, a preprint is rushed out with substandard evidence. Then, incentives are lacking to thoroughly study it; if the results actually are true, credit will largely be given to the flag planter.

[Dan Wilhelm @danwilhelm]

@davidcoffeeboy @abuchanlife @suchenzang Well, a PhD involves learning how to do research in a specific field while becoming an expert in a niche part of it. I assume experts at teaching tend to focus on teaching, not also becoming experts in other niche fields. But certainly one can do research outside universities!

[Dan Wilhelm @danwilhelm]

@davidcoffeeboy @abuchanlife @suchenzang Ideal students enjoy learning and "teaching themselves". The PhD gives them an opportunity to be surrounded by other skilled like-minded people who accelerate that learning and challenge their ideas.

[Dan Wilhelm @danwilhelm]

@xhluca @jxmnop Or the same papers would still be submitted, but the person's name would only be on those most meaningfully contributed to.

[Dan Wilhelm @danwilhelm]

We found a method that works well for eliciting hidden behaviors inserted during finetuning! It detects the behavior in the vast majority of current model organisms tested. Using the method as an investigator agent tool, we achieved a SoTA score on AuditBench. See below for more!

[Mo Abu Baker @m_shahoyi]

🤖New model organisms research 🐁

Most public MOs “leak” their finetuned behavior/beliefs, making them easy to identify by a simple contrastive method: complete random prefills and sort by perplexity difference wrt a reference model.

Top-ranked ones often give it away 🧵👇

[Dan Wilhelm @danwilhelm]

@mach1nelearning @yujink_ As you imply, doing supporting interpretability studies would likely involve first replicating the results on smaller open-weight models. IMO "self-preservation" may not be uniquely represented; it could just be the result of its normal training to be helpful.

[Dan Wilhelm @danwilhelm]

@DenisPaperno Can't this be accomplished without planning? For example, the embeddings themselves may contain rhyming features which are retained (w/ transforms) throughout. Info about the prior tokens is always copied into each newline. Then to rhyme, attention looks back at the last newline.

[Dan Wilhelm @danwilhelm]

@alexcdot @Dorialexander Not excusing it, but citations such as the following could be a legitimate placeholder. The year either due to misremembering or a preprint:

"Firstname Lastname and Others. Drivlme: A large-scale multi-agent driving benchmark, 2023. URL or arXiv ID to be updated."

[Dan Wilhelm @danwilhelm]

@GergelyOrosz Not only this, but many mobile games are intended to be mindless where you advance through thousands of similar-difficulty levels. Games seem most beneficial to me if you actively play them and get better, but these actively discourage that since there is minimal challenge.

[Dan Wilhelm @danwilhelm]

@TheAndersPaul @NPRinskeep False. Correlation does not imply causation, even in part. Putting sunglasses on someone will not cause them to buy ice cream. Even though sunglasses are associated with ice cream purchases (due to the sun/heat).

[Dan Wilhelm @danwilhelm]

@adamscochran The playbook always seems to be (1) spam random anti-left theories connected tangentially to the facts, then (2) amplify those that take off.

[Dan Wilhelm @danwilhelm]

@deedydas This paper has similar conclusions - LLMs were completely unable to solve Gold and Platinum-level USACO problems: https://t.co/XbbHcyQvMl

I also contributed the eval to Inspect Evals, so can easily run it on recent LLMs (307 problems): https://t.co/1xi8Q2XkVQ

[Dan Wilhelm @danwilhelm]

@Michael_Druggan @icmpressure Its founder made one, but unfortunately it didn't resonate and shut down in 2022: https://t.co/wKLhK182CD

[Dan Wilhelm @danwilhelm]

@panickssery @littmath @skdh Private funding tends to prioritize research that will have potential impact short-term. Unfortunately long term, risky, and theoretical research (with potentially huge reward) is difficult to justify commercially.

[Dan Wilhelm @danwilhelm]

@americansulla @scottastevenson Except that people don't learn those things deeply to be cool ... more likely they're passionate about the field. Deep knowledge is in short supply, and obtaining it (regardless of motivation) is a huge win that will pay off in spades.

[Dan Wilhelm @danwilhelm]

@ben_golub If they didn't mention hiring an editor and sent me detailed suggestions, I would be appreciative that they took such an interest. The weirdness would be if there was some expectation involved with them spending money on an editor.

[Dan Wilhelm @danwilhelm]

@JayBurris633948 @littmath @OGVlonaldTrutin Historically, research that seemed useless at the time became incredibly useful and led to breakthroughs in nearly every aspect of your life. For example, what was originally "pure math" forms the backbone of the internet, mobile phones, and online commerce.