daios

Moral choice in the hands of individuals.

Building a personalized and controllable values layer for machines.

[Megan Anne Agathon @agathomai]

@cosmos_inst @andrew_roci technical log: https://t.co/RBrucmeuuk

[Megan Anne Agathon @agathomai]

Can we post-train a machine with virtue? Both RLHF (consequentialist) and Constitutional (deontological) are limited. I think there's a third path: virtue training.

I received a @cosmos_inst grant to investigate with @andrew_roci, starting with sycophancy, the vice Aristotle attributes to the kolax.

Benchmark + adapters open-source. Lab notes linked below.

[Megan Anne Agathon @agathomai]

an article I wrote on how consuming ai generated content changes our perception of reality, making outliers imperceptible, and accelerating our culture towards the mediocre. enjoy ❤️

[daios @daiostech]

https://t.co/7XBmGkq2H4

[daios @daiostech]

More information on our views on open-source, negative values, AI policy, and the role of the non-aggression principle (NAP) in the daios method.

Link below.

[François Chollet @fchollet]

One thing that even relatively senior ML people often fail to grasp is that deep learning models are curves fitted to a data distribution. You cannot expect them to solve tasks outside of their training distribution (which is the sort of thing that you need intelligence for).

"Emergent learning" is an incorrect label -- if a model demonstrates performance on task A that it wasn't trained on, that simply means that there is significant overlap between A and all the data that you did train on. Competence doesn't magically emerge out of nowhere.

[Megan Anne Agathon @agathomai]

what if I told you that "the crowd is untruth"?

source: how anthropic created the HH (helpfulness and harmlessness) dataset, the cutting-edge of datasets used to align models with human values

[Percy Liang @percyliang]

model = learn(data)

Synthetic data is great, but it’s not data. It’s an intermediate quantity created by learn(). Data is created by people and has privacy and copyright considerations. Synthetic “data” does not - it’s internal to learn().

[Megan Anne Agathon @agathomai]

the approach towards definitions in computer science vs philosophy is interesting:

AI ethics academics used to complain that since we can't agree on the definition of ethics, we cannot progress further

the silence around this topic has been definitive since chatGPT was released

[Marc Andreessen 🇺🇸 @pmarca]

Often valid, but my experience is that people prioritize their ideological priors over their economic self interest an awful lot of the time.

[Alexander Doria @Dorialexander]

Since I’m not sure we realize how much LLM discourse is just training data discourse ⬇️

[daios @daiostech]

Turns out some startups are already working on this problem ;)

[gfodor.id @gfodor]

This is a profound insight.

I never considered it. It turns out, people have different values. And so aligning AI must be impossible because there are no universal values.

How has nobody ever made this point before?

[Paul Graham @paulg]

Oddly enough, this exercise suggests a way to solve the otherwise possibly intractable problem of what an AI's politics should be. Let the user choose what they want the reference group to be, and they can pick Oberlin undergrads or Freedom Caucus or whatever.

[Moritz Bierling @bierlingm]

We’re in the fast take off phase now.

Trusted, meaningful data becoming ever more important.

[andi (twocents.com) @Nexuist]

interesting post from an OpenAI employee claiming that all large language models reach the same endpoint regardless of training strategy or clever tricks

this is of course what the bitter lesson teaches us but useful to get an up to date confirmation that it still holds true

[daios @daiostech]

https://t.co/0hb3i3euT2

[Nat Friedman @natfriedman]

Enjoying the alignment memes from @anthrupad and others recently. Perfectly timed for the release of Sydney!

[daios @daiostech]

Halloween weekend throwback:
Andrew as RLHF 🙂

[daios @daiostech]

jailbreaking via "shadow alignment": https://t.co/9Zq7TbY3Nn

[daios @daiostech]

fine-tuning can override original safety mechanisms built into LLMs

original safety mechanisms prevent harmful behavior but often cauterize the model

what's better?

papers below

[Interconnects @interconnectsai]

Undoing RLHF and the brittleness of safe LLMs
Recent papers show most of the arguments about needing "safety" in releases of open LLM weights are nearly dead in the water. Yes, still release the parameters.
Read here: https://t.co/D8sFq6X9qG

[daios @daiostech]

fine-tuning compromising safety: https://t.co/9RqITlQH61