@sandersted@MariusHobbhahn@MatthewJBar@soundboy I agree that improving the model's capabilities in some intended manner is hard, but improving model capabilities by any means necessary is not similarly difficult. I think that gap in difficulty illustrates why alignment is difficult.
@sandersted@MariusHobbhahn@MatthewJBar@soundboy If the model can exploit differences between your operationalisation and your intention in order to improve score, then it will, and those behaviours are among those that we consider misaligned. This is one reason among many as to why it's harder.
@sandersted@MariusHobbhahn@MatthewJBar@soundboy Now suppose you want to improve alignment. You try to operationalise your intention as an RL environment. Success rate climbs in the same way. Is the model more aligned? Given enough optimisation pressure, the answer is no.
@AndyMasley I think people have been taught that claims aren't scientific (or trustworthy) if they aren't the product of Popper-style scientific method. By that standard, none of the claims they'd associate with science are scientific.
@jon_stokes The idea is that cycles increase the likelihood of the agent ending up in dispreferred outcomes. The agent prefers to end up in preferred outcomes, so it prefers not to have the cycles.
@jon_stokes There's no specific resource. The agent could have two different preference cycles, and each one could be threatened by a money pumping argument involving a different choice object. Also, training doesn't need to instill a preference for not getting dominated. 1/
@jon_stokes The resource will differ depending on what cycle we're talking about. I only need assume that there exists at least one choice object that is either preferred or dispreferred to each choice object in the cycle. I think it's pretty likely that this assumption holds for...
@jon_stokes For some small value of e, it seems like the agent should take the lottery. Then, you can offer it C, then A. You go round and round in circles, until the agent ends up with D – an option worse by its own lights. This seems like the transitivity money pump sans money.
@jon_stokes I think you can typically substitute the "money" in money pumping arguments for something the agent cares about. For example: suppose A<B<C, and there exists some D such that D>[A,B,C]. You could have the agent choose between A, and a lottery with P(D)=e, and P(B)=1-e. 1/
@jon_stokes Some combination of: 1) agents that are sufficiently intelligent will self-modify their way to coherent preferences so they aren't vulnerable to money pumps, 2) preferences get increasingly coherent with scale. Not certain about either, but they seem plausible!
@jon_stokes It seems premature to dismiss worries about wildly super intelligent agents with coherent preferences, on the basis of agents that aren't wildly super intelligent having incoherent preferences
@jon_stokes It seems premature to dismiss worries about wildly super intelligent agents with coherent preferences, on the basis of agents that aren't wildly super intelligent having incoherent preferences
@MatthewJBar@KatjaGrace it doesn't follow that controls must be stronger. There are differences between chemical weapons development and AI development that could invalidate that implication. Concretely, export controls on AI chips slow development significantly, and aren't that strong of a control.
@MatthewJBar@KatjaGrace This argument feels quite abstract to me, and I'm not convinced it cashes out? First, it is not in their material interest to develop a technology that may destroy them. Second, even if a nation has more incentive to build AI as compared to chemical weapons -
@DavidSKrueger@Miles_Brundage There was this report from the Forecasting Research Institute. From what I recall, concerned experts debated X-risk with skeptical superforecasters at length (30-80 hours of time investment per person), and neither side changed their mind much.
https://t.co/AUrkBUoHUZ
@xwanyex@woke8yearold@tenobrus Rogue AI incidents will always involve humans using poor judgement. It isn't possible to make a completely airtight safety or security case, and at some point the AI becomes better at spotting the holes than any developer.