A pattern I can’t help but notice in speaking with LLMs about general life advice is that they seem to be a lot more risk averse than I am. I think the average person probably has a more concave utility function than I do, and maybe this is a case of my utility function being an outlier and not the LLM’s. This would be in line with the existing studies on this, which are inconclusive and overall probably show a roughly human-like risk aversion.
Still, the fact that my AI is noticeably more risk averse than I am (I’ve had moments of being recommended extreme compliance/caution in instances where, intuitively, I’d only expect something like 95th+ percentile risk-averse humans to be compliant/cautious, for example advising against posting something mildly provocative) makes me want to defer to it a lot less, and I suspect that at this stage it’s given me on net negative-EV life advice cumulatively.
Overall it would be intuitive for this drift to continue, given that the incentive for a higher “risk aversion parameter” in post-training should be higher than the incentive for a higher risk aversion parameter when consuming the AI’s output – edge cases of extreme harm as a result of the AI’s advice in say 0.0…1% of consumers can cause meaningful regulatory damage to the AI lab, and it’s probably fair to say that advice-giving with the normal human utility function would clearly lead to extreme harm in at least 0.0…1% of consumers.
In the same light, I’ve noticed that AIs are meaningfully less Machiavellian in giving life advice than would be expected of an average person (if both were answering honestly), and certainly less Machiavellian than would be expected of an extremely successful person. This also often makes their life advice seem negative-EV – e.g. the model refuses to suggest one manipulates somebody for personal gain even if it might be optimal – these are more explicitly pursued in post-training than the risk parameters.
These run somewhat contrary to the common wisdom that we’ll gradually converge to a world in which an increasingly large number of people either outsource their life decisions to AIs, or fall behind in life outcomes? Could we not see a different drift in the near term – where people not making their own decisions become suboptimally risk-averse, suboptimally prosocial, etc. and the opposite gets rewarded?
Personal update: I've joined Anthropic. I think the next few years at the frontier of LLMs will be especially formative. I am very excited to join the team here and get back to R&D. I remain deeply passionate about education and plan to resume my work on it in time.