Srdjan Miletic

Into philosophy, rationality, economics & ethics

I write at Engineer @ Apollo Research

[Srdjan Miletic @SrdjanMiletic1]

There's also a more meta/ethical point about which group level predictors you think it's just/unjust to use when making predictions about individuals.

(Also, you can probably just skip this whole problem by selecting migrants for IQ/education/job offer with a high salary. You'd then get a migrant mix which would likely be better on most metrics than the average native citizen regardless of gender/age)

[Srdjan Miletic @SrdjanMiletic1]

Not necessarily. Assume that you want a migration policy which soely optimizes for selecting people who contribute the most to your economy/country. That might mean you should select for women over men. But insofar as age goes young people, while committing more crime, are usually much better fiscally than older people. They have more working years ahead of them. So overall you may want to prefer younger immigrants, even given their heightened crime risk.

[Srdjan Miletic @SrdjanMiletic1]

While I agree with the overall sentiment re gartner, I want to point out that this specific quadrant is for legacy code modernization tools, not for IDE's/coding agents. It's plausible to me that Claude Code/Codex are worse at modernizing large COBL codebases than something like rocket.

[Srdjan Miletic @SrdjanMiletic1]

@ben_j_todd What do you think about Embedded Evaluators? Specifically, do you think it's good for AI safety orgs to get involved in these programs or do you think on average it would be better to focus on other things (e.g: alignment science)

[Srdjan Miletic @SrdjanMiletic1]

Hmmmmm. Actually, I'm not that sure about this. How does the cost of a final good get distributed through the supply chain? Presumably something like
- if you have a competitive market then the cost of each node in the production tree is driven down to cost/normal profit
- the cost of each node depends on the tech/product
- some products have very different cost structures
- I assume that the final good of two different kinds of compute (AI vs BitCoin) could have very different cost allocations between energy and prior steps in the chain

[Srdjan Miletic @SrdjanMiletic1]

@robinhanson @VitalikButerin Couldn't it true that the kind of chips used for AI are just much harder to produce, meaning most of the cost concentrates in the chip manufacturing process and not energy/running the chip once it's bough?

[Srdjan Miletic @SrdjanMiletic1]

Engineer at an leading AI Safety Lab.

Team was already very AI pilled when I joined but is more and more so over time. PR output is drastically higher. The raw number of PR's is maybe 5x. But the scope/scale/complexity of features/PR's we're delivering is also drastically larger.

Spend is high. 4k/month per engineer wold not be strange. Much, much higher for researchers.

We've also struggled a bit with the standard problems
- dealing with the flood of PR's
- code being cheap => temptation to create dozens of features or cater to every customer request => realization that keeping your product surface area small is actually often better both for customer UX and your ability to maintain it
- how to use models without creating slop. Specifically, I've found that models are great at implementation but far worse at architecture/code clarity etc... Left to their own devices they can implement complex features in one shot but can't do so in a way that provides a good foundation to build on. The mix for us seems to be some level of high level human review/arch and low level implementation + review by models

For me personally it has taken a while to get used to trusting the models end to end and using `/goal` and clear objectives to have agent's hill climb on objectives. I more an more find I'm managing teams of agents and deciding on high level goals/arch. Can be exhausting but also exhilarating.

[Srdjan Miletic @SrdjanMiletic1]

Some rough thoughts on AI consciousness

First, a meta-problem.
- I'm pretty deeply unsure about the philosophy of consciousness.
- I tend towards illusionism, or at least some kind of anti-realism. I can't give a great argument for it, but it seems weird to posit the existence of something that is definitionally not ever possible to empirically verify and doesn't have any predictive power in our universe
- I do experience qualia (or think I do), have an internal monologue, etc.... At the same time I also experience a sense of good/evil and the intuition that certain moral claims are just objectively correct or false. I think on reflection that the latter moral intuition is something programmed into my by evolution and false. There is no objective moral reality. I suspect the same is true of qualia/consciousness.
- Hence talking about whether a given entity is conscious kinda doesn't make much sense. Similarly to how talking about how many angels can fit on the head of a pin doesn't make sense.

Okay, but I'm aware that illusionism is a minority position and not one most people share. It's also one I'm not all in on so in the interests of moral uncertainty/thinking about the problem from the average informed persons starting point:
- Let's say we reject illusionism. Assume qualia does exist, in an objective sense, and is a property certain kinds of agents can have. How do we know if AI's have qualia?
- How do I know if I have qualia? I can experience it.
- How do I know if other people have qualia? I can't. They may say they do, but they could be P-Zombies. I do think most people have qualia because most people's minds and brains are likely very, very similar to my own and share most traits with me.
- The further away you move from a mind like mine, the less certain I am. Dogs, cows, pigs, other large mammals... Fairly high confidence they have qualia. Small Mammals, somewhat high but less so. Fish? Much lower. Insects? Lower again.
- The scale here is a combination of two things. One is similarity to my mind, which is the only mind I know for sure has qualia. The other is complexity/size. I guess that smaller minds are less likely to have qualia or internal representations of the world/self. I'm very unsure about this. We may live in a universe where you could have creatures far smaller, dumber and simpler than us yet who experience things far more than we do.
- AI's are a hard case. On one hand their external behaviour seems to indicate consciousness. On the other hand they're trained on all human text and emulate humans speech/thought. On the other hand they're post-trained specifically to not claim/act conscious.
- Also, their cognitive architecture is completely different from our own. A fish is far, far closer to me in terms of how it's brain works and mind is structured than a modern LLM. So it's hard to use my general heuristic here
- So I guess my conclusion is deep uncertainty. Is Claude conscious? More specifically, what's my credence that Claude is more conscious than a fish? Or an insect? I don't really know.

[Srdjan Miletic @SrdjanMiletic1]

>There’s no reason to think anything non-biological is conscious.

I'd be curious to know why you think this. My probability on LLM's being conscious is fairly low (and to put my cards on the table I tend to lean towards illusionism anyway) but I would guess there's a higher chance of Claude being conscious than a simple insect such as a fruit fly?

A few other questions
- do you think a full simulation of a human brain, assuming accuracy down to the atomic level or something similarly high resolution, would be concious?
- what's your rough intuition/metric for assessing if another mind is conscious? You have access to your own experiences/know you have qualia and hence can assume your own consciousness. What about humans/animals/insects etc...

[Srdjan Miletic @SrdjanMiletic1]

Hmmmm. I get that the person affecting view doesn't care about future people not coming into existence and hence cares less about extinction/forfeiting the lightcone. Still, doesn't the standard AI risk argument of "High odds most currently living people die/get huge negative utility" work?

Or is it something like AI going well = currently living people could live an incredibly long time/get lots of utility. If everyone lives 100x longer/get's 100x more utility in the good AI scenario, then maybe we should take the AI pill even if it's 90% likely to lead to extinction/everyone's utility going to 0?

[Srdjan Miletic @SrdjanMiletic1]

Why do you think this is wrong?
- You think the risk of reoffending/harm to the public is still probably too high?
- You think it's unjust that they don't spend longer imprisoned?
- Something else?

If it's the risk, then probably the best way to assess this is to try to find data on re-offending rates of previously institutionalized for serious crime individuals.

If it's the injustice, then I think we may have different moral intuitions. Assuming the court/legal standard is correct and these people genuinely aren't culpable for their actions, I would not want them to be punished.

[Srdjan Miletic @SrdjanMiletic1]

> There is also a reference class problem. What exactly counts as an observer “like me”? A perfect duplicate? Someone with vaguely similar experiences? The same observer moment? A simulation with my memories but different future experiences? Unless that is made precise, you cannot just say there are 100 million relevant observers and treat the ratio as obvious

1. I think that only SSA suffers from the reference class problem, not SIA. SSA doesn't require you to define a reference class. It just asks how many observers are in your same epistemic position. Basically, if you could distinguish between yourself and another person from the inside view, then you're not in an identical position.
2. If you don't buy SIA then I agree the whole argument falls down.

> But a huge likelihood ratio does not rescue an arbitrarily tiny prior

I agree. I think that anthropics + SIA is an argument in favor of the simulation hypothesis. Not a definitive argument. Basically, I think it should cause you to update somewhat towards simulation. If your prior is low, which it may well be for many reasons, then the SIA update alone will not be sufficient to convince you

> Imagine Universe C contains no simulations at all, but because space is infinite, there are a trillion naturally occurring physical duplicates of me, with the same memories and epistemic position. Universe B contains your 100 million simulated copies. SIA would tell me to favor Universe C over Universe B. But notice what just happened: the exact same reasoning you presented now counts as evidence against the simulation hypothesis

Agree that buying SIA = buying evidence for large worlds , including worlds which are large for reasons other than simulation. I guess my intuition here is that if you're choosing between "base reality is the seemingly finite physical universe we see" and "base reality is a much vaster possible space that includes countless sims" SIA favors the latter.

[Srdjan Miletic @SrdjanMiletic1]

Really cool to see new projects like https://t.co/QFEgZi6wPn. Turns out you can just do things.

A few interesting features
- donors can delegate the decision on which grants to fund to experts (liquid democracy-esque)
- quick/light application process (similar to something like fast grants
- proportional cost sharing to resolve the last mover incentive (not that I'm convinced this is that big a barrier)

[Srdjan Miletic @SrdjanMiletic1]

Hmmmmm. What alternate source would you prefer as the foundation for moral arguments? For both realist and anti-realists moral intuitions tend to be what we ground ethics on. Either because (realist) they are the only accessible evidence we have of of an objective moral reality or (anti-realist) because moral preferences are a kind of preference and our intuitions illuminate the complex landscape of our moral preferences.

(I think there's an interesting question in why anti-realism cares about intuitions. My best guess is that ethics for an anti-realist is a process of coming to understand your own "true" ethical beliefs better, where "true" means something like considered or fully worked out)

[Srdjan Miletic @SrdjanMiletic1]

One hypothesis is that war crimes are activities that benefit from not happening.

E.g: imagine that poison gas is easy to manufacture and use and is equally useful for both sides in a war. Imagine that using it also ensures that at least 20% of soldiers become cripples for life. In this case, even maximally self interested parties to a conflict will have a strong incentive to coordinate with each other to not use gas.

I think this explains many rules/norms in wars. E.g: not targeting officers during the Napoleonic wars (symmetric, imposes greater personal risk on elites who are the ones making decisions), taking POW's instead of killing everyone (can trade back after war, alternative is you're both weaker to other nations/enemies)

[Srdjan Miletic @SrdjanMiletic1]

Doesn't the unkowability bite here too?

E.g:
- If the distant future equilibrium for populated regions of space is net negative (massive galaxy wide suffering), then extinction could be good
- If human civ is likely to work out universe destroying tech (e.g: stable new vacum) and less likely to control it than other civs that arise in the future, then extinction may be good

I guess if I was to make this more explicit my argument is that a certain action locking you into a stable know state doesn't help with unknowability because you're still need to know the counterfactual state you would have been in had you not taken the action in order to assess if the stable state is preferable.

[Srdjan Miletic @SrdjanMiletic1]

Hmmmm. I suspect that there isn't a single absolute threshold. Some intuition pumps for this
- "I have a drug that costs $1 and will cure cancer if you have it" => strong intuition should call even if the person you're calling only has a 1% chance of having cancer. Compare this to "I have a car insurance policy that will save you $10 per year if you have insurance" => strong intuition that calling is wrong even if there's a 20% chance the person you call has insurance.
- Calling someone who doesn't mind calls/enjoys chatting with salespeople (e.g: somewhat lonely and time rich old person) feels like it should be a higher threshold then calling someone who probably doesn't want to be disturbed. e.g: A busy CEO.

I guess if I try to systematize my intuitions here I would use one of two standards.
1. Expected value. At what interest % filter do we max overall utility generated.
2. Consent/respecting preferences. At what interest % filter do we best align with the calee's preferences. e.g: some optimization function over calling people who would want to be called and not calling people who would not. Maybe also taking into account the strength of preferences, but that get's tricky and preference aggregation/weighing is messy in various ways.

[Srdjan Miletic @SrdjanMiletic1]

@NathanpmYoung Hmmmm. Do you think the costs/effects of prediction markets are meaningfully different from gambling in general? Do you think prediction markets reached potentially problematic bettors who would otherwise have not gambled?