i love apple photos memories because it’ll be like hey remember when you found a parking spot in hayes valley in 2022? wasn’t that special?
and i’ll be like yeah it was. that’s exactly why i took a photo of that moment. thank you apple photos
@DavidSHolz@yassineyousfi_ we experimented with using slurm for everything, but the scheduler struggles to keep up past a few hundred jobs per second, and for short-lived tasks we often run thousands per second
@DavidSHolz@yassineyousfi_ we use https://t.co/CeAPPv0wld, the workers run as preemptible slurm tasks so we can easily switch between training and inference on a single cluster
@DavidSHolz .@yassineyousfi_ has done a bunch of solo 32-node runs, but even that is a lot of work. I’d guess 128 is close to the hard limit unless your infrastructure is _really_ good.
@KelseyTuoc@CharlesFLehman I feel like this is a real structural issue with prediction markets, there's very little signal for low-probability events since the inflation-adjusted EV of betting against is negative even if the true odds are 0%.