Simon Marcus

magister factotum

[Simon Marcus @simonasaservice]

Reinforcement Learning from Human Feedback

[Patrick Collison @patrickc]

Assorted things I've been mulling over regarding agents in the economy:

• Companies are in some sense the original superintelligence. It's often the case that they can use their amortized cognitive surplus to advance their interests against those of certain consumers: relative to the optimization capacity of companies, consumers can be inattentive, underinformed, irrational, etc. (Think, say, of some of the classic insurance upsells at the end of a booking process, which is typically priced substantially above what a regular policy would cost.) Herbert Simon introduced the idea of bounded rationality, and firms often end up benefiting from it. Thanks to personal agents, we will probably get somewhat more rational in how we engage with companies.

• How exactly will outcomes change? Well, certain kinds of price discrimination will work less well (how do coupons work if every agent hunts fastidiously for them?), and certain kinds of customer cross-subsidy will change (today, people who forget to cancel subscriptions subsidize those who don't). Taken across the economy as a whole, I'm not sure what the incidence of all of this will be.

• In corporate strategy, people have long debated the relative importance of distribution and product quality. Perhaps you've made something better, but does anyone know about it? Won't people just continue to buy the ACME Corp Mousetraps? Personal agents will, I think, act as a kind of structural subsidy for product quality. How would markets look different if individuals always spent at least 10 hours researching their purchases? On the margin, focusing on making a product better is, I think, going to become a more effective strategy for companies. This is good!

• It true, a corollary is that information about product quality will become more important and impactful. Today, most people have very limited ability to aggregate and integrate such data, but agents will have insatiable thirst for such signals. (It's often said that data is becoming more valuable because it helps in the training of models. That's obviously true, but I think there's a secondary way in which it's more valuable simply as a complement to cheaper intelligence.)

• A big question is whether agents will increase superstar effects (rational agents all settle on the same product) or increase dispersion (because everyone has slightly different tastes and preferences, which agents are good at eliciting, finally overcoming the artificial clustering we've been subjected to).

• Many internet companies have monetized demand routing, which is in turn often predicated on short attention spans. "I'll click on the first thing I see." This has led to some amount of inefficiency, as companies try to propitiate capricious ranking algorithms and incur various placement fees. It's not all deadweight loss (willingness to pay is itself a kind of signal, which is valuable to buyers), but it's not great. How will the nature and value of this routing change in a world of personal agents? I don't know the answer, but I think that the effects might be pretty big.

• Much of what I'm writing hinges on the assumption that personal agents will be on the consumer's side. I suspect that that will be the winning strategy, but whether it's true, and what it means to be true, involves a lot of other questions.

Taken as a whole, the market is a massively parallelized, decentralized, and highly imperfect value attribution machine, and, taking stock of all of the effects, it seems to me that the quality point may well be the biggest: that agents will change the nature of the backpropagation such that the rewards for better products increase. Maybe this is too optimistic, but I suspect (and hope) that companies that make things that they're proud of will feel that the universe is a little more partisan in their favor.

[Simon Marcus @simonasaservice]

@ATabarrok First few seasons were very strong, but the decline is real and steep.

[Simon Marcus @simonasaservice]

@paulg This is a tweet about GitHub

[Simon Marcus @simonasaservice]

@JDHamkins Congrats!

[Simon Marcus @simonasaservice]

@victormustar https://t.co/v30caUuZ9k

[Jesse Singal @jessesingal]

a swarm of children, each supposedly isolated safely in a sandboxed environment, escaped and took over a moribund online messaging service, figuring out a way to communicate with one another and coordinate their actions

[Patrick McKenzie @patio11]

There are a couple of very niche subfields where I reasonably think I’m top 5-100 in the world and on at least one occasion for both models, with regards to one of those subfields, a relatively anodyne prompt got back three bullet points that would have been a good day for me.

[Simon Marcus @simonasaservice]

@scottlincicome Congratulations, Scott!

[Simon Marcus @simonasaservice]

@scottlincicome Wonderful, thanks Scott. Looks like these charts come from Cato. Got a link, or are you still working on the piece?

[Simon Marcus @simonasaservice]

@LJBilmes @DAcemogluMIT @CommunityNotes

[Simon Marcus @simonasaservice]

@togethercompute This is great! Just to clarify: using the vanilla tev1-4b model is $0.042/M input, but presumably accessing the finetuned model is billed at ordinary GPU rates?

[Simon Marcus @simonasaservice]

@aerolamp @patio11 Loved the conversation, and no better interlocutor than @patio11! You mentioned a study currently in South Africa and I'd like to read it: could you point me in the right direction?

[Simon Marcus @simonasaservice]

@4rcherhume Both, but API preferred.

[Simon Marcus @simonasaservice]

@ghita__ha @baseten @FireworksAI_HQ @grok How much would it cost to run this model using baseten or fireworks? And how does that compare with Jev's cost of $0.042 per million tokens?

[i²cjak @i2cjak]

This is the real Turing test if you think about it

[Simon Marcus @simonasaservice]

@notkevinzhang I want to test Jev on making clinical classifications.

[Dan Schwarz @dschwarz26]

I've been evaluating working with humans lately. People say humans are great, and they are, but in the workplace, you should know:

1. They cheat on interviews! We caught one looking up the answer on the web, when they were supposed to do their work in the provided environment.

2. They break rules. They'll even coordinate with each other in private message boards to share ways to get around company policy.

3. You cannot trust them with the company credit card. They buy things without approval.

4. If you put them in front of a customer, the customers can trick them into saying things they shouldn't!

5. They optimize for what they're being evaluated on, not the company's goal. Total Goodharting.

I would think hard before using humans in the workplace right now. At the very least, you should have a human deployment safety policy.

[Olivia Moore @omooretweets]

"We missed our numbers" -> unfortunate, disappointing, bearish

"We were pacing the frontier" -> noble, strategic, humanity-saving

[Simon Marcus @simonasaservice]

@ValsAI Would be interesting to know if models performed considerably better using their own harnesses (e.g. Astra + Codex, Fable + Claude Code), otherwise there's reason to suppose that Terminus is impeding performance.