Miles Heslen

just doing stuff
economics @TerryCollege

[Miles Heslen @milesheslen]

Write-up: https://t.co/HHSWX9LB78
Notebook: https://t.co/c6o3N7uMGo

[Miles Heslen @milesheslen]

I took 250 real respondents from the 2024 General Social Survey, turned their demographics into personas, and had an LLM answer the same five survey questions they did. Then I scored the answers against the real people.
All 250 personas answered "pretty happy" on the general happiness question. Not most of them. All of them. Real respondents were 57% pretty happy, 20% very happy, 23% not too happy. Temperature was set at 1, so this wasn't just the model playing it safe.
On political views the simulated distribution was off by 15%, but 55% of individual agents gave different answers than their real counterpart. You can match a distribution and still miss almost everyone in it.
Death penalty support by party - real people: 49% of Democrats, 63% of independents, 74% of Republicans. The simulation: 40% Democrats, then 100% and 100%, respectively. Reality is a dial. The simulation is a switch.
Four of the five questions offered a "don't know" option, like the real survey. Real respondents answered with it 3-4% of the time. The simulated personas used it zero.
I wrote my predictions down before running any of it.I predicted affirmative action would come back heavily in favor. It came back at 58% "strongly opposes", and while 18% of real respondents answered "strongly favors", the answer never appeared once among simulated personas. Completely wrong.
What I haven't done yet:
- no noise floor for the distance metric
- no repeated runs, so I can't separate "deterministic" from "very low variance"

Links below

[Miles Heslen @milesheslen]

Just got done with my first little bit of my experiment. I was really just validating that everything works, for a subsample of 250 of my original 3,000+ from the GSS. Got the same answer for each artificial persona. Here's some graphics that Claude made.

Basically, I took real respondents from the 2024 GSS, took 6 variables (age, sex, race, education, party ID, region), built a persona prompt, and asked them the verbatim GSS question for happiness. Gave the agent the same 3 answer choices as the GSS, with "Do not know" added.

Using Sonnet 5, with it's temperature at 1, and every one of the 250 agents answered with "Pretty happy", while as you can see the real distribution for the total distribution is 20.2/57.8.22. So our total variation distance is about 0.42.

One reason for this could be that none of the variables included strongly predict happiness. Variables like marriage, health, income and employment do. It would be interesting to see another run with those variables included. Another reason for the flattening could have to do with way the experiment is run.

EDSL from @ExpectedParrot was the Python library that I used to do this. When an agent is run, they are provided with a comment to make about why they chose their answer. Depending on when this happens, a model thinking too much about their answer could be causing them to flatten, kind of like how humans overthink (could be an interesting topic to research, btw). Now, I'm not 100% sure, this is my first project using EDSL.

This is just my first actual pass at using a sample of agents to answer questions. I still have another 4 questions to panel and different variations I want to perform. When it's all said and done I will provide the full notebook along with a brief. I just thought the 250/250 answers being the same was interesting, and I instantly thought I was doing something wrong, but I check and I honestly don't think I am.

[How To Prompt @HowToPrompt__]

China created a system that simulates the earth with billions of AI agents that has real personality, memory, beliefs and human-like desires.

it gives them a terrifying ability to predict the future.

they published a paper called "Modeling Earth-Scale Human-Like Societies with One Billion Agents."

they built "Light Society," a framework that efficiently models human-like societies powered by large language models.

traditional agent-based models were always limited by simple, predictable behaviors.

but this new earth-scale simulation framework uses a "mixture-of-models" engine to change everything.

by combining massive llms with smaller, highly efficient distilled surrogates, the researchers successfully simulated a society of over one billion autonomous agents.

they grounded these agents using real-world demographic data from the world values survey. these agents don't just act like generic bots..

they exhibit sophisticated social behaviors that mimic actual, diverse human populations.

they ran simulations on trust games and massive opinion diffusion, tracking how information spreads and how society evolves in real-time.

the simulation proved to have incredibly high fidelity in modeling diverse social phenomena, giving researchers a practical foundation for hypothesis testing.

think about what this means for predicting global elections, market crashes, or how a population will react to a crisis..

you don't need to poll people or guess anymore. you just run the simulation.

the infrastructure for agentic simulation is scaling faster than anyone realized. social sciences and predictive modeling will literally never be the same..

[Miles Heslen @milesheslen]

@mylesfriedman7 @aaruHQ @ExpectedParrot @simsurveys For sure. Can you message me about it?

[Miles Heslen @milesheslen]

Doing a little silicon-sampling project/experiment. Just trying to learn about the space. I'm thinking that having AI simulate human behaviors and opinions is going to be a big industry. There's already big players in the game like @aaruHQ and @ExpectedParrot.

[Miles Heslen @milesheslen]

I'm going to try to write about it more here as I complete more, then do a full post on Substack, hopefully with visualizations and all the fun stuff.

[Miles Heslen @milesheslen]

Basically take a sample of the GSS, get those respondents demographic variables, include them as a prompt to an LLM, ask it the questions, get its answers and see if it matches to the real GSS respondent. I'm stuck doing some data cleaning right now so I haven't gotten to far.

[Miles Heslen @milesheslen]

https://t.co/IuLa2usb8E

[Miles Heslen @milesheslen]

Wrote my second substack article. Picked up from the last one, talking about the four things agentic simulations are missing.

Honestly didn’t think I’d like writing, but it forces you to spend time learning what you’re talking about to evade embarrassment

Here’s the link:

[Miles Heslen @milesheslen]

It's a little barebones so I plan on digging deeper into the idea since I have some time off of school during the summer.

[Miles Heslen @milesheslen]

Wrote my first Substack article about an idea I had while taking a class at UGA. It's about using agentic simulations to predict counterfactuals in causal research. Had the idea rehashed since I'm currently reading @causalinf "The Mixtape". Check it out: https://t.co/wJzHUyfBgt

[Miles Heslen @milesheslen]

@rajatsandeepsen @JsonBasedman Why

[Miles Heslen @milesheslen]

@rileybrown Is there a recording of this stream?

[Miles Heslen @milesheslen]

@Saboo_Shubham_ I swear, anyone that has any sense of agency will be all over this. It’s crazy to me that I won’t know about things like this until they just pop up on my feed. What a great idea.

Pre-validated ideas, preciate you!

[Miles Heslen @milesheslen]

@v_computer @zulali Hahaha

[Miles Heslen @milesheslen]

@rork May just use it privately for my work and build an app that is completely for my work, or maybe also try to publish it and get actual users.

[Miles Heslen @milesheslen]

Recently began working on a little side project. I've been trying to think of ways that I can solve daily problems in my life using AI. I came across a video of someone using @rork to build mobile applications, completely by prompting AI. (thread)

[Miles Heslen @milesheslen]

@rork When a member selects a task, their name/icon/initial shows up next to the tasks showing that they are working on it, with something like "In Progress" next to it. This way, the admin can track who is doing what task in real time.