Alexsandra Cavalcanti @cavalcantialexs - Twitter Profile

about 1 month ago

Google trained an AI to predict your neighbourhood's income by counting the coffee shops, bus stops, and high-rises on a map. Nobody told it what income was. The model is called S2Vec, and it was published by Google Research as part of their Earth AI initiative. It takes the built environment (every building, road, park, and business in an area) and converts it into a layered image. Three coffee shops and one park in a grid cell become pixel values. The AI then reads that image the same way a computer vision model reads a photograph. The training method is the part that matters. S2Vec uses masked autoencoding: you show the model a patch of a city with chunks missing, and it learns to fill in the gaps. Show it a cluster of high-rise apartments next to a subway station, mask out a section, and it predicts a grocery store belongs there. Do that millions of times across the globe and the model learns the deep spatial grammar of how cities organise themselves. No human ever labels a region as "financial district" or "suburban residential." The model figures out those groupings on its own from the geometry of what's built where. The output is an embedding, a string of numbers that acts as a mathematical fingerprint for any location on Earth. Feed those embeddings into a prediction task and S2Vec can estimate population density, median income, and carbon emissions for regions it has never seen before. On zero-shot geographic extrapolation (predicting for regions entirely absent from training data) S2Vec was typically the best-performing individual model. It matched or beat satellite imagery baselines like RS-MaMMUT and outperformed GEOCLIP on socioeconomic prediction. The best results came from combining S2Vec with satellite image embeddings. Built environment data alone couldn't capture vegetation, terrain, or transportation patterns well enough for environmental tasks like tree cover and elevation. But fused together, the two modalities outperformed everything else. The standard approach to geospatial ML has been hand-crafting indicators for every new problem. Predicting air quality meant building a bespoke feature set. Estimating housing prices meant building another one. S2Vec replaces that with a single general-purpose representation that transfers across tasks. The training data is map features, not satellite pixels. That distinction is pretty important to understand. It means: map data updates faster, costs less to process, and covers built infrastructure at a resolution satellite imagery can't always match. A satellite sees rooftops. S2Vec knows there are three cafes, a pharmacy, and a bus stop underneath them. Google's broader Earth AI pipeline now has three foundation models working in parallel. 1. PDFM for population dynamics. 2. RS-MaMMUT for satellite imagery. 3. S2Vec for the built environment. Stack them and you get a system that can read a neighbourhood the way a local understands it. More info on it here: https://t.co/vVJlLlfhc7

yohaniddawela's tweet photo. Google trained an AI to predict your neighbourhood's income by counting the coffee shops, bus stops, and high-rises on a map. Nobody told it what income was.

The model is called S2Vec, and it was published by Google Research as part of their Earth AI initiative. It takes the built environment (every building, road, park, and business in an area) and converts it into a layered image. Three coffee shops and one park in a grid cell become pixel values. The AI then reads that image the same way a computer vision model reads a photograph.

The training method is the part that matters. S2Vec uses masked autoencoding: you show the model a patch of a city with chunks missing, and it learns to fill in the gaps. Show it a cluster of high-rise apartments next to a subway station, mask out a section, and it predicts a grocery store belongs there.

Do that millions of times across the globe and the model learns the deep spatial grammar of how cities organise themselves. No human ever labels a region as "financial district" or "suburban residential." The model figures out those groupings on its own from the geometry of what's built where.

The output is an embedding, a string of numbers that acts as a mathematical fingerprint for any location on Earth. Feed those embeddings into a prediction task and S2Vec can estimate population density, median income, and carbon emissions for regions it has never seen before.

On zero-shot geographic extrapolation (predicting for regions entirely absent from training data) S2Vec was typically the best-performing individual model.

It matched or beat satellite imagery baselines like RS-MaMMUT and outperformed GEOCLIP on socioeconomic prediction. The best results came from combining S2Vec with satellite image embeddings. Built environment data alone couldn't capture vegetation, terrain, or transportation patterns well enough for environmental tasks like tree cover and elevation. But fused together, the two modalities outperformed everything else.

The standard approach to geospatial ML has been hand-crafting indicators for every new problem. Predicting air quality meant building a bespoke feature set. Estimating housing prices meant building another one. S2Vec replaces that with a single general-purpose representation that transfers across tasks.

The training data is map features, not satellite pixels.

That distinction is pretty important to understand. It means: map data updates faster, costs less to process, and covers built infrastructure at a resolution satellite imagery can't always match.

A satellite sees rooftops. S2Vec knows there are three cafes, a pharmacy, and a bus stop underneath them.

Google's broader Earth AI pipeline now has three foundation models working in parallel.

1. PDFM for population dynamics.
2. RS-MaMMUT for satellite imagery.
3. S2Vec for the built environment.

Stack them and you get a system that can read a neighbourhood the way a local understands it.

More info on it here: https://t.co/vVJlLlfhc7

13

992

148

1K

86K

Alexsandra Cavalcanti

@cavalcantialexs

Who to follow

Last Seen Users on Sotwe

Trends for you

Most Popular Users