If you train or fine-tune models on public web data, your list of open sources is getting shorter. We measured it.
On 25 September and 27 October we read robots.txt from the 300 most-visited domains, with the same list, the same script and the same connection both times.
151 of them publish a robots.txt. 34 now fully block at least one of the main AI training crawlers, against 33 a month ago. Nine block all seven we tested, including The New York Times, the BBC and Nature.
Sites treat training and search differently. 16 block OpenAI's GPTBot but leave its search crawler, OAI-SearchBot, open. OpenAI's own documentation says a site can do exactly that.
In one month, https://t.co/JcczwPjqah moved from a partial rule to a full block. CNN went the other way.
A different connection changes where a request comes from. It does not change what a site has agreed to.
Our new post sets out five checks to run before a page goes into a training set, with the full survey data:
https://t.co/0hyFF86XpR
Free seven-day pilots are open to teams building AI search, RAG or crypto data products. If you run a training pipeline, say so on the form: https://t.co/VrNpFiYJ3N
The future of the internet is an exciting space to explore. Keeping up with Perceptron's vision for decentralized infrastructure. 🌐
@PerceptronNTWK#NodeAndProud
It's time to bring your idea to life. Build your own workforce with an AI-powered Organisation.
@Vanarchain Foundry campaign is live now. @PerceptronNTWK has joined as an ecosystem launch partner to bring you special access.
🎁 7,000 free credits with code "Perceptron"
👉 https://t.co/P9ZiiSZpl7
Spending on AI agents is set to more than double next year, from $29.2B to $65.5B, per Gartner.
An agent acting on last month's data just makes last month's decisions faster.
Perceptron's network is capable of feeding agents live, ground-level data from the web as it moves.