This week Max taught the whole company how 100+ servers talk to each other without the public internet ever finding out.
Wisdom Wednesday is where one person at @byteful teaches everyone else something they know well. Max's was @Tailscale. One private network across datacentres, ISPs and cloud providers, nothing left open to the public web, and when a traffic rule changes you change it once instead of touching every machine.
Open source and written in Go, which visibly pleased Max more than the security did.
(We are still hiring a full-time Support Engineer to join us in person in Manchester. There will be more useful nerdiness.)
Bot detection keeps moving down the stackโฆ
user agents
JS fingerprints
TLS
now TCP
Before HTTP, the handshake can already leak the OS behind it so we rebuilt TCP in user space for ISP & Datacenter proxies.
Try the Beta here:
https://t.co/9CjyZfG0fJ
Formatting a proxy list by hand is the sort of small job that eats twenty minutes and teaches you nothing... one missing colon and the import fails. Free formatter, whole list at once. It won't tell you which proxies are live, just how the list is written.
One missing colon can stop a proxy list importing ๐ง
Our proxy formatter converts a whole list at once. Paste or upload, it works out the source format on its own, you pick the output, then copy or export.
It rewrites the list. It will not tell you which proxies are live.
The Byteful Startup Program is live๐
Free residential data for three months, then up to 50% off for two years. Three tiers for bootstrapped, VC or accelerator backed, and Y Combinator companies.
Plus a Slack channel with a senior Byteful developer.
The jump from 100 requests to 100,000 breaks things that looked fine in testing, and it's usually the boring stuff... timeouts set off the average, retries with no backoff, no real idea what your concurrency ceiling is.
A scraping job that works at 100 requests does not automatically work at 100,000 ๐
1๏ธโฃ Data consumption at volume
2๏ธโฃ Your timeout
3๏ธโฃ Your concurrency ceiling
4๏ธโฃ Retry strategy, with backoff
5๏ธโฃ Whether you can tell why something failed
Your browser can claim whatever it likes, but the TCP handshake underneath tells a different story... proxy servers run Linux. We rebuilt the stack in user space so you can set that layer yourself. Beta for now, so tell us if it does something odd.
Introducing TCP/IP Fingerprint Emulation, now live in beta on datacenter and ISP proxies!
Your browser says Windows. Does the TCP handshake agree? ๐ป
Windows, macOS, Linux, Android or iOS, set per request in your proxy username or once when you export your list.
Day two of @zytedata's Extract Summit is underway, and Byteful is here in Austin, Texas!
Today at 12:00pm, our founding engineer Joe Dye takes the stage at Brazos Hall for a talk on TCP/IP fingerprint emulation๐ซ
Come and say hello before or after๐
Two bot-traffic numbers came out this month and they do not agree. DataDome says bots and AI agents make up 26.5% of traffic across its customer base. Cloudflare puts automated traffic at roughly 57% of web requests.
My guess is both are low. The agentic era is real and exponential.
More than half the traffic reaching sites on Cloudflare is now automated, and AI agents are the fastest-growing part of it. We're giving site owners the tools to see who's visiting, decide who gets in, and charge for access. https://t.co/WzchYEKudA #BirthdayWeek
A very cool example of what web-scraped data can enable in academic research. Found more than 70% of the thousand most repeated questions were templates or boilerplate. FAQ copy, repeated page elements, the same question published over and over.
You can collect an enormous amount of the web quickly. Working out what the data actually matters is the tough bit.
In the agentic age, that is only getting harderโฆ๐
Bodhium Labs: From web pages to a question corpus
FineQuestions draws on 110 snapshots of FineWeb, the filtered English-language corpus derived from Common Crawl, spanning 2013โ2025. Extraction yielded roughly 13.4 billion raw questions filling 22 TB. Normalization and aggregation reduced these to approximately 1.5 billion unique questions.
sharing one login around a team was never a good answer, and it got worse once agents started holding the keys... organizations gives you roles and keys scoped to specific actions with an expiry. nothing to migrate, no extra charge.
Byteful accounts now support organizations ๐ฅ
A shared workspace for your team with its own proxies, billing and API keys. Each member gets a role, so no more sharing one password.
API keys can be scoped to specific actions with an expiry date, which is what you want when an agent holds the key.
selling concurrency in tiers always struck me as charging for the wrong thing... you pay for data or for ips, not for how many requests run at once. unlimited doesn't mean no safeguards though, and pretending otherwise would be dishonest.
Unlimited concurrent sessions means there is no concurrency tier to buy ๐
No plan-level cap to hit, no upgrade to reach for. You pay for data or for IPs, not for how many requests run at once.
Fair usage safeguards still exist for traffic that threatens stability. Rarely triggered.
๐๐บ๐๐ฎ๐๐ต ๐ญ.๐ฌ ๐ถ๐ ๐ต๐ฒ๐ฟ๐ฒ: a stable, open source modern successor to WordPress, built on Astro, with a secure, decentralized plugin registry!
iโm incredibly proud to have contributed to this release during my internship at Cloudflare โ ๏ธ
Read more: https://t.co/iHxnMvXukW
a page makes far more requests than the one you asked for, and most don't need a residential ip... paying residential rates for every asset a browser pulls is waste. smartpath decides per request, under 10ms. our own test last july: 40.79% less residential data.
Smartpath decides in under 10ms whether a request actually needs a residential IP.
If it does, it uses residential. If it does not, the request can go through datacenter instead without consuming residential data.
Most request latency comes after that routing decision.
A 1996 protocol still carries most of the proxy world. The detail people miss: SOCKS5 takes hostnames, not just IPs, so DNS resolves proxy-side, in the region doing the work. Old spec, still quietly correct.
SOCKS5 has been standardised since 1996 and defines three commands ๐
CONNECT for TCP. BIND when the client needs to accept a connection back. UDP ASSOCIATE for UDP datagrams.
It can also pass a hostname to the proxy for proxy-side DNS resolution instead of resolving locally.
Last week's Wisdom Wednesday ๐งโโ came directly out of work Denver Whiteley had been doing to further improve @byteful's security.
He gave a presentation on infrastructure hardening and firewalls after learning more about both through actual work. That is pretty much the ideal source for these sessions... someone runs into something, understands it, then teaches the rest of us.
Denver used a simple building analogy. An IP address is the street address, ports are the doors on the building, and behind each door sits a different application or service. Some doors need to be reachable for legitimate traffic. Others may be left exposed unnecessarily, which gives attackers another place to look.
A firewall guards the street by only allowing legitimate traffic through the doors.
That is the pattern with these sessions. Someone does the work, learns something doing it, then teaches the rest of us.