@GCNDiscs_@Waffl3x@endless_sine like just think about this, when you do
GET https://t.co/2bvkYrtqeB what does the server have to do? does it sound expensive? it shouldn't be. What is happening is the software isn't properly designed to scale and instead of fixing the problem we blame "AI crawlers"
@GCNDiscs_@Waffl3x@endless_sine NO ITS NOT lol thats what im fucking saying, a crawler should not take down your website. It doesn't matter what it is
Publicly visible info about git repos can be cached and mostly shouldn't change much and serving that billions of times should be AS CHEAP as anubis
@Waffl3x@endless_sine No. Serving web pages has a computational cost. The fact that your git host cannot be within a factor of 10 of anubis' cost is sad.
@Waffl3x@endless_sine Because the site was unoptimized... One server should be able to handle millions of concurrent clients at once with no trouble even if you are handling things like displaying fresh git content.