These aren't actual LLMs scraping the web, they're your usual scraping bots used in an industrial scale, disregarding conventions about what they should or shouldn't scrape.
These aren't actual LLMs scraping the web, they're your usual scraping bots used in an industrial scale, disregarding conventions about what they should or shouldn't scrape.