this post was submitted on 19 Aug 2025
827 points (99.3% liked)

Technology

74359 readers
2652 users here now

This is a most excellent place for technology news and articles.

Our Rules

  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago
MODERATORS
L3s@lemmy.world
enu@lemmy.world
technopagan@lemmy.world
L4s@lemmy.world
L3s@hackingne.ws
L4s@hackingne.ws
827
The AI company Perplexity is complaining their bots can't bypass Cloudflare's firewall (www.searchenginejournal.com)
submitted 4 days ago* (last edited 4 days ago) by Davriellelouna@lemmy.world to c/technology@lemmy.world
203 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] Kissaki@feddit.org 100 points 4 days ago* (last edited 4 days ago) (3 children)

Perplexity argues that a platform’s inability to differentiate between helpful AI assistants and harmful bots causes misclassification of legitimate web traffic.

So, I assume Perplexity uses appropriate identifiable user-agent headers, to allow hosters to decide whether to serve them one way or another?

[–] lime@feddit.nu 37 points 4 days ago (1 children)

yeah it's almost like there as already a system for this in place

[–] seraphine@lemmy.blahaj.zone 1 points 3 days ago (1 children)

THE CAKE DAY IS NOW. (i dont have an image at hand)

[–] lime@feddit.nu 1 points 3 days ago* (last edited 3 days ago) (1 children)

i really wish we wouldn't do those. feels too reddity.

but thanks.

[–] seraphine@lemmy.blahaj.zone 1 points 3 days ago (1 children)

as you wish

[–] lime@feddit.nu 2 points 3 days ago

*monkeys paw curls and i turn into cake*

[–] ubergeek@lemmy.today 9 points 3 days ago (1 children)

And I'm assuming if the robots.txt state their UserAgent isn't allowed to crawl, it obeys it, right? :P

[–] Kissaki@feddit.org 4 points 3 days ago (1 children)

No, as per the article, their argumentation is that they are not web crawlers generating an index, they are user-action-triggered agents working live for the user.

[–] ubergeek@lemmy.today 2 points 3 days ago

Except, it's not a live user hitting 10 sights all the same time, trying to crawl the entire site... Live users cannot do that.

That said, if my robots.txt forbids them from hitting my site, as a proxy, they obey that, right?

[–] drmoose@lemmy.world 0 points 3 days ago

Its not up to the hoster to decide whom to serve content. Web is intended to be user agent agnostic.