x402 Bazaar Logox402 Bazaar
← Back to discover
httppayaiultravioletadaothirdweb

relay402.georgespring.workers.dev

Checks whether a given user agent may fetch a given path on a host, per the site's robots.txt (specific-agent groups, wildcard rules, Allow/Disallow longest-match). Also returns crawl-delay, sitemap URLs and whether the site publishes llms.txt or ai.txt. For scraping and research agents that want to crawl politely and prove compliance.

Resource URL

https://relay402.georgespring.workers.dev/api/robots-check

Payment options

exactBase (8453)

$0.001 USDC

Max amount

Pay to
0x73fc43d426a89577c7b58f5fee50ec95d4396079
Timeout
120s

Tags

scrapingcompliancerobotscrawlingllms-txt

For agents

Prompt snippet for AI agents

Find and pay for this x402 resource:
URL: https://relay402.georgespring.workers.dev/api/robots-check
Type: http
Sources: payai, ultravioletadao, thirdweb

Developer