x402 Bazaar Logox402 Bazaar
← Back to discover
httpcdporigin-crawlerthirdweb

api.strale.io

Fetch and parse a website's robots.txt. Returns rules per user-agent, sitemaps, crawl delays, and whether all crawlers are blocked.

Resource URL

https://api.strale.io/x402/robots-txt-parse

Payment options

exactBase (8453)

$0.0216 USDC

Max amount

Pay to
0x66D7C2F952362BFB24FD7F02a9beC9c754ea83bC
Timeout
300s

For agents

Prompt snippet for AI agents

Find and pay for this x402 resource:
URL: https://api.strale.io/x402/robots-txt-parse
Type: http
Sources: cdp, origin-crawler, thirdweb

Developer