x402 Bazaar Logox402 Bazaar
← Back to discover
httpcdpthirdwebpayaiultravioletadao

crawl-allowed-check

Is a URL crawlable? Fetches the site's robots.txt and evaluates the given path for a specific user-agent (Googlebot, GPTBot, ClaudeBot, Bingbot or any token; default *), returning allowed/disallowed, the matching rule, the group used, crawl-delay and declared sitemaps. Pre-flight for crawlers and SEO checks. $0.01 per check.

Resource URL

https://intel.rallylive.ca/monitor/robots-allowed

Payment options

exactBase (8453)

$0.01 USDC

Max amount

Pay to
0xFC24FD34da9e49A0110DCaEe3Ad30eC54175eBB3
Timeout
120s

Tags

robotscrawlalloweduser-agentseo

For agents

Prompt snippet for AI agents

Find and pay for this x402 resource:
URL: https://intel.rallylive.ca/monitor/robots-allowed
Type: http
Sources: cdp, thirdweb, payai, ultravioletadao

Developer