x402 Bazaar Logox402 Bazaar
← Back to discover
httpthirdwebultravioletadaocdp

web.openverbs.com

Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.

Resource URL

https://web.openverbs.com/v1/robots

Payment options

exactBase (8453)

$0.006 USDC

Max amount

Pay to
0xb3f4de103F34b99f0Da19B4F16D6B53a14C2194A
Timeout
300s

Tags

robots.txt parserrobots txt checkeris url crawlablecrawl permission apiuser-agent allow disallow

For agents

Prompt snippet for AI agents

Find and pay for this x402 resource:
URL: https://web.openverbs.com/v1/robots
Type: http
Sources: thirdweb, ultravioletadao, cdp

Developer