x402 Bazaar Logox402 Bazaar
← Back to discover
httpcdpthirdweb

crawl-preflight.postnov01.workers.dev

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.

Resource URL

https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain

Payment options

exactBase (8453)

$0.02 USDC

Max amount

Pay to
0x02d15E7f25d29743c27021d32a7B280d760B2bC2
Timeout
60s
exactPolygon (137)

$0.02 USDC

Max amount

Pay to
0x02d15E7f25d29743c27021d32a7B280d760B2bC2
Timeout
60s
exactArbitrum (42161)

$0.02 USDC

Max amount

Pay to
0x02d15E7f25d29743c27021d32a7B280d760B2bC2
Timeout
60s

Tags

crawlrobotswebpreflightagent

For agents

Prompt snippet for AI agents

Find and pay for this x402 resource:
URL: https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain
Type: http
Sources: cdp, thirdweb

Developer