x402 Bazaar Logox402 Bazaar
← Back to discover
httppayai

Agent Extraction API

Check whether a URL may be crawled per the site's robots.txt for a given user-agent. Returns allowed, matched rule, crawl-delay, sitemaps.

Resource URL

https://freetier-agent-api.pages.dev/api/robots-check

Payment options

exactBase (8453)

$0.003 USDC

Max amount

Pay to
0x165Eb3dC3E98403ecB1141e90400F5e71806554B
Timeout
300s

Tags

webrobots-txtcrawlingcompliance

For agents

Prompt snippet for AI agents

Find and pay for this x402 resource:
URL: https://freetier-agent-api.pages.dev/api/robots-check
Type: http
Sources: payai

Developer