httpultravioletadao

x402-markdown.onrender.com

Before fetching, quoting or training on a web page, check the site's machine-readable AI/crawl signals for your crawler. GET /v1/crawl/permission?url=<public http(s) URL>&ua=<robots.txt product token, e.g. GPTBot, ClaudeBot, CCBot, Google-Extended; default *>. Evaluates robots.txt per RFC 9309 (group for ua else *, longest match, Allow wins ties; robots.txt 4xx = no restrictions, 5xx = treat as disallowed), ai.txt, TDMRep (/.well-known/tdmrep.json, tdm-reservation/tdm-policy headers and meta), X-Robots-Tag and <meta name=robots|<ua>> (noai, noimageai, noindex...). Returns verdict 'allowed' | 'disallowed' | 'unclear' plus reasons[] and every raw signal. Disallowed if robots.txt blocks the path for ua or any AI opt-out (noai, tdm-reservation=1, ai.txt disallow) is present; unclear if the page itself could not be checked. Errors are JSON {error:<code>, message, charged:false}. Not charged when robots.txt cannot be retrieved (DNS, connect, timeout) or the URL is not public (400 before payment). Not a legal opinion.