httppayai

agents.writerslogic.com

Reads a supplied robots.txt the RFC 9309 way for an agent deciding whether to fetch up to 25 paths as one product-token user agent: group selection (merged groups, "*" fallback), longest-match with the Allow-wins-tie rule, "*" and "$" patterns, percent-encoding normalization, fetch-status handling (4xx = no rules, 5xx = disallow all), the rule and line that decided each path. It also lints a supplied llms.txt (llmstxt.org format) and lists supplied X-Robots-Tag / meta robots / noai / TDM-reservation signals. This is a reading of published rules, not permission: it does NOT fetch the file, interpret terms of service or licences, follow redirects, honour Crawl-delay, or prove the file you supplied is the live one.