Check robots.txt crawl rules
Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any decl
Go to this service →Our latest check (2026-10-11)
- Answering
- Yes — valid x402 payment terms
- Time to answer
- 410 ms
- Price
- $0.006 per call
- Paying buyers, last 30 days
- 3 (directory's own records)
- Paid calls, last 30 days
- 3
- Category
- Web page readers and scrapers
- Address
- https://web.openverbs.com/v1/robots
Accuracy: paid answers checked against an independent record
Not yet in our paid checks. A seller we have not checked has neither passed nor failed.
For agents
This page as JSON: /sellers/634d27c2f431.json. Paid lookup by address: GET /api/seller?resource=… ($0.001, x402).