NetIntel
Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate s
Go to this service →Our latest check (2026-10-11)
- Answering
- Yes — valid x402 payment terms
- Time to answer
- 185 ms
- Price
- $0.010 per call
- Paying buyers, last 30 days
- 3 (directory's own records)
- Paid calls, last 30 days
- 4
- Category
- DNS and domains
- Address
- https://netintel.dev/robots-txt/analyze
Accuracy: paid answers checked against an independent record
Not yet in our paid checks. A seller we have not checked has neither passed nor failed.
For agents
This page as JSON: /sellers/da46c1b0d125.json. Paid lookup by address: GET /api/seller?resource=… ($0.001, x402).