Robots.txt Analyzer
Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate sitemaps before scraping.
Start with a quote-only recipe. Payment signing stays in your project.
const response = await fetch("https://netintel.dev/robots-txt/analyze", {
method: "GET",
});
console.log(response.status);
console.log(Object.fromEntries(response.headers));
console.log((await response.text()).slice(0, 1000));
// A valid 402 is a quote, not a completed paid call.This sends no payment. Inspect the 402 response before adding a wallet-enabled client.
- Live 402 quote confirmed
GET returned a protocol-valid 402 quote (402). No payment was made.
- Origin reachable
HEAD https://netintel.dev/ returned 200.
- Observed in cdp-bazaar
The registry record was observed and retained with provenance.
- Observed in 402-index
Discovered through bazaar.
- Registry record checked
Registry health: healthy.
EVIDENCE
Each record has an exact-route quote outcome. Unresolved templates and potentially mutating methods are labeled instead of being invoked without provider-specific test input. A quote is still separate from a settled paid call.