Robots.txt Analyzer
Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate sitemaps before scraping.
Start with a quote-only recipe. Payment signing stays in your project.
const response = await fetch("https://netintel-production-440c.up.railway.app/robots-txt/analyze", {
method: "GET",
});
console.log(response.status);
console.log(Object.fromEntries(response.headers));
console.log((await response.text()).slice(0, 1000));
// A valid 402 is a quote, not a completed paid call.This sends no payment. Inspect the 402 response before adding a wallet-enabled client.
- Exact route checked
http 402 invalid · accept-0:resource-mismatch;accepts-missing
- Origin reachable
HEAD https://netintel-production-440c.up.railway.app/ returned 200.
- Observed in cdp-bazaar
The registry record was observed and retained with provenance.
- Observed in 402-index
Discovered through bazaar.
- Registry record checked
Registry health: healthy.
EVIDENCE
Each record has an exact-route quote outcome. Unresolved templates and potentially mutating methods are labeled instead of being invoked without provider-specific test input. A quote is still separate from a settled paid call.