https://staging.402utils.com/v1/robots-check
Robots.txt compliance check for polite crawlers and AI agents (GPTBot, ClaudeBot, ...). Pass a page URL and optionally your crawler's product token; returns whether the URL may be fetched, the winning Allow/Disallow rule, crawl-delay and sitemap URLs. Full RFC 9309 matching: wildcards, $ anchors, longest-match, merged groups. A 4xx robots.txt allows all; 5xx or unreachable hosts report disallowed. Cached per host for 1 hour.
Start with a quote-only recipe. Payment signing stays in your project.
const response = await fetch("https://staging.402utils.com/v1/robots-check", {
method: "GET",
});
console.log(response.status);
console.log(Object.fromEntries(response.headers));
console.log((await response.text()).slice(0, 1000));
// A valid 402 is a quote, not a completed paid call.This sends no payment. Inspect the 402 response before adding a wallet-enabled client.
- Live 402 quote confirmed
GET returned a protocol-valid 402 quote (402). No payment was made.
- Origin reachable
HEAD https://staging.402utils.com/ returned 200.
- Observed in cdp-bazaar
The registry record was observed and retained with provenance.
- Observed in 402-index
Discovered through bazaar.
- Registry record checked
Registry health: healthy.
EVIDENCE
Each record has an exact-route quote outcome. Unresolved templates and potentially mutating methods are labeled instead of being invoked without provider-specific test input. A quote is still separate from a settled paid call.