Fetch and parse robots.txt for a domainGET /web/robots · x402 API by x402node
Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use ?domain=example.com.
What does this x402 API do?
Fetch and parse robots.txt for a domain. This is an x402-gated GET API resource from x402node. TOLL·402 confirmed a protocol-valid HTTP 402 quote on 2026-07-21. No normalized USD price is recorded; the listing names Base and Solana. No settled paid call has been verified.
- Published price
- No normalized USD price is recorded
- Networks
- Base · Solana
- Latest evidence
- TOLL·402 confirmed a protocol-valid HTTP 402 quote on 2026-07-21
- Resource ID
- 143eba6996cb33d75a400cf7
Start with a quote-only recipe. Payment signing stays in your project.
const response = await fetch("https://api.x402node.dev/web/robots", {
method: "GET",
});
console.log(response.status);
console.log(Object.fromEntries(response.headers));
console.log((await response.text()).slice(0, 1000));
// A valid 402 is a quote, not a completed paid call.This sends no payment. Inspect the 402 response before adding a wallet-enabled client.
- Live 402 quote confirmed
GET returned a protocol-valid 402 quote (402). No payment was made.
- Origin reachable
HEAD https://api.x402node.dev/ returned 404.
- Observed in cdp-bazaar
The registry record was observed and retained with provenance.
- Observed in 402-index
Discovered through bazaar.
- Registry record checked
Registry health: healthy.
What evidence supports this listing?
Each record has an exact-route quote outcome. Unresolved templates and potentially mutating methods are labeled instead of being invoked without provider-specific test input. A quote is still separate from a settled paid call. Read the discovery and verification methodology →
Which networks and payment options does it accept?
The normalized listing price is No normalized USD price is recorded. Raw protocol requirements remain visible below for implementation and audit use.