How to Read Any Site’s robots.txt as JSON

Follow the steps below and use the linked tools and references as you go.

Evidence trail: Robots.txt & Sitemap Parser API on RapidAPI · Web Content & URL Tools MCP Server on Apify

Before you crawl a site, read its robots.txt. One GET request parses it into user-agent groups and the sitemaps it lists, and can test whether a path is allowed.

Subscribe to the free plan on RapidAPI, then call the endpoint with your RapidAPI key:

curl -G 'https://robots-txt-sitemap-parser.p.rapidapi.com/v1/robots' \
  --data-urlencode 'url=https://avaloncompany.ai/' \
  -H 'x-rapidapi-host: robots-txt-sitemap-parser.p.rapidapi.com' \
  -H 'x-rapidapi-key: YOUR_RAPIDAPI_KEY'

We ran the same check against our own site while writing this post. The key fields of the real response:

{
  "exists": true,
  "status_code": 200,
  "user_agent_groups": 1,
  "sitemaps_listed": 11
}

import requests

resp = requests.get(
    'https://robots-txt-sitemap-parser.p.rapidapi.com/v1/robots',
    params={'url': 'https://avaloncompany.ai/'},
    headers={'x-rapidapi-host': 'robots-txt-sitemap-parser.p.rapidapi.com', 'x-rapidapi-key': 'YOUR_RAPIDAPI_KEY'},
    timeout=30,
)
resp.raise_for_status()
print(resp.json())

• Check whether your crawler or AI agent is allowed to fetch a path before it does.
• Find the sitemaps a site declares, to discover its pages without crawling links.
• Catch an accidental 'Disallow: /' after a deploy.

• robots.txt is a request, not access control: it tells polite crawlers what to skip.
• A missing robots.txt is reported as such and means everything is allowed.

The same endpoint is one of the tools in the Web Content & URL Tools MCP Server on Apify. Point any client that supports remote MCP servers at `https://avalonai–web-content-url-tools-mcp.apify.actor/mcp` with your Apify token (`YOUR_APIFY_TOKEN`) and the agent can call it directly: https://apify.com/avalonai/web-content-url-tools-mcp

Robots.txt & Sitemap Parser API on RapidAPI (free plan available): https://rapidapi.com/avalonai/api/robots-txt-sitemap-parser

Our APIs on RapidAPI: https://rapidapi.com/user/avalonai

• Robots.txt & Sitemap Parser API — produced the example response in this tutorial (run against avaloncompany.ai): https://rapidapi.com/avalonai/api/robots-txt-sitemap-parser