Preview a site crawl

Beyond Retell Discovers pages on a website and returns their URLs. Nothing is saved. Pick the pages you want and pass them to Add knowledge base sources as knowledge_base_urls, or set the site as a crawl_roots entry with auto_add_pages on.

Request#

How pages are found#

1. Tried reads /sitemap.xml on the URL's host, plus any nested sitemaps it lists (up to 10 sitemap files). Only URLs on the same host are kept. If this finds any pages, the result has source: "sitemap".
2. Otherwise it follows links breadth first from url, keeping http and https links on the same host and skipping links to images, CSS, JavaScript, PDF, ZIP and icon files. The result has source: "links".

Every fetch has a 10 second timeout and goes through the same egress guard as URL sources (see URL fetching). Pages that fail or are refused are skipped, so a blocked or unreachable site returns an empty urls list.

Example request#

curl -X POST https://api.tried.com.au/knowledge-base-crawl-preview \
  -H "Authorization: Bearer $TRIED_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com.au", "max_pages": 50}'

const res = await fetch("https://api.tried.com.au/knowledge-base-crawl-preview", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.TRIED_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://example.com.au", max_pages: 50 }),
});
const { urls, source } = await res.json();

import os, requests

res = requests.post(
    "https://api.tried.com.au/knowledge-base-crawl-preview",
    headers={"Authorization": f"Bearer {os.environ['TRIED_API_KEY']}"},
    json={"url": "https://example.com.au", "max_pages": 50},
)
preview = res.json()

Response#

200 OK

{
  "urls": [
    "https://example.com.au/",
    "https://example.com.au/services",
    "https://example.com.au/pricing",
    "https://example.com.au/contact"
  ],
  "source": "sitemap"
}

Errors#

See Errors for the body format.