Crawler

Crawl your website to populate the knowledge base automatically. Crawling runs in the background; poll the status endpoint for progress. Authentication required.

POST /api/v1/sites/{id}/crawl Auth

Start a full-site crawl. Returns 409 if one is already running.

Response

{ "success": true, "message": "Crawling started" }

POST /api/v1/sites/{id}/crawl-url Auth

Crawl a single URL (must belong to the site’s domain) and add it to the knowledge base.

Request body

{ "url": "https://acme.com/pricing" }

Response

{ "success": true, "title": "Pricing", "content_length": 4200, "tokens": 1050, "ai_processed": true }

GET /api/v1/sites/{id}/crawl-status Auth

Current crawl status and counts.

Response

{ "success": true, "status": "crawling", "pages_crawled": 12, "knowledge_chunks": 48, "last_crawled_at": "2026-06-01T10:00:00Z" }