Crawling

Crawling is the automatic browsing and capture of web pages or documents by a program. In messaging contexts, crawling is used to feed website content into an AI knowledge base automatically — instead of uploading every document manually.

What does crawling mean?

Crawling (from "to crawl") is the automatic process where a program (the "crawler" or "bot") systematically visits pages on the web, reads their content, and follows links. Google does it to build its index. AI platforms do it to populate a knowledge base automatically.

Crawling in the AI context

For an AI chatbot or AI agent to answer customer questions, it needs to know about your business: products, prices, shipping terms, FAQ, opening hours. Classically, this knowledge is uploaded laboriously, one document at a time. With crawling, a single input — the homepage URL — is enough. The crawler works through the linked subpages and stores them as searchable sources.

How an AI crawler works

1. Start URL: You provide the domain, e.g. www.yourcompany.com.
2. Check robots.txt: The crawler reads the robots.txt file and respects which areas it must not enter.
3. Use the sitemap: When available, it uses the XML sitemap as a map for efficient crawling.
4. Fetch pages: Each page is downloaded and the HTML parsed.
5. Extract content: Navigation, footers, cookie banners are stripped out; only the actual text and heading structure remain.
6. Index: Content is split into smaller, semantically meaningful chunks and stored in a vector database.
7. Updates: At configurable intervals the crawler re-checks whether content has changed.

What can go wrong with crawling

• JavaScript-rendered pages: Content loaded only via JavaScript is invisible to many crawlers. Fix: server-side rendering or a modern headless-browser crawler.
• Login-protected areas: Behind a login the crawler sees nothing — unless you provide credentials.
• Stale content: If the crawler runs only once, the knowledge base ages fast. Regular updates are mandatory.
• Structural issues: Without clear headings and semantic HTML, retrieval quality drops.

What robots.txt has to do with it

Every reputable crawler respects a domain's robots.txt. It declares which paths are off-limits. When crawling your own website, you set the rules. When crawling a third party, you must follow theirs.

Crawling vs. manual upload

Crawling at SendSeven

The SendSeven knowledge base can be filled from multiple sources: uploaded documents (PDF, DOCX), website crawls (with configurable update cadence), and direct entries. Crawled content is immediately available to the AI chatbot and AI agent as a basis for answers.