Web Content

This section allows you to add online information to your AI agent's knowledge base. You can index entire websites, individual webpages, downloadable files, or manage existing indexed content.

Existing Content

The Existing Content tab displays all web-based sources that have already been indexed.

Here you can:

View the list of indexed items
Check whether each item has been embedded
Identify the content type (website, webpage, or file URL)
Delete entries you no longer need

This section serves as a management table for all previously added web content.

Websites

Use this option when you want to index an entire website domain. The system will crawl the domain, locate all accessible pages, and allow you to choose which ones to embed.

How to index a website

Step 1 — Enter the domain

Add one or more website domains using the Add another button. All domains must begin with https:// for security reasons.

Step 2 — Process the website

Switch to the Embed Data tab. Click Process Now to begin crawling the domain. Once completed, a pop-up window will appear displaying all discovered URLs.

Step 3 — Select webpages to embed

In the pop-up:

Tick individual pages
Or use Select All / Deselect All
Click the copy icon beside any URL if you want to preview it in your browser

When finished, click Confirm Selection.

Selected pages will now appear in the Embed Data tab.

Webpages

Use this option to index specific webpages rather than an entire site.

Features

Add multiple URLs manually using Add another
All URLs must start with https://
Supports Bulk upload (CSV) — prepare a CSV containing webpage URLs, upload it, and the system will index them automatically

Once uploaded, the webpages will appear in Embed Data for embedding.