Website Crawling With Firecrawl

Maven AGI uses Firecrawl to crawl a website you point it at and bring that content into its own knowledge base, with no manual exports or uploads.

What Maven AGI Does With Firecrawl

For content that lives on a website rather than in a platform like Confluence or Notion, Firecrawl gives Maven a generic way in. You set the scope, root URL, and path filters for a crawl source, and Maven converts matching pages into clean markdown in that source's own knowledge base, recrawling on the schedule you choose.

Key Capabilities

• Smart Crawling. Configure crawl depth, path filters, and sitemap support to scope exactly which pages of a site get crawled.
• Clean Markdown, One KB per Source. Each crawl source gets its own knowledge base. Pages are converted to clean markdown, stripping ads, navigation, and boilerplate.
• Scheduled Refreshes. Recrawl each source daily, weekly, or monthly, or refresh it on demand, so your knowledge base keeps pace with the live site.

How It Works

1. Add a source. Enter a root URL and set the crawl's scope and markdown options.
2. Crawl. Firecrawl follows links within that scope and converts pages to markdown.
3. Stay current. Set a refresh cadence, or refresh on demand, to keep the knowledge base in sync with the live site.

Common Use Cases

• Bring a public docs site or help center into Maven's knowledge base without a dedicated platform connector.
• Keep AI answers current as marketing or documentation pages change.
• Combine crawled web content with other knowledge sources in one Maven agent.

Related Integrations

• GitHub
• Confluence
• Notion
• ReadMe