togotogo
Marketplace

Crawlee

Other

Web crawler (colly) → page text for RAG.

togo-framework
bash
togo install togo-framework/ai-crawlee

Install

bash
togo install togo-framework/ai-crawlee

ai-crawlee

A togo AI data-source plugin — crawl a website and extract page text for ai-rag ingest and agents. Crawlee is a Node library; this is a Go-native crawler (colly) serving the same purpose.

bash
togo install togo-framework/ai-crawlee

Use

  • Go: crawlee.FromKernel(k).Crawl(ctx, "https://example.com", 20) → []Page{URL,Title,Text}
  • REST: POST /api/ai/crawlee {"url":"…","maxPages":20}

Part of the togo AI kit. MIT.

<div align="center"> <h3>Premium sponsors</h3> <p> <a href="https://id8media.com"><strong>ID8 Media</strong></a> &nbsp;·&nbsp; <a href="https://one-studio.co"><strong>One Studio</strong></a> </p> <p><sub>Support togo — <a href="https://github.com/sponsors/fadymondy">become a sponsor</a>.</sub></p> </div>