Skip to main content

​
The problem

Competitors update pricing, features, and positioning constantly. Manually checking their sites doesn’t scale, and the info goes stale fast.

​
How Ragrails solves it

Crawl competitor sites and news on a schedule, recover failures automatically, and keep the index fresh.
  1. Full-crawl competitor sites with a dead-letter queue so partial failures are recoverable.
  2. Refresh on a schedule: re-ingest, and delete() pages that disappeared.
  3. Query or chat to compare and summarize.
from ragrails import RagRails, DLQ

rag = RagRails()
rag.setup_url()

result = rag.scrape(
    ["https://competitor-a.com", "https://competitor-b.com"],
    mode="full", max_pages=200,
    dlq=DLQ("files/dlq/competitors.json"),   # capture failures
)
# later: retry just the failures
rag.scrape(dlq="files/dlq/competitors.json")
Schedule the crawl (cron, a worker) and use edit()/delete() so answers always reflect the latest competitor state.

​
Features used

URL ingestion · Resilient ingestion · Keeping your index current