Build From a Website

Use this walkthrough when the source of truth is a public or internal website: documentation, help centers, marketing pages, handbooks, or knowledge portals.

What you will build

A website collection in your vector database, backed by crawled pages from https://example.com, with retrieval and chat on top.

Prerequisites

pip install "ragrails[url,voyage,qdrant]"
export VOYAGE_API_KEY="..."
export OPENAI_API_KEY="..."
docker run -p 6333:6333 qdrant/qdrant

URL crawling uses a browser runtime. Install one once:

ragrails setup-url --browser chromium

1. Ingest the website

from ragrails import RagRails

rag = RagRails(
    collection="website",
    vector_store={"provider": "qdrant", "url": "http://localhost:6333"},
    embedding={"provider": "voyage", "model": "voyage-3"},
    llm={"provider": "openai", "model": "gpt-4o-mini"},
)

result = rag.ingest(
    urls="https://example.com",
    ingestion={"urls": {"mode": "full", "max_depth": 3, "max_pages": 200}},
    chunking={"chunk_size": 900, "chunk_overlap": 120},
    storage={"collection": "website"},
)

print(result.sources, result.chunks, result.stored)

ragrails ingest \
  --source-url https://example.com \
  --vector-db qdrant \
  --collection website \
  --url http://localhost:6333 \
  --provider voyage \
  --model voyage-3 \
  --chunk-size 900 \
  --chunk-overlap 120

curl -X POST http://127.0.0.1:8000/v1/pipelines/ingest \
  -H "Content-Type: application/json" \
  -d '{
    "urls": "https://example.com",
    "ingestion": {"urls": {"mode": "full", "max_depth": 3, "max_pages": 200}},
    "chunking": {"chunk_size": 900, "chunk_overlap": 120},
    "embedding": {"provider": "voyage", "model": "voyage-3"},
    "storage": {"vector_db": "qdrant", "collection": "website", "url": "http://localhost:6333"}
  }'

For long crawls, use Streaming: rag.scrape_stream(...) or POST /v1/ingest/url/stream.

2. Verify retrieval

result = rag.query(
    "What does the site say about pricing?",
    retrieval={"top_k": 10, "rerank": {"enabled": True, "top_k": 5}},
)

for item in result.items:
    print(item.rerank_score, item.metadata.get("title"), item.text[:120])

ragrails query "What does the site say about pricing?" \
  --vector-db qdrant \
  --collection website \
  --url http://localhost:6333 \
  --provider voyage \
  --model voyage-3 \
  --top-k 10 \
  --rerank \
  --rerank-top-k 5

curl -X POST http://127.0.0.1:8000/v1/pipelines/query \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What does the site say about pricing?",
    "embedding": {"provider": "voyage", "model": "voyage-3"},
    "retrieval": {
      "vector_db": "qdrant",
      "collection": "website",
      "url": "http://localhost:6333",
      "top_k": 10,
      "rerank": {"enabled": true, "top_k": 5}
    }
  }'

3. Add chat

history = []
answer = rag.chat(
    "Summarize the pricing tiers.",
    history=history,
    persona="Answer only from the crawled website context.",
)
print(answer.answer)
print(answer.sources)

Keep it current

Websites change. Re-crawl on a schedule, store new chunks, and delete stale IDs using Knowledge Base Maintenance.

Get Started

Features

Capabilities

Use Cases

Walkthroughs

Build From a Website

What you will build

Prerequisites

1. Ingest the website

2. Verify retrieval

3. Add chat

Keep it current

Next steps

​What you will build

​Prerequisites

​1. Ingest the website

​2. Verify retrieval

​3. Add chat

​Keep it current

​Next steps

What you will build

Prerequisites

1. Ingest the website

2. Verify retrieval

3. Add chat

Keep it current

Next steps