Chat

Chat is part of the Query workflow. It retrieves relevant chunks, builds grounded context, calls an LLM, and returns an answer with sources, updated history, confidence metadata, and any stage errors. Use this page to understand chat behavior. Use the interface-specific pages when you need exact parameters, flags, request fields, or response shapes.

Chat is stateless. Ragrails does not persist conversations. Pass history into each turn and store the returned history wherever your application keeps session state.

Basic chat

from ragrails import RagRails

rag = RagRails(
    collection="docs",
    vector_store={"provider": "qdrant", "url": "http://localhost:6333"},
    embedding={"provider": "voyage", "model": "voyage-3"},
    llm={"provider": "openai", "model": "gpt-4o-mini"},
)

history = []
result = rag.chat("How do I authenticate?", history=history)
print(result.answer)

history = result.history

ragrails chat "How do I authenticate?"   --vector-db qdrant   --collection docs   --url http://localhost:6333   --embedder-provider voyage   --embedder-model voyage-3   --llm-provider openai   --llm-model gpt-4o-mini   --history-file files/chat-history.json

curl -X POST http://127.0.0.1:8000/v1/chat   -H "Content-Type: application/json"   -d '{
    "query": "How do I authenticate?",
    "llm_provider": "openai",
    "llm_model": "gpt-4o-mini",
    "embedder_provider": "voyage",
    "embedder_model": "voyage-3",
    "vector_db": "qdrant",
    "collection": "docs",
    "url": "http://localhost:6333",
    "history": []
  }'

Result:

{
  "answer": "Create an API key and include it as a Bearer token in the Authorization header.",
  "sources": [
    {
      "id": "auth#overview",
      "title": "Authentication",
      "score": 0.84,
      "metadata": {"source": "docs/auth.md"}
    }
  ],
  "history": [
    {"role": "user", "content": "How do I authenticate?"},
    {"role": "assistant", "content": "Create an API key and include it as a Bearer token in the Authorization header."}
  ],
  "retrieval": {
    "query": "How do I authenticate?",
    "search_query": "How do I authenticate?",
    "retrieved": 3,
    "failed": 0,
    "outputs": [],
    "errors": []
  },
  "llm": {"provider": "openai", "model": "gpt-4o-mini"},
  "errors": [],
  "retrieval_quality": {"status": "passed", "passed_chunks": 3},
  "answer_confidence": {"level": "high", "reason": "retrieval_quality_pass"},
  "compacted": false,
  "intent": "rag"
}

Key	Meaning
`answer`	Generated response. Empty when retrieval or generation fails before an answer can be produced.
`sources`	Source chunks used to build the grounded prompt. Use this for citations.
`history`	Updated chat history. Persist it and pass it into the next turn.
`retrieval`	Raw retrieval stage summary, including `search_query`, counts, outputs, and retrieval errors.
`llm`	LLM provider metadata returned by the generation stage.
`errors`	Chat, retrieval, quality, or generation errors.
`retrieval_quality`	Quality gate status after retrieval.
`answer_confidence`	UI-friendly confidence level and reason.
`compacted`	Whether old history was summarized during the turn.
`intent`	`"rag"` for retrieval-backed answers, or a direct intent such as greeting/thanks when retrieval is bypassed.

Streaming

chat_stream() yields progress events, token events, error events, and one final event containing the same ChatResult shape.

for event in rag.chat_stream("How do I authenticate?", history=[]):
    if event["type"] == "token":
        print(event["data"]["text"], end="")
    elif event["type"] == "final":
        result = event["data"]

# The CLI chat command prints the completed answer.
ragrails chat "How do I authenticate?"   --vector-db qdrant   --collection docs   --url http://localhost:6333

curl -N -X POST http://127.0.0.1:8000/v1/chat/stream   -H "Content-Type: application/json"   -d '{
    "query": "How do I authenticate?",
    "collection": "docs",
    "url": "http://localhost:6333",
    "history": []
  }'

Example event sequence:

{"type":"progress","stage":"chat","message":"Chat started","data":{"query":"How do I authenticate?"},"sequence":1}
{"type":"progress","stage":"retrieval","message":"Retrieval complete","data":{"retrieved":3,"failed":0,"search_query":"How do I authenticate?"},"sequence":3}
{"type":"token","stage":"generation","message":"","data":{"text":"Create"},"sequence":5}
{"type":"final","stage":"complete","message":"Chat complete","data":{"answer":"Create an API key..."},"sequence":32}

Conversation features

Feature	Default	What it does
History compaction	Enabled	Summarizes older turns when history grows past `history_limit`, keeping recent turns intact.
Query rewriting	Disabled	Rewrites follow-ups into standalone search queries before retrieval.
Intent routing	Enabled	Bypasses retrieval for simple greetings, thanks, farewells, and acknowledgements.
Retrieval quality	Enabled	Filters weak context and controls low-confidence answer behavior.
Reranking	Disabled	Reorders retrieved chunks before the answer prompt is built.

Query rewriting

Use query rewriting when follow-up questions need previous context to search correctly.

from ragrails import QueryRewriteConfig

result = rag.chat(
    "What about the second step?",
    history=history,
    query_rewrite=QueryRewriteConfig(
        enabled=True,
        session_context="The user is asking about onboarding setup.",
    ),
)

ragrails chat "What about the second step?"   --history-file files/chat-history.json   --rewrite-query   --rewrite-session-context "The user is asking about onboarding setup."   --vector-db qdrant   --collection docs   --url http://localhost:6333

curl -X POST http://127.0.0.1:8000/v1/chat   -H "Content-Type: application/json"   -d '{
    "query": "What about the second step?",
    "query_rewrite": {
      "enabled": true,
      "session_context": "The user is asking about onboarding setup."
    },
    "collection": "docs",
    "url": "http://localhost:6333",
    "history": []
  }'

Reranked chat

Reranking retrieves a wider candidate set, reorders it, and keeps the top reranked chunks for answer generation.

from ragrails.core.stg_05_retriever import RetrieverConfig

result = rag.chat(
    "What is the refund window?",
    history=history,
    retrieval_config=RetrieverConfig(use_rerank=True, rerank_top_k=5),
)

ragrails chat "What is the refund window?"   --vector-db qdrant   --collection docs   --url http://localhost:6333   --rerank   --reranker voyage   --reranker-model rerank-2-lite   --rerank-top-k 5

curl -X POST http://127.0.0.1:8000/v1/chat   -H "Content-Type: application/json"   -d '{
    "query": "What is the refund window?",
    "collection": "docs",
    "url": "http://localhost:6333",
    "rerank": true,
    "reranker": "voyage",
    "reranker_model": "rerank-2-lite",
    "rerank_top_k": 5,
    "history": []
  }'

Retrieval quality

Retrieval quality decides how chat behaves when the retrieved context is weak.

from ragrails import ChatRetrievalQualityConfig

result = rag.chat(
    "Can I get a refund after 30 days?",
    history=history,
    retrieval_quality=ChatRetrievalQualityConfig(
        min_retrieval_score=0.35,
        min_rerank_score=0.50,
        low_confidence_mode="answer_with_caution",
        max_context_chunks=5,
    ),
)

# The CLI does not expose retrieval-quality thresholds yet.
ragrails chat "Can I get a refund after 30 days?"   --vector-db qdrant   --collection docs   --url http://localhost:6333

curl -X POST http://127.0.0.1:8000/v1/chat   -H "Content-Type: application/json"   -d '{
    "query": "Can I get a refund after 30 days?",
    "collection": "docs",
    "url": "http://localhost:6333",
    "retrieval_quality": {
      "min_retrieval_score": 0.35,
      "min_rerank_score": 0.50,
      "low_confidence_mode": "answer_with_caution",
      "max_context_chunks": 5
    },
    "history": []
  }'

Low-confidence modes:

Mode	Behavior
`answer_with_caution`	Generate an answer, but confidence is low and the prompt asks for caution.
`ask_clarifying_question`	Ask the user for a narrower question.
`refuse_grounded_answer`	Refuse to answer from weak context.
`return_no_answer`	Return an empty answer and a quality error.

Possible errors

Most stage-level failures are returned in the result errors list instead of raising immediately. REST validation and setup failures may return an exception envelope instead. See Errors for the shared shapes.

{
  "answer": "",
  "sources": [],
  "history": [],
  "retrieval": {"retrieved": 0, "failed": 0, "outputs": [], "errors": []},
  "llm": {},
  "errors": [
    {
      "source": "",
      "source_kind": "chat",
      "stage": "validate",
      "error": "query must be a non-empty string",
      "isRetryable": false,
      "attempts": 1
    }
  ],
  "retrieval_quality": {"status": "not_evaluated"},
  "answer_confidence": {"level": "none", "reason": "errors"},
  "compacted": false,
  "intent": "rag"
}

{
  "answer": "",
  "sources": [],
  "history": [],
  "retrieval": {
    "retrieved": 0,
    "failed": 1,
    "outputs": [],
    "errors": [
      {
        "source": "How do I authenticate?",
        "source_kind": "query",
        "stage": "retrieve",
        "error": "Collection not found: docs",
        "isRetryable": true,
        "attempts": 1
      }
    ]
  },
  "llm": {},
  "errors": [
    {
      "source": "How do I authenticate?",
      "source_kind": "query",
      "stage": "retrieve",
      "error": "Collection not found: docs",
      "isRetryable": true,
      "attempts": 1
    }
  ],
  "retrieval_quality": {"status": "not_evaluated"},
  "answer_confidence": {"level": "none", "reason": "errors"},
  "compacted": false,
  "intent": "rag"
}

{
  "answer": "",
  "sources": [],
  "history": [{"role": "user", "content": "Can I get a refund after 30 days?"}],
  "retrieval": {"retrieved": 1, "failed": 0, "outputs": [], "errors": []},
  "llm": {},
  "errors": [
    {
      "source": "",
      "source_kind": "chat",
      "stage": "quality",
      "error": "retrieval quality below configured threshold",
      "isRetryable": false,
      "attempts": 1
    }
  ],
  "retrieval_quality": {"status": "low_confidence", "mode": "return_no_answer", "passed_chunks": 0},
  "answer_confidence": {"level": "none", "reason": "retrieval_quality_low_confidence"},
  "compacted": false,
  "intent": "rag"
}

Next steps

Use Retrieval when you want chunks without answer generation.
Use Pipeline Overview to see how chat fits into the full RAG flow.
Use Chat Tuning for deeper quality and prompt behavior.

Get Started

Features

Capabilities

Use Cases

Walkthroughs

Basic chat

Streaming

Conversation features

Query rewriting

Reranked chat

Retrieval quality

Possible errors

Next steps

​Basic chat

​Streaming

​Conversation features

​Query rewriting

​Reranked chat

​Retrieval quality

​Possible errors

​Next steps

Basic chat

Streaming

Conversation features

Query rewriting

Reranked chat

Retrieval quality

Possible errors

Next steps