Extraction

Extraction normalizes source content into document dicts that chunk() and ingest() can use.

from ragrails import RagRails

rag = RagRails()

Common Shape

All extraction methods return documents with the same core shape.

{
  "id": "doc_123",
  "text": "# Refund policy\n\nRefunds are available within 30 days.",
  "source": "files/refund-policy.pdf",
  "title": "Refund policy",
  "metadata": {"source_kind": "path", "file_type": "pdf"}
}

Parse Documents

Use parse() for local files, folders, remote file URLs, path dictionaries, or bytes from uploads/object storage.

result = rag.parse(files=["files/guide.pdf", "files/pricing.csv"])

result = rag.parse(folder="files/docs")

result = rag.parse(files="https://example.com/files/whitepaper.pdf")

with open("files/guide.pdf", "rb") as f:
    result = rag.parse(files=[{"content": f.read(), "filename": "guide.pdf", "title": "Guide"}])

Supported extensions include .pdf, .docx, .pptx, .xlsx, .csv, .md, .txt, .html, .json, .xml, .ipynb, .epub, .msg, .rss, .tsv, .xls, and .zip.

File URLs must end in a supported extension. Ragrails checks the URL path before downloading.

Scrape Websites

Website extraction needs the url extra and a browser runtime.

pip install "ragrails[url]"
ragrails setup-url --browser chromium

result = rag.scrape("https://example.com/help/refunds")

result = rag.scrape("https://example.com/docs", mode="full", max_depth=2, max_pages=50)

result = rag.scrape([
    "https://example.com/help/refunds",
    {"url": "https://example.com/docs", "mode": "full", "max_depth": 1, "max_pages": 25},
])

Always cap max_pages on full crawls.

Stream Website Extraction

for event in rag.scrape_stream("https://example.com/docs", mode="full", max_pages=50):
    if event["type"] == "page":
        print("scraped", event["data"]["url"])
    elif event["type"] == "error":
        print("error", event["message"])
    elif event["type"] == "final":
        result = event["data"]

scrape_stream() emits progress, page, error, and final events.

Dead-letter queue (DLQ)

Capture retryable scrape failures and retry only failed pages.

from ragrails import DLQ

first = rag.scrape("https://example.com/docs", mode="full", dlq=DLQ("files/dlq/web.json"))
retry = rag.scrape(dlq=first.dlq)
retry_from_file = rag.scrape(dlq="files/dlq/web.json")

Use non-streaming scrape(..., dlq=...) when you need DLQ capture.

Fetch REST APIs

fetch() turns API responses into document dicts. It supports headers, query params, JSON bodies, pagination config, timeouts, and batches of endpoints.

result = rag.fetch(
    url="https://api.example.com/refund-policy",
    title="Refund policy",
    headers={"Authorization": "Bearer token"},
)

result = rag.fetch(
    url="https://api.example.com/posts",
    pagination={"type": "page", "param": "page", "size_param": "per_page", "size": 100},
    max_pages=10,
)

result = rag.fetch(apis=[
    {"url": "https://api.example.com/posts", "title": "Posts"},
    {"url": "https://api.example.com/comments", "title": "Comments"},
])

Direct Markdown in `ingest()`

If your content is already Markdown, skip extraction methods and pass it directly to rag.ingest().

result = rag.ingest(markdown=[
    {"text": "# Refund policy\n\nCustomers can request a refund within 30 days.", "source": "policy.md"}
])

Save Outputs

All extraction methods can return in memory or save Markdown/JSON files.

result = rag.parse(
    files="files/refund-policy.pdf",
    output_format="json",
    output_dest="file",
    output_dir="files/output/docs",
)

result.outputs[0]["output_path"]

Result Fields

Result	Count fields	Data fields
`ScrapeResult`	`pages`, `failed`	`outputs`, `errors`, `dlq`
`ParseResult`	`documents`, `failed`	`outputs`, `errors`
`ApiIngestResult`	`documents`, `failed`	`outputs`, `errors`

Ingest

Run extraction through storage in one call.

Chunking

Split extracted documents into chunks.

Get Started

Workflows

Pipeline

Maintenance

Common Shape

Parse Documents

Scrape Websites

Stream Website Extraction

Dead-letter queue (DLQ)

Fetch REST APIs

Direct Markdown in `ingest()`

Save Outputs

Result Fields

Ingest

Chunking

​Common Shape

​Parse Documents

​Scrape Websites

​Stream Website Extraction

​Dead-letter queue (DLQ)

​Fetch REST APIs

​Direct Markdown in ingest()

​Save Outputs

​Result Fields

Ingest

Chunking

Common Shape

Parse Documents

Scrape Websites

Stream Website Extraction

Dead-letter queue (DLQ)

Fetch REST APIs

Direct Markdown in `ingest()`

Save Outputs

Result Fields