Extraction

Extraction is part of the Ingest workflow. It loads source content and normalizes it into Markdown the rest of the pipeline can use. Use this page to choose the right extraction path. Use the interface-specific pages when you need every parameter, flag, request field, or response shape.

Source forms

Method	Source	Best for	Notes
`scrape()`	Web pages and full sites	Docs sites, help centers, marketing pages	Needs `ragrails[url]` and one-time browser setup
`scrape_stream()`	Web pages with live events	Long crawls where you want progress	SDK and REST streaming are supported
`parse()`	Documents	PDFs, DOCX, PPTX, XLSX, HTML, Markdown, TXT, CSV, JSON, XML, ZIP, and more	Accepts paths, file URLs, folders, and bytes
`fetch()`	REST API responses	Product catalogs, support articles, OpenAPI JSON, paginated APIs	Supports headers, params, JSON bodies, pagination, and batch endpoints in SDK
`markdown`	Already-normalized text	Generated docs, fixtures, CMS exports	Used directly by `ingest()`, not a standalone extraction method

Website scraping needs pip install "ragrails[url]" and a one-time rag.setup_url() or ragrails setup-url in the environment that runs the scraper. The browser can be chromium, firefox, or webkit.

Common shape

All extraction methods return normalized document objects. Downstream chunking only needs the text field, but the metadata keeps source context for retrieval and citations.

{
  "id": "doc_123",
  "text": "# Refund policy\n\nRefunds are available within 30 days.",
  "source": "files/docs/refund-policy.pdf",
  "title": "Refund policy",
  "metadata": {
    "source_kind": "path",
    "file_type": "pdf"
  }
}

Scrape websites

Use scrape() for exact URLs or full-site crawls.

from ragrails import DLQ, RagRails

rag = RagRails()
rag.setup_url()

# One exact URL
one = rag.scrape("https://example.com/docs")

# Multiple exact URLs
many = rag.scrape([
    "https://example.com/docs",
    "https://example.com/blog",
])

# Per-URL config: mix exact pages and full crawls
mixed = rag.scrape([
    "https://example.com/docs/auth",
    {"url": "https://example.com/docs", "mode": "full", "max_depth": 2, "max_pages": 50},
])

# Save retryable failures, then retry only those pages later
first = rag.scrape("https://example.com/docs", mode="full", dlq=DLQ("files/dlq/web.json"))
retry = rag.scrape(dlq="files/dlq/web.json")

# Save extracted pages to JSON files
saved = rag.scrape(
    "https://example.com/docs",
    output_format="json",
    output_dest="file",
    output_dir="files/output/web/",
)

ragrails setup-url

# Exact URLs
ragrails scrape https://example.com/docs https://example.com/blog

# Full site crawl
ragrails scrape https://example.com/docs \
  --mode full \
  --max-depth 2 \
  --max-pages 50 \
  --output-dir files/output/web/

# Add YAML frontmatter to saved markdown output
ragrails scrape https://example.com/docs \
  --frontmatter \
  --output-dir files/output/web/

curl -X POST http://127.0.0.1:8000/v1/ingest/url \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/docs", "mode": "full", "max_depth": 2, "max_pages": 50}'

curl -X POST http://127.0.0.1:8000/v1/ingest/url \
  -H "Content-Type: application/json" \
  -d '{"url": ["https://example.com/docs", {"url": "https://example.com/blog", "mode": "full", "max_depth": 1}]}'

Always cap max_pages on a full crawl. Sites can be huge, and every page costs a fetch.

Dead-letter queue retry is SDK-first. REST returns errors and a dlq payload, but you resubmit failed URLs yourself. The CLI does not expose DLQ retry flags.

Stream website extraction

Use streaming when a crawl may take a while and the caller should see progress before the final result.

for event in rag.scrape_stream("https://example.com/docs", mode="full", max_pages=50):
    if event["type"] == "page":
        print("scraped", event["data"]["url"])
    elif event["type"] == "error":
        print("error", event["message"])
    elif event["type"] == "final":
        print("pages", event["data"]["pages"])

curl -N -X POST http://127.0.0.1:8000/v1/ingest/url/stream \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/docs", "mode": "full", "max_pages": 50}'

Streaming yields progress/page/error events and ends with a final event containing the same aggregate shape as scrape().

Parse documents

Use parse() for local files, folders, remote file URLs, path dictionaries with metadata, or raw bytes from uploads/object storage.

from ragrails import RagRails

rag = RagRails()

# One or more local files
local = rag.parse(files=["files/guide.pdf", "files/pricing.csv"])

# Folder: all supported files directly inside the folder
folder = rag.parse(folder="files/docs/")

# Remote file URL; the URL path must end with a supported extension
remote = rag.parse(files="https://example.com/files/whitepaper.pdf")

# Path dicts add title/description metadata
with_metadata = rag.parse(files=[
    {"path": "files/refund-policy.pdf", "title": "Refund policy", "description": "Customer refund rules"},
    {"path": "https://example.com/files/product-guide.pdf", "title": "Product guide"},
])

# Raw bytes, useful for web uploads or object storage
with open("files/guide.pdf", "rb") as f:
    data = f.read()
bytes_result = rag.parse(files=[{"content": data, "filename": "guide.pdf", "title": "Guide"}])

# Mixed inputs in one call
mixed = rag.parse(files=[
    "files/local.pdf",
    "https://example.com/files/remote.docx",
    {"content": data, "filename": "upload.pdf", "source": "s3://bucket/upload.pdf"},
])

# Return markdown with YAML frontmatter, or save JSON/markdown files
frontmatter = rag.parse(files="files/guide.pdf", frontmatter=True)
saved = rag.parse(files="files/guide.pdf", output_format="json", output_dest="file", output_dir="files/output/docs/")

# Folder
ragrails parse --folder files/docs/ --output-dir files/output/docs/

# Specific paths or file URLs
ragrails parse \
  --files files/guide.pdf \
  --files https://example.com/files/whitepaper.pdf \
  --output-dir files/output/docs/

# Add frontmatter to saved markdown output
ragrails parse --files files/guide.pdf --frontmatter --output-dir files/output/docs/

# Server-side path, folder, or remote file URL
curl -X POST http://127.0.0.1:8000/v1/ingest/docs \
  -H "Content-Type: application/json" \
  -d '{"files": ["files/guide.pdf", "https://example.com/files/whitepaper.pdf"]}'

curl -X POST http://127.0.0.1:8000/v1/ingest/docs \
  -H "Content-Type: application/json" \
  -d '{"folder": "files/docs/"}'

# Client upload through multipart
curl -X POST http://127.0.0.1:8000/v1/ingest/docs/upload \
  -F "files=@docs/guide.pdf" \
  -F "frontmatter=false" \
  -F "title=Guide" \
  -F "description=Product guide"

Use JSON /v1/ingest/docs for files reachable by the server process. Use multipart /v1/ingest/docs/upload when a client is uploading file bytes to the API.

File URLs must end in a supported extension such as .pdf, .md, .docx, or .xlsx; Ragrails checks the URL path before downloading.

Fetch REST APIs

Use fetch() for API responses. Each fetched page becomes a document, so paginated endpoints produce multiple documents.

from ragrails import RagRails

rag = RagRails()

# Single GET endpoint
posts = rag.fetch(url="https://api.example.com/posts", title="Blog posts")

# Headers, params, JSON body, method, timeout
search = rag.fetch(
    url="https://api.example.com/search",
    title="Search results",
    method="POST",
    headers={"Authorization": "Bearer token"},
    params={"locale": "en-US"},
    body={"query": "refund policy"},
    timeout=30.0,
)

# Page-number pagination
paged = rag.fetch(
    url="https://api.example.com/posts",
    title="Posts",
    pagination={"type": "page", "param": "page", "size_param": "per_page", "size": 100},
    max_pages=10,
)

# Offset pagination
offset = rag.fetch(
    url="https://api.example.com/events",
    pagination={"type": "offset", "param": "offset", "size_param": "limit", "size": 100},
    max_pages=20,
)

# Cursor pagination
cursor = rag.fetch(
    url="https://api.example.com/messages",
    pagination={"type": "cursor", "param": "cursor", "cursor_path": "meta.next_cursor"},
    max_pages=20,
)

# Batch endpoints are SDK-only
batch = rag.fetch(apis=[
    "https://api.example.com/posts",
    {"url": "https://api.example.com/comments", "title": "Comments", "max_pages": 5},
])

# Save output
saved = rag.fetch("https://api.example.com/posts", output_format="json", output_dest="file", output_dir="files/output/api/")

ragrails fetch https://api.example.com/posts \
  --title "Blog posts" \
  --description "Public posts API" \
  --method GET \
  --header "Authorization:Bearer TOKEN" \
  --param locale:en-US \
  --max-pages 10 \
  --output-dir files/output/api/

curl -X POST http://127.0.0.1:8000/v1/ingest/api \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://api.example.com/posts",
    "title": "Blog posts",
    "headers": {"Authorization": "Bearer token"},
    "params": {"locale": "en-US"},
    "pagination": {"type": "page", "param": "page", "size_param": "per_page", "size": 100},
    "max_pages": 10,
    "timeout": 30.0
  }'

Always set max_pages for paginated APIs. It stops pagination even if the upstream API keeps returning a next page.

The CLI covers common GET/header/query-param use. Use SDK or REST for JSON request bodies, cursor/offset pagination details, timeout control, or SDK batch apis=[...] ingestion.

Direct Markdown

If your content is already Markdown, skip extraction and pass it directly to the high-level ingest() workflow.

rag.ingest(markdown=[
    {
        "text": "# Refunds\n\nRefunds are available within 30 days.",
        "source": "policy.md",
        "metadata": {"title": "Refunds"},
    },
])

ragrails ingest \
  --markdown $'# Refunds\n\nRefunds are available within 30 days.' \
  --vector-db qdrant \
  --collection support \
  --url http://localhost:6333 \
  --provider voyage \
  --model voyage-3

curl -X POST http://127.0.0.1:8000/v1/pipelines/ingest \
  -H "Content-Type: application/json" \
  -d '{
    "markdown": [{"text": "# Refunds\n\nRefunds are available within 30 days.", "source": "policy.md", "metadata": {"title": "Refunds"}}],
    "embedding": {"provider": "voyage", "model": "voyage-3"},
    "storage": {"vector_db": "qdrant", "collection": "support", "url": "http://localhost:6333"}
  }'

Ingest pipeline

Use ingest() when extraction should immediately continue into chunking, embedding, and vector storage. The high-level pipeline accepts the same source families: docs, urls, api, and markdown.

result = rag.ingest(
    docs={"files": ["files/refund-policy.pdf", "files/pricing.csv"]},
    urls={"url": "https://example.com/docs", "mode": "full", "max_pages": 50},
    api={
        "url": "https://api.example.com/articles",
        "title": "Articles",
        "pagination": {"type": "page", "param": "page", "size_param": "per_page", "size": 100},
        "max_pages": 5,
    },
    markdown=[
        {"text": "# Manual note\n\nEscalations are handled by support.", "source": "manual.md"},
    ],
    concurrency="parallel",
    chunking={"chunk_size": 1200, "chunk_overlap": 120},
    embedding={"provider": "voyage", "model": "voyage-3", "batch_size": 64},
    storage={"vector_db": "qdrant", "collection": "support", "url": "http://localhost:6333"},
)

ragrails ingest \
  --docs files/refund-policy.pdf \
  --source-url https://example.com/docs \
  --api-url https://api.example.com/articles \
  --markdown $'# Manual note\n\nEscalations are handled by support.' \
  --concurrency parallel \
  --chunk-size 1200 \
  --chunk-overlap 120 \
  --provider voyage \
  --model voyage-3 \
  --batch-size 64 \
  --vector-db qdrant \
  --collection support \
  --url http://localhost:6333

curl -X POST http://127.0.0.1:8000/v1/pipelines/ingest \
  -H "Content-Type: application/json" \
  -d '{
    "docs": {"files": ["files/refund-policy.pdf", "files/pricing.csv"]},
    "urls": {"url": "https://example.com/docs", "mode": "full", "max_pages": 50},
    "api": {
      "url": "https://api.example.com/articles",
      "title": "Articles",
      "pagination": {"type": "page", "param": "page", "size_param": "per_page", "size": 100},
      "max_pages": 5
    },
    "markdown": [{"text": "# Manual note\n\nEscalations are handled by support.", "source": "manual.md"}],
    "concurrency": "parallel",
    "chunking": {"chunk_size": 1200, "chunk_overlap": 120},
    "embedding": {"provider": "voyage", "model": "voyage-3", "batch_size": 64},
    "storage": {"vector_db": "qdrant", "collection": "support", "url": "http://localhost:6333"}
  }'

concurrency="parallel" runs independent extraction sources at the same time before chunking. Use serial when source order or upstream rate limits matter more than speed.

The CLI pipeline exposes common extraction flags. Use SDK or REST when you need full per-source extraction options such as URL max_pages, API pagination details, request bodies, or document metadata dictionaries.

Output shapes

Each extraction method returns a small summary plus normalized document outputs. outputs is the part you pass to chunk() or inspect before running the full pipeline.

{
  "documents": 1,
  "failed": 0,
  "outputs": [
    {
      "id": "doc_123",
      "display_id": "refund_policy",
      "source": "files/refund-policy.pdf",
      "title": "refund-policy",
      "text": "# Refund policy\n\nRefunds are available within 30 days.",
      "metadata": {
        "source_kind": "docs",
        "file_type": "pdf",
        "description": ""
      }
    }
  ],
  "errors": []
}

{
  "pages": 1,
  "failed": 0,
  "outputs": [
    {
      "id": "url_123",
      "source": "https://example.com/docs/refunds",
      "title": "Refund policy",
      "text": "# Refund policy\n\nRefunds are available within 30 days.",
      "metadata": {
        "source_kind": "url",
        "mode": "each",
        "root_url": "https://example.com/docs/refunds"
      }
    }
  ],
  "errors": [],
  "dlq": null
}

{
  "documents": 1,
  "failed": 0,
  "outputs": [
    {
      "id": "api_123",
      "display_id": "001_api_example_com_articles",
      "source": "https://api.example.com/articles",
      "title": "Articles — page 1",
      "text": "# Articles — page 1\n\n{...}",
      "metadata": {
        "source_kind": "api",
        "file_type": "api",
        "method": "GET",
        "page": 1,
        "item_count": 25,
        "max_pages": 5
      }
    }
  ],
  "errors": []
}

{
  "sources": 3,
  "chunks": 12,
  "embedded": 12,
  "stored": 12,
  "source_results": {
    "docs": {"documents": 1, "failed": 0},
    "urls": {"pages": 1, "failed": 0},
    "api": {"documents": 1, "failed": 0}
  },
  "failed": 0,
  "errors": []
}

Key	Meaning
`documents`	Number of document outputs from `parse()` or `fetch()`.
`pages`	Number of page outputs from `scrape()`.
`outputs`	Normalized document dictionaries containing `text`, `source`, `title`, and `metadata`.
`failed`	Number of sources or stage items that failed.
`errors`	Structured validation, fetch, parse, crawl, or provider errors.
`dlq`	Retryable scrape failures when SDK dead-letter queue capture is enabled.
`sources`	Number of extracted documents sent into the high-level ingest pipeline.
`chunks`, `embedded`, `stored`	Downstream counts from chunking, embedding, and storage inside `ingest()`.
`source_results`	Per-source extraction summaries from `docs`, `urls`, `api`, and `markdown`.

Possible errors

Most stage-level failures are returned in the result errors list instead of raising immediately. REST validation and setup failures may return an exception envelope instead. See Errors for the shared shapes.

{
  "documents": 0,
  "failed": 1,
  "outputs": [],
  "errors": [
    {
      "source": "files/missing.pdf",
      "source_kind": "path",
      "stage": "validate",
      "error": "document path not found: files/missing.pdf",
      "isRetryable": false,
      "attempts": 1
    }
  ]
}

{
  "pages": 0,
  "failed": 1,
  "outputs": [],
  "errors": [
    {
      "source": "https://example.com/docs",
      "source_kind": "url",
      "stage": "crawl",
      "error": "Timeout while loading page",
      "isRetryable": true,
      "mode": "full",
      "root_url": "https://example.com/docs",
      "attempts": 1,
      "retry_input": {"url": "https://example.com/docs", "mode": "full"}
    }
  ]
}

{
  "documents": 0,
  "failed": 1,
  "outputs": [],
  "errors": [
    {
      "source": "https://api.example.com/products",
      "source_kind": "api",
      "stage": "request",
      "error": "HTTP 500 from upstream API",
      "isRetryable": true,
      "attempts": 1,
      "retry_input": {"url": "https://api.example.com/products", "method": "GET"}
    }
  ]
}

{
  "error": {
    "type": "ValueError",
    "message": "Provide either 'files' or 'folder'"
  }
}

Next steps

Use Chunking to split extracted documents into searchable passages.
Use Ingest when you want extraction, chunking, embedding, and storage in one call.
Use Resilient Ingestion when large crawls or paginated APIs need retry strategy.

Get Started

Features

Capabilities

Use Cases

Walkthroughs

Source forms

Common shape

Scrape websites

Stream website extraction

Parse documents

Fetch REST APIs

Direct Markdown

Ingest pipeline

Output shapes

Possible errors

Next steps

​Source forms

​Common shape

​Scrape websites

​Stream website extraction

​Parse documents

​Fetch REST APIs

​Direct Markdown

​Ingest pipeline

​Output shapes

​Possible errors

​Next steps

Source forms

Common shape

Scrape websites

Stream website extraction

Parse documents

Fetch REST APIs

Direct Markdown

Ingest pipeline

Output shapes

Possible errors

Next steps