REST Ingest

POST /v1/pipelines/ingest runs extraction, chunking, embedding, and storage together with a JSON request body. Use it when your application wants one request to build or refresh a collection from server-accessible paths, document URLs, websites, APIs, or Markdown. Use POST /v1/pipelines/ingest/mixed when the same document pipeline includes browser uploads plus JSON-addressable sources such as file paths, folders, or remote document URLs.

Request

curl -X POST http://127.0.0.1:8000/v1/pipelines/ingest \
  -H "Content-Type: application/json" \
  -d '{
    "docs": {"folder": "files/docs/"},
    "urls": ["https://example.com/docs"],
    "markdown": [
      {"id": "refund-policy", "text": "Customers can request a refund within 30 days of purchase."}
    ],
    "chunking": {"chunk_size": 1200, "chunk_overlap": 120},
    "embedding": {"provider": "voyage", "model": "voyage-3"},
    "storage": {"vector_db": "qdrant", "collection": "support", "url": "http://localhost:6333"},
    "concurrency": "parallel"
  }'

{
  "sources": 8,
  "chunks": 64,
  "embedded": 64,
  "stored": 64,
  "source_results": {},
  "chunk_result": {},
  "embed_result": {},
  "store_result": {},
  "failed": 0,
  "errors": []
}

Mixed Document Uploads

POST /v1/pipelines/ingest/mixed uses multipart form data with a JSON manifest, uploaded files, and optional file_refs. The manifest keeps the pipeline shape identical to JSON ingest while allowing uploaded bytes to join path, URL, and folder documents in one request.

Request

curl -X POST http://127.0.0.1:8000/v1/pipelines/ingest/mixed \
  -F 'manifest={
    "docs": [
      {"kind": "upload", "file_ref": "refund_pdf", "title": "Refund policy"},
      {"kind": "path", "path": "files/handbook.pdf"},
      {"kind": "url", "url": "https://example.com/files/guide.pdf"},
      {"kind": "folder", "folder": "files/policies/"}
    ],
    "chunking": {"chunk_size": 1200, "chunk_overlap": 120},
    "embedding": {"provider": "voyage", "model": "voyage-3"},
    "storage": {"vector_db": "qdrant", "collection": "support", "url": "http://localhost:6333"}
  }' \
  -F 'file_refs=refund_pdf' \
  -F '[email protected]'

file_refs is optional when file_ref matches the uploaded filename. When you provide it, repeat file_refs once per uploaded file in the same order as files.

Inputs

Field	Description
`docs`	File path, folder, file URL, upload item, or document config passed to document parsing.
`urls`	URL, list of URLs, or URL configs passed to website scraping.
`api`	API endpoint config passed to API fetching.
`markdown`	Raw markdown string, list of strings, or list of document dicts.

Document items can use kind: "path", kind: "folder", kind: "url", or kind: "upload". JSON ingest cannot carry upload bytes; use /v1/pipelines/ingest/mixed for upload items. You can combine inputs in one request. Ragrails merges extracted documents before chunking.

Config Blocks

Block	Controls
`ingestion`	Extraction options shared across source types.
`chunking`	Chunk size, overlap, and minimum chunk length.
`embedding`	Embedding provider, model, provider options, batch size, and input type.
`storage`	Vector database, collection, URL, batch size, and provider options.
`concurrency`	`"serial"` or `"parallel"` source extraction before chunking.

When To Use Stage Endpoints

Use stage endpoints instead of /v1/pipelines/ingest when you need to inspect extracted documents, review chunks before embedding, store artifacts outside the API response, or retry a specific stage.

Get Started

Workflows

Pipeline

Maintenance

Request

Mixed Document Uploads

Inputs

Config Blocks

When To Use Stage Endpoints