Extraction

CLI extraction commands normalize source content into JSON files that later flow into chunk or into the one-command ingest workflow. Add --output-dir when you want files for the next stage. Without it, commands return summary counts and errors.

Scrape Websites

Website extraction needs the url extra and browser setup.

pip install "ragrails[url]"
ragrails setup-url --browser chromium

ragrails scrape https://example.com/docs https://example.com/blog \
  --output-dir files/output/web/

ragrails scrape https://example.com/docs \
  --mode full \
  --max-depth 2 \
  --max-pages 50 \
  --output-dir files/output/web/

ragrails scrape https://example.com/docs \
  --frontmatter \
  --output-dir files/output/web/

Always cap --max-pages on a full crawl.

Option	Default	Description
`URL`	required	One or more positional URLs.
`--mode`	`each`	`each` exact URLs or `full` crawl.
`--max-depth`	`3`	Crawl depth for `mode=full`.
`--max-pages`	`200`	Max pages per site.
`--frontmatter`	off	Add YAML frontmatter.
`--output-dir`	none	Save output JSON files.

Dead-letter queue retry is SDK-only today. Use SDK extraction for DLQ capture and retry.

Parse Documents

# Folder
ragrails parse --folder files/docs/ --output-dir files/output/docs/

# Specific files or file URLs
ragrails parse \
  --files files/guide.pdf \
  --files https://example.com/files/whitepaper.pdf \
  --output-dir files/output/docs/

Option	Description
`--folder`	Directory to parse. Mutually exclusive with `--files`.
`--files`	File path or file URL. Repeatable.
`--frontmatter`	Add YAML frontmatter.
`--output-dir`	Save output JSON files.

Raw bytes input is SDK-only. Use SDK extraction for upload/object-storage bytes.

Fetch REST APIs

ragrails fetch https://api.example.com/posts \
  --title "Blog posts" \
  --header "Authorization:Bearer TOKEN" \
  --param limit:100 \
  --max-pages 10 \
  --output-dir files/output/api/

Option	Default	Description
`URL`	required	API endpoint.
`--title`	`API Response`	Document title.
`--description`	`""`	Description metadata.
`--method`	`GET`	`GET`, `POST`, `PUT`, `PATCH`, or `DELETE`.
`--header`	none	`KEY:VALUE`; repeatable.
`--param`	none	`KEY:VALUE`; repeatable.
`--max-pages`	`100`	Max pages to fetch.
`--frontmatter`	off	Add YAML frontmatter.
`--output-dir`	none	Save output JSON files.

Cursor/offset pagination strategies and JSON request bodies are configured through SDK extraction or REST extraction. The CLI covers headers and query params.

Next Stage

ragrails chunk --input-dir files/output/docs/ --output-dir files/output/chunks/

Ingest

Skip files and run extraction through storage.

Chunking

Chunk extraction output.

Get Started

Workflows

Pipeline

Maintenance

Scrape Websites

Parse Documents

Fetch REST APIs

Next Stage

Ingest

Chunking

​Scrape Websites

​Parse Documents

​Fetch REST APIs

​Next Stage

Ingest

Chunking

Scrape Websites

Parse Documents

Fetch REST APIs

Next Stage