> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hystersis.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Loaders

> Load documents from PDFs, audio, DOCX, and more

# Data Loaders

Hystersis supports loading data from various document formats for ingestion into memory.

## Production Ingestion API

Use the Sources API for production ingestion. It creates a durable source record, stores uploaded files in the configured blob backend, chunks extracted text, and writes searchable `source_chunk` memories with source attribution metadata.

```bash theme={null}
curl -X POST https://api.hystersis.com/sources/upload \
  -H "X-API-Key: your-api-key" \
  -F "file=@./runbook.txt;type=text/plain" \
  -F "org_id=org-123"
```

```bash theme={null}
curl -X POST https://api.hystersis.com/sources/ingest \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "url",
    "url": "https://example.com/docs",
    "org_id": "org-123"
  }'
```

Production object storage defaults to Cloudflare R2 when `STORAGE_PROVIDER=r2` and `R2_*` credentials are configured. Local filesystem storage is retained for development and self-hosted minimal installs.

## Supported Formats

| Format | Loader        | Description                          |
| ------ | ------------- | ------------------------------------ |
| PDF    | `PDFLoader`   | Extract text from PDF documents      |
| Audio  | `AudioLoader` | Transcribe audio with Whisper API    |
| DOCX   | `DocxLoader`  | Extract text from Word documents     |
| XLSX   | `XLSXLoader`  | Extract data from Excel spreadsheets |

## Quick Start

```go theme={null}
// Create multi-loader for multiple formats
loader := loaders.NewMultiLoader()
loader.Register(loaders.NewPDFLoader())
loader.Register(loaders.NewAudioLoader("https://api.openai.com/v1/audio/transcriptions"))
loader.Register(loaders.NewDocxLoader())
loader.Register(loaders.NewXLSXLoader())

// Load any supported format
doc, err := loader.Load(ctx, "/path/to/document.pdf")
```

## PDF Loader

```go theme={null}
loader := loaders.NewPDFLoader()
doc, err := loader.Load(ctx, "https://example.com/paper.pdf")

// Access extracted content
fmt.Println(doc.Content)

// Access chunks for processing
for _, chunk := range doc.Chunks {
    fmt.Printf("Chunk %d: %s\n", chunk.Index, chunk.Content)
}
```

## Audio Loader

```go theme={null}
// Configure with Whisper endpoint for transcription
loader := loaders.NewAudioLoader("https://api.openai.com/v1/audio/transcriptions")

doc, err := loader.Load(ctx, "/path/to/audio.mp3")
// Returns transcribed text in doc.Content
```

## Document Structure

```go theme={null}
type Document struct {
    Content  string                 // Full extracted text
    Source   string                 // Original source path/URL
    Title    string                // Extracted title
    Format   string                 // Document format (pdf, audio, etc.)
    Metadata map[string]interface{} // Additional metadata
    Chunks   []DocumentChunk       // Text chunks for processing
}

type DocumentChunk struct {
    Content  string                 // Chunk text
    Index    int                    // Chunk index
    Metadata map[string]interface{} // Chunk metadata
}
```

## Loading from URLs

All loaders support loading from HTTP/HTTPS URLs:

```go theme={null}
doc, err := loader.Load(ctx, "https://example.com/document.pdf")
```

## Custom Loaders

Implement the `Loader` interface for custom formats:

```go theme={null}
type Loader interface {
    Load(ctx context.Context, source string) (*Document, error)
    SupportedTypes() []string
}
```