> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hystersis.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Compression Engine

> Hystersis compression engine achieving 80-85% token reduction with 97% accuracy using ProMem extraction

# Compression Engine

Hystersis's proprietary compression engine achieves 80-85% token reduction with 97% accuracy, significantly outperforming Mem0's baseline compression through ProMem-style fact extraction and multi-model verification.

## How It Works

The compression engine uses a multi-stage pipeline:

```
Input Memory → LLM Router → Fast/Verify Path → Extracted Facts → Verified Facts → Compressed Memory
```

### Stage 1: LLM Router

Routes memories based on complexity:

* **Simple memories** (complexity \< 0.6) → Fast path (GPT-4o-mini): direct extraction
* **Complex memories** (complexity ≥ 0.6) → Verify path: fast extraction + Claude verification

```python theme={null}
from hystersis import Hystersis

client = Hystersis(api_key="your-api-key")

# Set compression mode
client.set_compression_mode("extract")  # or "balanced", "aggressive"
```

### Stage 2: ProMem Extraction

Based on the ProMem paper (arXiv:2601.04463), extraction follows four phases:

1. **Self-Question Generation** — "What does this memory tell us?"
2. **Answer Verification** — Validate answers against original content
3. **Gap Detection** — Find missing critical information
4. **Active Extraction** — Extract key facts, not summaries

### Stage 3: Verification

Complex memories are verified using a higher-accuracy model:

```
Input: "John works at Acme Corp as a senior engineer and prefers remote work"

Fast extraction (GPT-4o-mini):
  - John works at Acme Corp
  - John is a senior engineer
  - John prefers remote work

Verification (Claude):
  ✓ All facts verified against original
  ✓ No gaps detected
  → Confidence: 0.97
```

### Stage 4: Compression

Extracted facts are compressed and stored:

```
Original: 24 tokens → Compressed: 5 tokens → 79% reduction
```

## Compression Modes

| Mode         | Description                             | Token Reduction | Accuracy |
| ------------ | --------------------------------------- | --------------- | -------- |
| `extract`    | ProMem fact extraction + verification   | 80-85%          | 97%+     |
| `balanced`   | Moderate compression with summarization | 75-80%          | 95%+     |
| `aggressive` | Maximum compression                     | 85-90%          | 90%+     |

## Configuration

```bash theme={null}
# Environment variables
COMPRESSION_ENABLED=true
COMPRESSION_MODE=extract
COMPRESSION_LLM_FAST_PROVIDER=openai
COMPRESSION_LLM_FAST_MODEL=gpt-4o-mini
COMPRESSION_LLM_VERIFY_PROVIDER=anthropic
COMPRESSION_LLM_VERIFY_MODEL=claude-3-5-sonnet
COMPRESSION_COMPLEXITY_THRESHOLD=0.6
```

## API Usage

```python theme={null}
# Create memory with compression
memory = client.create_memory(
    content="Alice is a software developer at TechCorp. She works from home three days a week and visits the office on Tuesdays and Thursdays. She prefers async communication over meetings.",
    compression_mode="extract"
)

# Check compression stats
stats = client.get_compression_stats()
print(f"Token reduction: {stats.token_reduction}%")
print(f"Accuracy: {stats.accuracy_retention}%")
print(f"Total tokens saved: {stats.total_tokens_saved}")

# Test compression in playground
result = client.playground_compress(
    content="Long text to compress...",
    mode="extract"
)
```

## Algorithm Benchmarks

Run measured compression benchmarks before publishing compression claims:

```bash theme={null}
curl -X POST "$API_URL/compression/benchmarks/run" \
  -H "X-API-Key: $ADMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "corpus": "agent_memory",
    "algorithms": ["radix", "smart_radix", "smart_hybrid", "real_best", "gzip"],
    "iterations": 3,
    "min_retention": 0.9
  }'
```

Responses include `avg_reduction`, `avg_retention`, `p95_latency_ms`, `throughput_per_second`, `expansion_count`, `error_count`, and `retention_below_target` for each algorithm.

## Targets

| Metric             | Hystersis          | Mem0    | Advantage           |
| ------------------ | ------------------ | ------- | ------------------- |
| Accuracy Retention | 97%+ target        | 91%     | +6% target          |
| Token Reduction    | 80-85% target      | 80%     | +5% target          |
| P95 Latency        | under 200ms target | \~400ms | 2x faster target    |
| Write Impact       | under 5ms target   | N/A     | Non-blocking target |

## See Also

* [Compression API Reference](/api-reference/compression) for API endpoints
* [Spreading Activation](/features/spreading-activation) for retrieval
* [Compression Concepts](/concepts/compression) for architecture details