> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hystersis.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring Setup

> Set up monitoring, alerting, and observability for Hystersis in production

# Monitoring Setup

Complete guide to setting up monitoring, alerting, and observability for Hystersis in production.

## Overview

Hystersis exposes Prometheus metrics at `/metrics` and health checks at `/health` and `/ready`. This guide covers setting up Prometheus, Grafana, and alerting.

## Quick Setup

```bash theme={null}
# Check health endpoints
curl http://localhost:8080/health     # Liveness
curl http://localhost:8080/ready      # Readiness (checks Neo4j, Qdrant, Redis)

# View metrics
curl http://localhost:8080/metrics
```

## Prometheus Setup

### Docker Compose

```yaml theme={null}
# monitoring/docker-compose.monitoring.yml
version: '3.8'

services:
  prometheus:
    image: prom/prometheus:v2.48.0
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus/alerts:/etc/prometheus/alerts
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=30d'
      - '--storage.tsdb.retention.size=10GB'
      - '--web.enable-lifecycle'
    restart: unless-stopped

  grafana:
    image: grafana/grafana:10.2.0
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    restart: unless-stopped

  alertmanager:
    image: prom/alertmanager:v0.26.0
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
    restart: unless-stopped

volumes:
  prometheus-data:
  grafana-data:
```

### Prometheus Configuration

```yaml theme={null}
# monitoring/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    monitor: 'hystersis'

rule_files:
  - /etc/prometheus/alerts/*.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

scrape_configs:
  - job_name: 'hystersis-api'
    metrics_path: '/metrics'
    scrape_interval: 10s
    static_configs:
      - targets: ['host.docker.internal:8080']
        labels:
          environment: 'production'

  - job_name: 'neo4j'
    static_configs:
      - targets: ['host.docker.internal:2000']

  - job_name: 'redis'
    static_configs:
      - targets: ['host.docker.internal:6379']
```

### Alert Rules

```yaml theme={null}
# monitoring/prometheus/alerts/hystersis.yml
groups:
  - name: hystersis
    rules:
      - alert: APIHighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "API error rate above 5%"
          description: "Error rate is {{ $value | humanizePercentage }}"

      - alert: APIHighLatencyP95
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "API P95 latency above 2s"

      - alert: CompressionAccuracyDrop
        expr: hystersis_compression_accuracy_retention < 0.95
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Compression accuracy below 95%"

      - alert: MemoryUsageHigh
        expr: container_memory_usage_bytes{container="api-server"} / container_spec_memory_limit_bytes{container="api-server"} > 0.85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Memory usage above 85%"

      - alert: Neo4jQuerySlow
        expr: histogram_quantile(0.95, rate(hystersis_graph_query_duration_seconds_bucket[5m])) > 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Neo4j P95 query latency above 1s"

      - alert: APIInstanceDown
        expr: up{job="hystersis-api"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "API instance is down"
```

### Alertmanager Configuration

```yaml theme={null}
# monitoring/alertmanager/alertmanager.yml
route:
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'slack'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'
    - match:
        severity: warning
      receiver: 'slack'

receivers:
  - name: 'slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
        channel: '#hystersis-alerts'
        send_resolved: true

  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: 'YOUR_PAGERDUTY_SERVICE_KEY'
        severity: '{{ .GroupLabels.severity }}'
```

## Grafana Dashboards

### Datasource Provisioning

```yaml theme={null}
# monitoring/grafana/provisioning/datasources/datasources.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: true
```

### Key Dashboard Panels

| Panel                | Metric                                                                         | Description                   |
| -------------------- | ------------------------------------------------------------------------------ | ----------------------------- |
| Request Rate         | `rate(http_requests_total[5m])`                                                | Requests per second           |
| Error Rate           | `rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])` | Error percentage              |
| P95 Latency          | `histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))`     | 95th percentile response time |
| Memory Created       | `rate(hystersis_memories_created_total[5m])`                                   | Memory creation rate          |
| Search Latency       | `histogram_quantile(0.95, rate(hystersis_search_duration_seconds_bucket[5m]))` | Search performance            |
| Compression Accuracy | `hystersis_compression_accuracy_retention`                                     | Compression quality gauge     |
| Tier Distribution    | `hystersis_tier_*_count`                                                       | Memory tier breakdown         |
| Active Sessions      | `hystersis_active_sessions`                                                    | Current sessions              |

## Log Aggregation

### Structured Logging

```bash theme={null}
# Set logging configuration
LOG_LEVEL=info
LOG_FORMAT=json
LOG_OUTPUT=stdout
```

### Fluentd Configuration

```yaml theme={null}
# monitoring/fluentd/fluent.conf
<source>
  @type tail
  path /var/log/hystersis/*.log
  pos_file /var/log/fluentd/hystersis.pos
  tag hystersis.*
  format json
</source>

<filter hystersis.**>
  @type parser
  key_name log
  <parse>
    @type json
  </parse>
</filter>

<match hystersis.**>
  @type elasticsearch
  host elasticsearch
  port 9200
  index_name hystersis-logs
  flush_interval 5s
</match>
```

## Health Check Monitoring

```bash theme={null}
#!/bin/bash
# monitoring/health-check.sh
API_URL="http://localhost:8080"
ALERT_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK"

# Liveness check
HEALTH=$(curl -s -o /dev/null -w "%{http_code}" "$API_URL/health")
if [ "$HEALTH" -ne 200 ]; then
    curl -X POST "$ALERT_WEBHOOK" \
        -H "Content-Type: application/json" \
        -d "{\"text\": \"⚠️ API liveness check failed: HTTP $HEALTH\"}"
fi

# Readiness check (includes dependencies)
READY=$(curl -s "$API_URL/ready" | jq -r '.status')
if [ "$READY" != "ready" ]; then
    curl -X POST "$ALERT_WEBHOOK" \
        -H "Content-Type: application/json" \
        -d "{\"text\": \"⚠️ API readiness check failed: $READY\"}"
fi
```

## Start Monitoring

```bash theme={null}
cd monitoring
docker-compose -f docker-compose.monitoring.yml up -d

# Access services
# Prometheus: http://localhost:9090
# Grafana: http://localhost:3000 (admin/admin)
# Alertmanager: http://localhost:9093
```

## See Also

* [Deployment Monitoring](/deployment/monitoring) for infrastructure setup
* [Observability Feature](/features/observability) for available metrics
* [Performance Tuning](/performance-tuning) for optimization