Codakey
Play
← Back to openobserve

How to Deploy & Use openobserve

OpenObserve Deployment and Usage Guide

1. Prerequisites

Before deploying OpenObserve, ensure you have:

  • Runtime: Node.js 18+ (for web interface development)
  • Package Manager: npm or yarn
  • Storage: S3-compatible object storage (AWS S3, MinIO, etc.) for data persistence
  • Browser: Modern web browser for accessing the UI
  • Optional: Docker and Docker Compose for containerized deployment

Note: OpenObserve is primarily built in Rust and distributed as a single binary, but the web interface uses TypeScript/Node.js for development.

2. Installation

Quick Start (Single Binary - Recommended)

# Download the latest binary
curl -L https://github.com/openobserve/openobserve/releases/latest/download/openobserve-amd64 -o openobserve
chmod +x openobserve

# Run with minimal configuration
./openobserve --config ./config.yaml

From Source (Development)

# Clone the repository
git clone https://github.com/openobserve/openobserve.git
cd openobserve

# Install web dependencies
cd web
npm install

Docker

docker run -d \
  -v $(pwd)/data:/data \
  -p 5080:5080 \
  --name openobserve \
  public.ecr.aws/zinclabs/openobserve:latest

3. Configuration

Create a config.yaml file:

# Basic configuration
node_role: "all"
cluster_name: "openobserve"
web_port: 5080
http_port: 5080

# Storage configuration (S3-compatible)
storage:
  type: "s3"
  bucket_name: "your-bucket"
  region: "us-east-1"
  endpoint_url: "https://s3.amazonaws.com"
  access_key: "${AWS_ACCESS_KEY_ID}"
  secret_key: "${AWS_SECRET_ACCESS_KEY}"

# Local disk storage (alternative)
# storage:
#   type: "local"
#   data_dir: "./data"

# Authentication
auth:
  secret_key: "${SECRET_KEY}"  # Generate with: openssl rand -hex 32
  oidc_enabled: false

# Caching
cache:
  enabled: true
  size: 1024  # MB

# Log retention
retention:
  logs: "30d"
  metrics: "90d"
  traces: "7d"

Required Environment Variables:

# S3 Storage (if using S3)
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_REGION="us-east-1"

# Authentication
export SECRET_KEY=$(openssl rand -hex 32)

# Optional: External database for metadata
export DATABASE_URL="postgresql://user:password@localhost:5432/openobserve"

4. Build & Run

Development Mode (Web Interface)

cd web
npm run dev  # Starts development server on port 3000

Production Build (Web Interface)

cd web
npm run build  # Creates optimized build in dist/ directory

Run OpenObserve Server

# Development mode with hot reload
cargo run -- --config config.yaml

# Production binary
./openobserve --config config.yaml

# With custom ports
./openobserve --web-port 5080 --http-port 5080 --config config.yaml

Build from Source (Rust)

# Install Rust toolchain if needed
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Build release binary
cargo build --release

# Binary will be at: target/release/openobserve

5. Deployment

Single Node Deployment (Production)

# 1. Create systemd service
sudo nano /etc/systemd/system/openobserve.service

# 2. Add service configuration
[Unit]
Description=OpenObserve Observability Platform
After=network.target

[Service]
Type=simple
User=openobserve
WorkingDirectory=/opt/openobserve
ExecStart=/opt/openobserve/openobserve --config /etc/openobserve/config.yaml
Restart=always
Environment="AWS_ACCESS_KEY_ID=your-key"
Environment="AWS_SECRET_ACCESS_KEY=your-secret"

[Install]
WantedBy=multi-user.target

# 3. Enable and start
sudo systemctl enable openobserve
sudo systemctl start openobserve

High Availability (HA) Deployment

For mission-critical deployments, use the HA configuration:

# config.yaml for HA mode
node_role: "all"
cluster_mode: "ha"
cluster_name: "openobserve-ha"
cluster_nodes:
  - "node1:5080"
  - "node2:5080"
  - "node3:5080"

# Shared storage (required for HA)
storage:
  type: "s3"
  bucket_name: "openobserve-ha-data"

Kubernetes Deployment

# openobserve-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: openobserve
spec:
  replicas: 3
  selector:
    matchLabels:
      app: openobserve
  template:
    metadata:
      labels:
        app: openobserve
    spec:
      containers:
      - name: openobserve
        image: public.ecr.aws/zinclabs/openobserve:latest
        ports:
        - containerPort: 5080
        env:
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: openobserve-secrets
              key: aws-access-key
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: openobserve-secrets
              key: aws-secret-key
        volumeMounts:
        - name: config
          mountPath: /etc/openobserve
      volumes:
      - name: config
        configMap:
          name: openobserve-config

Supported Platforms

  • AWS: EC2, ECS, EKS with S3 storage
  • GCP: Compute Engine, GKE with Cloud Storage
  • Azure: VMs, AKS with Blob Storage
  • On-premises: Bare metal or VMs with MinIO/S3-compatible storage
  • Docker: Any Docker-supported platform

6. Troubleshooting

Common Issues and Solutions

Issue: Server fails to start with S3 connection errors

# Check S3 credentials and permissions
export AWS_ACCESS_KEY_ID="your-key"
export AWS_SECRET_ACCESS_KEY="your-secret"
export AWS_REGION="us-east-1"

# Test S3 connectivity
aws s3 ls s3://your-bucket --endpoint-url=https://s3.amazonaws.com

Issue: High memory usage

# Reduce cache size in config.yaml
cache:
  enabled: true
  size: 512  # Reduce from 1024 MB

Issue: Web interface not loading

# Check if web assets are built
cd web
npm run build

# Ensure web port is accessible
curl http://localhost:5080/api/health

Issue: Data ingestion failures

# Check OpenTelemetry collector configuration
# Verify ports are open: 5080 (HTTP), 5081 (gRPC)

# Test ingestion
curl -X POST http://localhost:5080/api/org/logs/ingest \
  -H "Content-Type: application/json" \
  -d '{"logs": [{"timestamp": "2024-01-01T00:00:00Z", "message": "test log"}]}'

Issue: Query performance issues

# Enable query caching and adjust settings
cache:
  enabled: true
  query_cache_size: 1000  # Number of queries to cache
  query_cache_ttl: "300s" # Time to live for cached queries

Issue: Authentication problems

# Generate new secret key
openssl rand -hex 32

# Update config.yaml and restart
auth:
  secret_key: "new-generated-key-here"

Logs and Monitoring

  • Check server logs: journalctl -u openobserve -f
  • Access metrics: http://localhost:5080/metrics
  • Health endpoint: http://localhost:5080/api/health
  • Prometheus metrics available at /metrics endpoint

Performance Tuning

  1. For high ingestion rates: Increase max_concurrent_uploads in config
  2. For query-heavy workloads: Increase cache size and enable compression
  3. For memory constraints: Reduce cache.size and max_file_size
  4. For storage optimization: Adjust Parquet compression settings in config

Getting Help

← Project overviewView alternatives to openobserve →