Codakey
Play
← Back to quickwit

How to Deploy & Use quickwit

Quickwit Deployment and Usage Guide

1. Prerequisites

System Requirements:

  • Rust toolchain (for building from source): Rust 1.70 or later
  • Operating System: Linux, macOS, or Windows (Linux recommended for production)
  • Memory: Minimum 2GB RAM, 4GB+ recommended for production
  • Storage: Local disk for temporary files, plus cloud storage for persistent data

Cloud Storage Accounts (choose one):

  • Amazon S3 (or S3-compatible storage like MinIO)
  • Azure Blob Storage
  • Google Cloud Storage

Optional Dependencies:

  • Kubernetes (for production deployment via Helm)
  • Docker (for containerized deployment)
  • Vector/Fluent Bit (for log ingestion)
  • Jaeger (for distributed tracing)
  • Grafana (for visualization with Quickwit data source plugin)

2. Installation

Quick Install (Binary)

# Linux/macOS
curl -L https://install.quickwit.io | sh

# Or download directly from GitHub releases
curl -LO https://github.com/quickwit-oss/quickwit/releases/latest/download/quickwit-$(uname -m)-unknown-linux-gnu.tar.gz
tar xzf quickwit-*.tar.gz
sudo mv quickwit /usr/local/bin/

Build from Source

# Clone the repository
git clone https://github.com/quickwit-oss/quickwit.git
cd quickwit

# Build with cargo (release build recommended)
cargo build --release --bin quickwit

# The binary will be at: ./target/release/quickwit

Docker

docker pull quickwit/quickwit:latest
docker run -p 7280:7280 quickwit/quickwit:latest

Package Managers

  • Homebrew (macOS): brew install quickwit
  • APT (Debian/Ubuntu): Repository coming soon
  • RPM (RHEL/CentOS): Repository coming soon

3. Configuration

Environment Variables

# Required for cloud storage
export AWS_ACCESS_KEY_ID="your-access-key"  # For S3
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_REGION="us-east-1"

# Or for Azure
export AZURE_STORAGE_ACCOUNT="your-account"
export AZURE_STORAGE_ACCESS_KEY="your-key"

# Or for GCP (service account JSON)
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"

# Quickwit-specific
export QW_CONFIG="./config/quickwit.yaml"  # Config file path
export QW_DATA_DIR="./data"  # Local data directory
export QW_LISTEN_ADDRESS="0.0.0.0:7280"  # REST API address

Configuration File (quickwit.yaml)

# Default configuration file
version: 0.8

node_id: "node-1"

# REST API server configuration
rest_listen_port: 7280
grpc_listen_port: 7281

# Metastore configuration (supports PostgreSQL, file-backed, or etcd)
metastore_uri: "postgresql://localhost:5432/quickwit"
# Alternative: file-backed (for single node)
# metastore_uri: "file:///path/to/metastore"

# Storage configuration
default_index_root_uri: "s3://my-bucket/quickwit-indexes"
# Or: azure://my-container/quickwit-indexes
# Or: gcs://my-bucket/quickwit-indexes

# Local cache directory
data_dir: "/var/lib/quickwit"

# Indexer configuration
indexer:
  max_concurrent_split_uploads: 4
  max_num_splits_per_index: 1000

# Searcher configuration
searcher:
  max_num_concurrent_searches: 10
  fast_field_cache_capacity: 1GB

Index Configuration

Create an index configuration file (e.g., hdfs-logs.yaml):

version: 0.8

index_id: "hdfs-logs"

doc_mapping:
  field_mappings:
    - name: timestamp
      type: datetime
      input_formats: ["unix_timestamp"]
      output_format: "unix_timestamp_secs"
      fast: true
    - name: severity
      type: text
      tokenizer: raw
      fast: true
    - name: message
      type: text
      tokenizer: default
      record: position
    - name: host
      type: text
      tokenizer: raw
      fast: true

  timestamp_field: timestamp

indexing_settings:
  commit_timeout_secs: 60

search_settings:
  default_search_fields: [message]

4. Build & Run

Development Build and Run

# Clone and build
git clone https://github.com/quickwit-oss/quickwit.git
cd quickwit

# Build in debug mode (faster compilation)
cargo build

# Run a single node with default config
./target/debug/quickwit run

# Or run specific services
./target/debug/quickwit run --service searcher
./target/debug/quickwit run --service indexer
./target/debug/quickwit run --service metastore
./target/debug/quickwit run --service control-plane

Production Build

# Optimized release build
cargo build --release --features "aws,gcp,azure,postgresql"

# Build with specific features
cargo build --release --no-default-features --features "aws,kubernetes"

Running Quickwit Services

Single Node Mode:

# Start all services in one process
quickwit run

# Or using Docker
docker run -p 7280:7280 \
  -v ./config:/etc/quickwit \
  -v ./data:/var/lib/quickwit \
  -e AWS_ACCESS_KEY_ID="your-key" \
  -e AWS_SECRET_ACCESS_KEY="your-secret" \
  quickwit/quickwit:latest

Distributed Mode:

# Start control plane
quickwit run --service control-plane --cluster listen-address=0.0.0.0:7282

# Start indexers
quickwit run --service indexer --cluster peer=node1:7282

# Start searchers
quickwit run --service searcher --cluster peer=node1:7282

CLI Operations

# Create an index
quickwit index create --index-config ./hdfs-logs.yaml

# List indexes
quickwit index list

# Ingest data
quickwit index ingest --index hdfs-logs --input-path ./logs.json

# Search
quickwit index search --index hdfs-logs --query "error"
quickwit index search --index hdfs-logs --query "severity:ERROR AND host:web-*"

# Delete index
quickwit index delete --index hdfs-logs --yes

5. Deployment

Kubernetes (Production Recommended)

# Add Quickwit Helm repository
helm repo add quickwit https://helm.quickwit.io
helm repo update

# Install Quickwit cluster
helm install quickwit quickwit/quickwit \
  --namespace quickwit \
  --create-namespace \
  --set storage.s3.bucket="my-quickwit-bucket" \
  --set storage.s3.region="us-east-1" \
  --set secrets.awsAccessKeyId="your-key" \
  --set secrets.awsSecretAccessKey="your-secret"

# With custom values file
helm install quickwit quickwit/quickwit -f values.yaml

Example values.yaml:

# values.yaml for production
cluster:
  nodeCount: 3
  replicaCount:
    indexer: 3
    searcher: 3
    controlPlane: 2

storage:
  s3:
    bucket: "production-quickwit-indexes"
    region: "eu-west-1"

resources:
  indexer:
    requests:
      memory: "4Gi"
      cpu: "2"
    limits:
      memory: "8Gi"
      cpu: "4"
  searcher:
    requests:
      memory: "2Gi"
      cpu: "1"
    limits:
      memory: "4Gi"
      cpu: "2"

ingress:
  enabled: true
  className: "nginx"
  hosts:
    - host: quickwit.example.com
      paths:
        - path: /
          pathType: Prefix

Docker Compose (Development/Testing)

# docker-compose.yaml
version: '3.8'

services:
  quickwit:
    image: quickwit/quickwit:latest
    ports:
      - "7280:7280"
    environment:
      - QW_CONFIG=/etc/quickwit/quickwit.yaml
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
      - AWS_REGION=${AWS_REGION}
    volumes:
      - ./config:/etc/quickwit
      - ./data:/var/lib/quickwit
    command: run

  # Optional: Add PostgreSQL for metastore
  postgres:
    image: postgres:15
    environment:
      POSTGRES_DB: quickwit
      POSTGRES_USER: quickwit
      POSTGRES_PASSWORD: quickwit
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

Cloud Platforms

AWS ECS/EKS:

  • Use EKS with Quickwit Helm chart
  • Configure IAM roles for S3 access
  • Use ALB/NLB for load balancing

Google Cloud GKE:

  • Deploy Helm chart on GKE
  • Use Workload Identity for GCS access
  • Configure Cloud Load Balancing

Azure AKS:

  • Deploy on AKS with Azure Blob Storage
  • Use Managed Identities for authentication
  • Configure Azure Application Gateway

Bare Metal/VMs

# Systemd service file: /etc/systemd/system/quickwit.service
[Unit]
Description=Quickwit Search Engine
After=network.target

[Service]
Type=simple
User=quickwit
Group=quickwit
Environment=AWS_ACCESS_KEY_ID=your-key
Environment=AWS_SECRET_ACCESS_KEY=your-secret
ExecStart=/usr/local/bin/quickwit run
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable quickwit
sudo systemctl start quickwit

6. Troubleshooting

Common Issues and Solutions

Issue: "Permission denied" on S3/GCS/Azure storage

# Solution: Check credentials and permissions
export AWS_ACCESS_KEY_ID="correct-key"
export AWS_SECRET_ACCESS_KEY="correct-secret"
export AWS_REGION="us-east-1"

# For S3, verify bucket policy allows:
# s3:ListBucket, s3:GetObject, s3:PutObject, s3:DeleteObject

Issue: "Metastore connection failed"

# Solution: Check metastore URI and connectivity
quickwit --config ./config.yaml metastore check

# For PostgreSQL, ensure:
# 1. Database exists
# 2. User has permissions
# 3. Network connectivity

Issue: High memory usage

# Solution: Adjust configuration
searcher:
  fast_field_cache_capacity: "512MB"  # Reduce cache size
  max_num_concurrent_searches: 5  # Reduce concurrency

indexer:
  max_concurrent_split_uploads: 2  # Reduce upload concurrency

Issue: Slow search performance

# Check index configuration
quickwit index describe --index my-index

# Ensure timestamp field is configured for time-based pruning
# Check if splits are too large (ideal: 1-10GB each)

Issue: Indexing failures

# Check logs
quickwit index ingest --index my-index --input-path data.json --verbose

# Common causes:
# 1. Schema mismatch in documents
# 2. Invalid timestamp formats
# 3. Document too large (>10MB)

# Enable debug logging
RUST_LOG=quickwit=debug quickwit run

Issue: Cluster nodes not joining

# Verify network connectivity
nc -zv node1 7282

# Check cluster configuration matches
# All nodes should use same cluster ID and peer addresses

# View cluster state
quickwit cluster members

Issue: "Shard scaling rate limited"

# This is expected behavior - Quickwit limits scaling operations
# to prevent rapid scaling up/down. Adjust rate limiter settings
# in source configuration if needed.

# Current limits (from source code):
# - Scale up: max 5 operations per minute
# - Scale down: max 1 operation per minute

Logging and Monitoring

Enable Debug Logging:

RUST_LOG=quickwit=debug,info quickwit run
RUST_LOG=quickwit=debug,actix=info quickwit run

Check Service Health:

# REST API health check
curl http://localhost:7280/api/v1/health

# Metrics endpoint (Prometheus format)
curl http://localhost:7280/metrics

# Cluster status
curl http://localhost:7280/api/v1/cluster/members

Common Metrics to Monitor:

  • quickwit_indexer_throughput_bytes: Indexing throughput
  • quickwit_searcher_query_latency: Search latency
  • quickwit_metastore_operation_duration: Metastore performance
  • quickwit_storage_operation_duration: Storage performance
  • System metrics: CPU, memory, disk I/O

Getting Help

← Project overviewView alternatives to quickwit →