Codakey
Play
← Back to xyne

How to Deploy & Use xyne

Xyne Deployment & Usage Guide

1. Prerequisites

Runtime & Tools

  • Node.js (v18 or higher)
  • Docker and Docker Compose (for containerized deployment)
  • PostgreSQL (v14 or higher)
  • Redis (v7 or higher)
  • Vespa Search Engine (for vector search and document storage)
  • Python 3.8+ (for OCR and layout parsing services)

External Accounts & API Keys

  • Google Workspace Service Account (for Google Workspace integration)
  • Slack App Credentials (for Slack integration - upcoming)
  • Atlassian API Tokens (for Jira/Confluence - upcoming)
  • LLM Provider API Key (OpenAI, Anthropic, AWS Bedrock, or local Ollama)
  • GitHub Personal Access Token (for repository indexing)

System Requirements

  • Minimum 8GB RAM (16GB recommended for production)
  • 4 CPU cores minimum
  • 50GB+ storage (scales with indexed data volume)

2. Installation

Clone the Repository

git clone https://github.com/xynehq/xyne.git
cd xyne

Install Dependencies

# Install server dependencies
cd server
npm install

# Install frontend dependencies
cd ../frontend
npm install

# Install shared dependencies
cd ../shared
npm install

Database Setup

# Using Docker for development
docker-compose up -d postgres redis

# Or install manually:
# PostgreSQL: https://www.postgresql.org/download/
# Redis: https://redis.io/download/

Vespa Search Engine Setup

# Using Docker (recommended for development)
docker run -d \
  --name vespa \
  -p 8080:8080 \
  -p 19071:19071 \
  vespaengine/vespa:latest

# Wait for Vespa to initialize (2-3 minutes)
curl http://localhost:19071/ApplicationStatus

3. Configuration

Environment Variables

Create a .env file in the server/ directory:

# Database
DATABASE_URL="postgresql://postgres:password@localhost:5432/xyne"
REDIS_URL="redis://localhost:6379"

# Vespa Search
VESPA_ENDPOINT="http://localhost:8080"
VESPA_DATA_ENDPOINT="http://localhost:8080"

# LLM Configuration (choose one provider)
OPENAI_API_KEY="sk-..."  # For OpenAI models
ANTHROPIC_API_KEY="sk-ant-..."  # For Claude models
AWS_ACCESS_KEY_ID="..."  # For AWS Bedrock
AWS_SECRET_ACCESS_KEY="..."
AWS_REGION="us-east-1"

# Model Selection
DEFAULT_BEST_MODEL="gpt-4-turbo-preview"  # or "claude-3-opus", "anthropic.claude-3"

# OCR & Layout Parsing (for PDF/image processing)
LAYOUT_PARSING_BASE_URL="http://localhost:8000"
LAYOUT_PARSING_TIMEOUT_MS="300000"

# Google Workspace Integration
GOOGLE_SERVICE_ACCOUNT_EMAIL="your-service-account@project.iam.gserviceaccount.com"
GOOGLE_SERVICE_ACCOUNT_KEY_PATH="./service-account-key.json"
GOOGLE_WORKSPACE_DOMAIN="yourcompany.com"

# Slack Integration (upcoming)
SLACK_CLIENT_ID="your-slack-client-id"
SLACK_CLIENT_SECRET="your-slack-client-secret"
SLACK_SIGNING_SECRET="your-slack-signing-secret"

# Server Configuration
NODE_ENV="development"
PORT="3000"
CORS_ORIGIN="http://localhost:5173"  # Frontend dev server

Google Workspace Service Account Setup

  1. Follow the Service Account integration guide
  2. Download the service account JSON key
  3. Place it in server/service-account-key.json
  4. Enable required Google APIs:
    • Google Drive API
    • Gmail API
    • Calendar API
    • People API

Vespa Schema Configuration

Deploy the Vespa application schema:

# From the project root
cd vespa-app
zip -r application.zip .
curl --header Content-Type:application/zip --data-binary @application.zip \
  http://localhost:19071/application/v2/tenant/default/prepareandactivate

4. Build & Run

Development Mode

# Start backend server (from server directory)
cd server
npm run dev

# Start frontend dev server (from frontend directory)
cd ../frontend
npm run dev

# Start OCR/layout parsing service (optional, for PDF/image processing)
docker run -d -p 8000:8000 xyne/layout-parsing:latest

Production Build

# Build all packages
npm run build  # From project root

# Or build individually:
cd server && npm run build
cd ../frontend && npm run build
cd ../shared && npm run build

Run with Docker Compose

# Using the provided docker-compose.yml
docker-compose up -d

# Or build and run:
docker-compose build
docker-compose up

Database Migrations

# Run database migrations
cd server
npm run db:migrate

# Generate new migration
npm run db:generate

5. Deployment

Cloud Deployment (AWS EC2)

# 1. Launch EC2 instance (t3.large or larger recommended)
# 2. Install Docker and Docker Compose
sudo yum update -y
sudo yum install docker -y
sudo service docker start
sudo usermod -a -G docker ec2-user
sudo curl -L "https://github.com/docker/compose/releases/download/v2.20.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

# 3. Clone and configure
git clone https://github.com/xynehq/xyne.git
cd xyne

# 4. Configure environment variables
cp .env.example .env
# Edit .env with production values

# 5. Deploy
docker-compose -f docker-compose.prod.yml up -d

Kubernetes Deployment

# Example deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: xyne
spec:
  replicas: 3
  selector:
    matchLabels:
      app: xyne
  template:
    metadata:
      labels:
        app: xyne
    spec:
      containers:
      - name: server
        image: xynehq/xyne:latest
        ports:
        - containerPort: 3000
        envFrom:
        - configMapRef:
            name: xyne-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: xyne-config
data:
  DATABASE_URL: "postgresql://postgres:password@postgres:5432/xyne"
  VESPA_ENDPOINT: "http://vespa:8080"
  # ... other environment variables

Docker Deployment

# Build Docker image
docker build -t xyne .

# Run with all dependencies
docker run -d \
  --name xyne \
  -p 3000:3000 \
  -v ./service-account-key.json:/app/service-account-key.json \
  --env-file .env \
  xyne:latest

6. Troubleshooting

Common Issues & Solutions

1. Database Connection Errors

# Check PostgreSQL is running
docker ps | grep postgres

# Test connection
psql "$DATABASE_URL" -c "SELECT 1"

# Reset database (development only)
npm run db:reset

2. Vespa Search Issues

# Check Vespa status
curl http://localhost:19071/ApplicationStatus

# View Vespa logs
docker logs vespa

# Re-deploy Vespa schema
cd vespa-app && ./redeploy.sh

3. OCR/Layout Parsing Service Failures

# Check service health
curl http://localhost:8000/health

# Increase timeout if processing large PDFs
LAYOUT_PARSING_TIMEOUT_MS=600000

# Check memory limits for image processing
docker run -d -p 8000:8000 --memory="4g" xyne/layout-parsing:latest

4. Google Workspace Authentication Errors

# Verify service account permissions
# Ensure domain-wide delegation is enabled
# Check Google APIs are enabled in Cloud Console

# Test service account
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://www.googleapis.com/drive/v3/about?fields=user"

5. LLM Provider Connection Issues

# Test API key validity
curl https://api.openai.com/v1/models \
  -H "Authorization: Bearer $OPENAI_API_KEY"

# Switch to local model via Ollama
DEFAULT_BEST_MODEL="deepseek-coder"
OLLAMA_BASE_URL="http://localhost:11434"

6. Memory Issues with Large Indexing

# Increase Node.js memory limit
NODE_OPTIONS="--max-old-space-size=4096"

# Configure Vespa memory limits
# In vespa-app/services.xml:
# <nodes>
#   <node hostalias="node1" memory="8Gb"/>
# </nodes>

7. Slow Search Performance

# Optimize Vespa configuration
# Increase thread pool size
# Add more content nodes for scaling

# Check query performance
curl -X POST http://localhost:8080/search/ \
  -H "Content-Type: application/json" \
  -d '{"yql": "select * from sources * where userQuery()", "query": "test", "timeout": "5s"}'

8. Frontend Connection Issues

# Check CORS configuration
CORS_ORIGIN="https://your-domain.com"

# Verify WebSocket connections
# Check browser console for errors
# Ensure proper HTTPS configuration for production

Logging & Monitoring

# View application logs
docker-compose logs -f

# Check specific service logs
docker-compose logs server
docker-compose logs frontend

# Enable debug logging
LOG_LEVEL="debug"
NODE_ENV="development"

Performance Tuning

  1. Database Indexing: Ensure proper indexes on frequently queried columns
  2. Vespa Tuning: Adjust ranking profiles and query timeout settings
  3. Connection Pooling: Configure PostgreSQL connection pool size
  4. Cache Optimization: Increase Redis memory limits for frequently accessed data
  5. Batch Processing: Use appropriate batch sizes for data ingestion

For additional help, join the Xyne Slack community or check the official documentation.

← Project overviewView alternatives to xyne →