Codakey
Play
← Back to airbyte

How to Deploy & Use airbyte

Airbyte Deployment and Usage Guide

1. Prerequisites

System Requirements

  • Docker and Docker Compose (required for running Airbyte Open Source)
  • Python 3.7+ (for connector development and customizations)
  • Git (for cloning the repository)
  • Java 11+ (for building and running certain components)
  • Node.js 14+ (for UI and some development tools)
  • At least 4GB RAM (8GB+ recommended for production)

Accounts & Credentials

  • Source Systems: API keys, OAuth credentials, or database credentials for your data sources (e.g., Google Ads, Salesforce, Shopify, HubSpot)
  • Destination Systems: Credentials for your data warehouses/lakes (Snowflake, BigQuery, Redshift, Postgres, etc.)
  • Optional: Docker Hub account if building custom connector images

2. Installation

Clone the Repository

git clone https://github.com/airbytehq/airbyte.git
cd airbyte

Quick Start with Docker (Recommended)

# Start Airbyte with Docker Compose
docker-compose up

# Or in detached mode
docker-compose up -d

Once running, access the Airbyte UI at http://localhost:8000

Manual Installation (Development)

# Install Python dependencies for connector development
pip install -e ".[dev]"  # From the root directory

# Build the project (requires Java and Node.js)
./gradlew build

# Start the platform components separately
./gradlew :airbyte-webapp:dev  # UI
./gradlew :airbyte-server:run  # API server
./gradlew :airbyte-worker:run  # Worker processes

3. Configuration

Environment Variables

Create a .env file in the root directory or set environment variables:

# Basic configuration
AIRBYTE_VERSION=0.50.0
DATABASE_URL=postgresql://airbyte:password@localhost:5432/airbyte
WORKSPACE_ROOT=/tmp/airbyte/workspace
LOCAL_ROOT=/tmp/airbyte/local
CONFIG_ROOT=/tmp/airbyte/config

# Optional: External database for production
# AIRBYTE_DATABASE_HOST=your-postgres-host
# AIRBYTE_DATABASE_PORT=5432
# AIRBYTE_DATABASE_USER=airbyte
# AIRBYTE_DATABASE_PASSWORD=secure_password

# Optional: S3/MinIO for logs and state storage
# AIRBYTE_MINIO_ENDPOINT=minio:9000
# AIRBYTE_MINIO_ACCESS_KEY=airbyte
# AIRBYTE_MINIO_SECRET_KEY=password123

Source Configuration Examples

Google Ads Connector (source-google-ads):

config:
  developer_token: "your_developer_token"
  client_id: "your_client_id"
  client_secret: "your_client_secret"
  refresh_token: "your_refresh_token"
  customer_id: "1234567890"
  start_date: "2024-01-01"

Salesforce Connector (source-salesforce):

config:
  client_id: "your_client_id"
  client_secret: "your_client_secret"
  refresh_token: "your_refresh_token"
  start_date: "2024-01-01T00:00:00Z"
  is_sandbox: false

Shopify Connector (source-shopify):

config:
  shop: "your-store-name"
  credentials:
    auth_method: "api_password"
    api_password: "your_api_password"
  start_date: "2024-01-01"

Authentication

Most connectors support multiple authentication methods:

  • API keys (Shopify, HubSpot)
  • OAuth 2.0 (Google Ads, Salesforce)
  • Bearer tokens (HubSpot)
  • Database credentials (Postgres, MySQL)

4. Build & Run

Running with Docker Compose

# Start all services
docker-compose up

# Check service status
docker-compose ps

# View logs
docker-compose logs -f airbyte-server
docker-compose logs -f airbyte-worker

# Stop services
docker-compose down

# Stop and remove volumes (clears all data)
docker-compose down -v

Development Build

# Build all Docker images
./gradlew build

# Build specific connector
cd airbyte-integrations/connectors/source-google-ads
docker build . -t airbyte/source-google-ads:dev

# Run tests for a connector
./gradlew :airbyte-integrations:connectors:source-google-ads:test

Custom Connector Development

# Create a new Python source connector
cd airbyte-integrations/connectors
python -m venv .venv
source .venv/bin/activate
pip install airbyte-cdk

# Use the Connector Builder UI (available at http://localhost:8000/connector-builder)
# Or develop with the low-code CDK

5. Deployment

Production Deployment Options

Option 1: Kubernetes (Recommended for production)

# Example Kubernetes deployment using Helm
helm repo add airbyte https://airbytehq.github.io/helm-charts
helm install airbyte airbyte/airbyte \
  --namespace airbyte \
  --set database.secretName=airbyte-database-secret \
  --set minio.enabled=true

Option 2: Docker Swarm

# Deploy stack
docker stack deploy -c docker-compose.yml airbyte

Option 3: VM/ Bare Metal

# Install dependencies
apt-get update
apt-get install -y docker.io docker-compose

# Clone and deploy
git clone https://github.com/airbytehq/airbyte.git
cd airbyte
docker-compose -f docker-compose.yml up -d

Production Considerations

  1. Database: Use external PostgreSQL for persistence
  2. Storage: Configure S3/MinIO for logs and state storage
  3. Scaling: Run multiple workers for parallel syncs
  4. Monitoring: Set up Prometheus metrics and logging
  5. Backups: Regular backups of PostgreSQL database and S3 storage

Cloud Deployment

  • Airbyte Cloud: Managed service (recommended for teams without DevOps resources)
  • AWS: Deploy on ECS/EKS with RDS and S3
  • GCP: Deploy on GKE with Cloud SQL and Cloud Storage
  • Azure: Deploy on AKS with Azure Database and Blob Storage

6. Troubleshooting

Common Issues

Issue: Connector fails with authentication errors

# Solution: Verify credentials and authentication method
# Check connector logs in the Airbyte UI
# Ensure OAuth tokens are refreshed if using OAuth

Issue: Docker containers failing to start

# Check available resources
docker system df
docker logs <container_id>

# Common fixes:
# 1. Increase Docker memory allocation
# 2. Clear Docker cache: docker system prune -a
# 3. Check port conflicts: netstat -tulpn | grep :8000

Issue: Syncs stuck or slow

# Check worker logs
docker-compose logs -f airbyte-worker

# Increase resources:
# 1. Add more workers in docker-compose.yml
# 2. Increase JVM memory: JAVA_OPTS="-Xmx4g"
# 3. Check network connectivity to sources/destinations

Issue: "No space left on device" errors

# Clear temporary files
docker system prune
rm -rf /tmp/airbyte/*

# Or change workspace location in .env:
WORKSPACE_ROOT=/path/to/larger/disk/airbyte/workspace

Issue: Connector development errors

# For Python connector issues:
pip install --upgrade airbyte-cdk
pytest tests/ -v  # Run tests

# For build issues:
./gradlew clean build

Getting Help

  1. Check Logs: Access logs via UI or docker-compose logs
  2. Community Support: Join Airbyte Community Slack
  3. GitHub Issues: Search existing issues or create new ones
  4. Documentation: Refer to docs.airbyte.com
  5. Debug Mode: Enable debug logging in connector configuration

Maintenance

# Regular maintenance tasks
docker-compose pull  # Update to latest version
docker-compose down && docker-compose up -d  # Restart services
docker system prune  # Clean up unused Docker resources

# Backup database
docker exec airbyte-db pg_dump -U airbyte airbyte > backup.sql

# Monitor resource usage
docker stats
docker-compose logs --tail=100
← Project overviewView alternatives to airbyte →