Celery Deployment and Usage Guide

1. Prerequisites

• Python: 3.8 or later (development branch may require newer features; check setup.py for exact version)
• pip: Python package installer
• Message Broker: One of the supported brokers:
  • RabbitMQ (recommended for production)
  • Redis
  • Amazon SQS
  • Apache Kafka (via transport)
  • Others listed in Celery documentation
• Result Backend (optional but recommended for task states/results):
  • Redis
  • RabbitMQ (RPC backend)
  • Database (SQLAlchemy, Django ORM)
  • Memcached
  • Elasticsearch
  • Others
• System Dependencies (for optional performance extensions):
  • librabbitmq (C library for RabbitMQ) - install via system package manager
  • libev or libuv (for event loop) - optional

2. Installation

From PyPI (stable release)

pip install celery

From Source (development branch)

# Clone the repository
git clone https://github.com/celery/celery.git
cd celery

# Install in editable mode with development dependencies
pip install -e .[redis,rabbitmq,sqlalchemy]

Verify Installation

python -c "import celery; print(celery.__version__)"

3. Configuration

Celery uses a configuration module or environment variables. Create a Python module (e.g., celery_app.py):

from celery import Celery

# Create Celery app
app = Celery('myapp')

# Configuration via dictionary
app.conf.update(
    broker_url='pyamqp://guest@localhost//',  # RabbitMQ
    result_backend='rpc://',                  # RPC result backend
    task_serializer='json',
    result_serializer='json',
    accept_content=['json'],
    timezone='UTC',
    enable_utc=True,
    task_track_started=True,
    worker_max_tasks_per_child=1000,          # Prevent memory leaks
    task_acks_late=True,                      # Tasks acknowledged after execution
    worker_prefetch_multiplier=1,             # One task per worker at a time
)

# Optional: Load from environment variables
import os
app.conf.broker_url = os.getenv('CELERY_BROKER_URL', app.conf.broker_url)
app.conf.result_backend = os.getenv('CELERY_RESULT_BACKEND', app.conf.result_backend)

# Define a sample task
@app.task(bind=True)
def add(self, x, y):
    return x + y

Key Configuration Options

• broker_url: Connection string to message broker (required)
• result_backend: Connection string to result backend (optional but recommended)
• task_serializer/result_serializer: json (default), pickle, yaml, etc.
• worker_concurrency: Number of worker processes/threads (default: number of CPU cores)
• task_acks_late: Acknowledge tasks after execution (ensures tasks are re-queued if worker crashes)
• task_reject_on_worker_lost: Reject tasks if worker terminates unexpectedly

4. Build & Run

Local Development

1. Start the broker (example with RabbitMQ):

   # Using Docker
   docker run -d --hostname rabbit --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3-management
   
   # Or install and start system service
   sudo systemctl start rabbitmq-server
   

2. Start a Celery worker:

   # From the directory containing celery_app.py
   celery -A celery_app worker --loglevel=info
   
   # With concurrency control
   celery -A celery_app worker --loglevel=info --concurrency=4
   
   # With eventlet/gevent (for I/O bound tasks)
   celery -A celery_app worker --loglevel=info --pool=eventlet --concurrency=1000
   

3. Send a task (in Python shell or separate script):

   from celery_app import add
   result = add.delay(4, 4)
   print(result.get(timeout=10))  # Blocks until result is ready
   

4. Monitor tasks (optional):

   # Flower (web-based monitoring)
   pip install flower
   celery -A celery_app flower --port=5555
   

Production Considerations

• Use a process manager (systemd, supervisord) to manage worker processes
• Set appropriate --concurrency based on task type (CPU vs I/O bound)
• Use --pool=prefork (default) for CPU-bound tasks, --pool=eventlet/gevent for I/O-bound
• Configure logging to file and rotate logs
• Set worker_max_tasks_per_child to prevent memory leaks
• Use --hostname to identify workers: celery -A celery_app worker --hostname=worker1@%h

5. Deployment

Option 1: Traditional Server (systemd)

1. Create a systemd service file /etc/systemd/system/celery-worker.service:

   [Unit]
   Description=Celery Worker Service
   After=network.target rabbitmq-server.service
   
   [Service]
   Type=forking
   User=celery
   Group=celery
   WorkingDirectory=/opt/myapp
   EnvironmentFile=/opt/myapp/.env
   ExecStart=/opt/myapp/venv/bin/celery multi start worker1 -A celery_app --loglevel=info --pidfile=/var/run/celery/%n.pid --logfile=/var/log/celery/%n.log
   ExecStop=/opt/myapp/venv/bin/celery multi stopwait worker1 --pidfile=/var/run/celery/%n.pid
   Restart=always
   
   [Install]
   WantedBy=multi-user.target
   

2. Enable and start:

   sudo systemctl daemon-reload
   sudo systemctl enable celery-worker
   sudo systemctl start celery-worker
   

Option 2: Docker

Dockerfile:

FROM python:3.10-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .

CMD ["celery", "-A", "celery_app", "worker", "--loglevel=info"]

docker-compose.yml:

version: '3.8'
services:
  rabbitmq:
    image: rabbitmq:3-management
    ports:
      - "5672:5672"
      - "15672:15672"
    environment:
      RABBITMQ_DEFAULT_USER: guest
      RABBITMQ_DEFAULT_PASS: guest
    volumes:
      - rabbitmq_data:/var/lib/rabbitmq

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data

  celery-worker:
    build: .
    command: celery -A celery_app worker --loglevel=info --concurrency=4
    environment:
      CELERY_BROKER_URL: amqp://guest:guest@rabbitmq:5672//
      CELERY_RESULT_BACKEND: redis://redis:6379/0
    depends_on:
      - rabbitmq
      - redis
    volumes:
      - .:/app
    deploy:
      replicas: 2

volumes:
  rabbitmq_data:
  redis_data:

Option 3: Kubernetes

deployment-worker.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: celery-worker
spec:
  replicas: 3
  selector:
    matchLabels:
      app: celery-worker
  template:
    metadata:
      labels:
        app: celery-worker
    spec:
      containers:
      - name: worker
        image: myapp/celery-worker:latest
        command: ["celery", "-A", "celery_app", "worker", "--loglevel=info"]
        env:
        - name: CELERY_BROKER_URL
          value: "amqp://guest:guest@rabbitmq:5672//"
        - name: CELERY_RESULT_BACKEND
          value: "redis://redis:6379/0"
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          exec:
            command:
            - celery
            - inspect
            - ping
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          exec:
            command:
            - celery
            - inspect
            - ping
          initialDelaySeconds: 5
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: celery-worker
spec:
  selector:
    app: celery-worker
  ports:
  - port: 5555
    name: flower

deployment-beat.yaml (for periodic tasks):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: celery-beat
spec:
  replicas: 1
  selector:
    matchLabels:
      app: celery-beat
  template:
    metadata:
      labels:
        app: celery-beat
    spec:
      containers:
      - name: beat
        image: myapp/celery-worker:latest
        command: ["celery", "-A", "celery_app", "beat", "--loglevel=info"]
        env:
        - name: CELERY_BROKER_URL
          value: "amqp://guest:guest@rabbitmq:5672//"
        - name: CELERY_RESULT_BACKEND
          value: "redis://redis:6379/0"

6. Troubleshooting

Common Issues

1. Worker cannot connect to broker

   • Check broker URL and credentials
   • Verify broker is running and accessible (use telnet <host> <port>)
   • Check firewall/security groups
   • Enable debug logging: celery -A celery_app worker --loglevel=debug

2. Tasks not being processed

   • Ensure worker is running: celery -A celery_app inspect ping
   • Check task is registered: celery -A celery_app inspect registered
   • Verify queue names match (default queue is celery)
   • Check for task rate limits or concurrency limits

3. Result backend errors

   • Ensure backend service is running
   • Check backend URL configuration
   • For RPC backend, ensure RabbitMQ is configured correctly (exchanges/queues)
   • For database backends, check connection and migrations

4. Worker crashes or memory leaks

   • Set worker_max_tasks_per_child to limit tasks