Codakey
Play
← Back to maxun

How to Deploy & Use maxun

Maxun: Web Data Platform Deployment Guide

This guide provides comprehensive instructions for deploying and using Maxun, an open-source no-code platform for web scraping, crawling, search, and AI data extraction.

1. Prerequisites

Before you begin, ensure you have the following installed:

  • Node.js: Version 18 or higher.
  • npm or Yarn: For package management.
  • Docker and Docker Compose: Recommended for simplified setup and production deployments.
  • PostgreSQL: A running PostgreSQL instance is required for Maxun's database.
  • Git: For cloning the repository.

2. Installation

Maxun can be installed and run using Docker Compose (recommended) or directly on your local machine.

2.1. Clone the Repository

First, clone the Maxun repository:

git clone https://github.com/getmaxun/maxun.git
cd maxun

2.2. Setup with Docker Compose (Recommended)

This method simplifies the setup by orchestrating all necessary services (database, Maxun server).

  1. Create .env file: Copy the example environment file and configure it.

    cp .env.example .env

    Refer to the Configuration section for details on environment variables.

  2. Start Services:

    docker-compose up -d

    This command will build the Docker images, create the necessary containers, and start them in detached mode.

2.3. Setup without Docker (Local)

This method requires you to manage dependencies and services manually.

  1. Install Dependencies:

    npm install

  2. Database Setup: Ensure you have a PostgreSQL database running and accessible. Create a database for Maxun.

  3. Create .env file: Copy the example environment file and configure it.

    cp .env.example .env

    Refer to the Configuration section for details on environment variables, especially database connection strings.

  4. Run Database Migrations:

    npm run migrate

3. Configuration

Maxun relies on environment variables for configuration. A .env.example file is provided in the repository root. Copy this to .env and modify the values.

Essential Environment Variables:

  • Database Connection:

    • DB_USER: PostgreSQL username.
    • DB_PASSWORD: PostgreSQL password.
    • DB_HOST: PostgreSQL host (e.g., localhost, db).
    • DB_PORT: PostgreSQL port (default: 5432).
    • DB_NAME: PostgreSQL database name (e.g., maxun).
    • DATABASE_URL: Full PostgreSQL connection string (e.g., postgresql://user:password@host:port/database). This often overrides individual DB_ variables if present.

  • Application Settings:

    • PORT: Port for the Maxun server (default: 3000).
    • JWT_SECRET: A strong, unique secret for JWT token signing.
    • ENCRYPTION_KEY: A strong, unique key for encrypting sensitive data (e.g., in type or press actions within workflows). This is used by the decrypt function in server/src/utils/auth.ts and server/src/workflow-management/classes/Interpreter.ts.
    • MINIO_ENDPOINT, MINIO_PORT, MINIO_ACCESS_KEY, MINIO_SECRET_KEY, MINIO_USE_SSL: Configuration for MinIO or S3-compatible storage for binary outputs (e.g., screenshots).
    • PG_BOSS_CONNECTION_STRING: Connection string for PgBoss, typically derived from the main database connection. Ensure this is correctly set for the PgBoss worker (server/src/pgboss-worker.ts).

Example .env (Docker Compose):

PORT=3000
DATABASE_URL=postgresql://user:password@db:5432/maxun
DB_USER=user
DB_PASSWORD=password
DB_HOST=db
DB_PORT=5432
DB_NAME=maxun
JWT_SECRET=your_jwt_secret_here
ENCRYPTION_KEY=your_encryption_key_here

# MinIO/S3 Configuration (optional, for binary outputs)
MINIO_ENDPOINT=minio
MINIO_PORT=9000
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin
MINIO_USE_SSL=false

4. Build & Run

4.1. Development Mode (Local)

To run Maxun in development mode (without Docker):

  1. Install dependencies and set up .env as described in Installation without Docker and Configuration.
  2. Start the server: 
    npm run dev
    This will typically start the server on http://localhost:3000 (or your configured PORT).

4.2. Production Mode

Using Docker Compose (Recommended)

If you followed the Setup with Docker Compose instructions, Maxun is already running in a production-ready setup.

  1. Ensure .env is configured for production.
  2. Start services: 
    docker-compose up -d

Building and Running without Docker

  1. Install dependencies and set up .env as described in Installation without Docker and Configuration.
  2. Build the application: 
    npm run build
  3. Start the server: 
    npm start

5. Deployment

Maxun can be deployed to various cloud platforms that support Docker containers and PostgreSQL databases.

5.1. Self-Hosting with Docker & Portainer

The official documentation suggests using Portainer for self-hosting with Docker. This provides a GUI for managing your Docker environment.

  1. Install Docker and Portainer on your server.
  2. Clone the Maxun repository and configure your .env file as described in Configuration.
  3. Deploy using Docker Compose: You can use Portainer to deploy a stack from your docker-compose.yml file. Upload the docker-compose.yml and your .env file, then deploy the stack.

5.2. Cloud Platforms (e.g., AWS, Google Cloud, Azure)

For cloud deployments, consider the following architecture:

  • Container Orchestration: Use services like AWS ECS/EKS, Google Kubernetes Engine (GKE), or Azure Kubernetes Service (AKS) to run Maxun's Docker containers.
  • Database: Utilize managed PostgreSQL services (e.g., AWS RDS, Google Cloud SQL for PostgreSQL, Azure Database for PostgreSQL) for high availability and scalability.
  • Object Storage: For storing binary outputs (screenshots, etc.), use AWS S3, Google Cloud Storage, or Azure Blob Storage, configured via the MINIO_ environment variables.
  • Load Balancer: Place a load balancer in front of your Maxun instances for traffic distribution and SSL termination.

General Steps for Cloud Deployment:

  1. Containerize Maxun: The Dockerfile in the repository allows building a production-ready Docker image. 
    docker build -t maxun-app .
  2. Push to Container Registry: Push your built image to a cloud container registry (e.g., AWS ECR, Google Container Registry).
  3. Provision Database: Set up a managed PostgreSQL instance.
  4. Configure Environment Variables: Ensure all necessary environment variables (especially database credentials and JWT_SECRET, ENCRYPTION_KEY) are securely passed to your container instances.
  5. Deploy Containers: Deploy Maxun containers to your chosen orchestration service, ensuring they can connect to the database and object storage.
  6. Set up Networking: Configure networking, including load balancers, firewalls, and domain mapping.

6. Troubleshooting

6.1. Database Connection Issues

  • Error Message: Failed to start pgboss worker: one or more required environment variables are missing. or similar database connection errors.
  • Solution:
    • Double-check your DB_USER, DB_PASSWORD, DB_HOST, DB_PORT, DB_NAME, and DATABASE_URL in your .env file.
    • Ensure your PostgreSQL database is running and accessible from the Maxun server. If using Docker Compose, verify the db service is healthy.
    • Check firewall rules if deploying to a remote server.

6.2. ENCRYPTION_KEY Missing

  • Error Message: Errors related to decryption failures or missing encryption key.
  • Solution: Ensure ENCRYPTION_KEY is set in your .env file. This key is crucial for decrypting sensitive information stored in workflows (e.g., inputs for type or press actions).

6.3. Playwright Browser Issues

  • Error Message: Errors related to browser launch, connection, or Playwright.
  • Solution:
    • Ensure your Docker environment has sufficient resources (memory, CPU).
    • If running locally, ensure all Playwright dependencies are installed. Maxun's Docker image includes necessary browser dependencies.
    • Check logs for more specific Playwright errors. The browser-management/classes/RemoteBrowser.ts and browser-management/controller.ts files handle browser interactions.

6.4. Workflow Execution Problems

  • Issue: Workflows are not executing as expected, or data is not being extracted.
  • Solution:
    • Check Maxun UI: The Maxun application provides logs and run details for executed robots.
    • Server Logs: Review the Maxun server logs for errors during workflow interpretation (workflow-management/classes/Interpreter.ts) or generation (workflow-management/classes/Generator.ts).
    • Rate Limiting/Blocking: Websites might block or rate-limit scraping attempts. Consider using proxies (Maxun supports proxy configuration) or adjusting the robot's speed.
    • Selector Issues: If using Recorder Mode, website changes can break selectors. Re-record or manually adjust the workflow.
    • Scrape List Limit: Note that scrapeList actions have a default limit of 5 in certain contexts if checkLimit is true (as seen in server/src/routes/storage.ts and server/src/workflow-management/classes/Interpreter.ts). This might affect the amount of data extracted.

6.5. PgBoss Worker Not Starting

  • Error Message: Failed to start pgboss worker: one or more required environment variables are missing. (specifically from server/src/pgboss-worker.ts).
  • Solution:
    • Verify that all DB_USER, DB_PASSWORD, DB_HOST, DB_PORT, and DB_NAME environment variables are correctly set and accessible to the PgBoss worker process.
    • Ensure the PostgreSQL database is running and the connection string is valid.

For further assistance, refer to the official Maxun documentation or join the Discord community.

← Project overviewView alternatives to maxun →