Logstash Deployment and Usage Guide

Prerequisites

Runtime Requirements

Java Development Kit (JDK): Version 11 or 17
- Set the JAVA_HOME environment variable to your JDK installation directory
- Example: set JAVA_HOME=<JDK_PATH>
JRuby: Version 9.2.x
- Recommended to use a Ruby version manager like RVM or [rbenv]

Development Tools

Rake: Install with gem install rake
Bundler: Install with gem install bundler
Gradle: Required for building dependencies

Optional Tools

Drip: For faster JVM startup during development
- Set JAVACMD=which drip`` to use drip

Installation

Clone the Repository

git clone https://github.com/elastic/logstash.git
cd logstash

Set Environment Variables

export LOGSTASH_SOURCE=1
export LOGSTASH_PATH=/YOUR/LOGSTASH/DIRECTORY

Install Dependencies

Using Gradle (Recommended)

# Install development dependencies
./gradlew installDevelopmentGems

# Install default plugins and other dependencies
./gradlew installDefaultGems

For Open Source Version Only

export OSS=true

Verify Installation

bin/logstash -e 'input { stdin { } } output { stdout {} }'

This starts Logstash with stdin input waiting for events. Type "hello world" to test.

Configuration

Environment Variables

JAVA_HOME: Path to JDK installation
LOGSTASH_SOURCE: Set to 1 for source builds
LOGSTASH_PATH: Path to Logstash directory
OSS: Set to true for open-source only version
JAVACMD: Set to which drip for drip launcher (optional)

Configuration Files

Logstash uses configuration files with .conf extension. Example structure:

input {
  stdin { }
}

filter {
  # Your filters here
}

output {
  stdout { }
}

Plugin Configuration

Logstash plugins are hosted in separate repositories under the logstash-plugins organization. Each plugin is a self-contained Ruby gem.

Build & Run

Running Locally (Development)

# Basic test run
bin/logstash -e 'input { stdin { } } output { stdout {} }'

# Run with a configuration file
bin/logstash -f your-config.conf

# Run with debug logging
bin/logstash --debug -f your-config.conf

Running Locally (Production)

# Use the packaged version
bin/logstash -f /path/to/production.conf --path.data /var/lib/logstash

Using Drip for Development

JAVACMD=`which drip` bin/logstash -e 'input { stdin { } } output { stdout {} }'

Deployment

Platform Recommendations

Docker

FROM docker.elastic.co/logstash/logstash:8.8.1

# Add your pipeline configuration file
COPY logstash.conf /usr/share/logstash/pipeline/logstash.conf

Kubernetes

Deploy using Helm charts or custom manifests with persistent storage for queue data.

Cloud Platforms

AWS: EC2 instances or ECS containers
Google Cloud: Compute Engine or GKE
Azure: Virtual Machines or AKS

Package Managers

Debian/Ubuntu: Use .deb packages from Elastic downloads
RHEL/CentOS: Use .rpm packages from Elastic downloads

Production Considerations

Use persistent storage for queue data
Configure proper JVM heap size
Set up monitoring and logging
Configure appropriate number of pipeline workers
Enable dead letter queue for error handling

Troubleshooting

Common Issues and Solutions

1. Java Version Mismatch

Issue: Logstash fails to start with Java version errors Solution: Ensure JDK 11 or 17 is installed and JAVA_HOME is set correctly

2. Ruby Version Issues

Issue: Ruby version conflicts during development Solution: Use RVM or rbenv to manage Ruby versions, ensure JRuby 9.2.x is active

3. Slow Startup

Issue: Logstash takes too long to start Solution: Use Drip launcher for development: JAVACMD=which drip``

4. Plugin Installation Failures

Issue: Plugins fail to install or load Solution: Ensure proper Ruby environment and network connectivity to RubyGems.org

5. Memory Issues

Issue: OutOfMemory errors Solution: Adjust JVM heap size with -Xmx and -Xms parameters

6. File Permissions

Issue: Cannot write to data directory Solution: Ensure proper permissions on data directories and configure --path.data

7. Queue Corruption

Issue: Persistent queue data becomes corrupted Solution: Check disk space, file system integrity, and consider using --path.data on different filesystem

Getting Help

Testing

Run unit tests using rspec for Ruby parts and junit for Java parts:

# Run all tests
bin/rspec

# Run specific test file
bin/rspec spec/path/to/test_spec.rb

How to Deploy & Use elastic/logstash