Twemproxy (Nutcracker) Deployment Guide

1. Prerequisites

System Requirements

Operating System: Linux, *BSD, OS X, or SmartOS (Solaris)
Compiler: GCC (newer versions recommended; older versions may have compilation issues)
Build Tools: automake, libtool, autoreconf
Backend Services: Running memcached or redis instances to proxy

Install Build Dependencies

Debian/Ubuntu:

sudo apt-get install build-essential automake libtool

RHEL/CentOS/Fedora:

sudo yum install gcc automake libtool
# or
sudo dnf install gcc automake libtool

macOS:

brew install automake libtool

2. Installation

Option A: Install from Distribution Tarball (Recommended for 0.5.0+)

# Download from https://github.com/twitter/twemproxy/releases
wget https://github.com/twitter/twemproxy/releases/download/0.5.0/twemproxy-0.5.0.tar.gz
tar -xzf twemproxy-0.5.0.tar.gz
cd twemproxy-0.5.0

./configure
make
sudo make install

Option B: Build from Source (Git)

git clone https://github.com/twitter/twemproxy.git
cd twemproxy
autoreconf -fvi
./configure
make
sudo make install

Verify Installation

make check        # Run unit tests
src/nutcracker -h # Display help/usage

Debug Build (Development)

# Full debug with symbols
CFLAGS="-ggdb3 -O0" ./configure --enable-debug=full
make
sudo make install

# Debug logs and assertions enabled
autoreconf -fvi
./configure --enable-debug=full
make

Compiler Optimization Flags

If encountering build issues, try alternative optimization levels:

CFLAGS="-O1" ./configure && make
# or
CFLAGS="-O3 -fno-strict-aliasing" ./configure && make

3. Configuration

Twemproxy uses a YAML configuration file to define server pools. Create a configuration file (e.g., /etc/nutcracker.yml):

Basic Configuration Structure

# Example pool configuration
alpha:
  listen: 127.0.0.1:22121
  hash: fnv1a_64
  distribution: ketama
  auto_eject_hosts: true
  redis: true
  server_retry_timeout: 2000
  server_failure_limit: 1
  servers:
   - 127.0.0.1:6379:1
   - 127.0.0.1:6380:1

beta:
  listen: 127.0.0.1:22122
  hash: fnv1a_64
  hash_tag: "{}"
  distribution: ketama
  auto_eject_hosts: false
  timeout: 400
  redis: true
  servers:
   - 127.0.0.1:6381:1
   - 127.0.0.1:6382:1

Key Configuration Parameters

Parameter	Description	Example
`listen`	Address:port or socket path for client connections	`127.0.0.1:22121` or `/var/run/nutcracker.sock`
`hash`	Hash function for key distribution	`fnv1a_64` (default), `md5`, `crc32`, `murmur`, `jenkins`
`hash_tag`	Two-character string for key hashing (enables mapping related keys)	`{}`, `$$`
`distribution`	Key distribution mode	`ketama`, `modula`, `random`
`servers`	List of backend servers with weight	`- 127.0.0.1:6379:1`
`auto_eject_hosts`	Remove failed nodes from pool	`true` or `false`
`redis`	Enable Redis protocol (vs memcached)	`true` or `false`
`timeout`	Connection timeout in milliseconds	`400`
`client_connections`	Max client connections (unlimited by default)	`10000`

Validate Configuration

nutcracker -t -c /etc/nutcracker.yml
# or
nutcracker --test-conf --conf-file=/etc/nutcracker.yml

4. Build & Run

Development Mode

Run in foreground with verbose logging:

src/nutcracker -c conf/nutcracker.yml -v 6

Production Mode

Run as daemon with logging and monitoring:

nutcracker \
  -d \
  -c /etc/nutcracker.yml \
  -o /var/log/nutcracker.log \
  -p /var/run/nutcracker.pid \
  -s 22222 \
  -a 0.0.0.0 \
  -i 30000 \
  -v 5

Command-Line Options

Flag	Description	Default
`-d, --daemonize`	Run as daemon	Disabled
`-c, --conf-file`	Configuration file path	`conf/nutcracker.yml`
`-o, --output`	Log file path	`stderr`
`-p, --pid-file`	PID file path	Disabled
`-s, --stats-port`	Stats monitoring port	`22222`
`-a, --stats-addr`	Stats bind address	`0.0.0.0`
`-i, --stats-interval`	Stats aggregation interval (ms)	`30000`
`-m, --mbuf-size`	Memory buffer chunk size (bytes)	`16384`
`-v, --verbose`	Logging level (0-11)	`5`
`-t, --test-conf`	Test configuration syntax and exit	-
`-D, --describe-stats`	Print stats description and exit	-

Memory Tuning ( mbuf )

For high-concurrency deployments, adjust mbuf size to balance memory usage vs syscalls:

# Small mbuf for many connections (512 bytes)
nutcracker -c /etc/nutcracker.yml -m 512

# Large mbuf for high throughput (default 16KB)
nutcracker -c /etc/nutcracker.yml -m 16384

Rule of thumb: Use smaller mbuf sizes (512-2048 bytes) when handling >10,000 concurrent connections; use default 16KB for lower connection counts with high throughput.

5. Deployment

Systemd Service

Create /etc/systemd/system/nutcracker.service:

[Unit]
Description=Twemproxy (Nutcracker) Proxy
After=network.target

[Service]
Type=forking
PIDFile=/var/run/nutcracker.pid
ExecStart=/usr/local/sbin/nutcracker -d -c /etc/nutcracker.yml -p /var/run/nutcracker.pid -o /var/log/nutcracker.log
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable nutcracker
sudo systemctl start nutcracker
sudo systemctl status nutcracker

Docker Deployment

FROM alpine:latest
RUN apk add --no-cache build-base automake libtool git && \
    git clone https://github.com/twitter/twemproxy.git && \
    cd twemproxy && \
    autoreconf -fvi && \
    ./configure && \
    make && \
    make install && \
    apk del build-base automake libtool git
COPY nutcracker.yml /etc/nutcracker.yml
EXPOSE 22121 22222
CMD ["nutcracker", "-c", "/etc/nutcracker.yml", "-m", "512"]

Production Checklist

Connection Limits: Ensure ulimit -n (file descriptors) > max client connections + backend connections
Monitoring: Enable stats port (-s 22222) for observability
Hash Tags: Configure hash_tag (e.g., {}) to ensure related keys map to same server
Auto-ejection: Enable auto_eject_hosts: true for fault tolerance
Logging: Use -o to redirect logs from stderr to file in daemon mode
Memory: Calculate memory usage: connections × mbuf_size × 2 (request + response buffers)

Load Balancer Integration

Place twemproxy behind your application load balancer or run locally on application servers:

App Server → Local Nutcracker (127.0.0.1:22121) → Redis/Memcached Cluster

6. Troubleshooting

Build Issues

Error: autoreconf: not found

# Install automake and libtool
sudo apt-get install automake libtool  # Debian/Ubuntu
sudo yum install automake libtool      # RHEL/CentOS

Error: gcc: internal compiler error or optimization failures

# Use lower optimization or disable strict aliasing
CFLAGS="-O1" ./configure && make
# or
CFLAGS="-O3 -fno-strict-aliasing" ./configure && make

Error: undefined reference to...

Ensure using GCC (not Clang) and version is recent (4.8+)

Runtime Issues

High Memory Usage

Cause: Default 16KB mbuf × many connections
Solution: Reduce mbuf size: nutcracker -m 512

Connection Refused Errors

Verify backend servers are running and accessible
Check firewall rules between twemproxy and backends
Verify listen address in config matches connection string

Keys Not Distributing Evenly

Verify hash function choice (fnv1a_64 recommended)
Check hash_tag configuration if using key patterns
Ensure server weights are balanced in config

Configuration Parse Errors

# Validate syntax before starting
nutcracker -t -c /etc/nutcracker.yml

Stats Monitoring Access stats via TCP for debugging:

echo "stats" | nc localhost 22222
# or
telnet localhost 22222
> stats
> quit

Hot Keys/Server Imbalance

Implement hash_tag to group related keys (e.g., user:{123}:profile and user:{123}:settings)
Consider consistent hashing (ketama) for better distribution

Debug Logging

Increase verbosity for troubleshooting (levels 0-11):

# Debug level logging
nutcracker -c /etc/nutcracker.yml -v 11 -o /tmp/nutcracker-debug.log

How to Deploy & Use twitter/twemproxy