Codakey
Play
← Back to alibaba/zvec

How to Deploy & Use alibaba/zvec

Zvec Deployment and Usage Guide

1. Prerequisites

Before installing Zvec, ensure your system meets the following requirements:

Operating Systems:

  • Linux (x86_64 or ARM64)
  • macOS (ARM64 - Apple Silicon)

Python (for Python package):

  • Python 3.10, 3.11, or 3.12
  • pip package manager

Node.js (for Node.js package):

  • Node.js runtime
  • npm package manager

For building from source:

  • C++17 compatible compiler (GCC, Clang, or MSVC)
  • CMake (version 3.10 or higher)
  • Git

2. Installation

Python Installation (Recommended)

pip install zvec

Node.js Installation

npm install @zvec/zvec

Building from Source

# Clone the repository
git clone https://github.com/alibaba/zvec.git
cd zvec

# Build the project
mkdir build && cd build
cmake ..
make -j$(nproc)

# For Python bindings
pip install -e ../python

3. Configuration

Zvec is an in-process vector database that requires minimal configuration. The primary configuration is done through the CollectionSchema when creating collections.

Environment Variables

No environment variables are required for basic operation. Zvec operates entirely within your application process.

Collection Schema Configuration

When creating a collection, define your schema with the appropriate vector dimensions and data types:

import zvec

# Define your collection schema
schema = zvec.CollectionSchema(
    name="my_collection",
    vectors=zvec.VectorSchema(
        name="embedding",  # Vector field name
        dtype=zvec.DataType.VECTOR_FP32,  # Data type
        dim=768  # Vector dimension
    ),
    # Optional: Add scalar fields for hybrid search
    scalars=[
        zvec.ScalarSchema("category", zvec.DataType.STRING),
        zvec.ScalarSchema("timestamp", zvec.DataType.INT64)
    ]
)

Storage Configuration

Zvec stores data locally in the specified directory:

# Data will be stored in ./my_zvec_data directory
collection = zvec.create_and_open(path="./my_zvec_data", schema=schema)

4. Build & Run

Development Setup

import zvec

# 1. Define schema
schema = zvec.CollectionSchema(
    name="test_collection",
    vectors=zvec.VectorSchema("embedding", zvec.DataType.VECTOR_FP32, 4)
)

# 2. Create collection
collection = zvec.create_and_open(path="./test_db", schema=schema)

# 3. Insert data
collection.insert([
    zvec.Doc(id="doc_1", vectors={"embedding": [0.1, 0.2, 0.3, 0.4]}),
    zvec.Doc(id="doc_2", vectors={"embedding": [0.2, 0.3, 0.4, 0.1]}),
    zvec.Doc(id="doc_3", vectors={"embedding": [0.3, 0.4, 0.1, 0.2]}),
])

# 4. Query data
results = collection.query(
    zvec.VectorQuery("embedding", vector=[0.4, 0.3, 0.3, 0.1]),
    topk=2
)

print(f"Search results: {results}")

Production Considerations

For production use:

  1. Memory Management: Monitor memory usage as Zvec loads vectors into memory for fast access
  2. Persistence: Data is automatically persisted to disk, but consider regular backups
  3. Concurrency: Zvec supports concurrent read operations; for write-heavy workloads, implement appropriate locking

Running Tests

# From the source directory
cd build
ctest --output-on-failure

5. Deployment

Deployment Options

1. Embedded in Applications (Recommended) Zvec is designed as an in-process database, making it ideal for:

  • Embedding directly in Python/Node.js applications
  • Microservices
  • Desktop applications
  • Edge computing devices

2. Container Deployment

FROM python:3.11-slim

# Install Zvec
RUN pip install zvec

# Copy your application
COPY app.py /app/
WORKDIR /app

# Run your application
CMD ["python", "app.py"]

3. Serverless Functions Zvec works well with serverless platforms:

  • AWS Lambda (Python/Node.js)
  • Google Cloud Functions
  • Azure Functions
  • Vercel/Netlify functions

4. Traditional Servers Deploy as part of your application stack on:

  • Bare metal servers
  • Virtual machines
  • Kubernetes pods

Scaling Considerations

  • Vertical Scaling: Increase memory for larger vector datasets
  • Sharding: Implement application-level sharding for datasets exceeding single-machine memory
  • Hybrid Architectures: Combine Zvec with traditional databases for metadata storage

6. Troubleshooting

Common Issues and Solutions

Issue: Installation fails on unsupported platform

Solution: Ensure you're using a supported platform (Linux x86_64/ARM64 or macOS ARM64).
For other platforms, build from source.

Issue: Memory errors with large datasets

Solution: 
1. Monitor memory usage: Zvec loads vectors into memory
2. Consider using smaller vector dimensions
3. Implement data sharding across multiple collections
4. Ensure sufficient swap space is configured

Issue: Slow query performance

Solution:
1. Check vector dimension matches schema definition
2. Ensure data is properly indexed (Zvec automatically creates indexes)
3. Consider batch queries for multiple search vectors
4. Verify system has sufficient CPU resources

Issue: Python import error

# Solution: Reinstall with verbose output
pip uninstall zvec -y
pip install zvec --verbose

# Check Python version compatibility
python --version  # Should be 3.10-3.12

Issue: Node.js binding errors

# Solution: Rebuild native modules
npm rebuild @zvec/zvec

# Check Node.js compatibility
node --version

Issue: Data persistence problems

Solution:
1. Ensure write permissions to the data directory
2. Check disk space availability
3. Verify proper shutdown sequence (collections auto-close on program exit)
4. For critical data, implement backup strategy

Getting Help

Performance Tips

  1. Use appropriate vector dimensions for your use case
  2. Batch insert operations when adding large datasets
  3. Consider using FP16 vectors if precision requirements allow
  4. Implement caching for frequent queries
  5. Use hybrid search with scalar filters to narrow result sets
← Project overviewView alternatives to alibaba/zvec →