Vault Deployment and Usage Guide

This guide provides comprehensive instructions for deploying and using HashiCorp Vault, a tool for secrets management, encryption as a service, and privileged access management.

1. Prerequisites

Before deploying or running Vault, ensure you have the following installed:

Go (Golang): Version 1.12.0 or newer is recommended for full feature support (e.g., undo logs require Vault 1.12.0+).
- Download and Install Go
- Set up your GOPATH environment variable.
- Set GOBIN to $GOPATH/bin and ensure $GOPATH/bin is in your system's PATH.

2. Installation

This section covers cloning the Vault repository and setting up the development environment.

Clone the Repository: Clone the Vault repository outside of your GOPATH to leverage Go Modules.
```
git clone https://github.com/hashicorp/vault.git
cd vault
```
Bootstrap Development Environment: Download any required build tools by running the bootstrap command.
```
make bootstrap
```

3. Configuration

Vault's configuration is primarily managed through configuration files (HCL or JSON) and environment variables.

Environment Variables

VAULT_ADDR: Specifies the address of the Vault server. Example: export VAULT_ADDR='http://127.0.0.1:8200'
VAULT_TOKEN: Specifies the Vault token to use for authentication.
VAULT_CACERT / VAULT_CAPATH: Paths to CA certificates for TLS verification.
VAULT_CLIENT_CERT / VAULT_CLIENT_KEY: Paths to client TLS certificates and keys for authentication.

Configuration Files

Vault uses a configuration file (e.g., config.hcl) to define its behavior, including storage backend, listener addresses, and TLS settings.

Example config.hcl for development with file storage:

storage "file" {
  path = "vault/data"
}

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = "true"
}

# For production, enable TLS and provide certificates
# listener "tcp" {
#   address     = "0.0.0.0:8200"
#   tls_cert_file = "/etc/vault/tls/vault.crt"
#   tls_key_file  = "/etc/vault/tls/vault.key"
# }

Raft Storage Configuration (for High Availability):

Vault can use Raft for integrated storage and high availability.

storage "raft" {
  path    = "vault/data"
  node_id = "node1" # Unique ID for each node
  # For multi-node setup, specify peers:
  # retry_join {
  #   leader_api_addr = "http://<leader_ip>:8200"
  #   leader_ca_cert  = "/etc/vault/tls/vault_ca.crt"
  # }
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = "true" # Use TLS in production
}

api_addr = "http://<node_ip>:8200" # Must be reachable by other nodes
cluster_addr = "http://<node_ip>:8201" # For Raft communication

4. Build & Run

Development Build and Run

To compile and run a development version of Vault:

Compile Vault:
```
make dev
```
This will place the vault binary in the bin/ and $GOPATH/bin directories.
Run Vault in Development Mode: For a quick local development server that stores data in memory and is automatically unsealed:
```
bin/vault server -dev
```
This will output an unseal key and a root token.
Run Vault with a Configuration File: Create a config.hcl (see Configuration section) and run:
```
bin/vault server -config=config.hcl
```

Production Build

For a production-ready binary, simply run make. This will compile Vault without development-specific flags.

make

The compiled binary will be located in the bin/ directory.

5. Deployment

Vault can be deployed on various platforms, often utilizing its high-availability features with integrated Raft storage or external storage backends like Consul, AWS S3, or Google Cloud Storage.

Recommended Deployment Platforms

Kubernetes: Ideal for containerized deployments, leveraging Helm charts or custom operators for managing Vault instances, auto-unsealing, and high availability.
Virtual Machines (VMs): For traditional server deployments, Vault can run on Linux VMs (e.g., AWS EC2, Azure VMs, Google Cloud Compute Engine).
HashiCorp Nomad: For orchestrating Vault alongside other applications in a cluster.

General Deployment Steps (VMs/Bare Metal)

Install Vault:
- Download the appropriate pre-compiled binary from the Vault releases page.
- Place the vault binary in a directory included in your system's PATH (e.g., /usr/local/bin).

Create Configuration Directory:

sudo mkdir /etc/vault
sudo mkdir /var/lib/vault/data # For file or Raft storage

Create Configuration File: Place your config.hcl (see Configuration section) in /etc/vault/config.hcl. Ensure it's configured for your chosen storage backend (e.g., Raft, Consul, S3) and listener addresses. Always enable TLS for production deployments.

Systemd Service (Linux): Create a systemd service file (e.g., /etc/systemd/system/vault.service) to manage the Vault process.

[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://developer.hashicorp.com/vault/docs
Requires=network-online.target
After=network-online.target

[Service]
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/local/bin/vault server -config=/etc/vault/config.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
LimitNOFILE=65536
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target

Create a vault user and group: sudo useradd --system --home /etc/vault --shell /bin/false vault
Set permissions: sudo chown -R vault:vault /etc/vault /var/lib/vault/data

Start and Enable Service:

sudo systemctl daemon-reload
sudo systemctl enable vault
sudo systemctl start vault

Initialize and Unseal Vault: After starting, Vault needs to be initialized and unsealed.
```
vault operator init
```
This command outputs Share Keys and a Root Token. Securely store these. Then, unseal Vault using the share keys (typically 3 out of 5, or as configured):
```
vault operator unseal <key_share_1>
vault operator unseal <key_share_2>
vault operator unseal <key_share_3>
```
Log in and Configure: Log in using the root token and start configuring secrets engines, auth methods, and policies.
```
vault login <root_token>
```

Kubernetes Deployment

Use the official HashiCorp Vault Helm chart for Kubernetes deployments.

Add HashiCorp Helm Repository:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

Install Vault: Customize values.yaml for your specific needs (e.g., storage backend, HA, TLS, auto-unseal).

helm install vault hashicorp/vault --version <chart_version> --values my-values.yaml

Example my-values.yaml for a basic Raft HA setup:

server:
  image:
    repository: hashicorp/vault
    tag: "1.16.0" # Specify your desired Vault version
  dataStorage:
    enabled: true
    size: 10Gi
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: true
  # For auto-unseal with KMS, uncomment and configure:
  # unseal:
  #   enabled: true
  #   secretShares: 1
  #   secretThreshold: 1
  #   kms:
  #     aws:
  #       accessKey: "YOUR_AWS_ACCESS_KEY"
  #       secretKey: "YOUR_AWS_SECRET_KEY"
  #       region: "us-east-1"
  #       kmsKeyId: "arn:aws:kms:us-east-1:123456789012:key/your-kms-key-id"

Initialize and Unseal: If auto-unseal is not configured, you will need to exec into one of the Vault pods to initialize and manually unseal.

kubectl exec -it vault-0 -- vault operator init
# ... securely store keys and root token ...
kubectl exec -it vault-0 -- vault operator unseal <key_share_1>
kubectl exec -it vault-0 -- vault operator unseal <key_share_2>
kubectl exec -it vault-0 -- vault operator unseal <key_share_3>

6. Troubleshooting

Common Issues and Solutions

Vault won't start or immediately exits:
- Check logs: Examine the Vault server logs for errors (journalctl -u vault.service for systemd, or kubectl logs <vault-pod-name> for Kubernetes).
- Configuration errors: Validate your config.hcl using vault validate config.hcl.
- Permissions: Ensure the Vault user has correct read/write permissions to the data directory and configuration file.
- Port conflicts: Verify that the configured listener port is not already in use.
Vault is sealed:
- Vault starts in a sealed state for security. It must be unsealed before use.
- Use vault operator unseal <key_share> with the required number of unseal keys.
- If using auto-unseal, check the configuration for the KMS provider and ensure Vault has the necessary permissions to access it.
Error: permission denied when accessing secrets:
- This indicates an issue with the Vault token's policies.
- Verify the policies attached to your token using vault token lookup.
- Check the ACL policies defined in Vault to ensure they grant the necessary permissions to the paths you are trying to access.
TLS handshake errors:
- Certificate paths: Ensure tls_cert_file and tls_key_file paths in config.hcl are correct and the files are readable by the Vault user.
- Client certificates: If using client TLS authentication, ensure VAULT_CLIENT_CERT and VAULT_CLIENT_KEY are correctly set and valid.
- CA certificates: If using custom CAs, ensure VAULT_CACERT or VAULT_CAPATH is correctly configured on the client side.
Raft cluster not forming/leader election issues:
- Network connectivity: Ensure all Raft nodes can communicate with each other on both the api_addr and cluster_addr ports. Check firewalls and security groups.
- node_id uniqueness: Each Raft node must have a unique node_id in its configuration.
- retry_join configuration: Verify that retry_join stanzas correctly point to existing cluster members.
- Clock skew: Significant clock skew between Raft nodes can cause issues. Ensure NTP is configured.
- Logs: Check the logs on all Raft nodes for messages related to leader election, peer communication, or snapshot failures.
Plugin errors (ErrPluginNotFound, ErrPluginBadType, etc.):
- Plugin directory: Ensure the plugin_directory in your Vault configuration points to the correct location where your external plugins are stored.
- Plugin registration: Verify that the plugin is registered with Vault using vault plugin catalog list.
- Checksums: Ensure the plugin's SHA256 checksum matches the one registered in the catalog.
- Plugin type: Confirm the plugin type (e.g., secret, auth) matches its usage.
- Dependencies: Check if the plugin has any external dependencies that are not met.

How to Deploy & Use hashicorp/vault