# Upgrades & monitoring

Day-2 operations: how to upgrade to a new chart version and what to monitor once the vault is running.

## Helm upgrade flow

Every released chart version pins exact image tags for all services. When Sodot publishes a new version, you upgrade by pulling the new chart and running `helm upgrade` on each cluster.

### Within a major version

You can upgrade directly from any minor or patch version to any later version within the same major:

```bash
helm registry login repo.sodot.dev -u <your-username>
helm pull oci://repo.sodot.dev/sodot-helm-charts/crypto-vault --version <new-version>

# Upgrade clusters in order: spokes first, hub last
helm upgrade crypto-vault ./crypto-vault-<new-version>.tgz \
    --kubeconfig ./out/cluster-1.kubeconfig \
    --namespace crypto-vault \
    -f ./out/values.yaml

helm upgrade crypto-vault ./crypto-vault-<new-version>.tgz \
    --kubeconfig ./out/cluster-3.kubeconfig \
    --namespace crypto-vault \
    -f ./out/values.yaml

# Hub last (Orchestrator is here; upgrade it after spokes are ready)
helm upgrade crypto-vault ./crypto-vault-<new-version>.tgz \
    --kubeconfig ./out/cluster-2.kubeconfig \
    --namespace crypto-vault \
    -f ./out/values.yaml
```

:::note
Upgrade spokes before the hub. The Orchestrator (on the hub) is upgraded last so that spoke Key Store and Policy Server pods are already running the new version before the Orchestrator restarts.
:::

### Across major versions

A chart upgrade is atomic: one `helm upgrade` rolls every service to the exact images pinned by the new chart version. You cannot mix images from different chart versions, so there is no separate reader/writer staging step to run by hand. Kubernetes performs a rolling update per service, and signing stays available throughout as long as the cross-cluster threshold holds.

Breaking format changes are confined to major versions. **Move one major at a time; you cannot skip a major version.**

Example: upgrading from `2.3.1` to `6.3.5` requires the path `2.3.1 -> 3.0.0 -> 4.0.0 -> 5.0.0 -> 6.3.5`.

The chart enforces this automatically. A preflight hook refuses any `helm upgrade` command that would skip a required major version and tells you which version to install next.

## Health and readiness checks

### Basic pod health

```bash
kubectl --kubeconfig ./out/cluster-1.kubeconfig -n crypto-vault get pods
kubectl --kubeconfig ./out/cluster-2.kubeconfig -n crypto-vault get pods
kubectl --kubeconfig ./out/cluster-3.kubeconfig -n crypto-vault get pods
```

All pods should be `Running` with all containers `Ready`. Key Store pods on Nitro nodes take longer to reach Ready on first start or after an upgrade; allow several minutes before escalating.

### Orchestrator health endpoint

```bash
curl -sS https://<your-hostname>/health
```

A `200` response confirms the Orchestrator is up and has reached all six Key Store and Policy Server instances. This endpoint is also used by `helm test`:

```bash
helm test crypto-vault --kubeconfig ./out/cluster-2.kubeconfig -n crypto-vault
```

## What to monitor

Set up alerts on the following signals:

| Signal | Recommended alert |
|---|---|
| Pod not `Ready` (any cluster) | Alert immediately; Key Store and Policy Server outages reduce the signing threshold headroom |
| Orchestrator `/health` returning non-200 | Alert immediately; vault is unavailable to users |
| NATS JetStream consumer lag | Alert if consumers fall significantly behind; indicates async processing is backed up |
| RDS instance reachability (any cluster) | Alert immediately; database loss stops all persistence |
| KMS key usage denied | Alert immediately; KMS access loss prevents Key Store from decrypting shares |
| Key Store pod restart (any cluster) | Alert and investigate; enclave restarts require re-establishing the enclave session |

## Replica guidance

Default replica counts provide high availability for stateless services. The Key Store and Policy Server replicas tolerate pod disruptions within a single cluster without affecting the cross-cluster MPC threshold.

| Component | Default replicas | Notes |
|---|---|---|
| Key Store | 2 | One unified enclave image per shard. Each pod requires its own Nitro-capable node; scale the Nitro node group to match. |
| Orchestrator | 2 | Active-active; stateless between requests. |
| Policy Server | 2 | Reader and writer are independently replicated. |
| UI | 2 | Stateless; scale freely. |

Horizontal Pod Autoscaler is available for Orchestrator, Policy Server, and UI when `autoscaling.enabled: true`. This requires the `metrics-server` addon on the hub cluster.

Pod Disruption Budgets (`maxUnavailable: 1`) are enabled by default and prevent simultaneous eviction of more than one replica per role during node drains or cluster upgrades.