# Upgrades & monitoring

Day-2 operations: how to upgrade to a new chart version and what to monitor once the vault is running.

## Helm upgrade flow

Every released chart version selects a tested set of service images. Keep those image coordinates
aligned with the chart. When Sodot publishes a new version, review its migration notes first, then
pull the new chart and run `helm upgrade` on each cluster. When no release-specific migration applies,
use the flow below.

### Within a major version

For a chart-only upgrade with no documented Terraform or data migration, you can upgrade directly
from any minor or patch version to any later version within the same major:

```bash
helm registry login repo.sodot.dev -u <your-username>
helm pull oci://repo.sodot.dev/sodot-helm-charts/crypto-vault --version <new-version>

# Upgrade clusters in order: spokes first, hub last
helm upgrade crypto-vault ./crypto-vault-<new-version>.tgz \
    --kubeconfig ./out/cluster-0.kubeconfig \
    --namespace crypto-vault \
    -f ./out/cluster-0.values.yaml

helm upgrade crypto-vault ./crypto-vault-<new-version>.tgz \
    --kubeconfig ./out/cluster-2.kubeconfig \
    --namespace crypto-vault \
    -f ./out/cluster-2.values.yaml

# Hub last (Orchestrator is here; upgrade it after spokes are ready)
helm upgrade crypto-vault ./crypto-vault-<new-version>.tgz \
    --kubeconfig ./out/cluster-1.kubeconfig \
    --namespace crypto-vault \
    -f ./out/cluster-1.values.yaml
```

If `chart_ha = false`, append `-f ./out/extra.values.yaml` to every command.

:::note
Upgrade spokes before the hub. The Orchestrator (on the hub) is upgraded last so that spoke Key Store and Policy Server pods are already running the new version before the Orchestrator restarts.
:::

### Across major versions

A routine chart upgrade uses one `helm upgrade` to roll every service to the chart-selected compatible
image set, including any explicitly qualified overrides. Keep that set aligned; there is no separate
reader/writer staging step to run by hand. Kubernetes performs a rolling update per service, and
signing stays available as long as the cross-cluster threshold holds. Release-specific infrastructure
or data migrations can require a maintenance window and take precedence over this routine flow.

Breaking format changes are confined to major versions. **Move one major at a time; you cannot skip a major version.**

Example: upgrading from `2.3.1` to `6.3.5` requires the path `2.3.1 -> 3.0.0 -> 4.0.0 -> 5.0.0 -> 6.3.5`.

The chart enforces this automatically. A preflight hook refuses any `helm upgrade` command that would skip a required major version and tells you which version to install next.

## Health and readiness checks

### Basic pod health

```bash
kubectl --kubeconfig ./out/cluster-0.kubeconfig -n crypto-vault get pods
kubectl --kubeconfig ./out/cluster-1.kubeconfig -n crypto-vault get pods
kubectl --kubeconfig ./out/cluster-2.kubeconfig -n crypto-vault get pods
```

All pods should be `Running` with all containers `Ready`. Key Store pods can take longer on first start
or after an upgrade while AWS launches the Nitro Enclave or GCP initializes the pod's Cloud KMS access.

### Orchestrator health endpoint

```bash
curl -sS https://<your-hostname>/health
```

A `200` response confirms the Orchestrator is up and has reached all six Key Store and Policy Server instances. This endpoint is also used by `helm test`:

```bash
helm test crypto-vault --kubeconfig ./out/cluster-1.kubeconfig -n crypto-vault
```

## What to monitor

Set up alerts on the following signals:

| Signal | Recommended alert |
|---|---|
| Pod not `Ready` (any cluster) | Alert immediately; Key Store and Policy Server outages reduce the signing threshold headroom |
| Orchestrator `/health` returning non-200 | Alert immediately; vault is unavailable to users |
| NATS JetStream consumer lag | Alert if consumers fall significantly behind; indicates async processing is backed up |
| RDS or Cloud SQL reachability (any cluster) | Alert immediately; database loss stops all persistence |
| KMS key usage denied | Alert immediately; KMS access loss prevents Key Store from decrypting shares |
| Key Store pod restart (any cluster) | Alert and investigate repeated restarts or readiness failures |

## Replica guidance

Default replica counts provide high availability for stateless services. The Key Store and Policy Server replicas tolerate pod disruptions within a single cluster without affecting the cross-cluster MPC threshold.

| Component | Default replicas | Notes |
|---|---|---|
| Key Store | 2 | One unified Key Store per shard. Scale the Nitro-capable node group on AWS or the Confidential node pool on GCP to fit. |
| Orchestrator | 2 | Active-active; stateless between requests. |
| Policy Server | 2 | Reader and writer are independently replicated. |
| UI | 2 | Stateless; scale freely. |

Horizontal Pod Autoscaler is available for Orchestrator, Policy Server, and UI when `autoscaling.enabled: true`. This requires the `metrics-server` addon on the hub cluster.

Pod Disruption Budgets (`maxUnavailable: 1`) are enabled by default and prevent simultaneous eviction of more than one replica per role during node drains or cluster upgrades.