# Provision infrastructure

The example Terraform module provisions the full three-cluster AWS topology (and all the cross-cluster
networking), then writes per-cluster Helm **handoff files** to `./out/`. Terraform **never installs the
chart** - you install it yourself in the next step, [Install with Helm](/deploy/helm/install). You can
also bring your own infrastructure (provision the resources below any way you like) and skip straight
to the Helm step.

## What must exist

| Resource | Where |
|---|---|
| EKS cluster with a Nitro-capable node group for the Key Store workload | All three sub-accounts |
| Standard node group for Policy Server (and on the hub: Orchestrator, NATS, relay, UI) | All three sub-accounts |
| RDS PostgreSQL with `mpc_ks_db` and `policy_db_N` logical databases | All three sub-accounts |
| RDS PostgreSQL with `orchestrator_db` logical database | Hub sub-account only |
| AWS KMS symmetric key (storage encryptor for the Key Store) | All three sub-accounts |
| IRSA IAM role granting `kms:Decrypt` and `kms:GenerateDataKey` to Key Store pods | All three sub-accounts |
| Cross-cluster network connectivity (see [Architecture](/deploy/architecture) for required ports) | Spans all three sub-accounts |
| Load balancer, TLS, and DNS for the hub UI and the orchestrator REST API | Hub sub-account only |
| Kubernetes Secrets for image pull credentials (`artifactory-dockerconfig`) | All three namespaces |

The example provisions all of this, plus the in-cluster add-ons (AWS Load Balancer Controller, Cluster
Autoscaler, the Nitro Enclaves device plugin) and a one-shot Job that bootstraps the databases. Because
every cross-cluster NLB and PrivateLink endpoint is a Terraform resource, the handoff values files are
complete before any Helm install - there is no post-install patching.

## The example Terraform module

:::warning\[Example module disclaimer]
The example Terraform is a concrete starting point, not production-ready code. It assumes one operator
with CLI access to all three sub-accounts. In a production environment, each sub-account should be
administered by a different person with separate credentials. Adapt or rewrite the module to match
your change-management and secrets-handling practices.
:::

### 1. Get a JFrog access token

The module pulls the chart images, the building-block Terraform module, and the measured enclave image
from `repo.sodot.dev`. Open the **Set Me Up** panel on the `sodot-terraform-modules` and
`sodot-helm-charts` repos in the JFrog UI to mint or copy an access token, then export it:

```bash
export ARTIFACTORY_USERNAME=<your-username>
export ARTIFACTORY_TOKEN=<your-access-token>
```

### 2. Download and unpack the example module

The example is published as a versioned zip artifact. Browse the available versions in JFrog at
[`sodot-terraform-modules/sodot/crypto-vault-example/aws`](https://repo.sodot.dev/ui/repos/tree/General/sodot-terraform-modules/sodot/crypto-vault-example/aws/)
and pick the **latest** `vX.Y.Z.zip`. Download it straight from the JFrog UI (select the artifact, then
**Download**), or with curl - replace `vX.Y.Z` with the version you picked:

```bash
# Replace vX.Y.Z with the latest version from the link above.
curl -fSL -u "$ARTIFACTORY_USERNAME:$ARTIFACTORY_TOKEN" \
  -o crypto-vault-example-aws.zip \
  https://repo.sodot.dev/artifactory/sodot-terraform-modules/sodot/crypto-vault-example/aws/vX.Y.Z.zip

unzip crypto-vault-example-aws.zip -d crypto-vault-example-tf
cd crypto-vault-example-tf
```

The unpacked directory is flat (`main.tf`, `variables.tf`, `terraform.tfvars.example`, `providers.tf`,
`outputs.tf`, `helm-handoff.tf`, `versions.tf`, plus `README.md` and `TERRAFORM.md`). It references the
building-block module from the Terraform registry, so Terraform needs registry credentials. The
simplest way is an environment variable Terraform reads automatically (host dots become underscores):

```bash
export TF_TOKEN_repo_sodot_dev="$ARTIFACTORY_TOKEN"
```

Alternatively run `terraform login repo.sodot.dev`, or add a `credentials "repo.sodot.dev" { token = "…" }`
block to `~/.terraformrc`.

:::warning\[Terraform 1.14+ required]
The example and building-block modules require Terraform `>= 1.14` (Sodot pins 1.15.x). `terraform init`
refuses to run on older versions. Check with `terraform version` and upgrade if needed.
:::

### 3. Configure AWS CLI profiles

Each of the three clusters resolves its AWS profile and region independently, so you choose the account
topology by how many profiles you configure:

* **Full MPC (recommended for production): one account per shard.** This is the trust boundary - a
  single account compromise then exposes at most one Key Store + one Policy Server, below the MPC
  threshold. Configure three profiles:

  ```ini
  # ~/.aws/config
  [profile cv-hub]
  sso_session    = <your-sso-session>
  sso_account_id = <hub-account-id>
  sso_role_name  = <role-with-infra-admin-permissions>
  region         = <aws-region>

  [profile cv-spoke-0]
  sso_session    = <your-sso-session>
  sso_account_id = <spoke-0-account-id>
  sso_role_name  = <role-with-infra-admin-permissions>
  region         = <aws-region>

  [profile cv-spoke-1]
  sso_session    = <your-sso-session>
  sso_account_id = <spoke-1-account-id>
  sso_role_name  = <role-with-infra-admin-permissions>
  region         = <aws-region>
  ```

* **Single-account PoC (for staging or dev envs): one profile.** All three clusters land in
  one account - still three separate VPCs/EKS clusters meshed over PrivateLink.

Log in and verify whichever profiles you configured:

```bash
aws sso login --sso-session <your-sso-session>   # if you use AWS SSO
aws sts get-caller-identity --profile cv-hub
aws sts get-caller-identity --profile cv-spoke-0
aws sts get-caller-identity --profile cv-spoke-1
```

:::warning\[Trust model]
Whenever two clusters resolve to the **same** account, that account's compromise can expose 2 of the 3
MPC shards - at or above the signing threshold. A single- or two-account layout is fine for a test, but
**do not ship a shared-account layout to production.**
:::

### 4. Configure `terraform.tfvars`

Copy the template and edit it:

```bash
cp terraform.tfvars.example terraform.tfvars
$EDITOR terraform.tfvars
```

The module's interface is **profile-based at the root** (it sets each cluster's `is_hub` / `shard_index`
internally - you do not). The variables you set:

| Variable | Required | Description |
|---|---|---|
| `aws_profile` | yes | Base AWS CLI profile used by all three clusters unless overridden. For a single-account PoC, set just this. |
| `aws_region` | no (default `us-east-1`) | Base region for all three clusters unless overridden. |
| `aws_profile_hub` / `aws_profile_spoke_0` / `aws_profile_spoke_1` | no | Per-cluster profile overrides. Set all three to three distinct accounts for the Full MPC topology. Any left unset inherit `aws_profile`. |
| `aws_region_hub` / `aws_region_spoke_0` / `aws_region_spoke_1` | no | Per-cluster region overrides (multi-region; cross-region PrivateLink adapts automatically). |
| `deployment_name` | no (default `crypto-vault`) | Prefix for all resource names in each account. |
| `artifactory_username` / `artifactory_password` | yes (sensitive) | Your JFrog username and access token (used to pull chart images). |
| `relay_api_key` | yes (sensitive) | Relay API key, shared by all clusters. Any non-empty string (e.g. `openssl rand -base64 16`). |
| `mpc_key_stores` | yes (sensitive) | Exactly **3** shard identities: `{ name, backupEncryptorPublicKey }` (plus optional `importedKeyEncryptorPem`). The MPC keypair is **self-generated by each node on first boot** - you supply only the name and the backup encryptor public key. |
| `policy_servers` | yes (sensitive) | Exactly **3** policy identities: `{ writerName, readerName }`. Identity keypairs are self-generated on first boot. |
| `load_balancer_enabled` / `internet_facing` | no (both default `true`) | Master switches for **both** front-door NLBs (vault UI + orchestrator API): whether to create them, and whether they are public. Overridable per-LB on the objects below. |
| `ui_load_balancer`, `ui_cert_domain` | yes (LB on by default) | UI front door: `host` plus a TLS cert. See [Expose the UI and API](#5-expose-the-ui-and-orchestrator-api-tls--dns). |
| `orchestrator_load_balancer`, `orchestrator_cert_domain` | yes (LB on by default) | Orchestrator API front door: its own `host` plus a TLS cert, distinct from the UI's. |
| `ui_dns_zone` | no | Public Route53 zone (hub account) for managed alias records of **both** hosts; leave empty to point your own DNS at the `ui_nlb_dns_name` / `orchestrator_nlb_dns_name` outputs. |
| `google_oauth_client_id` / `ui_google_oauth_client_secret` / `google_admin_emails` | no | Google OAuth client + Setup Admin emails for UI login. Leave empty for a chart-only deploy you won't log into. |
| `db_instance_type` | no (default `db.t3.small`) | RDS instance class per cluster. |
| `eks_node_instance_types` | no (default `["c5.xlarge"]`) | Nitro-Enclave-capable instance type(s) for the Key Store node group (non-burstable, non-metal, >=4 vCPU). |
| `eks_endpoint_public_access` | no (example default `true`) | Whether the EKS API endpoint is public (still IAM-gated). Set `false` for a private-only deploy run from inside the VPC/VPN. Tighten `eks_endpoint_public_access_cidrs` for anything beyond a test. |
| `db_deletion_protection` | no (default `true`) | Set `false` for a throwaway test so `terraform destroy` is painless. |
| `vpc_single_nat_gateway` / `db_multi_az` / `chart_ha` | no | Cost-vs-HA toggles. Defaults are production-shaped HA; set `vpc_single_nat_gateway = true`, `db_multi_az = false`, `chart_ha = false` for the cheapest test footprint. |

:::note\[Backup key gate]
`mpcKeyStores[].backupEncryptorPublicKey` must be set before any key can be created or imported - it is
a hard system gate. For a test you may use any valid 64-hex public key; for production, generate the
backup keypair offline and keep the private half out of the cluster (see [Backup & DR](/deploy/operations/backup-dr)).
:::

:::warning\[Terraform state security]
Terraform state files contain sensitive values (keys, passwords) in plaintext. Use an encrypted S3
backend with SSE-KMS, or equivalent, before running `terraform apply`. The generated `./out/` handoff
files also embed secrets - they are gitignored; treat them like credentials.
:::

### 5. Expose the UI and orchestrator API (TLS & DNS)

The hub serves **two** public front doors, each behind its own Network Load Balancer with a TLS listener
on **your** domain: the **vault UI** and the **orchestrator REST API**. Two master switches govern both,
each overridable per-LB:

```hcl
load_balancer_enabled = true   # create both front-door NLBs (false = create neither)
internet_facing       = true   # public; false = internal (VPC/VPN-only reach)

ui_load_balancer = {
  host = "vault.example.com"      # YOUR UI hostname (drives the OAuth redirect + WebAuthn origin)
  # certificate_arn = "arn:aws:acm:<hub-region>:<account>:certificate/<id>"   # explicit cert
  # enabled         = false       # per-LB override of load_balancer_enabled
  # internet_facing = false       # per-LB override of the master
}
ui_cert_domain = "*.example.com"  # ...or look the UI cert up by domain (alternative to certificate_arn)

orchestrator_load_balancer = {
  host = "vault-api.example.com"  # YOUR API hostname, distinct from the UI's
  # certificate_arn = "arn:aws:acm:<hub-region>:<account>:certificate/<id>"
}
orchestrator_cert_domain = "*.example.com"  # ...or look the API cert up by domain

# DNS: a public Route53 zone in the hub account gets managed alias records for BOTH hosts,
# or leave empty and point your own DNS at the *_nlb_dns_name outputs.
# ui_dns_zone = "example.com"
```

Both NLBs are **enabled and internet-facing by default**. Each requires its own `host` and a TLS cert
(`certificate_arn`, or the matching `*_cert_domain` to look one up in ACM **in the hub region**) - the
plan fails validation otherwise. Override `enabled` / `internet_facing` on either object to diverge from
the masters (e.g. expose the UI publicly but keep the API internal).

| Need | How |
|---|---|
| **TLS cert** covering each host | `*_load_balancer.certificate_arn` (an ISSUED ACM cert in the hub region) or `*_cert_domain` to look one up. Required for each enabled LB. |
| **DNS for each host** | Set `ui_dns_zone` to one public Route53 zone in the hub account for managed aliases of **both** hosts, or leave it empty and point your own DNS at `ui_nlb_dns_name` / `orchestrator_nlb_dns_name` (use `*_nlb_zone_id` for alias records). |
| **OAuth login** | Register `https://<ui-host>/auth/callback` on your `google_oauth_client_id`, or Google login fails with `redirect_uri_mismatch`. |
| **UI -> API** | The UI's browser **and server-side** calls reach the orchestrator at `orchestrator_load_balancer.host`; that host **must resolve** (managed alias or your own DNS) or the UI returns `500`s on every API call (`whoami`, key list, even passkey enrollment). |

To create neither front door, set `load_balancer_enabled = false` (then expose the Services yourself).

### 6. Apply

```bash
terraform init -upgrade
terraform apply
```

:::warning\[AWS SSO token refresh]
If your profiles use the newer `sso_session` config format and `terraform apply` fails immediately with
`failed to refresh cached SSO token` / `InvalidGrantException`, run `aws sso login --sso-session <name>`
**right before** `apply`. The Terraform AWS provider can't refresh a near-expiry `sso_session` token
mid-run (the AWS CLI can, which is why `aws sts get-caller-identity` still works); a fresh login mints a
token valid for the whole apply.
:::

A first apply takes roughly **25-35 minutes** - the three EKS clusters are the long pole and build in
parallel. A single `terraform apply` provisions everything (VPC, EKS, RDS, KMS, IAM, in-cluster add-ons,
the database bootstrap Job, and every cross-cluster NLB + PrivateLink endpoint), then writes the Helm
handoff files. It does **not** install the chart.

### 7. Handoff files and verification

`terraform apply` writes these to `./out/` (gitignored; the values files embed secrets):

| File | What it is |
|---|---|
| `out/cluster-0.values.yaml` | spoke-0 (shard 0) chart values |
| `out/cluster-1.values.yaml` | **hub** (shard 1) chart values - also carries `remoteMpcKeyStores[]` / `remotePolicyServers[]` and the NATS/relay NLB addresses |
| `out/cluster-2.values.yaml` | spoke-1 (shard 2) chart values |
| `out/cluster-{0,1,2}.kubeconfig` | per-cluster kubeconfig (`aws eks get-token` exec auth) |
| `out/extra.values.yaml` | only when `chart_ha = false` - the cheap-mode overlay, passed as an extra `-f` |

`terraform output helm_handoff` summarizes them, plus the chart ref/version and release name to use.
`terraform output ui_url` and `terraform output orchestrator_url` print the UI and API URLs (reachable
once their DNS resolves - via the managed Route53 alias or your own DNS). Confirm cluster access before
installing:

```bash
kubectl --kubeconfig out/cluster-1.kubeconfig get nodes      # hub; Nitro nodes tainted aws-nitro-enclaves=true:NoSchedule
kubectl --kubeconfig out/cluster-0.kubeconfig get nodes      # spoke-0
kubectl --kubeconfig out/cluster-2.kubeconfig get nodes      # spoke-1
```

Once all three return healthy node lists, proceed to [Install with Helm](/deploy/helm/install).

## Cross-cluster connectivity

The example wires connectivity using AWS PrivateLink, all created by Terraform before Helm:

* The hub exposes NATS (port 4222) and relay (port 80) as PrivateLink Endpoint Services, consumed by the spokes.
* Each spoke exposes its Key Store (port 8080) and Policy Server (port 3001) as PrivateLink Endpoint Services, consumed by the hub Orchestrator.

VPC peering is a supported alternative for environments with non-overlapping VPC CIDRs (`peering-aws/`).
For multi-region or multi-provider setups, bring your own VPN or SD-WAN; the chart has no opinion on the
transport layer.