# APIX Registry — Infrastructure Setup Guide

Complete walkthrough for deploying the APIX registry to a Hetzner VPS with Bunny.net CDN,
Prometheus/Grafana observability, and live Loki telemetry.

---

## Table of Contents

1. [Architecture Overview](#1-architecture-overview)
2. [Prerequisites](#2-prerequisites)
3. [VPS Provisioning (Hetzner)](#3-vps-provisioning-hetzner)
4. [DNS Configuration](#4-dns-configuration)
5. [Server Bootstrap](#5-server-bootstrap)
6. [Application Build](#6-application-build)
7. [Environment Configuration](#7-environment-configuration)
8. [Deploy the Stack](#8-deploy-the-stack)
9. [Caddy TLS Reverse Proxy](#9-caddy-tls-reverse-proxy)
10. [Bunny.net CDN Setup](#10-bunnynet-cdn-setup)
11. [Live Telemetry: Promtail → Loki](#11-live-telemetry-promtail--loki)
12. [Grafana Dashboards](#12-grafana-dashboards)
13. [Weekly Analytics (Bunny.net Logs)](#13-weekly-analytics-bunnynet-logs)
14. [Verification Checklist](#14-verification-checklist)
15. [Routine Operations](#15-routine-operations)

---

## 1. Architecture Overview

```
Internet
    │
    ▼
Bunny.net CDN (100+ PoPs, GDPR-compliant, European)
    │  Cache-Control headers respected; query-string vary cache enabled
    │  HIT: served from edge (~5ms).  MISS: forwarded to VPS
    │
    ▼
Hetzner VPS (Helsinki / Falkenstein)
    │
    ├── Caddy (80/443) ─── TLS termination, HTTPS redirect, rate limiting
    │       │
    │       ├── registry:8180  — REST API  (Quarkus, JVM)
    │       ├── portal:8081    — Web UI    (Quarkus, Qute templates)
    │       └── grafana:3000   — Dashboards (internal access only)
    │
    ├── db:5432          — PostgreSQL 16 (no public port)
    ├── spider:8082      — Liveness checker (no public port)
    ├── prometheus:9090  — Metrics scraper (no public port)
    └── promtail:9080    — Syslog receiver → Loki (port 5514 open)

Grafana Cloud (Loki)
    │  Real-time: Bunny.net → TCP Syslog → Promtail → Loki → Grafana
    └── Every CDN request visible within 1-2 seconds
```

**CDN governance constraint:** Cloudflare and AWS CloudFront must never be used.
Both are founding member candidates — operating infrastructure gives governance leverage
regardless of the founding charter. Approved providers: Bunny.net (primary), Fastly (fallback).

---

## 2. Prerequisites

### Local machine
| Tool | Version | Purpose |
|------|---------|---------|
| Java (Temurin) | 21 | Building application JARs |
| Maven | 3.9+ | Build system |
| Docker Desktop | latest | Local dev + image building |
| `curl` | any | Script calls to Bunny.net API |
| `python3` | 3.8+ | JSON parsing in setup scripts |

Install Java 21 via SDKMAN:
```bash
curl -s https://get.sdkman.io | bash
sdk install java 21.0.3-tem
```

### Accounts required
| Service | What you need |
|---------|---------------|
| Hetzner Cloud | API token for VPS creation (optional — can provision manually) |
| Bunny.net | Account + API key (`Account → API`) |
| Grafana Cloud | Free tier sufficient; Loki + Prometheus endpoints |
| Domain registrar | Control over DNS for `api-index.org` |

---

## 3. VPS Provisioning (Hetzner)

### Recommended spec
```
Type:     CPX21  (3 vCPU, 4 GB RAM)  — sufficient for MVP
Location: Helsinki (hel1) or Falkenstein (fsn1)
OS:       Ubuntu 24.04 LTS
Network:  Primary IPv4 + IPv6 dual stack
Backups:  Enable automatic backups (adds 20% to monthly cost)
```

The CPX21 comfortably runs the full Docker stack (registry + spider + portal + db +
prometheus + grafana + caddy + promtail) under MVP load. Upgrade to CPX31 if Prometheus
retention or portal traffic grows.

### Firewall rules (Hetzner firewall or `ufw`)

| Port | Protocol | Source | Purpose |
|------|----------|--------|---------|
| 22 | TCP | Your IP only | SSH |
| 80 | TCP | any | Caddy HTTP→HTTPS redirect |
| 443 | TCP+UDP | any | Caddy HTTPS + HTTP/3 |
| 5514 | TCP | Bunny.net IPs | Promtail syslog receiver |
| 9090 | TCP | VPS localhost | Prometheus (internal only) |
| 3000 | TCP | VPS localhost | Grafana (access via Caddy or SSH tunnel) |

**Bunny.net syslog source IPs:** Bunny.net does not publish a static IP list; open 5514
to `0.0.0.0/0` and rely on the Promtail pipeline to discard unexpected traffic.
The syslog format is the only authentication layer needed at this volume.

### SSH hardening (run as root after first login)
```bash
# Create deploy user
useradd -m -s /bin/bash deploy
usermod -aG sudo,docker deploy

# Copy your SSH public key
mkdir -p /home/deploy/.ssh
echo "YOUR_PUBLIC_KEY_HERE" > /home/deploy/.ssh/authorized_keys
chown -R deploy:deploy /home/deploy/.ssh
chmod 700 /home/deploy/.ssh
chmod 600 /home/deploy/.ssh/authorized_keys

# Disable root SSH + password auth
sed -i 's/^PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sed -i 's/^PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart sshd
```

---

## 4. DNS Configuration

### Required records

| Name | Type | Value | TTL |
|------|------|-------|-----|
| `api-index.org` | A | VPS IPv4 | 300 |
| `api-index.org` | AAAA | VPS IPv6 | 300 |
| `www.api-index.org` | CNAME | `api-index.org` | 3600 |

Set TTL to 300 (5 min) before the cutover so propagation is fast.
After CDN is live (Step 10), change the A/AAAA records to the Bunny.net CNAME instead.

### After CDN setup (replace A records)
```
api-index.org  CNAME  <bunnynet-cname>.b-cdn.net
```

Caddy still handles TLS on the VPS. Bunny.net terminates the edge TLS and forwards
to the VPS over HTTPS using the Caddy certificate.

---

## 5. Server Bootstrap

SSH in as `deploy` and run:

```bash
# 1. System update
sudo apt-get update && sudo apt-get upgrade -y

# 2. Docker (official repo — apt package is outdated)
curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

echo "deb [arch=$(dpkg --print-architecture) \
  signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] \
  https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

# 3. Add deploy user to docker group (logout + login to apply)
sudo usermod -aG docker deploy

# 4. Install Promtail (for Loki integration — see Step 11)
PROMTAIL_VERSION="3.0.0"
wget -q "https://github.com/grafana/loki/releases/download/v${PROMTAIL_VERSION}/promtail-linux-amd64.zip"
unzip -q promtail-linux-amd64.zip
sudo mv promtail-linux-amd64 /usr/local/bin/promtail
sudo chmod +x /usr/local/bin/promtail
rm promtail-linux-amd64.zip

# 5. Clone the repository
git clone https://gitea.your-server.example/botstandards/apix-mvp.git /opt/apix
# Or using GitHub mirror during MVP phase:
git clone https://github.com/your-org/apix-mvp.git /opt/apix

cd /opt/apix
```

---

## 6. Application Build

Build JARs locally and copy to the VPS, or build directly on the VPS if Java 21 is installed.

### Build locally (recommended for CI phase)
```bash
# On your dev machine:
cd bot-service-index/apix-mvp
mvn clean package -DskipTests

# Copy artifacts to VPS
scp apix-registry/target/quarkus-app/   deploy@<vps-ip>:/opt/apix/apix-registry/target/quarkus-app/  -r
scp apix-spider/target/quarkus-app/     deploy@<vps-ip>:/opt/apix/apix-spider/target/quarkus-app/    -r
scp apix-portal/target/quarkus-app/     deploy@<vps-ip>:/opt/apix/apix-portal/target/quarkus-app/    -r
```

### Build on VPS (MVP shortcut)
```bash
# Install Java 21 on VPS
sudo apt-get install -y wget
wget -q "https://github.com/adoptium/temurin21-binaries/releases/download/jdk-21.0.3%2B9/OpenJDK21U-jdk_x64_linux_hotspot_21.0.3_9.tar.gz" \
  -O /tmp/jdk21.tar.gz
sudo mkdir -p /opt/java
sudo tar xzf /tmp/jdk21.tar.gz -C /opt/java
sudo ln -sf /opt/java/jdk-21.0.3+9/bin/java /usr/local/bin/java
sudo ln -sf /opt/java/jdk-21.0.3+9/bin/javac /usr/local/bin/javac

# Install Maven
sudo apt-get install -y maven

# Build
cd /opt/apix
mvn clean package -DskipTests
```

### Docker images

Dockerfiles are defined in WORKLOG Block 5 (I-04 to I-06) and not yet created.
Until they exist, run the JARs directly via `quarkus:dev` or write minimal Dockerfiles:

```dockerfile
# infra/Dockerfile.registry  (placeholder until Block 5)
FROM eclipse-temurin:21-jre-alpine
WORKDIR /app
COPY apix-registry/target/quarkus-app/ quarkus-app/
EXPOSE 8180
ENTRYPOINT ["java", "-jar", "quarkus-app/quarkus-run.jar"]
```

Repeat for `Dockerfile.spider` (port 8082) and `Dockerfile.portal` (port 8081).

Build and tag:
```bash
cd /opt/apix
docker build -f infra/Dockerfile.registry -t apix-registry:latest .
docker build -f infra/Dockerfile.spider   -t apix-spider:latest   .
docker build -f infra/Dockerfile.portal   -t apix-portal:latest   .
```

---

## 7. Environment Configuration

Create `/opt/apix/.env` from the template below. This file is in `.gitignore` — never commit it.

```bash
# /opt/apix/.env — production values

# ── Database ──────────────────────────────────────────────────────────────────
APIX_DB_USER=apix
APIX_DB_PASSWORD=<generate: openssl rand -base64 32>
APIX_DB_NAME=apix
APIX_DB_PORT=5432    # Only exposed inside Docker network in production

# ── API security ─────────────────────────────────────────────────────────────
# Used to authenticate write requests (POST /services, PATCH /services/*, etc.)
# Rotate this key when onboarding new registrars.
APIX_API_KEY=<generate: openssl rand -hex 32>

# ── Registry identity ─────────────────────────────────────────────────────────
APIX_REGISTRY_BASE_URL=https://api-index.org
APIX_REGISTRY_NAME=APIX Registry
APIX_REGISTRY_DESCRIPTION=The open autonomous agent service discovery registry.

# ── Verification integrations ────────────────────────────────────────────────
GLEIF_API_URL=https://api.gleif.org/api/v1
OPENCORPORATES_API_KEY=<your OpenCorporates API key — leave blank if not yet contracted>
APIX_VERIFICATION_TIMEOUT_MS=5000

# ── Mail signing (Ed25519) ────────────────────────────────────────────────────
# Leave blank on first deploy — ephemeral key generated at startup.
# Set before production: openssl genpkey -algorithm ed25519 | ...
APIX_MAIL_SIGNING_PRIVATE_KEY=
APIX_MAIL_SIGNING_PUBLIC_KEY=
APIX_MAIL_SIGNING_KID=2026-05

# ── Spider ────────────────────────────────────────────────────────────────────
SPIDER_INTERVAL_MINUTES=15

# ── Grafana ───────────────────────────────────────────────────────────────────
GRAFANA_ADMIN_PASSWORD=<generate: openssl rand -base64 16>
GRAFANA_ROOT_URL=https://grafana.api-index.org   # or http://localhost:3000 if SSH tunnel only

# ── Logging ───────────────────────────────────────────────────────────────────
LOG_LEVEL=INFO
```

Generate secrets in one pass:
```bash
echo "APIX_DB_PASSWORD=$(openssl rand -base64 32)"
echo "APIX_API_KEY=$(openssl rand -hex 32)"
echo "GRAFANA_ADMIN_PASSWORD=$(openssl rand -base64 16)"
```

---

## 8. Deploy the Stack

```bash
cd /opt/apix/infra

# Start everything
docker compose --env-file ../.env up -d

# Watch startup logs
docker compose logs -f --tail=50

# Verify all services are healthy
docker compose ps
```

Expected healthy services after ~60 seconds:

| Service | Health endpoint | Expected |
|---------|----------------|---------|
| `db` | `pg_isready` | healthy |
| `registry` | `http://localhost:8180/q/health/live` | `{"status":"UP"}` |
| `spider` | `http://localhost:8082/q/health/live` | `{"status":"UP"}` |
| `portal` | `http://localhost:8081/q/health/live` | `{"status":"UP"}` |
| `prometheus` | `http://localhost:9090/-/healthy` | `Prometheus Server is Healthy.` |
| `grafana` | `http://localhost:3000/api/health` | `{"database":"ok"}` |

Quick smoke test (from VPS):
```bash
# Registry root (HATEOAS navigation)
curl -s http://localhost:8180/ | python3 -m json.tool

# Metrics endpoint (Prometheus scrape target)
curl -s http://localhost:8180/q/metrics | grep apix_search

# Search endpoint
curl -s "http://localhost:8180/services?capability=nlp" | python3 -m json.tool
```

### Liquibase note

Liquibase runs automatically at startup (`quarkus.liquibase.migrate-at-start=true`).
If the changelog is missing (`db/changelog/db.changelog-master.xml`), the registry will
fail to start. Check logs with `docker compose logs registry` and ensure migrations
are present (WORKLOG Block 1 / C-20 to C-24).

---

## 9. Caddy TLS Reverse Proxy

Create `infra/Caddyfile`:

```caddy
# infra/Caddyfile

api-index.org {
    # Public API — registry
    handle /services*         { reverse_proxy registry:8180 }
    handle /devices*          { reverse_proxy registry:8180 }
    handle /organizations*    { reverse_proxy registry:8180 }
    handle /mail-signing-keys { reverse_proxy registry:8180 }
    handle /                  { reverse_proxy registry:8180 }

    # Caddy does not forward /q/* to CDN — Quarkus internals only
    handle /q/*               { reverse_proxy registry:8180 }

    # Rate limiting (requires caddy-ratelimit plugin or enterprise)
    # Basic protection: Caddy's built-in connection limit
    header {
        Strict-Transport-Security "max-age=31536000; includeSubDomains"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"
    }

    log {
        output file /var/log/caddy/api-index.log
        format json
    }
}

# Portal — separate subdomain (optional)
portal.api-index.org {
    reverse_proxy portal:8081
    header Strict-Transport-Security "max-age=31536000; includeSubDomains"
}

# Grafana — restrict to internal access or require basic auth
grafana.api-index.org {
    basicauth {
        # htpasswd -nb admin <password>  — generate and paste hash here
        admin $2a$14$REPLACE_WITH_BCRYPT_HASH
    }
    reverse_proxy grafana:3000
}
```

Caddy fetches TLS certificates from Let's Encrypt automatically on first request.
No manual certificate management needed.

Rebuild the `caddy` container to pick up the new Caddyfile:
```bash
cd /opt/apix/infra
docker compose restart caddy
docker compose logs caddy -f
```

Verify TLS:
```bash
curl -sv https://api-index.org/ 2>&1 | grep -E "SSL|certificate|subject"
```

---

## 10. Bunny.net CDN Setup

The CDN sits in front of Caddy/registry and handles ~95% of read traffic from cache.

### One-time setup
```bash
cd /opt/apix

# With Loki log forwarding (strongly recommended for observability):
BUNNYNET_API_KEY=your-key \
ORIGIN_URL=https://api-index.org \
CUSTOM_HOSTNAME=api-index.org \
SYSLOG_HOST=<vps-public-ip> \
SYSLOG_PORT=5514 \
./scripts/setup-bunnynet.sh
```

The script:
1. Creates a pull zone pointing at `https://api-index.org`
2. Enables query-string vary cache (so `?capability=nlp` and `?capability=translation`
   are cached as separate entries — critical for correct cache behavior)
3. Sets edge TTL to follow origin `Cache-Control` headers (registry sets `max-age=30`
   on `/services` and `/devices`; `max-age=60` on `/`)
4. Adds the `api-index.org` custom hostname
5. Adds an edge rule to bypass cache for `/q/*` (Quarkus health/metrics endpoints)
6. Enables real-time syslog forwarding to Promtail (when `SYSLOG_HOST` is set)
7. Prints the CNAME value for your DNS record

### Update DNS to point to CDN edge
After the script prints the CDN hostname:
```
api-index.org  CNAME  apix-registry.b-cdn.net
```

Remove the A/AAAA records pointing directly to the VPS. The VPS is now origin-only.

### Verify CDN is caching

```bash
# First request — cache MISS (origin hit)
curl -sI https://api-index.org/services?capability=nlp | grep -i "cache\|x-cache\|age"

# Second request within 30s — cache HIT
curl -sI https://api-index.org/services?capability=nlp | grep -i "cache\|x-cache\|age"

# From an Asian machine or VPN endpoint (tests geographic edge)
curl -w "Total: %{time_total}s\n" -o /dev/null -s https://api-index.org/services?capability=nlp
# Target: <20ms after warm-up
```

---

## 11. Live Telemetry: Promtail → Loki

Provides real-time telemetry of all CDN traffic (hits + misses) during demos.
Origin Prometheus only sees cache misses — Loki sees everything.

### Grafana Cloud Loki credentials

In Grafana Cloud (`grafana.com`):
1. Go to your stack → Loki → Details
2. Note the **Push URL** (e.g. `https://logs-prod-eu-west-0.grafana.net/loki/api/v1/push`)
3. Create a service account with `logs:write` scope → copy the token

### Configure Promtail

```bash
# Copy config to system path
sudo cp /opt/apix/scripts/promtail-cdn-logs.yaml /etc/promtail/cdn-logs.yaml

# Fill in credentials
sudo sed -i 's|https://LOKI_PUSH_URL/loki/api/v1/push|https://logs-prod-eu-west-0.grafana.net/loki/api/v1/push|g' \
  /etc/promtail/cdn-logs.yaml
sudo sed -i 's|"LOKI_USERNAME"|"123456"|g'           /etc/promtail/cdn-logs.yaml  # stack user ID
sudo sed -i 's|"LOKI_PASSWORD"|"your-token-here"|g'  /etc/promtail/cdn-logs.yaml
```

Or edit directly: `sudo nano /etc/promtail/cdn-logs.yaml` — replace the three
`LOKI_PUSH_URL`, `LOKI_USERNAME`, `LOKI_PASSWORD` placeholders.

### Create systemd service

```bash
sudo tee /etc/systemd/system/promtail.service > /dev/null <<'EOF'
[Unit]
Description=Promtail — Bunny.net CDN log forwarder
After=network.target

[Service]
User=nobody
Group=nogroup
ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/cdn-logs.yaml
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now promtail
sudo systemctl status promtail
```

### Verify the pipeline end-to-end

```bash
# 1. Make a test request through the CDN
curl -s "https://api-index.org/services?capability=nlp" > /dev/null

# 2. Check Promtail received it (should appear within 1-2 seconds)
sudo journalctl -u promtail -f --no-pager

# 3. In Grafana Explore (Loki datasource):
#    {job="apix-cdn"} — should show a log line from the test request
```

---

## 12. Grafana Dashboards

### Add Loki datasource to Grafana

The stack's provisioned Prometheus datasource is auto-loaded. Add Loki manually or via provisioning:

```yaml
# infra/grafana/provisioning/datasources/loki.yml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    access: proxy
    url: https://logs-prod-eu-west-0.grafana.net
    basicAuth: true
    basicAuthUser: "123456"    # Grafana Cloud stack user ID
    secureJsonData:
      basicAuthPassword: "your-loki-token"
    isDefault: false
    editable: false
```

Restart Grafana to apply:
```bash
docker compose restart grafana
```

### Import the OpenClaw demo dashboard

```bash
# Copy the dashboard JSON to the provisioning directory
cp /opt/apix/scripts/grafana-demo-dashboard.json \
   /opt/apix/infra/grafana/provisioning/dashboards/demo-openclaw.json
```

Grafana auto-discovers dashboards in the provisioning path (30s poll interval per `provider.yml`).
No manual import needed.

Alternatively, import via UI:
1. Grafana → Dashboards → Import
2. Upload `scripts/grafana-demo-dashboard.json`
3. Select the Loki and Prometheus datasources when prompted

### Dashboard refresh for demo sessions

The demo dashboard is pre-configured with:
- **Refresh:** 5 seconds
- **Time range:** Last 15 minutes
- **Auto-play:** Enable via Grafana's kiosk mode for the demo screen

Kiosk mode URL (hides nav bar):
```
https://grafana.api-index.org/d/apix-demo-openclaw/apix-registry-demo?kiosk&refresh=5s
```

---

## 13. Weekly Analytics (Bunny.net Logs)

Bunny.net stores gzip access logs (one file per day). The `query-report.sh` script
downloads them, parses them, and produces a capability frequency report.

```bash
# Basic report — last 7 days
BUNNYNET_API_KEY=your-key \
PULL_ZONE_ID=$(cat /opt/apix/.bunnynet-pull-zone-id) \
./scripts/query-report.sh
```

With Prometheus Pushgateway (builds a weekly time-series in Grafana):
```bash
BUNNYNET_API_KEY=your-key \
PULL_ZONE_ID=$(cat /opt/apix/.bunnynet-pull-zone-id) \
PROMETHEUS_PUSH_URL=https://pushgateway.your-grafana.example/metrics/job/apix-cdn-report \
DAYS=7 \
./scripts/query-report.sh
```

### Weekly cron job (on VPS)

Pass `INSTALL_CRON=true` when running `setup-bunnynet.sh` and the cron entry is
installed automatically — no manual `crontab -e` needed:

```bash
BUNNYNET_API_KEY=your-key \
ORIGIN_URL=https://api-index.org \
INSTALL_CRON=true \
./scripts/setup-bunnynet.sh
```

The script installs a `crontab` entry for the current user (Mondays 06:00) and
deduplicates on re-run — safe to call again after rotating the API key.
Verify with: `crontab -l | grep query-report`

The report answers:
- Which capabilities are queried most (all requests, including CDN hits)
- Cache hit ratio per endpoint
- Geographic distribution (PoP breakdown)
- Top query string combinations

---

## 14. Verification Checklist

Run through this after each deployment.

### Registry
```bash
# HATEOAS root
curl -s https://api-index.org/ | python3 -m json.tool

# Search (returns empty array if no services registered yet — correct)
curl -s "https://api-index.org/services?capability=nlp"

# Health
curl -s https://api-index.org/q/health | python3 -m json.tool

# Cache-Control header on search endpoint
curl -sI "https://api-index.org/services?capability=nlp" | grep -i cache-control
# Expected: Cache-Control: public, max-age=30
```

### CDN
```bash
# Second request should be a cache HIT
curl -sI "https://api-index.org/services?capability=nlp" | grep -i cache
# Expected: X-Cache: HIT  (or similar from Bunny.net)

# Edge latency test — run from a machine outside Germany
curl -w "DNS: %{time_namelookup}s  Connect: %{time_connect}s  Total: %{time_total}s\n" \
     -o /dev/null -s "https://api-index.org/services?capability=nlp"
# Target: Total <20ms from Asia/US after warm-up
```

### Observability
```bash
# Prometheus scraping registry
curl -s http://localhost:9090/api/v1/targets | python3 -c \
  "import sys,json; [print(t['labels']['job'], t['health']) for t in json.load(sys.stdin)['data']['activeTargets']]"

# Loki receiving CDN logs
# In Grafana Explore: {job="apix-cdn"} | last 5 min should show entries
```

### TLS
```bash
curl -sv https://api-index.org/ 2>&1 | grep -E "issuer|subject|expire"
# Should show Let's Encrypt issuer
```

---

## 15. Routine Operations

### Update application
```bash
cd /opt/apix

# Pull latest code
git pull origin main

# Rebuild affected images
docker build -f infra/Dockerfile.registry -t apix-registry:latest .

# Rolling restart
docker compose up -d registry
docker compose logs registry -f --tail=50
```

### Rotate API key
```bash
NEW_KEY=$(openssl rand -hex 32)
# Update .env
sed -i "s/^APIX_API_KEY=.*/APIX_API_KEY=${NEW_KEY}/" /opt/apix/.env
# Restart registry to pick up new key
docker compose up -d registry
echo "New key: ${NEW_KEY}"
# Distribute to all registrar clients before the old key expires
```

### Database backup
```bash
# Manual backup
docker exec apix-infra-db-1 pg_dump -U apix apix | gzip > /opt/apix/backups/apix-$(date +%Y%m%d).sql.gz

# Automated daily backup via cron
0 2 * * * docker exec apix-infra-db-1 pg_dump -U apix apix | gzip \
          > /opt/apix/backups/apix-$(date +\%Y\%m\%d).sql.gz

# Restore
gunzip -c /opt/apix/backups/apix-20260101.sql.gz | docker exec -i apix-infra-db-1 psql -U apix apix
```

### View logs
```bash
# Live registry logs
docker compose -f /opt/apix/infra/docker-compose.yml logs registry -f

# Use the convenience scripts (from project root)
./scripts/logs.sh registry
./scripts/logs.sh spider
./scripts/logs.sh portal

# Promtail (CDN log forwarder)
sudo journalctl -u promtail -f
```

### Purge CDN cache (after deploying schema changes)
```bash
curl -sf -X POST "https://api.bunny.net/pullzone/$(cat /opt/apix/.bunnynet-pull-zone-id)/purgeCache" \
  -H "AccessKey: ${BUNNYNET_API_KEY}"
```

### Stop / restart the full stack
```bash
cd /opt/apix/infra
./scripts/stop.sh     # graceful stop
./scripts/restart.sh  # stop + start
./scripts/reset.sh    # WARNING: drops all volumes including DB data
```

---

## Appendix: Environment Variable Reference

All variables accepted by the `registry` container, sourced from `.env`:

| Variable | Default | Required | Description |
|----------|---------|----------|-------------|
| `QUARKUS_DATASOURCE_JDBC_URL` | `jdbc:postgresql://db:5432/apix` | yes | Database JDBC URL |
| `QUARKUS_DATASOURCE_USERNAME` | `apix` | yes | DB username |
| `QUARKUS_DATASOURCE_PASSWORD` | `apix` | yes | DB password — use a strong value |
| `APIX_API_KEY` | `dev-insecure-key-change-in-prod` | yes | Write-endpoint auth key |
| `APIX_REGISTRY_BASE_URL` | `http://localhost:8180` | yes | Used in HATEOAS links |
| `GLEIF_API_URL` | `https://api.gleif.org/api/v1` | no | O2 verification: GLEIF REST API |
| `OPENCORPORATES_API_KEY` | _(blank)_ | no | O2 verification: OpenCorporates |
| `APIX_VERIFICATION_TIMEOUT_MS` | `5000` | no | HTTP timeout for verification calls |
| `APIX_MAIL_SIGNING_PRIVATE_KEY` | _(blank)_ | no | Ed25519 private key, Base64; ephemeral if blank |
| `APIX_MAIL_SIGNING_PUBLIC_KEY` | _(blank)_ | no | Ed25519 public key, Base64 |
| `APIX_MAIL_SIGNING_KID` | `dev` | no | Key ID in signed payloads; rotate every 6 months |
| `SANCTIONS_CACHE_PATH` | `./sanctions-cache` | no | Local path for sanctions list cache |
| `LOG_LEVEL` | `INFO` | no | `DEBUG` / `INFO` / `WARNING` / `ERROR` |