Secret Management: Managed Secrets

Overview

Each AppPlatformRegistration provisions a platform namespace (<app>-platform) with its own encryption key and Flux sync. Secrets are created and managed through the dashboard — the operator handles SOPS encryption and Git commits automatically. Flux decrypts the committed secrets and applies them directly into the target namespaces.

┌────────────────────────────────────────────────────────────┐
│  AppPlatformRegistration: my-diet                          │
│                                                            │
│  Dashboard: Create Secret                                  │
│    name: my-diet-oidc                                      │
│    targetNamespaces: [my-diet-dev, my-diet-prod]           │
│    data: { oidc-client-secret: "..." }                     │
│         │                                                  │
│         ▼                                                  │
│  K8s Secret in my-diet-platform namespace                  │
│    labels: labrats.work/managed-secret=true                │
│    annotations: target-namespaces, target-name             │
│         │                                                  │
│         ▼  Operator reconciles                             │
│  Platform repo (my-diet-platform on GitHub):               │
│    secrets/my-diet-dev/my-diet-oidc.sops.yaml              │
│    secrets/my-diet-prod/my-diet-oidc.sops.yaml             │
│         │                                                  │
│         ▼  Flux syncs + SOPS decrypts                      │
│  ┌──────────────┐  ┌──────────────┐                        │
│  │ my-diet-dev  │  │ my-diet-prod │                        │
│  │              │  │              │                         │
│  │ Secret:      │  │ Secret:      │                        │
│  │ my-diet-oidc │  │ my-diet-oidc │                        │
│  │              │  │              │                         │
│  │ Pods ref it  │  │ Pods ref it  │                        │
│  └──────────────┘  └──────────────┘                        │
└────────────────────────────────────────────────────────────┘

Why This Pattern

• No manual SOPS: You never run sops --encrypt or commit secret files — the operator does it
• Isolation: Each app's secrets are in its own platform repo with its own encryption key
• SOPS at rest: Secrets are encrypted in Git, decrypted only at apply time by Flux
• Tiered encryption: Each platform gets its own AGE key — compromising one key doesn't expose other apps' secrets
• Direct delivery: Secrets are applied directly to target namespaces — no SecretStore or ExternalSecret indirection needed
• Audit trail: Every secret create/update/delete is a Git commit in the platform repo

Per-Platform AGE Keys

When an AppPlatformRegistration is created, the controller automatically generates a unique AGE X25519 keypair for that platform. This keypair is:

1. Stored as K8s Secret <app>-platform-age-key in the operator namespace
2. SOPS-encrypted (with the cluster AGE key) and committed to the Flux repo as apps/<app>-platform/age-secret.sops.yaml
3. Referenced in .sops.yaml at the root of the <app>-platform repo

The encryption chain is:

Cluster AGE key (in flux-system)
  └─ decrypts → Platform AGE private key (per-app)
                  └─ decrypts → Secrets in <app>-platform repo

The .sops.yaml file is auto-committed to the platform repo, and the AGE key is auto-provisioned — no manual setup required.

How It Works

Creating a Secret

1. In the dashboard, go to the Secrets page for your app
2. Click Create Secret and fill in:

- Name: Secret name (lowercase, alphanumeric, hyphens) - Target Namespaces: Which environment namespaces should receive this secret - Target Name: Name of the K8s Secret in the target namespace (defaults to the secret name) - Data: Key-value pairs (e.g., oidc-client-secret: mysecretvalue)

1. The dashboard creates a K8s Secret in the <app>-platform namespace with managed-secret labels
2. On the next operator reconciliation (triggered immediately by the Secret watch):

- The operator reads the secret data - Builds a K8s Secret YAML targeting each namespace (e.g., namespace: my-diet-prod) - SOPS-encrypts the YAML using the per-app AGE key - Commits the encrypted file to the platform repo at secrets/<target-ns>/<name>.sops.yaml - Sets the labrats.work/committed=true annotation on the K8s Secret

1. Flux syncs the platform repo, decrypts the SOPS files, and applies the secrets to the target namespaces

Updating a Secret

1. Edit the secret via the dashboard — update the key-value data
2. The operator resets committed=false, re-encrypts, and commits the updated file to the platform repo
3. Flux applies the updated secret on its next sync

Deleting a Secret

1. Delete the secret via the dashboard — this marks it with labrats.work/marked-for-deletion=true
2. On the next reconciliation, the operator:

- Deletes the encrypted files from the platform repo (secrets/<target-ns>/<name>.sops.yaml) - Deletes the K8s Secret from the platform namespace

1. Flux prunes the secret from the target namespaces

Platform Repo Structure

The operator manages the secrets/ directory in the platform repo. You never edit these files directly:

<app>-platform repo (on GitHub):
├── .sops.yaml                    # auto-committed by operator (AGE key config)
├── secrets/
│   ├── my-app-dev/
│   │   ├── my-app-oidc.sops.yaml
│   │   └── my-app-smtp.sops.yaml
│   └── my-app-prod/
│       ├── my-app-oidc.sops.yaml
│       └── my-app-smtp.sops.yaml
├── my-app-dev.yaml               # Flux Kustomization (app team manages)
├── my-app-prod.yaml              # Flux Kustomization (app team manages)
└── ...

Referencing Secrets in Your App

Pods in the target namespace reference the secret directly by name:

# k8s/base/deployment.yaml
spec:
  template:
    spec:
      containers:
        - name: my-app
          env:
            - name: OIDC_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: my-app-oidc      # ← matches targetName from dashboard
                  key: oidc-client-secret

No SecretStore or ExternalSecret needed — the secret is delivered directly to the namespace by Flux.

Exception: ghcr-pull-secret stays on ClusterSecretStore since it's a shared cluster credential, not app-specific.

Automatic Infrastructure

The following are managed entirely by the operator — you don't create or maintain these:

Component	What the operator does
AGE encryption key	Generated per-app, stored in operator namespace, SOPS-encrypted in Flux repo
Flux sync	Creates a Flux Kustomization in the Flux repo that syncs the platform repo with SOPS decryption
Secret-reader RBAC	Creates Role + RoleBinding in the platform namespace granting env namespace ServiceAccounts read access
Platform repo	Created on GitHub with deploy key, .sops.yaml auto-committed

Checklist for New Apps

1. Verify the AppPlatformRegistration is Ready (check dashboard health dot or kubectl get appplatformregistration)
2. Navigate to your app's Secrets page in the dashboard
3. Create secrets with the appropriate target namespaces
4. Wait for the committed status to show true (operator has encrypted and pushed to Git)
5. Reference the secrets in your deployment manifests via secretKeyRef
6. Keep ghcr-pull-secret ExternalSecret on ClusterSecretStore for container image pulls

Flux postBuild Variables

Environment overlays need Flux postBuild.substitute for variable substitution in manifests (e.g., ${HOSTNAME}, ${NAMESPACE}). These are set on the Flux Kustomization resources in the platform repo:

# Set on the Flux Kustomization (in the <app>-platform repo)
spec:
  postBuild:
    substitute:
      HOSTNAME: dev.my-app.hcl.labrats.work
      NAMESPACE: my-app-dev
      NODE_ENV: development
      LOG_LEVEL: debug

Each environment gets its own set of substitutions defined in its Flux Kustomization YAML file.