Operator Guide

This guide is for platform administrators who install, configure, and maintain the App Platform operator. It covers CRD configuration, operator deployment, Flux integration, Authelia OIDC setup, and operational procedures.

Architecture Overview

The App Platform consists of three components:

Component	Runtime	Purpose
Operator	Go controller-runtime	Reconciles CRDs, manages GitHub repos, Flux config, OIDC
API	Express + TypeScript	In-cluster K8s API proxy, OIDC auth, CRD CRUD
Frontend	Next.js	Dashboard for teams, resources, namespaces, docs

All three run in the app-platform namespace and are deployed via a Helm chart.

Operator  -> Reconciles AppPlatformRegistration + GitHubOperatorConfig
          -> GitHub API (repo creation, deploy keys)
          -> Flux repo (GitRepository, Kustomization YAML)
          -> Authelia OIDC client registration
          -> K8s (namespaces, secrets, RBAC)

API       -> In-cluster K8s API (ServiceAccount)
          -> Authelia OIDC (JWT cookie auth)

Frontend  -> /api/* proxied to Express API

Custom Resource Definitions

AppPlatformRegistration (cluster-scoped)

Declares an application on the platform. The operator reconciles this into:

A private GitHub repo (<name>-platform)
SSH deploy key (Ed25519)
AGE encryption keypair for SOPS
Platform namespace (<name>-platform)
Flux GitRepository + Kustomization in the Flux repo
RBAC RoleBindings in app namespaces

OIDC clients are managed independently by the OidcClientReconciler (see Authelia OIDC Setup).

Full spec:

apiVersion: labrats.work/v1alpha1
kind: AppPlatformRegistration
metadata:
  name: my-app
spec:
  displayName: "My Application"      # Required: human-readable name
  hostname: my-app.hcl.labrats.work  # Optional: public hostname
  namespaces:                         # Optional: declarative app namespaces
    - my-app-dev
    - my-app-prod
  access:                             # Required: group-to-role mappings
    - group: myapp-ops
      role: admin
    - group: myapp-dev
      role: developer
    - group: myapp-viewer
      role: reader

> Note: OIDC clients are no longer declared on the APR — they are > managed via labeled Secrets handled by the OidcClientReconciler > (see Authelia OIDC Setup).

> Note: All entries in spec.namespaces must start with the app name prefix (e.g., my-app-). The platform namespace (my-app-platform) is created automatically and should not be listed.

Status fields:

Field	Description
`ready`	Whether reconciliation completed successfully
`namespace`	Same as `metadata.name` (for convenience)
`repoURL`	HTTPS URL of the managed GitHub repo
`sshURL`	SSH URL of the managed GitHub repo
`deployKeyID`	GitHub deploy key ID
`agePublicKey`	Per-platform AGE public key
`namespaces`	List of associated app namespaces
`conditions`	Standard Kubernetes conditions

GitHubOperatorConfig (cluster-scoped)

Runtime configuration for the operator. Overrides environment variables without restarting:

apiVersion: labrats.work/v1alpha1
kind: GitHubOperatorConfig
metadata:
  name: default
spec:
  fluxRepo: labrats-work/labrats.work.hetzner.cluster.flux
  ageRecipient: age1y8fcdn5u...

Field	Env Var Override	Purpose
`fluxRepo`	`FLUX_REPO`	Flux GitOps repo (`owner/name`)
`ageRecipient`	`AGE_RECIPIENT`	AGE public key for SOPS encryption

Operator Environment Variables

Variable	Required	Description
`GITHUB_TOKEN`	Yes	PAT with `admin:org`, `repo` scopes
`FLUX_REPO`	No	Flux GitOps repo in `owner/name` format
`AGE_RECIPIENT`	No	AGE public key for SOPS encryption
`DEBUG`	No	Set to `true` for verbose logging

These can be overridden at runtime by a GitHubOperatorConfig CR.

Namespace Management

Namespaces are declared in the AppPlatformRegistration CRD under spec.namespaces. The operator ensures each listed namespace exists and is properly configured.

How It Works

For each namespace in spec.namespaces, the operator:

Creates the namespace (if not exists) with labels:

- labrats.work/app: <app-name> - labrats.work/namespace-role: app - app.kubernetes.io/managed-by: app-platform

Creates RBAC RoleBindings in the namespace based on spec.access entries
Creates secret-reader RBAC — a Role + RoleBinding in the platform namespace (<app>-platform) granting ServiceAccount:secret-reader from each app namespace read access to platform secrets

Platform Namespace

The platform namespace (<app>-platform) is created automatically by the operator and holds deploy keys, AGE keys, OIDC secrets, and Flux config. It is hidden from the dashboard and should not be listed in spec.namespaces.

Namespace Labels

Label	Values	Purpose
`labrats.work/app`	`<app-name>`	Associates namespace with an app
`labrats.work/namespace-role`	`app` or `platform`	Distinguishes workload vs infrastructure
`app.kubernetes.io/managed-by`	`app-platform`	Marks operator-managed namespaces

Namespace Validation

All namespace names must start with the app name prefix (e.g., my-app-dev, my-app-prod) or exactly match the app name (my-app). The operator rejects namespaces that don't follow this convention and sets an error condition.

Secret-Reader RBAC

When app namespaces need to access secrets from the platform namespace (e.g., OIDC client secrets), the operator creates:

A Role secret-reader in <app>-platform with secrets: [get, list, watch]
A RoleBinding secret-reader with subjects = ServiceAccount:secret-reader from each app namespace

Apps use the External Secrets Operator with a SecretStore pointing at the platform namespace to pull secrets into their own namespace.

Flux Integration

The operator commits YAML files to the Flux repo for each registered app:

apps/<app>-platform/
├── kustomization.yaml      # Kustomize resource list
├── secret.sops.yaml        # SOPS-encrypted deploy key
├── age-secret.sops.yaml    # SOPS-encrypted AGE key
├── gitrepository.yaml      # Flux GitRepository CR
└── flux-kustomization.yaml # Flux Kustomization CR

These resources are labeled with:

app.kubernetes.io/managed-by: app-platform
labrats.work/app: <app-name>

Resources with the managed-by label appear as read-only in the dashboard (no Edit/Delete buttons).

Flux Dependencies

Add a dependency in clusters/main/apps.yaml so Flux processes the platform config after the operator is running:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps-<app>-platform
  namespace: flux-system
spec:
  dependsOn:
    - name: apps-app-platform
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/<app>-platform
  prune: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age

Authelia OIDC Setup {#authelia-oidc-setup}

Prerequisites

Authelia deployed with OIDC enabled
The operator's Flux repo has write access to Authelia config

How It Works

OIDC clients are registered by creating a labeled Secret in the <app>-platform namespace:

apiVersion: v1
kind: Secret
metadata:
  name: my-app-oidc
  namespace: my-app-platform
  labels:
    labrats.work/oidc-client: my-app           # client_id in Authelia
    labrats.work/app: my-app
type: Opaque
stringData:
  hostname: app.my-app.hcl.labrats.work
  callbackPath: /api/auth/oidc/callback        # optional

The OidcClientReconciler then:

Generates a random clientSecret and writes it back into the Secret
Hashes it with pbkdf2-sha512 for Authelia config
Commits the OIDC client config to Authelia's config in the Flux repo
Reports status via the labrats.work/oidc-status annotation on the Secret

Multiple OIDC clients per app are supported — one Secret per client/hostname. Cleanup is finalizer-driven: deleting the Secret strips the corresponding entry from Authelia's config.

Manual OIDC Client Hashing

If you need to manually create an OIDC client:

kubectl -n authelia exec deployment/authelia -- \
  authelia crypto hash generate pbkdf2 --password '<plaintext-secret>'

RBAC Model

The operator creates RoleBindings in each app namespace based on spec.access. It binds groups to custom app-platform-* ClusterRoles defined in the Helm chart:

Role	ClusterRole	Capabilities
`admin`	`app-platform-admin`	Full CRUD on pods, services, secrets, configmaps, deployments, statefulsets, jobs, ingresses, network policies
`developer`	`app-platform-developer`	Read workloads, secrets, services; exec into pods, port-forward
`reader`	`app-platform-reader`	Read-only access to workloads, services, and pod logs

RoleBindings are reconciled on every operator loop — added groups get new bindings, removed groups lose theirs.

Helm Chart Configuration

The operator ships as a Helm chart at charts/app-platform/. Secrets are managed via pre-existing K8s Secrets (not passed as Helm values).

Prerequisites

Create the required secrets before installing:

# Operator token (GITHUB_TOKEN)
kubectl -n app-platform create secret generic app-platform-token \
  --from-literal=token=<github-pat>

# API secrets (OIDC + JWT)
kubectl -n app-platform create secret generic app-platform-api-secret \
  --from-literal=jwt-secret=<hmac-secret> \
  --from-literal=oidc-issuer=https://my-id.hcl.labrats.work \
  --from-literal=oidc-client-id=app-platform \
  --from-literal=oidc-client-secret=<oidc-secret> \
  --from-literal=oidc-redirect-uri=https://app-platform.hcl.labrats.work/api/auth/oidc/callback

Key Values

Value	Default	Description
`image.tag`	`""` (uses appVersion)	Operator image tag
`api.image.tag`	`""` (uses appVersion)	API image tag
`app.image.tag`	`""` (uses appVersion)	Frontend image tag
`existingSecret`	`app-platform-token`	K8s Secret with `token` key for GITHUB_TOKEN
`api.existingSecret`	`app-platform-api-secret`	K8s Secret with OIDC + JWT keys
`api.platformAdminGroup`	`""`	OIDC group for platform admin access
`config.fluxRepo`	`""`	Flux GitOps repo (`owner/name`)
`config.ageRecipient`	`""`	AGE public key for SOPS encryption
`ingress.hostname`	`app-platform.hcl.labrats.work`	Dashboard hostname

Operational Procedures

Registering a New App

Create Authelia groups (e.g., myapp-ops, myapp-dev)
Apply the AppPlatformRegistration CR
Add the Flux Kustomization entry in clusters/main/apps.yaml
Verify: kubectl get appplatformregistration <name> shows Ready

Deregistering an App

Deleting the AppPlatformRegistration triggers the finalizer, which runs best-effort cleanup: removes the GitHub platform repo, deploy key, GHCR packages, RBAC, and the platform namespace. Deleting the platform namespace cascades to all labeled OIDC Secrets, whose own finalizer strips the corresponding entries from Authelia's config.

Rotating a Deploy Key

Delete the deploy key Secret in the platform namespace. The operator will regenerate it on the next reconciliation:

kubectl -n <app>-platform delete secret <app>-platform-deploy-key

Checking Reconciliation Status

# Overall status
kubectl get appplatformregistrations

# Detailed conditions
kubectl get appplatformregistration <name> -o jsonpath='{.status.conditions}' | jq

# Operator logs
kubectl -n app-platform logs -l app.kubernetes.io/name=app-platform --tail=100

Troubleshooting

AppPlatformRegistration stuck at "not Ready"

Check conditions:

kubectl get appplatformregistration <name> -o yaml

Common causes:

Condition Reason	Cause	Fix
`RepoCreationFailed`	GITHUB_TOKEN lacks `repo` scope	Update the PAT
`DeployKeyFailed`	SSH key generation failed	Check operator logs
`EncryptionKeyFailed`	AGE key generation failed	Check AGE_RECIPIENT config

> Note: OIDC client registration failures show up on the labeled > Secret's labrats.work/oidc-status annotation, not on the APR.

> Note: Flux config commit failures are logged as non-fatal warnings and do not set a condition. Check operator logs if Flux resources are missing.

Flux not syncing platform repo

# Check GitRepository source
kubectl get gitrepository -n <app>-platform

# Check Kustomization
kubectl get kustomization -n <app>-platform

# Check Flux source controller logs
kubectl -n flux-system logs deploy/source-controller --tail=50

Dashboard shows "Unauthorized"

Verify the API's OIDC config matches Authelia
Check the JWT_SECRET is consistent across API restarts (stored in app-platform-api-secret)
Verify the user's Authelia groups match spec.access entries

Operator CrashLoopBackOff

# Check resource limits
kubectl -n app-platform describe pod -l app.kubernetes.io/name=app-platform

# Check logs for panic/OOM
kubectl -n app-platform logs -l app.kubernetes.io/name=app-platform --previous --tail=100

Common causes: missing GITHUB_TOKEN (check app-platform-token secret), invalid Flux repo reference, insufficient RBAC on the ServiceAccount.