Skip to content

Secret Management

Architecture

Secrets are stored in Akeyless — a zero-knowledge secrets management platform. The critical distinction is the customer fragment: a secret shard stored on-premise in a CipherTrust Manager VM. Akeyless SaaS holds only an encrypted fragment; without the customer shard it cannot reconstruct any secret.

flowchart LR
    dev["Operator"]
    ak["Akeyless SaaS\n(encrypted fragment only)"]
    cm["CipherTrust Manager\ncm.home.tillo.ch\ncustomer fragment"]
    eso["External Secrets Operator\nClusterSecretStore: cm-akeyless"]
    sec["K8s Secret"]
    pod["Pod"]

    dev -->|"store secret"| ak
    ak <-->|"customer fragment\nnever transmitted\nover internet"| cm
    eso -->|"fetch at runtime\n/mdapi/* paths"| cm
    eso --> sec --> pod

The gateway is embedded in the CipherTrust Manager VM and exposed at cm.home.tillo.ch. The External Secrets Operator (ESO) talks to the v2 API endpoint at /akeyless-api/v2.

Secret Lifecycle

  1. Write — operator creates/updates a secret via Akeyless CLI or UI; it is stored encrypted, split between Akeyless SaaS and the on-premise fragment.
  2. Fetch — ESO reads the ExternalSecret CR, authenticates to the gateway, and reconstructs the plaintext. The plaintext exists only in memory and in the resulting K8s Secret.
  3. Consume — pods reference the K8s Secret as environment variables or volume mounts; they never talk to Akeyless directly.
  4. Rotate — the secret is updated in Akeyless; ESO refreshes the K8s Secret on its next poll interval (default: 1 minute).
  5. Revoke — deleting the ExternalSecret CR causes the K8s Secret to be deleted on the next reconcile.

ExternalSecret Pattern

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-secret
  namespace: my-namespace
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: cm-akeyless
    kind: ClusterSecretStore
  target:
    name: my-secret
  data:
    - secretKey: MY_VAR
      remoteRef:
        key: /mdapi/my-service/my-secret/MY_VAR

Template field order

When templating a secret from multiple Akeyless keys, NAME (direct value) must come before SELECTOR (path reference) in template.data. Wrong order silently produces empty values — no error is shown.

Failure Modes

Symptom Cause Resolution
ExternalSecrets stuck in SecretSyncedError CipherTrust Manager VM unreachable Check VM health; restore from Longhorn snapshot if needed
/api/derived-key 404 CM upgrade broke the fragment endpoint Restore CM VM from last known-good Longhorn backup
500 on derived-key Fragment service crashed Same as above

The CipherTrust Manager VM has a Longhorn-backed persistent disk and is snapshotted on the weekly schedule. Recovery typically takes under 5 minutes.