Skip to main content

Operations: Backup, DR, Observability, Image Updates

Companion to: architecture.md, deployment.md

1. Backup Strategy​

Backup is per tenant. A TenantExport captures one tenant into one encrypted bundle in object storage: each app's databases (PostgreSQL and MariaDB dumps), buckets and volume claims, the tenant's realm, its mailboxes where the cluster runs its own mail server, and the rights that follow from nothing else. A BackupPolicy says where bundles go and which keys they are encrypted to. The platform takes a backup when it is asked for one; taking them on a schedule and expiring old ones is an add-on's, which creates the TenantExports. What a bundle carries per kind is in data-lifecycle.md; the reference is §9.

The vault, and the transit seal that unseals it, are the kernel stores the platform snapshots: on Raft storage, every day, encrypted to the backup recipient, into the bucket gentian-openbao-snapshots of the platform's object store, keeping the newest 14 of each. The operator writes how old each newest is on the Cluster claim's status (status.openbaoSnapshots, status.transitSnapshots) and as metrics (§6). An instance on file storage is reported and not snapshotted. Restoring the snapshots and moving the two to Raft are recovery.md §4 and §5.

Nothing else is backed up by the platform. There is no continuous archiving of a database (no WAL archive, no point-in-time recovery), no replication of the object store and no Velero; a cluster that wants any of these adds it itself. The rest of the kernel's own state is rebuilt, not restored (§4).

2. Tenant-Scoped Restore​

Each tenant has its own databases, buckets, realm and namespace, so one tenant is restored without touching another. A TenantRestore in the tenant's namespace does it from a bundle; scripts/recovery.sh and recovery.md are the procedure. Export can be started from the administration console; a restore is a cluster administrator's act. Restore is data-only — the tenant's shape is re-composed from its claim, never restored — and quiesces one app at a time, since the consistency boundary that matters is an app's database plus its bucket plus its PVC, not the tenant as a whole. What a bundle carries per kind and what no restore brings back are in data-lifecycle.md; the rule a restore decides by is in §9.4.

3. Tenant Migration Between Clusters​

Export on the source, import on the target, update DNS: kubectl gentian tenants import declares the tenant from the bundle's manifest, waits for the operator to provision it, and restores the data into it (commands.md §12a). Credentials are not carried; the target provisions its own.

3.1 Node Flavor Migration​

Replacing a cluster's worker nodes with a different flavor — sizing from usable rather than purchased memory, the drain order the kernel singletons require, and the CNPG and RWO cases that need manual steps — is covered in node-pool-migration.md.

4. Disaster Recovery​

For full-cluster DR (recovery.md §1):

  1. ./install.sh --recover <kit> on a fresh cluster, with the same deployments repository and cluster id. The recovery kit carries what git cannot hold: the master password and salt, the unseal material, the repository and registry credentials, the backup key.
  2. The install runs as normal: Argo CD, Crossplane and the Cluster claim bring the kernel back from git.
  3. Each tenant is imported from its newest bundle (§3).

With secretMode: derived (the default) the kernel's service credentials and each app's are derived from the master password and salt, so they come back with their original values; with secretMode: random they are generated anew (scripts/lib/bootstrap.sh, internal/kernel/secrets). An app's own secrets come back in both modes with the tenant's bundle, which carries them so that the data it brings back stays readable (security/design.md §6.4). People's and administrators' passwords are never derived: a realm export carries none, and every member resets theirs after a restore.

5. Observability via the K8s API​

Crossplane's MR status model gives uniform observability:

# Tenant health at a glance
kubectl get tenants
NAME   STATUS   APPS   READY   ADMIN               AGE
demo   Ready    2      2       admin@example.com   30d

# What is installed for a tenant (Component → helm Release, cluster-scoped)
kubectl get components -n tenant-demo
kubectl get releases.helm.crossplane.io -l gentianos.io/tenant=demo

# The tenant's composite (namespace, quota, policy)
kubectl get xtenant demo

# Integration contract health
kubectl get integrationbindings -n tenant-demo

# Kernel / GitOps (not per-tenant app charts)
kubectl get applications -n kernel-gitops

A tenant's apps are observed through the Component in tenant-{name} and its helm Release (an App claim beside it for a profile that brings its own Composition). Argo CD Applications cover the kernel's services, the claims and the deployments repository's tenant and catalog directories, not each tenant app install.

6. Metrics​

The operator exposes Prometheus metrics (internal/controller/metrics.go); the chart ships a ServiceMonitor for them, off unless metrics.serviceMonitor is enabled. The platform installs no Prometheus.

MetricDescription
gentianos_tenants_totalTenants managed
gentianos_tenant_apps_totalRequested apps per tenant
gentianos_provisioning_duration_secondsDuration of a tenant provisioning pass
gentianos_reconcile_errors_totalReconcile errors by controller
gentianos_credentials_age_secondsAge of last-provisioned credentials per tenant
gentianos_integration_bindings_statusBinding state by contract
gentianos_externalsecrets_sync_statusExternalSecret sync state per tenant
gentianos_tenant_export_total, gentianos_tenant_export_quiesce_duration_secondsExports that ended, and how long an app was paused for one
gentianos_openbao_snapshot_stateThe state of the snapshots of instance (openbao, openbao-transit), 1 on the one that holds: Current, Stale, Missing, NotRaft, Unknown, Disabled
gentianos_openbao_snapshot_age_secondsAge of the newest snapshot of instance; -1 while there is none

Crossplane, External Secrets and Argo CD expose their own upstream metrics.

6.1 Usage sampling​

The operator runs a leader-elected worker that records each tenant's enforced ceiling and what is committed under it into that tenant's own {tenant}_shell database, every usage.sampler.interval (15 minutes by default). It is what the administration console's Resources screen and kubectl gentian resources report read — see resource-plans.md.

# Is it running, and is any tenant being skipped?
kubectl logs -n kernel-control deployment/gentian-os | grep usage-sampler

A tenant without a desktop-database Secret in its namespace is skipped and logged; the others are unaffected. One tenant's database being unreachable never stops the pass, because a single broken tenant must not become a cluster-wide gap in the billing record.

6.2 Usage statistics report​

Gentian OS is free. A cluster says, in the open, what it runs, and nothing on the cluster is blocked by the answer: the operator reads the HTTP status of the reply and nothing else.

When. A leader-elected worker in the operator sends one report a few minutes after the operator starts and one every 24 hours after that. A report that does not arrive is retried with a doubling wait, from one minute up to one hour. No failure stops anything else.

Where. An HTTP POST to one https address fixed at install time (usageStatistics.url in the operator chart). A redirect is not followed.

What is sent.

{"version": 1, "sequence": 12, "sentAt": "2026-03-04T05:06:07Z",
 "cluster": {"id": "<cluster id>", "url": "https://<kernel domain>"},
 "tenants": [{"url": "https://<the tenant's host>", "uid": "<metadata.uid>",
              "issuer": "https://id.<kernel domain>/auth/realms/<realm>", "users": 40,
              "apps": [{"coordinate": "<catalogue>/<app>",
                        "digest": "sha256:…", "users": 25,
                        "addons": [{"coordinate": "<catalogue>/<addon>",
                                    "digest": "sha256:…", "users": 7}]}]}],
 "publicKey": "<base64 Ed25519 public key>"}
  • tenants[].users is the number of enabled accounts in the tenant's realm -- or, for a tenant people sign up to themselves (spec.signUp.enabled), the number of its members: an account made on the registration page is in no group of the tenant and is not a user of it.
  • tenants[].uid is the tenant's metadata.uid: random, never reused and not derived from its name, so a tenant deleted and made again under the same name is another tenant to the receiver. A store names its partner entry for the tenant's realm by it (store-contract.md §6.1).
  • tenants[].issuer is the tenant's realm: the issuer of the assertions its people's apps send a partner such as the store, which finds the tenant by it (AD-17).
  • apps lists only the apps installed through the App Store: the entries of Tenant.spec.apps that carry a digest. users is the number of people entitled to the app, the members of its group.
  • addons lists, under each listed app, the add-ons of it that are pinned to a build: the entries of that app's addonPins whose add-on is in its addons. users is the number of people entitled to the add-on, the members of its own group, counted as an app's are. The key is always sent, [] for an app with none. An add-on activated with no pin is not listed, as an app with no digest is not. The director pins an add-on only inside an app that is itself pinned, so a pinned add-on always has a listed app to appear under. A manifest written before that rule may still hold a pinned add-on inside an app with no digest; the report still omits it, because the app it would be listed under is not in the report and no entry is made up for it.
  • coordinate is null for an entry whose install did not record its catalog (spec.apps[].catalogue, addonPins[].catalogue), and a count that could not be had is null rather than a smaller number.
  • Apps are sorted by digest, and an app's add-ons by digest and then by name, so a cluster whose state has not changed says the same thing again.

The format does not forbid members a receiver does not know.

What is not sent. No names, e-mail addresses, user ids, group names, tenant display names or administrators' addresses, and nothing about apps or add-ons that were not installed at a stated build.

Signing. Each cluster has an Ed25519 key pair. The installer writes 32 random bytes to the vault (gentian-os/kernel/usage-statistics, signing_seed), an ExternalSecret hands them to the operator, and the operator derives the pair. The exact request body is signed: X-Gentian-Signature: ed25519=<base64>, with X-Gentian-Key-Id naming the key (the first 16 hex characters of the SHA-256 of the public key, which is in the body). The seed is never sent or logged. A cluster without the seed sends nothing and records why.

Seeing it. The last report exactly as sent, and what became of the last attempt, are in the ConfigMap gentian-usage-statistics in the control namespace. The usher serves the same to whoever holds can_audit on the cluster:

GET /v1/clusters/{cluster}/usage-statistics
{"enabled": true, "url": "https://…",
 "attempt": {"at": "…", "outcome": "accepted", "httpStatus": 202, "nextAt": "…"},
 "report":  {"sequence": 12, "body": "<the request body, byte for byte>",
             "signature": "ed25519=…", "keyId": "…"}}

outcome is accepted, failed (sent, and refused or not delivered; see reason, httpStatus, error), not-sent (reason is signing-key-absent, signing-key-invalid or inventory-unavailable) or sending. A cluster that does not report answers {"enabled": false}.

A cluster that does not report sends nothing, ever, and creates no signing key. It has no App Store app. A store serves a tenant only when the reports it has received list that tenant's address (store-contract.md §6.2), so the usher's tiles answer (GET /v1/tenants/{tenant}/tiles) carries "appStore": {"available": false, "reason": "usage-statistics-disabled"}, the operator places the App Store app on no tenant and removes it where it was, and no App Store tile is shown; the installer also leaves catalogue.storeUrl out of a new Cluster claim. (A cluster that reports and names no store has none either, and answers "reason": "no-store-configured"; one whose claim names its own App Store app's address answers "reason": "store-address-is-own-host".) Everything else runs as before, and apps are installed by command.

For whoever receives it. The request above is also written down as a machine-readable format, usage-statistics.openapi.yaml. It describes what the operator sends and changes nothing about it.

7. Image Updates via ArgoCD Image Updater​

7.1 Philosophy​

The kernel is a singular shared service: one kernel version per cluster, all tenants on the same version. This is intentional — kernel upgrades are platform-wide and atomic.

An app is not upgraded with the kernel. A tenant's install names the build of the profile it runs (spec.apps[].digest); the ComponentProfile is committed to the deployments repository at that digest and delivered by Argo CD, and so is every build any tenant installed, as a ProfileBuild (custom-catalogs.md §6). How an app moves to a newer build -- the platform inside the cluster's window, the cluster's administrator, or the tenant's -- is its update mode's to say, and the move itself exports first, verifies after and rolls back by itself (versioning.md).

The platform this cluster runs is on the Cluster claim's status (status.platform): the version line and the build the operator runs, and the newest release the platform's feed publishes (GENTIAN_RELEASE_FEED_URL; none on a cluster that reaches nothing). A profile declares the oldest line it runs on (spec.requires.platform), and the director refuses a build whose floor this line does not meet.

7.2 Per-Environment Update Policies​

Cluster-to-stage mapping and promotion workflows (simplified dev→prod and fortified dev→staging→prod) are documented in deployment.md.

Nothing rolls the kernel forward on its own. The gentian-os Application (in kernel-gitops) follows the ref the cluster tracks, and the operator's image is pinned to the build of the installer's checkout: a branch resolves to that commit's image, a release tag to its version. ./install.sh --only B-01 advances the pin.

Argo CD Image Updater is installed (step A-06), but the platform writes no ImageUpdater resource, so it updates nothing. A cluster that wants an image to roll on its own adds one in kernel-gitops, naming the Application and the tags it may take.

7.3 Tenant Impact​

Tenants are not parameterized by kernel version — when the kernel is updated, all tenants use the new kernel. This is the intended behavior: cluster admins manage kernel versions, tenant admins do not.

7.4 Upgrades and definitions​

A cluster holds two kinds of resource definition for this API group, and they reach it by different roads:

DefinitionIn the repositoryReaches a cluster by
The operator's own CRDs (Tenant, Component, ComponentProfile, …)charts/gentian-os/crds/The Argo CD sync of the gentian-os Application — the same sync that rolls the operator and the director, CRDs first.
The kinds Crossplane generates (App/XApp, XTenant, Cluster, Repository, …) and the Compositionscrossplane/xrds/, crossplane/compositions/The gentian-definitions Application in Argo CD, at the revision the cluster follows. On a first install, installer step B-06 applies them from the checkout before Argo CD syncs this repository: ./install.sh --only B-06.

An API server prunes what it does not know. When a definition on the cluster is older than the software — Argo CD following a branch that moved back under a pinned image, a plain helm upgrade (which never touches crds/), a new image rolled before the gentian-definitions Application synced — a field the software writes is dropped on write and nothing fails. So both binaries carry the definitions they were built with (internal/schemacheck, copied by make manifests), and the operator compares them with what the cluster serves: at start, then every five minutes, and every thirty seconds while something is wrong. For a Crossplane kind it reads the CRD Crossplane generated, claim and composite, because that is what the API server prunes against. It reads each definition by name and may read no other.

What a person sees when a definition is older than the software:

  • The operator's log names the kind, the fields that would be dropped and the remedy, and a Warning Event DefinitionsOutdated is recorded on the operator's pod.
  • Every object a held reconciler would have reconciled carries the condition DefinitionsCurrent=False, reason DefinitionsOutdated, with the same message. An outdated chart CRD holds every reconciler of the operator's own kinds; an outdated Crossplane kind holds the reconciler that writes it — XTenant the tenant's, App the component's. The operator keeps running.
  • The director answers 503 to a write that sets one of those fields, naming the definition and the remedy, and commits nothing. Reads, removals and writes that do not touch the field are served. If the director cannot learn the state at all — the operator is unreachable, has not checked yet, or is mid-rollout on another build — it refuses every write that sets a field and says the state could not be confirmed.
  • GET /v1/definitions on the operator's listener returns the whole finding: each kind, its state, the missing fields and the remedy. The director's and the usher's identities are admitted to it.

What to do. For a chart CRD, sync the gentian-os Application at the revision the image was built from. For a Crossplane kind, sync the gentian-definitions Application the same way; on a cluster that has none yet, run B-06 from a current checkout. Nothing has to be restarted: the next check lets the held reconcilers go, the condition comes off, and the director serves the writes it refused.

A definition that is absent rather than old — Crossplane generates a CRD a little after its XRD is applied — is reported as DefinitionsNotReady and waited for; it holds only the reconciler that writes that kind. On a fresh install this does not arise in the normal order: B-06 applies the XRDs and waits for them to be established, and the operator arrives at D-01.

8. Safety Guards​

  • The vault's Argo CD Applications (openbao, openbao-transit) deploy with prune: false and finalizers: [], so Argo CD never deletes them when their manifests are temporarily missing from git. The Applications of the claims and of the catalog directory also sync without pruning.
  • OpenBao paths managed by Crossplane use managementPolicies: [Observe, Create] — never overwrite live credentials. See security/design.md.
  • Backups are not verified by the platform. Nothing runs a restore or checks a bundle on a schedule, and nothing alerts. scripts/recovery.sh inspect reads a bundle and its key without writing anything, and commands.md §13 is the restore drill.

8.1 The object store's certificates and keys​

  • Certificates renew on their own. cert-manager reissues seaweedfs-server-tls and seaweedfs-shell-tls (90 days) at two thirds of their lifetime. The server reads its certificate files again every five hours (weed's default, WEED_TLS_CERT_REFRESH_INTERVAL) and the Secret is mounted whole, so no restart is needed; a Job reads its certificate when it starts.
  • While the vault is sealed nothing of gentian-ca is issued or renewed. A certificate's renewal starts with about 30 days left, so that is how long the vault may stay sealed before the store's certificates expire.
  • Replacing the internal CA is a new issuer in the vault's PKI engine: generate one at pki-internal (bao write pki-internal/root/generate/internal common_name=gentian-root-ca key_type=ec key_bits=256 ttl=87600h max_path_length=0) and make it the default (bao write pki-internal/config/issuers default=<its id>), as the root token or an administrator; then have cert-manager reissue every certificate of gentian-ca (cmctl renew --all in each namespace that holds one; for the store, system-s3). Until the server's next refresh, Jobs with a certificate of the new CA are refused: restart the store (kubectl rollout restart deploy/seaweedfs-all-in-one -n system-s3) to take the new CA at once. Re-run B-08 to write the new certificate to gentian-root-ca-tls; on a self-signed cluster every client that trusts the old CA has to be given the new one. The installer never does this itself: an existing CA is left as it is.
  • Rotating a signing key (gentian-os/kernel/storage/s3/jwt) is writing a new value there, re-syncing the ExternalSecret seaweedfs-jwt in system-s3, and restarting the store, which reads the keys from its environment at start. A token is valid for seconds, so only requests in flight fail. The keys are never generated again by the installer: an existing path is left as it is.

8.2 The vault's own certificate​

  • The serving certificate openbao-tls renews every 60 days, signed by the vault's CA openbao-ca, which every client trusts; no client changes. The vault reads its certificate only at start or on SIGHUP; the sidecar tls-reload in its pod sends the signal within a minute of the renewed file reaching the pod, and B-01 (./install.sh --only B-01) does too.
  • Replacing openbao-ca (ten years; renewed by cert-manager with the same key): delete its Secret openbao-ca-tls in kernel-secrets and the Certificate is issued again with a new key; then re-run B-01, which copies the new certificate to cert-manager's, the operator's and Crossplane's namespaces, signals the vault and restarts the custodian; restart the operator and provider-vault, which read it at start. External Secrets reads the CA on every connection.

9. Data Lifecycle: Reference​

What creating, backing up, restoring, importing, uninstalling, purging, retiring and deleting do with each kind of thing a tenant and an app own is in data-lifecycle.md: one table, the order, what a bundle does not hold, how each act fails, and its limits. Read that first. This section is the reference under it: detail that page leaves out, and nothing that page already says.

The inventory and the order are in internal/backup: inventory.go names each thing (InventoryOf, and AppVolumes for whose a volume claim is), teardown.go lists the kinds in the order provisioning makes them (AppKinds), says what every act does with each and how each is found once its app is uninstalled, builds the one Job that destroys each store, and holds the one question that says which databases are an app's. record.go is the record provisioning keeps of what it made. Teardown is provisioning's order reversed (TeardownOrder); nothing else defines an order. A test fails when a kind is added without saying what export and each teardown do with it, when something uninstalling keeps does not say how it is found afterwards, or when a store a profile can declare is left out of the record.

9.1 Per kind: what makes it, and the fine print​

In provisioning order.

KindMade byDetail
Provisioning recordsevery provisioning Job, and the operator's labeled Secrets, from the first step onJobs, their pods and labeled Secrets; found by the tenant's and the app's labels. A teardown's own Jobs are records too, which is why they go last.
Stored credentialsthe tenant reconciler's seeders, before each store and for generated secretsVault paths …/apps/<app> and …/apps/<app>-<extension>. A tenant's deletion removes them with the tenant's whole vault subtree.
Access group and membershipsthe tenant's identity Job, from Tenant.spec.appsCarried inside the realm export; a purge destroys it in the tenant's own realm.
Sign-in scopethe tenant's identity Job, from the OIDC pack the app's profile namesThe client scope with its mappers. Kept at an uninstall: taking it away there would need the uninstall to talk to the identity provider, and an uninstall is a commit nobody waits on. A purge destroys it unless an installed app names the same scope or it is one of Keycloak's own, and with it the app's client if one is left: the identity Job makes the client before the release is rendered, so an install that never got that far leaves one its uninstall does not remove.
Database and rolea role Job and a CloudNativePG Database, or a MariaDB setup JobA bundle holds a dump of the provisioned database and of every other database that is the app's (§9.3). A restore replaces the provisioned one; each other one is created if missing and replaced, under the provisioned name here when that is not the one in the bundle -- the tenant's database prefix differs between where the bundle was taken and here (<source>_x becomes <target>_x; on PostgreSQL any other name y becomes <target>_y); one that is the app's now and that the bundle does not hold is left, and named. A PostgreSQL database owned by another role is never replaced: the restore fails first.
Object storagethe bucket JobA restore makes the bucket, its user and its policy with the code install uses, then writes the objects, removing objects the bundle does not hold.
Cache userthe ACL JobA restored cache is stale, so none is carried.
Model keythe tenant reconciler, at the model gateway, where the cluster serves modelsA credential, registered again at install. A tenant's deletion removes the keys, then the tenant's team.
Sign-in clientthe component reconciler, or the identity JobWith its client role, its default-scope assignments and the group's mapping to the role. Keycloak removes what is part of the client with it.
Workloadsthe Helm release the component reconciler writesThe manifest records the chart version, the digest and the releases; a restore checks the installed build against them (§9.4). A Release or Object a gone composite left composed (an install cut short leaves one on an unfinished create, which its provider never deletes) is unblocked and deleted by the uninstall's tenant reconcile and by a purge, which waits for it; an install renders nothing beside one (internal/composed).
Filesthe app's chart, as volume claims of its releaseAn archive per claim that is the app's (AppVolumes), unpacked onto the claim of the same name. At a tenant's deletion they go with the namespace.
Materialized profilecommitted by the director under clusters/<cluster>/catalogue/ at the digest the install named, applied by Argo CDKept by every act: a purge of the app reads it to know what the app owns.
Bundle companionsapplied with the profile, from the same fileThe Composition, OIDC pack catalog, ConfigMaps and customization records a profile's bundle brings. Kept by every act.

So the teardown order is: files, workloads, sign-in client, model key, cache, object storage, database, sign-in scope, access group, stored credentials, provisioning records. A purge of an app runs the steps of that list that destroy (the workloads and the client went with the uninstall). A tenant's deletion removes the workloads first — its Components, and with each the App claim and the release — waits for them, then runs the store steps in the same order for every app the tenant ever had, with the same Jobs and scripts, then the realm; then it removes the namespace and waits until the API server no longer has it, removes the vault subtree and, last, the records.

The last two rows are not a tenant's. A profile and its companions belong to the cluster, are shared by every tenant that installs the app, and no act on a tenant or an app removes them; neither does a newer build that stops bringing a companion. They leave by one deliberate act of the cluster's administrator, one object at a time: the operator lists what no bundle owns any more and the profiles no tenant uses or retains data for, and removes an entry of that list on request (custom-catalogs.md §6).

9.2 What a tenant has that is no app's​

Several of these lie outside what a deletion sweeps by default — outside the tenant's namespace, realm and vault subtree, and without its label. The inventory lists them (TenantOwned) so that each says what removes it. What each act does with them is in data-lifecycle.md §2; the fine print:

  • The namespace. With Retain the Components and the operator's quota, limits and network policy are removed from it. With Delete the deletion waits until the namespace is gone.
  • The realm. Never deleted or disabled when it is the kernel realm, which a tenant only adopts.
  • The client a realm signed in to the kernel realm as (broker-<realm>), and the mapper on it. The platform makes none (iam.md §1.7); one that exists is removed from the kernel realm by the Job that deletes the realm.
  • The vault subtree. An operator with no vault that was not told to run without one (GENTIAN_WITHOUT_VAULT=true) fails the deletion here.
  • Mail. The DNS records are DNSEndpoint mail-<tenant>; a deletion also removes the DKIM key and the SMTP credentials.
  • The edge. A route that cannot be removed fails the deletion.

9.3 Finding what an uninstalled app left​

An uninstalled app has left the tenant's manifest and all its stores are still there. Provisioning therefore writes down what it makes: one ConfigMap per tenant in the provisioning namespace (tenant-<name>-provisioned-stores), one entry per app with the database engine and names, the bucket, the cache user and the model key, written before the Jobs that make them are handed over and never reduced by provisioning. A purge takes a kind off the record once it has destroyed it. The deletion of a tenant, a purge and the read of what uninstalled apps hold (GET /apps/retained) read it, and so does an export, which captures what an uninstalled app left (§9.4); the read answers present or absent for a bucket, a cache user and a MariaDB database from it without running anything, and unknown only when the record could not be read. A PostgreSQL database is also found through the CloudNativePG Database object; files, credentials and the access group by the claims, vault paths and group names themselves.

Which databases are an app's is one rule that export, restore and purge share. The provisioned database is. On PostgreSQL so is every other database the app's role owns — a role creates databases only when its profile asks (allowDynamicDatabaseCreation), owns what it creates, and the server records the owner.

On MariaDB a database has no owner and one server holds every tenant's, so the rule is by name, and the same names are all the app's user may touch. An app's databases are its provisioned database <database> and every database named <database>_… — the provisioned name, an underscore, anything. demo_crm_reports is demo_crm's; demo_crm2 and demoXcrm are not: the grant and the query escape the underscores, which MariaDB otherwise reads as "any one character", and compare byte for byte. The user of an app whose profile sets allowDynamicDatabaseCreation is granted all privileges on those names; every other app's user on its provisioned database alone; none is granted anything on the server (*.*). The rule and the grants are stated once, in internal/backup/mariadb.go.

Hyphens in a tenant's or an app's name become underscores in a database's, so one app's provisioned database can be named under another's prefix: demo_crm_extra is app crm-extra of tenant demo, and app extra of a tenant demo-crm. Two things keep the rule exact there. A database under an app's prefix that another account holds rights on is that account's: export, restore and purge leave it alone and say so. And provisioning refuses, under a lock on the server, to make the overlap where a grant would span it — an app whose database another account's rights already reach, and an app that asks to create databases while another account's database lies under its prefix. The setup Job fails with the reason and the app is not provisioned.

Clusters provisioned before this rule. The setup Job is run again when its script changes, and a run leaves the user with the grants above and nothing else: a user that held ALL PRIVILEGES ON *.* … WITH GRANT OPTION (every app with allowDynamicDatabaseCreation on MariaDB did), or a grant on its unescaped database name, has everything revoked and is granted again. Two things it cannot put right. A database such an app created under a name outside its prefix is no longer reachable by the app, and is in no export and no purge: rename it under the prefix or drop it by hand. And an account the app created for itself while it could (CREATE USER, GRANT) is not the platform's to find: compare mysql.global_priv with the tenants' apps. Until every app's Job has run again, an old unescaped grant of one app can make the Job of another fail with "within the rights of another account"; it passes once the first has run.

9.4 Restore​

What a restore puts back is what the bundle says it holds. The operator reads the bundle's manifest with the restore's own key and makes a plan once, before anything is changed; the plan is in the restore's status.

  1. A bundle is restored only into the tenant of its own name. One whose manifest names another tenant is refused before anything is changed (reason OtherTenant): a restore does not rename, and a bundle meant for another name is rewritten into it, by a tool that does so, first.

  2. Only an app the manifest lists is touched. An app the tenant has and the bundle does not is left exactly as it is.

  3. An app is restored whole or not at all. It is not restored when it is not installed; when its ComponentProfile is not on the cluster; when the build installed is older than the one that wrote the data, or cannot be compared with it (unless spec.skipVersionCheck); when the bundle holds a store of a kind or engine the installed app does not have; or when it holds a volume claim that is not one of the app's here. A newer build installed is restored: an app upgrades older data when it starts.

  4. Every app not restored is named with the reason in status.notRestored, and the restore ends Ready with status.complete: false and the reason PartiallyRestored. Nothing a bundle holds is dropped without saying so.

  5. With spec.apps, only the apps named are considered and each has to be restorable; otherwise the restore is refused before anything is changed.

What is guaranteed. The platform guarantees the round trip: a bundle restored by a platform of the version line that wrote it (operatorVersion), into the tenant it was taken of, on the same domain. A bundle of an older line, or one that records none, is restored as far as this platform reads it and nothing in it is converted; one taken on another domain is restored, and the apps' data still names the old domain where their restore.post commands do not rewrite it. A custom domain is not declared by a restore or an import: the message names it and the command that declares it again. Each way the restore carries the condition OutsideGuarantee (reason OtherVersion, OtherDomain or SeveralDifferences), whose message says what differs and is repeated in status.notes. A bundle of a newer line is refused, as a newer format is, by the restore and by the import's inspection alike.

Each artifact is fetched from the path the manifest gives and loaded into the store of that kind the installed app has, by the inventory's names — which differ from the bundle's wherever the tenant's database or bucket prefix (spec.isolation) here is not the one it had where the bundle was taken. A profile's restore.post commands run with GENTIAN_RESTORE_* variables naming both ends: tenant, domain, the cluster's own (kernel) domain, platform version, cluster, the app's chart versions, each store's name in the bundle and here -- the databases an app made itself too, which the restore's Job reports in its termination message and the restore keeps on the artifact (databases) -- and the app's database login (custom-catalogs.md §4.1). They carry no secret; each app's profile rewrites what its app cached, reading credentials the platform made new from the app's own Secrets. An app paused by scaling down is resumed before them, as they need a pod, and rolled (rollout restart) once they have run; one scaled down because its maintenance command could not run has its quiesce.post command run after them, since its data may have been taken in maintenance mode. Each of a profile's commands runs for at most 15 minutes (AppCommandTimeout); one still running then has failed, as one that exits non-zero has. An app is done once its workloads have rolled out on the restored data (at most 10 minutes, then a note).

Held back at the edge. Before a restore changes anything of an app — its secrets, its pause — it names the app in status.heldBack and waits until every route of the app is held: its entries, its sign-in sidecar's route and its published surfaces in the DMZ answer 503 with Retry-After: 60 from an Envoy Gateway HTTPRouteFilter (restore-hold) instead of reaching the app, behind the same session and bouncer as before (routing.md §11.1). An edge that has not held it within 2 minutes fails the restore with nothing of the app changed. The hold lasts until the restore ends, Ready or Failed, or is deleted — past the app's own restore.post and rollout and past the realm, which comes back last — so nobody signs in to an app before the people it matches on are back. It is read from the restore's status by the component reconciler, first in each pass, so an operator restart or a failing component does not keep a route held; status.heldBack is emptied when the restore ends.

The bundle format. The manifest (manifest.json, encrypted like every artifact) carries schemaVersion. Format 1 named each app and the kinds captured, nothing else. Format 2 adds per app one stores entry per artifact — kind (postgres, postgresOwned, mariadb, mariadbOwned, s3, volume), name (what it was captured from), path (where in the bundle), and for a volume the Helm release it recorded — and the app's digest, databaseEngine and releases; the tenant-wide captures are not listed among the apps. A format 1 bundle still restores: the names are derived from the tenant's prefixes the manifest records and the volume claims are the ones the apps have now, and the result says nameDerivation: derived. Format 3 adds retained on an app that was uninstalled with its data kept when the bundle was taken, and on each of its volumes the claim it was captured from (size, access modes, and the labels and annotations that say whose it is), from which the claim is made again where it is not; mailboxes (kind: mailboxes, name the mail domain, path); and rights, the entries of the rights store that follow from nothing else, as granted and withdrawn lists of user, relation, object, with the cluster they were read on. A format 2 bundle has none of them and restores as before. Format 4 adds to an app's stores an entry of kind: secrets, name the app, path secrets/<app>.json: the app's own secrets, a JSON document of secrets (name to value), extensions (extension, then name to value) and derived (the keys the profile declares under secrets.derived, key to value), with no path in it. A document without derived is read too. A format 3 bundle has none and restores as before, changing no stored secret (security/design.md §6.4). Also within format 4, and optional, the manifest says by what and where it was written: operatorVersion, the platform's version line (0.5); domain, the tenant's domain, its custom one where a TenantDomain bound it, and customDomain, that one alone; kernelDomain, the cluster's own domain, under which its sign-in issuer is; and cluster, the cluster's id in its deployments repository; and clusterUid, the cluster's random id (spec.clusterUID of its Cluster claim). A bundle without them reads as before. Beside the manifest, manifest.json.sig (JSON: algorithm ed25519, publicKey, signature, base64) is the cluster's signature over gentian-os bundle manifest v1\n followed by the decrypted manifest without its final newline; a bundle without it is unsigned, and a reader that ignores it reads the bundle as before. A manifest of a format newer than the platform reads is refused. A postgresOwned artifact is a tar.gz holding INDEX, the database names one per line, and <line number from 0>.pgc, each one's custom-format dump. A mailboxes artifact is a tar.gz holding INDEX, the mailboxes one per line by the part of the address before the @, and <line number from 0>/, each one's mail as a Maildir++ tree written by doveadm backup: every folder and message with its flags, UID and GUID.

Uninstalled apps. An app that was uninstalled with its data kept is captured after the installed ones, by the same units, from the stores the cluster still holds for it: the ones on the record of what was provisioned, a PostgreSQL database the cluster keeps a record of, and the volume claims that are the app's. Nothing is paused. A restore puts the data back into the stores the tenant still holds for the app and installs nothing; where the tenant holds none of it, it was purged, and the app is named in status.notRestored. Only a restore with spec.intoNewTenant, which an import sets, makes the stores: the record first, then the database with its role and vault record (or the MariaDB setup Job), the bucket with its user, and each volume claim from what the bundle recorded, on the cluster's default storage class.

Mailboxes. On a cluster that runs its own mail server (mail.serviceMode: system) a tenant's mailboxes are the directories below its mail domain's on the server's volume. They are copied by doveadm backup, in the mail server's own image, beside the volume and on the node the server holds it on: the server's own synchronization takes the server's locks, where an archive of a Maildir that is being written to can miss a message that is moving between new/ and cur/. A restore runs doveadm sync one way, from the bundle into the mailbox: what the bundle holds and the mailbox lacks is added, and nothing is removed. A mail domain that is the cluster's own (the user tenant of a single-tenancy cluster) is shared with the cluster's administrators; its mailboxes are neither copied nor destroyed, and status.notIncluded says so. On a cluster without a mail server there is no unit and nothing is said.

The archived mailboxes of removed people (mail.md §5c) are in the same artifact, under archived/ with an INDEX of their own, copied the same way from /var/mail/.archive/<domain>/. The manifest names them (archivedMailboxes: archive, address, when, by whom, size). A restore synchronizes each into /var/mail/.archive/<target domain>/<archive> and writes a MailboxRemoval for it that says it was restored, so that it is listed and can be deleted; the operator runs no Job for such a record. A tenant's deletion removes /var/mail/.archive/<domain> with the domain's mailboxes, and then the tenant's records.

Rights. The operator reads the tenant's entries in the rights store and writes into the manifest the ones its projection would not write, and the defaults the store no longer holds; no Job reads the store. A restore writes them after the realm is back and after the projection has attached the tenant. The withdrawn ones are withdrawn, under this cluster's name. The granted ones, and any entry naming a person, are written only from a bundle this cluster signed, taken on a cluster of this cluster's id (security/design.md §6.4); from any other they are named in status.rights.notBrought and in the notes, and status.rights.provenance says why: NoSigningKey, Unsigned, SignedByAnotherKey, BadSignature, NoClusterID or AnotherCluster.

Replacing the bundle signing key. The key is never replaced by the installer. To replace it, delete gentian-os/kernel/backup/signing in the vault, run ./install.sh --step B-08 --force, which seeds a new one, and export a new recovery kit. Every bundle signed before then reads as signed by another key: it still restores, with what was withdrawn withdrawn and no granted right brought back. Changing spec.clusterUID has the same effect on every earlier bundle.

What no restore brings back is the checklist in data-lifecycle.md §5, and is said on every result (status.notes). The detail behind it:

  • Stored credentials, but for each app's own secrets, which a bundle carries and a restore sets (security/design.md §6.4). A profile's spec.backup.boundSecrets are not carried, and an export of such an app says so.
  • Declared state. App grants are declared in git and come from there; integration bindings are derived from the installed apps and their profiles; authorization tuples are projections, but for the ones a bundle carries (above). On the same cluster none of the three is lost. A tenant imported into another cluster has its bindings and tuples made again and its app grants to set again.

An uploaded bundle (the import bucket, gentian-imports) is removed when a restore of it has run to its end, restored or failed. A restore refused before it changed anything leaves it, so the request can be made again; an upload that is never restored stays until it is removed by hand.

9.4a Where each step of a backup and a restore runs​

A step runs in the namespace where the credential it works with already is. No administrator credential of a database or of the identity provider is copied anywhere.

StepNamespaceReads there
PostgreSQL dump and load, an app's and the desktop'ssystem-postgresqlpostgres-admin
The desktop's database of the tenant that adopts the kernel realmkernel-datakernel-postgres-role-portal-shell: the database's own role, not an administrator
Mailbox copy and loadsystem-mailthe mail server's volume
MariaDB dump and loadsystem-mariadbmariadb-admin
Realm export and importthe identity namespace (kernel-authentication)keycloak-admin
Bucket archive and load, the manifest, a bundle's removalsystem-s3s3-admin, and the bundle's own credential and key
Volume archive and loadthe tenant's namespacethe claim
An app's own secrets, into the bundle and backno Job: the operator itselfthe vault, as the operator; the bundle's credential in system-s3; the key from the export's or the restore's own Secret

What a step needs besides is the credential the bundle is reached with and the key it is encrypted or opened with. Those exist beside the object store. For every other namespace a run has a step in, the operator stages one Secret holding both (tx-<tenant>-<run>-run-creds, …-rcreds for a restore) and removes it when the run ends: completed, failed or deleted. For a bundle on the platform's own storage the staged credential is the object store's administrator's.

The names of a run's Jobs and Secrets carry the tenant's name: those namespaces are shared, and a run's name is unique in its tenant only. A Job or Secret found under a run's name that another run made is never used.

The object store admits these pods by their label (gentianos.io/component=tenant-export) from those namespaces (security/design.md §2.5). A test builds every step with the reconcilers' own code and fails when a pod reads a Secret that is not in the namespace it runs in (TestEveryUnitFindsWhatItsPodReadsWhereItRuns).

9.5 Failing loudly​

What a person sees when an act fails is in data-lifecycle.md §6. Underneath: a destroy script ends in success only when what it was asked to remove is verifiably gone. A purge of an app stops at the first step that fails and answers with it (store-contract.md §8). A tenant's deletion is not waited for by anybody: a cleanup Job that fails, a vault or a model gateway that does not answer, an edge route or a provisioning Job that cannot be removed, is a reconcile error; the Tenant stays Terminating, and the failed step is run again on the next pass. The Tenant is gone only when its namespace is.

The scripts of a backup and a restore are held to the same rule by the same kind of test (TestNoUnitScriptDiscardsAFailure): each stops at the first command that fails and discards no failure. The realm import goes on to the end, names every person and membership it could not put back, and then fails. A step whose pod does not start is counted like one that failed, and a backup or a restore gives up after exportMaxAttempts of them, in every stage, starting the paused app again. A TenantRestore holds a finalizer: deleting one that runs stops its Jobs and starts the app again.

9.6 Known limits​

The limits that matter to a person running the platform are in data-lifecycle.md §7. The technical limits behind them:

  • Cache keys. The cache is one shared instance and an app's user may touch every key, so the keys an app wrote cannot be told from another's. A purge and a tenant's deletion remove the user and leave the keys.
  • spec.isolation.namespace. Export, restore, purge and the retained read take a tenant's namespace from the Tenant, as provisioning does. The rest of the platform does not: an export or a restore finds its tenant by stripping tenant- from its namespace, and the director never sets the field. A tenant placed in a namespace of another name is not supported end to end; an export or restore created for one is refused.
  • MariaDB databases an app creates for itself are the app's by their name (§9.3). Each database is dumped at one instant of its own; an export is consistent across an app's databases only because the app is paused. A database whose name has a control character in it fails the export. The read of what uninstalled apps hold reports the database kind as one — present or absent, from the record — and does not list the databases.
  • The MariaDB client is the server's image. The Jobs that provision, dump and load run the image the MariaDB chart pins. A newer mariadb-dump writes dumps that server refuses to load; the two are bumped together.
  • Files at a tenant's deletion go with the namespace, after the stores, not before them.
  • Renamed databases. A database an app made for itself is restored under another name where the tenant's database prefix here is not the one it had where the bundle was taken (§9.1). The app's own settings may still name the old one, as far as its restore.post commands do not rewrite it.
  • Bundle containers. A bundle is stored as a prefix in object storage: one age file per artifact, manifest.json.age written last, the signature and bundle-info.json in the clear beside it. The single file <tenant>-<export>.gentian is a tar of that prefix, streamed as the objects are read — nothing is decrypted and nothing is re-packed — and is what a download hands out and an import upload takes; an upload is laid out as a prefix in the import bucket again before anything reads it. A bundle is read where it is: from the tenant's own backup bucket, from the import bucket, or from a declared bundle source outside the cluster (data-lifecycle.md §4).
  • Encryption modes. Every artifact is an ordinary age file, and both modes open with age -d and no platform tooling. recipient, the default, encrypts to age public keys: with none named, to the cluster's configured recipients, whose identity is escrowed with the recovery kit so the platform can read the bundle when the person who took it is unreachable; with keys named, to those alone, which the platform has no identity for. passphrase encrypts under a passphrase from a Secret in the tenant's namespace, copied beside the capture Job and removed when the export finishes; the platform keeps no copy, so a lost passphrase is a lost bundle. The mode and the public keys used are recorded on the export's status and in bundle-info.json, with the command that opens it.