Gentian OS v0.4

Milestone: the customization ladder, tenant backup and restore, and an installer that converges. Compares against: v0.3.0 (2026-07-18) — 888 commits. Cut with: the runbook in deployment.md §5.4.

v0.3 finished the app story: apps installed, authenticated and, with the AI line, talked to a model. v0.4 is about what happens to an app after it is installed — how a customer changes it without forking it, how their data gets out and back in, and how the cluster that runs it is built from a description rather than from an operator's shell history.

Three subsystems carry that, and each replaces something that used to be implicit: the customization ladder replaces "patch it wherever you know how", tenant export/restore replaces "we have volume snapshots somewhere", and the step driver replaces install.sh + update.sh + uninstall.sh and the knowledge that lived only in whoever ran them.

---

1. The customization ladder

The headline, designed and delivered in this release: app-customization.md.

Gentian had customization mechanisms — extraValues, tenant config, sidecar addons, an Odoo fork — and no ordering over them. Nothing told a human or an agent which mechanism was the cheapest one that could express a change, which ones a given app supported, or what the change would cost at the next upstream release. v0.4 supplies that ordering.

• Seven rungs, L0–L6 (configure, drop-in, companion, extension, repackage, patch, fork) ordered by how much of the app's own delivery artifact Gentian ends up owning — which is what determines upgrade cost. Cluster hotfixes are Rung X and forbidden.
• A second axis, scope — tenant / profile / platform blast radius, chosen independently of the rung.
• AppProfile.spec.customization — each app declares which rungs are reachable and by which mechanism, plus an A–D readiness grade. Generic advice is useless; "Odoo supports L3 via addons, Collabora does not" is actionable.
• The Customization CRD — a DEP-3-inspired record required for every customization at L2 and above, carrying owner, origin, exit criteria and a review date, so a downstream delta is a debt with a due date rather than a decision. A CI validator and a debt report enforce it; the report surfaces in the Admin Console.
• The addon model (L3) — addons are AppProfiles that activate inside a base rather than deploying alone. They need no chart of their own, inherit the base's ladder, and are selected per tenant through Tenant.spec.apps[].addons and a PUT endpoint. Commercial addons are withheld until an entitlement grants them.
• Editions ce · me · ee — how an entry is licensed, not who publishes it. spec.author names the supplier, which may be the upstream organization or a third party. AppPackage presets preselect a curated addon set at install without becoming an artifact of their own.
• Tenant-scoped L1 drop-ins — a tenant admin can add a config or asset file through the console, rendered by the operator into a ConfigMap.

The framework is authorship-neutral by design: Customization.spec.origin records whether a customization came from Gentian, a tenant, a supplier or a partner from day one. v0.4 ships that data model; the delegation process (signing, entitlement, review SLAs) is deliberately deferred.

2. Tenant backup, export and restore

A whole subsystem that did not exist in v0.3, documented for the people who use it in tenant-backup.md.

A TenantExport captures a workspace — every app's database, its object storage, its PersistentVolumeClaims, its staged Secrets and its member accounts — into one encrypted bundle. TenantRestore puts it back. Around those:

• BackupPolicy / TenantBackupPolicy set destination and schedule as policy: cluster first, then tenant. A bundle can be sent to a destination the policy does not name, reusing the workspace credential or taking one-off keys.
• TenantExportSchedule takes exports on a cadence, anchored on the window it committed to rather than drifting with each firing.
• Encryption by recipient key or passphrase, so the platform can hold a bundle it cannot read.
• Quiescing per AppProfile.spec.backup — scale to zero, or run the app's own pre/post hooks in its own pod under its own role. Consistency is per-app by default, perStore where the app says so.
• minRestoreVersion refuses to restore a bundle into an app older than the version that wrote it; the reverse direction is allowed, because that just runs the app's migrations.

Schedule, retention, encryption and destination are deliberately absent from the profile: they are platform and tenant policy, and an app author must not be able to weaken them.

Who holds the key is a choice. The cluster gets an age key pair at first --export-recovery-kit, and never regenerates it — a second pair would orphan every bundle written to the first. The public half goes to OpenBao; whether the private half is escrowed there too is spec.backup.escrowIdentity, on by default because a lost recovery kit is the likelier disaster than a stolen cluster. Off is the stronger position against theft and makes the first kit irreplaceable. A tenant may encrypt its own schedule to a key the platform does not hold at all. The Admin Console reports which key a workspace is on without handing it over.

Recovery is a procedure, not a folder of notes. recovery.md and scripts/recovery.sh cover listing, inspecting and restoring a bundle from a rescue shell — including the case where the cluster that wrote it is gone. age is now a hard install requirement rather than a fallback, the kit export takes its own OpenBao token like every other caller, and the backup key prints as a scannable PNG rather than terminal art nobody can transcribe.

Volume capture runs where the volume is and gives up if it cannot start there; its upload authenticates to the destination the bundle goes to, not the one the platform happens to own. A restore resumes the app before its post-restore hooks, rolls its pods so a warmed cache cannot outlive the data under it, and heals ownership on a retry instead of wedging.

Cluster-level PITR and Velero are not in v0.4. What ships is tenant-scoped export and restore, the credential escrow, and the drills.

3. Identity provisioning moved onto Crossplane

In v0.3 a tenant's Keycloak configuration was written by a series of bootstrap Jobs. Those Jobs and the Compositions were two writers over the same objects, which flapped, and no single artifact described a tenant realm.

v0.4 moves the whole surface into tenant-default one object at a time, in a repeated observe → declare → manage → retire the Job sequence: the tenant realm and its user profile, the browser flow and login theme, the first-broker-login flow and its executions, the reverse broker into the kernel realm, the OIDC pack's client roles and protocol mappers, the entitlement groups and their grants, and the required actions.

Two resolution defects went with it: a mapper no longer has its external name seeded from its own name, and clientIdRef is re-resolved so a recreated client is followed rather than pointing at a client that no longer exists.

The Jobs are gone. Their names survive, in a single documented sweep (keycloak_retired_jobs.go), purely to delete what they left behind on clusters that ran them — a retired Job is not harmless where it sits, because its TTL expires and an older image or a stale Argo revision recreates it against objects that now have a declared owner.

4. The installer became a driver

install.sh no longer installs anything. It is a driver over scripts/steps/*.sh, each of which declares requires:, provides: and mutates: and implements check(), apply() and destroy(). Ordering is the filename prefix, so the sequence is visible from ls. See reference/install.md.

• uninstall.sh and update.sh are deleted. Reversal is destroy() per step (--purge also removes the data an uninstall keeps); update is a convergent re-run, because check() makes every step idempotent.
• One command, then one handover. The install writes the cluster's recovery kit itself, at the one moment when the derivation salt, the master password and the OpenBao recovery key all exist and none is durable yet, and records the handover in a ConfigMap. Leaving that to a command the operator had to know about meant it was routinely missed.
• Scoped runs — --phase, --from, --until, --skip, --only, --dry-run, --explain, --status, --verify-only. --validate answers the question that comes before all of them: is this configuration coherent, without contacting a cluster at all.
• credentials.yaml is the credential catalog: every secret that must come from outside the cluster, in one file, with the CredentialRequirement CRs generated from it and make verify-gen asserting the two carriers agree. Derived credentials (HMAC of the master password) and generated ones are absent by design — they need no human. Prompting is catalog-driven, so a cluster is only asked for what it actually uses, and Compositions gate on credential satisfaction rather than on shell ordering.
• A Credential Manager service with ESO satisfaction probes and per-field validation, so a wrong value is attributed to the field that carries it.
• versions.yaml pins every external component the installer pulls — Crossplane, cert-manager, ESO — once, as platform source rather than per-cluster configuration. Two clusters on the same release run the same versions.
• cluster-settings.env is retired. Its declarative half is on the Cluster claim; claims arrive through Argo CD. install.env is now only a pointer file — 75 lines, one per setting — and a cluster property left in it is reported rather than silently beating the claim.
• Repositories, secrets and roles all carry the gentian- prefix, and the app catalog's credential became a step of its own (B-12) instead of a special case inside catalog installation. envsubst is gone from the install path, and the gentian Argo CD AppProject renders from the bootstrap chart rather than being applied by hand.

5. Mail became a real service

MAIL_SERVICE_MODE=kernel now actually delivers, in both directions.

Postfix and Dovecot run in the kernel with per-tenant DKIM signing over published OpenDKIM tables, port 25 exposed so the cluster can receive internet mail, IMAP over TLS, and a mail password minted per user per client app instead of a cluster-wide master user. The outbound address is a cluster declaration rather than a hardcoded relay, Postfix is pinned to the node that sends, and external-dns publishes the tenant mail records — and the rest of the cluster's records — from the cluster itself.

6. Edge, gateway and infrastructure

• One Gateway serves every hostname, with the portal reachable on tenant hostnames and :80 redirecting to :443. Load-balancer addressing is provider-agnostic, with an OpenStack health monitor where the provider offers one.
• The edge split in two: EdgeDNSWriter and EdgeIngress. Routing a hostname and resolving it are one operation, and the kernel path had been doing only the first — every kernel hostname got a tunnel ingress rule and no DNS record. It stayed invisible because every cluster so far ran on a zone whose kernel records predated the install. On a fresh zone nothing resolved, D-03 waited out its full fifteen minutes, and the timeout blamed the Cloudflare credential, which was the one component working correctly.
• external-dns owns records by cluster, not by product, so two clusters sharing a zone stop fighting over the same names. The domain filter includes the zone rather than only the host, the nameserver is found by walking up from the kernel domain, tunnel hostnames publish as DNSEndpoints, and provider settings sit on the record rather than the object.
• One provider table, kernel/platforms.yaml, decides the cert-manager solver, the external-dns provider and the credential — replacing a hard-coded exception per provider. certificates.acmeEnv picks the Let's Encrypt endpoint (staging, production, or both to render each issuer).
• Kernel service manifests are templated on stage instead of copied per environment; cert-manager manifests render through Helm; a declarable trust anchor supports an offline self-signed CA.
• Dynamic database creation on PostgreSQL, generic privileged-role provisioning (PrivilegedRoleSpec) in place of app-specific code in the kernel, and Redis ACL usernames mapped through valueMapping. Apps declare their quirks; the operator no longer names apps.
• Argo CD 3.5.0, with the operator's RBAC completed and rollback no longer hanging.
• ResourcePlan — a tenant's CPU, memory and storage ceiling is a cluster object resolved from a plan catalog, not a hand-set quota. Design: resource-plans.md.
• Node pool migration is documented as a maintenance procedure: node-pool-migration.md.
• An opt-in Kyverno exception for MetalLB's speaker DaemonSet, for clusters that address their edge that way.
• Portal images reach the cluster again. Image Updater had been pinned to a moving tag, so the candidate set was one entry whose name never changed and no new portal build was ever selected; it now tracks the branch-scoped immutable tags CI publishes.

7. Portability and supply chain

The installer runs on stock bash 3.2, so macOS works: BSD/GNU compat primitives, a portability lint, and a macOS CI job that runs the driver on the runner's own bash. The operator image is multi-arch (linux/amd64, linux/arm64) with manifest-list digest pins and a lint that keeps them honest.

CI also gained step-contract linting, a recovery-kit round trip, credential catalog parity, authorization policy tests, and a live-identifier ratchet that counts the places this repository names one specific deployment. It runs on feature branches and never publishes an image from one.

---

8. New APIs

CRD	What it is
Customization	The record required for every customization at rung ≥ L2
AppPackage	A curated addon preset, preselected at install
TenantExport	One capture of a tenant into an encrypted bundle
TenantRestore	Putting a bundle back
TenantExportSchedule	Exports on a cadence
BackupPolicy	Cluster-level destination and retention
TenantBackupPolicy	The tenant's own override within the cluster's
CredentialRequirement	One entry of the credential catalog
ResourcePlan	A tenant's resource ceiling, as a plan

AppProfile gained spec.customization, spec.backup, spec.author, spec.edition and spec.provisioning.privilegedRole.

9. Upgrade notes

• uninstall.sh and update.sh no longer exist. Use ./install.sh for both directions: a convergent re-run to update, --purge to tear down.
• cluster-settings.env is retired — its settings live on the Cluster claim. install.sh reads install.env and the claim.
• GENTIAN_OS_BRANCH decides which gentian-os a cluster runs, which need not be the one you install from. Set it to v0.4.0 to pin this release — checking the tag out is not enough, and the installer refuses a detached checkout rather than guessing. It also refuses a ref the remote does not have. See deployment.md §4.
• LETSENCRYPT_EMAIL, KV_MOUNT and ACME_ENV moved to the Cluster claim (certManager.letsencryptEmail, openbao.kvMount, certificates.acmeEnv). They are still read from install.env and reported when they contradict the claim; KERNEL_DOMAIN warns the same way.
• The edition enum is ce/me/ee. pro is no longer accepted; one catalog profile carried it and was migrated. ee now means commercially licensed and entitlement-gated, with spec.author naming the supplier.
• spec.adminEmail is gone — a tenant admin's login is their address.
• age is required. A missing binary stops the install rather than silently falling back, because the fallback could not round-trip a recovery kit unattended.
• The inline OpenFGA token is gone. authzBridge.openfgaToken no longer exists; the credential is read from a Secret, and D-01 now checks that the bridge actually holds one.
• Removed because nothing read or enforced them: the init-Job sizing fields, the mail quota and rate limit, and the TURN plumbing.

10. Deliberately not in v0.4

• Cluster-level PITR and Velero (backup plan Phase 6).
• Automated customization-readiness grading — grades are maintainer-assigned; the rubric automation is roadmap item 2.13.
• Third-party customization delegation: signing, entitlement and review SLAs. The data model carries authorship; the process does not exist yet.
• Migrating tenant provisioning entirely to Crossplane. The identity surface moved; whether the rest should follow is roadmap item 2.1.

---

v0.4.1

Released from: 35 commits on develop after v0.4.0. Upgrading from v0.4.0: no action beyond the pin. No breaking changes, and every new claim field is optional with a default.

v0.4.0 was tagged before a cluster had been installed from it on a fresh zone. Doing that surfaced four faults in the install path that no existing cluster could have hit, because each one only appears where nothing was configured by hand first. Most of this release is those, plus the things a first real tenant deployment found.

The install path

• DNS-01 challenges waited forever on a new zone. cert-manager checked propagation through the cluster's own DNS, which cannot answer for the zone: the CoreDNS hairpin serves the kernel domain from a hosts block, and a hosts entry answers every query type for its name — so the apex's SOA and NS come back empty, cert-manager finds no authoritative nameservers, and no challenge ever completes. It now checks through public resolvers, configurable as certificates.dns01RecursiveNameservers for a zone public resolvers cannot see.
• Credential consumers latched before seeding and stayed latched, so a credential supplied at B-10 never reached the components waiting for it.
• D-07 could not tell a certificate name mismatch from an untrusted chain, and reported the wrong one — sending an operator after a CA problem that was really a hostname.
• C-01 withheld the wildcard when it could not issue one. Distribution and issuance are separate concerns: a cluster with a pre-existing certificate should still get it distributed.

Tenants

• A failed deploy could tear down a tenant it did not create. Rollback now removes only what that deploy made.
• The deletion chain resurrected a purged cleanup Job, so a tenant could come back after being removed.
• gtnctl scaffolded mail fields the Tenant CRD rejects, making a scaffolded tenant unappliable.

Edge and identity

• An addon may declare a hostname, and the edge now reads it: the addon's host reaches the gateway while the base app's primary host does not, and OIDC origins are allowed for addon-declared hosts too.
• Kernel-mail DNS is no longer published on clusters without kernel mail.

Resource plans

• The plan catalog sizes itself to the node actually ordered, rather than to a unit that assumed one node shape.

External inference providers

spec.llm.providers declares an external provider — name, API base, the credential's property, and the models it offers — beside the existing spec.llm.instances. The operator registers one LiteLLM model per entry and probes egress before it does.

This closes a gap rather than adding a surface: LiteLLM ran with STORE_MODEL_IN_DB=True and no config file, so any model beyond the claim's vLLM instances had to be typed into the Admin Console — present only in LiteLLM's Postgres, absent from git, and gone on a rebuild. The obvious alternative was worse, because it fails silently: adding the provider as an OpenAI connection inside a tenant's Open WebUI cannot work, since tenant isolation permits egress to 10.0.0.0/8:443 and nothing else, so the provider's /models request times out and the UI renders an empty model list rather than an error. The gateway in platform-kernel is the only thing that can reach a provider, and it is where the budgets and virtual keys already are.

Each provider carries its own credential, so they can be supplied one at a time, and the portal is told whether the LLM capability exists at all.

Build

• The delete-path envtest flake is fixed (a cleanup Job is now observed before it completes). It had been failing roughly one run in three, and because Docker build and push is gated on the Go job, each failure meant a commit published no image.