Skip to main content

Networking

How traffic is admitted inside the cluster: the network plugin, the NetworkPolicy tiers of a tenant namespace, the kernel's rules, egress, and the edge that real-time media takes. The edge itself -- Gateways, listeners, routes, certificates and DNS -- is routing.md; the controls each rule rests on, with their tests, are security/design.md; the mail edge is mail.md. The rules are AD-1, AD-6 and AD-9 in decisions/index.md.

1. The layers​

Each layer answers one question and is enforced by one component. A request passes them top to bottom; a lower layer never re-answers a higher one.

#LayerQuestionEnforced byIdentity it sees
L0TLS, DNS, rate limitis this traffic well-formed and within budget?the Envoy listener; a local limit per client address on sign-in posts, to a realm's sign-in pages and to a sign-in sidecar's answer path (BackendTrafficPolicy); the publishing proxy's own limits per client address on a published entry (security/design.md §2.14)none
L1Edge sessionwho is this, in which realm?Envoy SecurityPolicy.oidc: one confidential client per tenant zone in that tenant's realm, the kernel realm for the platform tenant's zone and the kernel UIs (routing.md §4.1)the realm's token: sub, realm
L1′Perimeterwhich paths, how much, and does a credential pass?the publishing proxy in the tenant's DMZ, a named enforcement point: it forwards the declared paths and nothing else, holds a request to one reading of its path, a method, a size and a time, removes every identity header, Cookie and Set-Cookie, and removes Authorization unless the entry was approved as authMode: app, where the app alone checks it. It verifies no callernone, or the credential the app issued
L2Reachabilitymay this person reach this component at all?the ext-auth bouncer, asking OpenFGA can_use on the app (or the relation the route names) over the stored membership projection (AD-12); cached per (sub, sid, route), evicted on OpenFGA's change logthe same token
L3App session and authorizationwhat may they do inside?the app: the identity headers the bouncer sets, or its own silent single sign-on with its own client and audience and its own cookie. The profile's sessionMaxAge caps how long the realm renews that client's sign-in; it does not end the app's own cookie session. The edge token itself is forwarded only on a route that declares forwardToken; a route that declares exchangeToken is handed a token exchanged for that app alone (§1.1)identity headers, or the app's own session
L5Networkmay these two pods talk at all?NetworkPolicy derived from requires, integrations and granted privileges (§2–§4); Kyverno for what a pod may beServiceAccount, namespace and pod labels

Two things follow. The edge decides reachability, the app decides everything finer: the Gateway never asks "may Alice edit document 4711", so its OpenFGA load scales with logins × apps, not with requests. And a perimeter route skips L1–L2 by declaration: its authMode is the whole of its authentication, which is why the field is mandatory and none is a word somebody wrote and somebody else approved.

1.1 What a backend is handed​

A backend behind the session is told who is asking in the identity headers (x-gentian-subject, -realm, -session, -email, -name), set by the bouncer after it removed what the client sent. It believes them only as far as it believes that nothing but the Gateway's Envoy pods reach it, which is the network rule of §2 and nothing in the request.

The session's own token is handed to no app: it is valid at the director and at every sibling. A route whose entry says forwardToken -- the desktop, which relays to the director -- receives it. A route whose entry says exchangeToken is handed a token of the person issued by the tenant's realm for that app alone: the bouncer exchanges the session's token (RFC 8693) through a client of the realm that exists for nothing else, gentian-edge-exchange, which signs nobody in, has no token of its own and is admitted to the exchange only for a session's token that names it. The bouncer holds that client's secret for each realm (Secret bouncer-exchange); it is the one platform component below the edge that holds a client secret. One exchange serves a session for the token's lifetime, and a request for which no exchanged token can be had is refused, never let through with the session's token or with none (internal/bouncer/exchange.go).

2. Tenant namespaces​

A tenant's namespace is closed by default and opened by declaration. The operator writes every policy of the namespace from the tenant's installs, bindings and grants (internal/kernel/netpolicy, BuildDesired); nobody writes one by hand.

PolicySelectsOpens
tenant-isolation (the baseline)every pod of the namespace, an adapter's exceptedIngress from the edge namespace -- its Envoy pods alone where the kernel's rules are on (§3) -- from the identity provider's namespace, which calls an app's back-channel logout address, and from the operator's namespace, which provisions into running apps. Egress to the cluster's DNS and to the API server. Nothing else in either direction
the export Job's policythe tenant's backup and export Jobswhat a backup of the tenant's stores needs
an adapter's policiesan adapter's pods, which the edge, the identity provider and the operator must not reacha floor of the adapter's own
kernel-access-<app>the app's podsEgress to each store the profile declares, on that store's port, and to the model gateway for requires.services.llm (security/design.md §2.6)
the app's in-app policythe app's podsthe programs of one app reach each other
component-<app>the app's podsthe egress rules of requires.privileges.egress that were granted by name on the install. A declared privilege nobody granted opens nothing
a contract's policiesconsumer and provider of an IntegrationBindingthe path between two apps that share a contract, as far as the AppGrant allows
the cache policiesthe apps that declare a Memcached cacheEgress to the tenant's Memcached instance on its port, and ingress at it. An app that declares Redis uses the shared cache store through kernel-access-<app> instead

Network rules only ever add: a pod the baseline selects cannot be closed again by another policy, which is why an adapter's pods are left out of the baseline instead of being narrowed afterwards.

What follows for an app that believes the identity headers: another app of the same tenant reaches it only where an integration between the two was granted, and a pod of another tenant never. The rule holds only on a cluster whose network plugin enforces NetworkPolicy; the installer refuses a cluster whose plugin does not (§3).

3. The kernel's rules​

Every kernel namespace refuses an ingress that an inventory of callers does not list. The inventory is one file (internal/kernel/kernelnet/inventory.yaml) -- per namespace, every pod set, every port, and for each port who is admitted -- and the policies are generated from it, so no rule exists without a line that says who it is for. OpenFGA admits the programs that ask it; the bouncer's authorization port admits the Envoy proxies alone; the vault and its seal admit their named clients; a tenant's pod reaches the director, the usher, the custodian and the registrar and no other port of the control namespace. The table is security/design.md §2.13.

The rules are on by default: KERNEL_NETWORK_POLICIES, read by the installer, is true when unset, after a check that the network plugin enforces NetworkPolicy. The installer applies them with the namespaces, before a pod exists in any of them. The operator is told the same switch, and the two rules of its own that rest on the same fact follow it: the tenant baseline's edge rule, which then admits the Envoy pods instead of the whole edge namespace, and the publishing proxy's ingress rule, which then admits the Envoy pods and nothing else.

Egress is open in the kernel and system namespaces. Argo CD, Crossplane and its providers, cert-manager, external-dns, Keycloak, the Envoy proxies and the operator reach the internet and the API server in ways that differ per cluster. A pod there is stopped at another namespace's door by that namespace's rules and on the way out by nothing. The exceptions are the edges that face the internet: the mail proxy in system-mail-dmz and the TURN edge in system-turn-dmz (§7), whose egress is restricted to what they relay to.

Plain networking.k8s.io/v1 NetworkPolicy throughout, naming no address block, so the rules mean the same on every plugin that enforces them. What a NetworkPolicy cannot say -- the API server, the kubelet, the install host -- and what stands in its place is in security/design.md §2.13.

4. The shared stores: one rule with two sides​

A tenant's namespace denies egress by default and an app's kernel-access-<app> policy opens the store its profile declares, on that store's port. Each store -- PostgreSQL, MariaDB, Redis, SeaweedFS -- carries the other side, a store-ingress policy on its server's pods: tenant namespaces, the pods of its own namespace (the operator's provisioning and destroy Jobs, and the steps of a backup and a restore that work on that store), and by name the operator, for SeaweedFS the backup and restore pods of the database and identity namespaces and, for PostgreSQL's instance manager, CloudNativePG's operator. The server's side keeps out everything that is not a tenant namespace or one of those; the tenant's side keeps out the apps that declared no such store. Who reaches which port, and what stays closed: security/design.md §2.7.

The kernel's own PostgreSQL carries the same kind of policy: Keycloak's and OpenFGA's namespaces, the registrar and the operator by label, tenant namespaces for the desktop of the tenant that adopts the kernel realm, and its own namespace. So do the two mail servers in system-mail, neither of which faces the internet: 25, 587 and 993 are taken by a TCP proxy in system-mail-dmz that holds no mail, no user, no certificate and no key, and passes each connection on with the client's address in a PROXY-protocol header, on ports of the servers that require the header and admit the proxy alone (security/design.md §2.8, mail.md).

The model gateway is the same rule as a store. An app that declares requires.services.llm is opened system-llm on the gateway's port by its kernel-access-<app> policy; the gateway's own policy admits tenant namespaces, the operator and the Gateway's Envoy pods (its console) on that port, and the gateway's database, cache and mock model server each admit the gateway alone (security/design.md §2.9). An app that did not declare the gateway has no path to it and no key for it.

5. Three levels of exposure​

Nobody writes a route. What is reachable without a session is declared at three levels with different owners, cardinalities and lifetimes, and the operator derives listeners, certificates, routes and DMZ proxies from them. Keeping the levels apart is what makes self-service and control compatible.

LevelObjectDeclared byHow manyLives
The profile's entrya surface: perimeter entry: name, host label, paths, deny paths, authModethe app's author, in the cataloga few per appwith the profile
The tenant's enablementthis entry of this install is published: owner, publish date, reviewAt, an optional expiresAt (routing.md §7). The authMode is not restated: it is the entry's, and an approval cannot weaken itthe tenant's perimeter approver, can_expose, through the directora handful per tenantuntil withdrawn or expired; regular review is the control
The app-level objecta share link, a guest meeting, a public formany user of the app, inside the appthousandsas the app's own policy says

A share never needs an administrator: the approver published share links once, and the app issues them. What the platform never does is learn about individual shares by routing: they are capabilities inside a declared surface, not routes. So the platform's inventory of what can be reached anonymously -- hosts, path prefixes, modes -- is the enablements and the routes derived from them, complete by construction, since nothing is routed without one. Which objects an app has made public is the app's to say.

6. Certificates and DNS​

Every wildcard is issued by DNS-01. ACME issues a wildcard by DNS-01 only. The kernel's *.<kernelDomain> and each tenant's *.<effectiveDomain> -- a domain of the tenant's own included -- are Certificates against the DNS-01 ClusterIssuer letsencrypt-dns01-<provider> (tenant_edge_tls.go, default letsencrypt-dns01-cloudflare). The zone of a tenant served on a domain of its own is listed on the Cluster claim (certificates.dnsZones) so that the DNS credential cert-manager and external-dns hold covers it (routing.md §3a). The HTTP-01 issuer is emitted beside the DNS-01 one, and the perimeter Gateway's port-80 listener answers its challenges at /.well-known/acme-challenge/*: with the realm endpoints it is one of exactly two kernel-owned surfaces without a session.

Two different things travel under the word "wildcard". The certificate's DNS-01 challenge is a TXT record at _acme-challenge.<domain>, written and removed by cert-manager. A wildcard address record is separate and unnecessary: external-dns publishes one record per routed hostname, so every host resolves without one. Only a deployment that chooses to rely on a wildcard address record has to care that a DNS wildcard matches a single label.

external-dns owns records; the operator does not write them. On a static-ip cluster its gateway-httproute source reads hostnames off HTTPRoutes and takes the target from the Gateway's status address, and the operator publishes nothing. On a tunneled cluster the Gateway never gets an address, so the operator publishes a DNSEndpoint naming the hostnames and the tunnel's target and external-dns reconciles that (edge_dnsendpoint.go): not the operator writing DNS, the operator declaring intent for the component that owns it. Mail and the TURN edge publish their load balancers' addresses the same way.

7. Real-time media​

A conference has two parts, and the platform treats them differently. Getting media through NAT is TURN's job, and every conferencing app needs the same thing from it, so it is one system service. Forwarding media between the participants of a call is a selective forwarding unit's, and every app brings its own, so the media server belongs to the app. Media never passes the Gateway: WebRTC goes to the TURN edge over UDP, and the controls there are the credential's lifetime, the app's room membership and the short list of peers the edge relays to.

7.1 The TURN edge​

  • Where. system-turn-dmz, tier system-dmz, with no inner part: TURN holds no data, so the whole service is edge (AD-9), the way the mail edge is in system-mail-dmz. The namespace is composed on demand: the first component that declares requires.services.turn puts the engine turn into the operator's set (system_engines.go), and it stays until a person purges it.
  • What runs. STUNner's daemon, stunnerd, standalone, from a configuration the operator writes (turn_edge.go). There is no STUNner operator, no resource definition of STUNner's and no permission outside the namespace. The configuration is the one thing that names the server software.
  • How it is reached. A LoadBalancer Service of its own, not a listener of the Gateway's, for the reason the mail edge has one: UDP 3478, TCP 3478, and TURN over TLS on TCP 443 for networks that let nothing else out. externalTrafficPolicy: Local keeps the client's address, which STUN reports back to the client. One name for the cluster, turn.<kernelDomain>; the operator publishes it from the Service's address as a DNSEndpoint, and the name's certificate is issued by DNS-01 into the namespace. Apart from the shared secret, that key is the only secret the namespace holds.
  • Where it relays to. The relay is headless: a relayed connection may reach the edge's own pods, which is how two participants behind NAT reach each other through one public port, and nothing else in the cluster. Two things enforce that: the daemon's list of peers, which is the headless Service of its own replicas, and the namespace's policy, whose egress admits the edge's own pods and DNS and nothing of the cluster or the internet.
  • Credentials. Time-limited TURN credentials (the TURN REST scheme): the username is <expiry>:<id>, the password is the HMAC-SHA1 of the username under a shared secret, and the app mints one per session. One secret for the cluster, minted once into the vault and delivered to the edge and to every component that declares the requirement.
  • Tunnel mode. A tunnel carries no UDP and no TURN. A component that declares the requirement on a tunnel cluster is not installed; it waits under SystemEngineNotReady with the reason named.

7.2 The contract​

requires.services.turn names no implementation. A component that declares it receives the Secret turn-credentials-<component> in its namespace (turn_credentials.go):

KeyValue
TURN_SERVERturn.<kernelDomain>
TURN_PORT, TURNS_PORT3478, and 443 for TURN over TLS
TURN_URISturn:…:3478?transport=udp, turn:…:3478?transport=tcp and turns:…:443?transport=tcp, comma-separated
STUN_URIstun:turn.<kernelDomain>:3478
TURN_SECRETthe shared secret

A chart mounts the Secret by that name, so the secret is never a release value. The Secret is removed once nothing the component runs declares the requirement. A requirement an add-on declares is served to the component the add-on activates into, because that is where the add-on runs: the Secret is named after the base. Nextcloud Talk's add-on declares it as turn: {}, and the base mounts turn-credentials-nextcloud-base-ce while the add-on is selected. A credential minted in the strict form above is accepted by STUNner, coturn and the other TURN servers that implement the scheme, so replacing STUNner changes the operator's edge and nothing a profile says. What it does change is the network: a server that needs a range of relay ports needs that range open on the load balancer, or host networking.