Multiple Environments on One Kubernetes Cluster: What a Namespace Suffix Doesn't Reach

One client needed three lower environments, dev, qa1 and qa2, to run on the same cluster, on AWS (EKS) and on GCP (GKE). Its services ran in shared namespace names such as backend and cronjobs, and internal service hostnames had the backend namespace written into them, which breaks as soon as two environments share a cluster.

The plan was suffixed namespaces rendered by the Helm charts and an ingress controller per environment. Two things a suffix does not reach took separate work: hostnames inside the application, and the ingress pieces that apply to the whole cluster. A chart review on the way turned up a PodDisruptionBudget guarding zero pods. Whether a second cluster would be worth it instead is its own question, and our post on multi-cluster Kubernetes failover covers what one costs. This post assumes you have decided to share.

Shared namespace names and hardcoded hostnames

Our engineer proposed suffixed namespaces for the lower environments: backend-dev, cronjobs-dev, backend-qa1, backend-qa2. That settles where each environment's objects land.

It does nothing for code that names a namespace itself. An internal hostname with backend in it keeps pointing at backend after the service it was written for has moved to backend-qa1. On a cluster where some environment still runs in the plain backend namespace, that call can reach the wrong environment and still succeed, which is harder to spot than a failed lookup.

We filed the hardcoded hostnames as an application change, and we have no record of it shipping yet. For a team in the same position, we would have the application read the target hostname from per-environment configuration, so the deploy that already knows the suffix can supply it.

Render the suffix in the chart

One option was to inject the suffix at deploy time through the GitHub workflows. Our engineer pushed back on it, since any version change could break the replacement. The suffix went into the charts instead, and every namespace is now rendered through it.

The two ways to roll it out had different costs, and we set them out before anyone chose. Suffixing only where proposed touched the GKE dev environment and nothing else. Making the suffix the default for all three lower environments would have forced changes and app redeploys in the GKE dev environment and in the three AWS lower environments as well.

The helper that builds the name skips an empty suffix, so an environment with no suffix set renders the plain backend and cronjobs names. We wrote down why it skips, because a helper that quietly returns the base name looks like a bug to the next reader. A minimal version of the pattern looks like this:

{{/* Illustrative only: every acme.* name is invented for this post */}}
{{- define "acme.ns" -}}
{{- if .suffix -}}
{{- printf "%s-%s" .base .suffix -}}
{{- else -}}
{{- .base -}}
{{- end -}}
{{- end -}}

# in a workload template
metadata:
  namespace: {{ include "acme.ns" (dict "base" "backend" "suffix" .Values.acme.envSuffix) }}

A PodDisruptionBudget with nothing behind it

We approved the team's chart change with one follow-up. The chart's PDB template was not gated the way its scaling template was, so on GCP a workload meant for AWS only rendered a PodDisruptionBudget with minAvailable: 1 over 0 pods.

The PodDisruptionBudget API reference says what that budget does from then on. The disruption controller sets its DisruptionAllowed condition to False with reason InsufficientPods when "the number of pods are either at or below the number required by the PodDisruptionBudget." Zero pods is below one, so the budget stays at DisruptionAllowed=False for good.

There is no pod for it to block, so it does no harm. It still shows up as noise in Wiz and kube-linter, and that alone makes it worth fixing: a finding that never clears trains people to skip that rule, and the next PDB finding could be a real one. Our Kubernetes production readiness checklist asks for a PDB on every production service; this is the opposite case, where the chart renders one for a workload that is not there.

The follow-up puts the PDB template behind the same condition as the scaling template, so the two render together or not at all:

{{/* Illustrative only: every acme.* name is invented for this post */}}
{{- if .Values.acme.scaling.enabled }}
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: acme-api
  namespace: {{ include "acme.ns" (dict "base" "backend" "suffix" .Values.acme.envSuffix) }}
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: acme-api
{{- end }}

To find the same leftovers on a running cluster, list every budget with the number of pods it counts. A row with 0 under EXPECTED is a budget over nothing:

kubectl get pdb --all-namespaces \
  -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,EXPECTED:.status.expectedPods,ALLOWED:.status.disruptionsAllowed

An ingress controller for each environment

Each environment then got its own IngressClass and its own namespace for the ingress controller: one controller per environment instead of one per cluster. Each ingress-nginx admission webhook was scoped to its own namespaces.

Both settings exist in the ingress-nginx chart's values file. controller.ingressClassResource.controllerValue is the value a controller uses to find the IngressClasses it should reconcile, and the chart passes the same value to the controller as --controller-class. With a distinct class name and controller value per environment, one controller does not pick up another environment's Ingresses. The same file notes that IngressClasses cannot be changed after creation, so the names are worth choosing before the first install.

The chart ships the webhook with failurePolicy: Fail and an empty namespaceSelector. In the ValidatingWebhookConfiguration reference, an empty selector "matches everything", and Fail "means that an error calling the webhook causes the admission to fail." Three controllers on those defaults put every environment's webhook in the path of every Ingress change on the cluster, so a qa2 webhook that cannot be reached can block an Ingress change in dev. A selector per environment keeps each webhook to its own namespaces:

# Example values for the qa1 controller. Keys are from the chart's values.yaml;
# the class name and controller value are placeholders.
controller:
  ingressClass: nginx-qa1
  ingressClassResource:
    name: nginx-qa1
    controllerValue: k8s.io/ingress-nginx-qa1
  admissionWebhooks:
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: In
          values:
            - backend-qa1   # list every namespace that holds qa1 Ingresses

The controller itself has a support problem that the split does not change. Kubernetes SIG Network and the Security Response Committee retired Ingress NGINX: best-effort maintenance ran until March 2026, and "afterward, there will be no further releases, no bugfixes, and no updates to resolve any security vulnerabilities that may be discovered." Existing deployments keep working, and the chart's repository has been read-only since it was archived on 24 March 2026. One controller per environment means more installs to move later. We would keep the same per-environment layout on the replacement, and our ingress-nginx retirement migration guide compares the options.

Secrets and data hostnames on GCP

Two changes applied only to the GCP side. GCP deploys now read secrets from the target environment's AWS account, and they carry the namespace suffix into the Postgres and Redis hostnames.

Those hostnames are set by the deploy, so the deploy change could carry the suffix. The service hostnames from the first section sit in application code, and that is why they need an application change. On a shared cluster, every string that names a namespace needs the suffix, including the ones in code.

What was merged, and what is still open

The per-environment ingress controllers were merged, with qa2 left out for now and the suffix removed for dev. The chart change was approved with the PDB follow-up. The application hostname change is the open item: we have no record of it shipping.

Before suffixing a single namespace on a cluster you plan to share, we would search the application code and its config for namespace names written into hostnames, because those are the calls a chart change cannot fix. If you are about to put more environments on one cluster, send us the layout and that search is where we would start.