Connect a private Kubernetes cluster to Operate

LOHA Operate lets a workspace see a small, read-only inventory of the Kubernetes workloads that its cluster administrator explicitly approves. During the pilot, the supported profile is metadata-minimal: Deployments, ReplicaSets, StatefulSets, Pods, Jobs, and CronJobs in named namespaces. It does not collect logs or Events, and it does not make changes to the cluster.

The agent makes an outbound HTTPS connection to the LOHA operations gateway. LOHA never opens an inbound connection to your cluster, and the setup never asks you to upload a kubeconfig, a Kubernetes token, a private key, or a proxy password.

The workspace administrator creates and confirms the LOHA connection. The cluster administrator prepares the cluster, reviews the rendered RBAC, and installs the agent. Keep both browser windows open: the enrollment token is displayed once and cannot be recovered.

Prerequisites

Use this guide only when Settings → Operate says that installation is available. That check confirms that LOHA has a reachable gateway, public CA bundle, immutable chart version and checksum, image repository and digest, and protocol version. If it says that installation is unavailable, use Retry later or ask the platform administrator responsible for enabling Operate. Do not substitute a hostname, chart, image, CA bundle, or latest tag from another environment.

This pilot supports the following observed qualification. It is not an upgrade mandate. The installer does not automatically detect your cluster, and it never uploads a kubeconfig or token.

Component	Verified support for this release
Kubernetes server	1.34.x only (>=1.34.0 <1.35.0), qualified on UAT v1.34.8+rke2r2. Kubernetes 1.25.x and 1.35+ are outside this release's verified range. Stop installation outside this range; use a supported pilot cluster or request compatibility validation.
Helm	3.19.0 minimum for Kubernetes 1.34; 3.19.x is recommended. The upstream Helm version-skew matrix says 3.19.x supports Kubernetes 1.34.x–1.31.x; that upstream range does not broaden LOHA agent support beyond Kubernetes 1.34.x. Another Helm 3 release must explicitly support Kubernetes 1.34; Helm 4 is not qualified in this flow.
kubectl	1.33.x–1.35.x with Kubernetes server 1.34; matching 1.34.x is recommended. The minimum is 1.33.0, subject to HA server skew in the Kubernetes version-skew policy. A kubectl 1.35 client does not make a Kubernetes 1.35 server supported.
Rancher / RKE2	The agent does not require Rancher and LOHA has no separate qualified Rancher minimum. Check the actual downstream Kubernetes version and vendor-supported combination. Rancher 2.7.4 does not qualify Kubernetes 1.25.12.
Installer shell	Bash or Zsh with helm, kubectl, mktemp, and sha256sum or shasum. No numerical shell minimum is required.

Requirement	What to verify
Supported profile	metadata-minimal with one or more explicit namespace names. Events, logs, actions, production access, wildcards, and cluster-wide inventory are outside this installation flow.
Cluster tools	The cluster administrator has kubectl and Helm 3, and can use a context for the intended non-production pilot cluster.
Namespace scope	Every approved namespace already exists. Record each namespace UID before installing; recreating a namespace with the same name but a new UID is intentionally quarantined.
Network	The agent can resolve the gateway hostname and reach it over outbound HTTPS/443. If your network requires a CONNECT proxy, the cluster administrator creates a Secret reference; its URL and credentials never belong in Helm values or commands.
Registry	Nodes can pull the digest-pinned agent image. If the registry needs credentials, arrange the cluster's approved image-pull mechanism separately; do not place registry credentials in this guide's values file.
Trust and storage	Save the public gateway CA bundle supplied by LOHA and provide encrypted persistent storage for the agent identity, either a StorageClass or an existing PVC.

Before any mutating command, check the active context and server version as well as the client version. These checks are read-only:

kubectl config current-context
kubectl version --output=yaml
helm version --short

The first command must name the intended cluster. Confirm the Kubernetes server is inside the verified range, then check namespace names with kubectl get namespace. Stop if any result is surprising.

Operate connections and Delivery bindings are separate. A connection can become active and report inventory while a project still cannot bind a workload. Binding also requires the platform's Delivery/Targets feature and a Delivery binding whose managedWorkload.namespace is explicit.

Prepare scope and create the connection

1. In Settings → Operate, choose Connect private cluster.
2. Give the connection a recognizable name, such as Payments staging, and enter only the namespaces the agent may read. The page removes duplicate names and rejects invalid Kubernetes namespace names.
3. Keep the pilot profile as metadata-minimal. It grants inventory only.
4. You may collect each namespace UID now, before creating the connection. This is read-only preparation; keep each result paired with its namespace name. Replace only the clearly marked local namespace name in the following command, once for every namespace selected in the UI:

kubectl get namespace 'YOUR_APPROVED_NAMESPACE' -o jsonpath='{.metadata.uid}'
printf '\n'

YOUR_APPROVED_NAMESPACE is your own approved namespace name, not a release coordinate. The wizard shows the UID fields only after the connection has been created, so retain each name-to-UID pair for the next step.

5. Create the connection. LOHA then shows its connection ID, the installation profile, a one-time enrollment token, and a UID field for each approved namespace. Enter and verify the collected UIDs there before the wizard generates values.yaml, render commands, or Helm install commands. Copy the resulting non-secret values.yaml into a local file. It contains the connection ID, the exact namespace UIDs, the public gateway URL and CA ConfigMap name, and an immutable image digest; it never contains the enrollment token.

Do not put the enrollment token in a ticket, chat, shell history, Helm values, a Pod environment variable, a Kubernetes Secret, a file, or a support request.

Prepare the cluster

The commands in the installation panel are generated from the available installation profile. Use those generated commands for the chart location, version, checksum, image digest, and gateway URL. Public documentation deliberately does not publish invented or environment-specific release coordinates.

Create the agent namespace and the public gateway CA ConfigMap before rendering the chart. Save the CA bundle shown by LOHA as a local, protected file and use its actual local path in the command below. PATH_TO_LOHA_PUBLIC_CA_BUNDLE is the only local-file placeholder in this example; it must contain public certificate material supplied by the installation panel, never a private key.

kubectl create namespace loha-operate-system --dry-run=client -o yaml | kubectl apply -f -
kubectl -n loha-operate-system create configmap loha-operate-gateway-ca \
  --from-file=ca.crt='PATH_TO_LOHA_PUBLIC_CA_BUNDLE' \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl -n loha-operate-system get configmap loha-operate-gateway-ca

The final command must show the ConfigMap. It is a public trust bundle; do not edit it into a client certificate or a secret.

Identity storage and optional proxy

The agent creates its private key and stores its quarantine and active certificates on /var/lib/loha/identity. Keep the default persistent volume enabled and select an encrypted StorageClass where policy requires it. To use a pre-provisioned customer volume, set only identity.persistence.existingClaim in the copied values file. Do not use emptyDir for a durable installation: a lost identity volume requires revoke and re-enrollment.

The copied values.yaml is the source of truth. Review these fields; do not replace values that Settings generated from the installation profile.

Field	Source or permitted choice	Notes
image.repository	Settings installation profile	Repository only. Do not add a tag or digest here.
image.digest	Settings installation profile	The immutable sha256: image digest. The chart refuses to render without it.
gateway.url	Settings installation profile	Credential-free HTTPS gateway URL.
gateway.caConfigMap	loha-operate-gateway-ca	The ConfigMap created from the public CA bundle.
connectionID	Settings connection detail	Identifies this one connection; it is not a credential.
allowedNamespaces	The approved namespace list	Every entry must be explicit; no wildcard or empty list.
namespaceUIDs	UIDs read from the live cluster	Comma-separated namespace=UID pairs must match allowedNamespaces.
identity.persistence.enabled	true	Keep identity durable.
identity.persistence.storageClass	Your approved local StorageClass	Optional only when the cluster default is suitable.
identity.persistence.existingClaim	Your approved local PVC name	Use instead of creating a chart-managed PVC; do not set both storage strategies.
proxy.existingSecret	Your approved local Secret name	Optional reference only; never a proxy URL or credential.
certificate.renewBefore and resources	Chart defaults unless policy requires a reviewed change	Do not change them to work around TLS or resource failures.

If outbound traffic must use a CONNECT proxy, the cluster administrator creates a Secret in loha-operate-system with HTTPS_PROXY and, optionally, NO_PROXY. The values file may name the Secret and keys only:

proxy:
  existingSecret: your-approved-proxy-reference
  httpsProxyKey: HTTPS_PROXY
  noProxyKey: NO_PROXY

your-approved-proxy-reference is the name of a Secret already created by your cluster administrator. Never put a proxy URL or plaintext credential in the values file, Helm command, command history, or support request.

Verify and install the agent

Use the generated Verify chart, Render, and Install cards from Settings in this order. They carry the actual immutable values for your connection. The verified archive flow is:

set -eu
CHART_DIR="$(mktemp -d)"
trap 'rm -rf "$CHART_DIR"' EXIT
# Run the generated `helm pull ... --version ... --destination "$CHART_DIR"` command here.
# Run the generated expected SHA-256 comparison here; it must print no mismatch.
# `ARCHIVE` is the single .tgz archive accepted by that generated verification command.

The generated command uses helm pull with the profile's exact OCI location and version, finds exactly one .tgz archive, then compares its computed sha256: digest with the profile checksum. A mismatch is a stop condition: discard the archive and obtain a newly generated installation profile. Do not bypass the check, retag an image, or replace the digest with latest.

The current profile verifies the chart with its SHA-256 checksum and the agent image with its immutable digest. If a future profile supplies a release signature and verification instruction, use that exact release-provided instruction as an additional check. Do not invent a signature URL, public key, or verification command.

After verification, use the generated local archive for both commands:

# Generated card: helm template loha-operate-agent "$ARCHIVE" \
#   --namespace loha-operate-system --values loha-operate-values.yaml
# Generated card: helm upgrade --install loha-operate-agent "$ARCHIVE" \
#   --namespace loha-operate-system --create-namespace --values loha-operate-values.yaml

These are intentionally comments, not copyable release placeholders. Copy the corresponding cards from Settings, where $ARCHIVE has already been verified in the same shell. The chart names the Deployment exactly loha-operate-agent-loha-operate-agent in the loha-operate-system namespace.

Review the rendered manifest before installation. The approved surface is namespaced get, list, and watch on Deployments, ReplicaSets, StatefulSets, Pods, Jobs, and CronJobs in the approved namespaces, plus cluster-scoped get namespaces restricted by those namespace names. It should also show one non-root Deployment, a projected service-account token, the identity PVC, the gateway CA ConfigMap, and an optional proxy Secret reference.

Reject a rendered manifest that grants access to Secrets, ConfigMaps, Nodes, namespace listing, Events, logs, exec, attach, port-forward, write verbs, impersonation, wildcard RBAC, cluster-admin, privileged containers, host networking, or host paths. Do not install it until the difference is understood.

Wait for the Pod and verify the expected Deployment exists:

kubectl -n loha-operate-system rollout status deployment/loha-operate-agent-loha-operate-agent
kubectl -n loha-operate-system get pvc

A Ready Pod means only that Kubernetes started the process. It does not prove that LOHA has enrolled, activated, received a heartbeat, or received an observation.

Enroll and confirm the identity

Keep the LOHA confirmation page open. When the Deployment is ready, enter the one-time token without echoing it. This Bash and Zsh command reads it from the terminal, sends it over stdin, and removes the shell variable afterward:

read -rs LOHA_ENROLL_TOKEN
printf '%s' "$LOHA_ENROLL_TOKEN" | kubectl -n loha-operate-system exec -i \
  deployment/loha-operate-agent-loha-operate-agent -- /operate-agent enroll
unset LOHA_ENROLL_TOKEN

The helper prints an identity ID and a 64-character lowercase hexadecimal fingerprint, then reports that enrollment is quarantined while it waits for confirmation. Copy the fingerprint only into the LOHA confirmation form. Confirm every namespace UID again from the values you read from the cluster. LOHA does not auto-confirm the identity.

After confirmation, keep the helper running until it reports activation. It exchanges the quarantine certificate for an active certificate and then the running agent claims its session. If a network interruption happens after the token was consumed and the identity volume is intact, resume without reading or creating another token:

kubectl -n loha-operate-system exec \
  deployment/loha-operate-agent-loha-operate-agent -- /operate-agent enroll

This resume command uses the quarantined identity already stored on the PVC. Do not delete the PVC or identity directory to solve a network problem. If the token expired before first use, was already consumed before a local identity existed, or the identity volume was permanently lost, revoke the pending connection in LOHA and create a new connection. Never reuse a token.

Verify health, inventory, and a runtime binding

In LOHA, wait through the onboarding stages: awaiting enrollment, awaiting confirmation, awaiting first heartbeat, awaiting observations, then active with observations. Check both Last seen and the observation freshness; an Active connection with no observation is not proof that workloads are available.

The cluster administrator can verify the expected least-privilege boundary. Replace YOUR_APPROVED_NAMESPACE only with one selected namespace:

kubectl auth can-i --as=system:serviceaccount:loha-operate-system:loha-operate-agent-loha-operate-agent \
  list secrets -n 'YOUR_APPROVED_NAMESPACE'
kubectl auth can-i --as=system:serviceaccount:loha-operate-system:loha-operate-agent-loha-operate-agent \
  list deployments -n 'YOUR_APPROVED_NAMESPACE'
kubectl auth can-i --as=system:serviceaccount:loha-operate-system:loha-operate-agent-loha-operate-agent \
  get namespace/'YOUR_APPROVED_NAMESPACE'

The expected results are no, yes, and yes. A namespace list must remain no.

To bind a Delivery target, first ensure Delivery/Targets is enabled for the workspace and create a Delivery binding with an explicit managedWorkload.namespace. Go to the project's Operate view and choose the exact observed workload UID. A name alone is not enough: the UID prevents a recreated workload from receiving the old binding. If there are no observed workloads, resolve heartbeat, RBAC, namespace UID, or freshness issues first.

Maintain, revoke, upgrade, and offboard

The agent renews its active client certificate before expiry using the identity stored on the PVC. If renewal fails until expiry, it fails closed; do not disable certificate verification or copy the identity files elsewhere.

For an upgrade, obtain a new available installation profile, verify its new chart archive and image digest, render it again, then run the generated helm upgrade --install card with the existing values and PVC. Start with one non-production connection. A rollback must retain the identity PVC and use an additive-compatible chart/application version; never roll back by deleting identity state.

To revoke access, revoke the connection in LOHA first. Then remove the namespaced RoleBindings and namespace-identity grant by uninstalling the Helm release, close the agent's egress route, and delete the identity PVC only when your retention policy permits it. Finally, run kubectl auth can-i again and record that the former ServiceAccount no longer has the approved namespace reads. A lost PVC or a revoked certificate requires a new, confirmed enrollment.

Troubleshooting

Symptom	Check and corrective action	Expected result
Installation is unavailable or the Operate feature is absent	Use Retry. Ask the platform administrator to check the available installation profile and feature enablement. Do not invent a gateway or release value.	Settings eventually shows an available profile, or the responsible administrator supplies the next action.
Chart pull or checksum fails	Use the generated helm pull command again and compare the archive to the generated sha256: checksum. Check approved registry access.	One archive matches exactly; only then render/install it.
Image cannot pull	Check the node's approved registry access and the digest in the copied values file. Configure the cluster's approved image-pull reference outside the agent values.	The Pod pulls the exact digest.
PVC is Pending	Inspect the PVC and StorageClass with kubectl -n loha-operate-system describe pvc. Correct capacity, access mode, or the selected approved StorageClass.	The identity PVC is Bound before relying on enrollment durability.
Gateway CA ConfigMap is missing	Run kubectl -n loha-operate-system get configmap loha-operate-gateway-ca and recreate it from the public bundle shown by Settings.	The Deployment mounts ca.crt; TLS validation remains enabled.
DNS, egress, proxy, or TLS fails	Check DNS resolution and the approved HTTPS/443 route from the cluster. Check only the proxy Secret reference and keys, then verify the CA bundle is the current public bundle from LOHA.	The agent reaches the gateway with normal certificate verification. Never use a TLS skip-verify option.
Forbidden or missing workloads	Render the same chart values and compare Roles/RoleBindings with the approved namespaces. Use kubectl auth can-i for the exact resource and namespace.	Only inventory reads in named namespaces are allowed; do not widen RBAC, use wildcards, or grant cluster-admin.
Namespace UID mismatch	Read the live UID again. A recreated namespace must be reviewed and confirmed as a new scope; do not edit an old pairing blindly.	The connection is re-confirmed with the intentional namespace identity.
Token expired or consumed	Do not retry it. If no usable quarantined identity is on the PVC, revoke the pending connection and create a new one.	A new one-time token is used once for a new connection.
Quarantine remains pending	Compare the displayed fingerprint and every namespace UID in LOHA, then have the workspace administrator confirm them. If interrupted after token use, run the resume command.	The helper reports activation and the UI advances to heartbeat.
Certificate revoked, expired, or activation aborts	Check the connection status in LOHA. Preserve the PVC for a transient outage; for revoke, expiry, or permanent loss, create a new confirmed enrollment.	The old identity remains fenced; the replacement has a new identity.
Active but no heartbeat, observation, or fresh data	Check Last seen, gateway reachability, namespace UID pairing, and inventory RBAC. Wait for the first full observation after a reconnect.	LOHA shows a recent heartbeat and observations with a current freshness state.
Delivery binding is empty	Confirm Delivery/Targets is enabled and the binding has an explicit managed workload namespace, then wait for an observed workload UID.	The project can select an exact observed workload UID.

Do not send kubeconfigs, enrollment tokens, proxy credentials, private keys, identity-volume contents, or plaintext Secret values to support. Share only redacted diagnostic information that your organization permits, such as a timestamp, the public connection ID, the stage shown in LOHA, and whether a safe command returned an expected result.