Installing IBM Rhapsody Systems Engineering (RSE) Using the Kubernetes Operator
A step-by-step installation and verification guide · Operator-based deployment via OLM · Audience: cluster administrators and platform engineers
Introduction
IBM Rhapsody Systems Engineering (RSE) is distributed
exclusively as a
Kubernetes Operator. You do not deploy RSE by applying individual YAML manifests. Instead, you tell the
Operator Lifecycle Manager (OLM) where to find the RSE operator, and OLM installs and manages everything from that point forward — including future upgrades and self-healing.
This guide walks you through every step in the order they must be performed, explains what each OLM concept actually does, and gives you a concrete verification command after every step so you always know whether to proceed or stop and investigate.
What is an Operator?
A Kubernetes Operator is a piece of software that runs
inside your cluster and acts as an automated human expert for a specific application. You describe the desired state of RSE using a
Custom Resource (CR), and the operator continuously reconciles the actual cluster state to match that desired state — creating pods, configuring secrets, running database migrations, and repairing drift automatically.
Understanding OLM Before You Start
OLM is the framework that packages, delivers, and upgrades operators on a Kubernetes cluster. On
OpenShift, OLM is built in. On vanilla Kubernetes (such as kind or kubeadm), you install OLM once as a cluster-level component. Before running a single command, understand the five objects that form the OLM pipeline.
CatalogSource --> Where does the operator come from? (OCI image index on IBM Container Registry)
| OLM polls every N minutes
v
PackageManifest --> Auto-generated list of operator packages and available version channels
| You choose a channel
v
OperatorGroup --> Which namespace(s) may this operator manage?
|
v
Subscription --> Subscribe to a channel. OLM generates an InstallPlan.
|
v
InstallPlan --> The concrete plan: list of resources to create (CRDs, RBAC, Deployment)
| Approved automatically or manually
v
ClusterServiceVersion (CSV) --> The running operator Deployment + all RBAC + CRD ownership
| CSV phase reaches Succeeded
v
RhapsodySEApp CR --> You apply one CR. The operator creates everything RSE needs.
CatalogSource — the operator's source of truth
A
CatalogSource is a pointer to an OCI image that IBM maintains and publishes to the IBM Container Registry (ICR). That image contains an index of every available version and channel of the RSE operator. OLM runs a small
catalog pod from that image and queries it on a configurable polling interval (typically every 10 minutes). You never pull this image yourself — OLM does it, using the pull secret you configure in the
CatalogSource.
An
OperatorGroup tells OLM which Kubernetes namespaces the operator is allowed to watch and manage. RSE uses a single-namespace scope: the operator runs in the operator namespace and manages resources there. The operand pods (the actual RSE application) are created in the application namespace by the operator after you apply the Custom Resource.
Subscription — channel tracking
A
Subscription ties your cluster to a specific release channel of the RSE operator (for example,
v1.8). Once a Subscription exists, OLM watches the
CatalogSource for new versions on that channel. When IBM releases a new patch version, OLM generates an
InstallPlan automatically. With
installPlanApproval: Automatic, OLM approves and executes the plan without any human action. With
Manual, a cluster administrator must review and approve the
InstallPlan before anything changes — useful for regulated environments.
InstallPlan — the execution manifest
An
InstallPlan is an auto-generated resource that lists every Kubernetes object OLM will create or update to bring the operator to the desired version: CRD definitions, RBAC rules, the operator Deployment, and so on. You can inspect an
InstallPlan before approving it. Once approved, OLM executes it. After execution, the
InstallPlan transitions to
Complete and a new
ClusterServiceVersion becomes active.
The
CSV is the richest OLM object. Think of it as the full specification of the operator at a given version. It bundles together:
- The operator Deployment (what image to run, resource limits, environment variables)
- All RBAC rules the operator needs to manage RSE resources
- The CRD (Custom Resource Definition) for
RhapsodySEApp — the schema of the object you will apply
- Registrations for any admission webhooks the operator uses to validate your CR before accepting it
- Descriptive metadata shown in OperatorHub
When the CSV phase reaches
=Succeeded=, the operator pod is running and ready to reconcile
RhapsodySEApp custom resources.
Prerequisites
Confirm all of the following before starting. Missing any one of them will cause a failure that is harder to diagnose mid-install than it is to resolve before you begin.
| Requirement |
Why it is needed |
| A running Kubernetes cluster |
RSE runs entirely inside Kubernetes. Supported distributions include OpenShift, vanilla Kubernetes (kubeadm, kind), and managed Kubernetes (EKS, AKS, GKE) where you have cluster-admin access. |
| OLM installed on the cluster |
OpenShift includes OLM by default. For vanilla Kubernetes, install OLM using the upstream installation script before proceeding. Verify with kubectl get pods -n olm. |
| IBM Container Registry access |
The RSE operator image and catalog image are hosted on IBM ICR. You need an IBM Entitlement Key from myibm.ibm.com, and an ICR API key for the catalog. Both are used to create pull secrets. |
| A ReadWriteMany (RWX) StorageClass |
RSE creates three shared PersistentVolumeClaims that multiple pods access concurrently. You need a StorageClass backed by NFS, CephFS, OpenShift Data Foundation, or an equivalent RWX provider. Know its name before you start. |
| A PostgreSQL database |
RSE stores all SysML v2 model data in PostgreSQL. The database must be reachable from the cluster, with a known service endpoint, username, password, and (if TLS is enabled) a CA certificate. RSE does not deploy its own database. |
| An OIDC provider |
RSE delegates all user authentication to an external OpenID Connect provider (for example, IBM Jazz Authorization Server, Keycloak, or Microsoft Entra). You need the well-known endpoint URL and a registered OIDC client ID and secret. |
| An IBM FlexNet licence server |
RSE checks out floating licences from an IBM FlexNet server. You need the hostname and ports for the lmgrd daemon and the ibmratl vendor daemon. |
| An Ingress controller |
RSE uses path-based routing through a Kubernetes Ingress. An NGINX Ingress Controller (or equivalent) must be installed and its IngressClass name must be known. |
| TLS secrets |
You need two TLS secrets in the RSE application namespace: one for browser-facing HTTPS (external) and one for pod-to-pod HTTPS (internal). How these are created — cert-manager, your corporate PKI, or manual — is your choice; the secrets must exist before you apply the CR. |
Step 1 — Create the Namespaces
RSE uses two separate Kubernetes namespaces with distinct responsibilities. Creating them explicitly — before any other object — gives you full control over RBAC, resource quotas, and network policies from the start.
The
operator namespace will hold OLM objects (
CatalogSource,
OperatorGroup, Subscription, CSV) and the operator pod itself. The
application namespace will hold all RSE pods,
PersistentVolumeClaims, secrets, and the
PostgreSQL database (if deployed in-cluster).
# Replace <operator-namespace> and <app-namespace> with your chosen names.
# Common convention: rse-operator for the operator, ibm-rse for the application.
kubectl create namespace <operator-namespace>
kubectl create namespace <app-namespace>
Verification
kubectl get namespace <operator-namespace> <app-namespace>
# Expected: both show STATUS Active
Step 2 — Create Image Pull Secrets
The RSE operator and its application images are hosted in IBM's private container registry (
cp.icr.io). The OLM catalog image is hosted in a separate IBM registry (
icr.io/cpopen). Both registries require authentication. Kubernetes pulls these images automatically, but only if the correct pull secrets are present in the right namespaces
before OLM tries to pull.
Two secrets are needed:
- Catalog pull secret in the operator namespace — used by OLM to pull the RSE catalog image from
icr.io/cpopen.
- Application pull secret in the application namespace — used by the operator to pull RSE application images from
cp.icr.io.
# Catalog pull secret (operator namespace)
# Server: icr.io | Username: iamapikey | Password: your ICR API key
kubectl create secret docker-registry <catalog-pull-secret-name> \
--docker-server=icr.io \
--docker-username=iamapikey \
--docker-password="<your-icr-api-key>" \
--namespace=<operator-namespace>
# Application pull secret (application namespace)
# Server: cp.icr.io | Username: cp | Password: your IBM Entitlement Key
kubectl create secret docker-registry <app-pull-secret-name> \
--docker-server=cp.icr.io \
--docker-username=cp \
--docker-password="<your-ibm-entitlement-key>" \
--namespace=<app-namespace>
Verification
kubectl get secret <catalog-pull-secret-name> -n <operator-namespace>
kubectl get secret <app-pull-secret-name> -n <app-namespace>
# Expected: both show TYPE kubernetes.io/dockerconfigjson
The
CatalogSource is the first OLM object you create. It tells OLM where to find all available versions of the RSE operator and instructs OLM to run a local
catalog server pod that answers queries about available packages and channels.
What happens when you apply a CatalogSource
OLM starts a pod in your operator namespace running the catalog image you specify. This pod serves a gRPC API. OLM queries it periodically (on the interval you configure) and creates
PackageManifest objects from the responses. A
PackageManifest is a read-only Kubernetes object — you never create it yourself — that lists which operator packages exist in the catalog and which channels and versions are available within each package.
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: <catalog-source-name>
namespace: <operator-namespace>
spec:
displayName: IBM RSE Operator Catalog
image: <ibm-rse-operator-catalog-image> # provided in your IBM product documentation
publisher: IBM
sourceType: grpc
secrets:
- <catalog-pull-secret-name> # the secret created in Step 2
updateStrategy:
registryPoll:
interval: 10m # how often OLM checks for new operator versions
Verification — wait until the catalog pod is RUNNING and the
PackageManifest appears
# The catalog pod should start in the operator namespace
kubectl get pods -n <operator-namespace>
# Expected: a pod named after your CatalogSource shows 1/1 Running
# PackageManifest confirms OLM can read the catalog
kubectl get packagemanifest -A | grep rhapsody
# Expected: a row showing ibm-rhapsody-system-engineering with a CATALOG column matching your CatalogSource name
If the catalog pod is in ImagePullBackOff
The pull secret name in
spec.secrets is wrong, or the ICR API key is invalid. Check with
kubectl describe pod -n <operator-namespace> <catalog-pod-name> and look at the Events section.
An
OperatorGroup is required in every namespace where you want to run an operator. It defines the set of namespaces the operator is permitted to watch and create resources in. OLM will refuse to install an operator in a namespace that does not have exactly one
OperatorGroup.
Single-namespace vs all-namespace OperatorGroup
Single-namespace (used for RSE):
spec.targetNamespaces lists the operator namespace. The operator's RBAC is tightly scoped to that namespace. RSE then creates its application pods in the
application namespace separately, after you apply the CR.
All-namespace:
spec.targetNamespaces is empty or omitted. The operator gets cluster-wide watch permissions. Do not use this for RSE — the single-namespace scope is the supported and recommended configuration.
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: <operator-group-name>
namespace: <operator-namespace>
spec:
targetNamespaces:
- <operator-namespace> # scope the operator to the operator namespace
Verification
kubectl get operatorgroup -n <operator-namespace>
# Expected: your OperatorGroup appears with AGE column showing it was just created
kubectl describe operatorgroup <operator-group-name> -n <operator-namespace>
# Expected: Status.Namespaces includes <operator-namespace>
Only one OperatorGroup per namespace
If you see an error like
"an OperatorGroup already exists in namespace", someone has already created one. Do not create a second — use
kubectl get operatorgroup -n <operator-namespace> to see what exists, and reuse the existing name in your Subscription (Step 5).
Step 5 — Create the Subscription
The Subscription is the object that actually triggers the operator installation. It tells OLM which package to install, which channel to track, where to find it, and how to handle future upgrades.
Channels and installPlanApproval
A
channel (e.g.,
v1.8) is a named stream of operator versions maintained by IBM. Subscribing to a channel means OLM will automatically offer you every new release IBM publishes to that channel, in order, without skipping versions.
- =installPlanApproval: Automatic= — OLM applies every new version without asking. Suitable for non-production or agile environments.
- =installPlanApproval: Manual= — OLM generates an InstallPlan and pauses. A human must inspect it and set
spec.approved: true before anything changes. Use this for production systems where change control is required.
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: <subscription-name>
namespace: <operator-namespace>
spec:
channel: <channel-name> # e.g. v1.8 — check your IBM product docs for current channel
name: ibm-rhapsody-system-engineering # package name in the catalog — do not change this
source: <catalog-source-name> # must match the CatalogSource name from Step 3
sourceNamespace: <operator-namespace>
installPlanApproval: Automatic # or Manual for controlled environments
After you apply this, OLM immediately creates an
InstallPlan. If approval is
Automatic, OLM also immediately executes it. The execution creates all CRDs, RBAC objects, and the operator Deployment — culminating in a new CSV object.
Verification — check the
InstallPlan and CSV status
# Check the InstallPlan was created and executed
kubectl get installplan -n <operator-namespace>
# Expected: STATUS column shows Complete (if Automatic) or RequiresApproval (if Manual)
# If Manual approval: inspect and approve
kubectl get installplan -n <operator-namespace>
# Note the InstallPlan name, then approve it:
kubectl patch installplan <installplan-name> \
--type merge \
--patch '{"spec":{"approved":true}}' \
-n <operator-namespace>
# Check the Subscription status
kubectl get subscription <subscription-name> -n <operator-namespace>
# Expected: PACKAGE shows ibm-rhapsody-system-engineering, SOURCE shows your catalog
Step 6 — Wait for the CSV to Reach Succeeded
After the
InstallPlan executes, OLM creates a
ClusterServiceVersion. The CSV progresses through several phases:
Pending ->
InstallReady ->
Installing ->
Succeeded. Only when it reaches
Succeeded is the operator pod running and ready to handle your Custom Resource.
What the CSV creates when it reaches Succeeded
- The operator Deployment — a single-replica pod with two containers:
manager (the RSE controller) and kube-rbac-proxy (secures the metrics endpoint).
- All RBAC objects the operator needs: ClusterRoles, Roles, ClusterRoleBindings, RoleBindings.
- The CRD for
RhapsodySEApp — from this point on, the cluster understands this custom resource type.
- A ValidatingWebhookConfiguration that intercepts every
RhapsodySEApp create or update, validates the CR fields, and rejects misconfigured resources with a clear error message before any reconciliation begins.
# Watch CSV status — may take 3-15 minutes on first install
kubectl get csv -n <operator-namespace> --watch
# Expected final state:
# NAME DISPLAY VERSION PHASE
# ibm-rhapsody-system-engineering.vX.Y.Z IBM RSE X.Y.Z Succeeded
Full operator health verification
# CSV must be Succeeded
kubectl get csv -n <operator-namespace>
# Operator pod must be 2/2 Running (manager + kube-rbac-proxy)
kubectl get pods -n <operator-namespace>
# CRD must exist
kubectl get crd rhapsodyseapps.rhapsodyse.ibm.com
# ValidatingWebhookConfiguration must exist
kubectl get validatingwebhookconfiguration | grep rhapsody
CSV stuck in Installing or Failed
- If the operator pod is in
ImagePullBackOff: the application pull secret is missing in the operator namespace, or the image reference in the CSV is incorrect.
- If the CSV shows
Failed with a webhook error: the webhook certificate secret was not injected. This typically means cert-manager (or your certificate tooling) is not reconciling the webhook secret. Restart the operator pod after confirming the TLS secret exists.
- Run
kubectl describe csv <csv-name> -n <operator-namespace> and read the Conditions section for the exact failure reason.
Step 7 — Prepare Secrets in the Application Namespace
The RSE operator reads secrets from the application namespace using a pattern called
secret reference resolution. In the Custom Resource, instead of writing sensitive values directly, you write a reference in the form:
<field>: secret::<secret-name>::<key>
At reconcile time, the operator resolves each reference to the actual value from the named Kubernetes Secret. This means
the secrets must exist before you apply the CR. If a referenced secret or key is missing, the reconciliation fails immediately and the CR status will show a descriptive error.
The following secrets are required in the application namespace:
| Secret name (your choice) |
Required keys |
Purpose |
| Your DB credentials secret |
dbPassword, dbAdminPassword |
PostgreSQL user and admin passwords. Referenced in the CR's database section. |
| Your DB CA cert secret |
ca-certificate.pem |
The CA certificate chain for validating the PostgreSQL TLS connection. The key name must be exactly ca-certificate.pem. |
| Your external TLS secret |
tls.crt, tls.key |
Browser-facing HTTPS certificate for the RSE hostname. Used by the Ingress controller. |
| Your internal TLS secret |
tls.crt, tls.key |
Certificate for pod-to-pod HTTPS within the cluster. SANs must include *.<app-namespace>.svc.cluster.local and localhost. |
# Database credentials
kubectl create secret generic <db-credentials-secret-name> \
--from-literal=dbPassword=<db-user-password> \
--from-literal=dbAdminPassword=<db-admin-password> \
-n <app-namespace>
# PostgreSQL CA certificate
# (Export the CA cert from your PostgreSQL provider and save it as ca-certificate.pem)
kubectl create secret generic <db-ca-cert-secret-name> \
--from-file=ca-certificate.pem=<path-to-your-ca-certificate.pem> \
-n <app-namespace>
# External TLS (browser-facing) — create from your certificate files
kubectl create secret tls <external-tls-secret-name> \
--cert=<path-to-external-tls.crt> \
--key=<path-to-external-tls.key> \
-n <app-namespace>
# Internal TLS (pod-to-pod)
kubectl create secret tls <internal-tls-secret-name> \
--cert=<path-to-internal-tls.crt> \
--key=<path-to-internal-tls.key> \
-n <app-namespace>
Verification — all four secrets must exist before continuing
kubectl get secret \
<db-credentials-secret-name> \
<db-ca-cert-secret-name> \
<external-tls-secret-name> \
<internal-tls-secret-name> \
<app-pull-secret-name> \
-n <app-namespace>
# Expected: all five secrets listed with no "not found" errors
Step 8 — Apply the RhapsodySEApp Custom Resource
With the operator running and all secrets in place, you now apply the single CR that drives the entire RSE deployment. Every field in this CR is a
desired state declaration. The operator reconciles the cluster to match it — and keeps reconciling any time the cluster drifts away from it.
What happens the moment you apply the CR
- The ValidatingWebhook intercepts the API call and checks all required fields. If anything is wrong, the apply is rejected immediately with a clear error message — nothing is created yet.
- On success, the operator's Reconcile() loop is invoked. It reads the CR, resolves all
secret::name::key references, and works through a deterministic sequence of create/patch operations.
Replace every
<placeholder> with your environment-specific values. Do not hardcode any passwords — use the
secret::name::key pattern for all sensitive fields.
apiVersion: rhapsodyse.ibm.com/v1
kind: RhapsodySEApp
metadata:
name: <cr-name> # a name of your choosing; used as prefix for all created resources
namespace: <app-namespace>
spec:
acceptLicense: true # REQUIRED — you must accept the IBM licence
imagePullSecret: <app-pull-secret-name> # pull secret name from Step 2
imageRepo: <rse-image-registry-path> # RSE image registry path — see your IBM product documentation
routing:
hostname: <rse-public-hostname> # the FQDN at which RSE will be accessible
tlsSecret: <external-tls-secret-name> # external TLS secret from Step 7
ingressClassName: <ingress-class-name> # IngressClass name of your ingress controller (e.g. nginx)
internalServiceTlsSecret: <internal-tls-secret-name> # internal TLS secret from Step 7
database:
server: <postgresql-connection-uri> # e.g. postgresql://<host>:5432
name: <database-name>
schema: <database-schema>
user: <db-username>
password: secret::<db-credentials-secret-name>::dbPassword
adminUser: <db-admin-username>
adminPassword: secret::<db-credentials-secret-name>::dbAdminPassword
cacert: secret::<db-ca-cert-secret-name>::ca-certificate.pem
rejectUnauthorizedTls: <true-or-false> # set false only if your DB uses a self-signed cert
oidc:
clientId: <oidc-client-id>
clientSecret: <oidc-client-secret>
signingKey: <random-signing-key-string> # a random string used to sign internal session tokens
customIdentityClaim: sub # the JWT claim used as the RSE username
wellKnownEndpoint: <oidc-well-known-url>
introspectionEndpoint: <oidc-introspection-url>
rejectUnauthorized: <true-or-false> # set false only if your OIDC provider uses a self-signed cert
storage:
storageClassName: <rwx-storage-class-name> # your ReadWriteMany StorageClass name
accessMode: ReadWriteMany
capacity: <storage-capacity> # e.g. 10Gi — applied per PVC
dbMigrateMode: update # run Liquibase on every reconcile — safe, idempotent
serverAdminUsernames:
- <oidc-admin-username> # the RSE super-admin user, must match a claim from your OIDC provider
serviceToken: <service-token-string> # a random token for internal OSLC API authentication
licensing:
server:
host: <licence-server-hostname>
port: <lmgrd-port> # default 27000
id: <licence-server-host-id> # from lmutil lmhostid on the licence server
vendor:
id: ibmratl
port: <ibmratl-vendor-port> # default 27002
linkIndexProvider:
isJASEnabled: <true-or-false>
ldxEndpoint: <ldx-endpoint-url> # only required if integrating with IBM ELM / DOORS Next
Verification — CR accepted by the webhook
# If the webhook rejects the CR, kubectl apply returns an error immediately.
# A successful apply shows:
rhapsodyseapp.rhapsodyse.ibm.com/<cr-name> created
Step 9 — Understand and Monitor the Reconciliation Loop
Once the CR is applied, the operator runs through a fixed sequence of operations. Each step in the reconciliation must succeed before the next begins. If any step fails, the operator sets the CR status to a degraded or failed phase and stops. Knowing the sequence tells you exactly where to look when something goes wrong.
- Resolve secret references. Every
secret::name::key token is resolved to its actual value. A missing secret or wrong key name aborts reconciliation here with a clear status message.
- Create / patch Secrets and ConfigMaps. The operator writes derived internal configuration objects (
rse-settings-config, rse-license-client-config) into the application namespace.
- Create / patch PersistentVolumeClaims. Three RWX PVCs are created using the StorageClass you specified. All three must reach
Bound status before the operator proceeds.
- Create / patch ServiceAccounts and RBAC. The
<cr-name>-rse-deploy ServiceAccount and its bindings are created in the application namespace.
- Run the database migration Job (
rse-db-migrator). Liquibase connects to PostgreSQL, creates the schema if it does not exist, and applies all pending changesets. The operator waits for this Job to complete with exit code 0 before proceeding. If the Job fails, the operator surfaces the error in the CR status.
- Create / patch Deployments. Four application Deployments are created:
rse-server (GraphQL/SysML v2 core), rse-client (Next.js web UI), rse-rest (OSLC REST API), and rse-license (FlexNet licence client).
- Create / patch Services. A ClusterIP Service is created for each Deployment, providing stable internal DNS names.
- Create / patch Ingress. A single Ingress object with path-based routing rules is created:
/ -> rse-client, /_server -> rse-server, /_rest -> rse-rest, /_license -> rse-license.
- Create NetworkPolicy. A NetworkPolicy that denies inbound cross-namespace traffic to RSE pods is applied, while permitting intra-namespace communication between RSE components.
- Update CR status. The operator writes phase and condition fields back to the CR (
phase: Running when all pods are healthy).
# Watch the operator logs — the most direct window into reconciliation progress
kubectl logs -n <operator-namespace> \
-l control-plane=controller-manager \
-c manager --follow
# Watch pods appear in the application namespace
kubectl get pods -n <app-namespace> --watch
# Check CR status phase and conditions
kubectl describe rhapsodyseapp <cr-name> -n <app-namespace>
# Check PVCs — all must reach Bound before Deployments are created
kubectl get pvc -n <app-namespace>
# Check database migration Job — must reach Succeeded
kubectl get job -n <app-namespace>
kubectl logs -n <app-namespace> \
-l job-name=<cr-name>-rse-db-migrator
# Look for: "Liquibase command 'update' was executed successfully"
Step 10 — Verify the Fully Healthy State
Run all of the following checks. Every item must pass before you consider the installation complete and hand the system over to users.
| Component |
Resource |
Expected status |
| Deployments (app namespace) |
<cr-name>-rse-client |
1/1 Running |
| Deployments (app namespace) |
<cr-name>-rse-server |
1/1 Running |
| Deployments (app namespace) |
<cr-name>-rse-rest |
1/1 Running |
| Deployments (app namespace) |
<cr-name>-rse-license |
1/1 Running |
| PVCs (app namespace) |
<cr-name>-shared-storage |
Bound |
| PVCs (app namespace) |
<cr-name>-shared-storage-public-ext |
Bound |
| PVCs (app namespace) |
<cr-name>-artifacts-storage |
Bound |
| OLM (operator namespace) |
CSV phase |
Succeeded |
| OLM (operator namespace) |
Operator pod |
2/2 Running |
| OLM (operator namespace) |
Subscription |
AtLatestKnown |
| Job (app namespace) |
<cr-name>-rse-db-migrator |
Succeeded |
# All RSE pods
kubectl get pods -n <app-namespace>
# All PVCs Bound
kubectl get pvc -n <app-namespace>
# Ingress rule created with correct hostname
kubectl get ingress -n <app-namespace>
# CR phase — expected output: Running
kubectl get rhapsodyseapp <cr-name> -n <app-namespace> \
-o jsonpath='{.status.phase}'
# OLM subscription tracking
kubectl get subscription <subscription-name> -n <operator-namespace>
# Expected: INSTALLEDCSV shows the current version, STATE shows AtLatestKnown
Troubleshooting — Common First-Install Failures
| Symptom |
Diagnostic command |
Cause and fix |
Catalog pod in ImagePullBackOff |
kubectl describe pod -n <operator-namespace> <catalog-pod> |
The catalog pull secret is wrong or missing. Check the secret name in the CatalogSource spec.secrets field matches the secret name created in Step 2, and that the ICR API key is valid. |
No PackageManifest for RSE after 10+ minutes |
=kubectl get packagemanifest -A \ |
grep rhapsody= |
The catalog pod is not healthy. Check its logs: kubectl logs -n <operator-namespace> <catalog-pod>. Also confirm the CatalogSource sourceType: grpc is set. |
InstallPlan stuck in RequiresApproval |
kubectl get installplan -n <operator-namespace> |
Expected when installPlanApproval: Manual. Approve it: kubectl patch installplan <name> --type merge --patch '{"spec":{"approved":true}}' -n <operator-namespace> |
CSV stuck in Installing |
kubectl describe csv <csv-name> -n <operator-namespace> |
Read the Conditions section. Most common cause: the operator image cannot be pulled. Ensure the application pull secret exists in the operator namespace and the entitlement key is valid. |
| CR apply rejected with webhook error |
kubectl describe validatingwebhookconfiguration |
The operator webhook server is not ready. Wait for the operator pod to reach 2/2 Running. If still failing, the webhook TLS secret may not exist — check =kubectl get secret -n <operator-namespace> \ |
grep webhook=. |
PVCs stuck in Pending |
kubectl describe pvc -n <app-namespace> |
The StorageClass name in the CR does not match any existing StorageClass, or the RWX provisioner is not running. Verify with kubectl get storageclass and check the Events on the PVC. |
DB migrator Job in Error / fails to complete |
kubectl logs -n <app-namespace> -l job-name=<cr-name>-rse-db-migrator |
(a) PostgreSQL not reachable — check network and connection string. (b) Wrong CA cert — the ca-certificate.pem key does not match the actual CA used by the DB server. (c) pg_hba.conf uses cert auth — RSE uses password (scram-sha-256) over TLS, not client certificates. |
rse-server in CrashLoopBackOff |
kubectl logs -n <app-namespace> -l app=<cr-name>-rse-server |
(a) DB migrator has not yet succeeded. (b) OIDC well-known endpoint unreachable from inside the cluster. (c) Internal TLS secret missing or has incorrect SANs. |
rse-client readiness probe failing |
kubectl describe pod -n <app-namespace> <rse-client-pod> |
rse-client has an initialDelaySeconds: 60 readiness probe — wait at least 90 seconds. If still failing after 2 minutes: confirm the external TLS secret exists and the hostname in routing.hostname matches the certificate's Subject Alternative Names. |
| Ingress returns 404 or times out |
kubectl get ingress -n <app-namespace> -o yaml |
The ingressClassName in the CR does not match the IngressClass installed on your cluster. Run kubectl get ingressclass to find the correct name and update the CR. |
rse-license in CrashLoopBackOff |
kubectl logs -n <app-namespace> -l app=<cr-name>-rse-license |
Network connectivity issue to the FlexNet licence server. Test from inside the pod: kubectl exec -n <app-namespace> <rse-license-pod> -- nc -zv <licence-server-host> <port>. Also verify the id field matches the licence server host ID (run lmutil lmhostid on the licence server). |
| Reconciliation never progresses; CR status stays blank |
kubectl logs -n <operator-namespace> -l control-plane=controller-manager -c manager |
The operator is not watching the application namespace. Check the OperatorGroup's targetNamespaces includes the operator namespace, and that the CRD exists: kubectl get crd rhapsodyseapps.rhapsodyse.ibm.com. |
How Upgrades Work After Installation
Once installed, the RSE operator manages its own upgrades through OLM. You do not need to re-run any of the steps above to apply a new patch version of RSE.
When IBM publishes a new operator bundle to the catalog image, the catalog pod in your cluster picks it up within the polling interval (default: 10 minutes). OLM generates a new
InstallPlan for the next version in the channel. If your Subscription uses
Automatic approval, the new operator version rolls out within those 10 minutes. Once the new CSV reaches
Succeeded, the operator reconciles all existing
RhapsodySEApp CRs, applies the new application images, and triggers a fresh database migration Job.
To move to a new
major channel (for example, from
v1.8 to
v1.9), patch the Subscription's channel field:
kubectl patch subscription <subscription-name> \
--patch '{"spec":{"channel":"<new-channel>"}}' \
--type=merge \
-n <operator-namespace>
To switch to Manual approval for controlled upgrades:
kubectl patch subscription <subscription-name> \
--patch '{"spec":{"installPlanApproval":"Manual"}}' \
--type=merge \
-n <operator-namespace>
# After OLM generates the new InstallPlan, inspect it:
kubectl describe installplan <installplan-name> -n <operator-namespace>
# When ready to proceed:
kubectl patch installplan <installplan-name> \
--type merge \
--patch '{"spec":{"approved":true}}' \
-n <operator-namespace>
For full TWiki text formatting rules, see
TWiki Text Formatting Rules.