TWiki > Deployment Web > DeploymentInstallingUpgradingAndMigrating > RhapsodySE > InstallingIBMRhapsodySystemsEngineeringViaKubernetesOperator
Revision 1 - 2026-09-23 - 05:56:57 - ArtaballavChaudhury

Installing IBM Rhapsody Systems Engineering (RSE) Using the Kubernetes Operator

A step-by-step installation and verification guide · Operator-based deployment via OLM · Audience: cluster administrators and platform engineers

Introduction

IBM Rhapsody Systems Engineering (RSE) is distributed exclusively as a Kubernetes Operator. You do not deploy RSE by applying individual YAML manifests. Instead, you tell the Operator Lifecycle Manager (OLM) where to find the RSE operator, and OLM installs and manages everything from that point forward — including future upgrades and self-healing.

This guide walks you through every step in the order they must be performed, explains what each OLM concept actually does, and gives you a concrete verification command after every step so you always know whether to proceed or stop and investigate.

What is an Operator?

A Kubernetes Operator is a piece of software that runs inside your cluster and acts as an automated human expert for a specific application. You describe the desired state of RSE using a Custom Resource (CR), and the operator continuously reconciles the actual cluster state to match that desired state — creating pods, configuring secrets, running database migrations, and repairing drift automatically.

Understanding OLM Before You Start

OLM is the framework that packages, delivers, and upgrades operators on a Kubernetes cluster. On OpenShift, OLM is built in. On vanilla Kubernetes (such as kind or kubeadm), you install OLM once as a cluster-level component. Before running a single command, understand the five objects that form the OLM pipeline.

CatalogSource       -->  Where does the operator come from? (OCI image index on IBM Container Registry)
    | OLM polls every N minutes
    v
PackageManifest     -->  Auto-generated list of operator packages and available version channels
    | You choose a channel
    v
OperatorGroup       -->  Which namespace(s) may this operator manage?
    |
    v
Subscription        -->  Subscribe to a channel. OLM generates an InstallPlan.
    |
    v
InstallPlan         -->  The concrete plan: list of resources to create (CRDs, RBAC, Deployment)
    | Approved automatically or manually
    v
ClusterServiceVersion (CSV)  -->  The running operator Deployment + all RBAC + CRD ownership
    | CSV phase reaches Succeeded
    v
RhapsodySEApp CR    -->  You apply one CR. The operator creates everything RSE needs.

CatalogSource — the operator's source of truth

A CatalogSource is a pointer to an OCI image that IBM maintains and publishes to the IBM Container Registry (ICR). That image contains an index of every available version and channel of the RSE operator. OLM runs a small catalog pod from that image and queries it on a configurable polling interval (typically every 10 minutes). You never pull this image yourself — OLM does it, using the pull secret you configure in the CatalogSource.

OperatorGroup — namespace scope

An OperatorGroup tells OLM which Kubernetes namespaces the operator is allowed to watch and manage. RSE uses a single-namespace scope: the operator runs in the operator namespace and manages resources there. The operand pods (the actual RSE application) are created in the application namespace by the operator after you apply the Custom Resource.

Subscription — channel tracking

A Subscription ties your cluster to a specific release channel of the RSE operator (for example, v1.8). Once a Subscription exists, OLM watches the CatalogSource for new versions on that channel. When IBM releases a new patch version, OLM generates an InstallPlan automatically. With installPlanApproval: Automatic, OLM approves and executes the plan without any human action. With Manual, a cluster administrator must review and approve the InstallPlan before anything changes — useful for regulated environments.

InstallPlan — the execution manifest

An InstallPlan is an auto-generated resource that lists every Kubernetes object OLM will create or update to bring the operator to the desired version: CRD definitions, RBAC rules, the operator Deployment, and so on. You can inspect an InstallPlan before approving it. Once approved, OLM executes it. After execution, the InstallPlan transitions to Complete and a new ClusterServiceVersion becomes active.

ClusterServiceVersion (CSV) — the operator itself

The CSV is the richest OLM object. Think of it as the full specification of the operator at a given version. It bundles together:

  • The operator Deployment (what image to run, resource limits, environment variables)
  • All RBAC rules the operator needs to manage RSE resources
  • The CRD (Custom Resource Definition) for RhapsodySEApp — the schema of the object you will apply
  • Registrations for any admission webhooks the operator uses to validate your CR before accepting it
  • Descriptive metadata shown in OperatorHub

When the CSV phase reaches =Succeeded=, the operator pod is running and ready to reconcile RhapsodySEApp custom resources.

Prerequisites

Confirm all of the following before starting. Missing any one of them will cause a failure that is harder to diagnose mid-install than it is to resolve before you begin.

Requirement Why it is needed
A running Kubernetes cluster RSE runs entirely inside Kubernetes. Supported distributions include OpenShift, vanilla Kubernetes (kubeadm, kind), and managed Kubernetes (EKS, AKS, GKE) where you have cluster-admin access.
OLM installed on the cluster OpenShift includes OLM by default. For vanilla Kubernetes, install OLM using the upstream installation script before proceeding. Verify with kubectl get pods -n olm.
IBM Container Registry access The RSE operator image and catalog image are hosted on IBM ICR. You need an IBM Entitlement Key from myibm.ibm.com, and an ICR API key for the catalog. Both are used to create pull secrets.
A ReadWriteMany (RWX) StorageClass RSE creates three shared PersistentVolumeClaims that multiple pods access concurrently. You need a StorageClass backed by NFS, CephFS, OpenShift Data Foundation, or an equivalent RWX provider. Know its name before you start.
A PostgreSQL database RSE stores all SysML v2 model data in PostgreSQL. The database must be reachable from the cluster, with a known service endpoint, username, password, and (if TLS is enabled) a CA certificate. RSE does not deploy its own database.
An OIDC provider RSE delegates all user authentication to an external OpenID Connect provider (for example, IBM Jazz Authorization Server, Keycloak, or Microsoft Entra). You need the well-known endpoint URL and a registered OIDC client ID and secret.
An IBM FlexNet licence server RSE checks out floating licences from an IBM FlexNet server. You need the hostname and ports for the lmgrd daemon and the ibmratl vendor daemon.
An Ingress controller RSE uses path-based routing through a Kubernetes Ingress. An NGINX Ingress Controller (or equivalent) must be installed and its IngressClass name must be known.
TLS secrets You need two TLS secrets in the RSE application namespace: one for browser-facing HTTPS (external) and one for pod-to-pod HTTPS (internal). How these are created — cert-manager, your corporate PKI, or manual — is your choice; the secrets must exist before you apply the CR.

Step 1 — Create the Namespaces

RSE uses two separate Kubernetes namespaces with distinct responsibilities. Creating them explicitly — before any other object — gives you full control over RBAC, resource quotas, and network policies from the start.

The operator namespace will hold OLM objects (CatalogSource, OperatorGroup, Subscription, CSV) and the operator pod itself. The application namespace will hold all RSE pods, PersistentVolumeClaims, secrets, and the PostgreSQL database (if deployed in-cluster).

# Replace <operator-namespace> and <app-namespace> with your chosen names.
# Common convention: rse-operator for the operator, ibm-rse for the application.

kubectl create namespace <operator-namespace>
kubectl create namespace <app-namespace>

Verification

kubectl get namespace <operator-namespace> <app-namespace>

# Expected: both show STATUS Active

Step 2 — Create Image Pull Secrets

The RSE operator and its application images are hosted in IBM's private container registry (cp.icr.io). The OLM catalog image is hosted in a separate IBM registry (icr.io/cpopen). Both registries require authentication. Kubernetes pulls these images automatically, but only if the correct pull secrets are present in the right namespaces before OLM tries to pull.

Two secrets are needed:

  • Catalog pull secret in the operator namespace — used by OLM to pull the RSE catalog image from icr.io/cpopen.
  • Application pull secret in the application namespace — used by the operator to pull RSE application images from cp.icr.io.

# Catalog pull secret (operator namespace)
# Server: icr.io  |  Username: iamapikey  |  Password: your ICR API key
kubectl create secret docker-registry <catalog-pull-secret-name> \
  --docker-server=icr.io \
  --docker-username=iamapikey \
  --docker-password="<your-icr-api-key>" \
  --namespace=<operator-namespace>

# Application pull secret (application namespace)
# Server: cp.icr.io  |  Username: cp  |  Password: your IBM Entitlement Key
kubectl create secret docker-registry <app-pull-secret-name> \
  --docker-server=cp.icr.io \
  --docker-username=cp \
  --docker-password="<your-ibm-entitlement-key>" \
  --namespace=<app-namespace>

Verification

kubectl get secret <catalog-pull-secret-name> -n <operator-namespace>
kubectl get secret <app-pull-secret-name>     -n <app-namespace>

# Expected: both show TYPE kubernetes.io/dockerconfigjson

Step 3 — Create the CatalogSource

The CatalogSource is the first OLM object you create. It tells OLM where to find all available versions of the RSE operator and instructs OLM to run a local catalog server pod that answers queries about available packages and channels.

What happens when you apply a CatalogSource

OLM starts a pod in your operator namespace running the catalog image you specify. This pod serves a gRPC API. OLM queries it periodically (on the interval you configure) and creates PackageManifest objects from the responses. A PackageManifest is a read-only Kubernetes object — you never create it yourself — that lists which operator packages exist in the catalog and which channels and versions are available within each package.

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: <catalog-source-name>
  namespace: <operator-namespace>
spec:
  displayName: IBM RSE Operator Catalog
  image: <ibm-rse-operator-catalog-image>    # provided in your IBM product documentation
  publisher: IBM
  sourceType: grpc
  secrets:
    - <catalog-pull-secret-name>              # the secret created in Step 2
  updateStrategy:
    registryPoll:
      interval: 10m                           # how often OLM checks for new operator versions

Verification — wait until the catalog pod is RUNNING and the PackageManifest appears

# The catalog pod should start in the operator namespace
kubectl get pods -n <operator-namespace>
# Expected: a pod named after your CatalogSource shows 1/1 Running

# PackageManifest confirms OLM can read the catalog
kubectl get packagemanifest -A | grep rhapsody
# Expected: a row showing ibm-rhapsody-system-engineering with a CATALOG column matching your CatalogSource name

If the catalog pod is in ImagePullBackOff

The pull secret name in spec.secrets is wrong, or the ICR API key is invalid. Check with kubectl describe pod -n <operator-namespace> <catalog-pod-name> and look at the Events section.

Step 4 — Create the OperatorGroup

An OperatorGroup is required in every namespace where you want to run an operator. It defines the set of namespaces the operator is permitted to watch and create resources in. OLM will refuse to install an operator in a namespace that does not have exactly one OperatorGroup.

Single-namespace vs all-namespace OperatorGroup

Single-namespace (used for RSE): spec.targetNamespaces lists the operator namespace. The operator's RBAC is tightly scoped to that namespace. RSE then creates its application pods in the application namespace separately, after you apply the CR.

All-namespace: spec.targetNamespaces is empty or omitted. The operator gets cluster-wide watch permissions. Do not use this for RSE — the single-namespace scope is the supported and recommended configuration.

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: <operator-group-name>
  namespace: <operator-namespace>
spec:
  targetNamespaces:
    - <operator-namespace>    # scope the operator to the operator namespace

Verification

kubectl get operatorgroup -n <operator-namespace>
# Expected: your OperatorGroup appears with AGE column showing it was just created

kubectl describe operatorgroup <operator-group-name> -n <operator-namespace>
# Expected: Status.Namespaces includes <operator-namespace>

Only one OperatorGroup per namespace

If you see an error like "an OperatorGroup already exists in namespace", someone has already created one. Do not create a second — use kubectl get operatorgroup -n <operator-namespace> to see what exists, and reuse the existing name in your Subscription (Step 5).

Step 5 — Create the Subscription

The Subscription is the object that actually triggers the operator installation. It tells OLM which package to install, which channel to track, where to find it, and how to handle future upgrades.

Channels and installPlanApproval

A channel (e.g., v1.8) is a named stream of operator versions maintained by IBM. Subscribing to a channel means OLM will automatically offer you every new release IBM publishes to that channel, in order, without skipping versions.

  • =installPlanApproval: Automatic= — OLM applies every new version without asking. Suitable for non-production or agile environments.
  • =installPlanApproval: Manual= — OLM generates an InstallPlan and pauses. A human must inspect it and set spec.approved: true before anything changes. Use this for production systems where change control is required.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: <subscription-name>
  namespace: <operator-namespace>
spec:
  channel: <channel-name>                   # e.g. v1.8 — check your IBM product docs for current channel
  name: ibm-rhapsody-system-engineering     # package name in the catalog — do not change this
  source: <catalog-source-name>             # must match the CatalogSource name from Step 3
  sourceNamespace: <operator-namespace>
  installPlanApproval: Automatic            # or Manual for controlled environments

After you apply this, OLM immediately creates an InstallPlan. If approval is Automatic, OLM also immediately executes it. The execution creates all CRDs, RBAC objects, and the operator Deployment — culminating in a new CSV object.

Verification — check the InstallPlan and CSV status

# Check the InstallPlan was created and executed
kubectl get installplan -n <operator-namespace>
# Expected: STATUS column shows Complete (if Automatic) or RequiresApproval (if Manual)

# If Manual approval: inspect and approve
kubectl get installplan -n <operator-namespace>
# Note the InstallPlan name, then approve it:
kubectl patch installplan <installplan-name> \
  --type merge \
  --patch '{"spec":{"approved":true}}' \
  -n <operator-namespace>

# Check the Subscription status
kubectl get subscription <subscription-name> -n <operator-namespace>
# Expected: PACKAGE shows ibm-rhapsody-system-engineering, SOURCE shows your catalog

Step 6 — Wait for the CSV to Reach Succeeded

After the InstallPlan executes, OLM creates a ClusterServiceVersion. The CSV progresses through several phases: Pending -> InstallReady -> Installing -> Succeeded. Only when it reaches Succeeded is the operator pod running and ready to handle your Custom Resource.

What the CSV creates when it reaches Succeeded

  • The operator Deployment — a single-replica pod with two containers: manager (the RSE controller) and kube-rbac-proxy (secures the metrics endpoint).
  • All RBAC objects the operator needs: ClusterRoles, Roles, ClusterRoleBindings, RoleBindings.
  • The CRD for RhapsodySEApp — from this point on, the cluster understands this custom resource type.
  • A ValidatingWebhookConfiguration that intercepts every RhapsodySEApp create or update, validates the CR fields, and rejects misconfigured resources with a clear error message before any reconciliation begins.

# Watch CSV status — may take 3-15 minutes on first install
kubectl get csv -n <operator-namespace> --watch

# Expected final state:
# NAME                                      DISPLAY   VERSION   PHASE
# ibm-rhapsody-system-engineering.vX.Y.Z   IBM RSE   X.Y.Z     Succeeded

Full operator health verification

# CSV must be Succeeded
kubectl get csv -n <operator-namespace>

# Operator pod must be 2/2 Running (manager + kube-rbac-proxy)
kubectl get pods -n <operator-namespace>

# CRD must exist
kubectl get crd rhapsodyseapps.rhapsodyse.ibm.com

# ValidatingWebhookConfiguration must exist
kubectl get validatingwebhookconfiguration | grep rhapsody

CSV stuck in Installing or Failed

  • If the operator pod is in ImagePullBackOff: the application pull secret is missing in the operator namespace, or the image reference in the CSV is incorrect.
  • If the CSV shows Failed with a webhook error: the webhook certificate secret was not injected. This typically means cert-manager (or your certificate tooling) is not reconciling the webhook secret. Restart the operator pod after confirming the TLS secret exists.
  • Run kubectl describe csv <csv-name> -n <operator-namespace> and read the Conditions section for the exact failure reason.

Step 7 — Prepare Secrets in the Application Namespace

The RSE operator reads secrets from the application namespace using a pattern called secret reference resolution. In the Custom Resource, instead of writing sensitive values directly, you write a reference in the form:

<field>: secret::<secret-name>::<key>

At reconcile time, the operator resolves each reference to the actual value from the named Kubernetes Secret. This means the secrets must exist before you apply the CR. If a referenced secret or key is missing, the reconciliation fails immediately and the CR status will show a descriptive error.

The following secrets are required in the application namespace:

Secret name (your choice) Required keys Purpose
Your DB credentials secret dbPassword, dbAdminPassword PostgreSQL user and admin passwords. Referenced in the CR's database section.
Your DB CA cert secret ca-certificate.pem The CA certificate chain for validating the PostgreSQL TLS connection. The key name must be exactly ca-certificate.pem.
Your external TLS secret tls.crt, tls.key Browser-facing HTTPS certificate for the RSE hostname. Used by the Ingress controller.
Your internal TLS secret tls.crt, tls.key Certificate for pod-to-pod HTTPS within the cluster. SANs must include *.<app-namespace>.svc.cluster.local and localhost.

# Database credentials
kubectl create secret generic <db-credentials-secret-name> \
  --from-literal=dbPassword=<db-user-password> \
  --from-literal=dbAdminPassword=<db-admin-password> \
  -n <app-namespace>

# PostgreSQL CA certificate
# (Export the CA cert from your PostgreSQL provider and save it as ca-certificate.pem)
kubectl create secret generic <db-ca-cert-secret-name> \
  --from-file=ca-certificate.pem=<path-to-your-ca-certificate.pem> \
  -n <app-namespace>

# External TLS (browser-facing) — create from your certificate files
kubectl create secret tls <external-tls-secret-name> \
  --cert=<path-to-external-tls.crt> \
  --key=<path-to-external-tls.key> \
  -n <app-namespace>

# Internal TLS (pod-to-pod)
kubectl create secret tls <internal-tls-secret-name> \
  --cert=<path-to-internal-tls.crt> \
  --key=<path-to-internal-tls.key> \
  -n <app-namespace>

Verification — all four secrets must exist before continuing

kubectl get secret \
  <db-credentials-secret-name> \
  <db-ca-cert-secret-name> \
  <external-tls-secret-name> \
  <internal-tls-secret-name> \
  <app-pull-secret-name> \
  -n <app-namespace>
# Expected: all five secrets listed with no "not found" errors

Step 8 — Apply the RhapsodySEApp Custom Resource

With the operator running and all secrets in place, you now apply the single CR that drives the entire RSE deployment. Every field in this CR is a desired state declaration. The operator reconciles the cluster to match it — and keeps reconciling any time the cluster drifts away from it.

What happens the moment you apply the CR

  1. The ValidatingWebhook intercepts the API call and checks all required fields. If anything is wrong, the apply is rejected immediately with a clear error message — nothing is created yet.
  2. On success, the operator's Reconcile() loop is invoked. It reads the CR, resolves all secret::name::key references, and works through a deterministic sequence of create/patch operations.

Replace every <placeholder> with your environment-specific values. Do not hardcode any passwords — use the secret::name::key pattern for all sensitive fields.

apiVersion: rhapsodyse.ibm.com/v1
kind: RhapsodySEApp
metadata:
  name: <cr-name>                          # a name of your choosing; used as prefix for all created resources
  namespace: <app-namespace>
spec:
  acceptLicense: true                      # REQUIRED — you must accept the IBM licence
  imagePullSecret: <app-pull-secret-name>  # pull secret name from Step 2
  imageRepo: <rse-image-registry-path>     # RSE image registry path — see your IBM product documentation

  routing:
    hostname: <rse-public-hostname>         # the FQDN at which RSE will be accessible
    tlsSecret: <external-tls-secret-name>   # external TLS secret from Step 7
    ingressClassName: <ingress-class-name>  # IngressClass name of your ingress controller (e.g. nginx)
    internalServiceTlsSecret: <internal-tls-secret-name>   # internal TLS secret from Step 7

  database:
    server: <postgresql-connection-uri>     # e.g. postgresql://<host>:5432
    name: <database-name>
    schema: <database-schema>
    user: <db-username>
    password: secret::<db-credentials-secret-name>::dbPassword
    adminUser: <db-admin-username>
    adminPassword: secret::<db-credentials-secret-name>::dbAdminPassword
    cacert: secret::<db-ca-cert-secret-name>::ca-certificate.pem
    rejectUnauthorizedTls: <true-or-false>  # set false only if your DB uses a self-signed cert

  oidc:
    clientId: <oidc-client-id>
    clientSecret: <oidc-client-secret>
    signingKey: <random-signing-key-string> # a random string used to sign internal session tokens
    customIdentityClaim: sub               # the JWT claim used as the RSE username
    wellKnownEndpoint: <oidc-well-known-url>
    introspectionEndpoint: <oidc-introspection-url>
    rejectUnauthorized: <true-or-false>     # set false only if your OIDC provider uses a self-signed cert

  storage:
    storageClassName: <rwx-storage-class-name>  # your ReadWriteMany StorageClass name
    accessMode: ReadWriteMany
    capacity: <storage-capacity>           # e.g. 10Gi — applied per PVC

  dbMigrateMode: update                   # run Liquibase on every reconcile — safe, idempotent

  serverAdminUsernames:
    - <oidc-admin-username>               # the RSE super-admin user, must match a claim from your OIDC provider

  serviceToken: <service-token-string>    # a random token for internal OSLC API authentication

  licensing:
    server:
      host: <licence-server-hostname>
      port: <lmgrd-port>                  # default 27000
      id: <licence-server-host-id>        # from lmutil lmhostid on the licence server
    vendor:
      id: ibmratl
      port: <ibmratl-vendor-port>         # default 27002

  linkIndexProvider:
    isJASEnabled: <true-or-false>
    ldxEndpoint: <ldx-endpoint-url>       # only required if integrating with IBM ELM / DOORS Next

Verification — CR accepted by the webhook

# If the webhook rejects the CR, kubectl apply returns an error immediately.
# A successful apply shows:
rhapsodyseapp.rhapsodyse.ibm.com/<cr-name> created

Step 9 — Understand and Monitor the Reconciliation Loop

Once the CR is applied, the operator runs through a fixed sequence of operations. Each step in the reconciliation must succeed before the next begins. If any step fails, the operator sets the CR status to a degraded or failed phase and stops. Knowing the sequence tells you exactly where to look when something goes wrong.

  1. Resolve secret references. Every secret::name::key token is resolved to its actual value. A missing secret or wrong key name aborts reconciliation here with a clear status message.
  2. Create / patch Secrets and ConfigMaps. The operator writes derived internal configuration objects (rse-settings-config, rse-license-client-config) into the application namespace.
  3. Create / patch PersistentVolumeClaims. Three RWX PVCs are created using the StorageClass you specified. All three must reach Bound status before the operator proceeds.
  4. Create / patch ServiceAccounts and RBAC. The <cr-name>-rse-deploy ServiceAccount and its bindings are created in the application namespace.
  5. Run the database migration Job (rse-db-migrator). Liquibase connects to PostgreSQL, creates the schema if it does not exist, and applies all pending changesets. The operator waits for this Job to complete with exit code 0 before proceeding. If the Job fails, the operator surfaces the error in the CR status.
  6. Create / patch Deployments. Four application Deployments are created: rse-server (GraphQL/SysML v2 core), rse-client (Next.js web UI), rse-rest (OSLC REST API), and rse-license (FlexNet licence client).
  7. Create / patch Services. A ClusterIP Service is created for each Deployment, providing stable internal DNS names.
  8. Create / patch Ingress. A single Ingress object with path-based routing rules is created: / -> rse-client, /_server -> rse-server, /_rest -> rse-rest, /_license -> rse-license.
  9. Create NetworkPolicy. A NetworkPolicy that denies inbound cross-namespace traffic to RSE pods is applied, while permitting intra-namespace communication between RSE components.
  10. Update CR status. The operator writes phase and condition fields back to the CR (phase: Running when all pods are healthy).

# Watch the operator logs — the most direct window into reconciliation progress
kubectl logs -n <operator-namespace> \
  -l control-plane=controller-manager \
  -c manager --follow

# Watch pods appear in the application namespace
kubectl get pods -n <app-namespace> --watch

# Check CR status phase and conditions
kubectl describe rhapsodyseapp <cr-name> -n <app-namespace>

# Check PVCs — all must reach Bound before Deployments are created
kubectl get pvc -n <app-namespace>

# Check database migration Job — must reach Succeeded
kubectl get job -n <app-namespace>
kubectl logs -n <app-namespace> \
  -l job-name=<cr-name>-rse-db-migrator
# Look for: "Liquibase command 'update' was executed successfully"

Step 10 — Verify the Fully Healthy State

Run all of the following checks. Every item must pass before you consider the installation complete and hand the system over to users.

Component Resource Expected status
Deployments (app namespace) <cr-name>-rse-client 1/1 Running
Deployments (app namespace) <cr-name>-rse-server 1/1 Running
Deployments (app namespace) <cr-name>-rse-rest 1/1 Running
Deployments (app namespace) <cr-name>-rse-license 1/1 Running
PVCs (app namespace) <cr-name>-shared-storage Bound
PVCs (app namespace) <cr-name>-shared-storage-public-ext Bound
PVCs (app namespace) <cr-name>-artifacts-storage Bound
OLM (operator namespace) CSV phase Succeeded
OLM (operator namespace) Operator pod 2/2 Running
OLM (operator namespace) Subscription AtLatestKnown
Job (app namespace) <cr-name>-rse-db-migrator Succeeded

# All RSE pods
kubectl get pods -n <app-namespace>

# All PVCs Bound
kubectl get pvc -n <app-namespace>

# Ingress rule created with correct hostname
kubectl get ingress -n <app-namespace>

# CR phase — expected output: Running
kubectl get rhapsodyseapp <cr-name> -n <app-namespace> \
  -o jsonpath='{.status.phase}'

# OLM subscription tracking
kubectl get subscription <subscription-name> -n <operator-namespace>
# Expected: INSTALLEDCSV shows the current version, STATE shows AtLatestKnown

Troubleshooting — Common First-Install Failures

Symptom Diagnostic command Cause and fix
Catalog pod in ImagePullBackOff kubectl describe pod -n <operator-namespace> <catalog-pod> The catalog pull secret is wrong or missing. Check the secret name in the CatalogSource spec.secrets field matches the secret name created in Step 2, and that the ICR API key is valid.
No PackageManifest for RSE after 10+ minutes =kubectl get packagemanifest -A \ grep rhapsody= The catalog pod is not healthy. Check its logs: kubectl logs -n <operator-namespace> <catalog-pod>. Also confirm the CatalogSource sourceType: grpc is set.
InstallPlan stuck in RequiresApproval kubectl get installplan -n <operator-namespace> Expected when installPlanApproval: Manual. Approve it: kubectl patch installplan <name> --type merge --patch '{"spec":{"approved":true}}' -n <operator-namespace>
CSV stuck in Installing kubectl describe csv <csv-name> -n <operator-namespace> Read the Conditions section. Most common cause: the operator image cannot be pulled. Ensure the application pull secret exists in the operator namespace and the entitlement key is valid.
CR apply rejected with webhook error kubectl describe validatingwebhookconfiguration The operator webhook server is not ready. Wait for the operator pod to reach 2/2 Running. If still failing, the webhook TLS secret may not exist — check =kubectl get secret -n <operator-namespace> \ grep webhook=.
PVCs stuck in Pending kubectl describe pvc -n <app-namespace> The StorageClass name in the CR does not match any existing StorageClass, or the RWX provisioner is not running. Verify with kubectl get storageclass and check the Events on the PVC.
DB migrator Job in Error / fails to complete kubectl logs -n <app-namespace> -l job-name=<cr-name>-rse-db-migrator (a) PostgreSQL not reachable — check network and connection string. (b) Wrong CA cert — the ca-certificate.pem key does not match the actual CA used by the DB server. (c) pg_hba.conf uses cert auth — RSE uses password (scram-sha-256) over TLS, not client certificates.
rse-server in CrashLoopBackOff kubectl logs -n <app-namespace> -l app=<cr-name>-rse-server (a) DB migrator has not yet succeeded. (b) OIDC well-known endpoint unreachable from inside the cluster. (c) Internal TLS secret missing or has incorrect SANs.
rse-client readiness probe failing kubectl describe pod -n <app-namespace> <rse-client-pod> rse-client has an initialDelaySeconds: 60 readiness probe — wait at least 90 seconds. If still failing after 2 minutes: confirm the external TLS secret exists and the hostname in routing.hostname matches the certificate's Subject Alternative Names.
Ingress returns 404 or times out kubectl get ingress -n <app-namespace> -o yaml The ingressClassName in the CR does not match the IngressClass installed on your cluster. Run kubectl get ingressclass to find the correct name and update the CR.
rse-license in CrashLoopBackOff kubectl logs -n <app-namespace> -l app=<cr-name>-rse-license Network connectivity issue to the FlexNet licence server. Test from inside the pod: kubectl exec -n <app-namespace> <rse-license-pod> -- nc -zv <licence-server-host> <port>. Also verify the id field matches the licence server host ID (run lmutil lmhostid on the licence server).
Reconciliation never progresses; CR status stays blank kubectl logs -n <operator-namespace> -l control-plane=controller-manager -c manager The operator is not watching the application namespace. Check the OperatorGroup's targetNamespaces includes the operator namespace, and that the CRD exists: kubectl get crd rhapsodyseapps.rhapsodyse.ibm.com.

How Upgrades Work After Installation

Once installed, the RSE operator manages its own upgrades through OLM. You do not need to re-run any of the steps above to apply a new patch version of RSE.

When IBM publishes a new operator bundle to the catalog image, the catalog pod in your cluster picks it up within the polling interval (default: 10 minutes). OLM generates a new InstallPlan for the next version in the channel. If your Subscription uses Automatic approval, the new operator version rolls out within those 10 minutes. Once the new CSV reaches Succeeded, the operator reconciles all existing RhapsodySEApp CRs, applies the new application images, and triggers a fresh database migration Job.

To move to a new major channel (for example, from v1.8 to v1.9), patch the Subscription's channel field:

kubectl patch subscription <subscription-name> \
  --patch '{"spec":{"channel":"<new-channel>"}}' \
  --type=merge \
  -n <operator-namespace>

To switch to Manual approval for controlled upgrades:

kubectl patch subscription <subscription-name> \
  --patch '{"spec":{"installPlanApproval":"Manual"}}' \
  --type=merge \
  -n <operator-namespace>

# After OLM generates the new InstallPlan, inspect it:
kubectl describe installplan <installplan-name> -n <operator-namespace>

# When ready to proceed:
kubectl patch installplan <installplan-name> \
  --type merge \
  --patch '{"spec":{"approved":true}}' \
  -n <operator-namespace>

For full TWiki text formatting rules, see TWiki Text Formatting Rules.

Edit | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r1 | More topic actions
This site is powered by the TWiki collaboration platformCopyright © by IBM and non-IBM contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our Terms of Use. Please read the following disclaimer.
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.