Deployment example: IBM Jazz.net self-host development environment

This article describes the architecture, topology, and specification of the Jazz.net self-host development environment.

The Jazz development team's self-host production IBM Engineering Lifecycle Management (ELM) system is called "JazzDev." JazzDev provides applications for Change and Configuration Management (CCM) using IBM Engineering Workflow Management (EWM), Quality Management (QM) using IBM Engineering Test Management (ETM), Requirements Management (RM) using IBM Engineering Requirements Management DOORS Next (DOORS Next), Global Configurations (GC) using IBM Engineering Global Configuration Management (GCM), Reporting using IBM Engineering Report Builder (ERB), data collection and data warehouse for reporting using IBM Engineering Data Warehouse (EDW), indexed data for reporting using Lifecycle Query Engine (LQE), and cross-product linked indexes using Link Index Provider (LDX).

There are two staging/test environments for the JazzDev self-host environment. The first -- also known as "BluesDev" -- is for continuous daily sprint milestone ELM builds and is available for the development teams to test daily builds in a JazzDev staging environment. The second staging environment -- also known as "SoulDev" -- is used for staging and testing the final declared sprint milestone ELM build before deploying to (upgrading) production JazzDev.

Production JazzDev Application Servers - nine.

Application server hardware/system - Eight separate ELM servers and one user authorization server

Server 1: Primary ELM Server - Separate Liberty Java application profiles for Jazz Team Server (JTS), Clustered Engineering Workspace Management (EWM) primary node for Jazzdev (/jazz), Engineering Requirements Management DOORS Next (DOORS Next), and Clustered Engineering Test Management (ETM) primary node

• Virtual machine (host name: jazzclm01)
• 64-bit x86; Intel (R) Xeon(R) CPU E5-2665 0 @ 2.40GHz; 20 cores; 48 GB RAM
• Red Hat Enterprise Linux Server release 8.10 (0otpa)
• WebSphere Liberty Server 25.0.0.6

Server 2: ELM Server - Separate Liberty application profiles for Engineering Data Warehouse (EDW) and Global Configuration Management (GCM)

• Virtual machine (host name: jazzclm02)
• 64-bit x86; Intel (R) Xeon(R) CPU E5-2665 0 @ 2.40GHz; 16 cores; 20 GB RAM
• Red Hat Enterprise Linux Server release 8.10 (0otpa)
• WebSphere Liberty Server 25.0.0.6

Server 3: ELM Server - Engineering Report Builder (ERB) on single Liberty application profile

• Virtual machine (host name: jazzclm03)
• 64-bit x86; Intel (R) Xeon(R) CPU E5-2665 0 @ 2.40GHz; 4 cores; 20 GB RAM
• Red Hat Enterprise Linux Server release 8.10 (0otpa)
• WebSphere Liberty Server 25.0.0.6

Server 4: ELM Server - JTS (Liberty profile) and three instances of EWM applications (Jazzdev02 (/jazz02), Jazzdev03 (/jazz03), and Jazzdev04 (/jazz04)) on three separate Liberty profiles

• Virtual Machine (host name: jazzclm04)
• 64-bit x86; Intel (R) Xeon(R) CPU E5-2665 0 @ 2.40GHz; 8 cores; 36 GB RAM
• Red Hat Enterprise Linux Server release 8.10 (0otpa)
• WebSphere Liberty Server 25.0.0.6

Server 5: ELM Server -Lifecycle Query Engine (LQE) application on Liberty profiles

• Physical machine (host name: jazzclm09)
• 64-bit x86; Intel (R) Xeon(R) Gold CPU @ 2.10GHz; 64 cores; 1056 GB (1 TB) RAM
• Red Hat Enterprise Linux Server release 8.10 (0otpa)
• WebSphere Liberty Server 25.0.0.6

Server 6: ELM Server - Clustered EWM secondary node application (Jazzdev02 (/jazz02), Jazzdev03 (/jazz03), and Jazzdev04 (/jazz04)) on three separate WebSphere Liberty profiles

• Virtual machine (host name: jazzclm06)
• 64-bit x86; Intel (R) Xeon(R) CPU E5-2665 0 @ 2.40GHz; 20 cores; 32 GB RAM
• Red Hat Enterprise Linux Server release 8.10 (0otpa)
• WebSphere Liberty Server 25.0.0.6

Server 7: ELM Server - Clustered EWM secondary node (Jazzdev (/jazz)) and Clustered ETM secondary node applications on separate Liberty profiles

• Virtual machine (host name: jazzclm07)
• 64-bit x86; Intel (R) Xeon(R) CPU E5-2665 0 @ 2.40GHz; 6 cores; 32 GB RAM
• Red Hat Enterprise Linux Server release 8.10 (0otpa)
• WebSphere Liberty Server 25.0.0.6

Server 8: ELM Server - Link Index Provider (LDX) application on Liberty profile

• Virtual machine (host name: jazzclm08)
• 64-bit x86; Intel (R) Xeon(R) CPU E5-2665 0 @ 2.40GHz; 6 cores; 24 GB RAM
• Red Hat Enterprise Linux Server release 8.10 (0otpa)
• WebSphere Liberty Server 25.0.0.6

Server 9: Jazz Authorization Server (JAS) - Single Sign-On (SSO) LDAP service on Liberty profile

• Virtual machine (host name: jazzauth01)
• 64-bit x86; Intel (R) Xeon(R) CPU E5-2665 0 @ 2.40GHz; 2 cores; 4 GB RAM
• Red Hat Enterprise Linux Server release 8.10 (0otpa)
• WebSphere Liberty Server 25.0.0.6

WebSphere Liberty Application Server JVM Settings:

• Verbose Garbage Collection: enabled on all servers
  

• Primary ELM Server (jazzclm01):
  • Initial heap (Xms): JTS: 8GB, clustered EWM node: 12GB, clustered ETM node: 8GB, DOORS Next: 6GB
  • Max heap (Xmx): JTS: 8GB, clustered EWM node: 12GB, clustered ETM node: 8GB, DOORS Next: 6GB
  • Generic JVM arguments
    • -Xgcpolicy:gencon
    • -Xcompressedrefs
    • -Xgc:preferredHeapBase=0x100000000
    • - XX:MaxDirectMemorySize=2G Increases native memory (java/nio) to avoid explicit gc()

• ELM server (jazzclm02):
  • Initial heap (Xms): EDW: 8GB, GCM: 4GB
  • Max heap (Xmx): EDW: 8GB, GCM: 4GB
  • Generic JVM arguments
    • -Xgcpolicy:gencon
    • -Xcompressedrefs
    • -Xgc:preferredHeapBase=0x100000000
    • - XX:MaxDirectMemorySize=1G

• ELM server (jazzclm03):
  • Initial heap (Xms): 8GB
  • Max heap (Xmx): 8GB
  • Generic JVM arguments
    • -Xgcpolicy:gencon
    • -Xcompressedrefs
    • -Xgc:preferredHeapBase=0x100000000
    • - XX:MaxDirectMemorySize=1G

• ELM server (jazzclm04):
  • Initial heap (Xms): 6GB for each (four) profile -- JTS and the three clustered EWM nodes: jazzdev02, jazzdev03, jazzdev04
  • Max heap (Xmx): 6GB for each (four) profile -- JTS and the three clustered EWM nodes: jazzdev02, jazzdev03, jazzdev04
  • Generic JVM arguments
    • -Xgcpolicy:gencon
    • -Xcompressedrefs
    • -Xgc:preferredHeapBase=0x100000000
    • - XX:MaxDirectMemorySize=1G

• ELM server (jazzclm09):
  • Initial heap (Xms): 300GB for LQE profile (-Xms300g)
  • Max heap (Xmx): 300GB for LQE profile (-Xmx300g)
  • Nursery heap=50GB for LQE profile (-Xmn50g)
  • Generic JVM arguments
    • -Xgcpolicy:gencon
    • -Xcompressedrefs
      
    • -Xgc:preferredHeapBase=0x100000000
      
    • - XX:MaxDirectMemorySize=1G
      

• ELM server (jazzclm06):
  • Initial heap (Xms): 6GB for each (three) profile -- clustered EWM nodes: jazzdev02, jazzdev03, jazzdev04)
  • Max heap (Xmx): 6GB for each (three) profile -- clustered EWM nodes: jazzdev02, jazzdev03, jazzdev04)
  • Generic JVM arguments
    • -Xgcpolicy:gencon
    • -Xcompressedrefs
    • -Xgc:preferredHeapBase=0x100000000
    • - XX:MaxDirectMemorySize=1G

• ELM server (jazzclm07):
  • Initial heap (Xms): clustered EWM node: 12GB
  • Max heap (Xmx): clustered ETM node: 12GB
  • Generic JVM arguments
    • -Xgcpolicy:gencon
    • -Xcompressedrefs
      
    • -Xgc:preferredHeapBase=0x100000000
      
    • - XX:MaxDirectMemorySize=1G
      

• ELM server (jazzclm08):
  • Initial heap (Xms): LDX: 12GB
  • Max heap (Xmx): LDX: 12GB
  • Generic JVM arguments
    • -Xgcpolicy:gencon
    • -Xcompressedrefs
      
    • -Xgc:preferredHeapBase=0x100000000
      
    • - XX:MaxDirectMemorySize=1G
      

• JAS server (jazzauth01):
  • Initial heap (Xms): 1536 MB
  • Max heap (Xmx): 1536 MB
  • Generic JVM arguments
    • Defaults (none set)

Production Database Servers for JazzDev - Three

DB server hardware/system:

• Two virtual machines (jazzdb01, jazzdb02)
• Red Hat Enterprise Linux Server 8.10 (Ootpa)
• x86_64 bit OS; Intel (R) Xeon(R) CPU E5-2665 0 @ 2.40GHz, 16 cores each; 64 GB RAM each
• 7.9 TB local RAID 5 array for data, 100 GB for "/" partition, mirrored

• One Baremetal machine (jazzdb03)
• Red Hat Enterprise Linux release 9.6 (Plow)
• x86_64 bit OS; Intel (R) Xeon(R) CPU Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz
• 24 TB local RAID 6 array for data, 100 GB for "/" partition
• 1 TB RAM

Db2 (jazzdb01, jazzdb02, jazzdb03):

• Db2 11.5.9.0 with the Latest Fix Mod Pack
• "Automatic" settings, except automatic table space growth are not allowed. You can monitor disk usage and not get caught short of disk space
• Monitor regularly for sufficient disk space and table space
• Daily online backups using IBM Spectrum Protect (formerly Tivoli Storage Manager)

Production Load Balancers - two

Load balancer hardware/system:

• Two virtual machines (jazzdispatch1)
• Red Hat Enterprise Linux release 8.10 (Ootpa)
• x86_64 bit OS; Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GHz; 2 cores each; 16 GB RAM each

• Two virtual machines ( jazzdispatch2)
• Red Hat Enterprise Linux 7.9 (Maipo)
• x86_64 bit OS; Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz; 2 cores each; 16 GB RAM each

IBM HTTPS Server (IHS) load balancer:

• WebSphere Edge Load Balancer
• Setup as provided in the ELM documentation

Production Reverse Proxy - two

Reverse proxy hardware/system:

• Two virtual machines (jazzproxy1)
• Red Hat Enterprise Linux release 8.10 (Ootpa)
• x86_64 bit OS;Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz; 4 cores each VM; 16 GB RAM each VM
• Active/Active configuration

• Two virtual machines (jazzproxy2)
• Red Hat Enterprise Linux Server release 7.9 (Maipo)
• x86_64 bit OS; Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz; 4 cores each VM; 16 GB RAM each VM
• Active/Active configuration

Reverse proxy:

• HAProxy (Red Hat Enterprise Linux)

Production MQTT Broker (Eclipse Amlen) - one

MQTT (Eclipse Amlen) server hardware/system for Clustered EWM and ETM:

• One virtual machine (jazzmqtt01)
• Red Hat Enterprise Linux release 8.10 (Ootpa)
• x86_64 bit OS;Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz; 2 cores; 12 GB RAM
• Active configuration

Eclipse Amlen:

• Amlen Server 1.0.0.2, Amlen WebUI 1.0.0.2

Production Apache Web Servers - four

Apache server hardware/system:

• Four virtual machines (jazzweb1, jazzweb2, jazzweb3, jazzweb4)
• Red Hat Enterprise Linux 7.9 (Maipo)
• x86_64 bit OS; Intel (R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz (on two VMs) and Intel (R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz (on two VMs); 8 cores each VM; 64 GB RAM each VM

Apache servers:

• Apache 2.4.6 (Red Hat Enterprise Linux)
• Apache Tomcat/9.1
  • JVM Settings
    • Initial heap=2048 MB
    • Max heap=8192 MB
• mod_proxy module to handle reverseProxy configuration
• Apache is shared with other applications.

Firewall and VPN Host Servers (two):

• Two virtual machines (jazzgate01, jazzgate02)
• SoftLayer Vyatta 5600
• x86_64 bit OS; Intel(R) Xeon(R) CPU E3-1270 V2 @ 3.50GHz (two VM), 8 cores each VM; 32 GB RAM each VM
• Active/Active configuration
• Provides virtual firewall across all zones
• Provides Virtual Private Network (VPN) for tunnelling between SoftLayer and Raleigh Technical Park (RTP) site

Architecture Diagrams (Latest)

• 2021 Jazz.net Self-Host Infrastructure Overview:
  

• 2021 Jazz.net Environment System Topology Architecture:
  

• 2021 Staging BluesDev environment for continuous daily ELM builds:
  

• 2021 Staging SoulDev environment for Jazzdev:
  

• 2021 Jazz.net Environment System Topology LDAP Architecture:
  

Related topics: None

External links:

• None

Additional contributors: None