RTC Build agent on z/OS SSL communication RC:410 and RC:420

Sathya moorthy (35●2●9●13) | asked Jun 14 '14, 2:51 p.m.
retagged Sep 01 '17, 11:41 a.m. by Ken Tessier (841●1●7)

Hi,

We have installed RTC Build Agent on z/OS. now we are planning to do SSL settings for Build agent.

This is Our SSL part of bfagent.conf

May use a gskkyman key database instead of SAF keyring

gsk_ssl_key_location /etc/jazz405/ccm/bfagentssl.kdb
gsk_ssl_kdb_password 93a2686a707787d400f7f2d401194145b4e0c3cb695d0e1b58ec
gsk_keyring_label bfacert
gsk_ssl_protocol ALL
gsk_ssl_cipher_v2 6321
gsk_ssl_cipher_v3 0906030201
gsk_ssl_client_authentication false
gsk_password_encrypt true

We are getting the below error, when we test the connection on the bfagent.log debug.

Sat Jun 14 18:32:33 2014 [16842915] main.c          : 519: === STARTING DAEMON PROCESS ===
Sat Jun 14 18:32:34 2014 [16842915] bf_ipv6.c       : 780: bf_listener_new: [0.0.0.0/5555] listening
Sat Jun 14 18:32:34 2014 [16842915] daemon.c        : 53: background: orphaning daemon
Sat Jun 14 18:32:34 2014 [16842915] daemon.c        : 45: --- EXITING --- (create orphan)
Sat Jun 14 18:32:34 2014 [   65700] daemon.c        : 45: --- EXITING --- (create orphan)
Sat Jun 14 18:32:41 2014 [33620131] daemon.c        : 131: [/5555] accepting [161.178.198.170/49830] (161.178.198.170)
Sat Jun 14 18:32:41 2014 [33620131] daemon.c        : 156: [161.178.198.170/49830]: attached to pid 16842916
Sat Jun 14 18:32:41 2014 [16842916] daemon.c        : 149: === NEW AGENT ===
Sat Jun 14 18:32:41 2014 [16842916] platform.c      : 192: ICONV ok [IBM-1047]
Sat Jun 14 18:32:41 2014 [16842916] platform.c      :2132: LOCALE ok [] -> [C]
Sat Jun 14 18:32:41 2014 [16842916] io.c            : 482: In start_SSL : initialize system ssl environment
Sat Jun 14 18:32:41 2014 [16842916] io.c            : 291: gsk_ssl_kdb_password : 93a2686a707787d400f7f2d401194145b4e0c3cb695d0e1b58ec

Sat Jun 14 18:32:41 2014 [16842916] bfcryptloader.c : 569: Password decoded.
Sat Jun 14 18:32:41 2014 [16842916] io.c            : 300: decode_password :***
Sat Jun 14 18:32:42 2014 [16842916] io.c            : 365: SSL ssl_key_location=/etc/jazz405/ccm/bfagentssl.kdb
Sat Jun 14 18:32:42 2014 [16842916] io.c            : 378: SSL decode_password=
Sat Jun 14 18:32:42 2014 [16842916] io.c            : 415: Server: Setting protocol to all.

Sat Jun 14 18:32:42 2014 [16842916] io.c            : 418: SSL ssl_key_ring_label=bfacert
Sat Jun 14 18:32:42 2014 [16842916] io.c            : 430: SYSTEM SSL env init ok
Sat Jun 14 18:32:42 2014 [16842916] io.c            : 496: Calling secure socket open
Sat Jun 14 18:32:42 2014 [16842916] io.c            : 517: SSL GSK_SESSION_TYPE=GSK_SERVER_SESSION
Sat Jun 14 18:32:42 2014 [16842916] io.c            : 524: SSL cipher_specs_V2=6321
Sat Jun 14 18:32:42 2014 [16842916] io.c            : 529: SSL cipher_specs_V3=0906030201
Sat Jun 14 18:32:42 2014 [16842916] io.c            : 533: Calling gsk_secure_socket_init() to start ssl handshake with client.
Sat Jun 14 18:34:05 2014 [33620131] daemon.c        : 131: [*/5555] accepting [10.26.125.254/17502] (10.26.125.254)
Sat Jun 14 18:34:06 2014 [33620131] daemon.c        : 156: [10.26.125.254/17502]: attached to pid 65702
Sat Jun 14 18:34:06 2014 [   65702] daemon.c        : 149: === NEW AGENT ===
Sat Jun 14 18:34:06 2014 [   65702] platform.c      : 192: ICONV ok [IBM-1047]
Sat Jun 14 18:34:06 2014 [   65702] platform.c      :2132: LOCALE ok [] -> [C]
Sat Jun 14 18:34:06 2014 [   65702] io.c            : 482: In start_SSL : initialize system ssl environment
Sat Jun 14 18:34:06 2014 [   65702] io.c            : 291: gsk_ssl_kdb_password : 93a2686a707787d400f7f2d401194145b4e0c3cb695d0e1b58ec

Sat Jun 14 18:34:06 2014 [   65702] bfcryptloader.c : 569: Password decoded.
Sat Jun 14 18:34:06 2014 [   65702] io.c            : 300: decode_password :
Sat Jun 14 18:34:06 2014 [   65702] io.c            : 365: SSL ssl_key_location=/etc/jazz405/ccm/bfagentssl.kdb
Sat Jun 14 18:34:06 2014 [   65702] io.c            : 378: SSL decode_password=
Sat Jun 14 18:34:06 2014 [   65702] io.c            : 415: Server: Setting protocol to all.

Sat Jun 14 18:34:06 2014 [   65702] io.c            : 418: SSL ssl_key_ring_label=bfacert
Sat Jun 14 18:34:06 2014 [   65702] io.c            : 430: SYSTEM SSL env init ok
Sat Jun 14 18:34:06 2014 [   65702] io.c            : 496: Calling secure socket open
Sat Jun 14 18:34:06 2014 [   65702] io.c            : 517: SSL GSK_SESSION_TYPE=GSK_SERVER_SESSION
Sat Jun 14 18:34:06 2014 [   65702] io.c            : 524: SSL cipher_specs_V2=6321
Sat Jun 14 18:34:06 2014 [   65702] io.c            : 529: SSL cipher_specs_V3=0906030201
Sat Jun 14 18:34:06 2014 [   65702] io.c            : 533: Calling gsk_secure_socket_init() to start ssl handshake with client.
Sat Jun 14 18:37:41 2014 [33620131] daemon.c        : 131: [*/5555] accepting [10.26.125.254/17660] (10.26.125.254)
Sat Jun 14 18:37:41 2014 [16842916] io.c            : 536: SSL_accept rc=410 (0 is good, >=1 is handshake failure or fatal)
Sat Jun 14 18:37:41 2014 [16842916] io.c            : 539: SSLErrorHandshake: failed to Call gsk_secure_socket_init() to.
Sat Jun 14 18:37:41 2014 [16842916] agent.c         : 761: cleanup
Sat Jun 14 18:37:41 2014 [33620131] platform.c      : 260: SIGCHLD(20) 16842916 -> bfdaemon: status=0
Sat Jun 14 18:37:41 2014 [33620131] daemon.c        : 156: [10.26.125.254/17660]: attached to pid 65705
Sat Jun 14 18:37:41 2014 [   65705] daemon.c        : 149: === NEW AGENT ===
Sat Jun 14 18:37:41 2014 [   65705] platform.c      : 192: ICONV ok [IBM-1047]
Sat Jun 14 18:37:41 2014 [   65705] platform.c      :2132: LOCALE ok [] -> [C]
Sat Jun 14 18:37:41 2014 [   65705] io.c            : 482: In start_SSL : initialize system ssl environment
Sat Jun 14 18:37:41 2014 [   65705] io.c            : 291: gsk_ssl_kdb_password : 93a2686a707787d400f7f2d401194145b4e0c3cb695d0e1b58ec

Sat Jun 14 18:37:41 2014 [   65705] bfcryptloader.c : 569: Password decoded.
Sat Jun 14 18:37:41 2014 [   65705] io.c            : 300: decode_password :
Sat Jun 14 18:37:42 2014 [   65705] io.c            : 365: SSL ssl_key_location=/etc/jazz405/ccm/bfagentssl.kdb
Sat Jun 14 18:37:42 2014 [   65705] io.c            : 378: SSL decode_password=**
Sat Jun 14 18:37:42 2014 [   65705] io.c            : 415: Server: Setting protocol to all.

Sat Jun 14 18:37:42 2014 [   65705] io.c            : 418: SSL ssl_key_ring_label=bfacert
Sat Jun 14 18:37:42 2014 [   65705] io.c            : 430: SYSTEM SSL env init ok
Sat Jun 14 18:37:42 2014 [   65705] io.c            : 496: Calling secure socket open
Sat Jun 14 18:37:42 2014 [   65705] io.c            : 517: SSL GSK_SESSION_TYPE=GSK_SERVER_SESSION
Sat Jun 14 18:37:42 2014 [   65705] io.c            : 524: SSL cipher_specs_V2=6321
Sat Jun 14 18:37:42 2014 [   65705] io.c            : 529: SSL cipher_specs_V3=0906030201
Sat Jun 14 18:37:42 2014 [   65705] io.c            : 533: Calling gsk_secure_socket_init() to start ssl handshake with client.
Sat Jun 14 18:37:42 2014 [   65705] io.c            : 536: SSL_accept rc=420 (0 is good, >=1 is handshake failure or fatal)
Sat Jun 14 18:37:42 2014 [   65705] io.c            : 539: SSLErrorHandshake: failed to Call gsk_secure_socket_init() to.
Sat Jun 14 18:37:42 2014 [   65705] agent.c         : 761: cleanup
Sat Jun 14 18:37:42 2014 [33620131] platform.c      : 260: SIGCHLD(20) 65705 -> bfdaemon: status=0

Please let us know what we are missing.

Regards,
Sathya

Comments

Donald Nong commented Jun 15 '14, 11:18 p.m.

Check the corresponding error messages for the two return code.
http://publib.boulder.ibm.com/infocenter/zvm/v5r4/topic/com.ibm.zvm.v54.kijl0/hcsk7b3040.htm#wq211
RC 410 means "incorrectly-formatted message received from peer application". Maybe you can use tcpdump to capture the network traffic and see what was being transferred.

Sathya moorthy commented Jun 16 '14, 2:08 a.m.

Thanks Donald Nong. Yes i had found the reason code what it means for RC 410 and RC 420, but not sure why its happening. I'll check on tcpdump.

Regards,
Sathya

Eric Kung commented Aug 24 '17, 7:22 p.m. | edited Aug 24 '17, 9:10 p.m.

Hi, Sathya,

Bring this old discussion back again, we are struggling with same setup. Trying to get z/OS Rational build agent connect to RTC server with TLS1.2, got exactly same error as you had.

Did you end up with working solution ? are you able to share if you did.

Sathya moorthy commented Aug 25 '17, 1:25 a.m.

Hi Eric,

Yes, it was resolved and we were able to use TLS1.2. I'm not able to recollect them now. can you share your log and conf details.

Regards,
Sathya

Eric Kung commented Aug 27 '17, 6:00 p.m. | edited Aug 27 '17, 6:44 p.m.

Hi, Sathya,

Thanks for your reply.

I followed this link to set it up, use kdb file as certificate

https://jazz.net/wiki/bin/view/Main/BFAgentZOSSystemSSL#With_a_kdb_file

SSL section of bfagent.conf

# Note: If using SSL, create or obtain PEM keystores from engine and set the proper key password.

#ssl_key_location /usr/local/bin/buildForgeKey.pem

#ssl_key_password <password>

#ssl_cert_location /usr/local/bin/buildForgeCert.pem

#ssl_ca_location /usr/local/bin/buildForgeCA.pem

# Note: The ssl_protocol can be TLSv1,TLSv1.1 or TLSv1.2. You select only one.

#ssl_protocol TLSv1

#ssl_protocol TLSv1.1

ssl_protocol TLSv1.2

#ssl_cipher_group ALL

#ssl_cipher_override <cipher_list>

#ssl_client_authentication true

#ssl_client_auth_sufficient true

fips_enabled true

# Note: If using password encryption, see "Export Key File" in console to obtain keys in bfpwcrypt.conf.

#password_encrypt_module ./bfcrypt.dll;./bfpwcrypt.conf

#digest_algorithm SHA2

# System SSL support using a SAF keyring, just like [userid]/[keyring] SAF

#gsk_ssl_key_location root/cert SAF

# OR

# May use a gskkyman key database instead of SAF keyring

gsk_ssl_key_location /etc/jazz603/ccm/ek_rba.kdb

gsk_ssl_kdb_password 9b16d9abab33fcb200f7fa63be9a9c5289d522e5058157f5e738

gsk_keyring_label EKCERT01

gsk_ssl_protocol ALL

#gsk_ssl_protocol TLSV1_2

#gsk_ssl_cipher_v2 6321

#gsk_ssl_cipher_v3 0906030201

gsk_ssl_client_authentication true

gsk_password_encrypt true

bfagent.log :

Fri Aug 25 05:13:33 2017 [ 65881] platform.c : 192: ICONV ok [IBM-1047]

Fri Aug 25 05:13:33 2017 [ 65881] platform.c :2154: LOCALE ok [] -> [C]

Fri Aug 25 05:13:33 2017 [ 65881] main.c : 519: === STARTING DAEMON PROCESS ===

Fri Aug 25 05:13:33 2017 [ 65881] bf_ipv6.c : 780: bf_listener_new: [0.0.0.0/5555] listening

Fri Aug 25 05:13:33 2017 [ 65881] daemon.c : 53: background: orphaning daemon

Fri Aug 25 05:13:33 2017 [ 65881] daemon.c : 45: --- EXITING --- (create orphan)

Fri Aug 25 05:13:33 2017 [ 65882] daemon.c : 45: --- EXITING --- (create orphan)

Fri Aug 25 05:14:51 2017 [16843097] daemon.c : 131: [/5555] accepting [9.32.157.211/57246] (igartctrain02.swg.usma.ibm.com)

Fri Aug 25 05:14:51 2017 [16843097] daemon.c : 156: [9.32.157.211/57246]: attached to pid 33620315

Fri Aug 25 05:14:51 2017 [33620315] daemon.c : 149: === NEW AGENT ===

Fri Aug 25 05:14:51 2017 [33620315] platform.c : 192: ICONV ok [IBM-1047]

Fri Aug 25 05:14:51 2017 [33620315] platform.c :2154: LOCALE ok [] -> [C]

Fri Aug 25 05:14:51 2017 [33620315] io.c : 482: In start_SSL : initialize system ssl environment

Fri Aug 25 05:14:51 2017 [33620315] io.c : 291: gsk_ssl_kdb_password : 9b16d9abab33fcb200f7fa63be9a9c5289d522e5058157f5e738

Fri Aug 25 05:14:51 2017 [33620315] bfcryptloader.c : 569: Password decoded.

Fri Aug 25 05:14:51 2017 [33620315] io.c : 300: decode_password :***

Fri Aug 25 05:14:52 2017 [33620315] io.c : 365: SSL ssl_key_location=/etc/jazz603/ccm/ek_rba.kdb

Fri Aug 25 05:14:52 2017 [33620315] io.c : 378: SSL decode_password=**

Fri Aug 25 05:14:52 2017 [33620315] io.c : 415: Server: Setting protocol to all.

Fri Aug 25 05:14:52 2017 [33620315] io.c : 418: SSL ssl_key_ring_label=EKCERT01

Fri Aug 25 05:14:52 2017 [33620315] io.c : 430: SYSTEM SSL env init ok

Fri Aug 25 05:14:52 2017 [33620315] io.c : 496: Calling secure socket open

Fri Aug 25 05:14:52 2017 [33620315] io.c : 514: SSL GSK_SESSION_TYPE=GSK_SERVER_SESSION_WITH_CL_AUTH

Fri Aug 25 05:14:52 2017 [33620315] io.c : 524: SSL cipher_specs_V2=631

Fri Aug 25 05:14:52 2017 [33620315] io.c : 529: SSL cipher_specs_V3=090201

Fri Aug 25 05:14:52 2017 [33620315] io.c : 533: Calling gsk_secure_socket_init() to start ssl handshake with client.

Fri Aug 25 05:19:51 2017 [33620315] io.c : 536: SSL_accept rc=410 (0 is good, >=1 is handshake failure or fatal)

Fri Aug 25 05:19:51 2017 [33620315] io.c : 539: SSLErrorHandshake: failed to Call gsk_secure_socket_init() to.

Fri Aug 25 05:19:51 2017 [33620315] agent.c : 761: cleanup

Fri Aug 25 05:19:51 2017 [33620315] agent.c : 763: cleanup, then unmap_drives

One other thing I don't understand.

If I issue a bfagent -v command in z/OS USS shell, it show system ssl v1

UISC196:/u/jazz603/ccm: >bfagent -v

IBM Rational Build Forge Agent

8.0.0.3 GA (Build 0007)

Platform: Unix

SYSTEM SSL: TLSv1

UISC196:/u/jazz603/ccm: >

Our z/OS environment has system SSL TLS1.2

Sathya moorthy commented Aug 28 '17, 11:14 a.m.

I had a copy my conf file. I have enabled trace where you can get more info, reg the issue.

gsk_trace_file /tmp/rtc/gskssl.trc
gsk_trace 0xFF
gsk_ssl_key_location /etc/jazz405/ccm/bfagentssl.kdb
gsk_ssl_kdb_password 93a2686a707787d400f7f2d401194145b4e0c3cb695d0e1b58ec
gsk_keyring_label bfacert
gsk_ssl_protocol ALL
gsk_ssl_cipher_v2 6321
gsk_ssl_cipher_v3 0906030201
gsk_ssl_client_authentication false
gsk_password_encrypt true

You can comment this ssl_protocol TLSv1.2.

After the change, can you restart the build agent and check the logs and trace.

Regards,
Sathya

Eric Kung commented Aug 28 '17, 7:12 p.m. | edited Aug 28 '17, 7:13 p.m.

Hi, Sathya,

The 2 bfagent.conf lines gsk_trace_file and gsk_trace has no effect. therefore I added this line to startbfa.sh

export GSK_TRACE 0xFF, according to this manual https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.gska100/sssl2dia1023934.htm

The gsk trace shows system ssl v3, although a PTF OA39422 has been installed to enable TLS1.2

08/28/2017-17:21:26 Thd-0 INFO gsk_svc_init(): System SSL Version 3, Release 23, Service level OA39422

08/28/2017-17:21:26 Thd-0 INFO gsk_svc_init(): LE runtime level 0x410d0000, 31-bit addressing mode

08/28/2017-17:21:26 Thd-0 INFO gsk_svc_init(): STDOUT handle=1, STDERR handle=2, TRACE handle=4

08/28/2017-17:21:26 Thd-0 INFO gsk_dll_init_once(): Using variant character table for code set IBM-1047

08/28/2017-17:21:26 Thd-0 INFO gsk_dll_init_once(): Using local code page IBM-1047

08/28/2017-17:21:26 Thd-0 INFO gsk_dll_init_once(): Using ISO8859-1 for TELETEX string

08/28/2017-17:21:27 Thd-0 ERROR gsk_dll_init_once(): Unable to load crypto DLL

EDC5205S DLL module not found.

08/28/2017-17:21:27 Thd-0 INFO crypto_init(): SHA-1 crypto assist is available

08/28/2017-17:21:27 Thd-0 INFO crypto_init(): SHA-224 crypto assist is available

08/28/2017-17:21:27 Thd-0 INFO crypto_init(): SHA-256 crypto assist is available

08/28/2017-17:21:27 Thd-0 INFO crypto_init(): SHA-384 crypto assist is available

08/28/2017-17:21:27 Thd-0 INFO crypto_init(): SHA-512 crypto assist is available

08/28/2017-17:21:27 Thd-0 INFO crypto_init(): DES crypto assist is available

08/28/2017-17:21:27 Thd-0 INFO crypto_init(): DES3 crypto assist is available

08/28/2017-17:21:27 Thd-0 INFO crypto_init(): AES 128-bit crypto assist is available

08/28/2017-17:21:27 Thd-0 INFO crypto_init(): AES 256-bit crypto assist is available

08/28/2017-17:21:27 Thd-0 INFO crypto_init(): ICSF services are not available

08/28/2017-17:21:27 Thd-0 INFO gsk_dll_init_once(): Job name BLZBFA4, Process 05010114

08/28/2017-17:21:27 Thd-0 INFO gsk_dll_init_once(): GSKSRVR communication area at 00000000

Do you still have a gsk trace file available for us to compare ?

Can you access your z/OS USS shell to issue the command bfagent -v ? would like to see what SYSTEM SSL level in the command output.

Thank you for your great help.

Sathya moorthy commented Aug 29 '17, 6:56 a.m.

I don't have anymore access to the RTC Build agent. so i can't check the logs or issue commands. I'm answering to your questions based on my experience here.

I see on gsktrace shows the following:
ERROR gsk_dll_init_once(): Unable to load crypto DLL EDC5205S DLL module not found.

This means that the Security Level 3 FMID JCPT3D1 is not installed and that means you will only be able to use Base Security Level ciphers.

You will need to order FMID JCPT3D1 to use the stronger ciphers.

Are you doing everything on Mainframe or ADCD package?

Regards,
Sathya

Eric Kung commented Aug 29 '17, 8:46 p.m. | edited Aug 29 '17, 8:46 p.m.

Hi, Sathya

We are testing this on ADCD package, before move it to production mainframe.

Unfortunately ADCD package only have z/OS 1.13, there are many difference from a real mainframe environment.

Our sysprog also identified the problem with missing FMID. he has installed it.

We progressed a little, still have cipher issue though.

Thank you for your help :)

showing 5 of 9 show 4 more comments

4 answers

Most liked answers ↑|Newest answers|Oldest answers

permanent link

Jean-Bernard Curmi (141●6●2) | answered Aug 31 '17, 12:01 p.m.
JAZZ DEVELOPER

Could you please add

gsk_ssl_cipher_v3 0a

in your bfagent.conf file

Comments

Eric Kung commented Aug 31 '17, 7:11 p.m. | edited Aug 31 '17, 7:11 p.m.

Hi, Jean

We have resolved cipher issue, but stuck with RTC side can not validate Build agent certificate.

SSL trace message

ERROR read_v3_alert(): SSL V3 alert 46 received from 9.32.157.211[56180]

Do you think RTC server need to be restarted once a cerficate is imported according to this article https://jazz.net/library/article/607

permanent link

Jean-Bernard Curmi (141●6●2) | answered Sep 01 '17, 5:55 a.m.
JAZZ DEVELOPER

The RTC server doesn't need to be restarted.

I am afraid the doc you followed to create the certificate is not up to date,

The certificate must be generated with a Rsa Key Size of 2048 and Signature Digest Type of SHA-256

ie

permanent link

Jean-Bernard Curmi (141●6●2) | answered Sep 01 '17, 5:56 a.m.
JAZZ DEVELOPER

       RSA Key Size

   1 - 1024-bit key
   2 - 2048-bit key
   3 - 4096-bit key

Select RSA key size (press ENTER to return to menu): 2

       Signature Digest Type

   1 - SHA-1
   2 - SHA-224
   3 - SHA-256
   4 - SHA-384
   5 - SHA-512

Select digest type (press ENTER to return to menu): 3

permanent link

Eric Kung (11●1) | answered Sep 01 '17, 6:09 a.m.

Yes, I am aware about the newer digest type and stronger key length is required.

I used exactly what you listed.

This is part of the certificate details

Signature algorithm: sha256WithRsaEncryption

Issuer unique ID: None

Subject unique ID: None

Public key algorithm: rsaEncryption

Public key size: 2048

I still think the cert isn't imported in RTC server side correctly.

Your answer

Register or log in to post your answer.

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.

It's all about the answers!

Related questions

RTC Build agent on z/OS SSL communication RC:410 and RC:420

May use a gskkyman key database instead of SAF keyring

4 answers

Your answer