IEBCOPY failure to copy load modules during promotion
We have a build service account (SVC_JAZZ) that we use to run our mainframe builds. We were trying to setup a "primer" component promotion definition that would seed the QA stream from the Production stream. When the component promotion was running it was able to copy source files from the prod to the QA PDSs but when it tried to copy the load modules, it failed with a return code 20 as follows:
BPXM023I (SVC_JAZZ) Gateway Frontend started - z/OS V2R1 01JAN12 Base
BPXM023I (SVC_JAZZ) CGI_CEATSO is FALSE
BPXM023I (SVC_JAZZ) ISPZINT invoking original gateway interface module ISPZINO
* [exec] Copy failed for member 'JKEMLIST' Return Code : 20
* [exec] ISRZ002 : IEBCOPY interface failed
* [exec] Copy failed for member 'JKEMLIST' Return Code : 20
* [exec]
* [exec] ISPD117
* [exec] The initially invoked CLIST ended with a return code = 20
Digging through the system logs we see the following messages around the same time that the above promotion failed:
BPXM023I (SVC_JAZZ) Gateway Frontend started - z/OS V2R1 01JAN12 Base
BPXM023I (SVC_JAZZ) CGI_CEATSO is FALSE
BPXM023I (SVC_JAZZ) ISPZINT invoking original gateway interface module ISPZINO
Suspecting a permissions issue, I switched to use my RACF ID for the build agent and the promotion ran successfully. I don't recall seeing anything in the setup instructions about needing to configure RACF for CEATSO for the build service account. Anyone have any suggestions on what we need to do? We can open a PMR or work item on jazz.net if need be.
8 answers
Hi Alex
The CEATSO facility is new to ISPF, and the gateway, as configured for RTC has not yet been tested with it. It is on my todo list. However I see your invocation is using the original Gateway code by the messages above. A return code 20 from the LMCOPY is a severe error, for which unfortunately there are no additional messages. You said you changed IDs and then the promotion ran. So can you reproduce by switching the agent id back to the original? Are there any RACF messages in the system log following the ISPZINT message you spotted?
Liam
The CEATSO facility is new to ISPF, and the gateway, as configured for RTC has not yet been tested with it. It is on my todo list. However I see your invocation is using the original Gateway code by the messages above. A return code 20 from the LMCOPY is a severe error, for which unfortunately there are no additional messages. You said you changed IDs and then the promotion ran. So can you reproduce by switching the agent id back to the original? Are there any RACF messages in the system log following the ISPZINT message you spotted?
Liam
Comments
Is there something different with the service userid. Does it have a valid TSO segment in RACF? Is there a difference between it and your id with relation to the logon proc specified? Or any other settings in the TSO segment?
I don't see those messages on my z/OS 2.1 system. I am checking with the ISPF team if they know anything.
I don't see those messages on my z/OS 2.1 system. I am checking with the ISPF team if they know anything.
Comments
The fact that this seems to be a permissions issue, I am surprised there is nothing in the system log. Does the functional ID have the authority to copy to the target data set? RTC cannot suppress system messages such as RACF errors or E37 for example. It does at times find it more difficult to capture them in the build output. Although we are tryiung to address that as there is a recent enhancement to capture *37 messages.
I can send you a module with Trace turned on, and some extra diagnostics, maybe ZERRLM has more info. Can you tell me what version of RTC you are running.
The CEATSO messages are a red herring. Basically some of the ISPF Gateway support for CEATSO exists in z/OS 2.1 but is disabled and CSI_CEATSO is set to FALSE always. You are only seeing that because you are getting an error running the Gateway.
Are you saying that the functional ID works OK for non load modules, and only fails for load modules? Are you sure there are no RACF (or whatever security server you arre using) profiles that are protecting the load library?
I can send you a module with Trace turned on, and some extra diagnostics, maybe ZERRLM has more info. Can you tell me what version of RTC you are running.
The CEATSO messages are a red herring. Basically some of the ISPF Gateway support for CEATSO exists in z/OS 2.1 but is disabled and CSI_CEATSO is set to FALSE always. You are only seeing that because you are getting an error running the Gateway.
Are you saying that the functional ID works OK for non load modules, and only fails for load modules? Are you sure there are no RACF (or whatever security server you arre using) profiles that are protecting the load library?
Comments
Hi Alex , you are right , in my environment I had exactly the same error but using a JCL that invoke the BUZTOOL clist In my environment I solved my problem simply adding the region=0M statement at step level.
But , in your environment (a promotion build in RTC EE ) it's important to understand HOW we can change the region (and/or increase the memory ) during the buztool.sh invocation.
I'm searching for a solution but I need to prepare my environment : I'm thinking to investigate in this area :
Build definitions --> z/OS dependency build --> Ant with Enterprise extension configuration ( maybe ANT arguments, but I'need to verify )
But , in your environment (a promotion build in RTC EE ) it's important to understand HOW we can change the region (and/or increase the memory ) during the buztool.sh invocation.
I'm searching for a solution but I need to prepare my environment : I'm thinking to investigate in this area :
Build definitions --> z/OS dependency build --> Ant with Enterprise extension configuration ( maybe ANT arguments, but I'need to verify )
Hi Alex,
The promotion module didn't get the TRACE upgrade like the Packaging/Deployment stuff because it is a lot simpler and had not changed until recently due to some upgrades for Sequential file support. They are correct and there is a REXX involved, BLZPRMCP, and it is compiled. But it does not have the logic to turn tracing on currently. I will create a Task to put the trace stuff in.
I have sent a test load module with trace activated to your email.
I can reproduce a RC=20 for stuff like space issues. I get to see the D37, where as you only get IEBCOPY interface failed which is the short message from ISPF. (The ZERRLM is in this version only, the rest is as shipped) :
Promote Manifest file : /u/dohertl/Test/Promotion502/promotionInfo.xml
Copying the following members from 'DOHERTL.BUILD502.DBRM' to 'DOHERTL.PROD502.DBRM'
Copy EPSCMORT
Copying the following members from 'DOHERTL.BUILD502.BMSCPYBK' to 'DOHERTL.PROD502.BMSCPYBK'
Copy EPSMLIS
Copy EPSMORT
Copying the following members from 'DOHERTL.BUILD502.OBJ' to 'DOHERTL.PROD502.OBJ'
Copy EPSMLIST
Copy EPSNBRVL
Copying the following members from 'DOHERTL.BUILD502.LOAD' to 'DOHERTL.PROD502.LOAD'
Copy EPSMLIST
ZERRLM : A system abend occurred. Press Help key for further information.
Copy failed for member 'EPSMLIST' Return Code : 20
ISRZ002 : System abend '0D37'
Copy failed for member 'EPSMLIST' Return Code : 20
So you can see the D37 for example. But maybe there are some other issues that don't put a message out. Like incompatible load library types? PDSE to PDS for example?
As it only happens with load modules can you send me the attributes of both your source and target data sets? Screen captures from 3.4 would be good. Are they both PDS or PDSE or are they different?
Additionally can you see if there are any data sets on your system that have a low level qualifier of IEBCOPY, for example DOHERTL.SPF123.IEBCOPY. ISPF will sometimes create these to hole the IEBCOPY messages if an ISPF COPY fails.
One orther thing that may (or may not) be relevant. We noticed an interesting issue with z/OS 2.1 on an IEBCOPY (which LMCOPY uses under the covers). It was actually in an SMP/E ACCEPT from a PDSE to a PDS. If the region size was too small it abended. The underlying issue was the REGION size being used was insufficient. In the IEBCOPY case they it was picking up a really small default region size. In the ACCEPT, the REGION size was being capped by the IEFUSI. So do you have the IEFUSI exit active on your system?
I don't think the Trace is going to give us any extra information, but lets check anyway. I am more interested in the attributes of your source and target load libraries.
The promotion module didn't get the TRACE upgrade like the Packaging/Deployment stuff because it is a lot simpler and had not changed until recently due to some upgrades for Sequential file support. They are correct and there is a REXX involved, BLZPRMCP, and it is compiled. But it does not have the logic to turn tracing on currently. I will create a Task to put the trace stuff in.
I have sent a test load module with trace activated to your email.
I can reproduce a RC=20 for stuff like space issues. I get to see the D37, where as you only get IEBCOPY interface failed which is the short message from ISPF. (The ZERRLM is in this version only, the rest is as shipped) :
Promote Manifest file : /u/dohertl/Test/Promotion502/promotionInfo.xml
Copying the following members from 'DOHERTL.BUILD502.DBRM' to 'DOHERTL.PROD502.DBRM'
Copy EPSCMORT
Copying the following members from 'DOHERTL.BUILD502.BMSCPYBK' to 'DOHERTL.PROD502.BMSCPYBK'
Copy EPSMLIS
Copy EPSMORT
Copying the following members from 'DOHERTL.BUILD502.OBJ' to 'DOHERTL.PROD502.OBJ'
Copy EPSMLIST
Copy EPSNBRVL
Copying the following members from 'DOHERTL.BUILD502.LOAD' to 'DOHERTL.PROD502.LOAD'
Copy EPSMLIST
ZERRLM : A system abend occurred. Press Help key for further information.
Copy failed for member 'EPSMLIST' Return Code : 20
ISRZ002 : System abend '0D37'
Copy failed for member 'EPSMLIST' Return Code : 20
So you can see the D37 for example. But maybe there are some other issues that don't put a message out. Like incompatible load library types? PDSE to PDS for example?
As it only happens with load modules can you send me the attributes of both your source and target data sets? Screen captures from 3.4 would be good. Are they both PDS or PDSE or are they different?
Additionally can you see if there are any data sets on your system that have a low level qualifier of IEBCOPY, for example DOHERTL.SPF123.IEBCOPY. ISPF will sometimes create these to hole the IEBCOPY messages if an ISPF COPY fails.
One orther thing that may (or may not) be relevant. We noticed an interesting issue with z/OS 2.1 on an IEBCOPY (which LMCOPY uses under the covers). It was actually in an SMP/E ACCEPT from a PDSE to a PDS. If the region size was too small it abended. The underlying issue was the REGION size being used was insufficient. In the IEBCOPY case they it was picking up a really small default region size. In the ACCEPT, the REGION size was being capped by the IEFUSI. So do you have the IEFUSI exit active on your system?
I don't think the Trace is going to give us any extra information, but lets check anyway. I am more interested in the attributes of your source and target load libraries.
Comments
showing 5 of 6
show 1 more comments
Liam,
We just discovered one thing that could be the culprit. The functional ID we have has an 8 character name. Although RACF doesn't have a problem with an 8 character name and, in fact, that appears to be the recommended user name length for service accounts, it appears that TSO does have a problem with it and only supports 7 character names. We will try requesting a 7 character user name and see if that helps with this issue. If it turns out to be the case, you may want to mention a blurb about this in your docs (even though it's really a TSO issue but may not be obvious that your functional IDs need to use short names IFF we intend to use them for ISPF gateway processing.
Comments
Liam,
Good news! We finally got it working. So it was a combination of the shortened ID and a modification in RACF to set the prefix for that user. I don't have the exact details but hopefully you know how to do that so you can properly document it as part of the required set up steps. Also, please provide the ZERRLM tracing as an option in the product. We wouldn't have been able to diagnose this problem without it.