It's all about the answers!

Ask a question

IEBCOPY failure to copy load modules during promotion


Alex Akilov (1211224) | asked Jan 29 '15, 5:41 p.m.
edited Jan 29 '15, 5:45 p.m.
 We have a build service account  (SVC_JAZZ)  that we use to run our mainframe builds.  We were trying to setup a "primer" component promotion definition that would seed the QA stream from the Production stream.  When the component promotion was running it was able to copy source files from the prod to the QA PDSs but when it tried to copy the load modules, it failed with a return code 20 as follows:

*      [exec]   Copy failed for member 'JKEMLIST' Return Code : 20 
*      [exec]   ISRZ002  : IEBCOPY interface failed
*      [exec]   Copy failed for member 'JKEMLIST' Return Code : 20 
*      [exec]    
*      [exec]   ISPD117                             
*      [exec]   The initially invoked CLIST ended with a return code = 20

Digging through the system logs we see  the following messages around the same time  that the above promotion failed:

BPXM023I (SVC_JAZZ) Gateway Frontend started - z/OS V2R1 01JAN12 Base
BPXM023I (SVC_JAZZ) CGI_CEATSO is FALSE
BPXM023I (SVC_JAZZ) ISPZINT invoking original gateway interface module ISPZINO

Suspecting a permissions issue, I switched to use my RACF ID for the build agent and the promotion ran successfully.  I don't recall seeing anything in the setup instructions about needing to configure RACF for CEATSO for the build service account.  Anyone have any suggestions on what we need to do?  We can open a PMR or work item on jazz.net if need be.



8 answers



permanent link
Liam Doherty (2312) | answered Jan 29 '15, 9:45 p.m.
JAZZ DEVELOPER
Hi Alex

The CEATSO facility is new to ISPF, and the gateway, as configured for RTC has not yet been tested with it. It is on my todo list. However I see your invocation is using the original Gateway code by the messages above. A return code 20 from the LMCOPY is a severe error, for which unfortunately there are no additional messages. You said you changed IDs and then the promotion ran. So can you reproduce by switching the agent id back to the original? Are there any RACF messages in the system log following the ISPZINT message you spotted?

Liam



Comments
Alex Akilov commented Jan 30 '15, 12:08 p.m.

 Liam,


Here's the response from our sysprog:

At the time that the 3 messages appear on the system log, there are no RACF messages appearing at all.  Those three messages seem to appear in isolation; there are no other messages of any type that look like they could be related to them.

That said, there must be something with the service id that is triggering the difference; i.e., we do not get those messages when you run the promotion under your userid, they only appear when running under the service id.  And, likewise, the IEBCOPY of the load modules (and ONLY the load modules) only fails then running under the service id.



Alex Akilov commented Jan 30 '15, 12:08 p.m.

 Rest of the response from the sysprog:

So, if the IBM guys on the forum could give us some idea of what difference in the RACF userid might be triggering whatever logic is generating the CEATSO message, that would help to identify what we need to look at specifically with the userid.


I will continue to look at things on my end, but I’m really hoping that they can provide some information to give me a better direction in which to look.


Alex Akilov commented Jan 30 '15, 12:15 p.m.

 Liam,

Since the promotion ran all the way through using my streams and PDSs, it doesn't look like it thinks it has anything left to do if I now rerun it again using the service account.  However,  if need be, we can still replicate the issue on my colleague's streams/datasets since he hasn't yet modified his build agent to run with his credentials and allow the promotion to complete.


permanent link
Liam Doherty (2312) | answered Feb 02 '15, 3:19 a.m.
JAZZ DEVELOPER
Is there something different with the service userid. Does it have a valid TSO segment in RACF? Is there a difference between it and your id with relation to the logon proc specified? Or any other settings in the TSO segment?

I don't see those messages on my z/OS 2.1 system. I am checking with the ISPF team if they know anything.

Comments
Alex Akilov commented Feb 03 '15, 12:38 p.m.

 Are you saying that a service account should be able to sign on to TSO to perform ISPF gateway types of processing?  I'm concerned that this is probably going to be viewed as a security exposure.  Hence, I cannot sign in to TSO with the service account like I can with my RACF account.

Neither I nor our sys admin know the RACF rules in place for a regular user vs. this service account user.  Security is administered by our security group.  If we need to engage them, is there a list of RACF settings that are required for the service account that I need to pass along to them?


Alex Akilov commented Feb 03 '15, 3:25 p.m.

 I was told that our service account does have a TSO segment although the login proc is a bit different from the one a human user uses.  One thing we ran into in the past had to do with the allocation of a dataset like SVC_JAZZ.ISPF.VCMISPF.xxxe but found that setting team.enterprise.build.ant.myISPFBinPath to a concatenation allowed us to override the default HLQ that should be used (since SVC_JAZZ is not a human account, it doesn't have an HLQ/alias allotted to it).  However, setting that property didn't seem to help in this case for some reason. 


Alex Akilov commented Feb 03 '15, 3:31 p.m.
I was told that our service account does have a TSO segment although the login proc is a bit different from the one a human user uses.  

We suspect that this issue has something to do with the ISPF gateway wanting to allocate a dataset like SVC_JAZZ.ISPF.VCMISPF.xxx which was failing since SVC_JAZZ is not a human account, and as such it doesn't have an HLQ/alias allotted to it.  We found that setting team.enterprise.build.ant.myISPFBinPath to a concatenation allowed us to override the default HLQ that should be used. This worked for us in the past with a dependency build trying to call out to a Rexx. However, setting that property now in the promotion definition didn't seem to help for some reason. 
BTW, the information about that allocation failing is only visible in the system log and not replicated in the build log.  This is an issue and you guys need to figure out a way to capture that information and communicate it in the build log.  Otherwise,  the perception is that the build feature isn't as powerful as JCL where all info related to the failure is available

Alex Akilov commented Feb 03 '15, 4:42 p.m.

 Update: looks the the ISPF.VCMISPF error was not relevant.  It only happened when we didn't have the myISPFBinPath set but the load module IEBCOPY failure seems to occur with the service account regardless of that allocation issue.  Something unique to copying load modules (since other types copy just fine) but we can't seem to pinpoint what it is since the error messages that are being returned are not very helpful (RC=20 and no other info). Help!


permanent link
Liam Doherty (2312) | answered Feb 04 '15, 12:48 a.m.
JAZZ DEVELOPER
The fact that this seems to be a permissions issue, I am surprised there is nothing in the system log. Does the functional ID have the authority to copy to the target data set? RTC cannot suppress system messages such as RACF errors or E37 for example. It does at times find it more difficult  to capture them in the build output. Although we are tryiung to address that as there is a recent enhancement to capture *37 messages.

I can send you a module with Trace turned on, and some extra diagnostics, maybe ZERRLM has more info. Can you tell me what version of RTC you are running.

The CEATSO messages are a red herring. Basically some of the ISPF Gateway support for CEATSO exists in z/OS 2.1 but is disabled and CSI_CEATSO is set to FALSE always. You are only seeing that because you are getting an error running the Gateway.

Are you saying that the functional ID works OK for non load modules, and only fails for load modules? Are you sure there are no RACF (or whatever security server you arre using)  profiles that are protecting the load library?

Comments
Alex Akilov commented Feb 04 '15, 9:53 a.m.

 Liam,


We're on v5.0.2.  Yes, I'm saying that the ID is only not able to copy load modules.  We're going through Robin's/Jorge's EE build admin workshop trying to do the primer promotion (lab 4).  The promotion job (running under the functional ID) created the target QA datasets and copied sources fine but failed on copying the load modules.


Alex Akilov commented Feb 05 '15, 12:52 p.m.
Liam,

This is not a permissions issue.  We've confirmed this by simply running the IEBCOPY using the functional ID via JCL and it copied the LOAD modules just fine.  Can you please send that trace module you mentioned?  

Also, IBM (in a PMR) suggested that we set a build property called TRACE with a value of TRACE(ALL), any ideas if this is relevant in the RTC case?  They seem to think that there is a REXX involved in the promotion process and setting that build property will turn on tracing for any REXXes that are running.


permanent link
giuseppe ruggeri (3137) | answered Feb 04 '15, 5:04 a.m.
Hi Alex , I had the same problem and I've solved it adding the statement REGION=0M to the step


Comments
Alex Akilov commented Feb 04 '15, 9:56 a.m.

giuseppe,


We're running a promotion build in RTC EE so I'm not exactly sure where you added that step.  Can you please clarify exactly what you did?


Liam Doherty commented Feb 04 '15, 9:56 p.m.
JAZZ DEVELOPER

Alex, what is the logon region size used by both IDs? Is the one that worked greater than the functional ID that failed?


permanent link
giuseppe ruggeri (3137) | answered Feb 05 '15, 6:52 a.m.
Hi Alex , you are right , in my environment I had  exactly the same error but using  a JCL that invoke   the BUZTOOL clist In my environment I solved my problem simply adding the region=0M statement at step level.

 But  , in your environment (a promotion build in RTC EE ) it's important to understand HOW  we can change the region (and/or increase the memory ) during the buztool.sh invocation.

I'm searching for a solution but I need to prepare my environment : I'm thinking to investigate in this area :
 Build definitions --> z/OS dependency build --> Ant with Enterprise extension configuration ( maybe ANT arguments, but I'need to verify ) 

permanent link
Liam Doherty (2312) | answered Feb 05 '15, 11:06 p.m.
JAZZ DEVELOPER
Hi Alex,

The promotion module didn't get the TRACE upgrade like the Packaging/Deployment stuff because it is a lot simpler and had not changed until recently due to some upgrades for Sequential file support. They are correct and there is a REXX involved, BLZPRMCP, and it is compiled. But it does not have the logic to turn tracing on currently. I will create a Task to put the trace stuff in.

I have sent a test load module with trace activated to your email.

I can reproduce a RC=20 for stuff like space issues. I get to see the D37, where as you only get IEBCOPY interface failed which is the short message from ISPF. (The ZERRLM is in this version only, the rest is as shipped) :

Promote Manifest file : /u/dohertl/Test/Promotion502/promotionInfo.xml                      
Copying the following members from 'DOHERTL.BUILD502.DBRM' to 'DOHERTL.PROD502.DBRM'        
Copy EPSCMORT                                                                               
Copying the following members from 'DOHERTL.BUILD502.BMSCPYBK' to 'DOHERTL.PROD502.BMSCPYBK'
Copy EPSMLIS                                                                                
Copy EPSMORT                                                                                
Copying the following members from 'DOHERTL.BUILD502.OBJ' to 'DOHERTL.PROD502.OBJ'          
Copy EPSMLIST                                                                               
Copy EPSNBRVL                                                                               
Copying the following members from 'DOHERTL.BUILD502.LOAD' to 'DOHERTL.PROD502.LOAD'        
Copy EPSMLIST                                                                               
ZERRLM : A system abend occurred. Press Help key for further information.                   
                                                                                            
                                                                                            
                                                                                            
Copy failed for member 'EPSMLIST' Return Code : 20                                          
ISRZ002  : System abend '0D37'                                                              
Copy failed for member 'EPSMLIST' Return Code : 20                                          
                                                                                            
So you can see the D37 for example. But maybe there are some other issues that don't put a message out. Like incompatible load library types? PDSE to PDS for example?

As it only happens with load modules can you send me the attributes of both your source and target data sets? Screen captures from 3.4 would be good. Are they both PDS or PDSE or are they different?

Additionally can you see if there are any data sets on your system that have a low level qualifier of IEBCOPY, for example DOHERTL.SPF123.IEBCOPY. ISPF will sometimes create these to hole the IEBCOPY messages if an ISPF COPY fails.

One orther thing that may (or may not) be relevant. We noticed an interesting issue with z/OS 2.1 on an IEBCOPY (which LMCOPY uses under the covers). It was actually in an SMP/E ACCEPT from a PDSE to a PDS. If the region size was too small it abended. The underlying issue was the REGION size being used was insufficient. In the IEBCOPY case they it was picking up a really small default region size. In the ACCEPT, the REGION size was being capped by the IEFUSI. So do you have the IEFUSI exit active on your system?

I don't think the Trace is going to give us any extra information, but lets check anyway. I am more interested in the attributes of your source and target load libraries.

Comments
Alex Akilov commented Feb 06 '15, 9:53 a.m. | edited Feb 06 '15, 10:07 a.m.

 Liam,


Here are the screenshots you requested (they are both PDS/Es):

Target Load Mod


Source Load Mod

Again, I don't understand how space or type of data set would matter if I can simply switch the build user from the functional ID to my own and then the promotion works (using the same data sets).


Alex Akilov commented Feb 06 '15, 5:10 p.m.

 Liam,

I sent you the build log offline with the tracing turned on.  From it, we see that during the failure, there was an attempt to allocate a listing dataset for IEBCOPY.  The naming convention it was using for this dataset was invalid on our system so it failed to allocate.   Do you know how the dataset name is computed?  Is it based on the HLQ for the user ID under whose credentials the build is running?  I don't see any IEBCOPY data sets under my user's HLQ so I'm wondering if it only attempts to allocate it if there was a failure (or if it deletes it at the completion of the copy).  Can you shed some light on what the lifecycle of these datasets is?  Is this documented somewhere?


Alex Akilov commented Feb 06 '15, 5:15 p.m.

 Also, can you clarify how the region size comes into play here?  As I indicated,  the functional ID does not logon to TSO, it's only used via USS/omvs.  In which case,  when is the region size used?  At any rate, we checked and even though the functional ID isn't allowed to logon it has the default region size of 4M defined (which is the same size used by the other human user IDs we were able to substitute for the functional ID and complete the promotion successfully).


Liam Doherty commented Feb 08 '15, 10:33 p.m.
JAZZ DEVELOPER

Hey Alex, yeah I realized after I posted that the data set type should not be the issue as it works with your normal ID. The listing failure is because ISPF is trying to allocate that file based on the userid running the build. I am guessing that userid, called BUILD(?) does not have an Alias so cannot allocate any data sets? Can you do a TSO PROFILE and see what the PREFIX is set to for the ID. Any chance you can set the prefix to a HLQ that userid can write to? What happens is the IEBCOPY is failing and ISPF is trying to write the IEBCOPY messages out to a data set. These IEBCOPY messages will hopefully tell us what is wrong. The data set only gets allocated by ISPF (not RTC) in case of a failure and will hang around until you delete them. We need to see the contents of the IEBCOPY failure data set.

I will also send you a test REXX that will hopefully set this up so we can run it from the userid directly and see if we can reproduce there. I think you answered your own question on the region size, it is defined in the TSO segment in RACF. Although you say the ID is not allowed to log on? Can you log into it (BUILD) for a test?


Liam Doherty commented Feb 08 '15, 11:06 p.m.
JAZZ DEVELOPER

In fact in doing a test where I set my prefix to something I cannot write to, I can get the same error. Even though if I know the LMCOPY would work OK, the fact it cannot allocate the listing causes the problem. So I suspect that is the issue here. Because BUILD cannot allocate a data set the IEBCOPY fails. The peculiar thing is why the LMCOPYs work OK for non-LOAD types but fail for LOADs.

Is it possible to log in to the BUILD userid and change it's TSO prefix to a HLQ it can write to?


Liam Doherty commented Feb 09 '15, 12:14 a.m.
JAZZ DEVELOPER

So I wonder if  IEBCOPY, under the covers, is allocating some kind of work data set for the copy of LOAD modules? Although running IEBCOPY in batch I can't see any allocations. If I run a copy of a DBRM with an invalid  prefix, the copy works just fine, so this points to the fact that something different is happening under the covers for LOAD modules.

showing 5 of 6 show 1 more comments

permanent link
Alex Akilov (1211224) | answered Feb 10 '15, 5:23 p.m.
 Liam,

We just discovered one thing that could be the culprit.  The functional ID we have has an 8 character name.  Although RACF doesn't have a problem with an 8 character name and, in fact, that appears to be the recommended user name length for service accounts, it appears that TSO does have a problem with it and only supports 7 character names.  We will try requesting a 7 character user name and see if that helps with this issue.  If it turns out to be the case, you may want to mention a blurb about this in your docs (even though it's really a TSO issue but may not be obvious that your functional IDs need to use short names IFF we intend to use them for ISPF gateway processing.

Comments
Liam Doherty commented Feb 10 '15, 8:09 p.m.
JAZZ DEVELOPER

Hi Alex, That could be the issue, or even an additional issue. I will be interested to see what happens. In reproducing I used my normal ID, with an invalid prefix. So I suspect there are a number of things around the userid used that may cause problems. Whatever occurs I checked our doc and can't see any words relating to this. So once we are square with the cause I will raise a Doc defect to get it fixed.


Alex Akilov commented Feb 12 '15, 11:26 a.m.

 We've made some progress although not fully there yet.  Shortening the userid name seems to have helped some.  We're not getting the Invalid TSO user error anymore and even though we specified a TSO profile that uses a valid prefix for the user, it seems to default to a prefix that's now the new user name (instead of BATCH which it was using previously).  So now we need to figure out why the profile isn't taking effect. 


permanent link
Alex Akilov (1211224) | answered Feb 20 '15, 11:08 a.m.
 Liam,

Good news!  We finally got it working.  So it was a combination of the shortened ID and a modification in RACF to set the prefix for that user.  I don't have the exact details but hopefully you know how to do that so you can properly document it as part of the required set up steps.  Also, please provide the ZERRLM tracing as an option in the product.  We wouldn't have been able to diagnose this problem without it.

Your answer


Register or to post your answer.