It's all about the answers!

Ask a question

About error messages with https:///ccm/scr & https:///rm/scr


long TRUONG (363489129) | asked Apr 19 '16, 12:16 p.m.
 We get same issue as per post 120457, however our is hanging our server (RTC/RRC 5.0.2 Windows/Tomcat) every 2 hrs or so requiring a restart. And there was no real answer to above post.

We have elevated one of our existing PMR to severity 1 server down, but hoping we may be lucky get an answer here before getting a call from support.

It started with a hang when the wrong port (143) was inadvertently saved for the DB ccm on the ccm DB connection web page. We fixed the port (to 1433) in ccm teamserver.properties and restarted the app (yes, it required a restart, we couln't get back in the DB connection page to correct the mistake). It restarted OK.

But then all hell broke loose, it has been hanging about every 2 hrs or so with CPU (x4 on a VM) peaked and persisted at 100%. It has to be restarted every time. 

Typical errors around the hangs (time) in JTS log:
Line 92537: 2016-04-18 15:58:03,077 [http-bio-443-exec-568 @@ 15:37     
e2xjeya <Initial Page Load@9f67d68c-edd8-42e9-a73a-04c762188750>        
/ccm/web/projects/PAS] ERROR license.behavior.                          
SCRBasedCompatibleLicenseBehavior  - Error fetching the scr document    
for application at url: "https://rtc-web1.ent.rt.csaa.com/ccm/scr"      
Line 92700: 2016-04-18 15:58:31,406 [http-bio-443-exec-580 @@ 15:57    
gg27pan <Jakarta Commons-HttpClient/3.1@10.37.128.136> /ccm/process-    
runtime/operation-reports] ERROR license.behavior.                      
SCRBasedCompatibleLicenseBehavior  - Error fetching the scr document    
for application at url: "https://rtc-web1.ent.rt.csaa.com/ccm/scr"      
Line 92863: 2016-04-18 15:58:35,586 [http-bio-443-exec-621 @@ 15:53    
<unauthenticated> <Requirements Management/5.0.2@172.26.167.65>         
/rm/process-security/operations/com.ibm.rrs.saveComment/report] ERROR   
license.behavior.SCRBasedCompatibleLicenseBehavior  - Error fetching    
the scr document for application at url: "https://rtc-web1.ent.rt.csaa. 
com/rm/scr"                                                             
Line 93023: 2016-04-18 15:59:03,541 [http-bio-443-exec-551 @@ 15:39    
gq6ilam <Initial Page Load@efa528c8-ee6d-4a77-9407-2473035c65c5>        
/rm/web] ERROR license.behavior.SCRBasedCompatibleLicenseBehavior  -    
Error fetching the scr document for application at url: "https://rtc-   
web1.ent.rt.csaa.com/ccm/scr"                                           
Line 93186: 2016-04-18 16:00:22,645 [http-bio-443-exec-492 @@ 15:48    
<unauthenticated> <Initial Page Load@eeb058c9-13ef-4151-b8af-           
a59659fe489f> /ccm/auth/authrequired] ERROR license.behavior.           
SCRBasedCompatibleLicenseBehavior  - Error fetching the scr document    
for application at url: "https://rtc-web1.ent.rt.csaa.com/ccm/scr"      
Line 93349: 2016-04-18 16:01:20,114 [   jts: Remote license updater]   
WARN ory.service.internal.license.FloatingLicenseClient  - CRJAZ1272I   
The floating license available from "https://rtc-web1.ent.rt.csaa.      
com/jts/service/com.ibm.team.repository.service.internal.license.       
IFloatingLicenseService/licenses/com.ibm.team.rrc.analyst.floating-     
v3_b92a5960-cb65-11df-bd3b-0800200c9a66?includeFullJar=true" could not  
be downloaded. The floating license at that location will be ignored.   
Line 93372: 2016-04-18 16:01:20,426 [http-bio-443-exec-455 @@ 15:29    
<unauthenticated> <Initial Page Load@5d86d8da-4352-4bf7-8b8a-           
7effa4f3e741> /ccm/auth/authrequired] ERROR license.behavior.           
SCRBasedCompatibleLicenseBehavior  - Error fetching the scr document    
for application at url: "https://rtc-web1.ent.rt.csaa.com/rm/scr"       
Line 93532: 2016-04-18 16:02:41,686 [http-bio-443-exec-513 @@ 15:28    
<unauthenticated> <Initial Page Load@c2a919a8-bfa9-4e3f-bd23-           
692a2a9a7752> /jts/auth/authrequired] ERROR license.behavior.           
SCRBasedCompatibleLicenseBehavior  - Error fetching the scr document    
for application at url: "https://rtc-web1.ent.rt.csaa.com/ccm/scr"      
Line 93695: 2016-04-18 16:03:28,236 [http-bio-443-exec-580 @@ 16:01    
easgovi <com.ibm.team.workitem.viewWorkItem@84e1c660-61ea-4858-89bb-    
182d9888c2bb> /ccm/service/com.ibm.team.workitem.common.internal.rest.  
IWorkItemRestService/workItemEditableProperties] ERROR license.behavior.
SCRBasedCompatibleLicenseBehavior  - Error fetching the scr document    
for application at url: "https://rtc-web1.ent.rt.csaa.com/ccm/scr"      
Line 93884: 2016-04-18 16:04:02,836 [http-bio-443-exec-464 @@ 15:01    
e2xjeya <Initial Page Load@1bde0ec6-dfb8-4419-b998-c04c063ba89b>        
/ccm/web/projects/PAS] ERROR license.behavior.                          
SCRBasedCompatibleLicenseBehavior  - Error fetching the scr document    
for application at url: "https://rtc-web1.ent.rt.csaa.com/ccm/scr"      
Line 94047: 2016-04-18 16:04:02,992 [http-bio-443-exec-247 @@ 14:10    
gjexbas <Initial Page Load@3516a49f-36d6-4e0b-8245-9633ee301bf7>        
/ccm/web/projects/PAS] ERROR license.behavior.                          
SCRBasedCompatibleLicenseBehavior  - Error fetching the scr document    
for application at url: "https://rtc-web1.ent.rt.csaa.com/rm/scr"       
Line 94209: 2016-04-18 16:04:05,925 [http-bio-443-exec-247 @@ 14:10    
gjexbas <Initial Page Load@3516a49f-36d6-4e0b-8245-9633ee301bf7>        
/ccm/web/projects/PAS]  WARN .team.repository.servlet.                  
AbstractTeamServerServlet  - CRJAZ1163I LicenseNotGrantedException      
processing GET request for com.ibm.team.repository.service.internal.    
license.ILicenseRestService.getAssertLicense(). CRJAZ1170I The request  
was made by user "rm_user" from "P01APW437.ent.rt.csaa.com".CRJAZ1166I  
The stack trace hash is AE2A456ECD912724104FC0D5C83E45E901DCA905.       
Line 94382: 2016-04-18 16:06:03,548 [  Start Level Event Dispatcher]   
INFO nternal.app.servlet.util.ServletDispatchingHandler  - CRJZS0315I   
Servlet Dispatching Enabled                                             
Line 94383: 2016-04-18 16:06:03,548 [  Start Level Event Dispatcher]   
INFO nternal.app.servlet.util.ServletDispatchingHandler  - CRJZS0318I   
Servlet Loopback Disabled                                               
Line 94384: 2016-04-18 16:06:09,898 [Component Resolve Thread (Bundle  
94)]  INFO nternal.OverridablePropertyFileServerConfiguration  -        
Line 94387: 2016-04-18 16:06:09,898 [Component Resolve Thread (Bundle  
94)]  INFO nternal.OverridablePropertyFileServerConfiguration  -        
CRJAZ1363I Loading the configuration from "file:/D:                     
/IBM/rtc_502/JazzTeamServer/server/conf/jts/teamserver.properties".     
Line 94388: 2016-04-18 16:06:43,438 [          http-bio-443-exec-74]   
INFO pository.service.internal.ServerDescriptionService  - CRJAZ1779I   
This server is configured as a JTS.                                     
Line 94389: 2016-04-18 16:06:44,545 [          http-bio-443-exec-74]   
INFO rvice.mapping.internal.ServerRenameProgressService  - CRJAZ2558I   
Setting the local server rename state to false and the                  
openServerDescriptionServiceTemporarily state to false.                 
    



Comments
Donald Nong commented Apr 19 '16, 9:45 p.m.

If you're referring to the long running PMR#10054, I don't think the cause-effect is the same as you describe here. A mistake in the CCM DB connection should not affect RM, for whatever reason (the above message shows that /rm/scr also has the same problem). Also, the issue does not appear to be the same as post 120457, as that one clearly points to a network issue - org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection. The fact that the other system could recover within a few minutes can be explained as the result of recycled network connections.

To fix the potentially corrupted index in your PRD, have you considered building it in a separate environment (such as TST) and swap it in in PRD during a restart? I'm not sure whether the approach is technically correct, and you can discuss further with IBM Support.


long TRUONG commented Apr 19 '16, 10:24 p.m.

 Thx Don,

The issue has been with IBM escalation all day and no word from them yet. 
Wonder if anybody knows what
 com.ibm.team.repository.service.internal.RepositoryRemoteService.fetchOrRefreshItems
is supposed to do: <0.2 secs of duration for 3 of them on the active services spiked CPU to a flat plateau of 100% for good & sluggish responses & a Site Scope alert.

<o:p> </o:p>

Accepted answer


permanent link
long TRUONG (363489129) | answered Apr 28 '16, 2:15 a.m.
 An answer to the root cause of the severity 1 issue (on long running PMR#10054) which prompted this question can be found on post 220347. It will answer this question in identifying these errors and others as false due to maxed out CPU condition.
Ralph Schoon selected this answer as the correct answer

Your answer


Register or to post your answer.