About error messages with https:///ccm/scr & https:///rm/scr
We get same issue as per post 120457, however our is hanging our server (RTC/RRC 5.0.2 Windows/Tomcat) every 2 hrs or so requiring a restart. And there was no real answer to above post.
We have elevated one of our existing PMR to severity 1 server down, but hoping we may be lucky get an answer here before getting a call from support.
It started with a hang when the wrong port (143) was inadvertently saved for the DB ccm on the ccm DB connection web page. We fixed the port (to 1433) in ccm teamserver.properties and restarted the app (yes, it required a restart, we couln't get back in the DB connection page to correct the mistake). It restarted OK.
But then all hell broke loose, it has been hanging about every 2 hrs or so with CPU (x4 on a VM) peaked and persisted at 100%. It has to be restarted every time.
Typical errors around the hangs (time) in JTS log:
Line 92537: 2016-04-18 15:58:03,077 [http-bio-443-exec-568 @@ 15:37
e2xjeya <Initial Page Load@9f67d68c-edd8-42e9-a73a-04c762188750>
/ccm/web/projects/PAS] ERROR license.behavior.
SCRBasedCompatibleLicenseBehavior - Error fetching the scr document
for application at url: "https://rtc-web1.ent.rt.csaa.com/ccm/scr"
Line 92700: 2016-04-18 15:58:31,406 [http-bio-443-exec-580 @@ 15:57
gg27pan <Jakarta Commons-HttpClient/3.1@10.37.128.136> /ccm/process-
runtime/operation-reports] ERROR license.behavior.
SCRBasedCompatibleLicenseBehavior - Error fetching the scr document
for application at url: "https://rtc-web1.ent.rt.csaa.com/ccm/scr"
Line 92863: 2016-04-18 15:58:35,586 [http-bio-443-exec-621 @@ 15:53
<unauthenticated> <Requirements Management/5.0.2@172.26.167.65>
/rm/process-security/operations/com.ibm.rrs.saveComment/report] ERROR
license.behavior.SCRBasedCompatibleLicenseBehavior - Error fetching
the scr document for application at url: "https://rtc-web1.ent.rt.csaa.
com/rm/scr"
Line 93023: 2016-04-18 15:59:03,541 [http-bio-443-exec-551 @@ 15:39
gq6ilam <Initial Page Load@efa528c8-ee6d-4a77-9407-2473035c65c5>
/rm/web] ERROR license.behavior.SCRBasedCompatibleLicenseBehavior -
Error fetching the scr document for application at url: "https://rtc-
web1.ent.rt.csaa.com/ccm/scr"
Line 93186: 2016-04-18 16:00:22,645 [http-bio-443-exec-492 @@ 15:48
<unauthenticated> <Initial Page Load@eeb058c9-13ef-4151-b8af-
a59659fe489f> /ccm/auth/authrequired] ERROR license.behavior.
SCRBasedCompatibleLicenseBehavior - Error fetching the scr document
for application at url: "https://rtc-web1.ent.rt.csaa.com/ccm/scr"
Line 93349: 2016-04-18 16:01:20,114 [ jts: Remote license updater]
WARN ory.service.internal.license.FloatingLicenseClient - CRJAZ1272I
The floating license available from "https://rtc-web1.ent.rt.csaa.
com/jts/service/com.ibm.team.repository.service.internal.license.
IFloatingLicenseService/licenses/com.ibm.team.rrc.analyst.floating-
v3_b92a5960-cb65-11df-bd3b-0800200c9a66?includeFullJar=true" could not
be downloaded. The floating license at that location will be ignored.
Line 93372: 2016-04-18 16:01:20,426 [http-bio-443-exec-455 @@ 15:29
<unauthenticated> <Initial Page Load@5d86d8da-4352-4bf7-8b8a-
7effa4f3e741> /ccm/auth/authrequired] ERROR license.behavior.
SCRBasedCompatibleLicenseBehavior - Error fetching the scr document
for application at url: "https://rtc-web1.ent.rt.csaa.com/rm/scr"
Line 93532: 2016-04-18 16:02:41,686 [http-bio-443-exec-513 @@ 15:28
<unauthenticated> <Initial Page Load@c2a919a8-bfa9-4e3f-bd23-
692a2a9a7752> /jts/auth/authrequired] ERROR license.behavior.
SCRBasedCompatibleLicenseBehavior - Error fetching the scr document
for application at url: "https://rtc-web1.ent.rt.csaa.com/ccm/scr"
Line 93695: 2016-04-18 16:03:28,236 [http-bio-443-exec-580 @@ 16:01
easgovi <com.ibm.team.workitem.viewWorkItem@84e1c660-61ea-4858-89bb-
182d9888c2bb> /ccm/service/com.ibm.team.workitem.common.internal.rest.
IWorkItemRestService/workItemEditableProperties] ERROR license.behavior.
SCRBasedCompatibleLicenseBehavior - Error fetching the scr document
for application at url: "https://rtc-web1.ent.rt.csaa.com/ccm/scr"
Line 93884: 2016-04-18 16:04:02,836 [http-bio-443-exec-464 @@ 15:01
e2xjeya <Initial Page Load@1bde0ec6-dfb8-4419-b998-c04c063ba89b>
/ccm/web/projects/PAS] ERROR license.behavior.
SCRBasedCompatibleLicenseBehavior - Error fetching the scr document
for application at url: "https://rtc-web1.ent.rt.csaa.com/ccm/scr"
Line 94047: 2016-04-18 16:04:02,992 [http-bio-443-exec-247 @@ 14:10
gjexbas <Initial Page Load@3516a49f-36d6-4e0b-8245-9633ee301bf7>
/ccm/web/projects/PAS] ERROR license.behavior.
SCRBasedCompatibleLicenseBehavior - Error fetching the scr document
for application at url: "https://rtc-web1.ent.rt.csaa.com/rm/scr"
Line 94209: 2016-04-18 16:04:05,925 [http-bio-443-exec-247 @@ 14:10
gjexbas <Initial Page Load@3516a49f-36d6-4e0b-8245-9633ee301bf7>
/ccm/web/projects/PAS] WARN .team.repository.servlet.
AbstractTeamServerServlet - CRJAZ1163I LicenseNotGrantedException
processing GET request for com.ibm.team.repository.service.internal.
license.ILicenseRestService.getAssertLicense(). CRJAZ1170I The request
was made by user "rm_user" from "P01APW437.ent.rt.csaa.com".CRJAZ1166I
The stack trace hash is AE2A456ECD912724104FC0D5C83E45E901DCA905.
Line 94382: 2016-04-18 16:06:03,548 [ Start Level Event Dispatcher]
INFO nternal.app.servlet.util.ServletDispatchingHandler - CRJZS0315I
Servlet Dispatching Enabled
Line 94383: 2016-04-18 16:06:03,548 [ Start Level Event Dispatcher]
INFO nternal.app.servlet.util.ServletDispatchingHandler - CRJZS0318I
Servlet Loopback Disabled
Line 94384: 2016-04-18 16:06:09,898 [Component Resolve Thread (Bundle
94)] INFO nternal.OverridablePropertyFileServerConfiguration -
Line 94387: 2016-04-18 16:06:09,898 [Component Resolve Thread (Bundle
94)] INFO nternal.OverridablePropertyFileServerConfiguration -
CRJAZ1363I Loading the configuration from "file:/D:
/IBM/rtc_502/JazzTeamServer/server/conf/jts/teamserver.properties".
Line 94388: 2016-04-18 16:06:43,438 [ http-bio-443-exec-74]
INFO pository.service.internal.ServerDescriptionService - CRJAZ1779I
This server is configured as a JTS.
Line 94389: 2016-04-18 16:06:44,545 [ http-bio-443-exec-74]
INFO rvice.mapping.internal.ServerRenameProgressService - CRJAZ2558I
Setting the local server rename state to false and the
openServerDescriptionServiceTemporarily state to false.
|
Accepted answer
|
An answer to the root cause of the severity 1 issue (on long running PMR#10054) which prompted this question can be found on post 220347. It will answer this question in identifying these errors and others as false due to maxed out CPU condition.
Ralph Schoon selected this answer as the correct answer
|
Your answer
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.
Comments
If you're referring to the long running PMR#10054, I don't think the cause-effect is the same as you describe here. A mistake in the CCM DB connection should not affect RM, for whatever reason (the above message shows that /rm/scr also has the same problem). Also, the issue does not appear to be the same as post 120457, as that one clearly points to a network issue - org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection. The fact that the other system could recover within a few minutes can be explained as the result of recycled network connections.
To fix the potentially corrupted index in your PRD, have you considered building it in a separate environment (such as TST) and swap it in in PRD during a restart? I'm not sure whether the approach is technically correct, and you can discuss further with IBM Support.
Thx Don,
<o:p> </o:p>