Upgrading RTC 5.0.2 to 6.0.1 results in JRS not being accessible anymore.
I just upgraded RTC 5.0.2 to 6.0.1 and in the process managed to break JRS. I now get the following error in diagnostics:
Application: the discovery resource at http://xxx.xxx.xxx.com:9080/rs/rootservices for "/rs" cannot be accessed:
The returned HTTP status was 500
The response body was
Error 500: java.lang.NoSuchMethodError: com/ibm/team/jfs/app/http/client/JazzHttpClient.execute(Lorg/apache/http/HttpHost;Lorg/apache/http/HttpRequest;Lorg/apache/http/protocol/HttpContext;)Lorg/apache/http/client/methods/CloseableHttpResponse;
I followed in the interactive guide to a T (at least I think I did). Tried it many times but to no avail. I also looked at other posts such as:
https://jazz.net/forum/questions/203488/application-the-discovery-resource-error-rs-cannot-be-accessed?redirect=%2Fforum%2Fquestions%2F203488%2Fapplication-the-discovery-resource-error-rs-cannot-be-accessed
but that did not help. Here's a more detailed listing if I try and access http://xxx.xxx.xxx.com/rs/service. Perhaps it has more clues. Any help to solve this would be much appreciated. (I apologize for the format below)
@prefix err: <http://jazz.net/xmlns/prod/jazz/foundation/1.0/> .
[ err:errorCause [ err:detailedMessage "java.lang.NoSuchMethodError: com/ibm/team/jfs/app/http/client/JazzHttpClient.execute(Lorg/apache/http/HttpHost;Lorg/apache/http/HttpRequest;Lorg/apache/http/protocol/HttpContext;)Lorg/apache/http/client/methods/CloseableHttpResponse;\r\n\tat com.ibm.team.jfs.app.http.client.JazzHttpClient.execute(JazzHttpClient.java:293)\r\n\tat org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:919)\r\n\tat com.ibm.team.jfs.app.discovery.internal.DiscoveryHelper.executeServiceQuery(DiscoveryHelper.java:144)\r\n\tat com.ibm.team.jfs.app.discovery.internal.DiscoveryHelper.getService(DiscoveryHelper.java:72)\r\n\tat com.ibm.team.integration.reporting.common.restservices.BaseRestService.getOAuthProviderProperties(BaseRestService.java:733)\r\n\tat com.ibm.team.integration.reporting.common.restservices.BaseRestService.getOauthProviderProperties(BaseRestService.java:701)\r\n\tat com.ibm.team.integration.reporting.common.restservices.BaseRestService.service(BaseRestService.java:128)\r\n\tat com.ibm.team.integration.reporting.gatewayservices.FrontResultService.service(FrontResultService.java:226)\r\n\tat com.ibm.team.jfs.app.servlet.AppContainerServlet.dispatchRequest(AppContainerServlet.java:155)\r\n\tat com.ibm.team.jfs.app.servlet.AppContainerServlet.service(AppContainerServlet.java:282)\r\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:668)\r\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)\r\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)\r\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)\r\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)\r\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)\r\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:79)\r\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:960)\r\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1064)\r\n\tat com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.dispatch(WebAppRequestDispatcher.java:1385)\r\n\tat com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java:194)\r\n\tat com.ibm.team.integration.reporting.common.auth.JTSAuthenticator.getUserID(JTSAuthenticator.java:89)\r\n\tat com.ibm.team.integration.reporting.common.auth.JTSAuthenticator.authenticate(JTSAuthenticator.java:42)\r\n\tat com.ibm.team.integration.reporting.gatewayservices.Front.handleRequest(Front.java:168)\r\n\tat com.ibm.team.integration.reporting.gatewayservices.Front.doGet(Front.java:64)\r\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:575)\r\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:668)\r\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)\r\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)\r\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)\r\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)\r\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)\r\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)\r\n\tat com.ibm.team.integration.reporting.common.services.EncodingFilter.doFilter(EncodingFilter.java:90)\r\n\tat com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)\r\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)\r\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:960)\r\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1064)\r\n\tat com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3837)\r\n\tat com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)\r\n\tat com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:981)\r\n\tat com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)\r\n\tat com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)\r\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)\r\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)\r\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:312)\r\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:283)\r\n\tat com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)\r\n\tat com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)\r\n\tat com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)\r\n\tat com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)\r\n\tat com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)\r\n\tat com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)\r\n\tat com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)\r\n\tat com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)\r\n\tat com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)\r\n\tat com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1864)\r\n"^^<http://www.w3.org/2001/XMLSchema#string> ;
err:errorMessage "java.lang.NoSuchMethodError: com/ibm/team/jfs/app/http/client/JazzHttpClient.execute(Lorg/apache/http/HttpHost;Lorg/apache/http/HttpRequest;Lorg/apache/http/protocol/HttpContext;)Lorg/apache/http/client/methods/CloseableHttpResponse;"^^<http://www.w3.org/2001/XMLSchema#string> ;
err:errorStatus "500"^^<http://www.w3.org/2001/XMLSchema#long>
] ;
err:errorMessage "Internal Error"^^<http://www.w3.org/2001/XMLSchema#string> ;
err:errorStatus "500"^^<http://www.w3.org/2001/XMLSchema#long>
5 answers
The conflicting class appears to be org.apache.http.impl.client.AbstractHttpClient, not com.ibm.team.jfs.app.http.client.JazzHttpClient.
The class com.ibm.team.jfs.app.http.client.JazzHttpClient has no changes regarding the execute() function between 5.0.2 and 6.0.1.
As for org.apache.http.impl.client.AbstractHttpClient, CLM 5.0.2 should come with org.apache.http.client 4.1.2 package, which contains the execute() function that matches the stack shown in the OP. But, CLM 6.0.1 should come with org.apache.http.client 4.5.0 package, which no longer has the execute() function in the AbstractHttpClient class.
So it appears that an older version of the org.apache.http.client package was loaded when JRS started. Apart from ensuring application paths and library paths are correct, make sure that the class loading policy is set to "Parent Last" for the JRS application in WAS.
The class com.ibm.team.jfs.app.http.client.JazzHttpClient has no changes regarding the execute() function between 5.0.2 and 6.0.1.
As for org.apache.http.impl.client.AbstractHttpClient, CLM 5.0.2 should come with org.apache.http.client 4.1.2 package, which contains the execute() function that matches the stack shown in the OP. But, CLM 6.0.1 should come with org.apache.http.client 4.5.0 package, which no longer has the execute() function in the AbstractHttpClient class.
So it appears that an older version of the org.apache.http.client package was loaded when JRS started. Apart from ensuring application paths and library paths are correct, make sure that the class loading policy is set to "Parent Last" for the JRS application in WAS.
Hello Stephane,
this could be related to the configuration of the Shared libraries references (assuming you are using WAS).
I suggest to review the configuration described in the Configure the Report Builder (RS) application section of
Deploying applications for the Rational solution for Collaborative Lifecycle Management on WebSphere Application Server
http://www-01.ibm.com/support/knowledgecenter/SSYMRC_6.0.1/com.ibm.jazz.install.doc/topics/t_deploy_was.html?lang=en
Best Regards,
Francesco Chiossi
this could be related to the configuration of the Shared libraries references (assuming you are using WAS).
I suggest to review the configuration described in the Configure the Report Builder (RS) application section of
Deploying applications for the Rational solution for Collaborative Lifecycle Management on WebSphere Application Server
http://www-01.ibm.com/support/knowledgecenter/SSYMRC_6.0.1/com.ibm.jazz.install.doc/topics/t_deploy_was.html?lang=en
Best Regards,
Francesco Chiossi
Stephane,
Have you been able to find a solution?
Your error message contains this:
So this is not about a missing library. It must be due to a version incompatibility somewhere.
There are essentially three sources for such an incompatibility:
1. WAS cached files not cleared during upgrade
2. Wrong SW component versions being brought in.
My advice is that you add this custom Java flag to WAS:
-verbose:class
Then stop/start the server
Then look for JazzHttpClient in the output and check that the versions loaded match what you expect for your JRS version. If you find that this references a file that did not come from the latest time you deployed JRS, then you should re-deploy, clean out temp files and caches on WAS.
Example native_stderr.log output with -verbose:class:
class load: com.ibm.team.jfs.app.http.client.JazzHttpClient from: file:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/Node01Cell/rs_war.ear/rs.war/WEB-INF/lib/jfs-sdk.jar
But the version incompatibility could also come from elsewhere in the callstack. other classes to check based on your callstack:
So look up the classes higher in the call stack in the "class load" info from native_stderr.log after you enable -verbose:class. See if you find a class that is being loaded from the wrong place.
Here is a technote about a similar problem by the way (but since that technote doesn't show the full callstack we can't be sure that it's the exact same problem
http://www-01.ibm.com/support/docview.wss?uid=swg21713659
Hope this helps
Erik
Have you been able to find a solution?
Your error message contains this:
java.lang.NoSuchMethodError: com/ibm/team/jfs/app/http/client/JazzHttpClient.execute(Lorg/apache/http/HttpHost;Lorg/apache/http/HttpRequest;Lorg/apache/http/protocol/HttpContext;)Lorg/apache/http/client/methods/CloseableHttpResponse
This is NOT saying that the class JazzHttpClient does not exist. It is saying that this class has no method called "execute" that takes the parameters listed. (I am now researching this same problem but for a different "missing" function for a customer.
So this is not about a missing library. It must be due to a version incompatibility somewhere.
There are essentially three sources for such an incompatibility:
1. WAS cached files not cleared during upgrade
2. Wrong SW component versions being brought in.
My advice is that you add this custom Java flag to WAS:
-verbose:class
Then stop/start the server
Then look for JazzHttpClient in the output and check that the versions loaded match what you expect for your JRS version. If you find that this references a file that did not come from the latest time you deployed JRS, then you should re-deploy, clean out temp files and caches on WAS.
Example native_stderr.log output with -verbose:class:
class load: com.ibm.team.jfs.app.http.client.JazzHttpClient from: file:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/Node01Cell/rs_war.ear/rs.war/WEB-INF/lib/jfs-sdk.jar
But the version incompatibility could also come from elsewhere in the callstack. other classes to check based on your callstack:
com.ibm.team.jfs.app.discovery.internal.DiscoveryHelper
org.apache.http.impl.client.AbstractHttpClient
So look up the classes higher in the call stack in the "class load" info from native_stderr.log after you enable -verbose:class. See if you find a class that is being loaded from the wrong place.
Here is a technote about a similar problem by the way (but since that technote doesn't show the full callstack we can't be sure that it's the exact same problem
http://www-01.ibm.com/support/docview.wss?uid=swg21713659
Hope this helps
Erik
Here are the findings from IBM summarized below:
1) For background JRS (Report Builder) is installed in the same WAS profile as RTC, DCC, JTS, LQE, RQM and DNG. Apparently, this topology is known to cause issues because not all applications use the same version of common libraries such as the Apache httpcore.jar and httpclient.jar. The hierarchical class loader in WAS can inadvertently load classes from the wrong version of the library.
2) From the stack trace in the WAS SystemOut.log the JazzHttpClient method execute(HttpHost, HttpRequest, HttpContext) is being called from the JazzHttpClient class in the jsf-sdk.jar library. The JazzHttpClient class actually extend the org.apache class DefaultHttpClient which in turn extends the org.apache class AbstractHttpClient. The method which does not exist but is being called is actually defined in the AbstractHttpClient class. Both DefaultHttpClient and AbstracthttpClient are included the common Apache library httpclient.jar.
Therefore, the WAS class loader (for the rs_war application) is picking up the wrong version of that httpclient.jar library. Report Builder 6.0.1 requires version 4.5 of that library and the actual jar file is httpclient-4.5.jar. In 5.0.2 the version of that library was httpclient-4.1.2.jar. I think WAS itself also has those HttpClient classes included in one of its own libraries and that version is not the same as the one Report Builder requires. So, it appears that the class loader from Report Builder (rs_war application) is picking up the wrong version of those Http Client library classes.
The next step for me is to provision a separate VM for JRS & DCC. As an FYI this is a development environment that has worked just fine until now running on a single server. I will provide an additional post if that solution solved the problem.