Why is CLM 6.0.2 repotools_jts.sh -setup SNAFU-ed by a "General Authentication" error with localhost:9443
In automated download, deployment, installation, and setup of CLM 6.0.2, I frequently (I can't recall if "always") encounter this error when running ./repotools_jts.sh -setup <with admin credentials>:
Repo Tools Provisioning using "./conf/jts/provision_profiles". CRJAZ2906E The following URL could not be reached because of a general authentication error: https://localhost:9443/jts
I cannot track down what is causing repotools_jts to believe that the credentials are not appropriate. The setup is attempted directly after installation of the files after the CLM services are up and running. The website for JTS is accessible both at its public external address and through the https://localhost:9443/jts address. Client browsers can run through the web setup and curl/wget on the server can access the JTS website.
For this particular installation, the databases are all on an external DB2 server that has been prepped with the necessary Databases (but may or may not have existing Tables for Jazz). However, I ran into the same issue when attempting to use the OOTB Derby services.
What should I investigate to determine why the shell script cannot gain access?
One answer
Attempting an interactive setup through the web UI gave me a clue to what may be the problem.
The db2 server is not always recreated from scratch on each deployment trial. It may have Tables for JTS from a prior attempt.
Our teamserver.properties files DO NOT record the RepoLockId -- as this would only be applicable for one particular installation and not for automated installation of serveral deployments.
Therefore, at repotools-jts.sh -setup time, it can happen that the setup script encounters a JTS database which has a RepoLockId while the provided teamserver.properties file does not have RepoLockId parameter value.
If this is indeed the problem -- and a repotools-jts.sh --resetRepoLockId exercise suggests that it is -- then the warning message for CRJAZ2906E should be changed to "JTS setup cannot be attempted without a RepoLockId property in teamserver.properties for a JTS Database which already contains a RepoLockId with value ABCDEF12345."
That would be immediately useful versus the current ambiguous remark about connectivity issues.
That would be immediately useful versus the current ambiguous remark about connectivity issues.
Comments
The RepoLockId issue may be one of the reasons but it is not the only reason. The General Authentication error can recur even when the DB2 Database instance has virgin tables and the teamserver.properties file has no RepoLockId property.
Something additional is awry here.
2016-09-02 04:06:35,374 Repo Tools
2016-09-02 04:06:35,374 java.version=1.7.0
2016-09-02 04:06:35,375 java.runtime.version=pxa6470_27sr3fp30-20160112_01 (SR3 FP30)
2016-09-02 04:06:35,379 Provisioning using "./conf/jts/provision_profiles".
2016-09-02 04:06:35,420 repotools-jts -setup adminPassword=****** adminUserId=ADMIN parametersFile=/opt/IBM/JazzTeamServer/server/co
nf/jts/parameters.properties
2016-09-02 04:06:37,658 CRJAZ2906E The following URL could not be reached because of a general authentication error: https://localho
com.ibm.team.repository.common.transport.ServiceNotAvailableException: CRJAZ0106E The server is temporarily unavailable. The request
for the following URL was denied: /jts/service/com.ibm.team.repository.common.internal.IRepositoryRemoteService.
at com.ibm.team.repository.transport.client.RemoteTeamServer.executeMethod(RemoteTeamServer.java:1266)
It is interesting and probably relevant that, when the CRJAZ2906E issue is reported by repotools_jts.sh -setup that after then having run through a Web interactive setup to completion that I can return to the command-line repotools_jts.sh -setup request, retry that, and the CRJAZ2906E error is not encountered.
This suggests either:
- there is something capricious in the environment and that the CRJAZ2906E error may or may not occur regardless of what I as administrator may be attempting
- there is something in the parameters.properties or initial teamserver.properties settings that is incorrect which the Web setup corrects.
repotools-jts.sh -setup can also run into this error midway:
Caused by: com.ibm.team.repository.common.transport.TeamServiceException: CRJAZ0099E An HTTP error occurred when this URL was being accessed: /jts/service/com.ibm.team.repository.service.internal.IServerConfigurationRestService/configurationUpdates. Error details: java.net.SocketTimeoutException.
at com.ibm.team.repository.transport.client.RemoteTeamServer.executePrimitiveRequest(RemoteTeamServer.java:1918)
I suspect that there are several scalability issues that are snafu-ing the repotools-jts.sh -setup process.
The SocketTimeoutException may simply be due to the amount of information being exchanged between the Jazz server and the DB2 server. If so, extending the default socket timeout limit may help.
sudo sysctl -w net.ipv4.tcp_syn_retries=8
Meanwhile, the original "general authentication" error may be a pam_limits restriction. Some google traffic suggests that Linux developers refer to rejection in their app code during PAM checking as "general authentication" errors.
If so, limits can be monitored and logged with the auditctl and ausearch system-call-tracing services.
The entire JTS service is launched from either an init.d script or an upstart script using start-stop-daemon and this daemonizer does not participate in the PAM services.
One has to set explicit "limit" stanzas in the upstart jts.conf script.
showing 5 of 7
show 2 more comments