About reindexing SQLserver 2012 DB's for CLM 4.0.6 ?
Have started the march to upgrade our RTC/RRC 4.0.6 on Windows server 2008 to 5.0.2 with un upgrade on the TST env (closely shadowing PRD):
- TST server was shut down
- TST DB's get refreshed with the latest full back up of PRD DB's (jts, ccm, and dw), including the VNF_32K.ndf index file
- We had no backup (at matched time) of the indices folder on the app install area on the appServer.
- jts was copied into the to-be-new rm DB for 5.0.2.
- When login to the (refreshed) DB's from the SQ:server Management studio, got popup error index out of bounds of array
-
- Not sure how the index works between the index file on DBserver and the indices folder on the appServer, with this error, and with IBM support advising to reindex before upgrade if there is no backup of indices on the appServer side: we decided to reindex and make sure 4.0.6 still working before attempting upgrade
- When reindexing "repotools-jts -reindex all" dumped javacores and prodiced heapdumps, see below.
- Looked into the repotools-jts.bat script, found only 1 instance of -Xmx1500M, so blindly fixed it to -Xmx4500M and hope for the best. (Line 109: set VMARGS=%VMARGS% -Xmx1500M)
Our (many) questions:
- Would this out of bounds error, if persistent, affect the upgrade or come back to bite us when in operation (either upgraded to 5.0.2 or backed out at 4.0.6) ?
- In this TST env, could we go ahead with the upgrade without reindexing ? Considering we can afford the reindexing time (our ccm rebuildTestIndices took more than 8 hrs) and on PRD the in-situ indices at upgrade should be up-to-date.
- If so, do we need a post-upgrade reindexing ?
- How does RTC relate/sync/work with an index file on the DBserver and the indices on the appServer ?
- What would the "repotools-APP -[reindex|rebuildTextIndices]" do? index on both sides? only on the appServer side (as IBM support seems to say) ?
- Is the index on the DBserver side used by the RTC ? In the past, we didn't know better, thought repotools reindex would be exactly same as an actual reindexing by a DBA: And those reindexing did boost performance significantly.
- Would the upgrade process take care of everything needed for the new rm DB, or it needs to be reindexed post-upgrade ?
- Do you think our blind fix of Xmx would work ?
CRJZS5578I Reindexing : Deleting "triple history" indices for /rm ...
Reindexing : 95.34% - Estimated remaining time: 1h:48m:31sJVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2015/02
/20 20:08:01 - please wait.
JVMDUMP032I JVM requested Heap dump using 'D:\IBM\rtc_406\JazzTeamServer\server\heapdump.20150220.200801.4492.0001.phd' in response to an event
JVMDUMP010I Heap dump written to D:\IBM\rtc_406\JazzTeamServer\server\heapdump.20150220.200801.4492.0001.phd
JVMDUMP032I JVM requested Java dump using 'D:\IBM\rtc_406\JazzTeamServer\server\javacore.20150220.200801.4492.0002.txt' in response to an event
JVMDUMP010I Java dump written to D:\IBM\rtc_406\JazzTeamServer\server\javacore.20150220.200801.4492.0002.txt
JVMDUMP032I JVM requested Snap dump using 'D:\IBM\rtc_406\JazzTeamServer\server\Snap.20150220.200801.4492.0003.trc' in response to an event
JVMDUMP010I Snap dump written to D:\IBM\rtc_406\JazzTeamServer\server\Snap.20150220.200801.4492.0003.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
CRJZS5856E An unexpected error occurred during a write transaction in the index.
java.lang.RuntimeException: CRJZS5856E An unexpected error occurred during a write transaction in the index.
Reindexing : 95.40% - Estimated remaining time: 2h:34m:55sJVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2015/02
/20 20:09:38 - please wait.
JVMDUMP032I JVM requested Heap dump using 'D:\IBM\rtc_406\JazzTeamServer\server\heapdump.20150220.200938.4492.0004.phd' in response to an event
JVMDUMP010I Heap dump written to D:\IBM\rtc_406\JazzTeamServer\server\heapdump.20150220.200938.4492.0004.phd
JVMDUMP032I JVM requested Java dump using 'D:\IBM\rtc_406\JazzTeamServer\server\javacore.20150220.200938.4492.0005.txt' in response to an event
JVMDUMP010I Java dump written to D:\IBM\rtc_406\JazzTeamServer\server\javacore.20150220.200938.4492.0005.txt
JVMDUMP032I JVM requested Snap dump using 'D:\IBM\rtc_406\JazzTeamServer\server\Snap.20150220.200938.4492.0006.trc' in response to an event
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2015/02/20 20:09:44 - please wait.
JVMDUMP010I Snap dump written to D:\IBM\rtc_406\JazzTeamServer\server\Snap.20150220.200938.4492.0006.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
JVMDUMP032I JVM requested Heap dump using 'D:\IBM\rtc_406\JazzTeamServer\server\heapdump.20150220.200944.4492.0007.phd' in response to an event
JVMDUMP010I Heap dump written to D:\IBM\rtc_406\JazzTeamServer\server\heapdump.20150220.200944.4492.0007.phd
JVMDUMP032I JVM requested Java dump using 'D:\IBM\rtc_406\JazzTeamServer\server\javacore.20150220.200944.4492.0008.txt' in response to an event
JVMDUMP010I Java dump written to D:\IBM\rtc_406\JazzTeamServer\server\javacore.20150220.200944.4492.0008.txt
JVMDUMP032I JVM requested Snap dump using 'D:\IBM\rtc_406\JazzTeamServer\server\Snap.20150220.200944.4492.0009.trc' in response to an event
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2015/02/20 20:09:50 - please wait.
CRJZS5856E An unexpected error occurred during a write transaction in the index.
JVMDUMP010I Snap dump written to D:\IBM\rtc_406\JazzTeamServer\server\Snap.20150220.200944.4492.0009.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
JVMDUMP032I JVM requested Heap dump using 'D:\IBM\rtc_406\JazzTeamServer\server\heapdump.20150220.200950.4492.0010.phd' in response to an event
JVMDUMP010I Heap dump written to D:\IBM\rtc_406\JazzTeamServer\server\heapdump.20150220.200950.4492.0010.phd
JVMDUMP032I JVM requested Java dump using 'D:\IBM\rtc_406\JazzTeamServer\server\javacore.20150220.200950.4492.0011.txt' in response to an event
JVMDUMP010I Java dump written to D:\IBM\rtc_406\JazzTeamServer\server\javacore.20150220.200950.4492.0011.txt
JVMDUMP032I JVM requested Snap dump using 'D:\IBM\rtc_406\JazzTeamServer\server\Snap.20150220.200950.4492.0012.trc' in response to an event
CRJAZ1065I Failed to execute preInvoke.
Reindexing : 95.40% - Estimated remaining time: 2h:34m:55sJVMDUMP010I Snap dump written to D:\IBM\rtc_406\JazzTeamServer\server\Snap.20150220.200950.4
492.0012.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
java.lang.RuntimeException: CRJZS5856E An unexpected error occurred during a write transaction in the index.
Exception in thread "jena.query.monitor" java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.entrySet(HashMap.java:553)
at com.ibm.team.jfs.rdf.internal.jena.provider.JenaSimplifiedQueryMonitor.run(JenaSimplifiedQueryMonitor.java:221)
at java.lang.Thread.run(Thread.java:761)
Exception in thread "jts: Floating lease extending timer" CRJZS5856E An unexpected error occurred during a write transaction in the index.
java.lang.OutOfMemoryErrorCRJAZ1065I Failed to execute preInvoke.
: Exception in thread "pool-2-thread-1" CRJAZ1065I Failed to execute preInvoke.
Exception in thread "jts.jfs.indexer./rm.text.live" CRJAZ1065I Failed to execute preInvoke.
java.lang.RuntimeException: CRJZS5856E An unexpected error occurred during a write transaction in the index.
Exception in thread "Rational Common Licensing Floating LicenCRJAZ1065I Failed to execute preInvoke.
se Operation Handler" java.lang.OutOfMemoryErrorJava heap spaceCRJAZ1065I Failed to execute preInvoke.
java.lang.OutOfMemoryErrorException in thread "jts: Remote license updater" Exception in thread "Rational Common Licensing Floating License Operation
Handler" Exception in thread "Un-Available FLoating Licenses scanner" : Java heap space: java.lang.OutOfMemoryErrorjava.lang.OutOfMemoryError
Java heap space at java.lang.OutOfMemoryErrorjava.lang.Throwable.: : : fillInStackTrace
at Java heap space(Native Method) at
at Java heap spacejava.lang.Throwable.
com.ibm.team.jfs.indexing.service.internal.task.AbstractIndexAgent
Java heap space at java.lang.Class.java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject<init> at (.run.Throwable.javagetMethodImpl:a
ddConditionWaiter((java.lang.J9VMInternals5(Native Method)6AbstractQueuedSynchronizer.javaAbstractIndexAgent.java:.1
at )newInstanceImpl
(Native Method)
at 3java.lang.Class at :
1 at java.lang.Class3.1.)8
getMethod at newInstancejava.lang.Thread1.(java.lang.Throwable2(java.lang.J9VMInternalsClass.java.:newInstanceImpl)runClass.java(Native Method)1
5.:<init>
at (java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject9(4Thread.java.
at Throwable.javaawaitNanos:1java.lang.Class.9:76(newInstance)776(AbstractQueuedSynchronizer.java1))
at :)
Class.java2: at
021org.apache.commons.logging.LogFactorysun.io.Converters
7)
5.. at 4 at java.util.concurrent.DelayQueuejava.lang.OutOfMemoryError9getContextClassLoader.)newConverter<init>.
at (((OutOfMemoryError.javaLogFactory.java:Converters.javasun.io.Converters:4take.6:)4
at (newConverter(3Converters.java:33338com.ibm.team.repository.service.internal.license.ManagedFloatingLicenseClient9.)9DelayQueue.javaextendL
eases()ManagedFloatingLicenseClient.java::)
at
sun.io.Converters at sun.io.Converters13
at .org.apache.commons.logging.LogFactory$1972newConverter)5.(newConverterConverters.java(.
at runcom.ibm.team.repository.service.internal.license.ManagedFloatingLicenseClient$1)Converters.java.:run((
at 3:java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue36.ManagedFloatingLicenseClient.javaLogFactory.java:463:)take694
))2
( at at
at sun.io.ByteToCharConverterScheduledThreadPoolExecutor.java.sun.io.ByteToCharConvertergetConverter2(ByteToCharConverter.java2java.util.Timer
$TimerImpl.)run:(.Timer.java::getConverter161(2
at java.security.AccessController729ByteToCharConverter.java)
: at java.lang.StringCoding06.)
.1)makeDecoder1doPrivileged(
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:613)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:928)
at java.lang.Thread.run(Thread.java:761)
7)
at java.lang.StringCoding.makeDecoder(StringCoding.java:72)
at java.lang.StringCoding$DecoderCache$1.Reindexing : 95.4(AccessController.java:273)me: 1days(s) 10h:50m:21sStringCoding.java
at org.apache.commons.logging.LogFactory.getFactory(LogFactory.java:218)
at :org.apache.commons.logging.LogFactoryrun7.2getLog)((
LogFactory.java at java.lang.StringCoding$DecoderCache$1StringCoding.java.::run(1StringCoding.java3:361)3
5 at 61)
)
at java.security.AccessController at org.apache.http.impl.conn.DefaultClientConnection.java.security.AccessController..<init>doPrivilege
d(doPrivilegedAccessController.java((:3AccessController.java2:7DefaultClientConnection.java)
at 3:java.lang.StringCoding$DecoderCache72.getDecoder7()StringCoding.java
at :1java.lang.StringCoding$DecoderCache43.3getDecoder))(
StringCoding.java at :java.lang.StringCoding.
1getDecoder3 at (3StringCoding.java:)38org.apache.http.impl.conn.DefaultClientConnectionOperator
at 4)java.lang.StringCoding.
at .java.lang.StringCodinggetDecoder.(createConnectiondecodeStringCoding.java(:StringCoding.java3:8449)6
at ()java.lang.StringCoding
. at decodeDefaultClientConnectionOperator.javajava.lang.StringCoding(.StringCoding.java:decode(1:StringCoding.java:4590765)6)
at )java.lang.String
at .java.lang.StringCoding<init>.(decodeString.java:
at org.apache.http.impl.conn.AbstractPoolEntry.17<init>(4()AbstractPoolEntry.javaStringCoding.java
at ::com.ibm.jvm.io.ConsolePrintStream.59write(2ConsolePrintStream.java)7:
at 16org.apache.http.impl.conn.tsccm.BasicPoolEntry.7<init>7())BasicPoolEntry.java
at java.lang.String.
at :<init>(java.io.PrintStreamString.java.:write(9PrintStream.java1:7415)4
at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.createEntry(ConnPoolByRoute.java:5)5
at 43))sun.nio.cs.StreamEncoder$CharsetSE
at .org.apache.http.impl.conn.tsccm.ConnPoolByRoutewriteBytes(.
StreamEncoder.java at :com.ibm.jvm.io.ConsolePrintStreamgetEntryBlocking3.5(5writeConnPoolByRoute.java)(
at sun.nio.cs.StreamEncoder$CharsetSE:ConsolePrintStream.java3:.implWrite(1StreamEncoder.java:777)
at java.io.PrintStream.write(PrintStream.java:454)
at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(StreamEncoder.java:355)
at sun.nio.cs.StreamEncoder$CharsetSE.implFlushBuffer(StreamEncoder.java:425)
at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:138)
at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:187)
at java.io.PrintStream.write(PrintStream.java:501)
at java.io.PrintStream.print(PrintStream.java:643)
at com.ibm.team.repotools.util.SystemErrorRecorder$1.print(SystemErrorRecorder.java:100)
at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:803)
at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:797)
at java.lang.Thread.uncaughtException(Thread.java:1241)
6)
at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300)
at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:401)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
at com.ibm.team.repository.service.internal.license.FloatingLicenseClient.executeMethod(FloatingLicenseClient.java:418)
at com.ibm.team.repository.service.internal.license.FloatingLicenseClient.getFloatingLicenseList(FloatingLicenseClient.java:121)
at com.ibm.team.repository.service.internal.license.FloatingLicenseClient.getFloatingPolicies(FloatingLicenseClient.java:151)
at com.ibm.team.repository.service.internal.license.ManagedFloatingLicenseClient.getFloatingPolicies(ManagedFloatingLicenseClient.java:756)
at com.ibm.team.repository.service.internal.license.LicenseService.updateRemoteFloatingLicenses(LicenseService.java:915)
at com.ibm.team.repository.service.internal.license.LicenseService.access$8(LicenseService.java:913)
at com.ibm.team.repository.service.internal.license.LicenseService$4$1.run(LicenseService.java:634)
at com.ibm.team.repository.service.internal.TeamServiceContext.runAsAdmin(TeamServiceContext.java:141)
at com.ibm.team.repository.service.internal.license.LicenseService$4.run(LicenseService.java:631)
at java.lang.Thread.run(Thread.java:761)
416)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:159)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:229)
at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:122)
at java.io.PrintStream.write(PrintStream.java:500)
at java.io.PrintStream.print(PrintStream.java:643)
at com.ibm.team.repotools.util.SystemErrorRecorder$1.print(SystemErrorRecorder.java:100)
at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:803)
at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:797)
at java.lang.Thread.uncaughtException(Thread.java:1241)
CRJAZ1065I Failed to execute preInvoke.
CRJAZ1065I Failed to execute preInvoke.
CRJAZ1065I Failed to execute preInvoke.
CRJAZ1065I Failed to execute preInvoke.
CRJAZ1065I Failed to execute preInvoke.
CRJAZ1065I Failed to execute preInvoke.ng time: 13m:39s
CRJZS5473E An unrecoverable error stopped "triple history" indexing
The user "ADMIN" has logged out of the database "//N01DSW227:1433;databaseName=jts;user=xxxxxxxx;password=xxxxxxxx".
CRJZS5652I Indexing services are being shut down ...
CRJZS5651I Indexing services finished shutting down in 16084 ms
Accepted answer
If you don't follow the suggested strategies: https://jazz.net/wiki/bin/view/Deployment/BackupCLM then you will run into trouble.
The other issue is that the index files and the indexes on the database are two completely different things and there are different reindex commands in the repotools of the various apps to reindex. You need to run the right one to recreated the indexes. See https://jazz.net/library/article/1272 and the related articles about indexing.
The other issue is that the index files and the indexes on the database are two completely different things and there are different reindex commands in the repotools of the various apps to reindex. You need to run the right one to recreated the indexes. See https://jazz.net/library/article/1272 and the related articles about indexing.
Comments
long TRUONG
Feb 21 '15, 7:32 p.m.Our last question was answered by the action of fixing -Xmx to 4500M: It worked.