It's all about the answers!

Ask a question
QuestionsTagsUsersBadges
Follow this question
Question details

×42,788
×5,828
×4,289
×3,952
×2,339

question asked: Feb 21 '15, 12:45 a.m.
question was seen: 3,686 times
last updated: Feb 23 '15, 6:44 a.m.

Related questions

FAQ - How this Forum works

About reindexing SQLserver 2012 DB's for CLM 4.0.6 ?

0
long TRUONG (365●4●118●146) | asked Feb 21 '15, 12:45 a.m.
 Have started the march to upgrade our RTC/RRC 4.0.6 on Windows server 2008 to 5.0.2 with un upgrade on the TST env (closely shadowing PRD):

  • TST server was shut down
  • TST DB's get refreshed with the latest full back up of PRD DB's (jts, ccm, and dw), including the VNF_32K.ndf index file
  • We had no backup (at matched time) of the indices folder on the app install area on the appServer.
  • jts was copied into the to-be-new rm DB for 5.0.2.
  • When login to the (refreshed) DB's from the SQ:server Management studio, got popup error index out of bounds of array
  • out of array bounds
  • Not sure how the index works between the index file on DBserver and the indices folder on the appServer, with this error, and with IBM support advising to reindex before upgrade if there is no backup of indices on the appServer side: we decided to reindex and make sure 4.0.6 still working before attempting upgrade
  • When reindexing "repotools-jts -reindex all" dumped javacores and prodiced heapdumps, see below.
  • Looked into the repotools-jts.bat script, found only 1 instance of -Xmx1500M, so blindly fixed it to -Xmx4500M and hope for the best. (Line 109: set VMARGS=%VMARGS% -Xmx1500M)
Our (many) questions:
  • Would this out of bounds error, if persistent, affect the upgrade or come back to bite us when in operation (either upgraded to 5.0.2 or backed out at 4.0.6) ?
  • In this TST env, could we go ahead with the upgrade without reindexing ? Considering we can afford the reindexing time (our ccm rebuildTestIndices took more than 8 hrs) and on PRD the in-situ indices at upgrade should be up-to-date.
  • If so, do we need a post-upgrade reindexing ?
  • How does RTC relate/sync/work with an index file on the DBserver and the indices on the appServer ?
  • What would the "repotools-APP -[reindex|rebuildTextIndices]" do? index on both sides? only on the appServer side (as IBM support seems to say) ? 
  • Is the index on the DBserver side used by the RTC ? In the past, we didn't know better, thought repotools reindex would be exactly same as an actual reindexing by a DBA: And those reindexing did boost performance significantly.
  • Would the upgrade process take care of everything needed for the new rm DB, or it needs to be reindexed post-upgrade ? 
  • Do you think our blind fix of Xmx would work ?
CRJZS5578I Reindexing : Deleting "triple history" indices for /rm ...
Reindexing : 95.34% - Estimated remaining time: 1h:48m:31sJVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2015/02
/20 20:08:01 - please wait.
JVMDUMP032I JVM requested Heap dump using 'D:\IBM\rtc_406\JazzTeamServer\server\heapdump.20150220.200801.4492.0001.phd' in response to an event
JVMDUMP010I Heap dump written to D:\IBM\rtc_406\JazzTeamServer\server\heapdump.20150220.200801.4492.0001.phd
JVMDUMP032I JVM requested Java dump using 'D:\IBM\rtc_406\JazzTeamServer\server\javacore.20150220.200801.4492.0002.txt' in response to an event
JVMDUMP010I Java dump written to D:\IBM\rtc_406\JazzTeamServer\server\javacore.20150220.200801.4492.0002.txt
JVMDUMP032I JVM requested Snap dump using 'D:\IBM\rtc_406\JazzTeamServer\server\Snap.20150220.200801.4492.0003.trc' in response to an event
JVMDUMP010I Snap dump written to D:\IBM\rtc_406\JazzTeamServer\server\Snap.20150220.200801.4492.0003.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
CRJZS5856E An unexpected error occurred during a write transaction in the index.
java.lang.RuntimeException: CRJZS5856E An unexpected error occurred during a write transaction in the index.
Reindexing : 95.40% - Estimated remaining time: 2h:34m:55sJVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2015/02
/20 20:09:38 - please wait.
JVMDUMP032I JVM requested Heap dump using 'D:\IBM\rtc_406\JazzTeamServer\server\heapdump.20150220.200938.4492.0004.phd' in response to an event
JVMDUMP010I Heap dump written to D:\IBM\rtc_406\JazzTeamServer\server\heapdump.20150220.200938.4492.0004.phd
JVMDUMP032I JVM requested Java dump using 'D:\IBM\rtc_406\JazzTeamServer\server\javacore.20150220.200938.4492.0005.txt' in response to an event
JVMDUMP010I Java dump written to D:\IBM\rtc_406\JazzTeamServer\server\javacore.20150220.200938.4492.0005.txt
JVMDUMP032I JVM requested Snap dump using 'D:\IBM\rtc_406\JazzTeamServer\server\Snap.20150220.200938.4492.0006.trc' in response to an event
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2015/02/20 20:09:44 - please wait.
JVMDUMP010I Snap dump written to D:\IBM\rtc_406\JazzTeamServer\server\Snap.20150220.200938.4492.0006.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
JVMDUMP032I JVM requested Heap dump using 'D:\IBM\rtc_406\JazzTeamServer\server\heapdump.20150220.200944.4492.0007.phd' in response to an event
JVMDUMP010I Heap dump written to D:\IBM\rtc_406\JazzTeamServer\server\heapdump.20150220.200944.4492.0007.phd
JVMDUMP032I JVM requested Java dump using 'D:\IBM\rtc_406\JazzTeamServer\server\javacore.20150220.200944.4492.0008.txt' in response to an event
JVMDUMP010I Java dump written to D:\IBM\rtc_406\JazzTeamServer\server\javacore.20150220.200944.4492.0008.txt
JVMDUMP032I JVM requested Snap dump using 'D:\IBM\rtc_406\JazzTeamServer\server\Snap.20150220.200944.4492.0009.trc' in response to an event
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2015/02/20 20:09:50 - please wait.
CRJZS5856E An unexpected error occurred during a write transaction in the index.
JVMDUMP010I Snap dump written to D:\IBM\rtc_406\JazzTeamServer\server\Snap.20150220.200944.4492.0009.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
JVMDUMP032I JVM requested Heap dump using 'D:\IBM\rtc_406\JazzTeamServer\server\heapdump.20150220.200950.4492.0010.phd' in response to an event
JVMDUMP010I Heap dump written to D:\IBM\rtc_406\JazzTeamServer\server\heapdump.20150220.200950.4492.0010.phd
JVMDUMP032I JVM requested Java dump using 'D:\IBM\rtc_406\JazzTeamServer\server\javacore.20150220.200950.4492.0011.txt' in response to an event
JVMDUMP010I Java dump written to D:\IBM\rtc_406\JazzTeamServer\server\javacore.20150220.200950.4492.0011.txt
JVMDUMP032I JVM requested Snap dump using 'D:\IBM\rtc_406\JazzTeamServer\server\Snap.20150220.200950.4492.0012.trc' in response to an event
CRJAZ1065I Failed to execute preInvoke.
Reindexing : 95.40% - Estimated remaining time: 2h:34m:55sJVMDUMP010I Snap dump written to D:\IBM\rtc_406\JazzTeamServer\server\Snap.20150220.200950.4
492.0012.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
java.lang.RuntimeException: CRJZS5856E An unexpected error occurred during a write transaction in the index.
Exception in thread "jena.query.monitor" java.lang.OutOfMemoryError: Java heap space
        at java.util.HashMap.entrySet(HashMap.java:553)
        at com.ibm.team.jfs.rdf.internal.jena.provider.JenaSimplifiedQueryMonitor.run(JenaSimplifiedQueryMonitor.java:221)
        at java.lang.Thread.run(Thread.java:761)
Exception in thread "jts: Floating lease extending timer" CRJZS5856E An unexpected error occurred during a write transaction in the index.
java.lang.OutOfMemoryErrorCRJAZ1065I Failed to execute preInvoke.
: Exception in thread "pool-2-thread-1" CRJAZ1065I Failed to execute preInvoke.
Exception in thread "jts.jfs.indexer./rm.text.live" CRJAZ1065I Failed to execute preInvoke.
java.lang.RuntimeException: CRJZS5856E An unexpected error occurred during a write transaction in the index.
Exception in thread "Rational Common Licensing Floating LicenCRJAZ1065I Failed to execute preInvoke.
se Operation Handler" java.lang.OutOfMemoryErrorJava heap spaceCRJAZ1065I Failed to execute preInvoke.
java.lang.OutOfMemoryErrorException in thread "jts: Remote license updater" Exception in thread "Rational Common Licensing Floating License Operation
Handler" Exception in thread "Un-Available FLoating Licenses scanner" : Java heap space: java.lang.OutOfMemoryErrorjava.lang.OutOfMemoryError
Java heap space at java.lang.OutOfMemoryErrorjava.lang.Throwable.: : : fillInStackTrace

        at Java heap space(Native Method)       at
        at Java heap spacejava.lang.Throwable.
com.ibm.team.jfs.indexing.service.internal.task.AbstractIndexAgent
Java heap space at java.lang.Class.java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject<init>  at (.run.Throwable.javagetMethodImpl:a
ddConditionWaiter((java.lang.J9VMInternals5(Native Method)6AbstractQueuedSynchronizer.javaAbstractIndexAgent.java:.1
        at )newInstanceImpl
(Native Method)
        at 3java.lang.Class     at :
1       at java.lang.Class3.1.)8
getMethod       at newInstancejava.lang.Thread1.(java.lang.Throwable2(java.lang.J9VMInternalsClass.java.:newInstanceImpl)runClass.java(Native Method)1
5.:<init>
        at (java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject9(4Thread.java.
        at Throwable.javaawaitNanos:1java.lang.Class.9:76(newInstance)776(AbstractQueuedSynchronizer.java1))
        at :)
Class.java2:    at
021org.apache.commons.logging.LogFactorysun.io.Converters
7)
5..     at 4    at java.util.concurrent.DelayQueuejava.lang.OutOfMemoryError9getContextClassLoader.)newConverter<init>.
        at (((OutOfMemoryError.javaLogFactory.java:Converters.javasun.io.Converters:4take.6:)4
        at (newConverter(3Converters.java:33338com.ibm.team.repository.service.internal.license.ManagedFloatingLicenseClient9.)9DelayQueue.javaextendL
eases()ManagedFloatingLicenseClient.java::)
        at
sun.io.Converters       at sun.io.Converters13
        at .org.apache.commons.logging.LogFactory$1972newConverter)5.(newConverterConverters.java(.
        at runcom.ibm.team.repository.service.internal.license.ManagedFloatingLicenseClient$1)Converters.java.:run((
        at 3:java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue36.ManagedFloatingLicenseClient.javaLogFactory.java:463:)take694
))2
(       at      at
        at sun.io.ByteToCharConverterScheduledThreadPoolExecutor.java.sun.io.ByteToCharConvertergetConverter2(ByteToCharConverter.java2java.util.Timer
$TimerImpl.)run:(.Timer.java::getConverter161(2
        at java.security.AccessController729ByteToCharConverter.java)
:       at java.lang.StringCoding06.)
.1)makeDecoder1doPrivileged(
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:613)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:928)
        at java.lang.Thread.run(Thread.java:761)
7)
        at java.lang.StringCoding.makeDecoder(StringCoding.java:72)
        at java.lang.StringCoding$DecoderCache$1.Reindexing : 95.4(AccessController.java:273)me: 1days(s) 10h:50m:21sStringCoding.java
        at org.apache.commons.logging.LogFactory.getFactory(LogFactory.java:218)
        at :org.apache.commons.logging.LogFactoryrun7.2getLog)((
LogFactory.java at java.lang.StringCoding$DecoderCache$1StringCoding.java.::run(1StringCoding.java3:361)3
5       at 61)
)
        at java.security.AccessController       at org.apache.http.impl.conn.DefaultClientConnection.java.security.AccessController..<init>doPrivilege
d(doPrivilegedAccessController.java((:3AccessController.java2:7DefaultClientConnection.java)
        at 3:java.lang.StringCoding$DecoderCache72.getDecoder7()StringCoding.java
        at :1java.lang.StringCoding$DecoderCache43.3getDecoder))(
StringCoding.java       at :java.lang.StringCoding.
1getDecoder3    at (3StringCoding.java:)38org.apache.http.impl.conn.DefaultClientConnectionOperator
        at 4)java.lang.StringCoding.
        at .java.lang.StringCodinggetDecoder.(createConnectiondecodeStringCoding.java(:StringCoding.java3:8449)6
        at ()java.lang.StringCoding
.       at decodeDefaultClientConnectionOperator.javajava.lang.StringCoding(.StringCoding.java:decode(1:StringCoding.java:4590765)6)
        at )java.lang.String
        at .java.lang.StringCoding<init>.(decodeString.java:
        at org.apache.http.impl.conn.AbstractPoolEntry.17<init>(4()AbstractPoolEntry.javaStringCoding.java
        at ::com.ibm.jvm.io.ConsolePrintStream.59write(2ConsolePrintStream.java)7:
        at 16org.apache.http.impl.conn.tsccm.BasicPoolEntry.7<init>7())BasicPoolEntry.java
        at java.lang.String.
        at :<init>(java.io.PrintStreamString.java.:write(9PrintStream.java1:7415)4
        at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.createEntry(ConnPoolByRoute.java:5)5
        at 43))sun.nio.cs.StreamEncoder$CharsetSE
        at .org.apache.http.impl.conn.tsccm.ConnPoolByRoutewriteBytes(.
StreamEncoder.java      at :com.ibm.jvm.io.ConsolePrintStreamgetEntryBlocking3.5(5writeConnPoolByRoute.java)(
        at sun.nio.cs.StreamEncoder$CharsetSE:ConsolePrintStream.java3:.implWrite(1StreamEncoder.java:777)
        at java.io.PrintStream.write(PrintStream.java:454)
        at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(StreamEncoder.java:355)
        at sun.nio.cs.StreamEncoder$CharsetSE.implFlushBuffer(StreamEncoder.java:425)
        at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:138)
        at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:187)
        at java.io.PrintStream.write(PrintStream.java:501)
        at java.io.PrintStream.print(PrintStream.java:643)
        at com.ibm.team.repotools.util.SystemErrorRecorder$1.print(SystemErrorRecorder.java:100)
        at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:803)
        at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:797)
        at java.lang.Thread.uncaughtException(Thread.java:1241)
6)
        at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300)
        at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:401)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
        at com.ibm.team.repository.service.internal.license.FloatingLicenseClient.executeMethod(FloatingLicenseClient.java:418)
        at com.ibm.team.repository.service.internal.license.FloatingLicenseClient.getFloatingLicenseList(FloatingLicenseClient.java:121)
        at com.ibm.team.repository.service.internal.license.FloatingLicenseClient.getFloatingPolicies(FloatingLicenseClient.java:151)
        at com.ibm.team.repository.service.internal.license.ManagedFloatingLicenseClient.getFloatingPolicies(ManagedFloatingLicenseClient.java:756)
        at com.ibm.team.repository.service.internal.license.LicenseService.updateRemoteFloatingLicenses(LicenseService.java:915)
        at com.ibm.team.repository.service.internal.license.LicenseService.access$8(LicenseService.java:913)
        at com.ibm.team.repository.service.internal.license.LicenseService$4$1.run(LicenseService.java:634)
        at com.ibm.team.repository.service.internal.TeamServiceContext.runAsAdmin(TeamServiceContext.java:141)
        at com.ibm.team.repository.service.internal.license.LicenseService$4.run(LicenseService.java:631)
        at java.lang.Thread.run(Thread.java:761)
416)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:159)
        at java.io.OutputStreamWriter.write(OutputStreamWriter.java:229)
        at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:122)
        at java.io.PrintStream.write(PrintStream.java:500)
        at java.io.PrintStream.print(PrintStream.java:643)
        at com.ibm.team.repotools.util.SystemErrorRecorder$1.print(SystemErrorRecorder.java:100)
        at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:803)
        at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:797)
        at java.lang.Thread.uncaughtException(Thread.java:1241)
CRJAZ1065I Failed to execute preInvoke.
CRJAZ1065I Failed to execute preInvoke.
CRJAZ1065I Failed to execute preInvoke.
CRJAZ1065I Failed to execute preInvoke.
CRJAZ1065I Failed to execute preInvoke.
CRJAZ1065I Failed to execute preInvoke.ng time: 13m:39s
CRJZS5473E An unrecoverable error stopped "triple history" indexing
The user "ADMIN" has logged out of the database "//N01DSW227:1433;databaseName=jts;user=xxxxxxxx;password=xxxxxxxx".
CRJZS5652I Indexing services are being shut down ...
CRJZS5651I Indexing services finished shutting down in 16084 ms

Comments
long TRUONG commented Feb 21 '15, 7:32 p.m.

 Our last question was answered by the action of fixing -Xmx to 4500M: It worked.

Accepted answer

0
permanent link
Ralph Schoon (63.1k●3●36●46) | answered Feb 23 '15, 2:27 a.m.
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER
If you don't follow the suggested strategies: https://jazz.net/wiki/bin/view/Deployment/BackupCLM then you will run into trouble.

The other issue is that the index files and the indexes on the database are two completely different things and there are different reindex commands in the repotools of the various apps to reindex. You need to run the right one to recreated the indexes. See https://jazz.net/library/article/1272 and the related articles about indexing.
long TRUONG selected this answer as the correct answer
Comments
long TRUONG commented Feb 23 '15, 6:44 a.m.

 Thx Ralph,


Still need to get educated on the index files and the DB index, though already learned a lot during the last 3 weeks. IBM support had not been of great help on comprehensive distinguishing the indices on the two sides.

Your answer

Register or to post your answer.


Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.