Dashboards and JFS Storage -- recovery ?
On Friday past one of our JTS was trying to index data and got this error:
2012-09-05 20:00:05,960 [jts.jfs.indexer.internal.triple] ERROR com.ibm.team.jfs - CRJZS5663 E Unable to persist tripe index com.hp.hpl.jena.tdb.base.block.BlockException: BlockMgrDirect.put at com.hp.hpl.jena.tdb.base.block.BlockMgrDirect.put(BlockMgrDirect.java:74) at com.hp.hpl.jena.tdb.base.block.BlockMgrSync.put(BlockMgrSync.java:52) at com.hp.hpl.jena.tdb.base.block.BlockMgrCache.syncFlush(BlockMgrCache.java:214) at com.hp.hpl.jena.tdb.base.block.BlockMgrCache.sync(BlockMgrCache.java:151) at com.hp.hpl.jena.tdb.index.bplustree.BPlusTree.sync(BPlusTree.java:329) at com.hp.hpl.jena.tdb.index.TupleIndexRecord.sync(TupleIndexRecord.java:224) at com.hp.hpl.jena.tdb.index.TupleTable.sync(TupleTable.java:172) at com.hp.hpl.jena.tdb.nodetable.NodeTupleTableConcrete.sync(NodeTupleTableConcrete.java:157) at com.hp.hpl.jena.tdb.store.QuadTable.sync(QuadTable.java:103) at com.hp.hpl.jena.tdb.store.DatasetGraphTDB.sync(DatasetGraphTDB.java:263) at com.hp.hpl.jena.tdb.store.DatasetGraphTDB.sync(DatasetGraphTDB.java:257) at com.hp.hpl.jena.tdb.TDB.syncObject(TDB.java:183) at com.hp.hpl.jena.tdb.TDB.sync(TDB.java:160) at com.hp.hpl.jena.tdb.TDB.sync(TDB.java:153) at com.ibm.team.jfs.rdf.internal.jenatdb.JenaTdbProvider.persist(JenaTdbProvider.java:1472) at com.ibm.team.jfs.rdf.internal.jenatdb.JenaRdfService.persist(JenaRdfService.java:222) at sun.reflect.GeneratedMethodAccessor2335.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:370) at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:356) at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportPro xyServiceRecord.java:56) at $Proxy477.persist(Unknown Source) at com.ibm.team.jfs.indexing.service.internal.task.TripleIndexAgent.persist(TripleIndexAgent.java:231) at com.ibm.team.jfs.indexing.service.internal.task.AbstractIndexAgent.idling(AbstractIndexAgent.java:478) at com.ibm.team.jfs.indexing.service.internal.task.TripleIndexAgent.idling(TripleIndexAgent.java:217) at com.ibm.team.jfs.indexing.service.internal.task.AbstractIndexAgent.run(AbstractIndexAgent.java:1062) at java.lang.Thread.run(Thread.java:736) Caused by: java.io.IOException: A file cannot be larger than the value set by ulimit. at sun.nio.ch.FileDispatcher.pwrite0(Native Method) at sun.nio.ch.FileDispatcher.pwrite(FileDispatcher.java:57) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:108) at sun.nio.ch.IOUtil.write(IOUtil.java:83) I fixed the ulimit and did the repotools-jts.sh -reindex and dashboards were then visible. Now one project area is complaining that dashboard isn't the same. I have nightly backups of the JFS storage, but this JTS has 6-7 CCM applications registered to it. Is there any hope of a recovery w/o ruination of other project's dashboards ? |
One answer
I need to ask a few questions to find out what happened here:
1. Is it a CCM project dashboard that's having the issue? If not, what type of app is it? 2. What version are the app and JTS? 3. Did the dashboard appear to lose all its data and revert back to its default, out-of-the box state, or did it change in some other way? The dashboard uses JFS to query for the dashboard resources when one is opened. If it doesn't find one then it assumes it hasn't been created yet and creates one. If there was an intermittent problem with the query service, this can result in the creation of an extra dashboard. However, if the query problem is fixed then it's supposed to come back because it takes the oldest one. So possibilities here are that either a user changed the dashboard (perhaps while trying to fix it), or there is a bug in dashboard, or there is a lingering problem in the query service. Either way, there are some manual steps to recover dashboards without reverting from a backup, however they're a bit involved and depend on the questions above. Comments 1) CCM ( upgraded from V2 , so /jazz ;-) ) 2) All CLM at 3.0.1.3 3) User provided zip file with at least 1 png showing the dashboard as of late August, the same dashboard is there but with different (fewer) tabs and one of the tabs at least now has different content. I tried to see if any Dashboard events are recorded for the project area (none), but recently a new iteration was set, but not much else. Thanks, given what you've described it doesn't sounds like the dashboard problem is related to JFS query or indexing. That only affects how it finds the dashboards. The actual state of dashboards is stored in the main database. Ignoring the possibility of a problem in the DB that would only affect this one resource (which seems like a long shot), the only possibility I can think of is that someone changed it. Unfortunately there is no auditability for dashboards in 3.x so no simple way to see who changed what. The history is actually recorded but there is no UI to tap into the history (this feature is tracked in https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/149850), although there is limited support in 4.x described in https://jazz.net/wiki/bin/view/Main/DashboardRevisionHistory. Well, looking at some of the TSM backup history for the jfs storage, some of the files there now are siginificantly smaller than what got backed up circa Sept 5-7. Would the JTS registered applications also need to get -reindex'ed ? Not for dashboards - all dashboards, even for applications, are stored on the Jazz Team Server. But some applications store their own data in their local JFS, though fairly minimal. It actually sounds like there may be a DB issue here requiring more expertise than I can provide.. I guess it's PMR time.... I checked the database server for issues; don't see full file system, etc. Thanks for your time. I did find numerous PUT by a member of the project area for at least one dashboard: X.77.156.73 - hsheppard@XX.XX.XX [06/Sep/2012:13:46:17 -0400] "PUT /jazz/proxy?uri=https%3A%2F%2Frtc3srv1%3A9443%2Fjts%2Fdashboards%2F2736%3FcontentType%3Dapplication%252Frdf%2525xml HTTP/1.1" 200 - which I think is https://rtc3srv1:9443/jts/dashboards/2736 AH HA .... That's the one. Any suggestions on further help here. So far PMR hasn't produced much success and customer is becoming irate. I did pose the work around of adding their "lost" dashboard uri to their main dashboard under "Useful Links" widget. From your previous comment, it sounded like you had found the culprit as someone was modifying the dashboard. Was that not the cause after all? My current understanding of the situation is that you have one project dashboard that is missing some content - it's not reverting back to its default state, but it just lost some tabs. Is this correct? I'm not sure if adding a link to the dashboard would help here, since it sounds like the dashboard was found, it's just missing some things. Well, that is true, the dashboard is located, but it's now a personal dashboard rather
I'd like to help but I'm still having trouble understanding the symptoms of the problem. It would help if I could get some specific examples, like the URL of the dashboard, or screenshots of before and after. Here is my understanding so far (please correct if wrong):
showing 5 of 10
show 5 more comments
|
Your answer
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.