It's all about the answers!

Ask a question

Dashboards and JFS Storage -- recovery ?


0
1
Kevin Ramer (4.4k6162183) | asked Sep 10 '12, 11:15 a.m.
On Friday past one of our JTS was trying to index data and got this error:

2012-09-05 20:00:05,960 [jts.jfs.indexer.internal.triple] ERROR com.ibm.team.jfs - CRJZS5663
E Unable to persist tripe index
com.hp.hpl.jena.tdb.base.block.BlockException: BlockMgrDirect.put
at com.hp.hpl.jena.tdb.base.block.BlockMgrDirect.put(BlockMgrDirect.java:74)
at com.hp.hpl.jena.tdb.base.block.BlockMgrSync.put(BlockMgrSync.java:52)
at com.hp.hpl.jena.tdb.base.block.BlockMgrCache.syncFlush(BlockMgrCache.java:214)
at com.hp.hpl.jena.tdb.base.block.BlockMgrCache.sync(BlockMgrCache.java:151)
at com.hp.hpl.jena.tdb.index.bplustree.BPlusTree.sync(BPlusTree.java:329)
at com.hp.hpl.jena.tdb.index.TupleIndexRecord.sync(TupleIndexRecord.java:224)
at com.hp.hpl.jena.tdb.index.TupleTable.sync(TupleTable.java:172)
at com.hp.hpl.jena.tdb.nodetable.NodeTupleTableConcrete.sync(NodeTupleTableConcrete.java:157)
at com.hp.hpl.jena.tdb.store.QuadTable.sync(QuadTable.java:103)
at com.hp.hpl.jena.tdb.store.DatasetGraphTDB.sync(DatasetGraphTDB.java:263)
at com.hp.hpl.jena.tdb.store.DatasetGraphTDB.sync(DatasetGraphTDB.java:257)
at com.hp.hpl.jena.tdb.TDB.syncObject(TDB.java:183)
at com.hp.hpl.jena.tdb.TDB.sync(TDB.java:160)
at com.hp.hpl.jena.tdb.TDB.sync(TDB.java:153)
at com.ibm.team.jfs.rdf.internal.jenatdb.JenaTdbProvider.persist(JenaTdbProvider.java:1472)
at com.ibm.team.jfs.rdf.internal.jenatdb.JenaRdfService.persist(JenaRdfService.java:222)
at sun.reflect.GeneratedMethodAccessor2335.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:370)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:356)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportPro
xyServiceRecord.java:56)
at $Proxy477.persist(Unknown Source)
at com.ibm.team.jfs.indexing.service.internal.task.TripleIndexAgent.persist(TripleIndexAgent.java:231)
at com.ibm.team.jfs.indexing.service.internal.task.AbstractIndexAgent.idling(AbstractIndexAgent.java:478)
at com.ibm.team.jfs.indexing.service.internal.task.TripleIndexAgent.idling(TripleIndexAgent.java:217)
at com.ibm.team.jfs.indexing.service.internal.task.AbstractIndexAgent.run(AbstractIndexAgent.java:1062)
at java.lang.Thread.run(Thread.java:736)
Caused by:
java.io.IOException: A file cannot be larger than the value set by ulimit.
at sun.nio.ch.FileDispatcher.pwrite0(Native Method)
at sun.nio.ch.FileDispatcher.pwrite(FileDispatcher.java:57)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:108)
at sun.nio.ch.IOUtil.write(IOUtil.java:83)

I fixed the ulimit and did the repotools-jts.sh -reindex and dashboards were then visible.  Now one project area is complaining that dashboard isn't the same.  I have nightly backups of the JFS storage, but this JTS has 6-7 CCM applications registered to it.   Is there any hope of a recovery w/o ruination of other project's dashboards ?


One answer



permanent link
Curtis d'Entremont (1.3k3) | answered Sep 10 '12, 11:35 a.m.
FORUM MODERATOR / JAZZ DEVELOPER
I need to ask a few questions to find out what happened here:

1. Is it a CCM project dashboard that's having the issue? If not, what type of app is it?
2. What version are the app and JTS?
3. Did the dashboard appear to lose all its data and revert back to its default, out-of-the box state, or did it change in some other way?

The dashboard uses JFS to query for the dashboard resources when one is opened. If it doesn't find one then it assumes it hasn't been created yet and creates one. If there was an intermittent problem with the query service, this can result in the creation of an extra dashboard. However, if the query problem is fixed then it's supposed to come back because it takes the oldest one. So possibilities here are that either a user changed the dashboard (perhaps while trying to fix it), or there is a bug in dashboard, or there is a lingering problem in the query service. Either way, there are some manual steps to recover dashboards without reverting from a backup, however they're a bit involved and depend on the questions above.

Comments
Kevin Ramer commented Sep 10 '12, 11:42 a.m.

1) CCM ( upgraded from V2 , so /jazz ;-) ) 2) All CLM at 3.0.1.3 3) User provided zip file with at least 1 png showing the dashboard as of late August, the same dashboard is there but with different (fewer) tabs and one of the tabs at least now has different content.

I tried to see if any Dashboard events are recorded for the project area (none), but recently a new iteration was set, but not much else.


Curtis d'Entremont commented Sep 10 '12, 11:54 a.m.
FORUM MODERATOR / JAZZ DEVELOPER

Thanks, given what you've described it doesn't sounds like the dashboard problem is related to JFS query or indexing. That only affects how it finds the dashboards. The actual state of dashboards is stored in the main database. Ignoring the possibility of a problem in the DB that would only affect this one resource (which seems like a long shot), the only possibility I can think of is that someone changed it.

Unfortunately there is no auditability for dashboards in 3.x so no simple way to see who changed what. The history is actually recorded but there is no UI to tap into the history (this feature is tracked in https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/149850), although there is limited support in 4.x described in https://jazz.net/wiki/bin/view/Main/DashboardRevisionHistory.


Kevin Ramer commented Sep 10 '12, 1:19 p.m.

Well, looking at some of the TSM backup history for the jfs storage, some of the files there now are siginificantly smaller than what got backed up circa Sept 5-7.

Would the JTS registered applications also need to get -reindex'ed ?


Curtis d'Entremont commented Sep 10 '12, 1:24 p.m.
FORUM MODERATOR / JAZZ DEVELOPER

Not for dashboards - all dashboards, even for applications, are stored on the Jazz Team Server. But some applications store their own data in their local JFS, though fairly minimal. It actually sounds like there may be a DB issue here requiring more expertise than I can provide..


Kevin Ramer commented Sep 10 '12, 2:00 p.m.

I guess it's PMR time.... I checked the database server for issues; don't see full file system, etc. Thanks for your time.


Kevin Ramer commented Sep 10 '12, 5:21 p.m.

I did find numerous PUT by a member of the project area for at least one dashboard:

X.77.156.73 - hsheppard@XX.XX.XX [06/Sep/2012:13:46:17 -0400] "PUT /jazz/proxy?uri=https%3A%2F%2Frtc3srv1%3A9443%2Fjts%2Fdashboards%2F2736%3FcontentType%3Dapplication%252Frdf%2525xml HTTP/1.1" 200 -

which I think is https://rtc3srv1:9443/jts/dashboards/2736

AH HA ....

That's the one.


Kevin Ramer commented Sep 24 '12, 3:07 p.m.

Any suggestions on further help here. So far PMR hasn't produced much success and customer is becoming irate. I did pose the work around of adding their "lost" dashboard uri to their main dashboard under "Useful Links" widget.


Curtis d'Entremont commented Sep 24 '12, 5:29 p.m.
FORUM MODERATOR / JAZZ DEVELOPER

From your previous comment, it sounded like you had found the culprit as someone was modifying the dashboard. Was that not the cause after all? My current understanding of the situation is that you have one project dashboard that is missing some content - it's not reverting back to its default state, but it just lost some tabs. Is this correct? I'm not sure if adding a link to the dashboard would help here, since it sounds like the dashboard was found, it's just missing some things.


Kevin Ramer commented Oct 08 '12, 5:11 p.m.

Well, that is true, the dashboard is located, but it's now a personal dashboard rather
than Project dashboard.  Users are becoming irate that my PMR is getting no response and I don't want to cause further damage.

Since it's now a personal dashboard its widgets cannot be copied onto another Project dashboard because that would be 'cross server'.   (jts has different host name than ccm)





Curtis d'Entremont commented Oct 09 '12, 11:45 a.m.
FORUM MODERATOR / JAZZ DEVELOPER

I'd like to help but I'm still having trouble understanding the symptoms of the problem. It would help if I could get some specific examples, like the URL of the dashboard, or screenshots of before and after. Here is my understanding so far (please correct if wrong):

1. User had a personal dashboard with content. The URL to the dashboard was something like /jts/dashboards/123.
2. Something happened, now when they go back to that same dashboard with the same URL, there is less content than before - some of it was lost.

Feel free to email me directly with screenshots/data (curtispd@ca.ibm.com) if you're concerned about exposing sensitive data. Here's what would help me diagnose the issue:

1. Before and after screenshot, including full URL visible in address bar.
2. Was it a one time issue triggered by the original error described in this post, or is it an ongoing issue where dashboard data continues to be lost?
3. Error log from Jazz Team Server - jts.log.

showing 5 of 10 show 5 more comments

Your answer


Register or to post your answer.