It's all about the answers!

Ask a question

RTC 6.0.1 - CCM goes offline after plugin deployment


vishnudharan manivannan (1182039) | asked May 12 '16, 3:46 a.m.
edited May 12 '16, 3:51 a.m.
 Hello team,

We are using RTC 6.0.1 with iFix003. (JTS + CCM + JRS)

SLES 11 SP4
IBM DB2 10.5
WAS 8.5.5 base profile

Single server topology.

Now we have developed several custom plugins for workitems and deployed in the RTC server. However we are facing issue now with the same.

Problem :

After deploying a plugin and starting the server, CCM goes offline. JTS & JRS is up and running as normal.

Steps we follow to deploy the plugin :

•Stop the jazz server(/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/stopServer.sh)
•Copy the testplugin-site folder to /opt/IBM/JazzTeamServer_6.0.1/server/conf/ccm/sites
•Copy the testplugin-site.ini file to /opt/IBM/JazzTeamServer_6.0.1/server/conf/ccm/provision_profiles
•Start the jazz server (/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/startServer.sh)
•Open URL https://hostname:9443/ccm/admin?internal=true
•Click on Request server reset
•Restart the app server
•Login to RTC server https://hostname:9443/ccm/web and make ensure RTC up and running.

Now once we follow the above steps and open the ccm/web link, we see CCM is offline.

These are the errors in the ccm.log file

 Launch callback handler] ERROR eam.repository.provision.internal.ProvisionService - CRJAZ0288E The "testplugin_Feature" profile feature could not be installed from the "file:ccm/sites/testplugin_Site" update site that is referenced in the "/opt/IBM/JazzTeamServer_6.0.1/server/conf/ccm/provision_profiles/testplugin-site.ini" profile file.

2016-05-12 09:19:50,672 [ Launch callback handler] ERROR eam.repository.provision.internal.ProvisionService - CRJAZ0287I Failed to install the bundle "file:/opt/IBM/JazzTeamServer_6.0.1/server/conf/ccm/sites/testplugin_Site/plugins/testplugin_1.0.0.201605061241.jar".
org.osgi.framework.BundleException: Error converting plugin at /fs1/WebSphere/AppServer/profiles/AppSrv01/temp/serverNode01/server1/ccm_war/ccm.war/eclipse/configuration/org.eclipse.osgi/bundles/1085/1/bundlefile.

The strange thing is that the same plugin works in our development environment (Similar setup). But when we deploy the plugin in our INT environment, we see this problem.

Any thoughts what could be the rootcause ?

Thanks,
Vishnu M


Comments
Donald Nong commented May 12 '16, 4:31 a.m.

It's related to OSGi. You should check the Eclipse log file for more details. That's /fs1/WebSphere/AppServer/profiles/AppSrv01/temp/serverNode01/server1/ccm_war/ccm.war/eclipse/workspace/.metadata/.log.

Also check the OSGi installation log. You can find it by
1. Go to /fs1/WebSphere/AppServer/profiles/AppSrv01/temp/serverNode01/server1/ccm_war/ccm.war/eclipse/
2. Run below command
find . -print | grep install.log


vishnudharan manivannan commented May 12 '16, 11:35 p.m. | edited May 12 '16, 11:53 p.m.

  Hello Donald,


There wasn't any .log under /fs1/WebSphere/AppServer/profiles/AppSrv01/temp/serverNode01/server1/ccm_war/ccm.war/eclipse/workspace/.metadata

However we had a install.log wee see the below error.

CRJAZ0288E The "testplugin_Feature" profile feature could not be installed from the "file:ccm/sites/testplugin_Site" update site that is referenced in the "/opt/IBM/JazzTeamServer_6.0.1/server/conf/ccm/provision_profiles/testplugin_Profile.ini" profile file.
com.ibm.team.repository.provision.ProvisionException: CRJAZ0287I Failed to install the bundle "file:/opt/IBM/JazzTeamServer_6.0.1/server/conf/ccm/sites/testplugin_Site/plugins/testplugin-New_1.0.0.201605121422.jar".
at com.ibm.team.repository.provision.internal.ProvisionService.installFeature(ProvisionService.java:575)
at com.ibm.team.repository.provision.internal.ProvisionService.installFromProfileProperties(ProvisionService.java:1139)
at com.ibm.team.repository.provision.internal.ProvisionService.installFromProfiles(ProvisionService.java:1196)
at com.ibm.team.repository.provision.internal.ProvisionService.installFromProfileNoWait(ProvisionService.java:1332)
at com.ibm.team.repository.provision.internal.ProvisionService.installFromProfile(ProvisionService.java:1269)
at com.ibm.team.repository.provision.internal.ProvisionService.installFromContextConfigurationUrl(ProvisionService.java:1351)
at com.ibm.team.jfs.app.install.Activator$1.run(Activator.java:177)
at java.lang.Thread.run(Thread.java:773)
Caused by: org.osgi.framework.BundleException: Error converting plugin at /fs1/WebSphere/AppServer/profiles/AppSrv01/temp/smtca00654Node01/server1/ccm_war/ccm.war/eclipse/configuration/org.eclipse.osgi/bundles/1052/1/bundlefile.
at org.eclipse.core.runtime.internal.adaptor.EclipseStorageHook.generateManifest(EclipseStorageHook.java:446)
at org.eclipse.core.runtime.internal.adaptor.EclipseStorageHook.getGeneratedManifest0(EclipseStorageHook.java:423)
at org.eclipse.core.runtime.internal.adaptor.EclipseStorageHook.getGeneratedManifest(EclipseStorageHook.java:393)
at org.eclipse.core.runtime.internal.adaptor.EclipseStorageHook.createCachedManifest(EclipseStorageHook.java:388)
at org.eclipse.core.runtime.internal.adaptor.EclipseStorageHook.getManifest(EclipseStorageHook.java:507)
at org.eclipse.osgi.internal.baseadaptor.BaseStorage.loadManifest(BaseStorage.java:306)
at org.eclipse.osgi.internal.baseadaptor.BundleInstall.begin(BundleInstall.java:82)
at org.eclipse.osgi.framework.internal.core.Framework.installWorkerPrivileged(Framework.java:921)
at org.eclipse.osgi.framework.internal.core.Framework$1.run(Framework.java:837)
at java.security.AccessController.doPrivileged(AccessController.java:280)
at org.eclipse.osgi.framework.internal.core.Framework.installWorker(Framework.java:888)
at org.eclipse.osgi.framework.internal.core.Framework.installBundle(Framework.java:832)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.installBundle(BundleContextImpl.java:167)
at com.ibm.team.repository.provision.internal.ProvisionService.installFeature(ProvisionService.java:567)
... 7 more
Caused by: org.eclipse.osgi.service.pluginconversion.PluginConversionException: Could not find a META-INF/MANIFEST.MF, plugin.xml or a fragment.xml in /fs1/WebSphere/AppServer/profiles/AppSrv01/temp/smtca00654Node01/server1/ccm_war/ccm.war/eclipse/configuration/org.eclipse.osgi/bundles/1052/1/bundlefile.
at org.eclipse.core.runtime.internal.adaptor.PluginConverterImpl.fillPluginInfo(PluginConverterImpl.java:110)
at org.eclipse.core.runtime.internal.adaptor.PluginConverterImpl.convertManifest(PluginConverterImpl.java:711)
at org.eclipse.core.runtime.internal.adaptor.EclipseStorageHook.generateManifest(EclipseStorageHook.java:443)
... 20 more
Caused by: java.util.zip.ZipException: invalid END header (bad central directory offset)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:150)
at java.util.zip.ZipFile.<init>(ZipFile.java:166)
at org.eclipse.core.runtime.internal.adaptor.PluginConverterImpl.findPluginManifest(PluginConverterImpl.java:179)
at org.eclipse.core.runtime.internal.adaptor.PluginConverterImpl.fillPluginInfo(PluginConverterImpl.java:108)
... 22 more
An error occurred while installing the feature "testplugin_Feature" from update site "file:ccm/sites/testplugin_Site".  The installation of that feature will be attempted again if other profiles are installed successfully.


Please advise..

Thank you,
Vishnu M

Accepted answer


permanent link
vishnudharan manivannan (1182039) | answered May 18 '16, 4:48 a.m.
  Hello Ralph & Donald,

We did the deployment in our Dev server. It worked and during deploying in our INT server we had this problem.

The latest log still states "CRJAZ0288E The "testplugin_Feature" profile feature could not be installed from the "file:ccm/sites/testplugin_Site" update site that is referenced in the "/opt/IBM/JazzTeamServer_6.0.1/server/conf/ccm/provision_profiles/testplugin_Profile.ini" profile file.
com.ibm.team.repository.provision.ProvisionException: CRJAZ0287I Failed to install the bundle "file:/opt/IBM/JazzTeamServer_6.0.1/server/conf/ccm/sites/testplugin_Site/plugins/testplugin-New_1.0.0.201605121422.jar"."

So we tried to restore our Linux file system to an earlier time (Before this problem started). The file system was restored successfully. Now we created a new bundle & re-deployed the plugin. This time the plugin was deployed successfully.

I see this problem is finally solved.

Thanks for your support & answers!

Regards,
Vishnu M
Ralph Schoon selected this answer as the correct answer

Comments
Ralph Schoon commented May 18 '16, 5:01 a.m.
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER

Thanks for sharing, glad you were able to solve the problem

2 other answers



permanent link
Donald Nong (14.3k211) | answered May 12 '16, 11:57 p.m.
Based on what you found in the install.log, the file /fs1/WebSphere/AppServer/profiles/AppSrv01/temp/smtca00654Node01/server1/ccm_war/ccm.war/eclipse/configuration/org.eclipse.osgi/bundles/1052/1/bundlefile is corrupted, and I believe it's just a copy of the  testplugin-New_1.0.0.201605121422.jar file.

What you can do is to make sure the file 
testplugin-New_1.0.0.201605121422.jar is in good shape, remove the folder fs1/WebSphere/AppServer/profiles/AppSrv01/temp/serverNode01/server1/ccm_war/ccm.war while restarting the server and see if the issue can be resolved.

Comments
vishnudharan manivannan commented May 13 '16, 4:03 a.m.

 Hello Donald,


We tried deleting the folder fs1/WebSphere/AppServer/profiles/AppSrv01/temp/serverNode01/server1/ccm_war/ccm.war while restarting the server but it's still the same.

we even removed WAS cache by following the below steps as mentioned in the readme.txt file provided by IBM.

Remove the contents from these directories under the profile root(e.g. <WAS_HOME>/<AppServer>/profiles/<Profile>), except for the noted help directory: temp, tranlog, wstemp, workspace. 
Note: Do not remove this directory, which is required for the CLM local help: temp/{jts_node}/{jts_server}/clmhelp_war.

But CCM still goes offline when we deploy. Any other thoughts ?


permanent link
Ralph Schoon (54.9k23642) | answered May 13 '16, 4:10 a.m.
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER
Remove the profile.ini file and the site for the extension. Enforce a reread https://rsjazz.wordpress.com/2014/06/12/is-the-extension-deployed-how-can-i-redeploy/
Make sure the server starts OK after removing the plugin.

Make sure your plugin works in Jetty.

Review all dependencies and make sure not to have any client API and unused dependencies.

Delete every file except the site.xml from the update site. Rebuild the update site and try to redeploy.

Comments
vishnudharan manivannan commented May 13 '16, 5:17 a.m.

Hello Raplh,


The following steps were followed while removing the Plugins.

1. All the sites files of the plugin were removed 
2. All the ini files of the plugins  were removed
3. deleted the builton.txt file
4. started the server.

The server is still not starting after removing the plugin. We have tried removing the WAS cache as well. but CCM is still down. Any other points to care about?



Ralph Schoon commented May 13 '16, 5:24 a.m. | edited May 13 '16, 5:30 a.m.
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER

 If the issue is due to that extension, that should have fixed it. You would have to look at the errors you see to find out more.


It is always a good idea to test deployment on a test server before going productive. You did and it works. So, maybe the problem is not the extension?  

I would review the new log and try to understand what is going on.

Your answer


Register or to post your answer.