Problem starting UNIX build tasks from Windows Client
I am running into a very weird error when I attempt to run a build on a UNIX machine; this suddenly started happening as this process has been working for the past month.
I request a build using a build definition that is quite simple, a command line that runs "pwd".
On the unix build machine (suse Linux64), the JBE was started with the -verbose option. Upon waking, the JBE finds and completes a "null" job. This is what I see from the build machine JBE output:
Within the RTC 3.0.1.1 rich client, the failed build returns the following in Event Details:
I have tried re-installing the JBE and buildsystem on the linux machine, created new build definitions, new build engines, without success. I am trying to understand what is meant by the errornull argument:A part's title tool tip must be non-null . Any help would be greatly appreciated because, for now, we cannot use the RTC build engine for any UNIX builds.
I request a build using a build definition that is quite simple, a command line that runs "pwd".
Command: echo
Arguments: "We are on `hostname` at `pwd`"
On the unix build machine (suse Linux64), the JBE was started with the -verbose option. Upon waking, the JBE finds and completes a "null" job. This is what I see from the build machine JBE output:
Java Version:
J2RE 1.5.0 IBM J9 2.3 Linux amd64-64 j9vmxa6423ifx-20101008 (JIT enabled)
J9VM - 20101007_66049_LHdSMr
JIT - 20100623_16197ifx1_r8
GC - 20100211_AA
2012-03-14 09:05:08 Running build loop...
2012-03-14 09:05:09 Not using a proxy to reach https://ips-rtc.swg.usma.ibm.com/jazz
2012-03-14 09:05:12 Searching for build request...
2012-03-14 09:05:12 No requests found.
2012-03-14 09:05:12
2012-03-14 09:05:12 Sleeping for 30 seconds...
2012-03-14 09:05:42 Searching for build request...
2012-03-14 09:05:45 Build "null" complete.
2012-03-14 09:05:45
2012-03-14 09:05:45 Sleeping for 30 seconds...
Within the RTC 3.0.1.1 rich client, the failed build returns the following in Event Details:
Unable to create editor ID com.ibm.team.build.ui.editors.buildResultExplorer: null argument:A part's title tool tip must be non-null
org.eclipse.core.runtime.AssertionFailedException: null argument:A part's title tool tip must be non-null
at org.eclipse.core.runtime.Assert.isNotNull(Assert.java:85)
at org.eclipse.ui.internal.PartTester.testWorkbenchPart(PartTester.java:109)
at org.eclipse.ui.internal.PartTester.testEditor(PartTester.java:37)
at org.eclipse.ui.internal.EditorReference.createPartHelper(EditorReference.java:679)
at org.eclipse.ui.internal.EditorReference.createPart(EditorReference.java:462)
at org.eclipse.ui.internal.WorkbenchPartReference.getPart(WorkbenchPartReference.java:595)
at org.eclipse.ui.internal.PartPane.setVisible(PartPane.java:313)
at org.eclipse.ui.internal.presentations.PresentablePart.setVisible(PresentablePart.java:180)
at org.eclipse.ui.internal.presentations.util.PresentablePartFolder.select(PresentablePartFolder.java:270)
at org.eclipse.ui.internal.presentations.util.LeftToRightTabOrder.select(LeftToRightTabOrder.java:65)
at org.eclipse.ui.internal.presentations.util.TabbedStackPresentation.selectPart(TabbedStackPresentation.java:473)
at org.eclipse.ui.internal.PartStack.refreshPresentationSelection(PartStack.java:1256)
at org.eclipse.ui.internal.PartStack.setSelection(PartStack.java:1209)
at org.eclipse.ui.internal.PartStack.showPart(PartStack.java:1608)
at org.eclipse.ui.internal.PartStack.add(PartStack.java:499)
at org.eclipse.ui.internal.EditorStack.add(EditorStack.java:103)
at org.eclipse.ui.internal.PartStack.add(PartStack.java:485)
at org.eclipse.ui.internal.EditorStack.add(EditorStack.java:112)
at org.eclipse.ui.internal.EditorSashContainer.addEditor(EditorSashContainer.java:63)
at org.eclipse.ui.internal.EditorAreaHelper.addToLayout(EditorAreaHelper.java:225)
at org.eclipse.ui.internal.EditorAreaHelper.addEditor(EditorAreaHelper.java:213)
at org.eclipse.ui.internal.EditorManager.createEditorTab(EditorManager.java:778)
at org.eclipse.ui.internal.EditorManager.openEditorFromDescriptor(EditorManager.java:677)
at org.eclipse.ui.internal.EditorManager.openEditor(EditorManager.java:638)
at org.eclipse.ui.internal.WorkbenchPage.busyOpenEditorBatched(WorkbenchPage.java:2854)
at org.eclipse.ui.internal.WorkbenchPage.busyOpenEditor(WorkbenchPage.java:2762)
at org.eclipse.ui.internal.WorkbenchPage.access$11(WorkbenchPage.java:2754)
at org.eclipse.ui.internal.WorkbenchPage$10.run(WorkbenchPage.java:2705)
at org.eclipse.swt.custom.BusyIndicator.showWhile(BusyIndicator.java:70)
at org.eclipse.ui.internal.WorkbenchPage.openEditor(WorkbenchPage.java:2701)
at org.eclipse.ui.internal.WorkbenchPage.openEditor(WorkbenchPage.java:2685)
at org.eclipse.ui.internal.WorkbenchPage.openEditor(WorkbenchPage.java:2668)
at com.ibm.team.build.internal.ui.tooltips.jobs.OpenEditorJob.runInUIThread(OpenEditorJob.java:58)
at org.eclipse.ui.progress.UIJob$1.run(UIJob.java:95)
at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35)
at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:134)
at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:3885)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3506)
at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:2405)
at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2369)
at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:2221)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:500)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:493)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:113)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:194)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:368)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:559)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:514)
at org.eclipse.equinox.launcher.Main.run(Main.java:1311)
I have tried re-installing the JBE and buildsystem on the linux machine, created new build definitions, new build engines, without success. I am trying to understand what is meant by the error
10 answers
On 3/14/12 6:41 , herbertg wrote:
It looks like the build is running fine, but then the error comes when
you try to open the build result editor in the Eclipse client.
I am guessing from the messages (but this is just a wild guess) that the
build label is null. But I can't think how that would happen. Do you
define the property buildLabel in either the build definition or the
build engine? Can you list the build results in the Builds view? If so,
what is in the Label column in that view?
--
David Olsen | IBM Rational | Jazz Process Team
2012-03-14 09:05:42 Searching for build request...
2012-03-14 09:05:45 Build "null" complete.
Within the RTC 3.0.1.1 rich client, the failed build returns the
following in Event Details:
Unable to create editor ID
com.ibm.team.build.ui.editors.buildResultExplorer: null argument:A
part's title tool tip must be non-null
org.eclipse.core.runtime.AssertionFailedException: null argument:A
part's title tool tip must be non-null
It looks like the build is running fine, but then the error comes when
you try to open the build result editor in the Eclipse client.
I am guessing from the messages (but this is just a wild guess) that the
build label is null. But I can't think how that would happen. Do you
define the property buildLabel in either the build definition or the
build engine? Can you list the build results in the Builds view? If so,
what is in the Label column in that view?
--
David Olsen | IBM Rational | Jazz Process Team
On 3/15/12 8:27 , herbertg wrote:
Was no work done? Or was work done but you can't see the results because
the build result editor won't open?
From the data that you have provided (and similar data from another
post yesterday), it seems that the JBE can't compute a build label when
it is run as root. And then the problem is compounded because the build
result editor can't handle an undefined build label. (I normally would
not recommend running the JBE as root, but RTC Build should not have
problems when you do.)
You should file two bugs against Build: The first for the JBE not
defining a build label when run as root. The second for the build
result editor not opening when there is no build label.
As a workaround, you could try using the buildResultPublisher Ant task
from the build toolkit to specify a build label. (I haven't tried this
myself, so I can't guarantee that it will work.)
The advice you saw for fixing the editor problem doesn't apply here
because it requires changing the source code for the editor, which you
don't control.
--
David Olsen | IBM Rational | Jazz Process Team
Thanks for the reply. To add to the information the build seems to be
picked up by the JBE, but no work is actually done - the "Build
'null' complete." is accurate only in that the task was
attempted.
Was no work done? Or was work done but you can't see the results because
the build result editor won't open?
I have discovered that if I start the JBE as a sudo (root) process,
this error occurs (ie sudo /jbe.sh -e EngineName -u userId
-p encryptedPwdFile); if started without sudo, the error does
not occur.
From the data that you have provided (and similar data from another
post yesterday), it seems that the JBE can't compute a build label when
it is run as root. And then the problem is compounded because the build
result editor can't handle an undefined build label. (I normally would
not recommend running the JBE as root, but RTC Build should not have
problems when you do.)
You should file two bugs against Build: The first for the JBE not
defining a build label when run as root. The second for the build
result editor not opening when there is no build label.
As a workaround, you could try using the buildResultPublisher Ant task
from the build toolkit to specify a build label. (I haven't tried this
myself, so I can't guarantee that it will work.)
The advice you saw for fixing the editor problem doesn't apply here
because it requires changing the source code for the editor, which you
don't control.
--
David Olsen | IBM Rational | Jazz Process Team
David,
Thanks for the reply. To add to the information the build seems to be picked up by the JBE, but no work is actually done - the "Build 'null' complete." is accurate only in that the task was attempted.
I have discovered that if I start the JBE as a sudo (root) process, this error occurs (ie sudo /jbe.sh -e EngineName -u userId -p encryptedPwdFile); if started without sudo, the error doesnot occur. All perplexing because the sudo based process initiation has been working for well over a month and have been using this off and on since October 2011. All that said, we need to be able to initiate this process by sudo/root in order that we can correctly set ownership and permissions, plus be able to start the JBE upon system restart/boot.
We do have a number of properties defined, butbuildLabel is not one of them (we do have a build.label however). I was very careful not to create any property that conflicted with the documented built-in properties.
On the build results line, the label column is empty.
I did find on the WWW that this same problem showed up in Eclipse, but the suggested solution (below which worked) does not seem to be useable in this situation.
Thanks for the reply. To add to the information the build seems to be picked up by the JBE, but no work is actually done - the "Build 'null' complete." is accurate only in that the task was attempted.
I have discovered that if I start the JBE as a sudo (root) process, this error occurs (ie sudo /jbe.sh -e EngineName -u userId -p encryptedPwdFile); if started without sudo, the error does
We do have a number of properties defined, but
On the build results line, the label column is empty.
I did find on the WWW that this same problem showed up in Eclipse, but the suggested solution (below which worked) does not seem to be useable in this situation.
Check your implementation of IEditorInput for the editor. The getToolTipText() method of the input must return a non-null entity, but the default implementation if you use an Eclipse wizard to create the class will implement a method that returns null.
On 3/15/12 8:27 , herbertg wrote:
Was no work done? Or was work done but you can't see the results because
the build result editor won't open?
No work was actually done as the build "work" is to run an ANT script which creates a (unique named) directory and populate it; the directory is not created/populated. And yes, you are correct that I am unable to see the results because the build result editor won't open due to the lack of a buildLabel being created.
From the data that you have provided (and similar data from another
post yesterday), it seems that the JBE can't compute a build label when
it is run as root. And then the problem is compounded because the build
result editor can't handle an undefined build label. (I normally would
not recommend running the JBE as root, but RTC Build should not have
problems when you do.)
You should file two bugs against Build: The first for the JBE not
defining a build label when run as root. The second for the build
result editor not opening when there is no build label.
I'm not so sure the first item is a "real" bug because, as mentioned earlier, this process has been working (in one form or another) without problems since October 2011. The 2nd is definitely a problem. How does one go about filing a bug against Build?
On 3/15/12 11:57 , herbertg wrote:
Your builds are not running and you don't know how to fix them. That's a
bug. Even if the problems ends up being on your side (maybe you
misconfigured something), it is a bug that RTC is not telling you what
went wrong.
Go to the RTC page on jazz.net
(https://jazz.net/projects/rational-team-concert/) and click on the
"Submit a bug or request" link
(https://jazz.net/jazz/web/projects/Rational%20Team%20Concert#action=com.ibm.team.workitem.newWorkItem).
The Filed Against field should be Build.
--
David Olsen | IBM Rational | Jazz Process Team
I'm not so sure the first item is a "real" bug because, as
mentioned earlier, this process has been working (in one form or
another) without problems since October 2011.
Your builds are not running and you don't know how to fix them. That's a
bug. Even if the problems ends up being on your side (maybe you
misconfigured something), it is a bug that RTC is not telling you what
went wrong.
How does one go about filing a bug against Build?
Go to the RTC page on jazz.net
(https://jazz.net/projects/rational-team-concert/) and click on the
"Submit a bug or request" link
(https://jazz.net/jazz/web/projects/Rational%20Team%20Concert#action=com.ibm.team.workitem.newWorkItem).
The Filed Against field should be Build.
--
David Olsen | IBM Rational | Jazz Process Team
Thanks David, I will enter bug tickets for these issues.
UPDATE: The below turned out to be a running process which was not noticed in the ps listing. Once it was killed this file went away.
I have some more information on the problem. In the JBE data area I've specified, there is a hidden file.nfs0000000001aa017300000033 which I am not able to delete. When I cat the contents of this file I get the following:
What I find interesting is thebuildStatus section followed by StaleDataException . I'm not really certain what this file is, but it seems to be something used by the JBE process. When the JBE process is killed, shouldn't this go away? Any idea on how to get rid of it, because when i try to delete it I get Device busy or resource busy.
I have some more information on the problem. In the JBE data area I've specified, there is a hidden file
2012-03-16 11:40:31 [Jazz build engine] Sleeping for 30 seconds...
2012-03-16 11:41:02 [Jazz build engine] Exception occurred in build loop: CRJAZ0322I 1 row was expected but 0 rows were found for handle "com.ibm.team.build.internal.common.model.impl.BuildResultImpl@32fb32fb (stateId: [UUID _eONL8G9-EeG1H7LA2kyvaQ], itemId: [UUID _dTVfIG9-EeG1H7LA2kyvaQ], origin: <unset>, immutable: true) (contextId: [UUID _kgdAsJ17Ed2rf8J-YqCQHw], modified: 2012-03-16 11:41:18.255, workingCopy: <unset>) (mergePredecessor: null, workingCopyPredecessor: <unset>, workingCopyMergePredecessor: <unset>, predecessor: [UUID _eEZvsG9-EeG1H7LA2kyvaQ]) (buildStatus: ERROR, buildState: INCOMPLETE, label: null, buildTimeTaken: 1032, buildStartTime: 1331912477223, summary: , ignoreWarnings: true, tags: , deleteAllowed: true, personalBuild: false)".
com.ibm.team.repository.common.StaleDataException: CRJAZ0322I 1 row was expected but 0 rows were found for handle "com.ibm.team.build.internal.common.model.impl.BuildResultImpl@32fb32fb (stateId: [UUID _eONL8G9-EeG1H7LA2kyvaQ], itemId: [UUID _dTVfIG9-EeG1H7LA2kyvaQ], origin: <unset>, immutable: true) (contextId: [UUID _kgdAsJ17Ed2rf8J-YqCQHw], modified: 2012-03-16 11:41:18.255, workingCopy: <unset>) (mergePredecessor: null, workingCopyPredecessor: <unset>, workingCopyMergePredecessor: <unset>, predecessor: [UUID _eEZvsG9-EeG1H7LA2kyvaQ]) (buildStatus: ERROR, buildState: INCOMPLETE, label: null, buildTimeTaken: 1032, buildStartTime: 1331912477223, summary: , ignoreWarnings: true, tags: , deleteAllowed: true, personalBuild: false)".
at com.ibm.team.repository.service.internal.dataaccess.AbstractRow.oneOrStaleData(AbstractRow.java:320)
at com.ibm.team.repository.service.internal.dataaccess.UpdateItemCurrentRow.validateResult(UpdateItemCurrentRow.java:73)
at
blah blah blah
What I find interesting is the
Usually when you get: Build "null" complete, it indicates that JBE failed trying to create the temporary build log file, which it creates in the current working directory in effect when JBE is launched.
Some things to check:
- Does the build result have a log file contribution? If so, what does it say?
- Is there a build-<timestamp>.log file in the current working directory? If it did have write access but couldn't publish the file to the build result, it's left on disk.
- Does the current working directory change between the sudo and non-sudo cases? It's odd that it's the sudo case failing, as I'd expect root to be able to write to any dir.
As for the .nfs... file, this looks like the console output from JBE. Not sure how it got there. The StaleDataException indicates that someone else tried to modify the BuildResult item after JBE read it but before it tried to save it.
This can happen if a script is issuing the request for the build (e.g. using the requestTeamBuild Ant task) and then immediately trying to modify the build result (e.g. using the buildResultPublisher Ant task) rather than waiting for it to be started. A better approach is to let the target build do any such modifications.
Some things to check:
- Does the build result have a log file contribution? If so, what does it say?
- Is there a build-<timestamp>.log file in the current working directory? If it did have write access but couldn't publish the file to the build result, it's left on disk.
- Does the current working directory change between the sudo and non-sudo cases? It's odd that it's the sudo case failing, as I'd expect root to be able to write to any dir.
As for the .nfs... file, this looks like the console output from JBE. Not sure how it got there. The StaleDataException indicates that someone else tried to modify the BuildResult item after JBE read it but before it tried to save it.
This can happen if a script is issuing the request for the build (e.g. using the requestTeamBuild Ant task) and then immediately trying to modify the build result (e.g. using the buildResultPublisher Ant task) rather than waiting for it to be started. A better approach is to let the target build do any such modifications.
Assuming the issue is due to lack of write permissions on the cwd, the following items cover it, and have been fixed in 4.0:
171059: Cannot view build result in Eclipse because of null tool-tip
171377: Poor feedback / diagnostic information when JBE unable to create build log files
I've also added a couple of extra checks in BuildResultEditorInput, piggy-backing on #171059.
171059: Cannot view build result in Eclipse because of null tool-tip
171377: Poor feedback / diagnostic information when JBE unable to create build log files
I've also added a couple of extra checks in BuildResultEditorInput, piggy-backing on #171059.
Assuming the issue is due to lack of write permissions on the cwd, the following items cover it, and have been fixed in 4.0:
171059: Cannot view build result in Eclipse because of null tool-tip
171377: Poor feedback / diagnostic information when JBE unable to create build log files
I've also added a couple of extra checks in BuildResultEditorInput, piggy-backing on #171059.
As it turns out, WI171377 was the cause of everything. I was building a command line (via a script) in order to start the JBE and using full paths for the command as well as option parameters, inadvertently overflowing the maximum length of a command line. This silently truncated the -data <path>, resolving to a non-writeable location; it's interesting to note that when you look in the log file, the -data <path> argument is not included although all the other arguments are:
Supplied arguments:
-repository https://myrepos_path/jazz -userID userID@myco.com -passwordFile ./pwd_userID@myco.com -engineId MyEngineID -verbose
By modifying my script to use relative paths, the -data <path> is kept intact and the problem described in this forum trail no longer occurs.
Thanks to all for their assistance.