Jenkins build stays on "pending"

Simon Eickel (1.1k●7●54●57) | asked Jul 16 '13, 4:46 a.m.
edited Nov 04 '13, 5:23 p.m. by Stephanie Taylor (241●1●5)

Hi there,

we are facing the problem that our jenkins jobs stay only in state "pending".
We can start the jobs from jenkins and see the correct status in RTC when using the jenkins plugin.

But when we start the build job from RTC than it seems that no request left the RTC server - or arrives the build server...

In the logfiles nothing is written. I have put the log for log4j.category.com.ibm.rational.connector.hudson to TRACE but the only info I see in the logs are connection infos for when running "test connection".

It seems for me that the requested build comes not to the build engine as there is nothing within the build activity of the build engine.

Any idea?

Greetings,
Simon

Comments

Spencer Murata commented Jul 16 '13, 9:30 a.m.

FORUM MODERATOR / JAZZ DEVELOPER

Can you try putting the log4j to ALL? You should see more than the test connection information. If the logging is done correctly you should see a continuous stream of logging as the loops look for updates and builds to run.

~Spencer

Simon Eickel commented Jul 17 '13, 1:47 a.m.

Hi Spencer,
logging is now "log4j.category.com.ibm.rational.connector.hudson=ALL, hudson" but at the moment nothing new in the logfile ... will investigate if there are some more entries after some time...

Simon Eickel commented Jul 17 '13, 7:05 a.m.

one entry is noticeable I think:
2013-07-17 12:03:32,400 [WebContainer : 8 @@ 12:03 <username> /ccm/service/com.ibm.rational.connector.hudson.internal.common.IHudsonService] WARN com.ibm.rational.connector.hudson - The Hudson/Jenkins job <jobname> was not parameterized with the RTC build parameter "buildResultUUID". Added the parameter to the given job.

But I think as this is only a warning it's no problem.

7 answers

Most liked answers ↑|Newest answers|Oldest answers

permanent link

Ralph Schoon (63.5k●3●36●46) | answered Jul 16 '13, 7:40 a.m.
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER

Which RTC version? How have you set up the Jenkins connection, the first solution we had, the old Jenkins plugin or the newest Jenkins Plugin?

Comments

Simon Eickel commented Jul 16 '13, 8:57 a.m. | edited Jul 16 '13, 8:58 a.m.

Hi Ralph,
this is RTC 4.0.3 with the actual IBM Jenkins Plugin.
The Buildserver is a Windows 2008 so I think same setup as John said.

Unfortunately what John said didn't help :(
The Build activity list still appears empty ...
Greetings,
Simon

permanent link

John Carolan (716●1●6) | answered Jul 16 '13, 8:48 a.m.

I saw the same behaviour in RTC 4.0.3 using the latest plugin on Windows 2008 Server.
It happened immediately after I setup the new Build Definition.

However, when I restarted Jenkins and requested a new build, the operation was successful.

Within Jenkins UI, I did Jenkins->Manage Jenkins->"Reload Configuration from Disk"

The original 'pending' build did not succeed, but the builds requested after the Jenkins refresh worked fine.

Comments

Simon Eickel commented Jul 16 '13, 8:57 a.m.

hi John,
thanks for your answer - I checked it but unfortunately it did not work for us :(
Any other ideas?

John Carolan commented Jul 16 '13, 9:12 a.m.

Hi Simon,

Is it possible that the build is setup to require a parameter in Jenkins?

Regards,

John

Spencer Murata commented Jul 16 '13, 9:17 a.m.

FORUM MODERATOR / JAZZ DEVELOPER

As of 4.0.3 it is required for the Jenkins jobs to have parameters. We now use the Jenkins parameters to track which Jenkins job is linked to which RTC build result.

~Spencer

John Carolan commented Jul 16 '13, 9:18 a.m.

It may also help to click on the refresh icon to the right of the Builds tab in RTC Eclipse Client - that will update the 'Progress' and 'Estimated Completion' columns with the latest status from Jenkins.

John Carolan commented Jul 16 '13, 9:28 a.m.

I was thinking of additional custom parameters as configured in Jenkins. For example, if I require a new string parameter 'test' but don't supply that parameter within the Build Definition Properties tab, or within the 'Build Properties' section of the Request Build dialog, then the build remains 'pending' until I supply the value and refresh.

Simon Eickel commented Jul 17 '13, 1:55 a.m.

Hi John, Hi Spencer,
refreshing changes nothing. The build queue stays in pending and the build activity on the build engine stays empty. The "Show Builds for engine" stays empty, too. I have no manually defined properties inside my build.
When taking a look at the build engine sometimes I have a warning that this engine seems to be not running ... but this warning I have seen on working builds, too.

build definition properties:

build queue:

build activity (build engine):

John Carolan commented Jul 17 '13, 7:19 a.m.

Hi Simon,

I think that warning might be important. If you open up the Build Engine, does it have the 'Build engine process polls for requests' toggle checked on?

Do you see the same behaviour if you create a new Build Engine using all default values?

John

Simon Eickel commented Jul 18 '13, 8:18 a.m. | edited Jul 18 '13, 8:24 a.m.

Hi John,
yes, this option is enabled:

The warning is gone when creating a build engine and not to connect it to a build definition. As soon as I connect it the warning appears.

Simon Eickel commented Jul 18 '13, 8:36 a.m.

when creating a request using the predefined properties (e.g. take the build engine supported with) he tells me the following:

John Carolan commented Jul 18 '13, 11:24 a.m.

Hi Simon,

That particular message was reported before on a different OS (https://jazz.net/forum/questions/33874/build-enginee-process-is-not-running) and it was related to restrictive permissions on the run area. I am not sure if it's applicable to your case - but perhaps running the build engine as an Administrator may workaround such a problem.
Similarly, making sure that all the services are running with the same JRE as the JTS may help.

John

Simon Eickel commented Jul 22 '13, 3:15 a.m.

Hi John,
as the build were running on friday fine and just since monday they didn't work anymore I doubt that this is a permissions problem within the team area.
But thanks for this tipp :)

showing 5 of 11 show 6 more comments

permanent link

Nick Edgar (6.5k●7●11) | answered Jul 17 '13, 11:03 a.m.
JAZZ DEVELOPER

The "The Hudson/Jenkins job <jobname> was not parameterized with the RTC build parameter "buildResultUUID". Added the parameter to the given job." warning is informational, not actually a warning. It's letting you know that it successfully added the buildResultUUID parameter to the Jenkins job (you could double-check to ensure this is the case). It should really be changed to INFO instead of WARNING. I've filed an issue for this.

Here are a few more questions / things to try:

do the RTC builds stay in pending, or are they showing as in-progress (but not actually making progress)?
in the build engine config, ensure the Jenkins URL has a trailing slash (known regression in 4.0.3)
do you have security enabled, i.e. are you using user id/password to login? If so ensure permissions are granted to be able to read/write the relevant job, their configs, and their builds (job config case is a known regression in 4.0.3, but you wouldn't have gotten log message above if it couldn't update the job).
if the buildResultUUID parameter was not in fact added automatically, add it manually to the Jenkins job (its name is case sensitive; default value should be blank)
check for any errors in the RTC build result's logs
check for any other errors in the RTC server log
check CCM admin web UI status page for any errors indicating if any services have stopped due to errors

Comments

Simon Eickel commented Jul 18 '13, 8:49 a.m.

Hi Nick
1) it stays in pending - nothing happens anymore no further progress
2) trailing slash is there but changes nothing
3) security is enabled and checked - everythings fine with
4) buildRequestUUID prop is set manually within build definition - value is empty - but changes nothing
5) there are no build results log's as the job does neither start nor finish
6) there are errors/warnings in the log files but for really other times and as I can say they do not belong to this problem:
ERROR .service.internal.save.notify.ChangeEventGenerator - CRRTC0255E: An error occurred while registering the approval request on a work item. The approval was saved, but a notification will not be sent to approvers.
All the warnings are warnings belonging to work items
7) no errors appears within CCM admin web UI

This is a very hard problem as no personal builds are running when created as Jenkins builds ...

permanent link

Simon Eickel (1.1k●7●54●57) | answered Jul 22 '13, 3:19 a.m.

Hi there,

thanks for all your ideas.
Since today the Jenkins builds are running fine again.

What did I do? - Nothing

Every week on Monday morning (1 AM) our productive server gets restarted.
Both the database server and the application server. I did a restart of the application server last week but this didn't resolve this issue.
The restart of both - database and application server seemed to get it working again.
Any idea what the issue could be?

Maybe a inconsistent entry in the db which got slapped by rebooting?

Greetings,
Simon

permanent link

Simon Eickel (1.1k●7●54●57) | answered Oct 16 '13, 5:13 a.m.

unfortunately this error occured again.
Since monday all our Builds stays on pending when starting them from the RTC client.

When starting using the Jenkins dashboard everything is fine.

When throwing an eye on the "active services" of ccm I noticed that the task "com.ibm.rational.hudson.team.internal.service.HudsonSyncLoopScheduledTask.executeTask" is open twice.

A restart of both servers (DB and Application) did not help this time.

Any more ideas would be great :)

Comments

Nick Edgar commented Oct 16 '13, 8:58 a.m.

JAZZ DEVELOPER

The two tasks shown are actually different. The BuildLoop one checks for new builds in RTC and sends them to Hudson/Jenkins. The SyncLoop one monitors the status for existing builds, updating them with the latest info from Hudson/Jenkins.

They both show as having been running for almost a day and a half, so must be hung up on something. Can click Show Details on both, and include the stack traces here?

Nick Edgar commented Oct 16 '13, 9:00 a.m.

JAZZ DEVELOPER

Can you also check the RTC server log? Probably called ccm.log or jazz.log under tomcat/server/logs (if using Tomcat).

Simon Eickel commented Oct 17 '13, 1:22 a.m.

Hi Nick,
thanks for clarification. Yes, Right after posting I noticed that those tasks are different. But when checking our "quality assurance" server I noticed that there is just the HudsonSyncLoopScheduledTask and on this server everything is working fine. But on both server those tasks seems to get started when the server starts up and running from then on. So I don't think they are stuck.
Here is the stack for the HudsonSyncLoopScheduledTask:

Simon Eickel commented Oct 17 '13, 1:23 a.m.

Here is the stack for the HudsonBuildLoopScheduledTask:

Simon Eickel commented Oct 17 '13, 1:34 a.m.

I restarted the server again tonight to have a clean overview of the logs and as expected there are not many entries inside.
Those are the only hudson entries:

I set my log4j that all hudson entried will be logged into an own file.

Nick Edgar commented Oct 17 '13, 8:23 a.m.

JAZZ DEVELOPER

If the tasks show as being running for any length of time greater than a few minutes, then they're definitely stuck. They normally run once every 15 seconds, as a separate invocation each time. Can you please check whether there are any Hudson/Jenkins type build engines defined whose URL may refer to an unresponsive server? Test Connection should indicate this, but you could also try the URL in a browser.

If you're not sure whether there are others on that RTC server, please have an admin user access: https://HOST:PORT/ccm/resource/virtual/build/engines?_prettyPrint=true

which lists the engines in JSON format. Search for any with a "hudson" config element. The "id" attribute gives the engine ID.

Simon Eickel commented Oct 18 '13, 2:43 a.m.

Hi Nick,
thanks for that information.
You're right this task should run every 15 second. But why does everything work on our qual server even when this task seems to be stuck?

At the moment everything is working again on the prod system - but we did nothing.

Noting you're tip it could mean, that if someone is using a build engine definition whose URL may refer to an unresponsive server this could use our system to crash ...
Have to check this and will come back with my results ... maybe again in 3 months :D

not really a good part.

Erik anderson commented Oct 30 '13, 5:39 p.m.

We are having the same problem and it was working fine up until about the time the "com.ibm.rational.hudson.team.internal.service.HudsonBuildLoopScheduledTask.executeTask" service started (it's been running for over 3 hours). Now our builds can not be started from RTC (they stay in pending state). Starting the job from the jenkins side works fine and results are visible in RTC.

I've followed all advise given in this thread without luck. Short of restarting the RTC server, is there anything else we can do? Good opportunity to debug this if development is up for it.

Simon Eickel commented Oct 31 '13, 4:00 a.m.

Hi Erik,
one suggestion is to check all your existing build engine (use administrative account for it to see all).
Is there one which could stuck? E.g. One definition with two build engines where one engine uses a Jenkins server on which the Jenkins job is not established?

Erik anderson commented Oct 31 '13, 3:19 p.m.

We tried that (removed all engines and definitions) but the BuildLoop ("com.ibm.rational.hudson.team.internal.service.HudsonBuildLoopScheduledTask.executeTask" )service remains wedged and we are not able to start jobs from RTC.

The SyncLoop is working fine as I can start jobs in jenkins and see the results in RTC.

Erik anderson commented Nov 01 '13, 8:46 a.m.

Restarting the server fixed our problem (starting jobs in RTC kicks off the jenkins job)

Simon Eickel commented Nov 04 '13, 1:17 a.m.

Hi Erik,
yes, that's the behavior we had, too. But unfortunately the last time a restart did not fix the problem (restarted both - db server and application server more than one time). :(

showing 5 of 12 show 7 more comments

permanent link

John Carolan (716●1●6) | answered Oct 16 '13, 10:46 a.m.

Hi Simon,

I noticed this defect in the recent Jenkins distributions: https://issues.jenkins-ci.org/browse/JENKINS-12994

If you're using 'Quiet period' in the Advanced Project Options of the build configuration in Jenkins, it might be worthwhile unchecking that toggle in case it is causing a problem.

I hope that helps,

John

Comments

Simon Eickel commented Oct 17 '13, 1:13 a.m.

Hi John,
thanks for this information. "Unfortunately" we are not using "Quite period". So this is not our problem. Never the less I tried it using this function but this does even not work.

permanent link

Leonardo Marzo (249●6●48●52) | answered Jul 23 '14, 2:13 p.m.

We're having the same issue with RTC 4.0.5. Adding the trailing slash fix it. That's weird because it was supposed to be fixed in 4.0.4:

https://jazz.net/jazz/web/projects/Rational%20Team%20Concert#action=com.ibm.team.workitem.viewWorkItem&id=272761

Your answer

Register or log in to post your answer.

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.

It's all about the answers!

Related questions

Jenkins build stays on "pending"

7 answers

Your answer