It's all about the answers!

Ask a question

Run multiple personal builds of same build definition in parallel on one jenkins

Christian Giesa (13137) | asked Aug 03 '16, 1:46 a.m.
Is it possible to run multiple personal builds of the same build definition in parallel on one jenkins with multiple slaves?

Multiple developer are working on the same stream. Before they deliver their changes they would like to trigger a personal build. In a huge project the build takes approx. 1 h. If every single developer is requestion a personal build with the same build definition there will be a large queue on jenkins side. That means, they have to wait a long time until their build is finished. Is there any way to run these jobs in parallel without creating a single build definition for each developer on the same jenkins with multiple slaves?

Thanks for any input and advice.

Accepted answer

permanent link
Raghu NS (311) | answered Aug 03 '16, 4:46 a.m.
Jenkins allows for parallel execution of builds for a Job.

Job configuration page has a check box, "Execute concurrent builds if necessary". Also, in the master node configuration set the "# of executors" field to more than 1. Once these two are done, parallel job execution is enabled. This should work for triggering personal builds from RTC build definition as well.

To confirm I tried the mentioned scenario with no slaves, but was able to successfully trigger, execute and complete multiple personal builds triggered by different users from the same build definition. Each of the builds had their own workspace, with workspace name being appended with "@<number>". This is explained in context help of the "Execute concurrent builds if necessary". Also they warn about using this setting with custom workspace, in which case the workspace might be overwritten.
Christian Giesa selected this answer as the correct answer

Christian Giesa commented Aug 03 '16, 5:33 a.m.

Thank you as well for clarification and testing. This is exactly what I was looking for.

One other answer

permanent link
Ralph Schoon (63.2k33646) | answered Aug 03 '16, 2:04 a.m.
edited Aug 03 '16, 2:06 a.m.
There is no free lunch. If a build takes an hour, running 4 in parallel will take (more than) 4 hours.

The only option you have is to add more build engines and likely more build hardware. There are ways to do that in Jenkins. Not my specialty though. You can probably run more than one build process/engine on one server without over-committing it, but there will be limits.

If you have multiple build engines, each build engine will pick up the next pending request if it finishes the current request.

Also consider a caching proxy between the repository and the build engines.

This might be also of interest:

Christian Giesa commented Aug 03 '16, 4:44 a.m.

Thanks, Ralph. The option I was looking for in the jenkins job configuration is "Execute concurrent builds if necessary". This was disabled by default in our configuration. Now the builds are running in parallel on the slaves and do not block the other ones.

Nevertheless thank you for the hints and sharing your blog entry. We will think about using different build engines for personal builds and integration builds by using the property value "requestFilter" for personalBuild.

Ralph Schoon commented Aug 03 '16, 4:55 a.m.

Christian, please be aware that Jenkins also supports master slave configurations. Please be aware that there can be racing conditions if you run the same build with the same repository workspace in parallel and accept changes into this. You can create such conditions by specific automation, It should not happen in your scenario, but you should be aware.

Christian Giesa commented Aug 03 '16, 5:31 a.m.

Thanks for the information. As I understood when running a personal build the user will specify his own repository workspace, but when running for example an integration build always the same repository given in the build definition will be taken. Due to that it would be good to create two different jenkins jobs and build definitions one for personal builds and one for the integration builds. For personal builds I would allow "execute concurrent builds if necessary" for the other one I will avoid running this job in parallel. This would avoid conflicts or inconsistencies in the repository workspace of the integration build job. Am I right?

Ralph Schoon commented Aug 03 '16, 5:40 a.m.

The personal builds are not an issue, as the repository workspace is just loaded.

The issue would be if you run two integration builds in parallel, where the second build is launched and accepts incoming changes while the first build still does the same. You will then get an exception.

Your answer

Register or to post your answer.

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.