Jobs queued for one server in 7.1.2 do not seem to be FIFO

We have been investigating the order of execution of build jobs that have been submitted to Build Forge. Our builds typically take several hours each, so it is important to us that the builds be executed in the order in which they were submitted.

However, instead of seeing this First In First Out (FIFO) behaviour, we are seeing a much more random order of execution which is very far from FIFO.

An experiment is described below, in which we essentially fire jobs into Build Forge at 2 minute intervals where each job lasts about 15 minutes and can only run on one possible server, with the experiment configured to ensure each job runs to completion before the next job can start execution.

We would expect to see BUILD_1 run to completion, then BUILD_2, then BUILD_3, then BUILD_4 and so on.

The actual execution order we are seeing is clearly demonstrated by a time-ordered list of the job directories created by Build Forge:

Oct 13 09:40 BUILD_1
Oct 13 09:55 BUILD_4
Oct 13 10:10 BUILD_14
Oct 13 10:23 BUILD_13
Oct 13 10:38 BUILD_8
Oct 13 10:53 BUILD_25
Oct 13 11:09 BUILD_3
Oct 13 11:24 BUILD_11
Oct 13 11:39 BUILD_21
Oct 13 11:54 BUILD_9
Oct 13 12:09 BUILD_15
Oct 13 12:24 BUILD_20
Oct 13 12:39 BUILD_6
Oct 13 12:54 BUILD_24
Oct 13 13:09 BUILD_18
Oct 13 13:24 BUILD_5
Oct 13 13:39 BUILD_2
and so on...

We would welcome any comments. FOr example:
Is Build Forge supposed to execute jobs as FIFO?
Have we misunderstood what Build Forge is supposed to do?
Have we made a fundamental mistake in the setup used in the experiment (see below for the details)?

Thanks
David Parrott


Experiment

In order to test Build Forge job queueing, we have set up a very simple arrangement as follows:


One server called "nosem1", with:
Max Jobs=1


The collector for this server is called "nosem1" and sets the following:
BF_RESERVE
STREAM: type=Set Value; value=#davep000#


We have a selector called "davep000" with the following condition:
STREAM must contain #davep000#


We have a project called "davep000" with the following properties:
Max Threads=unlimited
Run Limit=unlimited
Sticky=Sticky
Selector=davep000

The project has two simple steps:
Step 1: execute "set" to show the environment
Step 2: sleep 15 minutes


Finally we have the Scheduler set to run project "davep000" every 2 minutes. We make this schedule active for around an hour for this experiment then make the schedule inactive again.

So essentially we are firing a new "davep000" job every two minutes. Only one job can run at a time, because of the combination of "max jobs = 1", BF_RESERVE and Sticky. Each job takes a little over 15 minutes thanks to the sleep in step 2.