Differing build queue times with different settings and versions
Accepted answer
Starting with the specified engine vs any build engine, the difference in queue time is caused by the "any build engine" option testing all supported engines before selecting a candidate engine, whereas specifying an engine will just check that one engine. Depending on the latency times for the supported engines and how many supported engines there are, this can cause an increased queue time.
For the queue time differences between 3.0.1.x and 4.0.x, there was a code change between the two versions. Previously we would check engine availability only if there was a queued build against that engine, so the loop responsible for picking up builds would only have to test those build engines that had a potential build request. However in 4.0.x, there is a change that will cause the engines to always test the connectivity as often as the threshold settings for the engine specified, which is a setting that was ignored previously. The extra checks though for the threshold, especially with the default time of three minutes can add overhead to the build request loop, which leads to longer queue times. To return to the previous 3.0.1.x behavior, disable the threshold checking, and the queue time should fall to the previous levels.
~Spencer
Comments
What is the benefit of the 4.0.x behavior over the 3.0.2.x behavior?
As of 4.0.x we are actually checking ALL BF agent engine availablities, not just those that have a request against them. Previously it would be possible for a engine to be down for a long time, and it would only be reflected in the UI when a build was put against it. Essentially we now honor the threshold settings on the BF agent engines properly, instead of only updating it when we happen to look at them.
~Spencer