Controlling Rational Build Forge resource usage through throttle mechanisms
IBM Rational Build Forge has several ways to control the amount of processing it attempts at one time. Use these mechanisms to either limit the impact your management console has on the rest of your environment or to increase throughput if your system can handle it. You can apply throttles at one of three levels: global, project, and server.
This document covers these topics:
Global-level throttles- AutoClean Log Settings
- Database Size Threshold and Database Size Threshold Notification
- Max Console Procs
- Run Queue Size
- Max Simultaneous Purges
- Max Simultaneous Server Tests
Global-level throttles
The global-level throttles affect the entire Rational Build Forge system. Setting these throttles too high places undue stress on the host system. Setting the throttles too low reduces the number of concurrent actions a Rational Build Forge system can perform.
Access the global-level throttles in the web interface through the Administration > System menu. Some of the throttles are internal and accessible only through the database.
Note: Change the internal throttles with extreme caution, or if directed to do so by IBM Rational support.
AutoClean Log Settings
The AutoClean log settings are set very high by default. As a result, Rational Build Forge tends to store a large amount of the system messages within the database. Modify these numbers to fit your company’s use case. There are four parameters to consider:
AutoClean Audit Log Days
Audit messages capture changes to Build Forge artifacts and administration messages. In a typical system, audit messages account for around 20 percent of the total message count. The default setting is 365 days.
AutoClean Error Log Day
Error messages capture information about when the engine is unable to complete a task, such as when the engine cannot send an email notification. These messages typically account for less than 1 percent of the total message count. The default value is 0, indicating the system never deletes these messages. Because the impact of this message type is so low, leave the value set at 0.
AutoClean Info Log Days
Information messages account for approximately 70 percent of the overall message count. This setting has the most impact in your environment. The default count is 120 days, which is too long for most installations. Typically, you should keep 7 to 14 days of information.
AutoClean Warning Log Days
The warning messages account for around 10 percent of the overall message count. By default, these messages are also kept for 120 days; however, the impact is much less than the information messages.
Database Size Threshold and Database Size Threshold Notification
These settings serve as an early warning system. Rational Build Forge sends a notice to the address specified in Database Size Threshold Notification when the database exceeds the disk size in the Database Size Threshold setting.
The Database Size Threshold setting has a default value of 2 GB, which is too low for most Rational Build Forge environments. Set the value to 80% of the maximum amount of the disk space available to the data files of the database. Using this value provides time to take corrective actions before running out of disk space.
Max Console Procs
This setting controls the maximum number of processes the console runs at one time. As of version 7.1.2, only buildforge.exe and bfproject.exe are considered processes started by the console. For optimal behavior, use a value at least 1 higher than Run Queue Size, which is covered next. By default, this setting is 25.
Run Queue Size
This setting controls how many individual builds can run at the same time. Setting this value too low limits the ability of your console to run projects in parallel. Setting this value too high might have a negative impact on your system performance. The default value of 3 is likely too low for any environment. Increase this value incrementally until either you are able to perform the tasks required of the Build Forge system, or you are overtaxing the Build Forge host system. Ranges for this value have been anywhere between 10 and 200. The value you choose largely depends on your environment and the types of builds produced. For example, a single project with 100 threaded steps is similar in processing and memory needs to 10 projects with 10 threaded steps each. A run queue size of 10 in the former case could potentially be too high as it would be equivalent to a run queue size of 100 in the second example. In both cases, understanding your environment is required to set this value properly. Using an incremental approach has proven successful in the past. You could start at 5 and step up by 5 to determine the best value for your use.
Max Simultaneous Purges
This setting is new with Build Forge 7.1.2. With this setting, you can control how many purge threads run concurrently. Setting this number higher allows a larger number of purges to occur concurrently; however, it also reduces build performance. The default setting is 20. Change this setting only if you need a lower value to control the purge process more strictly. As an alternative to this setting, consider using class purge schedules, which greatly reduce the impact that purges have in your environment and are better for controlling purging.
Max Simultaneous Server Tests
As of Rational Build Forge version 7.1.2, server tests are completed through the Java services layer. As such, the tests have a much smaller footprint on the system resources, Consequently, you can use a number for this setting larger than the default value of 6. Typically, you need a higher number of concurrent server tests if your environment has a large number of computers restart often. In this case, you require constant server tests to tell Rational Build Forge when the servers are back online.
Project-level throttles
Set these properties when defining or modifying a project.
Run Limit
This setting determines how many instances of a project can run at one time. If you launch a project and the number of active jobs equals the Run Limit, then the new job waits until at least one job completes.
Max Threads
The Max Threads setting on a project serves as the thread limit. The setting limits how many threaded steps in that project try to run at one time.
Server-level throttles
Max Jobs
Each build server has a limit for how many steps it runs at one time. Steps translate to job slots on a server. The Max Jobs default value is 3. However, in some cases, you can increase this number if the computer has the resources to complete the additional work.
This setting is potentially dangerous in situations where one step consumes more than 1/Max Jobs of the system resources. Consider a compile step that uses 80 percent of the CPU with Max Jobs set at 3. The step is using much more than 1/3 of the system resources. Increasing the Max Jobs in this scenario can quickly overload the server. On the other hand, a server that performs minor or low-cost commands can usually take quite a few more concurrent steps than 3.
Set this property when defining or modifying a server.
command_output_cache
In the bfagent.conf file on each server where an agent runs, the command_output_cache setting determines how much data that server’s agent stores before sending the data to the management console. The default value, which is also the minimum allowed value, is 2048 (bytes). Setting this value higher can significantly improve agent performance and reduce network overhead. For optimal performance, set this value to 16384 or 32768.
Internal throttle mechanisms
Several internal parameters can have a global influence on the performance of a Build Forge system. Use extreme caution when modifying any of these parameters; changing their values could have undesirable side effects.
purge_log_count
The purge_log_count option exists within the bf_sysconfig table, however it is not visible or modifiable through the Admin > System menu in the Build Forge user interface. The purge mechanism, as stated previously, is now running through the Java services layer. One of the changes involves how many log lines are deleted in a batch. The default value is 100;you can increase this value in 100-row increments to purge more at a time. The main goal with this scheme is controlling the size of an impact a purge of a large number of log lines has on the overall Build Forge environment and host system. It is generally recommended to leave this number as is unless you have large step logs for your builds.
Perl logging internal controls
As of version 7.1.2, the step logs are written through the services layer. This approach takes advantage of the transactional nature of modern RDBMS. As a result of going through the services layer, two throttle options were added through the bf_sysconfig table to control the log write process. These are hidden bf_syscofig options that are not present in any BF database by default. To implement the controls, you must insert rows manually.
use_sl_logs
For the engine: A setting of 0 or higher turns this option on. A setting of -1 or lower disables services layer logging and uses the legacy Perl logging.
For the services layer: The value determines the number of lines to receive before a forced flush. This number must be at least 300. Setting it higher allows more logs to be force flushed at one time.
To add this control to your database, use this statement:
insert into bf_sysconfig values ('1','N','use_sl_logs','300','','','',NULL,'integer','',(SELECT MAX(bf_modified) from bf_users)) use_sl_bulk_logs
This parameter only affects the Build Forge engine. The parameter turns on bulk log sending from the engine to the services layer. The value is the number of lines per step part that are sent in batch to the services layer for processing. This setting only comes into play if the use_sl_logs setting is also enabled.
The lowest and default value for this setting is 100.
To add this control to your database, use this statement:
insert into bf_sysconfig values ('2','N','use_sl_bulk_logs','100','','','',NULL,'integer','',(SELECT MAX(bf_modified) from bf_users)) Note: For Oracle databases, you might need to change back-to-back single quotation marks (”) to space-separated single quotation marks (‘ ‘) because Oracle interprets NULLs and empty strings differently.
Using these settings
These settings are based on a producer-consumer model. You control how much data the producer, bfproject, sends to the consumer, the SL or services layers. By controlling the data flow, you control how much CPU is used for each individual process. In addition, you must control how many step logs are committed at one time. The number to be most careful with is the use_sl_logs setting. Using too large a value can artificially inflate both the CPU consumption for bfproject for large step logs and the INSERT time for the step log.
Increasing the use_sl_bulk_logs setting produces only minimal increases in performance. However, it is available to be modified in different environments as needed.
Safe ranges for these settings:
use_sl_logs: 300-2000
use_sl_bulk_logs:100-1000
Use caution if using higher settings and always carefully evaluate the settings in your environment.
The settings take effect with the next build. You do not need to restart Rational Build Forge.
db_pool
The services layer opens and maintains a pool of connections to the database. This pool is drawn from for every API command received from the user interface and engine. In high-use environments, with either 50+ users or 50+ builds concurrently running, this pool can be stressed. The db_pool entry in the buildforge.conf file allows for the adjustment of how many open connections are maintained by the services layer to the database. The format is db_pool <number>. For example: db_pool 150.
Use caution when determining a value for the setting because the setting keeps these connections open and available for use with Rational Build Forge requests. Setting the number too high can exhaust the open connections allowed by the database. You might need to configure the database as well.
For more information
© Copyright 2011 IBM Corporation