How to Diagnose a Build Forge Test Connection Error (310 * EXEC: /bin/bash: 13)?
Hi,
Let me first apologize as this is a rather lengthy post. But as you will see, I've done a lot in trying to get this error fixed.
I have setup a Build Forge 8.0.0.17 server on RHEL 7.8. I have builds running on Windows and AIX from it. I am having trouble with the "Test Connection" to a RHEL 7.8 VM, where the agent is also at the 8.0.0.17 level.
Here's the output from the Test Connection:
200 HELLO - BuildForge Agent v8.0.0.17-0-131
320 AUTH AuthOk["optimbuild"]
320 SET EnvSet["BF_AGENT_VERSION","8.0.0.17-0-131"]
320 SET EnvSet["BF_AGENT_PLATFORM","linux 3.10.0-1127.el7.x86_64 #1 SMP Tue Feb 18 16:39:12 EST 2020"]
320 EXEC Locale["C"]
320 PTY Pty[*PtyPipe]
320 EXEC Locale["C"]
320 EXEC ExecShellPath["/bin/bash"]
300 HEARTBEAT 1
310 * SUNIL: BFBUFSIZ:[20000]
310 * EXEC: /bin/bash: 13
320 EXEC LeavingTemp["/tmp/bffJS2iN"]
310 PLAT linux 3.10.0-1127.el7.x86_64 #1 SMP Tue Feb 18 16:39:12 EST 2020
251 RESULT 13
260 EOR
You'll note from the above that the temp file is kept as I uncommented the leave_tmp_file line in /etc/bfagent.conf as a result of reading this: https://jazz.net/forum/questions/144146/buildforge-agent-status-functional-failure
I have read the documentation about invoking this from the command line. Here is what I get from that (the same error):
optimbld:/export/local>telnet opvdtstr01.utp.unicomglobal.com 5558
Trying 10.130.45.21...
Connected to opvdtstr01.utp.unicomglobal.com (10.130.45.21).
Escape character is '^]'.
200 HELLO - BuildForge Agent v8.0.0.17-0-131
username optimbuild
password *********
cmd ping
go
320 AUTH AuthOk["optimbuild"]
320 SET EnvSet["BF_AGENT_VERSION","8.0.0.17-0-131"]
320 SET EnvSet["BF_AGENT_PLATFORM","linux 3.10.0-1127.el7.x86_64 #1 SMP Tue Feb 18 16:39:12 EST 2020"]
320 EXEC Locale["C"]
320 PTY Pty[*PtyPipe]
320 EXEC Locale["C"]
320 EXEC ExecShellPath["/bin/bash"]
300 HEARTBEAT 1
310 * SUNIL: BFBUFSIZ:[20000]
310 * EXEC: /bin/bash: 13
320 EXEC LeavingTemp["/tmp/bfm9eIHn"]
310 PLAT linux 3.10.0-1127.el7.x86_64 #1 SMP Tue Feb 18 16:39:12 EST 2020
251 RESULT 13
260 EOR
quit
210 GOODBYE - BuildForge Agent v8.0.0.17-0-131
Connection closed by foreign host.
optimbld:/export/local>
I can execute /tmp/bfm9eIHn when logged into the Agent VM (opvdtstr01) with the optimbuild ID:
[optimbuild@opvdtstr01 /]$ /bin/bash /tmp/bfm9eIHn
[optimbuild@opvdtstr01 /]$ echo $?
0
[optimbuild@opvdtstr01 /]$ /tmp/bfm9eIHn
[optimbuild@opvdtstr01 /]$ echo $?
0
[optimbuild@opvdtstr01 /]$
Note that the sysconf command works:
optimbld:/export/local>telnet opvdtstr01.utp.unicomglobal.com 5558
Trying 10.130.45.21...
Connected to opvdtstr01.utp.unicomglobal.com (10.130.45.21).
Escape character is '^]'.
200 HELLO - BuildForge Agent v8.0.0.17-0-131
username optimbuild
password *
cmd sysconf
go
320 AUTH AuthOk["optimbuild"]
310 PLAT linux 3.10.0-1127.el7.x86_64 #1 SMP Tue Feb 18 16:39:12 EST 2020
310 * CPU_ARCH X86
310 * CPU_MANUFACTURER Intel
310 * CPU_MODEL 686
310 * CPU_MHZ 2000
310 * CPU_LOAD1 0.000
310 * CPU_LOAD5 0.010
310 * CPU_LOAD15 0.050
310 * NUM_CPU 4
310 * MEM_PAGESIZE 4096
310 * MEM_TOTAL 64266
310 * MEM_FREE 59759
310 * MEM_LOAD 7
310 * NET_FQDN opvdtstr01.utp.unicomglobal.com
310 * NET_HWADDR 00:50:56:90:25:95
310 * NET_IFACE ens192
310 * NET_IPV 4
310 * NET_IPV4 10.130.45.21
310 * NET_IPV6 fe80::250:56ff:fe90:2595%ens192
310 * OS_HOSTID 820
310 * OS_SYSNAME Linux
310 * OS_VERSION #1 SMP Tue Feb 18 16:39:12 EST 2020
310 * OS_RELEASE 3.10.0-1127.el7.x86_64
310 * OS_MACHINE x86_64
320 EXEC SysconfDone
251 RESULT 0
260 EOR
quit
210 GOODBYE - BuildForge Agent v8.0.0.17-0-131
Connection closed by foreign host.
optimbld:/export/local>
So I looked at the pam. If I remove /etc/pam.d/bfagent, I get this:
com.buildforge.services.common.api.APIException: CRRBF0158I: unknown account. Unable to set user account to (optimbuild). Please check whether username and password in ServerAuth are right.
So it appears that pam is working. It appears that it was initially set as a copy of sshd:
[optimbuild@opvdtstr01 pam.d]$ diff sshd bfagent
[optimbuild@opvdtstr01 pam.d]$
I also found this information: https://www.ibm.com/support/pages/node/142873 But I get this:
[root@opdvl6 db2inst1]# cd /home/buildforge/Platform
[root@opdvl6 Platform]# export BF_AGENT_DEBUG=1
[root@opdvl6 Platform]# export BF_SERVERTEST_DEBUG=1
[root@opdvl6 Platform]# export BFDEBUG_SQL_PREPARE=1
[root@opdvl6 Platform]# export BFDEBUG_SQL_DO=1
[root@opdvl6 Platform]# env | grep BF
BFDEBUG_SQL_DO=1
BFDEBUG_SQL_PREPARE=1
BF_SERVERTEST_DEBUG=1
BF_AGENT_DEBUG=1
[root@opdvl6 Platform]# ./bfservertest opvdtstr01.utp.unicomglobal.com
install_driver(DB2) failed: Can't load 'auto/DBD/DB2/DB2.so' for module DBD::DB2: auto/DBD/DB2/DB2.so: cannot open shared object file: No such file or directory at /PerlApp/DynaLoader.pm line 212.
at perlapp line 1018
Perhaps a required shared library or dll isn't installed where expected
at /PerlApp/BuildForge/DB/Handle.pm line 110
[root@opdvl6 Platform]#
Also note that from this same server, I can successfully "Test Connection" to a 7.1.3.2 agent running on RHEL 4:
200 HELLO - BuildForge Agent v7.1.3.2-0-0012
320 AUTH AuthOk["optimbld"]
320 SET EnvSet["BF_AGENT_VERSION","7.1.3.2-0-0012"]
320 SET EnvSet["BF_AGENT_PLATFORM","linux 2.6.9-78.ELsmp #1 SMP Wed Jul 9 15:39:47 EDT 2008"]
320 EXEC Locale["en_US.UTF-8"]
320 PTY Pty[*PtyPipe]
320 EXEC Locale["en_US.UTF-8"]
320 EXEC ExecShellPath["/bin/bash"]
300 HEARTBEAT 1
310 * [H[J
310 PLAT linux 2.6.9-78.ELsmp #1 SMP Wed Jul 9 15:39:47 EDT 2008
320 PING PingOk
251 RESULT 0
260 EOR
And I can also "Test Connection" to a 7.1.2.2 agent runnning RHEL 5.1:
200 HELLO - BuildForge Agent v7.1.2.2-1-0100
320 AUTH AuthOk["optimbld"]
320 SET EnvSet["BF_AGENT_VERSION","7.1.2.2-1-0100"]
320 SET EnvSet["BF_AGENT_PLATFORM","linux 2.6.18-371.11.1.el5 #1 SMP Mon Jun 30 04:53:12 EDT 2014"]
320 EXEC Locale["C"]
320 PTY Pty[*PtyPipe]
320 EXEC Locale["C"]
320 EXEC ExecShellPath["/bin/bash"]
300 HEARTBEAT 1
310 * TERM environment variable not set.
310 PLAT linux 2.6.18-371.11.1.el5 #1 SMP Mon Jun 30 04:53:12 EDT 2014
320 PING PingOk
251 RESULT 0
260 EOR
As return code 13 signifies a permissions error, I've even tried changing the permissions on the /home/optimbuild directory and the .bash* files within the directory.
So, again, I can successfully "Test Connection" to agents on RHEL 4 and 5, but not to the one on 7.8.
HELP!!
Thanks,
Rick Spagna
|
One answer
I opened a ticket at IBM and with their help, this issue is resolved. It turns out to be that SELINUX is running on the VM and it needs to be configured for BF Agent.
Here's what they gave me (between the * lines):
*
1. A. Install Build Forge Agent.
2. B. Create /etc/systemd/system/bfagent.service as suggested by customer
3. C. Install SELinux Policy 1. 1. cd /root
2. 2. touch /etc/selinux/targeted/contexts/files/file_contexts.local
3. 3. semanage login --list
4. 4. sepolgen --application /usr/local/bin/bfagent
5. 5. ./bfagent.sh [http://bfagent.sh/]
6. 6. semodule -i bfagent.pp
7. 7. semodule --enable bfagent
8. 8. semodule -l | grep bfagent
4. D. After this, use systemctl command 1. "systemctl enable bfagent.service"
2. "systemctl start bfagent.service"
5. E. Verify that bfagent process is running at the desired port 1. ps -aefZ | grep bfagent
2. netstat -naplZ | grep 5555
6. If the connection to agent fails due to Functional Failure, please follow the steps mentioned below and update.
1. 1. cd /root
2. 2. grep bfagent /var/log/audit/audit.log > /tmp/bfagent-policy.txt
3. 3. cat /tmp/bfagent-policy.txt | audit2allow -M bfagent-New-policy
4. 4. semodule -i bfagent-New-policy.pp
5. 5. Restart bfagent
1. "systemctl stop bfagent.service"
2. "systemctl start bfagent.service"
6. 6. Try to connect to console.
**
I started executing step 3.C, but when I got to 3.c.4, sepolgen was not found. I searched for it on the system and it does not exist.
So I jumped down to step 6 and executed the instructions there. One modification I did is that after step 6.2 (grep bfagent /var/log/audit/audit.log > /tmp/bfagent-policy.txt) I edited /tmp/bfagent-policy.txt to only contain the last set of lines (8) for the latest connection attempt. There were many many lines in this file. I then completed step 6.3.
Tests Connection now completes successfully!
I've done this on both VMs where I was not previously able to get a successful Test Connection and they are now both connected.
Regards,
Rick
|
Your answer
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.