It's all about the answers!

Ask a question

How to Diagnose a Build Forge Test Connection Error (310 * EXEC: /bin/bash: 13)?


Rick Spagna (112) | asked Dec 31 '20, 2:04 p.m.
Hi,

Let me first apologize as this is a rather lengthy post. But as you will see, I've done a lot in trying to get this error fixed.

I have setup a Build Forge 8.0.0.17 server on RHEL 7.8. I have builds running on Windows and AIX from it. I am having trouble with the "Test Connection" to a RHEL 7.8 VM, where the agent is also at the 8.0.0.17 level.

Here's the output from the Test Connection:

    200 HELLO - BuildForge Agent v8.0.0.17-0-131
    320 AUTH AuthOk["optimbuild"]
    320 SET EnvSet["BF_AGENT_VERSION","8.0.0.17-0-131"]
    320 SET EnvSet["BF_AGENT_PLATFORM","linux 3.10.0-1127.el7.x86_64 #1 SMP Tue Feb 18 16:39:12 EST 2020"]
    320 EXEC Locale["C"]
    320 PTY Pty[*PtyPipe]
    320 EXEC Locale["C"]
    320 EXEC ExecShellPath["/bin/bash"]
    300 HEARTBEAT 1
    310 * SUNIL: BFBUFSIZ:[20000] 
    310 * EXEC: /bin/bash: 13
    320 EXEC LeavingTemp["/tmp/bffJS2iN"]
    310 PLAT linux 3.10.0-1127.el7.x86_64 #1 SMP Tue Feb 18 16:39:12 EST 2020
    251 RESULT 13
    260 EOR

You'll note from the above that the temp file is kept as I uncommented the leave_tmp_file line in /etc/bfagent.conf as a result of reading this: https://jazz.net/forum/questions/144146/buildforge-agent-status-functional-failure

I have read the documentation about invoking this from the command line. Here is what I get from that (the same error):

    optimbld:/export/local>telnet opvdtstr01.utp.unicomglobal.com 5558
    Trying 10.130.45.21...
    Connected to opvdtstr01.utp.unicomglobal.com (10.130.45.21).
    Escape character is '^]'.
    200 HELLO - BuildForge Agent v8.0.0.17-0-131
    username optimbuild
    password *********
    cmd ping
    go
    320 AUTH AuthOk["optimbuild"]
    320 SET EnvSet["BF_AGENT_VERSION","8.0.0.17-0-131"]
    320 SET EnvSet["BF_AGENT_PLATFORM","linux 3.10.0-1127.el7.x86_64 #1 SMP Tue Feb 18 16:39:12 EST 2020"]
    320 EXEC Locale["C"]
    320 PTY Pty[*PtyPipe]
    320 EXEC Locale["C"]
    320 EXEC ExecShellPath["/bin/bash"]
    300 HEARTBEAT 1
    310 * SUNIL: BFBUFSIZ:[20000]
    310 * EXEC: /bin/bash: 13
    320 EXEC LeavingTemp["/tmp/bfm9eIHn"]
    310 PLAT linux 3.10.0-1127.el7.x86_64 #1 SMP Tue Feb 18 16:39:12 EST 2020
    251 RESULT 13
    260 EOR
    quit
    210 GOODBYE - BuildForge Agent v8.0.0.17-0-131
    Connection closed by foreign host.
    optimbld:/export/local>
    
I can execute /tmp/bfm9eIHn when logged into the Agent VM (opvdtstr01) with the optimbuild ID:

    [optimbuild@opvdtstr01 /]$ /bin/bash /tmp/bfm9eIHn
    [optimbuild@opvdtstr01 /]$ echo $?
    0
    [optimbuild@opvdtstr01 /]$ /tmp/bfm9eIHn
    [optimbuild@opvdtstr01 /]$ echo $?
    0
    [optimbuild@opvdtstr01 /]$

Note that the sysconf command works:

    optimbld:/export/local>telnet opvdtstr01.utp.unicomglobal.com 5558
    Trying 10.130.45.21...
    Connected to opvdtstr01.utp.unicomglobal.com (10.130.45.21).
    Escape character is '^]'.
    200 HELLO - BuildForge Agent v8.0.0.17-0-131
    username optimbuild
    password *
    cmd sysconf
    go
    320 AUTH AuthOk["optimbuild"]
    310 PLAT linux 3.10.0-1127.el7.x86_64 #1 SMP Tue Feb 18 16:39:12 EST 2020
    310 * CPU_ARCH X86
    310 * CPU_MANUFACTURER Intel
    310 * CPU_MODEL 686
    310 * CPU_MHZ 2000
    310 * CPU_LOAD1 0.000
    310 * CPU_LOAD5 0.010
    310 * CPU_LOAD15 0.050
    310 * NUM_CPU 4
    310 * MEM_PAGESIZE 4096
    310 * MEM_TOTAL 64266
    310 * MEM_FREE 59759
    310 * MEM_LOAD 7
    310 * NET_FQDN opvdtstr01.utp.unicomglobal.com
    310 * NET_HWADDR 00:50:56:90:25:95
    310 * NET_IFACE ens192
    310 * NET_IPV 4
    310 * NET_IPV4 10.130.45.21
    310 * NET_IPV6 fe80::250:56ff:fe90:2595%ens192
    310 * OS_HOSTID 820
    310 * OS_SYSNAME Linux
    310 * OS_VERSION #1 SMP Tue Feb 18 16:39:12 EST 2020
    310 * OS_RELEASE 3.10.0-1127.el7.x86_64
    310 * OS_MACHINE x86_64
    320 EXEC SysconfDone
    251 RESULT 0
    260 EOR
    quit
    210 GOODBYE - BuildForge Agent v8.0.0.17-0-131
    Connection closed by foreign host.
    optimbld:/export/local>

So I looked at the pam. If I remove /etc/pam.d/bfagent, I get this:

    com.buildforge.services.common.api.APIException: CRRBF0158I: unknown account. Unable to set user account to (optimbuild). Please check whether username and password in ServerAuth are right.

So it appears that pam is working. It appears that it was initially set as a copy of sshd:

    [optimbuild@opvdtstr01 pam.d]$ diff sshd bfagent
    [optimbuild@opvdtstr01 pam.d]$

I also found this information: https://www.ibm.com/support/pages/node/142873 But I get this:

    [root@opdvl6 db2inst1]# cd /home/buildforge/Platform
    [root@opdvl6 Platform]# export BF_AGENT_DEBUG=1
    [root@opdvl6 Platform]#  export BF_SERVERTEST_DEBUG=1
    [root@opdvl6 Platform]# export BFDEBUG_SQL_PREPARE=1
    [root@opdvl6 Platform]# export BFDEBUG_SQL_DO=1
    [root@opdvl6 Platform]# env | grep BF
    BFDEBUG_SQL_DO=1
    BFDEBUG_SQL_PREPARE=1
    BF_SERVERTEST_DEBUG=1
    BF_AGENT_DEBUG=1
    [root@opdvl6 Platform]# ./bfservertest opvdtstr01.utp.unicomglobal.com
    install_driver(DB2) failed: Can't load 'auto/DBD/DB2/DB2.so' for module DBD::DB2: auto/DBD/DB2/DB2.so: cannot open shared object file: No such file or directory at /PerlApp/DynaLoader.pm line 212.
     at perlapp line 1018
    Perhaps a required shared library or dll isn't installed where expected
     at /PerlApp/BuildForge/DB/Handle.pm line 110
    [root@opdvl6 Platform]# 

Also note that from this same server, I can successfully "Test Connection" to a 7.1.3.2 agent running on RHEL 4:

    200 HELLO - BuildForge Agent v7.1.3.2-0-0012
    320 AUTH AuthOk["optimbld"]
    320 SET EnvSet["BF_AGENT_VERSION","7.1.3.2-0-0012"]
    320 SET EnvSet["BF_AGENT_PLATFORM","linux 2.6.9-78.ELsmp #1 SMP Wed Jul 9 15:39:47 EDT 2008"]
    320 EXEC Locale["en_US.UTF-8"]
    320 PTY Pty[*PtyPipe]
    320 EXEC Locale["en_US.UTF-8"]
    320 EXEC ExecShellPath["/bin/bash"]
    300 HEARTBEAT 1
    310 * 
    310 PLAT linux 2.6.9-78.ELsmp #1 SMP Wed Jul 9 15:39:47 EDT 2008
    320 PING PingOk
    251 RESULT 0
    260 EOR
    
And I can also "Test Connection" to a 7.1.2.2 agent runnning RHEL 5.1:

    200 HELLO - BuildForge Agent v7.1.2.2-1-0100
    320 AUTH AuthOk["optimbld"]
    320 SET EnvSet["BF_AGENT_VERSION","7.1.2.2-1-0100"]
    320 SET EnvSet["BF_AGENT_PLATFORM","linux 2.6.18-371.11.1.el5 #1 SMP Mon Jun 30 04:53:12 EDT 2014"]
    320 EXEC Locale["C"]
    320 PTY Pty[*PtyPipe]
    320 EXEC Locale["C"]
    320 EXEC ExecShellPath["/bin/bash"]
    300 HEARTBEAT 1
    310 * TERM environment variable not set.
    310 PLAT linux 2.6.18-371.11.1.el5 #1 SMP Mon Jun 30 04:53:12 EDT 2014
    320 PING PingOk
    251 RESULT 0
    260 EOR

As return code 13 signifies a permissions error, I've even tried changing the permissions on the /home/optimbuild directory and the .bash* files within the directory.

So, again, I can successfully "Test Connection" to agents on RHEL 4 and 5, but not to the one on 7.8.

HELP!!

Thanks,
Rick Spagna

One answer



permanent link
Rick Spagna (112) | answered Jan 14 '21, 10:30 a.m.
edited Jan 14 '21, 10:34 a.m.

 I opened a ticket at IBM and with their help, this issue is resolved. It turns out to be that SELINUX is running on the VM and it needs to be configured for BF Agent.


Here's what they gave me (between the * lines):

*
 1. A. Install Build Forge Agent.
 2. B. Create /etc/systemd/system/bfagent.service as suggested by customer
 3. C. Install SELinux Policy 1. 1. cd /root
     2. 2. touch /etc/selinux/targeted/contexts/files/file_contexts.local
     3. 3. semanage login --list
     4. 4. sepolgen --application /usr/local/bin/bfagent
     5. 5. ./bfagent.sh [http://bfagent.sh/]
     6. 6. semodule -i bfagent.pp
     7. 7. semodule --enable bfagent
     8. 8. semodule -l | grep bfagent
    
 4. D. After this, use systemctl command 1. "systemctl enable bfagent.service"
     2. "systemctl start bfagent.service"
    
 5. E. Verify that bfagent process is running at the desired port 1. ps -aefZ | grep bfagent
     2. netstat -naplZ | grep 5555
    
 6. If the connection to agent fails due to Functional Failure, please follow the steps mentioned below and update.
1. 1. cd /root
     2. 2. grep bfagent /var/log/audit/audit.log > /tmp/bfagent-policy.txt
     3. 3. cat /tmp/bfagent-policy.txt | audit2allow -M bfagent-New-policy
     4. 4. semodule -i bfagent-New-policy.pp
     5. 5. Restart bfagent
     1. "systemctl stop bfagent.service"
         2. "systemctl start bfagent.service"
        
        
     6. 6. Try to connect to console.
**
I started executing step 3.C, but when I got to 3.c.4, sepolgen was not found. I searched for it on the system and it does not exist.


So I jumped down to step 6 and executed the instructions there. One modification I did is that after step 6.2 (grep bfagent /var/log/audit/audit.log > /tmp/bfagent-policy.txt) I edited /tmp/bfagent-policy.txt to only contain the last set of lines (8) for the latest connection attempt. There were many many lines in this file. I then completed step 6.3.


Tests Connection now completes successfully!


I've done this on both VMs where I was not previously able to get a successful Test Connection and they are now both connected.

Regards,
Rick

Your answer


Register or to post your answer.


Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.