[closed] lscm (scm) command line is very slow (solved)

Brett Waldo (110●15●21) | asked Sep 30 '14, 10:25 a.m.
closed Jan 19 '15, 3:15 a.m. by Krzysztof Kaźmierczyk (7.5k●4●80●103)

The sounds like a duplicate from my searches, but it is so slow I feel like something has to be configured wrong. Every command takes at least 10 seconds, many take 30-40 seconds.

I'm using RTC5.0.1, which has supposed improved command line. As is on my system, it would be faster to print out the oracle database each time and just look at the table data. We are new to RTC so our server has very little data.

System Spec:

Client: RHEL6 32 bit, 16 cores, 14GB Ram, java -version = java version "1.7.0_25"

Server RHEL6, 64 bit, 24 cores, 50 GB Ram

Database Server RHEL6, 64 bit, 24 cores, 50GB Ram, Oracle backend

1GB network, same switch

Here is short session with the output of how long each command took, note working with a very small component:

time lscm login -r https://[host]:9443/ccm -n local --cache

real 0m38.958s

time lscm list workspaces -r local

[returns all 5 records]

real 0m15.882s

time lscm list streams -r local

[returns all 4 records]

real 0m15.993s

time lscm list components -r local

[returns first 10 records]

real 0m16.073s

time lscm load -r local -d checkout 1001 1006

real 4m10.082s

du -hs checkout/

179M checkout/

time lscm status

real 0m53.962s

time lscm logout -r local

real 0m15.715s

Inspection of the log file shows the following error message printed 3 times / second. Even while no commands are executing:

<logger>com.ibm.team.filesystem.client</logger>

<level>SEVERE</level>

<class>com.ibm.team.filesystem.client.internal.LoggingHelper</class>

<message>com.ibm.team.filesystem.client.FileSystemException: Error loading map info</message>

</record>

Comments

Remy Suen commented Sep 30 '14, 11:18 a.m.

If you use regular 'scm' instead of 'lscm' is it just as slow?

Brett Waldo commented Sep 30 '14, 12:16 p.m.

My answer must be really short. Only allowed about 300 characters.

Short answer is no. I would still argue its too slow though. 
login
0m7.132s
list workspaces
0m5.950s
list streams
0m5.989s
list components
0m5.961s
load
4m2.269s
status
0m18.998s
logout
0m3.025u

Remy Suen commented Sep 30 '14, 2:21 p.m.

Do your coworkers also have the same problem with 'lscm'?

Brett Waldo commented Sep 30 '14, 2:49 p.m.

I'm really the only person using it at this point. Our organization is pushing migrating to RTC and I've kind of been the guinea pig. But in talking with other people on other servers, it seems to be pretty standard that scm is really slow. I'm hoping to try GIT integration at some point so we have a command line available. Not sure what we lose in doing that however.

Donald Nong commented Oct 01 '14, 1:13 a.m.

The fact that even the login process took a such long time leads me to believe that the issue is network related. Looking at those numbers, I have a feeling that you hit a timeout of 15 seconds. It should be easy to capture network trace using tcpdump and analyze it with Wireshark in order to figure it out.

Brett Waldo commented Oct 01 '14, 6:59 a.m.

So this morning when I came in to try the tcpdump, but speeds were dramatically increased. I presume this was due to the nightly reboot of the client machine. Time to list workspaces/components/streams was < 1 sec. Logout was instant, First login was 5-10 seconds, subsequent ones were < 5 seconds.

The load command is still pretty slow. The network is GB and downloading 200MB should take much less than 4 minutes. After the load, I couldn't do a status command with the error message about another process has locked the file and could not initialize. Running a ps -aef | grep scm showed two instances of

/tcs/cots/rtc/jazz/scmtools/eclipse/scm --config /home/bwaldo/.jazz-scm daemon start --connection-timeout 120000 --inactive-timeout 7200000

I've seen the number of these grow to 4 or 5. I killed this process and then lscm got much slower.

So the question is, what is this process, should there be multiple of them? Why does killing one cause subsequent ones to be so slow?

Evan Hughes commented Nov 18 '14, 11:56 a.m.

JAZZ DEVELOPER

Those processes are intended to speed up the operation of 'lscm', making it faster to run than 'scm'. Each one runs as a daemon and should only be associated with one sandbox at a time. The next time you see slowness or multiple 'scm' instances, could you get a listing of the daemons along with their associated sandboxes?

You do that by running 'scm list daemons', e.g:

$ scm list daemons
Port  Description
58099 org.eclipse.ui.ide.workbench
  /home/echughes/work/jazz/workspaces/rtc-current
44203 org.eclipse.ui.ide.workbench
  /home/echughes/work/jazz/workspaces/code-review

That shows two daemons running in Eclipse, each with a single sandbox.

In order of 'lscm' to find daemons, it queries directories in ~/.jazz-scm. Does your home directory exist, and is it on a filesystem that provides consisting locking behaviour? Is your sandbox on a filesystem that supports consistent locking?

showing 5 of 7 show 2 more comments

The question has been closed for the following reason: "Duplicate Question" by krzysztofkazmierczyk Jan 19 '15, 3:15 a.m.

3 answers

Most liked answers ↑|Newest answers|Oldest answers

permanent link

Arun K Sriramaiah (3.2k●2●36●77) | answered Oct 01 '14, 7:56 a.m.

Hi Brett,

Could you please try with the below details.

LSCM_USE_JAVA_FEC EV and that seems to have speed things up considerably.
Set EV to any value i.e
set LSCM_USE_JAVA_FEC=xyz.
This can be done via the windows environment variables or at the command line before running the lscm commands.

Here is an example of a batch file that you can use to start run the initial login using lscm

set LSCM_USE_JAVA_FEC=xyz
@echo %LSCM_USE_JAVA_FEC% Fec
set PATH=C:\OPT\eclipse\RTC\RTC-Client-Win-4.0.5\jazz\scmtools\eclipse;%PATH%
@echo Start Bat file: %time%
lscm login -c -n r -r https://server:9444/ccm -u user -P password

Regards,

Arun.

Comments

Brett Waldo commented Oct 01 '14, 12:07 p.m.

That throws an error,
export LSCM_USE_JAVA_FEC=xyz
lscm login -r [host]:9443/ccm -n local --cache
Unrecognized option: -Xquickstart

permanent link

Brett Waldo (110●15●21) | answered Oct 01 '14, 12:58 p.m.

This question was duped. Arne posted on the other topic. Pasting back here for completeness

Arne Bister (2.1k●3●15) | answered 1 hour ago
JAZZ DEVELOPER

Brett,

please cp. https://jazz.net/wiki/bin/view/Deployment/RTCClientPerformance#Command_line_interfaces_CLIs to references how to enable Metronom and additional logging for command line to pinpoint exactly what is causing the performance issue you are observing.

- Arne

Comments

Brett Waldo commented Oct 01 '14, 1:15 p.m.

Metronom doesn't appear to work on lscm (invalid option). Running on scm, produced:

Start VM: -Xmx512m -Xshareclasses:nonfatal -Xquickstart /OutOfMemoryError,request=exclusive+prepwalk -os linux -ws gtk -arch x86 -showsplash -data @noDefault -debug .debugoptions list workspaces -r local -vm /usr/java/jdk1.6.0_26/bin/../jre/lib/i386/client/libjvm.so -vmargs -Xmx512m -Xshareclasses:nonfatal -Xquickstart -Dosgi.requiredJavaVersion=1.6 -Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError,request=exclusive+prepwalk -Djava.class.path=/cots/rtc/jazz/scmtools/eclipse/plugins/org.eclipse.equinox.launcher_1.1.1.R36x_v20101122_1400.jar

/org.eclipse.osgi_3.6.50.R36x_v20120315-1500.jar Splash location: null Debug options: file:/cots/rtc/jazz/scmtools/eclipse/.debugoptions loaded Time to load bundles: 4 Starting application: 558 (1000) (1033) Service Trip Statistics: Interface/method Calls Time IRepositoryRemoteService 4 212ms fetchOrRefreshItems 4 212ms IScmRestService......... 1 161ms getWorkspaces 1 161ms -- Total time in service calls: 373ms

real 0m5.602s

Anthony Kesterton commented Oct 01 '14, 2:59 p.m.

JAZZ DEVELOPER

Just looking at the output above - are you using a Java 6 VM from the installation of RTC, or one from elsewhere? Any chance you can point to the JVM supplied with the RTC client (if you installed this part of the client) - or try a Java 7 VM.

Brett Waldo commented Oct 17 '14, 9:33 a.m.

WHY ARE COMMENTS RESTRICTED TO LIKE 300 CHARACTERS?!?!? Is this a twitter forum??

Sorry, took me while to get back to this. We are looking to just switch to git since that command line generally works. but to answer your question, I don't know how to tie to the RTC VM, as it isn't installed anywhere I can tell. I tried again with Java 7, it actually seemed slightly better.

-vm /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25/bin/../jre/lib/i386/client/libjvm.so
-vmargs
-Xmx512m
-Xshareclasses:nonfatal
-Xquickstart
-Dosgi.requiredJavaVersion=1.6
-Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError,request=exclusive+prepwalk

Service Trip Statistics:
Interface/method           Calls  Time
  IRepositoryRemoteService     3  20ms
    fetchOrRefreshItems        3  20ms
  IScmRestService.........     1  56ms
    getWorkspaces              1  56ms
-- Total time in service calls: 76ms

The problem seems to largely be lscm, which doesnt support metronome.  
lscm list workspaces takes 10 seconds minimum (at times closer to a minute)
scm list workspaces takes 4 seconds

permanent link

Brett Waldo (110●15●21) | answered Jan 16 '15, 10:15 a.m.

Eventually tracked this down to NFS. I had read that RTC doesn't really support NFS, generally when the NFS doesn't support good locking. I found a workaround was to use

lscm --config /tmp/rtcconfig-$user

This put the .jazz5 directory on a local filesystem and it is now responsive.

Comments

Pavel Dubovitsky commented Jan 16 '15, 10:33 a.m.

Another option could be to mount NFS filesystem with -llock (Solaris/AIX/HP-UX) or -nolock (Linux) option. It would force to use local locks for the NFS filesystem. It is safe when the same build workspace is used from one UNIX/Linux host at a time.

Brett Waldo commented Jan 16 '15, 10:40 a.m.

RTC would not run at all without the -nolock option on the mount command. Added the option allowed it to run, just really, really slow. Using a local filesystem was a significant improvement (or an NFS filesystem that supported locks)

It's all about the answers!

Related questions

[closed] lscm (scm) command line is very slow (solved)

The question has been closed for the following reason: "Duplicate Question" by krzysztofkazmierczyk Jan 19 '15, 3:15 a.m.

3 answers