It's all about the answers!

Ask a question

Caching Proxy Solution Squid


Simon Eickel (1.1k75457) | asked Apr 05 '13, 9:56 a.m.
retagged May 09 '13, 10:39 a.m. by Arne Bister (2.6k12832)
Hello together,

currently I'm facing a problem getting the caching proxy solution with squid running.
I know this article Using content caching proxies for Jazz SCM and tried to setup a squid server running on RHEL 5.8
My Jazz server is running on Windows Server 2008 RC2 and is working on a WAS 7.
On the same server I have a reverse proxy running so that the URL of the RTC is https://jazzdev.rsint.net/ccm

After setting up the Squid solution described in the article above we could connect to it using cURL on the same machine. When using the Jazz SCM or the Eclipse or Browser on another windows client we where not able to connect to the Jazz server using the proxy.

We search for any issues and extended the configuration of squid for the parts we found on the www.
Our configuration is this now:
debug_options ALL,1 33,2 28,9 44,3
cache_replacement_policy heap GDSF
memory_replacement_policy heap GDSF
cache_dir aufs /usr/local/squid/var/cache 6000 256 256
cache_mem 2000 MB
cache_store_log none
cache_peer jazzdev.rsint.net parent 443 0 no-query originserver name=httpsAccel ssl login=PROXYPASS sslflags=DONT_VERIFY_PEER
cache_peer_access httpsAccel allow all
coredump_dir /usr/local/squid/var/cache
http_access allow all
http_port 3128 transparent
https_port 3129 transparent cert=/usr/local/squid/etc/server.pem accel key=/usr/local/squid/etc/privkey.pem
refresh_pattern . 0 20% 4320
cachemgr_passwd disable all
maximum_object_size 1024 MB
maximum_object_size_in_memory 16 MB
buffered_logs on
visible_hostname t-wsg01.rsint.net
access_log /usr/local/squid/var/logs/access.log squid
#
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
#
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
#
#http_access allow manager localhost
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports

Now I can connect to the proxy and can download files using Eclipse.
The Jazz scm refuses the connection with a "forbidden" error message.

But my main problem is: Even when loading files using Eclipse the caching proxy is caching .... nothing ....

Anybody any ideas why the cache is not getting filled ?

Greetings,
Simon

Comments
John Camelon commented Apr 08 '13, 7:49 a.m.
JAZZ DEVELOPER

 Can you post a sample of your access log?  It is saying "TCP_MISS" ?


Simon Eickel commented Apr 09 '13, 1:22 a.m.

Hi John,

yes, very much TCP_MISS messages:


Greetings,
Simon


John Camelon commented Apr 10 '13, 7:50 a.m.
JAZZ DEVELOPER

I don't see any lines in that output which say IVersionedContentService ... can you grep/show the part of the log where that service is being accessed?   


Simon Eickel commented Apr 10 '13, 8:22 a.m.

Hi John, there are now such entries inside the log...


John Camelon commented Apr 10 '13, 1:26 p.m.
JAZZ DEVELOPER

For clarity : The lines with IVersionedContentService say TCP_MISS ?  


Simon Eickel commented Apr 11 '13, 1:14 a.m.

Hi John, sorry there was a char too much inside my answer ...
I meant "there are no such entries inside the log".

I just see TCP_MISS entries ... nothing else.

At the moment I connect the client using the proxy entries inside network configuration.
In eclipse e.g. I connect directly to the jazz server and have the network configured to use a proxy. Is this wrong?

Running squid -v prints:
rbroot@t-wsg01# sbin/squid -v
Squid Cache: Version 3.0.STABLE26
configure options:  '-prefix=/usr/local/squid' '--with-pthreads' '--enable-storeio=ufs,aufs' '--enable-removal-policies=lru,heap' '--enable-ssl' '--with-large-files' 'CFLAGS=-O2 -pipe -m64 -march=core2 -fomit-frame-pointer -s' 'LDFLAGS=-m64 -s -Wl,-O1' 'CXXFLAGS=-O2 -pipe -m64 -march=core2 -fomit-frame-pointer -s'

showing 5 of 6 show 1 more comments

5 answers



permanent link
sam detweiler (12.5k6195201) | answered Apr 10 '13, 11:54 p.m.
hm.. here is my squid.conf
<code>
cache_replacement_policy heap GDSF
memory_replacement_policy heap GDSF
cache_dir aufs /usr/local/squid/var/cache 10240 256 256
cache_mem 1024 MB
cache_store_log stdio:/usr/local/squid/var/logs/cachelog
cache_peer 192.168.2.106 parent 9743 0 no-query originserver name=httpsAccel ssl login=PROXYPASS sslflags=DONT_VERIFY_PEER
cache_peer 192.168.2.106 parent 9843 0 no-query originserver name=httpsAccel1 ssl login=PROXYPASS sslflags=DONT_VERIFY_PEER
cache_peer 192.168.2.106 parent 9643 0 no-query originserver name=httpsAccel2 ssl login=PROXYPASS sslflags=DONT_VERIFY_PEER
https_port 9743 cert=/usr/local/squid/etc/server.pem accel key=/usr/local/squid/etc/privkey.pem vhost
https_port 9843 cert=/usr/local/squid/etc/server.pem accel key=/usr/local/squid/etc/privkey.pem vhost
https_port 9643 cert=/usr/local/squid/etc/server.pem accel key=/usr/local/squid/etc/privkey.pem vhost
acl ccm1 port 9743
acl ccm2 port 9843
acl ccm3 port 9643
#cache_peer_access httpsAccel allow ccm1
#cache_peer_access httpsAccel deny  ccm2 ccm3
#cache_peer_access httpsAccel allow all
#cache_peer_access httpsAccel1 allow ccm2
#cache_peer_access httpsAccel1 deny  ccm1 ccm3
#cache_peer_access httpsAccel2 allow ccm3
#cache_peer_access httpsAccel2 deny  ccm1 ccm2

coredump_dir /usr/local/squid/var/cache
http_access allow ccm1
#http_access allow ccm2
#http_access allow ccm3

refresh_pattern .              0       20%     40320
cachemgr_passwd disable all
maximum_object_size 5120 MB
maximum_object_size_in_memory 16 MB
buffered_logs on
visible_hostname 192.168.2.32

</code>



permanent link
sam detweiler (12.5k6195201) | answered Apr 10 '13, 10:03 a.m.
can u comment out your ACLs for a test?

I don't have any on mine, and can connect just fine from my build machines

Comments
Simon Eickel commented Apr 10 '13, 12:49 p.m.

Hi Sam, even after outcommenting those statements there are only "TCP_MISS" inside the logfile.


permanent link
sam detweiler (12.5k6195201) | answered Apr 11 '13, 12:18 a.m.
edited May 09 '13, 11:38 a.m.
Oh, there was also a bug introduced in squid v 3. I had to fall back to 2.8 I think
my build server is talking to the proxy as if it was the RTC server..

https://192.168.2.32:9743

Comments
Simon Eickel commented Apr 11 '13, 1:10 a.m.

This means you take the URL of your proxy at the client and the proxy rewrites this to the address of the jazz server - right?

This means inside of Eclipse you don't need the proxy configuration you just connect directly to the proxy?


sam detweiler commented May 09 '13, 11:37 a.m.

correct.. do not use the Eclipse proxy config..

all apps talk directly to the squid proxy as if it IS the RTC server


permanent link
sam detweiler (12.5k6195201) | answered Apr 11 '13, 7:49 a.m.
correct.. the proxy IS the RTC server from all clients.  we do not use the formal 'passthru proxy' configuration.

on my dev system, I changed the DNS entry for the RTC server to use the IP address of the squid server and the clients didn't have to do anything.

the squid url rewrite cannot change the structure of the URL.. Eclipse is sensitive to that..
so
https://server:port/context

must remain
https://realserver/context,

it cannot change to
https://realserver
or
https://realserver/context/context2

Comments
Simon Eickel commented Apr 15 '13, 2:27 a.m.

Hi Sam,

thanks for the answer, but as I understood this article from IBM correctly, this should normally not be needed.
You describe a scenario where everybody is using the same proxy, as the proxy DNS name is the old application DNS name. But in our environment we need more than one proxy - one proxy for each location. This is why I would prefer using this solution without a "DNS rename".

Next is - shouldn't it be possible to use the proxy cache with an own address which is beeing used in the "https_proxy" environment variable? Means that I still connect to the application server but the indication weather I use a caching proxy or not should be only through this variable or the network settings inside of Eclipse.

As I understand the article and the how IBM talks about it, this way should be possible and in our case this would be the most preferred way.

Greetings,
Simon


permanent link
sam detweiler (12.5k6195201) | answered Apr 15 '13, 11:17 p.m.
edited Apr 15 '13, 11:25 p.m.
the article says

Using a Reverse Accelerator Proxy

 You can also configure squid or apache httpd (with the mod_cache module) to run as a accelerator proxy server against your Jazz RTC Team Server.  The strategy here is that instead of referencing the Jazz RTC Team Server URI from your client, you would connect instead to a proxy server which forwards all requests onto the Jazz RTC Team Server. 

Note: For best results in using a reverse accelerator proxy, clients should start new eclipse, command line or visual studio sandboxes.  There can be issues in switching between different repository URIs pointing to the same repository from within the clients. 

Setting up Squid with HTTP as Accelerator Proxy



there is also talk of the forward proxy, at the beginning, but not much. The accelerator reverse caching proxy works.. we will have a number of them thruout our network. some near the servers in our central site, and one at some of our remote locations.  the proxy servers will also talk to each other as siblings (cache peers) 

this is a typical multi-proxy design used for web technology for years.

the design is twin purposed: 1. remove as much redundant traffic from the server as possible, and 2. for remote locations, provide as close to local lan speeds as possible.

For builds this reduces the loading time from hours to minutes. slightly slower that the original local lan setup the users had before moving to the central admin structure

the DNS server at the locations (local or remote) will direct the users application (eclipse, Windows, or Browser, or SCM tools) to the caching proxy nearest them, to provide the fastest speeds possible.

Your answer


Register or to post your answer.


Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.