Caching Proxy Solution Squid
currently I'm facing a problem getting the caching proxy solution with squid running.
I know this article Using content caching proxies for Jazz SCM and tried to setup a squid server running on RHEL 5.8
My Jazz server is running on Windows Server 2008 RC2 and is working on a WAS 7.
On the same server I have a reverse proxy running so that the URL of the RTC is https://jazzdev.rsint.net/ccm
After setting up the Squid solution described in the article above we could connect to it using cURL on the same machine. When using the Jazz SCM or the Eclipse or Browser on another windows client we where not able to connect to the Jazz server using the proxy.
We search for any issues and extended the configuration of squid for the parts we found on the www.
Our configuration is this now:
debug_options ALL,1 33,2 28,9 44,3
cache_replacement_policy heap GDSF memory_replacement_policy heap GDSF cache_dir aufs /usr/local/squid/var/cache 6000 256 256 cache_mem 2000 MB cache_store_log none cache_peer jazzdev.rsint.net parent 443 0 no-query originserver name=httpsAccel ssl login=PROXYPASS sslflags=DONT_VERIFY_PEER cache_peer_access httpsAccel allow all coredump_dir /usr/local/squid/var/cache http_access allow all http_port 3128 transparent https_port 3129 transparent cert=/usr/local/squid/etc/server.pem accel key=/usr/local/squid/etc/privkey.pem refresh_pattern . 0 20% 4320 cachemgr_passwd disable all maximum_object_size 1024 MB maximum_object_size_in_memory 16 MB buffered_logs on visible_hostname t-wsg01.rsint.net access_log /usr/local/squid/var/logs/access.log squid # acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12 # RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network # acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT # #http_access allow manager localhost http_access deny !Safe_ports http_access deny CONNECT !SSL_ports |
Now I can connect to the proxy and can download files using Eclipse.
The Jazz scm refuses the connection with a "forbidden" error message.
But my main problem is: Even when loading files using Eclipse the caching proxy is caching .... nothing ....
Anybody any ideas why the cache is not getting filled ?
Greetings,
Simon
5 answers
<code>
cache_replacement_policy heap GDSF
memory_replacement_policy heap GDSF
cache_dir aufs /usr/local/squid/var/cache 10240 256 256
cache_mem 1024 MB
cache_store_log stdio:/usr/local/squid/var/logs/cachelog
cache_peer 192.168.2.106 parent 9743 0 no-query originserver name=httpsAccel ssl login=PROXYPASS sslflags=DONT_VERIFY_PEER
cache_peer 192.168.2.106 parent 9843 0 no-query originserver name=httpsAccel1 ssl login=PROXYPASS sslflags=DONT_VERIFY_PEER
cache_peer 192.168.2.106 parent 9643 0 no-query originserver name=httpsAccel2 ssl login=PROXYPASS sslflags=DONT_VERIFY_PEER
https_port 9743 cert=/usr/local/squid/etc/server.pem accel key=/usr/local/squid/etc/privkey.pem vhost
https_port 9843 cert=/usr/local/squid/etc/server.pem accel key=/usr/local/squid/etc/privkey.pem vhost
https_port 9643 cert=/usr/local/squid/etc/server.pem accel key=/usr/local/squid/etc/privkey.pem vhost
acl ccm1 port 9743
acl ccm2 port 9843
acl ccm3 port 9643
#cache_peer_access httpsAccel allow ccm1
#cache_peer_access httpsAccel deny ccm2 ccm3
#cache_peer_access httpsAccel allow all
#cache_peer_access httpsAccel1 allow ccm2
#cache_peer_access httpsAccel1 deny ccm1 ccm3
#cache_peer_access httpsAccel2 allow ccm3
#cache_peer_access httpsAccel2 deny ccm1 ccm2
coredump_dir /usr/local/squid/var/cache
http_access allow ccm1
#http_access allow ccm2
#http_access allow ccm3
refresh_pattern . 0 20% 40320
cachemgr_passwd disable all
maximum_object_size 5120 MB
maximum_object_size_in_memory 16 MB
buffered_logs on
visible_hostname 192.168.2.32
</code>
my build server is talking to the proxy as if it was the RTC server..
https://192.168.2.32:9743
Comments
This means you take the URL of your proxy at the client and the proxy rewrites this to the address of the jazz server - right?
This means inside of Eclipse you don't need the proxy configuration you just connect directly to the proxy?
correct.. do not use the Eclipse proxy config..
all apps talk directly to the squid proxy as if it IS the RTC server
on my dev system, I changed the DNS entry for the RTC server to use the IP address of the squid server and the clients didn't have to do anything.
the squid url rewrite cannot change the structure of the URL.. Eclipse is sensitive to that..
so
https://server:port/context
must remain
https://realserver/context,
it cannot change to
https://realserver
or
https://realserver/context/context2
Comments
Hi Sam,
thanks for the answer, but as I understood this article from IBM correctly, this should normally not be needed.
You describe a scenario where everybody is using the same proxy, as the proxy DNS name is the old application DNS name. But in our environment we need more than one proxy - one proxy for each location. This is why I would prefer using this solution without a "DNS rename".
Next is - shouldn't it be possible to use the proxy cache with an own address which is beeing used in the "https_proxy" environment variable? Means that I still connect to the application server but the indication weather I use a caching proxy or not should be only through this variable or the network settings inside of Eclipse.
As I understand the article and the how IBM talks about it, this way should be possible and in our case this would be the most preferred way.
Greetings,
Simon
Using a Reverse Accelerator Proxy
You can also configure squid or apache httpd (with the mod_cache module) to run as a accelerator proxy server against your Jazz RTC Team Server. The strategy here is that instead of referencing the Jazz RTC Team Server URI from your client, you would connect instead to a proxy server which forwards all requests onto the Jazz RTC Team Server.Note: For best results in using a reverse accelerator proxy, clients should start new eclipse, command line or visual studio sandboxes. There can be issues in switching between different repository URIs pointing to the same repository from within the clients.
Setting up Squid with HTTP as Accelerator Proxy
there is also talk of the forward proxy, at the beginning, but not much. The accelerator reverse caching proxy works.. we will have a number of them thruout our network. some near the servers in our central site, and one at some of our remote locations. the proxy servers will also talk to each other as siblings (cache peers)
this is a typical multi-proxy design used for web technology for years.
the design is twin purposed: 1. remove as much redundant traffic from the server as possible, and 2. for remote locations, provide as close to local lan speeds as possible.
For builds this reduces the loading time from hours to minutes. slightly slower that the original local lan setup the users had before moving to the central admin structure
the DNS server at the locations (local or remote) will direct the users application (eclipse, Windows, or Browser, or SCM tools) to the caching proxy nearest them, to provide the fastest speeds possible.
Comments
John Camelon
JAZZ DEVELOPER Apr 08 '13, 7:49 a.m.Can you post a sample of your access log? It is saying "TCP_MISS" ?
Simon Eickel
Apr 09 '13, 1:22 a.m.Hi John,
yes, very much TCP_MISS messages:
Greetings,
Simon
John Camelon
JAZZ DEVELOPER Apr 10 '13, 7:50 a.m.I don't see any lines in that output which say IVersionedContentService ... can you grep/show the part of the log where that service is being accessed?
Simon Eickel
Apr 10 '13, 8:22 a.m.Hi John, there are now such entries inside the log...
John Camelon
JAZZ DEVELOPER Apr 10 '13, 1:26 p.m.For clarity : The lines with IVersionedContentService say TCP_MISS ?
Simon Eickel
Apr 11 '13, 1:14 a.m.Hi John, sorry there was a char too much inside my answer ...
I meant "there are no such entries inside the log".
I just see TCP_MISS entries ... nothing else.
At the moment I connect the client using the proxy entries inside network configuration.
In eclipse e.g. I connect directly to the jazz server and have the network configured to use a proxy. Is this wrong?
Running squid -v prints:
rbroot@t-wsg01# sbin/squid -v
Squid Cache: Version 3.0.STABLE26
configure options: '-prefix=/usr/local/squid' '--with-pthreads' '--enable-storeio=ufs,aufs' '--enable-removal-policies=lru,heap' '--enable-ssl' '--with-large-files' 'CFLAGS=-O2 -pipe -m64 -march=core2 -fomit-frame-pointer -s' 'LDFLAGS=-m64 -s -Wl,-O1' 'CXXFLAGS=-O2 -pipe -m64 -march=core2 -fomit-frame-pointer -s'