Thursday 17 September 2015

How to allow squid proxy to cache IDM downloads


I was having problems using squid to cache file downloads on my IDM.  After months of resrearch i found out that IDM was splitting the file into multiple pieces and requesting each of those pieces as different HTTP requests & the Squid cannot cache partial files by default, only whole file can be cached.


I came across Syed Jahanzaib Blog (https://aacable.wordpress.com/2012/08/24/squid-dont-cache-idm-downloads/)

I tried the offset limit rule (range_offset_limit -1) which works sometimes but the major challenge was that it comes with a bandwidth monster who eats up all your bandwidth. Therefore this is not good for you. 

According to  Syed Jahanzaib Blog

"CAUTION:Use this directive with care, because in some cases it can give you undesirable results including excessive bandwidth usage, e.g: If a user starts downloading any file of any size , lets suppose 600 MB, and he cancel the download in middle or after 10MB, the  SQUID will continue to download the whole file in the cache. This will cause a bigger problem , if user downloading a file with
download manager ,let’s say in 6 segments , squid will start 6 download threads for the same file each one from its beginning. & it will  consume 6 times bandwidth than really needed. so it can really become bottleneck in some cases, specially if you have small bandwidth pipe.So be careful using this option.  Most people prefer the opposite, to make Squid abort downloads immediately.because this kind of continued downloads can easily chew up a considerable amount of bandwidth for a very log hit ratio.
Regard’s
Syed Jahanzaib"

Yesterday i had the eureka moment and was like what what if i restrict IDM to having only one connection at a time instead of downloading using multiple connections at a time.

So i tried this below and voila it works like magic ( I can eat my cake and also have it at  the same time)

Using webmin

Go to Access Control


 Scroll down selecte maximum connections form the drop down and click Create new ACL
ACL Name i used IDM, Maximum Concurrent Request 1
And thats it

Using CLI add 
acl IDM maxconn 1


Result

First download before web cache ( Notice The Single Connetion on the IDM status bar)

Second download this time from web cache (Notice increase in speed and multiple connection since its not going to the internet)

Thank you

Azeez Badmus

3 comments: