I was having problems using squid to cache file downloads on my IDM. After months of resrearch i found out that IDM was splitting the file into multiple pieces and requesting each of those pieces as different HTTP requests & the Squid cannot cache partial files by default, only whole file can be cached.
I came across Syed Jahanzaib Blog (https://aacable.wordpress.com/2012/08/24/squid-dont-cache-idm-downloads/)
I tried the offset limit rule (range_offset_limit -1) which works sometimes but the major challenge was that it comes with a bandwidth monster who eats up all your bandwidth. Therefore this is not good for you.
According to Syed Jahanzaib Blog
"CAUTION:Use this directive with care, because in some cases it can give you undesirable results including excessive bandwidth usage, e.g: If a user starts downloading any file of any size , lets suppose 600 MB, and he cancel the download in middle or after 10MB, the SQUID will continue to download the whole file in the cache. This will cause a bigger problem , if user downloading a file with
download manager ,let’s say in 6 segments , squid will start 6 download threads for the same file each one from its beginning. & it will consume 6 times bandwidth than really needed. so it can really become bottleneck in some cases, specially if you have small bandwidth pipe.So be careful using this option. Most people prefer the opposite, to make Squid abort downloads immediately.because this kind of continued downloads can easily chew up a considerable amount of bandwidth for a very log hit ratio.Regard’s
Syed Jahanzaib"
Yesterday i had the eureka moment and was like what what if i restrict IDM to having only one connection at a time instead of downloading using multiple connections at a time.
So i tried this below and voila it works like magic ( I can eat my cake and also have it at the same time)
Using webmin
Go to Access Control
ACL Name i used IDM, Maximum Concurrent Request 1
And thats it
Using CLI add
acl IDM maxconn 1
Result
First download before web cache ( Notice The Single Connetion on the IDM status bar)
Second download this time from web cache (Notice increase in speed and multiple connection since its not going to the internet)
Thank you
Azeez Badmus
Its free .I mean the squid
ReplyDeleteYes its free install on ubuntu of centos
DeleteInteresting... enlighten us more please
ReplyDelete