[Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download

[Michael Scherer]

Le mardi 11 janvier 2011 à 21:45 -0500, andre999 a écrit :
> Michael Scherer a écrit :
> >
> > Le mardi 11 janvier 2011 à 20:03 +0100, Marcello Anni a écrit :
> >> hi all,
> >> i have one question (maybe it can be a proposal): is it possible to implement
> >> the torrent protocol to faster download the updates of the distro? it could be
> >> an interesting features for the coming Mageia releases
> >
> > I think the issue of faster download could be simply taken care by
> > having more mirror, or faster one.
> >
> > I had under the impression that some ISP throttle down bittorrent, and
> > that it may not be very nat and firewall friendly..
> 
> Some suggestions for faster downloads without bittorrent.
> 1) use aria2c (or a similar application), which uses multiple 
> connections, defaulting to 5, and allows multiple mirrors.
> By default it starts by allocating space for the file to be downloaded, 
> which allows non-sequential downloading of the file, facilitating faster 
> downloading from multiple sites.
> 
> 2) use mirrors which allow multiple connexions.
> (Of course, with download software that takes advantage of this.)
> 
> 3) use multiple mirrors.
> (Again, according to download software.)

Theses 3 suggestions basically put X time the load of the mirror for
each client. ( or on more mirror, for that matters ).

And that's quite bad from the point of view of a mirror manager.

For example, distrib-coffee could blacklist you if you do this, if you
are not alone on your network connexion. And when we deployed this
measure to protect the server, the limit was 2 connexion per address,
since this was taking too much ressources on the old server ( each http
request taking 1 process and so memory ). Hopefully, the hardware was
upgraded but not everybody can afford 32g of ram and 8*2 ghz CPU. 

[root at distrib-coffee ~]# grep -B 6 -A 3
MaxConnPerIP  /etc/httpd/conf.d/distrib-coffee.conf

<Directory /var/ftp/pub>
   order allow,deny
   Allow from all
   Options +Indexes +MultiViews +SymLinksIfOwnerMatch
   <IfModule mod_limitipconn.c>
       MaxConnPerIP 5
       ErrorDocument 503 "5 connections at the same time only allowed."
   </IfModule>
</Directory>

So I think pissing off mirror maintainers is likely the wrong way of
solving the problem ( who was not properly explained nor looked at
besides "it should be faster" ).

> 4) use ftp instead of http

Based on ?

If this is based on using d-c, again, that's our custom QOS rules. If
this is because of throttling on your provider, not everybody have the
same provider, and so the same throttling.
The only difference between http and ftp is that ftp server will likely
scale better server side. But that should have no impact on file serving
and download ( both nowadays use fast syscall, such as sendfile, etc ).

> 5) use closer mirrors.  (less delay in handshaking, etc.)

I think tcp handshake is not much a problem, given the fact it happen
once per rpm, compared to the number of tcp packet for the rest of the
download. Use wireshark to see. 


> In my case, using aria2c with 2 mirrors and the default 5 connexions is 
> at least 3 times as fast as a single connexion (to my closest mirror).
> And a much greater improvement over other download options I've tried.
> 
> I also have configured rpmdrake to use aria2c -- it seems to give me 
> faster and more reliable updating, but I don't have any figures.
> 
> aria2c is a console app, but it works well enough for me that I haven't 
> (yet) bothered to install the available GUI frontend.

That's because it worked in your case that it would work for every
possible case, especially without giving a proper analysis of the issue
on your side.

-- 
Michael Scherer