[Mageia-dev] Proposal for Mageia: implement bitorrent protocol to allow updates download

[andre999]

Olivier Thauvin a écrit :
> * andre999 (andr55 at laposte.net) wrote:
>> Romain d'Alverny a écrit :
>>>
>>> On Wed, Jan 12, 2011 at 04:29, Michael Scherer<misc at zarb.org>   wrote:
>>>
>>> However, if 1) was to open 5 connections on 5 distinct servers, that
>>> would make more sense, no?
>>
>> Right.
>> Another way to look at the question :
>> If 1000 people are downloading from mirrors allowing a total of 2000
>> connexions, if no-one uses multiple connexions, then 1000 connexions are
>> wasted.  These unused connexions would likely be from faster mirrors.
>
> The upload given by a serveur is split in term of:
> - connection available
> - bandwidth available.
>
> If the mirrors is connected to Gb (which is more likelly the size of
> bandwidth for the whole university...), and you split this Gb/s onto
> 2000 connection you obtain 500kb/s.

Please note that my example was hypothetical, for multiple sites :)
Sorry if that wasn't clear.  I don't know of _any_ single sites able to 
do that.  (Although I don't doubt that they exist.)
(When I started the example, I wrote 20 mirrors, but then erased it 
because it didn't seem relevant to my point.)

> However, I know only few server in the world really able to read Gb/s
> from their disk. The top rate I obtain on dc is 400mbis/s (the memory
> cache help a lot to obtain more).
>
> The number of connection is set high because some people download at
> only 50kb/s and other 2Mbit/s, so the spare bandwidth from the 50kb/s
> can be given to others. Nothing more.

exactly

> According the graph on distrib-coffee:
> http://distrib-coffee.ipsl.jussieu.fr/munin/distrib-coffee/distrib-coffee/index.html
> there is an average of 100 http download at a time, if each connection
> become 5 (500), you'll reach the limit (200 on this server).
> What will be the gain on your side ?

In that context, obviously.  The idea wasn't that most sites would have 
5 connexions - rather to allow up to 5 (depending on the traffic at the 
moment.  I would expect most sites to have one or 2 connexions.  And 
preferably use multiple mirrors, as recommended by aria2c documentation.

> I talked here only about http, but apply the same to ftp in the same
> internet connection, so it mean I have 100mbits/s to share between 400
> connections.
>
> As Michael explain, I voluntary limit the count of connection per IP,
> before I did this, the server was overload 12 hour a day, which mean
> stop serving !

A very good idea.  I would do the same thing in your situation.
Actually, aria2c would work somewhat better if all sites with more 
limited capacity did the same thing.  Because the benefit comes from 
making use of the unused capacity (of the moment) of mostly larger 
mirror sites.  And trying to access sites already at capacity slows 
somewhat the downloading.
Unfortunately not all download software monitors the connexion speed to 
dynamically choose the fastest connexions, as does aria2c.

>> With aria2c, 3 mirrors which support a total of 5 connexions gives me
>> optimal speed.  (The limit being the speed of my computer.)
>
> Can we know the speed of your connection ?

Fast, but I don't know exactly.  It is at the local library, via wi-fi. 
  (I use it for downloading ISOs because my home connexion has limited 
bandwidth.)
It is on the municipal network with telephony and all, and I access it 
in the evening, when essentially only the library is open.  I have seen 
another user downloading at over 900 KB/sec while I was downloading, at 
a somewhat slower speed.  (A while back.)

At the library with up to 5 connexions on 2 or 3 mirrors, it is always 
over 600 KB/sec, often over 700 KB/sec.  The last time I did a 
comparison (1 connexion/1 mirror vs max 5 connexions/2 mirrors) for a 
DVD, the ratio of speed was approx 3,5.  (From summary notes on my 
computer.)

> Maybe your connection is good enough to provide a mirror for Mageia ?
> And then you'll be able to test what you're suggesting.

Unfortunately, my home connexion tops out at 400KB/sec (officially 
512KB/sec), but usually is less than half that.  (Variable-speed slow 
DSL.)  And the upload speed is severely limited by my FAI, as well as a 
limited bandwidth.
Wouldn't mind trying a mirror when better connexions become available.
(Don't know if I would go for something as big as Mageia, though.)

> What is the result with 5 mirrors and 1 connection per mirror ?
At the library, it tops out with 3 mirrors and 5 connexions.  The best 
was over 700KB/sec.  (4 mirrors wasn't any faster.)
(I borrowed a faster computer from someone once, and it averaged well 
over 800 KB/sec on 1 mirror/5 connexions, for download of a DVD.)

> Obviously, any mirror in France (I am in France) is able to fulfill
> my personnal connection... (10mbits/s).

J'aimerais bien ça :)
Where I live (banlieue de Montréal), I couldn't get more than 2Mbits/s 
(officially), if that.  And fairly expensive, as well.  With very 
limited upload speeds.

-- 
André