[Mageia-dev] Will this work for a build system?

[Pascal Terjan]

On Sun, Sep 26, 2010 at 02:38, Michael Scherer <misc at zarb.org> wrote:
> Le samedi 25 septembre 2010 à 20:56 -0700, herman a écrit :
>> On Sat, 2010-09-25 at 10:44 -0700, R James wrote:
>> > Yeah, if I move the physical server about 2 miles. I can plug it into
>> > a better ISP.
>> BTW, I once calculated (test plus extrapolation) how long it would take
>> to rebuild every package in Mandriva on a low end 2 GHz Celeron server
>> that I had available and it came to about 80 days.
>>
>> Usually one won't rebuild every package all in one go, but one has to do
>> that the first time.  So, a build system needs to be a big ass piece of
>> iron, otherwise it will be frustrating to use.
>
> Well, for the moment, Mandriva has ( for i586 ) 5 servers with 4 core at
> 2.8 ghz
> ( fast scsi disk, 2 g of ram ). So by your estimate and using similar
> hardware ( which date back to 3/4 years ago ), it would take
> 80/(5*4*1.5) = 3 days
>
> now, that doesn't take in account several factors, like :
> - others bottleneck ( memory, i/o )
> - overhead of using iurt and chroot regeneration
> - overhead of uploading to the mirror
> - overhead of a non linear repartition ( kde, kernel, openoffice,
> sagemath, etc, take more time ).

Using a nice machine (24 'cpu', actually 2 * 6 cores + HT, 12GB RAM),
using few GB tmpfs for the build, I rebuilt main with iurt for one
arch in less than a day.
This can be optimized (not extracting chroot each time will improve
time for small packages, building several in parallel may improve as
nothing requires 24 cpus 99% of the time and we don't have disk seek
issue anyway with tmpfs) but that's to give an idea.

The upload to the mirror part can be done in parallel so it does not
slow the overall process.