[Mageia-dev] Will this work for a build system?

[Giuseppe Ghibò]

2010/9/27 herman <herman at aeronetworks.ca>


> On Sun, 2010-09-26 at 18:32 -0700, Frank Griffin wrote:
> > Giuseppe Ghibò wrote:
> > > IMHO one of the building problems was not massive automatic rebuilding
> > > but avoid bottenlecks to the users when building goes wrong.
> > I really like the concept of a distributed build system.
>
> The problem with a distributed system is the enormous increase in
> complexity.  As long as a single big server with about 24 cores can
> compile the lot in one day, then a distributed system is not really
> needed.
>
>
I agree there would be an ENORMOUS increase in complexity, but you are
forgetting that first 24 cores someone cited were not 24 cores, but a dual
exacore machine (I guess a dual Xeon 5650) so 12 core with hyperthreads,
which is not exactly the same as 24 native core. Such core with 12GB of
memory are just "peanuts" in an environment with plenty of developers. Half
of that machine (e.g. with a core i7 985X, or AMD 1055T) is actually a
medium/top PC which you can build in your home.

So you have to distinguish automatic rebuilding of the distro, which
operates on "working" packages from the svn, from new packages built from
the first time. The first task of a massive rebuilding is a automated task
which can be done sequentially (but 1 day is only for the main, then there
is contrib, then you have 2 archs, 32 and 64bits, and backports); the second
task instead has a lot of stop and go. If for instance a build goes wrong
because a packager couldn't test a parallel build in his own development
machine (or because the number of cores is different and you get race
conditions problems), and have to redo the work, but in the meanwhile the
building system is busy, or has other kind of problems (wait for library to
propagate, etc.), then he have to spend a lot of time fighting against the
system and babysitting a package, rather than concentrate on packaging or
developing. And for sure he would fly away (packagers are not of iron with
infinite patience).

Also consider that the phase of LZMA compression of the RPM building won't
operate in parallel.

Bye
Giuseppe.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/mageia-dev/attachments/20100927/d8e0e4e8/attachment.html>