[Mageia-dev] Mirror tree structure

Buchan Milne bgmilne at multilinks.com
Thu Oct 21 18:16:06 CEST 2010


On Thursday, 21 October 2010 06:37:37 Olivier Thauvin wrote:
> * J.A. Magallón (jamagallon at ono.com) wrote:
> > On Wed, 20 Oct 2010 18:34:24 +0200, Olivier Thauvin 
<nanardon at nanardon.zarb.org> wrote:
> > > Hi,
> > 
> > ...
> > 
> > > Now come the question: "what is a valid mirror ?", eg, what a mirror
> > > should have as file to be valid ?
> > 
> > Some ideas (probably silly ;)).
> > 
> > Why don't you split mirrors in 2 categories:
> > - Software (RPM) repo mirrors, available for network install or urpmi.
> > 
> >   (distrib+people+software).
> > 
> > - ISO mirrors. Avaliable for torrenting or ftp. MDV2010.1 release isos
> > are
> > 
> >   about 15Gb. With 2 releases per year, thats 30Gb per year, 200 Gb on 6
> >   years. Some places can just mirror this...you lower the 700Gb to 500,
> >   not bad.
> > 
> > so mirroring can be done by version (or was this what you wanted to
> > avoid?).
> 
> Yes, I tried to avoid it because I perfectly know mirrors will take care
> to mirror new version every 6 months (ibiblio example again, nobody had
> a look for several years, at least since 2008).

This will result in fewer mirrors (those that cannot or choose not to dedicate 
~ 1TiB). I would rather consider other solutions to this problem. On my own 
mirror (which has a total of 820GB available at present), I use exclude 
options to rsync in my mirror scripts (but, I have to change these after each 
release).

Maybe instead we could provide exclude-list files on the mirror, so mirror 
maintainers can use --exclude-from=/path/to/mageia/mirror_cfg/old_releases.cfg

Also, in order to try and expedite getting what most users need out to mirrors 
first, I would prefer either:
-development tree separate from releases
-development release should be selected last by mirroring tools (e.g. follow 
alphabetically all stable releases if not separate)


> 
> > BTW
> > <nitpickin>
> > - could you kill the final 's' on all the names ?
> > 
> >   distribs -> distrib
> 
> Done

Maybe 'releases' or 'release' instead?

> 
> >   iso (you didnt named it 'isos')
> >   peoples -> people
> 
> Done
> 
> >   software (not softwares)
> 
> Is already 'software'
> 
> > - could you not mix case in names (SRPMS <-> x86_64), just srpms...
> 
> This part depend on BS and how this team working on it will organise the
> distrib. I'll suggest them.
> 
> > - could be the arch names more uniform ? in my personal scripts/setups
> > 
> >   I use x86-32 and x86-64.

Is x86-32 a valid architecture for rpm etc.? While uniformity might be nice, 
unfortunately vendors don't necessarily choose uniform architecure names, and 
it might be better to match the repo structure to values that can be 
determined directly (and not heuristcally) . 

I've also never seen 'uname -m' report x86-32 or x86_32.

> >   Moreover, perhaps in a not so near future some
> >   adventurous soul builds Mageia on ARM or Sparc, so why not sort things
> >   like
> >   
> >     distrib/cauldron/srpm
> >     distrib/cauldron/x86/32/iso
> >     
> >                            /rpm
> >                         
> >                         /64
> >                     
> >                     /arm/32
> >                     
> >                         /64

I don't know if memory address space is a useful differentiator here, as 
features differ substantially in different ARM cores of the same family or 
architecture version. E.g., Fedora has an 'armv5tel' architecture, N900 ships 
.deb's with 'armel' as the architecture. See 
http://en.wikipedia.org/wiki/ARM_architecture

> >                     
> >                     /sparc/32
> >                     
> >                           /64

AFAIK, the valid architecture names for sparc are sparc,sparc64,sparcv9.

> 
> This depend again on BS part.
> 
> >   The drawback is that ISOs are in the same tree so no separate mirroring
> >   is possible, but I think you didn't want this anyways.
> 
> I'll take care to this.
> 
> > All this renaming in case it is not some kind of standard for tree layout
> > or the like...
> > </nitpickin>
> > 
> > Just my 2€cent, you will judge...

The readme file mentions that mirrors must sync every 2 hours. For tier1 
mirrors, I would agree, however you may need to stagger sync intervals to 
lower tiers (e.g. 4 hours at tier2 and 12 hours at tier3).

It may be worthwhile to try and estimate the actual rsync bandwidth that must 
be achieved to make a specific tier.

(I am not able to sync every 2 hours, probably not 4, maybe 12, though my 
mirror currently syncs daily - for Mandriva I could sync 'official' more 
regularly, but not 'devel').


Regards,
Buchan


More information about the Mageia-dev mailing list