[Mageia-dev] Mirror layout, round two

[andre999]

Michael scherer a écrit :
> On Sat, Nov 27, 2010 at 08:00:17PM +0200, Thomas Backlund wrote:
>    
>> Michael scherer skrev 27.11.2010 10:43:
>>      
>>> On Fri, Nov 26, 2010 at 10:29:14PM +0200, Thomas Backlund wrote:
>>>        
>> [...]
>>      
>>>> Then we come to the "problematic" part:
>>>>          
>>> This part look really too complex to me.
>>>
>>>        
>>>> ------
>>>> /x86_64/
>>>>         /media/
>>>>               /codecs/ (disabled by default)
>>>>          
>>> so, ogg, webm, being codec, should go there or not ?
>>> What about patents problem about something else than codec ?
>>> ( freetype, image such as gif, DRM stuff )
>>>
>>>        
>> Actually this is the "maybe_legal_greyzone" repo,
>> but since flagging it as "codecs" would really make people
>> react, I named it so for now...
>>      
> Sorry to be so direct, but that's doesn't answer the question :/
>
>    
>>>>               /core/ (old main+contrib)
>>>>                    /backports/ (disabled by default)
>>>>                    /backports_testing/ (disabled by default)
>>>>                    /release/
>>>>                    /testing/ (disabled by default)
>>>>          
> Shall I suggest to name this one "updates_testing", for consistency ?
> ( consistency with backport_testing, and because this explain what goes in
> more clearly. This also look simpler ).
>
>    
>>>>                    /updates/
>>>>               /extra/ (unmaintained, disabled by default)
>>>>          
>>> If used by people, then why no one step to maintain anything ?
>>>        
>> Yeah, thats the problem.
>>      
> If this is the problem, how does it help to have people to maintain
> the application ?
>
> So far, the only way that really work is
> "someone take care or we shoot the do^W rpm".
> So maybe we could just be more active with cleaning ?
>
>    
>> And reality shows we have a lot of packages assigned to nomaintainer@ ...
>>
>>      
>>>>               /firmware/ (disabled by default)
>>>>          
>>> Why separate firmware from non_free ? What does it bring ?
>>> Since both of them are disabled by default, they can be simply merged.
>>>
>>>        
>> Well, this suggestion is partly based on the fact that we have users
>> that want a firmware free install, wich this would satisfy...
>>      
> I do not think this warrant a full media, maybe just a way to filter package.
>
> Using a media seems overkill to me, since this bring complexity in dialog box, from
> easyurpmi to rpmdrake and installer, and since it bring complexity on mirror, on BS
> and on our policy.
>
> Maybe we could find a way to tag them "firmware", like a rpmgroup.
>    
Exactly.
The filtering will be more involved, but rpmdrake/urpmi need overhauling 
anyway.
We just need to add an rpm group "system/firmware", and move firmware 
packages from "system/kernel+hardware".
> The benefit is the complexity will only be on rpmdrake side, not on mirroring and BS
> side.
>
> More ever, this would much more flexible ( ie, see the games option I propose later ).
>
>    
>> But yes, if we ignore those suggestions, we split the firmwares in
>> GPL ->  /core/ and the rest to /non-free/
>>
>>      
>>>>               /games/ (disabled by default)
>>>>          
>>> That's a simplification that make no sense.
>>> Not all games are big, not all big packages are games ( tetex, openoffice ).
>>>        
>> It's not only a size question, its also a nice option for companies
>> to not mirror games ("employees should work, not play...")
>>      
> Such companies likely already have admins to prevent users from installing games.
> Maybe we could add feature in rpmdrake for that ( like "do not show package
> that match such conditions : group =~ games/, maintainer =~ nomaintainer@, requires =~ python ).
>    
Excellent idea.  I like the nomaintainer option :)
You can set some urpmi options by editing a config file.  There has 
already been suggestions to make this directly accessible in rpmdrake.  
Once done, we just need to add more options.

> The problem of private internal companies mirrors is really not our concern.
> And their software policy, even if they may decide to apply it on a public mirror,
> should not leak on our side.
>    

Right.  No point in confusing issues.
>> And we have some contributors that already have stated that they
>> plan to add all possible games so it will grow.
>> and we all know games are the fastest growing /space demanding...
>>      
> Well, so either that will cause a problem on our side, in which case this will
> just be unhelpful on our primary mirrors, or it will only cause issues on some mirrors,
> and in this case, there is lots of other thing that can take space that we do not
> take in account :
> - debug
> - source code ( except that a GPL requirement )
> - adding another arch ( like arm/mips )
> - adding more iso ( something that is asked each time, like 64 bits one, etc )
>
> So if we decide "mirrors will not handle the load, so we need to split games", then we
> should also say "mirrors will not handle the load, so we need to do less iso/offer to not
> mirror debug/offer to not mirror some architecture", and we end with a non consistent
> network of mirror, with lots of complexity on our side to handle the possible choice
> made by mirrors. I am not sure that users
> will truly benefit from this. And I am sure that we will not benefit from the complexity.
>
> If the space is a issue ( and I think that's one of the main one ), then we should decide
> based on metrics. Ie, we plan to have no more than X% growth in mirror size for 1 year.
> If we hit some soft limit, then we investigate and decide ( ie, stop adding big backport,
> stop adding new package, etc ).
>
> And decide the metrics based on mirrors input, and based on packagers input.
> But so far, apart from Olivier and Wolfgang, we do not have much metrics and
> requirements :/
>
>    
>>>>               /non-free/ (disabled by default)
>>>>               /debug_*/ (disabled by default)
>>>>          
>>>
>>> And what are the relation of requirements ?
>>> Ie, what can requires non_free, codecs, games, etc ?
>>>
>>>        
>> IMHO /core/ should be selfcontained.
>> We are promoting open source after all.
>>      
> Yes, but what about the others ?
> Ie, can a game requires a codec or not ? a package in extra ?
> If we remove a package from extra, do we remove everything
> that requires it ?
>
>    
>>> And what about something that can goes in both media, ie a non_free
>>> game goes where ? A unmaintained codecs goes where ?
>>>        
>> Yeah, to be precise, that would need a games_non-free
>>      
> another media ? Really, I think most users are already lost with the
> current media selection.
> For core, we have 15/20 medias ( src + debug + binary ( 1 or 2 ) * update/release/testing/backport/
> backport testing ). Each media we add at the level of core will therefore add 15 to 20 medias too.
> So firmware, game, extras, codecs, non_free, that would make the total around 80 to 90 medias for a single
> arch ( I assume that firmware may not have debug_* )
>
> While it can be partially solved with a better interface for selecting media,
> we cannot do miracles if there is too much things :/
>
> So let's try to think how we can reduce the number of media.
>
> We have 2 kind of issue we try to solve at mirror level :
> - the concern of mirror admins
> - the concern of users.
> with impact on BS and packagers
>
> Mirror admins are concerned by :
> - size and growth ( see Wobo mail in the past thread )
> - content ( or at least, we think )
>
> Content part is mainly legal matter, but I didn't heard any admin
> telling "we can't do that", so that's my interpretation. The concern is
> mainly around DCMA and EUCD, even if lesser know laws also exist around
> the world ( like the Paragraph 202C of German law, who ban "hacking tools" ).
> For DMCA, there is some protection for them :
> http://www.benedict.com/Digital/Internet/DMCA/DMCA-SafeHarbor.aspx .
> For EUCD and the rest, I do not know.
>
>
> Users are concerned with a wide range of issues, some contradictory :
> - some want newer stuff, some don't
> - some want stable stuff, some do not care as much
> - some want non_free, some don't want it
> - some want firmware, some don't
> - etc
>
> Yet, the users concern mainly evolve around 2 things :
> - package availiability
> - package filtering, based on packages content
>
> The first part is already solved by the subdivision ( release, etc ). We
> need to split them for build reason. So we can't really avoid adding
> medias on this part.
>
> The second part is more tricky. And in fact, I think we can avoid creating media
> for this. Ie, do not let the concern of filtering appearing on
> the BS and mirrors, and push this on endusers system.
> Some people do not want firmware on their system, they do not really care about
> the firmware being in a separate directory on mirrors, as long as they can
> disable them easily from the list of package they can install ( at
> perl-urpm level, IMHO ).
>
> Same goes for non_free, or for nomaintained software. Or even games.
>
> So if we push the users issues on endusers system, we only have to manage the
> mirror admins issue on mirror.
>    

Exactly.
> And so here is a proposal that start by the size issue :
>
> - discuss with mirror admin, decide on a size that everybody would agree to mirror
> for core/ for the next release, or the 2 next one. Ie, every year or every 6 months,
> we do a survey of our mirrors, to see if everything goes well for them.
> - discuss also of the growth of core in term of size
> - decide on a limit size
> - if anything goes off limit for mirror, add a overflow/ to hold the packages
> that will not be mirrored by everybody. Overflow will be treated like core, in all points.
> Only difference is that mirroring is optional ( but strongly encouraged )
> - put everything in core, except what goes to overflow.
> - let users filter on their system, with something urpmi side ( I suggest a filtering
> when we do urpmi.update, but the exact details of how to do it are not relevent now ).
>
> Overflow will be filled with packages that :
> 1) are not required by anything else ( thus games data would likely fit,
> but not only )
> 2) have triggered the limit of size
>
> After the limit of core size is raised ( ie after all mirror have agreed ),we can readd packages
> from overflow to core, based on
> criteria not defined yet ( first come first serve, try to make most useful first ?
> or some wild guesstimate based on some mirrors stats ? ). But being in core or
> overflow should not change anything for both enduser and packagers. This is
> a mirror only concern, and so should be kept there only.
> And this should avoid discussion about the location of packages by packagers.
>
> This mean that both core and overflow should be by default on users system.
>    
Agreed.
> ( and I would not be against a better name, but I didn't found one )
>    
I like extra, which would fit nicely with the approach I'm about to suggest.

I would take a similar but somewhat different approach, which would 
probably have at least as good results.
First, decide what is *essential* to a fully functioning desktop or 
server or development system.  That goes into core.
Then decide what would be *very useful* in a typical such system.  Add 
that to core.
Of course, only the free packages.  Those not free would remain in non-free.

Core should then have the various kernels, the usual Linux utilities and 
development tools, the drak* and associated utilities.  Various 
pilotes.  The compete desktops, such as Gnome, KDE, LXDE, etc.  And 
certain common applications, such as LibreOffice, Firefox.
This leaves a lot of other packages, to go into extra.  (or overflow, if 
you prefer.)
Games would generally be in extra.  Or non-free.  (There may be a few 
small exceptions.)

This leaves many applications now in main, as well as virtually 
everything now in contrib, which would be in extra.
So core would be (probably much) smaller than main, and thus extra 
bigger than contrib.
And core would be just that : the core of Mageia.

Besides any advantage for space-limited mirrors which may exclude extra, 
we could collectively focus on ensuring first that everything in core 
works, to help ensure that user's systems would always be functional.  
Being reliable won't hurt our reputation.
>
> In order to reduce number of media, another question is :
> - should non_free have it own media ?
>
> Having them in core would simplify the BS, the upload and the mirroring.
>
> Having it separated would be better from various points of view ( political,
> communication, etc ). Maybe some people will refuse to help us if we don't,
> maybe there is some further restriction on some non-free software leading us
> to create another media whatever we do, I do not know.
> To me, as long as we can filter on user side, it would be ok.
>
> I cannot really tell what I prefer for that :/
>    
I think it is better to keep non-free.  It makes it very obvious to 
everyone, and avoids the down sides you mention.
We still avoid adding 3 sets of repositories :)
>
> So the only important mirror issue left to solve is the greyzone area.
> And well, that's quite complex.
>
> So we can either :
>
> 1) decide to not care ( ie everything in core )
> 2) decide to not offer them at all ( aka offload to PLF )
> 3) decide to add a media ( aka the "codecs" media )
>
> 1 is the simplest. But maybe not really a good idea.
>
> If we care, then what indeed should be done is another media, and let admins
> choose to mirrors it or not. I would even propose to revise the idea of
> separation every year, because if all mirrors have the
> 2 medias, no need to split in reality ( but I doubt it will happen, but
> at least, this would show that we try to revise our fondation on a regular
> basis ). And at least, we should revise the packages present in such medias.
> If there is some packages that can be moved to core,
> then they should.
>
> We could also simplify a bit the BS by placing non-free packages there
> ( instead of either having a non_free media, or the non_free pacakges in core ).
> It would sadden me a little to blur the line between "free with patents problems"
> from "non free", but my PLF experience showed that most people do not care, and that
> it requires more than a media separation.
>
> So, in the end, we would have :
>
> core/
>    release
>    updates
>    updates_testing
>    
improved name.
>    backports
>    backports_testing
>
> "overflow"/<- big packages, just for mirroring issues
>    

Which I would prefer to call "extra".  (Note that I suggest above a 
somewhat different contents, which would probably make it larger, and 
core smaller.)
Thus a mirror dropping it would save more space.

> restricted/<- with non_free, firmware, "codecs"
>    

It seems to me that totally free (of patents, etc) codecs would be 
better in core.
But the name allows containing packages that are nominally free, which 
is excellent.
> with the 5 directories under them, and with src, debug, binary.
> Imho, 3 upper medias is the simplest we can have ( besides debug/src, that
> I would place also on the same level than the binaries, but my
> mail is already long enough :/ )
>    

Long but very useful :)
To save additional space, maybe minimal mirrors could drop ISOs as well.
And maybe we could have other minimal mirrors with only all current ISOs.

In any case, at this point the important to decide what repositories to 
have.
Essentially I agree with your proposals.
>> For codecs either a extra_codecs or simply drop after a grace period.
>> but I guess codecs are important to people, so hopefully they wont
>> get orphaned...
>>      
> Unfortunately, there is not always a relation between "being important
> to users" and "someone want to take the burden of maintaining it" :/
> For example, something like etherpad would be nice for users,
> yet no one will take time to maintain it.

The proposed Mageia-app-db will hopefully help Mageia respond better to 
user's needs/desires.

my 2 cents :)

- André