[Mageia-dev] bug 2317 revisited: --update option should behave like --search-media

[Claire Robinson]

On 21/06/12 22:01, AL13N wrote:
> Op donderdag 21 juni 2012 22:21:52 schreef Sander Lepik:
>> On Jun 21, 2012 9:10 PM, "AL13N"<alien at rmail.be>
>>
>>> so, there's 2 options:
>>>
>>> - testing i586 with backports enabled
>>> - testing x86_64 without backports enabled
>>>
>>> this is still 2 tests, and this is sufficient.
>>
>> Are you serious?
>> I've seen bugs were i586 and x86_64 doesn't work quite the same. Every arch
>> + repo must be tested separately (be it tainted or release, i'm still not
>> mixing backports with updates ... until you promise to do all the testing
>> here and not bother QA;)).
>
> I see...
>
> however, as long as backports is installed, it could still be that due to an
> update a new dependency from release is pulled, which could conflict (or not
> work correctly) with some of the installed backports.
>
> if we want to have supported both backports and updates, these cases should
> still be tested. And if you want to support cherrypicking backports, the
> possibilities are even bigger.
>
> It seems to me that people don't really see what kind of QA resources
> backports will need if all this need to be supported. This is completely
> irrelevant to this solution however. whatever solution we pick for bug 2317
> (A, B, or even doing nothing), it seems you think this solution has any effect
> on QA resources, but it doesn't, it's only enabling backports that does it.
>
> Let me give you an overview on options for the amount of support in backports
> and it's impact on QA (also with example) for only 1 release:
>
> (package X has update A&  backport B;
> package Y has update C and backport D;
> package Z has update E and backport F)
>
>
> A. backports is fully supported + cherry-picking backports is also supported
> (for only mga2)
>
> for update validation of package X (let's call it update A2):
> 1. testing combination: A,C,E for arch i586
> 2. testing combination: A,D,E for arch i586
> 3. testing combination: A,C,F for arch i586
> 4. testing combination: A,D,F for arch i586
> 5. testing combination: A,C,E for arch x86_64
> 6. testing combination: A,D,E for arch x86_64
> 7. testing combination: A,C,F for arch x86_64
> 8. testing combination: A,D,F for arch x86_64
>
> for backport validation of package X (let's call it backport B2):
> 1. testing combination: B,C,E for arch i586
> 2. testing combination: B,D,E for arch i586
> 3. testing combination: B,C,F for arch i586
> 4. testing combination: B,D,F for arch i586
> 5. testing combination: B,C,E for arch x86_64
> 6. testing combination: B,D,E for arch x86_64
> 7. testing combination: B,C,F for arch x86_64
> 8. testing combination: B,D,F for arch x86_64
>
> Validations required: 2*2^(number of backported packages - 1)
> ==>  this is completely impossible
>
> B. backports is fully supported, but cherry-picking isn't
>
> for update validation of package X (let's call it update A2):
> 1. testing combination: A,C,E for arch i586
> 2. testing combination: A,D,F for arch i586
> 3. testing combination: A,C,E for arch x86_64
> 4. testing combination: A,D,F for arch x86_64
>
> for backport validation of package X (let's call it backport B2):
> 1. testing combination: B,C,E for arch i586
> 2. testing combination: B,D,F for arch i586
> 3. testing combination: B,C,E for arch x86_64
> 4. testing combination: B,D,F for arch x86_64
>
> Validations required: 4 for each package
> ==>  this is quadrupling the QA workload
>
> B2. i thought perhaps that testing these on one arch would be ok:
>
> for update validation of package X (let's call it update A2):
> 1. testing combination: A,C,E for arch i586
> 2. testing combination: A,D,F for arch x86_64
>
> OR
>
> 1. testing combination: A,D,F for arch i586
> 2. testing combination: A,C,E for arch x86_64
>
> for backport validation of package X (let's call it backport B2):
> 1. testing combination: B,C,E for arch i586
> 2. testing combination: B,D,F for arch x86_64
>
> OR
>
> 1. testing combination: B,D,F for arch i586
> 2. testing combination: B,C,E for arch x86_64
>
> Validations required: 2 for each package
> =>  this could be doable by QA, even though it's perhaps not completely tested
>
> C. perhaps backports being semi-supported
>
> for update validation of package X (let's call it update A2):
> 1. testing combination: A,C,E for arch i586
> 2. testing combination: A,C,E for arch x86_64
>
> for backport validation of package X (let's call it backport B2):
> 1. testing combination: B,C,E for arch i586
> 2. testing combination: B,C,E for arch x86_64
>
> Validations required: 2 for each package
> =>  this could be doable by QA, but even though a package might work, it's
> possible that an update (or backport), might not be cleanly installable and
> give errors.
>
> D. not supporting backports
>
> for update validation of package X (let's call it update A2):
> 1. testing combination: A,C,E for arch i586
> 2. testing combination: A,C,E for arch x86_64
>
> for backport validation of package X (let's call it backport B2):
> No testing
>
> Validations required: 2 for each update
> =>  this is how it is now
>
> --------------------------------------------------
> I would implore all of you to look at this above and try to understand.
>
> (of course, you can tell me if i'm wrong here)
>
>
> It seems to me that all of you had thought that C was good enough validation
> (except for tv, i guess he saw this from the beginning and figured D would be
> best or even E, no backports at all).
>
> I'm was thinking that if C was good enough for you, then perhaps B2 would also
> be good, as i think it doesn't give any extra load on QA.
>
>
> IMPORTANT: Again, i'm stating that this does NOT matter whatever solution for
> bug 2317 is chosen.
>
> even my preferred solution on bug 2317 ONLY has more testing requirement for
> the backport packager
>
>
> Is this more clear?


All this assumes that backport media will be treated as a normal update 
media. That is certainly not my impression. My impression of backports 
are being able to install a new blender for example, not having a system 
where backports are just another update media and replace everything 
available. The QA task for that scenario would be ridiculously huge. If 
you want to have backports which go any further than backports testing 
then you seriously need to rethink this idea.

I don't think you understand quite how short handed we are in QA. For 
the life of Mageia 1 it was mainly just two people. This bug is a major 
bug for QA and has been since it was first discovered almost a year ago. 
It adds complexity and extra workload to an already severely overloaded 
team.

We have a few (very few) new people since Mageia 2 was released who are 
beginning to help out, understand the process and how to find ways to 
test things. Thankyou all, you know who you are. The complexity of 
working around this bug is something they shouldn't have to learn. The 
extra workload involved in working around this bug is something we can't 
continue with, even in our current state with no backport medias. I 
spent most of yesterday on it for example instead of testing updates. 
Validating updates is now taking twice as long because there is twice as 
much testing involved for two releases - plus the extra time spent 
working around bug 2317 for each release. Yesterday we even found its 
effects are different for some rpm's on i586 than they are on x86_64 
(see p11-kit bug 6502).

Anything which, at this stage, slows this process further will lead to 
updates being further and further delayed. That is a situation nobody 
wants to see, and personally I'd like to be able to take a day off now 
and again too. I'm sure others feel the same.

The aim of fixing this bug is to reduce the complexity and extra 
workload of working around it for QA. This assumption and solution 
actually has the opposite effect, dramatically increasing the complexity 
and workload. As I've explained, that is simply not possible if we want 
to release timely updates.

I hope this makes the situation clearer. There is a workable solution 
but I'm afraid it isn't this one, for the reasons given above.

Thanks
Claire