[Mageia-dev] ANN: Recent changes in QA updates validation

[Samuel Verschelde]

Hi,

During summer QA has not stopped working, and is still trying to improve the 
process so that updates are pushed as quickly as possible.

First, the QA team now uses http://mageia.madb.org/tools/updates to have an 
overview of current updates candidates. Major security updates are first in 
list, in bold (or even red bold for highly critical ones). Those are usually 
pushed very fast.

Other changes include :

--- Taking into account packager tests for updates they provided ---

Until now, packagers could not test their own updates. We decided to change 
this rule because packagers often know their packages better than QA team. So 
now packagers can test their own updates, within the following conditions: 
only one arch per Mageia version, and final update validation must still be 
done by a QA team member.

So when you provide an update, you can tell us what you have tested, on which 
version and arch, and how (step by step instructions much appreciated!). Then 
a QA team member will just have to test the other arch and validate the 
update, hopefully very fast if a procedure is provided. 

This policy change can be reviewed later if problems arise.

For the record, here is how QA tests update candidates: 
https://wiki.mageia.org/en/QA_process_for_validating_updates#Test

--- Identification of update candidates coming with a testing procedure ---

Update candidates that have a testing procedure (either provided by the 
packager or by QA team members) are validated really faster than updates 
candidates without a procedure. Indeed, most of QA's time is put in trying to 
find how a package works, so we really appreciate testing instructions.

We added a new "has_procedure" whiteboard marker in bugzilla to identify 
updates that have a testing procedure. They show in 
http://mageia.madb.org/tools/updates

--- Identification of update candidates pending packager input ---

We added a new "feedback" whiteboard marker in bugzilla to identify updates 
for which packager input has been requested. They have a gray background in 
the aforementioned page. 

Update candidates with that marker are held for a few days in hope that the 
packager will answer (a simple "please push regardless of this issue" is 
sometimes enough as a feedback), and QA members give priority to other updates 
in the meantime. 

Note : they are not held forever, unless the problem is found to be blocking. 
We review stalled updates regularly, although we don't like pushing updates 
when the packager doesn't answer our questions at al, of coursel. Critical 
security fixes are usually not held this way. When questions are answered, QA 
team usually removes the feedback marker itself, but don't hesitate to do so 
yourself when answering!

Do you have any questions or remark?

Best regards

Samuel Verschelde