Improving the handling of PR:s

From the hideous flamewar-about-everything thread:

> We clearly have a disconnect on PRs.  More come in than we have any way
> to handle.  This is clearly most distressing when the PRs contain
> patches and/or test cases.  Again, I'm open to ideas on how to set up
> something where more people can participate.

Eric Anderson posted a suggestion that I read and won't comment on other than 
to say that from the perspective of the non-committer it sounds like a good 
idea.

I have another suggestion.

The fundamental problem, I think everyone agrees, is that there is simply not 
enough commit bit developer time available to attend to all the PR:s. This is 
understandable.

But understandable or not, the problem becomes particularly frustrating when 
it affects PR:s that contain patches. Such PR:s constitute direct 
contributions to the project. In cases where such patches are correct/good 
enough, the non-application of those patches have what I believe to be 
significant effects.

The one perhaps lesser effect is that other people are bitten by the same bug 
(which translates into loss of time on potential future contribotors) even 
though a fix is available.

The second, possibly worse effect, is that the original submitter, I believe, 
is less inclined to spend more time in the future contributing, if said 
submitter feels the work is wasted because no one commits or even comments on 
the PR.

In the case of PR:s with patches, why do they not get enough attention? 
Obviously there is no technical reason why the application of the patch 
itself should be a problem (that could be completely automated even). The 
problem, presumably, will be that the time it takes to process the PR is much 
more than the mechanical act of applying a patch because (and this list is 
probably not exhaustive):

* The patch must be looked at and understood prior to application or even 
testing. Does it comply with style guidelines? Does it break something else 
as a side-effect? What are other implications? And so on.

* The committer may not have access to the hardware, or may not have a 
software setup that allows for testing. This means doing such testing 
suddenly requires a lot more effort.

* The committer may not be familiar with the code and, even though the patch 
looks good generally, it is felt that someone familiar with that part of the 
system needs to have a look. This means the committer can either spend 
additional time on understanding the surrounding codebase, or leave it for 
the subset of committers that have that understanding already.

One observation is that there is no realistic way for the community of 
non-committers to help effectively the committer in the sense of lessening 
the amount of work required to process the PR.

What I suggest is a system where a group of non-committers systematically 
pre-process PR:s, to provide feedback and additional quality assurance to the 
committer that is to process the PR.

This group of people could either be a self-proclaimed group of volunteers, or 
perhaps a group of people satisfying the criteria of "guy we kinda trust to 
do testing and provide a useful indication of sanity and correctness, but not 
with a commit bit".

Given a fairly simple PR with a fairly simple patch, would not the work of the 
committer be greatly lessened if there was a track record of the 
above "pre-processors" shows that <n> people have performed building and 
testing of the patched version on architectures/environments, <x, y and z>, 
and that they generally like the patch (or that they don't)?

One possible answer to this that I have gotten in the past with another 
project, is that noone is stopping me from just grabbing a bunch of PR:s and 
posting follow-ups saying I've tested something, or otherwise giving 
feedback.

The problem is that there is no real indication that this will cause anything 
to happen other than generating more E-Mail traffic.

What I suggest may make the difference is to have a *system* for this. 
Preferably with software to support it.

For example, patches with a high confidence of correctness due to many people 
affirming that they have tested it could be automatically prioritized for 
committers to deal with, and there will be a clear and systematic record of 
what testing was done, and by who. And the individual contributor (for 
testing) can really feel that the time spent was worthwhile, because the 
feedback was taken by the system and used for actual effect.

When I talk about priority above, I do not mean to imply that any committer 
should work on things they do not want to work on. But I have to assume that 
a lot of patches get committed because a committer decided to go and pick a 
few PR:s to process, rather than the committer necessarily has a burning 
interest in that particular patch.

As such, if said committer could spend those <n> minutes closing five times as 
many PR:s because the up-front confidence in the correctness of the patch is 
so much higher, perhaps it would help the system to "scale", moving some of 
the burden off the committers.

Perhaps one can either go so far as to have "full committers" give their 
blessing for a particular patch, possibly based on feedback such as above, 
but leave the act of actually commiting the patch and doing the actual smoke 
testing (that it builds etc) to someone who does have a commit bit and is 
trusted with it, but is not allowed to make any commits that are not directly 
blessed by a "full" committer.

There is some precedent for similar systems. Squeak had the concept 
of "harvesters". They seem to have changed things a bit nowadays, but 
seemingly use a similar approach still (I am no longer up to date):

   http://wiki.squeak.org/squeak/3152

(see the comment/review/vote period)

-- 
/ Peter Schuller

PGP userID: 0xE9758B7D or 'Peter Schuller <peter.schuller_at_infidyne.com>'
Key retrieval: Send an E-Mail to getpgpkey_at_scode.org
E-Mail: peter.schuller_at_infidyne.com Web: http://www.scode.org