Re: Improving the handling of PR:s

On Fri, Jan 11, 2008 at 07:35:41PM +0100, Peter Schuller wrote:
> But understandable or not, the problem becomes particularly frustrating when 
> it affects PR:s that contain patches. Such PR:s constitute direct 
> contributions to the project. In cases where such patches are correct/good 
> enough, the non-application of those patches have what I believe to be 
> significant effects.

I think you make some excellent points -- especially with how
understandable it is for someone who's submitted a patch which got
ignored, to not do the work to submit the next one.

With respect to patches, the two things that I've tried -- and they have
been insufficient -- are the following:

 - weekly email of "PRs containing patches"

 - weekly email of "PRs recommended by the bugbuster team"

We can make the following observations:

 - the "push" paradigm doesn't work particularly well.  Partly this is
   due to fatigue if you try to read through all this stuff (see below).

 - the "recommended" list has been slightly effective, but it's going
   to take some time for it to take off.  For one thing, it has been
   underpublicized; for another, we don't have the culture of committers
   getting "warm fuzzies" from committing PRs.  (Since no one working
   on that kind of stuff gets paid AFAIK, that's the only positive
   feedback they're going to get.)

Last week someone made the observation that if these things were instead
updated daily and posted to a web page, that it would be far more useful.
I've taken that up as a task.

Your other point about "latest PRs" is also a good one.  That should be
relatively easy to do.  I'll take up that task.

> * The committer may not be familiar with the code and, even though the patch 
> looks good generally, it is felt that someone familiar with that part of the 
> system needs to have a look.

In particular, this hurts for areas of the system that are unmaintained.

> What I suggest is a system where a group of non-committers systematically 
> pre-process PR:s, to provide feedback and additional quality assurance to the 
> committer that is to process the PR.
> 
> This group of people could either be a self-proclaimed group of volunteers, or 
> perhaps a group of people satisfying the criteria of "guy we kinda trust to 
> do testing and provide a useful indication of sanity and correctness, but not 
> with a commit bit".

So far it hasn't happened.  We've set up the freebsd-bugbusters_at_ mailing
list and the #freebsd-bugbusters IRC channel on EFNet (and please join
us!) and the latter is where our last 2 bugathons took place.

It's clear that there are several people who want to help process the
PRs, and we don't have a good answer for them on "how can I contribute?".
The existing tool, and social conventions, don't allow for non-committers
to change PR states.  As far as we've done in the past is to grant people
"GNATS access" rights but not "commit rights", on an experimental basis.
We've done this twice, and although it has worked well, just two people
isn't enough.  (One has gone on to become a full committer -- which is
great!; the other current does not have as much time for FreeBSD work).
Several hundred PRs were dealt with by these two folks, so I consider the
experiment a success.

What we used as a qualification was "track record of responding to PRs and
questions on mailing lists", fwiw.

> One possible answer to this that I have gotten in the past with another 
> project, is that noone is stopping me from just grabbing a bunch of PR:s and 
> posting follow-ups saying I've tested something, or otherwise giving 
> feedback.

Yes, but that's open-loop as well.  It's the same situation as with submitting
the patch in the first place: it's going to get frustrating if no committer
picks it up and commits it.

There's not too many people so thick-skinned and persistent as to keep
going forward when no one's using their work.

> For example, patches with a high confidence of correctness due to many people 
> affirming that they have tested it could be automatically prioritized for 
> committers to deal with, and there will be a clear and systematic record of 
> what testing was done, and by who.

Right.  The "priority" field in GNATS has been so abused as to become
useless.  (People assume that setting it higher will cause some kind of
magic to happen.)

My current thinking is that what committers ought to see is a metric of
correctness, and a metric of priority, _as set by someone who's vetted
the PR_.  The weekly mailings are too poor an approximation of either
to be useful.

Adding the second metric would cure one problem that you don't mention --
which is that few people have the interest and patience to plow through
N-thousand PRs.  It's not humanly possible to look at them all -- even
the new ones as they come in.  There's simply too many.  So, you create
an expectation "why bother, there's so many anyways".  We need to break
that chain of expectation.  A good fix is a good fix.  The PR count will
never get to zero; I (with bugmaster hat on) would be thrilled if we can
get to the point of just steady-state.

> When I talk about priority above, I do not mean to imply that any committer 
> should work on things they do not want to work on. But I have to assume that 
> a lot of patches get committed because a committer decided to go and pick a 
> few PR:s to process, rather than the committer necessarily has a burning 
> interest in that particular patch.

I think most get committed because a committer sees a PR come in on the
mailing list and grabs it.  Much less often do committers go through the
database looking for things to fix.  Again, the lousy "search/browse"
capabilities of the existing tool let us down here.

> As such, if said committer could spend those <n> minutes closing five times as 
> many PR:s because the up-front confidence in the correctness of the patch is 
> so much higher, perhaps it would help the system to "scale", moving some of 
> the burden off the committers.

Right.  What we need is to start small and create a _culture_ of where
it's fun (or at least intellectually interesting) to work on this stuff.
I think the IRC channel may still be the best bet for this.

Once I finish fiddling around with the sparc64-6 and sparc64-7 release
builds (the first approximation is finished, but I believe we can still
add some more packages), I intend to task-switch over to looking towards
defining a PR workflow and floating some proposals.  I hope to have
something concrete to present at BSDCan.  Please join us on bugbusters_at_
to discuss any more ideas, too.

mcl