Survey on the Fairness of Pull Requests Assessment

The pull requests (PR) assessment process is critical to the sustainability of open source contributions. The scientific literature indicates that only 13% of contributions are rejected for technical reasons. Some contributors experience positive circumstances, but others do not. I set out to find out what makes an evaluation of a PR fair and what contributors see as unfair. As PR nonacceptance often results in demotivated contributors who may cease contributing to the community, recruitment and maintenance of new contributors is critical.

I will analyze all “fair” and “unfair” PRs cases and draw conclusions on the factors motivating contributors for creating a more welcoming environment for people from different backgrounds. The results will be shared with the ROS community as soon as they become available. The survey is only 9 questions. Please note that participating in the survey implies supplying two PRs links (one you perceive being judge fairly and a second one which you perceive being judge unfairly). Please participate, link to the survey.

3 Likes

It requires you to link to a pull request that you consider to be fairly accessed. I don’t think you’re going to get good results from the survey because of this requirement.

Seeing as the analysis the OP wants to perform is going to be based on actual PRs, I don’t see how that could be avoided @John_Tapsell.

Would you have a suggestion on how you’d approach this?

It could show the code of an actual PR, and give the rejection reason, and have us judge whether it was fair, whether we agree, etc.

I might suggest that the question about a PR come much earlier in the survey. It feels like I’m being jerked around when I’m asked to answer a simple 9 question survey, and the first 8 questions are routine, and then the last one is essentially “should you be participating in this survey at all”?

1 Like

Thanks for the feedback. The survey was scientifically designed and the reason behind asking for PR examples, is to analyze the content (e.g. the discussion, the actions and the decisions) made during the interaction of the PR. The literature suggests that most of a PR interaction is social, hence the importance of the PR’s content.

1 Like

Thanks for the feedback. Good point, which we learnt after few days of the survey being live. However, traditionally, the demographic questions usually come first. To be more transparent, I’ll clarify in the original post that participating implies supplying two PRs links.

1 Like

Dear ROS community,

Thanks to those who participated. We received a significant number. This is just a friendly reminder to participate. The survey will be closed Friday the 1st of May.