We studied PR evaluation in 15 open source communities, including ROS. We captured contributors’ and maintainers’ attitudes, beliefs and principles used to evaluate PRs.
Overall, the PR evaluation process is not only technical. Although, the aim of the evaluation process is to achieve quality, communities in our sample bring their social norms and beliefs to the process which influence the outcome and the experience of the parties involved on the PR review. These norms are informal and expressed through social customs or folkways. They are more prominent in some communities (e.g., Linux) compared to others (e.g., ROS).
We found three distinct attitudes, or styles, in processing pull requests: protective, equitable and lenient. While some communities adhere strictly to one style (e.g., Coala, Linux and ROS) others seem to be combining two styles, mainly equitability and leniency (e.g., OpenSUSE), see attached figure.
-
The protective style exhibits a defensive attitude. During the evaluation process values such as trust, relationships and the reliability of the contributor are considered. Although this style is unique to the Linux community, we also find its aspects elsewhere, e.g., jQuery. The motivation of this style is to mitigate the risks of unreliable contributors and substandard quality, i.e., only trusted, and reliable contributors can deliver a quality contributor within the expectations of the community.
-
The lenient style of PR evaluation rests on the belief that no contribution should be ignored. Each contribution carries enthusiasm that should be leveraged for the benefit of the community. This strategic behavior shows prioritizing growth and openness of the community with the main purpose of attracting and retaining contributors. This choice does not imply “spoon feeding” the contributor. Contributors are expected to show willingness to learn and meet the community expectations on quality and adhering to prescribed standards. This governance style does not compromise quality by being lenient. Less experienced contributors are mentored by either maintainers or more experienced community members and in some communities, they are assigned a “mentor” to help and guide them.
-
The equitable governance style believes in being fair and impartial regardless of who is the contributor. The evaluation of a PR is concerned more with technicalities and less with social aspects. The purpose remains high-quality contributions with the belief that the contribution matters and not the individual. The reviewers and the maintainer focus on the technical merits of the PR.
We find that the ROS community shows strong adherence to the equitable style. While a good news, this is also surprising. We see little willingness to mentor contributors with limited experience with the ROS ecosystem and the technical expectations in ROS regarding quality. This is surprising given the high complexity of the ROS ecosystem and the learning path for newcomers could be long and not straightforward.
We do not have concrete recommendations for open source communities overall, because every community is unique and has a different history. These PR governance styles exist for a reason. We think it is up to the community and its leaders to decide how to go about it. However, If the priority of the ROS community is to remain sustainable (i.e., prolonged participation of contributors and attracting new contributors) then, the community should balance between leniency and equitability. i.e., invest in the enthusiasm of newcomers; they may become contributors in the long run. This could be implemented through a mentoring program. We also observed in equitable communities that the expectations are not always documented. Contributors are entitled to know what is expected from them to meet quality and other technical expectations to make the effort they invested in producing the PR worthy.
Happy to answer questions. We would like to thank the survey participants from the ROS community for making the study possible.