REP 2004 3.v.b has the recommendation that QL1 and QL2 packages should be registered “with a centralized list of ‘Level N’ packages, if one exists, to allow for peer review of the claim”.
Having looked through a few QL1 and QL2 packages and their quality declarations, I haven’t actually found any reference to such a “centralized” list.
Does anyone know if such a list exists already? If not, is there interest in starting one?
I understand the QL system is relatively new, and that 3.v.b is listed as a recommendation rather than a requirement. But I still think having such a public list would be beneficial to the community, both to give a sense of the overall maturity of ROS 2, and to potentially guide development focus towards maturing specific core packages.
I would be in favor of having such a list, but only if it is automatically generated nightly from the sources. Any manually-curated list would get out-of-date extremely quickly, and would be just one more place that we need to update when we change quality levels.
There’s bad news and good news about the automatic generation. The bad news is that the QUALITY_DECLARATION.md files in the ROS 2 core are human-readable, not machine-readable. So there are no guarantees on the format. The good news is that, at least in the ROS 2 core, the QUALITY_DECLARATION.md files are sufficiently similar that you can parse the data out pretty quickly. I wrote GitHub - clalancette/walk-qds a while ago which will find the quality level of a package and all of its dependencies. It works by walking a directory structure and looking for all of the QUALITY_DECLARATION.md files, then applying a regex to find a certain phrase.
Going forward, I think we either want to more strictly define the wording that defines the quality phrase, or define a machine-readable section in the QUALITY_DECLARATION.md that can be parsed more exactly.
I don’t 100% agree with this. If we just want a list of other packages that have a quality declaration, then this is fine, but the point of that suggestion in REP-2004 is so that we form a peer-review system (maybe an informational REP, maybe just a file in a repository that requires reviews to merge into), and importantly staying on that list cannot be automatic, because some of the requirements for quality declarations are on-going efforts, like are the change control processes being followed? is the code coverage being maintained? etc…
The idea was that any list would be curated and kept up-to-date, probably having a separate list for each ROS distribution.
Obviously this is a lot more work, but it’s (in my opinion) the only reasonable way to ensure the things on the list are actually at the quality level they declare. And if it is important enough to the community that process will be set in place and staffed, otherwise we’ll continue not to have one (which is ok I think until people actually want/need it).
But either way, that’s the intent of 3.v.b in REP-2004, not just to index the packages with a QD.
I had done the same, and had noticed that all of the existing declarations were consistent in the quality level declaration line, for what its worth. But I agree that this could be tricky in the future and may warrant some stricter formatting.
I also agree that the intent should be to have some peer-oversight… but perhaps as an intermediate solution, there is room for a middle ground approach? E.g. maintaining a list with three columns: package name, declared quality level, and “verified” quality level. The declared quality level could be auto-populated, for a low-effort quick reference for the community, and the “verified” quality level could require manual review whenever the declared level doesn’t match the verified level.
Yeah, that would be useful and could be automated I think.
The verified column would need to draw from some list of packages that were reviewed, so that could be the “just a file in a repository that requires reviews to merge into” that I mentioned, like a json file or something, but I still think it shouldn’t be a persistent honor. At the very least it should be re-evaluated each ROS release.
This is again more work (which we generally try to avoid), but if a package is to maintain quality but the effort the cannot be found to review its QD, then it seems unlikely to meet the other goals, to be honest.