in collaboration with researchers of the Carnegie Mellon University and the Vrije Universiteit Amsterdam, we defined a set of 49 guidelines about how to architect ROS-based systems. You can find the guidelines and the steps we performed for extracting them here.
Since we are just starting to do this kind of research on ROS, any possible feedback is very welcome!
What do you think about this kind of research? Do you find it useful? Is there some other aspect of ROS which you think is worth this kind of activities?
The full dataset we used for extracting the guidelines from GitHub repositories is available here.
Many thanks for sharing this! I read the whole paper and found lots of useful information! I will allow to share it on Weekly Robotics as I’m sure many people will find it useful.
Here are some things I think I would find useful mingled with some suggestions:
I’d be very interested to learn about the depth of launch files used in the analyzed projects. Quite often we have launchfiles including other launchfiles. Wonder what is the maximum depth that can be found in the wild
I’d also mention some guidelines on adding external packages. Ideally we want to list them as dependency and make rosdep handle it for us but sometimes packages are not released or we need to fork them in which case submodules are helpful (what I usually use but I know some people use some other repo managers)
What would be a killer for me is if someone took top 5-10 “top repos of platforms” and recorded bag files with representative data. I think this would be very useful for ‘good practices’ as you can quickly see how the whole stack works what kind of sensor data is provided and in what format etc.
It’s a pleasure to be included in the Weekly Robotics mailing list.
About your mentioned points:
Launch files: there is a paper from 2015 about how ROS launch files are developed, see it here
External packages: I agree, this may also create maintainability issues since then you need to manually check if the added packages have been updated, right?
Bag files recording: this is a very good idea and it is technically doable. I am wondering about how to (i) choose the “top platforms” in a replicable and unbiased manner and (ii) how to generate scenarios which are “representative” of the common usages of those platforms. I will think about it in the coming days.
worried about continuity (ie: external repositories disappearing): fork and point to your fork
worried about reproducability (ie: avoid tracking moving upstreams): specify versions (ie: commits/tags, not branch names) in scripts/.rosinstall/.repos files (avoid using scripts for seeding workspaces btw)
The second bullet is also less of a problem when using Git submodules, but I tend to not use those (for various reasons).
Comment/suggestion: this would be a great post for #quality category, instead of here in #general. You could consider moving the post there.
@iivanoo: could you perhaps also say a bit more on the repository selection process? The paper doesn’t seem to explain how you selected the initial set of repositories (ie: before you started pruning).
Right, it all makes sense, thanks Gijs for the hints.
Yes, indeed, it started as an announcement and it is becoming a full discussion now
Sure. The initial set of target repositories has been defined by reusing two other research contributions:
Rosmap: it is an analysis tool for scanning dependencies among ROS packages in GitHub, GitLab, and Bitbucket. The first step of the analysis tool consists in performing queries in those three platform and look for repositories taggeg with the ROS topic. Also, rosmap checks the official ROS Index in the rosdistro-repository and considers all repositories mentioned in there as well. In my research I am using rosmap only for the search step.
GHTorrent: it is a database (either as MySQL or mongodb) containing a periodically-built mirror of all projects’ metadata in GitHub. I am querying GHTorrent by selecting all repositories which are either tagged with the “ROS” topic or their name (or description) contain the “ROS” substring. Here you can see the specific SQL queries I performed in GHTorrent.
Many thanks for the launch files paper! Will definitely have a look at it!
External packages: yes, if you use submodules then you need to manually sync them with upstream, especially if there are some interesting developments in there for you.
Bagfiles: I know that chossing “top platforms” in unbiased manner is a challenge but maybe you can select them based on number of stars in the repo? Or you can be biased and say: these platforms do everything correctly in our view, check out examples of their raw data.
I’m now realizing that the bagfile data project I’m proposing might not necessarily be fit for a research project but having something similar to AOSAA book but for Robotics could be very interesting. @Katherine_Scott maybe OpenRobotics could lead such effort? I’d be up for writing a chapter on some platform if this was to become a thing.
You are welcome! And thanks for the clarification on the external packages.
In empirical software engineering there are many ways in which this bias may be controlled. For example, here is a paper about the do’s and dont’s related to the treatment of third-party GitHub repositories for scientific purposes.
This is extremely interesting for me (and for my research group) and we are definitely willing to help and put effort in this!
This is probably in the paper and I just missed it, but do you have any recommendations on the best way to document the system architecture? The paper says:
… documented architecture (i.e. the system is described in terms of nodes, services, topics, and their configuration).
Are these best described in a single README file or as ROS node graphs with text descriptions or do you have some other example of great architecture documentation that the rest of us can follow?
Is there one particularly excellent architecture that could be used as a starting point for most other robot architectures? Perhaps something like the Niryo One five layers architecture of external communication, command, motion planning, control, and hardware? Or are most robots unique enough to require independently designed architectures? Thanks!
Thanks Peter, here are my thoughts on your points.
My experience with this study is that people document the architecture of their systems in a very heterogeneous way, ranging from raw node communication graphs, to box-and-line diagrams done in Powerpoint, etc. Actually, in the dataset I collected a direct link to each Architecture documentation, you can refer to the “SA documentation” column here and explore.
About recommendations, I think that the best architectures (in terms of documentation) are the ones combining a graphical representation (of both nodes, topics, and pub/subs) and a textual description of the main responsibilities of each node. Personally, I also like those in which you have a high-level architecture description at the beginning and then details are provided in a top-down fashion, such as in MoveIt.
This is difficult to say. I think it is doable to have some kind of reference architecture for robots belonging to specific domains (e.g., delivery robots, or industrial arms), but it really depends on the requirements and regulations of each domain. If we want to be generic and cover all application domains, maybe the Niryo One layered architecture is a good candidate, but I doubt it will be able to cover all possible needs. Funnily enough, in these weeks I am working with a group of researchers between Italy and Sweden on a generic platform for (1) specifying reference architectures of arbitrary application domains (e.g., delivery robots) and (2) automatically checking the compliance of specific architectures against the reference architecture of their domain. I will keep you posted about it, if you are interested
The MoveIt documentation looks fantastic. I imagine it takes a lot of manual work to make it look that nice. Do you think it is possible to generate system architecture documentation automatically, even if it does not look quite that good, in a standardized way that makes it easy to compare architectures and subsystem compatibility? Maybe something like rqt_graph. I do not know how much could be determined statically and how much would need runtime information. Or does proper documentation require high level information that you cannot get from a set of packages alone? It seems like if you have enough information to check the compliance of an architecture then you would have enough information to automatically create documentation for it.
as soon as we will publish the paper I will come back to you with a copy of it.
About the automatic generation. There are approaches and some of them are pretty cool (see Haros by André in his other message), but the problem is still very hard, specially with ROS where the “architectural connectors” emerge only at run-time, basically when you publish/subscribe to specific topics. Interestingly, in ROS it is “fairly simple” to identify the main components of the architecture, but the connects are difficult to extract.
And yes, it could be fantastic (also for us, researchers) to have a standard notation for representing the architecture of robotics systems and to be able to compare architectures across systems, types of robots, and application domains. This would open up for large-scale studies where we could extract even more details information from the system, even more than architecting guidelines!