Source code layout revision

As I said over in the repositories topic,

related to splitting the repositories is reorganising the package structure

This is the topic for discussing that.

The current structure has several problems that make it hard to manage and maintain, let alone understand for new contributors, such as:

  • It mixes several software projects in a single repository when they should be in separate repositories related by an organisation. For example, nodes for driving a car, and a tablet-based UI.
  • You have to dig down a long way to get to the actual source code, e.g. five directories deep to get to the messages, and often several more to get to node source files.
  • The layout in no way conforms to the proposed best practice for ROS 2 repositories.

We need to reorganise packages as part of splitting up the repositories. However we should also take the opportunity to reorganise the files as well. This includes cleaning up the structure of a package, moving files to other packages, splitting and/or merging packages, and so on.

@shinpei0208 has been pushing a repository reorganisation that you can find here (188.9 KB). However, this preserves the problems stated above so I am not in favour if adopting it wholesale.

I think that reorganising the packages and the code is something to be tackled over time during the 1.12 iteration, and possibly into 1.14 as well. It is not something to hurry. It is also something we should do a little at a time, not all at once. So I am not concerned with having a plan ready to go by the 14th of March for the 1.12 release startup period. However I do want us to start talking about it now and gradually work towards a consensus.

In particular, I want a consensus on the guidelines for how to organise a repository and how to organised a package. A good starting point is the ROS 2 guidelines for organising a repository and organising a package. Also worth consulting are large projects like MoveIt2 and navigation2, although both are under active development so consider that their layouts may change.

Based on the guides above and the fact that we want to transition to ROS 2, here are some principles I have come up with. I will edit these as the discussion continues to reflect the developing consensus.

Repository guidelines:

  • If a repository is intended to contain a single package, that package should be at the top level directory, not in a subdirectory.
    • I do not expect us to be in this situation, but for completeness I’ve included it.
  • In general, the repository should contain packages in a flat structure.
  • If there is an argument for it, a repository may contain one level of subdirectories that divide up packages by some categorisation. However only one level is allowed and the subdirectories are purely for repository organisation.

Package guidelines:

  • Follow the ROS 2 recommended layout.
  • Package content is determined by relatedness, not by a rule such as “only nodes in a package” or “this package must only contain a library, the nodes go elsewhere.”
  • If a package contains a library that has no ROS dependencies, it should be a pure CMake package such that it could be built on its own using non-ROS, non-colcon tools.
    • Packages like this are prime candidates for being their own repository and treated like any non-ROS software.
  • Messages go in their own package, in keeping with ROS 2 best practices.
  • URDF files and other hardware description-related stuff should go in its own package. For example, the URDF for a Velodyne should be its own package.
  • Launch files not related solely to the nodes in one package (e.g. a launch file to bring up all of Autoware, or a particular perception pipeline) are candidates for having their own package.

Please add any additional guidelines or modifications to these ones in the comments below.

2 Likes

Here is a rough estimate of the packages we have in Autoware. The first two columns are for deciding where to put them. I will fill them in as the discussion progresses.

If the discussion in the repository thread goes against splitting the repositories, then the second column should be read as “top level directory” instead.

(Done as a Google Sheet because Discourse does not like wide tables.)

1 Like

Quite a few features have been put up for 1.12 that will touch a lot of code, such as messages refactoring, amentifying packages and changing the architecture for localisation.

Since touching a lot of code in a PR will be complicated if we are also moving that code around at the same time, I’d like to rethink a little when we do package moving.

I can think of three ways to do it:

  1. Move all packages to reasonable choices for their new home now as soon as possible, accepting we might move some packages again later, or delete or split them. At 131 packages, I think if two people sit together (virtually) down and spend a day on it we can get it done.
  2. Wait until after feature freeze, then move all packages during the bug fix period. I’m not keen on this because it complicates testing.
  3. Prepare the move in a fork, then do it in one hit immediately after 1.12 is released.

I’m in favour of option 1 or 3, with no strong preference for either.

Whichever we choose, I plan to move the messages as-is to their new repository in the next few days anyway.

The lack of integration tests complicates this but I think we just have to accept that.

Fortunately the move to using vcs means that most users should not notice any difference once that has been fully tested and documentation written.

I will update the spreadsheet soon based on the comments given so far.

Voting for 1

So PRs targetting 1.12 will be redirected to the correct repository.

I can virtually help :upside_down_face:

Ya 1 might be best, just to pull the band-aid off as fast as possible :smiley:
I have a list of small fixes and things that I want open PRs for, but I’m waiting for the repo split to happen.
Probably the sooner the better to get it over with.

OK, there are a bunch of people at Tier IV who have not responded here, so I will do a quick poll of them on Monday. If none of them object then I will try and rush through a rough split by the end of the week. I’ll probably need @amc-nu to help out.

2 Likes

I’m in favor of 1, we certainly won’t get it right on the first try, but at least it gets the ball rolling.

2 would complicate testing as you say, plus other projects that depend on Autoware may need to have a provisional vision now of what the layout may end up looking.

3 sounds good, but it’d either require everyone to stop working on Autoware until the new layout is done or it’d incur in more work later to reconcile all the conflicts with the merge.

Regarding messages and the rest of the packages, since the plan is to get both Autoware.AI and Autoware.Auto under the same organization, we’ll have to evaluate if we need to prefix all the repositories so they don’t collide. Autoware.Auto uses a single repository now, though, so perhaps it’s not that urgent.

I’ve updates the spreadsheet with a first run of moving packages. I had to make informed guesses for many packages.

Many of the packages were commented as being unnecessary, due to things like forks of existing ROS packages that are no longer needed and old packages that are just no longer used. I haven’t removed these yet, only moved them. We can deal with removing older packages next.

The packages have been moved to their new repositories, but the PRs for removing them from the autoware repository have not yet been merged. We are going to try and get the new locations building independently first, or at least make sure Autoware as a whole works from the new repositories, before we remove the packages from the autoware repository. @sgermanserrano has been working hard on setting up GitLab CI for the new repositories.

1 Like