Splitting the Autoware.AI repository and changing the organisation

[gbiggs]

All very good points, @davetcoleman! Thanks for the input.

@davetcoleman

I’ve loved the merging of MoveIt code into few repos, and I’d really like to merge more. I think a great example of a single-repo ROS project is the new navigation2 project - its way more consolidated than MoveIt is.

I also like the navigation2 repository’s layout. I think it’s very clean and I prefer the same packages-at-the-top-level approach. However, one of problems we have is the sheer number of packages. While that number may go down (I hope), we are probably still going to have a lot. navigation2 has 20 packages. We have 131.

@davetcoleman

By splitting repositories, you’re going to reduce the directory hierarchy by likely just one level. Is this really worth it? Software projects get complex, but its not the repo layout that makes it easier.

Well that’s why we’re here. We have this proposal, we need to decide if it is worth doing.

@davetcoleman

I really don’t see how separating the code across the internet (different github repos) makes finding code easier for new contributors. I’d argue its the opposite.

This is true, although if they know they are looking for a perception algorithm then it is fairly obvious which repository to go for. However the same could be said for top-level directory divisions.

@davetcoleman

Even with split up repositories you probably should test it against the other repos every time also, to ensure the whole system builds. Otherwise you’re going to have to test it against the debians last time they were synced, which is every ~3 months? This means you can’t change the API of repos with each other because one of the other repos will always be out of sync.

I’m hoping to get nightly debians going with our own little package repository eventually which will allow us to test against recent versions of other repositories using binaries. Same for tags, so testing against the most recent release of each repository when the release is made rather than needing to wait for a sync of the OSRF ones. And relatedly, in general I currently think that the master of every repository should build against the master of the others. PRs that require changes in another repository should be coordinated with those other PRs to be merged as simultaneously as possible. The CI would get messy and manual in these cases but in theory the split we have chosen breaks things along black-box lines so such multi-repository changes should be rare. I would welcome evidence against this and do not consider it a watertight case.

@davetcoleman

This is a limitation of the bloom-release tools, not of single git repos. If you manually released debians the way most other Ubuntu packages do, this limitation wouldn’t exist.

Yes, another true point. But with 131 packages I really want to automate releases.

@davetcoleman

Also, do you really need different versions of software within the Autoware project? Isn’t it one large software project?

Not really. A large part of the project is algorithms and users do tend to pick and choose. We also have commercial entities wanting to build custom combinations of bits of Autoware and they may want the latest perception algorithms but stick with a set of control packages that they know work.

@davetcoleman

I don’t see how this difficulty really goes away with separate repos. Someone could still put unsafe code in the wrong places in either scenarios. Its always up to your PR reviewers to enforce this.

I was more referring to being able to say “this whole repository is safety-critical code”, as an example. But it’s not really relevant as the proposed split doesn’t really split along these lines anyway.

@davetcoleman

Side note: I have used some pretty cool Bazel rules to restrict access to certain parts of a code base from other parts.

I’d love to see those!