Map Library for Autoware.Auto

[mitsudome-r]

In the first Map Working Group Meeting, attendees(including myself) have agreed that Lanelet2 was only realistic solution for map library.

Discussion was very short and the consensus was that Lanelet2 is the only real contender .

As a result, Autoware.ai would start supporting Lanelet2 library from v1.13. However, now that we have more understanding about the library, I would like to discuss once more if we should also Lanelet2 library for Autoware.Auto.

Here are my positive and negative feedback about the library.
pros:

• map data structure is flexible. (Adding new tags is very simple)
• The library is organized into modules(packages) for different objectives (core, routing, io, etc.)
• Documents and sample code exists explain how to customize the library. (e.g. adding new parser, adding regulatory element, etc)
• It seems to cover minimal uses cases so far as we were able to replace Autoware.ai functionaity.(see also map functional requirements thread)

cons:

• Expression of road objects are geometry driven. i.e.) geometry is not attribute of road objects, but object type is attribute of geometry. You cannot tell the object class unless you look into tags.
• Since all attributes are defined as optional tags, developers must be aware of what tags are present.
• Some objects have alternative ways of expression its geometry. (e.g. trafficlights can be polygon or linestring) This makes code to be relatively complex.
• IMO there are excess abstraction done for the primitive objects. Linestrings3d has 5 inheritance for example. This makes harder to track down the underlying data when debugging.

Overall, I thought the library was easy to use once I get used to it if we want to just use their API. However, underlying implementation is quite complex, and it would be hard to track down the issue if we find a bug. Also, Lanelet2 library is owned by third-party organization(not AWF member) so we cannot make changes to upstream if we cannot persuade them. For example, we have a report that Lanelet2 cannot be built on arm architecture due to mrt-cmake-modules package which Lanelet2 depends on, but the author have no intention of removing the dependency.

Therefore, I think that we should create our own simpler library and maintain in the AWF gitlab repository, and I think we should be starting soon for Hackathon in April.
Any thoughts?

[msmcconnell]

I have been working with Lanelet2 over the past few weeks (including the pre-release autoware integration) and generally agree with the Pro/Con breakdown by @mitsudome-r. I have found the tagging mechanism needs more documentation/standardization for it to be used without the potential for error. One way to mitigate that may be for the AWF to define what set of tags are to be used in Autoware as well as create the TrafficRules objects for the US/UK/Japan as needed (Germany is the available default).

However, I think we should be careful about rushing too quickly into building our library from scratch. That was essentially the idea originally proposed for Autoware.ai here https://github.com/autowarefoundation/autoware/issues/1701
That received a large amount of push back from the community which wanted a standard format to be used.
Some points to consider

1. Would an AWF implemented library still use the lanelet2 conceptual format (lanelets/areas/regulatory elements etc.)? I would prefer that over switching to an alternative without a very good reason.
2. Would forking the Lanelet2 repo be sufficient for making the kinds of changes which might be needed?
3. The AWF probably represents the single largest user community which would want to work with Lanelet2. Can we use that to get the lanelet2 developers directly involved?
4. Regardless of the backend library, it would be nice if the interface used by downstream components could be largely preserved to ease porting of Autoware.ai components to Autoware.Auto. What is the practicality of doing that?
5. Are there any functional requirements missing from lanelet2 which cannot be added without reworking the library? Pretty much all of the “cons” listed are surface level issues which represent a challenge in learning the lanelet2 API but do not represent bugs or blockers on usage of the library.

[Ian_Colwell]

I just went through the process of manually creating a small lanelet2 map last week in JOSM. I then upgraded the map according to this doc to conform to the autoware lanelet2 extension (Great job @mitsudome-r on that doc btw).

Here is my feedback from the process:

• It was a little complicated to learn all the tags/relations and how to apply them in JOSM (took me maybe 4-6 hours to fully grasp it all during the process of map creation), but once you understand, it’s pretty straightforward. A walk-through guide explaining the map creation procedure would make this much simpler and faster. (I may create this guide if I have time, but no guarantees).
• Overall it was an easy experience since JOSM has so many existing features/plugins for map creations.
• I did not look into the library source code at all, so I can’t comment on the complexity of that.

Here is my take on some of the cons you list:

Expression of road objects are geometry driven. i.e.) geometry is not attribute of road objects, but object type is attribute of geometry. You cannot tell the object class unless you look into tags.

Can you elaborate on why that is a con? Does it make code more complicated, less efficient, etc?

Since all attributes are defined as optional tags, developers must be aware of what tags are present.

I don’t see this as an issue, there are validators that exist and can be extended to check for more things/tags.

Overall I agree with @msmcconnell’s hesitation to abandon lanelet2.
My suggestions would be to try to upstream as many improvements as we can back to lanelet2 with the support of the autoware community. If 20 github users from autoware comment on a feature request to make arm compilation easier, and open a PR to make it happen, I doubt they’d really want to decline it.

[kenji-miyake]

First, I post general comments on Lanelet2, and later, will write about concerns by @msmcconnell and @Ian_Colwell.

I think Lanelet2 is a good library as discussed in https://github.com/autowarefoundation/autoware/issues/1701.
I know and agree with this conclusion, but it doesn’t mean Lanelet2 the best.

It said just that Lanelet2 is better than Autoware Maps Format, and it was for Autoware.AI, NOT for Autoware.Auto.
We must accomplish the production level in Autoware.Auto, but I feel we can’t do it with Lanelet2(including Lanelet2 as backend).

Of course, we shouldn’t be in a hurry, we have to list up enough motivations to move to a new format.
However, I think we also need to consider for what reasons we continue to use Lanelet2.

I mean, we have to consider:

• What are the requirements for the map of Autoware.Auto?
• Does Lanelet2 meet the requirements, or can meet them at low cost?
• If not, how will we meet the requirements?

I believe this is the Maps WG’s mission.
Currently, I feel everybody is saying that “Lanelet2 looks good” just with their feelings, and that the core discussions are not taken place.
Let’s consider them more deeply together!

And as for a new format, I’d like to develop a prototype because we probably can’t judge good or bad without a concrete example.
I believe I can show you in a few weeks, so please give me feedback then.

It would be:

• GIS-based, which is more standard than OSM
• Easy to extend(probably than Lanelet2)
• OpenDRIVE subset(+ some extensions?)

and in the future, targeting:

• At least Lanelet2 equivalent libraries(e.g. geographic, routing)
• OpenDRIVE converter(at least, from OpenDRIVE)

[mitsudome-r]

@msmcconnell

Thank you for your feedback.

Would an AWF implemented library still use the lanelet2 conceptual format (lanelets/areas/regulatory elements etc.)? I would prefer that over switching to an alternative without a very good reason.
Would forking the Lanelet2 repo be sufficient for making the kinds of changes which might be needed?

I think the overall concept behind the library could be kept (like separating road objects layers and routing modules). Major change that I would like to have is slimming out implementation of core primitives. About forking, since the part I am concerned is core primitives of the library, we wouldn’t get much merits from forking.

The AWF probably represents the single largest user community which would want to work with Lanelet2. Can we use that to get the lanelet2 developers directly involved?

If we are going to use Lanelet2 for Autoware.auto, I think asking lanelet2 developers to join the WG would be nice.

Regardless of the backend library, it would be nice if the interface used by downstream components could be largely preserved to ease porting of Autoware.ai components to Autoware.Auto. What is the practicality of doing that?

I agree that ideally the interface of the library used by downstream should be independent of backend library. (That’s one of the reasons why we wanted to list up all the functional requirements we need.)
About porting the Autoware.ai components to Autoware.Auto, I don’t think we want to do that in first place. Many of the components(lane_rule, velocity_set, decision_maker) that use HD Maps have very bad implementation, and they need refactoring anyways.

Are there any functional requirements missing from lanelet2 which cannot be added without reworking the library? Pretty much all of the “cons” listed are surface level issues which represent a challenge in learning the lanelet2 API but do not represent bugs or blockers on usage of the library.

I believe all of the functional requirements we have seen so far is achievable using lanelet2. My major concern is the challenge to learn the core implementation behind the API. Maybe it’s only me, but I had hard time looking deep into them. (e.g. How they implement attributeMap? What’s the logic behind registration of new Regulatory Elements?)
It might be good to come up with non functional requirements first.

[mitsudome-r]

@Ian_Colwell
Thank you for your feedback.

Expression of road objects are geometry driven. i.e.) geometry is not attribute of road objects, but object type is attribute of geometry. You cannot tell the object class unless you look into tags.

Can you elaborate on why that is a con? Does it make code more complicated, less efficient, etc?

IMO, it was more intuitive to have geometry as an attribute of road objects not vice versa, and I thought this could be an entry barrier for most of the developers and map makers. But I guess having good guide could solve the problem.

Since all attributes are defined as optional tags, developers must be aware of what tags are present.

I don’t see this as an issue, there are validators that exist and can be extended to check for more things/tags.

Can we assume that all users of Autoware.Auto to run the validator? I guess we can automatically run validation at loader, but I still want to have less ambiguity when accessing to tags(AttributeMaps) of an object. Otherwise, we would have to check existence every time we access to tags.

If 20 github users from autoware comment on a feature request to make arm compilation easier, and open a PR to make it happen, I doubt they’d really want to decline it.

That maybe true, but we probably don’t want to do it every time we need a critical change to the library.

[msmcconnell]

Thanks for providing your responses. Here are some more thoughts.

Would an AWF implemented library still use the lanelet2 conceptual format (lanelets/areas/regulatory elements etc.)? I would prefer that over switching to an alternative without a very good reason.
Would forking the Lanelet2 repo be sufficient for making the kinds of changes which might be needed?

I think the overall concept behind the library could be kept (like separating road objects layers and routing modules). Major change that I would like to have is slimming out implementation of core primitives. About forking, since the part I am concerned is core primitives of the library, we wouldn’t get much merits from forking.

I am more concerned with how the road is represented then the library structure. Will we still divide lanes into lanelets which are built from 2 linestrings which are built from points following the lanelet2 standard. Or are you proposing that we rework that structure as well? It is important that we are clear when discussing road description standard, the file format, or the access library.

I believe all of the functional requirements we have seen so far is achievable using lanelet2. My major concern is the challenge to learn the core implementation behind the API.

I agree that the way the library is implemented is hard to follow (lots and lots of templates). Is this the reason you believe it should not be used in Autoware.Auto?

@kenji-miyake

It said just that Lanelet2 is better than Autoware Maps Format, and it was for Autoware.AI, NOT for Autoware.Auto.
We must accomplish the production level in Autoware.Auto, but I feel we can’t do it with Lanelet2(including Lanelet2 as backend).

Can you elaborate on why exactly Lanelet2 cannot be used? Is it because the library implementation is hard to follow?

GIS-based, which is more standard than OSM

OSM is GIS based as each point is described by lat/lon, so I am not sure what you mean by this point. OSM is also just the file format used to store lanelet2 maps and does not directly relate to how the lanelet2 road description standard is defined.

OpenDRIVE subset(+ some extensions?)

Are you suggesting we use the OpenDRIVE road description standard instead of lanelet2? That is a fundamentally different way of representing road geometry.

[mch]

Sorry if this is a late interference. This is an awesome discussion. I’ve read that with great curiosity.
I’d like to point out a couple of potential problems I see with lanelet2. I’m speaking from personal experience as we are using in-house made mapping system very similar to lanelet.
Lanelet is a geo-points based solution, meaning that map lines (road sides, road marking lines, stop lines e.t.c.) represented by a set of points (https://github.com/fzi-forschungszentrum-informatik/Lanelet2/blob/master/lanelet2_core/doc/LaneletPrimitives.md).
With this kind of representation performance of some basic map operations will not be satisfactory.
Let me give examples:

1. One of basic operations: Defining if a geo-point is within specific lanelet. This task boils down to the checking if the point is inside the polygon defined by points of linestrings forming lanelet. This is a standard algorithm and it requires O(N) operations where N is a number of points in polygon. So checking single point is O(N) and in typical driving situation such check happens hundreds of times per second which loads CPU badly. We certainly do some precomuptation/caching to mitigate the load but problem is still there.
2. Another basic operation: For a geo-point, find closest point on the linestring. In general case it requires iterating all the points inside the linestring, meaning O(N) and again this operation is executed very often while driving.

If I have task to redesign the system from scratch now. I’d seriously consider using vector(polynomial) representation of linestring instead of approximated breadcrumb points. It’s more complicated math for sure, but if done right, I believe it can give you O(1) for those cases I described.

Hope it make sense and can help on final solution.

[kenji-miyake]

Can you elaborate on why exactly Lanelet2 cannot be used? Is it because the library implementation is hard to follow?

Exactly.
It supports many use cases and well tested, so it has no problem as long as we use basic features.
However, once we start to use advanced features, to customize it, or to guarantee the code quality, it will be very hard without the maintainers’ help.
What if the Lanelet2 maintainers stop supporting?
If we continue to use Lanelet2 in Autoware.Auto, I think we must invite the maintainers to Autoware Foundation, at least.

OSM is GIS based as each point is described by lat/lon, so I am not sure what you mean by this point. OSM is also just the file format used to store lanelet2 maps and does not directly relate to how the lanelet2 road description standard is defined.

OSM can be said a kind of GIS, but its expression is quite different from the usual GIS formats. As @mitsudome-r says, OSM is geometry-driven, but GIS is feature-driven.
One problem of that is OSM requires special editors. Please see https://learnosm.org/en/osm-data/data-overview/

If we switch to GIS-base, we can use other editors like QGIS or ArcGIS, which I think are more useful than JOSM. Also, we can use many libraries, for example, GDAL, geopandas(shapely, fiona).

Are you suggesting we use the OpenDRIVE road description standard instead of lanelet2? That is a fundamentally different way of representing road geometry.

Partly yes, but I’m not saying that we should you the “reference path + offset” way.
As discussed in the Maps WG, we must support OpenDRIVE to integrate with simulators.
What I’d like to do is to create more general and AD-friendly data structures, and to develop a loader/converter from OpenDRIVE to the data structures.
Of course, it can be done with Lanelet2, but it needs to be extended(e.g. lane-connection because Lanenet2 re-calculates it in runtime).

[mitsudome-r]

I am more concerned with how the road is represented then the library structure. Will we still divide lanes into lanelets which are built from 2 linestrings which are built from points following the lanelet2 standard.

I don’t think we have to change the data structure that much, but apparently @mch has a comment on that. I haven’t done much investigation about computation time as he has done.

I agree that the way the library is implemented is hard to follow (lots and lots of templates). Is this the reason you believe it should not be used in Autoware.Auto?

Yes. I am not sure if we want to use the library that we can’t fully understand. If we notice something going wrong in the library, I’m not confident to track down the root cause.

[Hatem]

Hello everyone,
I just finished addition lanelet2 load/save functions to ASSURE map editing tool. feedback is strongly appreciated.

Practically using lanelet2 library is painful:

• It took me one week to get used to the deep layered interfaces and templates.
• There is multiple ways to access traffic objects. don’t know which is efficient and safe and which is not.
• There is a lot of hidden functionality.
• I wasted several hours facing a bug , when creating a new object with LineStrings3d with and id, it allocates huge amount of memory. (I need to formulate a bug and report it to lenelet2)

From the semantic point of view:

• Because the map editor internal map structure (OpenPlanner RoadNetwok map) is center line based. - which means lanes are represented as center line waypoints -, exact conversion is really difficult , what I did is approximation. Meaning:
  – modules that uses the map such as (planning, localization, … )) should follow the lanelet2 structure. and this is very strong assumption
  – Loading vector maps, OpenDRIVE and convert them to lanelet2 will face lots of problems, main issue will be the contextual representation difference.

From coordinates system point of view:

• lanelet2 need manual setting of the Origin lat/long from the GPS grid. even if we add a tag in the autoware extension, we can prevent people from creating .osm map using JSOM.

About keeping lanelet2 or creating a custom format for Autoware.Auto, I am in favor of creating our simple format that satisfy the current Autoware functionality efficiently, and develop a converter for each standard (lanelet2, OpenDRIVE, Vector map) in which we can be sure that the output map meets both functional/nonfunctional requirements for Autoware.ai and Autoware.auto.

I will try to summarize these points in tomorrow’s meeting.

[kenji-miyake]

Expression of road objects are geometry driven. i.e.) geometry is not attribute of road objects, but object type is attribute of geometry. You cannot tell the object class unless you look into tags.

Can you elaborate on why that is a con? Does it make code more complicated, less efficient, etc?

If it’s hidden in the IO layer, it’s not a big problem for application layers. However, it makes the loader unnecessarily complicated and inefficient.
Please see the source code. I think it’s a sophisticated C++ code, but I think it can’t be said as “practical”.

Since all attributes are defined as optional tags, developers must be aware of what tags are present.

I don’t see this as an issue, there are validators that exist and can be extended to check for more things/tags.

Which do you think is better? I prefer the latter.

1. Force every developer to use a validator
2. Prevent inputting invalid tags in the digitizing phase, and developers don’t need to care about simple mistakes

For example, if we use an Database-based way, we can realize the method 2.

[kenji-miyake]

Adding a complement to @mitsudome-r 's comment.

• Would an AWF implemented library still use the lanelet2 conceptual format (lanelets/areas/regulatory elements etc.)? I would prefer that over switching to an alternative without a very good reason.

As @mitsudome-r says, we can keep the overall concept.
However, as @Hatem says, if we use Lanelet2, applications depend on the lanelet2 structure, so we should avoid directly using Lanelet, RegulatoryElement, and so on.

We can remove dependencies on Lanelet2 data structures by defining APIs. (i.e. Using Lanelet2 as a backend)
However, I believe that creating a simple library from scratch is much easier and better than doing that.

• Would forking the Lanelet2 repo be sufficient for making the kinds of changes which might be needed?

Since it requires to change significant amount of lanelet2_core, I think forking is not a good idea.
It can’t be done without the maintainers’s help.

[kenji-miyake]

@Hatem Thank you very much for your great feedback!
I strongly agree with your thought, but could you explain these points a bit more detail?

• – Loading vector maps, OpenDRIVE and convert them to lanelet2 will face lots of problems, main issue will be the contextual representation difference.

Have you tried loading/converting OpenDRIVE?
If so, it would be very useful if you share the experience with us. (but in another thread?)

• lanelet2 need manual setting of the Origin lat/long from the GPS grid. even if we add a tag in the autoware extension, we can prevent people from creating .osm map using JSOM.

I’m sorry, but I couldn’t understand this part: “we can prevent people from creating .osm map using JSOM.”
What do you mean?

[Hatem]

@kenji-miyake
Sure, I want to share that experience.
currently I am trying to have different format of the same map , which can all work with Autoware.
for example:

• Moriyama vector map -> lanelet2.
• CARLA OpenDRIVE maps -> lanelet2.

Until now conversion is not correct due to:

• GPS coordinates conversion , and how that can align exactly with the PointCloud maps.
• Missing some basic information such as lane width (which is essential for the creation of lanelet)

I’m sorry, but I couldn’t understand this part: “we can prevent people from creating .osm map using JSOM.”
What do you mean?

sorry , I mean we can’t prevent people from creating .osm maps that don’t follow our standard ! and once they load it to Autoware they face all kind of troubles, and start complaining.

[kenji-miyake]

sorry , I mean we can’t prevent people from creating .osm maps that don’t follow our standard ! and once they load it to Autoware they face all kind of troubles, and start complaining.

Thank you, I understand.
I think we should regulate the digitizing process more strictly so that people can’t create wrong maps.
To do that, I propose moving to GIS-based way, which can limit the tag names and values(e.g. enum).

In addition to that, we need to check more complicated things,
for example, “whether adjacent lanes are aligned”.
I think this is the role of the validator.

[LiyouZhou]

The Arm build issue is fixed in https://github.com/KIT-MRT/mrt_cmake_modules and merged to master. The SHA will soon be propagated to autoware.ai.

[JWhitleyAStuff]

Thank you! Very much appreciated!