Simulation software requirements

Autoware
1

Hello,

We would like to kickstart a discussion around simulation requirements for users of Autoware. While my team specifically is working on LGSVL Simulator, a more general discussion on simulation software needs and requirements for Autoware users can hopefully begin a useful conversation around how developers are currently using Autoware and what they are lacking. What do you need in a simulator for Autoware? What do you feel is missing with current solutions? What are some specific requirements in a simulator for your use case?

4 Likes
2

A few things off the top of my head that I think we need from one or more simulators. I say “one or more” because it’s likely that different simulators will be better at different things.

  • Deterministic execution. When executing deterministically, the same starting conditions should give exactly the same results.
  • Ability to control the simulation step (pause, step forwards a specific time, step forwards one loop of the control algorithms, run faster than 1:1 time, etc.).
  • Headless mode for running as part of a CI pipeline.
  • Modular framework (links to the above requirement, but broader). We need to be able to do things like plug in different pedestrian simulation algorithms, different sensor simulation models (even for the same sensor types), and so on.
  • Fault simulation in all parts of the car, from wheels to engine to sensors.
  • Sensor noise and interference simulation.
  • Simulation of different times of day and different weather conditions, including their impact on road conditions and thus the response of the car to control.
4 Likes
3

@bshin-lge we looked at this extensively couple of months back and we also talked to lots of simulation companies and none could really full-fill what we were asking for.

In short these would be our prioritized requests:

The entire, non-prioritized list would be as follows:

  • Computer hardware requirements for simulation should be listed
  • Interfaces (mocks) supported by simulation (CAN, ethernet, …)
  • Crisply defined software APIs
  • Support/integration for ROS. Here I would like to point to this discussion https://github.com/CPFL/Autoware/issues/1724#issuecomment-443079012 that we already had with LG guys and I felt that our opinion was not really taken seriously.
  • Fidelity of sensor models
    • E.g. do you accurately model different surface reflectivity/absorptivity, incidence angle, etc…
  • How to create our own sensor models?
  • Map formats supported
  • Maps that come with it?
  • How to include our own maps?
  • Precision/ real-time capabilities in general
  • Use for stress testing, e.g. an extremely high sensor load (>10 cameras, >10 lidars, etc) in realtime, precision/ real-time capabilities/guarantees during stress testing
  • Can simulation run in faster than real time?
  • How is the behavior of other cars governed in simulation?
    • Are there API’s/plugins that we can define to modify this behavior
  • How are traffic scenarios generated?
  • Is there any randomness in the simulations?
    • If yes, can we specify sources of randomness?
  • Is there collision detection?
  • Can simulation run in a headless state?
    • E.g. Can it run without a GUI?
  • Can we programmatically access the simulation state?
    • Or do we have to interact via filesystem (e.g. xml)
  • Are car dynamics simulated?
    • E.g. suspension, slip etc…
  • Can the simulator be integrated with the CI runner and does it output simulation/test results in e.g. junit format for test result validation?
4 Likes
4

Headless mode, logger (if the accident happens in the simulator, the logger logs this situation) and scenario mode is necessary for running simulation in CI.
I want to use simulator for CI.

5

CARLA hits several of the requirements that we need.

6

Here’s another one for when simulating light-based sensors: simulating things like sun glare behind a traffic light or a bright reflection from a glass pane into a camera for an instant.

7

These are all really good points. What are the specific use cases of simulators as it pertains to people working on Autoware right now? As with @Dejan_Pangercic’s post, I think it’s helpful from our perspective (the LGSVL Simulator team) at least to think about prioritization of features from users’ point of view. We know about some of the use cases of the Autoware Foundation, but it would be interesting to hear from any additional active Autoware developers and what their needs are for their specific use case. As @gbiggs entioned, maybe it will help more clearly define different simulators’ roles and their differences as well.

As for LGSVL Simulator specifically, I’ve addressed some of the feature requests above as they relate to our roadmap.

Currently, our immediate priorities are:

  • API implementation
  • New map generation/import

Some requests (hardware specification description, which exact sensor models are supported) are a documentation need, and we will continuously add to the documentation (https://www.lgsvlsimulator.com/docs).

Control/scripting capability of simulator:
We are currently working on implementing a Python API to run the simulator - this includes start/stop, run-time configuration, stepping through/running faster than real-time, controlling the environment, controlling ego/non-ego agents, state/sensor data retrieval.
This will let us more easily define or use standardized scenario descriptions.

Determinism:
LGSVL Simulator is based on Unity, and to my knowledge the PhysX Engine in Unity does guarantee the same simulation result when all inputs are the same. This is certainly a whole topic itself as to the degree of determinism that is required for use cases - the act of running on different machines (if in cloud, for example) can change behavior, and the extent to which it does and whether it’s acceptable is probably an important question.

Headless mode for CI:
We do plan to enable running the simulator in non-rendering mode.

Parallelizing simulation runs:
This is on our roadmap (we need to upgrade Unity version and implement multi-threading)

Behavior models of non-ego agents:
@Dejan_Pangercic Do you mean dynamics models of agents as well or higher-level behavior?
This can be enabled by the API (agents can have specified behavior such as following waypoints or randomness).
Currently by default, traffic vehicles follow the vector map, and pedestrians walk random paths.

Scenario database:
We are currently working on being able to support scenarios, and after this definitely plan to look at generating test suites of scenarios, integrating with existing scenario databases, etc.

8

@bshin-lge I’d like a plan for simulation support and usage to be put together. Every simulator has its strengths and weaknesses, and that is not going to change no matter how much one simulator tries. It’s not just a question of features but also of underlying technologies used, how features are implemented, etc. So we need to understand what each simulator is best at and where it can be best applied. For example, if I were making a demonstration video, I would probably use LG’s simulator. If I were doing simulated cameras, Cognata’s simulator produces images that are more photorealistic. If I were doing statistical data gathering of planning performance across a range of scenarios, I would also probably use Cognata’s because they have a good system for generating scenarios procedurally and deterministically, and executing them deterministically. LG’s simulator and Carla appear to have better physics simulation and so are good for control testing. Things like this need to be considered and each simulator developer needs to decide on what sort of usage they wants to focus on. Then we can decide how to apply each one.

2 Likes
9

@bshin-lge thanks for the reply first. To reply to above question first, it would make sense to have non-ego agents behave lawfully as well unlawfully (e.f. jay walking, crossing on red, etc.). So a higher level behavior indeed.

Further, do you for any of above features also have a timeline?

10

Hi,

I would like to propose CARLA to be the official AutoWare simulation platform. Let me explain why i think CARLA is the best option right now.

Let me start from the non-technical aspect:

  • Totally open source: code and assets are free to use for both non-commercial and commercial purposes

  • CARLA is sponsored by Intel, Toyota Research Institute, General Motors and KPIT, large corporations that help to keep CARLA aligned with the needs of the driving industry

  • In addition the CARLA challenge, is backed up by Waymo, Uber, AWS, Audi and AlphaDrive

  • CARLA has a strong community of users around the world within academia and industry

  • Multiple startups are building services around CARLA to provide advance behaviour models, simulation in the cloud, procedural generation of maps, etc. Our ecosystem is large and it keeps growing.

  • CARLA has a strong team: more than 20 engineers working full time in the platform
    We provide monthly releases with new features and bug fixes

From the perspective of technical features:

  • A modular server - multi-client architecture to distribute simulation load. Multiple clients, running in different nodes can connect to the same server. Feature releases will include a multi-server - multi-client architecture.

  • Flexible and extensible python API. CARLA itself is designed to be an API.

  • Everything can be controlled in CARLA using the API: maps, actors (vehicles, pedestrians), weather, traffic lights.

  • Full-stack simulation, from sensor simulation, planning simulation to control. Now we also expose an interface to the low-level physic parameter of vehicles, so that users can define frictions, mass, etc.

  • Standard mechanism to define maps and navigability based on OpenDrive

  • We now have a high-performance mode (sensor simulation off) that enables simulation at 800-1000 Hz. Hours of driving acquired in minutes

  • Sensor simulation: multiple cameras, LIDARs, GPS, IMU and others. Moreover, we have plans to extend these sensors with RADAR and more accurate sensors models

  • New logging and playback mode: It is possible now to log the entire simulation and resume it back at from any timestamp. This enables to try different stacks or variations of an algorithm (the simulation can be resumed and diverge from the log).

  • CARLA provides now a query engine to automatically find interesting events that happened during the simulation round. For instance, find collisions and then resume the simulation just at the interesting part.

  • Simulation can run synchronously or asynchronously. This simplifies prototyping and enables the use of a large number of sensors if the user is willing to give up real-time. Total control of the simulation step (tick).

  • We have created workflows and tools to simplify the generation of new maps. Now users can make use of RoadRunner to semi-automatically (and most recently totally automatically) create new maps. Those maps are now ready to ingest in CARLA.

  • The new 0.9.4 release can ingest maps on the fly without recompilation

  • We also have an engine for traffic simulation to generate interesting traffic scenarios based on templates and decision trees. This engine is being used for the CARLA challenge, creating pseudo-random traffic situations on the fly inspired by the NHTSA pre-crash typology.

  • CARLA supports ROS via the CARLA-ROS bridge

  • CARLA is integrated with AutoWare

  • CARLA is being integrated with Apollo

  • Headless mode. It works in docker too.

I hope this information helps.

Best regards,
German

2 Likes
11

Just to clarify, we won’t have a single “official” simulation platform. We intend to support as many as we can based on technical capability to use a simulator and resources to maintain a connection.

However, we will have a default simulator that will be used in tutorials and quick start guides, etc. My opinion is that licensing reasons make CARLA the best option here. @esteve has also made good arguments for why CARLA is technically the best option. As always, we are open to other opinions. In the long-run, what we want most is a healthy ecosystem of simulators that cover each other’s weaknesses.

12

Hi @gbiggs, we are all in the same page. That is what I meant.

Let us know how we can help.

Best regards,
German

13

We’ve been running CARLA to do comparison with LGSVL Simulator and there are few points worth mentioning:

  • CARLA performance was very poor, it was impossible to run complete Autoware or Apollo setup with CARLA in real-time
  • CARLA building and setup process was fairly complicated
  • CARLA uses Unreal engine which prohibits any commercial usage without registering all the revenues and paying 5% https://www.unrealengine.com/en-US/release Unreal keeps the right to perform audits to make sure agreement terms are followed. So it does not really matter if CARLA code is under MIT license.

I’m not sure having tutorials and quick-start guides with the simulator which is unable to run Autoware in real-time, difficult to setup and requires to pay 5% if you provide it to customers is a good idea.

BR
Dmitry Z.

2 Likes
14

@German_Ros I also agree with @gbiggs
We will not have “single” official simulator platform.
But, we want to integrate and add Documentation for some simulators in terms of usability.

15

@German_Ros I have some opinion to CARLA.
I try to integrate carla_bridge into Autoware and it was very very hard.

  1. Simulator is very heavy.
  2. Simulator is very very hard to build.
  3. Vehicle dynamics is strange.
  4. bridge software was not released as ROS package.
  5. Why bridge software depends PythonAPI source codes in CARLA simulator?
    Why do not you to distribute PythonAPI .egg file as binary?
16

In those reasons, I had to integrate carla bridge as Docker image, and I think it damages usability of carla and Autoware.
This is the feature branch for Carla Integration.

17

Hi,

Are you referring to the ROS-bridge or to our AutoWare integration?

  1. What do you mean by CARLA being very “heavy”?

  2. The simulator is hard to build given the current instructions or just takes many steps? I mean, are there points where you got blocked and therefore need attention or is it just that since it is a UE4 project it deviates from a common ROS-based compilation recipe?

  3. Vehicle dynamics is strange sounds like a very vague description. What problems are you having with vehicle dynamics? In fact the dynamics of the vehicles are configurable by the user through the API.

  4. Right, the bridge is not a ROS package yet. We can work on making this happen

  5. The PythinAPI is a super-set of the egg. There are some additional functionalities. But we remove this dependency.

18

Thanks for chiming in Germán. As @gbiggs said, we are not going to have an official simulator, but a set of recommended ones. It may happen that a given simulator is better suited for a specific scenario than others and it’ll be labeled as the default, very much like what ROS2 did with the DDS vendors. For example, ROS2 has no official DDS vendor, but there’s a default one that users can just assume will work well in most cases. Users are still free to use another DDS vendor if they want (e.g. they already have a license, their usecase is very specific, they want to run ROS2 on an unsupported OS/platform, etc.)

We just want to make it easier for our users to decide which simulator to use. In order to do that, we want to ensure that all the recommended simulators are well integrated with Autoware and with good performance, they all use open standards (e.g. Lanelet2, OpenDrive), have clear licensing that doesn’t restrict its use, etc.

On the more technical side, both @Dejan_Pangercic and @gbiggs have listed many of the features we’d like simulators to have. I’d say the highlights are being able to run faster than realtime, integration with CI (including scripting and headless) and a fairly realistic physics engine, but many others are also very important.

As I said, this is not a competition, so the more simulators that are well integrated with Autoware that our users can pick from, the better.

Having a clear, well-understood license that does not restrict the use is essential for all the simulators that we intend to collaborate with. I’m not a lawyer, but it seems that the Unreal license wouldn’t be a problem here, royalties are collected by Epic only if you sell the product that uses the Unreal engine, which doesn’t seem to be the case with Carla since it’s entirely opensource. Carla has a very common license for the code (MIT) and the assets (Creative Commons-BY). That doesn’t preclude anyone (including third parties) from taking Carla, repackaging it as a proprietary product and sell it, as long as they pay royalties to Epic, but that’s not a goal the Autoware Foundation has.

19

If we are going down the path discussing licensing issue, let’s at least get on the same page with definitions: Open-Source != Free (free as a freedom, not as a free beer). Here is an article about that matter The Free Software Definition - Wikipedia

@esteve Again Epic charges 5% from ALL gross revenues produced by products built with Unreal:

Example:

  • You sell Autoware to customer and include CARLA Simulation DEMO to prove it works.
    You need to pay 5% from the whole amount of the deal
  • You run Autoware and CARLA Simulator demo in the cloud and charge customers for getting access to Simulator.
    You need to pay 5% from the whole amount you are charging

Please read all the details below. Even if CARLA source code is MIT, CARLA can’t control usage of the simulator, it is very much restricted by Unreal Terms.

Generally, you are obligated to pay to Epic 5% of all gross revenue after the first $3,000 per game or application per calendar quarter, regardless of what company collects the revenue. For example, if your product earns $10 from sales on the App Store, the royalty due is $0.50 (5% of $10), even though you would receive roughly $7 from Apple after they deduct their distribution fee of roughly $3 (30% of $10).
Royalty payments are due 45 days after the close of each calendar quarter. Along with the payment, you must send a royalty report on a per-product basis. For more information, see here.

20

I didn’t mean to get into a debate about what Open Source means (I pasted a link to the definition of the Open Source Initiative The Open Source Definition – Open Source Initiative), but was just trying to reframe the discussion to give it a more positive and constructive spin to foster collaboration from Autoware with as many simulator teams as we can.

By the way, from the link you pasted:

Despite the philosophical differences between the free-software movement and the open-source-software movement, the official definitions of free software by the FSF and of open-source software by the OSI basically refer to the same software licences, with a few minor exceptions

The Open Source Definition was copied verbatim from the Debian Free Software Guidelines, the only difference between Free Software and Open Source, is that the former is politically and philosophically motivated, whereas the latter is more about pragmatism.

As for the Unreal license, it seems fairly clear (at least to me) that only when you’re selling your game or application (i.e. you’re getting revenue from selling licenses) the royalties clause applies.

But in any case, neither you or I are lawyers, so let’s better leave it to the experts and focus on how Autoware can collaborate with the many simulation teams out there.