Simulation software requirements

Hi,

I would like to propose CARLA to be the official AutoWare simulation platform. Let me explain why i think CARLA is the best option right now.

Let me start from the non-technical aspect:

  • Totally open source: code and assets are free to use for both non-commercial and commercial purposes

  • CARLA is sponsored by Intel, Toyota Research Institute, General Motors and KPIT, large corporations that help to keep CARLA aligned with the needs of the driving industry

  • In addition the CARLA challenge, is backed up by Waymo, Uber, AWS, Audi and AlphaDrive

  • CARLA has a strong community of users around the world within academia and industry

  • Multiple startups are building services around CARLA to provide advance behaviour models, simulation in the cloud, procedural generation of maps, etc. Our ecosystem is large and it keeps growing.

  • CARLA has a strong team: more than 20 engineers working full time in the platform
    We provide monthly releases with new features and bug fixes

From the perspective of technical features:

  • A modular server - multi-client architecture to distribute simulation load. Multiple clients, running in different nodes can connect to the same server. Feature releases will include a multi-server - multi-client architecture.

  • Flexible and extensible python API. CARLA itself is designed to be an API.

  • Everything can be controlled in CARLA using the API: maps, actors (vehicles, pedestrians), weather, traffic lights.

  • Full-stack simulation, from sensor simulation, planning simulation to control. Now we also expose an interface to the low-level physic parameter of vehicles, so that users can define frictions, mass, etc.

  • Standard mechanism to define maps and navigability based on OpenDrive

  • We now have a high-performance mode (sensor simulation off) that enables simulation at 800-1000 Hz. Hours of driving acquired in minutes

  • Sensor simulation: multiple cameras, LIDARs, GPS, IMU and others. Moreover, we have plans to extend these sensors with RADAR and more accurate sensors models

  • New logging and playback mode: It is possible now to log the entire simulation and resume it back at from any timestamp. This enables to try different stacks or variations of an algorithm (the simulation can be resumed and diverge from the log).

  • CARLA provides now a query engine to automatically find interesting events that happened during the simulation round. For instance, find collisions and then resume the simulation just at the interesting part.

  • Simulation can run synchronously or asynchronously. This simplifies prototyping and enables the use of a large number of sensors if the user is willing to give up real-time. Total control of the simulation step (tick).

  • We have created workflows and tools to simplify the generation of new maps. Now users can make use of RoadRunner to semi-automatically (and most recently totally automatically) create new maps. Those maps are now ready to ingest in CARLA.

  • The new 0.9.4 release can ingest maps on the fly without recompilation

  • We also have an engine for traffic simulation to generate interesting traffic scenarios based on templates and decision trees. This engine is being used for the CARLA challenge, creating pseudo-random traffic situations on the fly inspired by the NHTSA pre-crash typology.

  • CARLA supports ROS via the CARLA-ROS bridge

  • CARLA is integrated with AutoWare

  • CARLA is being integrated with Apollo

  • Headless mode. It works in docker too.

I hope this information helps.

Best regards,
German

2 Likes

Just to clarify, we won’t have a single “official” simulation platform. We intend to support as many as we can based on technical capability to use a simulator and resources to maintain a connection.

However, we will have a default simulator that will be used in tutorials and quick start guides, etc. My opinion is that licensing reasons make CARLA the best option here. @esteve has also made good arguments for why CARLA is technically the best option. As always, we are open to other opinions. In the long-run, what we want most is a healthy ecosystem of simulators that cover each other’s weaknesses.

Hi @gbiggs, we are all in the same page. That is what I meant.

Let us know how we can help.

Best regards,
German

We’ve been running CARLA to do comparison with LGSVL Simulator and there are few points worth mentioning:

  • CARLA performance was very poor, it was impossible to run complete Autoware or Apollo setup with CARLA in real-time
  • CARLA building and setup process was fairly complicated
  • CARLA uses Unreal engine which prohibits any commercial usage without registering all the revenues and paying 5% https://www.unrealengine.com/en-US/release Unreal keeps the right to perform audits to make sure agreement terms are followed. So it does not really matter if CARLA code is under MIT license.

I’m not sure having tutorials and quick-start guides with the simulator which is unable to run Autoware in real-time, difficult to setup and requires to pay 5% if you provide it to customers is a good idea.

BR
Dmitry Z.

2 Likes

@German_Ros I also agree with @gbiggs
We will not have “single” official simulator platform.
But, we want to integrate and add Documentation for some simulators in terms of usability.

@German_Ros I have some opinion to CARLA.
I try to integrate carla_bridge into Autoware and it was very very hard.

  1. Simulator is very heavy.
  2. Simulator is very very hard to build.
  3. Vehicle dynamics is strange.
  4. bridge software was not released as ROS package.
  5. Why bridge software depends PythonAPI source codes in CARLA simulator?
    Why do not you to distribute PythonAPI .egg file as binary?

In those reasons, I had to integrate carla bridge as Docker image, and I think it damages usability of carla and Autoware.
This is the feature branch for Carla Integration.

Hi,

Are you referring to the ROS-bridge or to our AutoWare integration?

  1. What do you mean by CARLA being very “heavy”?

  2. The simulator is hard to build given the current instructions or just takes many steps? I mean, are there points where you got blocked and therefore need attention or is it just that since it is a UE4 project it deviates from a common ROS-based compilation recipe?

  3. Vehicle dynamics is strange sounds like a very vague description. What problems are you having with vehicle dynamics? In fact the dynamics of the vehicles are configurable by the user through the API.

  4. Right, the bridge is not a ROS package yet. We can work on making this happen

  5. The PythinAPI is a super-set of the egg. There are some additional functionalities. But we remove this dependency.

Thanks for chiming in Germán. As @gbiggs said, we are not going to have an official simulator, but a set of recommended ones. It may happen that a given simulator is better suited for a specific scenario than others and it’ll be labeled as the default, very much like what ROS2 did with the DDS vendors. For example, ROS2 has no official DDS vendor, but there’s a default one that users can just assume will work well in most cases. Users are still free to use another DDS vendor if they want (e.g. they already have a license, their usecase is very specific, they want to run ROS2 on an unsupported OS/platform, etc.)

We just want to make it easier for our users to decide which simulator to use. In order to do that, we want to ensure that all the recommended simulators are well integrated with Autoware and with good performance, they all use open standards (e.g. Lanelet2, OpenDrive), have clear licensing that doesn’t restrict its use, etc.

On the more technical side, both @Dejan_Pangercic and @gbiggs have listed many of the features we’d like simulators to have. I’d say the highlights are being able to run faster than realtime, integration with CI (including scripting and headless) and a fairly realistic physics engine, but many others are also very important.

As I said, this is not a competition, so the more simulators that are well integrated with Autoware that our users can pick from, the better.

Having a clear, well-understood license that does not restrict the use is essential for all the simulators that we intend to collaborate with. I’m not a lawyer, but it seems that the Unreal license wouldn’t be a problem here, royalties are collected by Epic only if you sell the product that uses the Unreal engine, which doesn’t seem to be the case with Carla since it’s entirely opensource. Carla has a very common license for the code (MIT) and the assets (Creative Commons-BY). That doesn’t preclude anyone (including third parties) from taking Carla, repackaging it as a proprietary product and sell it, as long as they pay royalties to Epic, but that’s not a goal the Autoware Foundation has.

If we are going down the path discussing licensing issue, let’s at least get on the same page with definitions: Open-Source != Free (free as a freedom, not as a free beer). Here is an article about that matter The Free Software Definition - Wikipedia

@esteve Again Epic charges 5% from ALL gross revenues produced by products built with Unreal:

Example:

  • You sell Autoware to customer and include CARLA Simulation DEMO to prove it works.
    You need to pay 5% from the whole amount of the deal
  • You run Autoware and CARLA Simulator demo in the cloud and charge customers for getting access to Simulator.
    You need to pay 5% from the whole amount you are charging

Please read all the details below. Even if CARLA source code is MIT, CARLA can’t control usage of the simulator, it is very much restricted by Unreal Terms.

Generally, you are obligated to pay to Epic 5% of all gross revenue after the first $3,000 per game or application per calendar quarter, regardless of what company collects the revenue. For example, if your product earns $10 from sales on the App Store, the royalty due is $0.50 (5% of $10), even though you would receive roughly $7 from Apple after they deduct their distribution fee of roughly $3 (30% of $10).
Royalty payments are due 45 days after the close of each calendar quarter. Along with the payment, you must send a royalty report on a per-product basis. For more information, see here.

I didn’t mean to get into a debate about what Open Source means (I pasted a link to the definition of the Open Source Initiative The Open Source Definition – Open Source Initiative), but was just trying to reframe the discussion to give it a more positive and constructive spin to foster collaboration from Autoware with as many simulator teams as we can.

By the way, from the link you pasted:

Despite the philosophical differences between the free-software movement and the open-source-software movement, the official definitions of free software by the FSF and of open-source software by the OSI basically refer to the same software licences, with a few minor exceptions

The Open Source Definition was copied verbatim from the Debian Free Software Guidelines, the only difference between Free Software and Open Source, is that the former is politically and philosophically motivated, whereas the latter is more about pragmatism.

As for the Unreal license, it seems fairly clear (at least to me) that only when you’re selling your game or application (i.e. you’re getting revenue from selling licenses) the royalties clause applies.

But in any case, neither you or I are lawyers, so let’s better leave it to the experts and focus on how Autoware can collaborate with the many simulation teams out there.

Not only selling the game, but ANY revenue produced with Unreal Engine triggers royalties, where the product is included. I’m posting the whole piece of the FAQ here, to remove all misunderstanding.

What do I need to do when releasing a product?

You must notify Epic when you begin collecting revenue or ship your product; see here for more details.

If I release a commercial product, what royalties are due to Epic, and when?

Generally, you are obligated to pay to Epic 5% of all gross revenue after the first $3,000 per game or application per calendar quarter, regardless of what company collects the revenue. For example, if your product earns $10 from sales on the App Store, the royalty due is $0.50 (5% of $10), even though you would receive roughly $7 from Apple after they deduct their distribution fee of roughly $3 (30% of $10).
Royalty payments are due 45 days after the close of each calendar quarter. Along with the payment, you must send a royalty report on a per-product basis. For more information, see here.

What about downloadable content, in-app purchases, microtransactions, virtual currency redemption, and SUBSCRIPTION FEES, as well as in-app advertising and affiliate program revenue?

Revenue from these sources is included in the gross revenue calculation above.

Why does Epic think it’s fair to ask for a percentage of a developer’s product revenue?

Our aim is to provide powerful tools, a scalable and productive workflow, advanced features, and millions of lines of C++ source code that enables developers to achieve more than they would otherwise be able to, so that this structure works to everyone’s benefit.

In this business model, Epic succeeds only when developers succeed using UE4. Many of the industry’s leading developers and publishers have signed up to license the Unreal Engine with royalty-based terms over the years, and now this level of access is open to everyone. And, don’t forget, we continue to offer custom terms.

Do I need to report royalties forever?

No, you only need to report royalties when you are making more than $3,000 per quarter from your product. If your game no longer is being sold, or no longer makes that amount of money, no royalty reports are due.

What if my project requires custom licensing terms?

If you require terms that reduce or eliminate the 5% royalty in exchange for an upfront fee, or if you need custom legal terms or dedicated Epic support to help your team reduce risk or achieve specific goals, we’re here to help. See the custom licensing page for details.

What if my product is released through a publisher or distributor?

You’re free to release Unreal Engine products through a publisher or distributor, and the EULA gives you the right to sublicense the necessary parts of the Unreal Engine to them so they can release your game.
When negotiating terms with publishers, please keep in mind that the royalty remains 5% of the product’s gross revenue after the first $3,000 per game per calendar quarter from users. In this scenario, feel free to refer your publisher to Epic during discussions, as it may be advantageous to all if the publisher obtains a custom-negotiated, multi-product Unreal Engine license covering your product.

What if my project wins cash awards?

You do not have to pay royalties on award winnings.

What if my product obtains crowdfunding via Kickstarter or another source?

Royalties are due on revenue from Kickstarter or other crowdfunding sources when the revenue is actually attributable to your product. For example, if the user is required to purchase a particular funding package to obtain access (now or later) to your product, or if that package gives the buyer benefits within the product such as in-game items or virtual currency.
Here’s an example of what we mean by “attributable”: Assume you provide two tiers of offers, a signed poster for $20, and a signed poster plus game access for $50. No royalties are due on ancillary products like posters, so no royalty is due on the $20 tier. On the $50 tier, the user is paying for the poster with a $20 value, and that implies that the remaining $30 of value is attributable to the product. So, for each $50 tier sale, you’d pay a royalty of $1.50 (5% of $30).

Just based on that I suggest to remove CARLA simulator from the candidates list for Autoware.Auto in order to protect final users and OURSELVES of being trapped into unexpected royalties by Epic after sharing demo build with customer.

I agree there is no competition and we should include as many simulators as possible as far as it is not posessing a danger to Autoware.Auto end users.

1 Like

@esteve @German_Ros @zelenkovsky thanks for your replies but lets stop here discussing about licenses. Instead I suggest for AWF to hire a lawyer that can read Unreal, Unity and LG licenses and come back with the legal advice that would also include answers to all of the questions above. Do you agree?

Instead lets move back to the technical discussion and get to the bottom of questions like this: Simulation software requirements - #15 by hakuturu583.

In addition @German_Ros wrt your

Where is the integration code and some tutorials? Because I could find nothing in GitHub - autowarefoundation/autoware: Autoware - the world's leading open-source software project for autonomous driving or AutowareAuto / AutowareAuto · GitLab.

@Dejan_Pangercic: https://github.com/carla-simulator/carla-autoware

1.heavy means simulator calculation compare to other simulators such as LGSVL Simulator. I felt your simulator is too heavy when we use CARLA with GUI in Linux.

2.I try to build in Linux/Windows and it was very hard… So, I felt it is very hard to modify source code and contribute to your sim.

3.Sorry for my vague answer, but when I drive a vehicle in your sim, and I felt that the vehicle turns to much rapidly compare to real cars. I think friction calculation or something like that is strange.

4.It sounds good!! It improves CARLA usability.

5.Great

I forget to say a big problem in CARLA.
CARLA Lidar collision detection for the Ego Vehicle is a Box Collider.
I think it is not good for autonomous driving software development.
Lidar Data is very very important data for Autoware.

Please provide details of your testing method and the results you got for LGSVL and CARLA. Otherwise neither simulator will be able to validate your claims, find problems, and fix them.

Please be specific. What was hard? Where did you encounter difficulties? If you provide this information, the CARLA developers can improve their documentation and perhaps their build process. If you don’t, nothing can be improved.

Can you provide a video of this? And a video of another simulator to show how it compares.

As a side note, StreetDrone just opensourced their URDF model to be used in simulation in Gazebo.

https://streetdrone.com/open-source-simulation/

This is an example of the collaboration we’d like to foster between simulator teams, hardware providers and developers.

Here’s the pull request that integrates their model with Autoware:

Kudos to @Efimia_Panagiotaki and @Antonis_Skardasis!

2 Likes

Hi,

@zelenkovsky, where is your information coming from? May you be misinterpreting UE4 licensing? There are already people selling cloud services around CARLA and they are not paying any fees to Epics.

Have you checked with a group of industrial lawyers? We have had these conversations many times for many projects and partners. I think it is not as simple as reading through the license. You need the right legal background to interpret the license. Has your legal team looked into that recently?

Regarding the free software being free as in freedom we all agree :slight_smile:

In any case, the whole proposition value has always been “nobody tries to sell CARLA”. If a given party can sell a service or product done with/over CARLA without selling CARLA. In fact, many companies are doing that at the moment. This model seems to work.

So overall, I understand your concern, since we had it too before starting the project. However, after many discussions with different legal teams, we are confident this model works.

After several conversations with Epicgames, what it seems to be their near goal is to tap into this new sector of autonomous driving robotics. So, they are planning to find formulas to monetarize UE4 there, probably with new licenses or business models. But that is yet to be discovered.

Best regards,
German

For what it is worth, I decided not to consider using Unreal or Unity in rviz (when we were porting to ROS 2) partially due to the same license uncertainty. If it’s not licensed under something recognized by the OSI (https://opensource.org/), then it’s not sufficient for us. Now, if it’s optional that would be ok by me, and the ignition-rendering API may allow us to do something like that in the future (where something like Ogre3D or directly using OpenGL is the default, but Unreal is an alternative).

That sounds like they’d be moving towards a solution that is less acceptable for a top-to-bottom OSS stack, for those who care about that. Which was one of my concerns when initially evaluating it, that they were more likely to change their license to be further away from an OSI approved license rather to one that is approved. So you’d then be locked in to some degree into a proprietary solution.

2 Likes

I understand these concerns. To have a completely open source simulator that does not depend on any license deviating from MIT / BSD / etc. would be ideal. A dream come true.

From the CARLA project perspective we care about bringing state-of-the-art simulation to the community (both academia and industry). Here state-of-the-art means to enable a set of critical features identified by the driving community:

  • Realistic sensor simulation of: cameras, LIDARs, RADAR, IMUs and other yet-to-be-invented devices
  • Corner case discovery
  • Traffic simulation / Traffic scenarios based on AI
  • Automatic ingestion of maps and scenarios

The uses cases requiring these features are typically i) full-stack verification and ii) assisting in the R&D cycle of the perception stack and the planner.

In order to enable some of these features a state-of-the-art game engine, such as Unity or UE4, brings an important value. Cameras are progressively becoming more critical in AV stacks and decent/good real-time multi-camera simulation with a PBR workflow is required.

The only totally open-source alternative to these engines coming to my mind right now (for the purpose of real-time PBR rendering) would be the Eevee engine used by Blender: https://docs.blender.org/manual/en/dev/render/eevee/index.html

However, Eevee is not mature enough yet to be integrated in CARLA due to performance constraints in multiple-camera setups.

Long story short, the community needs state-of-the-art tools to make progress. Right now the largest open-source efforts to provide these simulation tools are CARLA and LGSVL and AirSim. All of them are using a game engine. We need to accept this or else go with an alternative that does not provide all the needed functionalities.

In the case of CARLA --the platform that I know best-- the only limitation we have at the moment is if people try to sell CARLA itself. In that case they would have to contact Epicgames. For practical purposes this doesn’t seem to be a limitation. There is no need to “sell” CARLA, you just use it and build on top.Toyota, GM, Valeo and others have not found this aspect to be limiting so far.

In the future, there will be new open source real-time PBR-based rendering engines and we will integrate CARLA to use those. But right now we are trying to be reasonable and provide solutions to existing Autonomous Driving problems.

2 Likes