Introducing robo-gym: An Open Source Toolkit for Distributed Deep Reinforcement Learning on Real and Simulated Robots

Dear ROS Community,

Together with my colleagues, I am happy to introduce to you robo-gym: an open source toolkit for distributed reinforcement learning on real and simulated robots.

robo-gym provides a collection of reinforcement learning environments involving robotic tasks applicable in both simulation and real world robotics. Additionally, we provide the tools to facilitate the creation of new environments featuring different robots and sensors.

Main Features

  • OpenAI Gym interface for all the the environments
  • simulated and real robots interchangeability, which enables a seamless transfer from training in simulation to application on the real robot.
  • built-in distributed capabilities, which enable the use of distributed algorithms and distributed hardware
  • based only on open source software, which allows to develop applications on own hardware and without incurring in cloud services fees or software licensing costs
  • integration of 2 commercially available industrial robots: MiR 100, UR 10 (more to come)
  • the provided tasks have been successfully solved both in simulation and on the real robot using a DRL algorithm trained exclusively in the simulation environments

robo-gym simulated environments are built with Gazebo and use ROS controllers making it easy to expand the library of RL Environments with new robots and sensors.

Getting Started

You can find all the documentation directly in the GitHub repo, it should be quite easy to start and play around with the already existing environments.

We also included some basic information on how to integrate new environments, but if you are interested in integrating you own robot, sensor or task please reach out, we would be happy to support you with that!

How to contact us

If you encounter any issues with robo-gym the best way to contact us is to directly open a new issue on GitHub.

If you are interested in expanding the framework or start a collaboration please drop us an email at

Matteo Lucchi
matteo.lucchi@joanneum.at

Friedemann Zindler
friedemann.zindler@joanneum.at

12 Likes

Hi,

the toolkit looks pretty nice and I’m looking forward to more examples!
After looking through the repository and the paper, I still wonder why you did not mention the (http://wiki.ros.org/openai_ros)[OpenAI_ROS] package, which has been another ROS interface to OpenAI? As far as I understand, it does not provide distributed training, but it looks pretty extensive and similarly structured.
Since I am pretty new in this field, there might be other reasons for not using this package?

Thanks a lot, Christian

2 Likes

Hi Christian!

Thank you for the interest and your nice words!

You are right, we should have probably mentioned this package as well in our comparison, but I will try to explain the benefits of robo-gym over this package.

Premise: I haven’t tried out the OpenAI_ROS package by myself, I base my knowledge on the documentation provided with it, so if I say something wrong about it please correct me.

  1. To the best of my knowledge, there is no example or description on how to exploit the models trained with the OpenAI_ROS pkg with the real robots, so I cannot say whether this is possible or not. This is a very important thing for us, because we don’t only want to have something looking nice in simulation but we want to use the trained models on the real robots and we aim at actually deploying the trained models in a industrial production scenario.

  2. As you already saw, there is no integrated support for distributed training, and again this is really important for us. In our paper we used the D4PG algorithm training on 20 different simulations at a time and we got nice models after a couple of hours of training. Training a non distributed algorithm (e.g. DDPG) to solve the same task on a single robot simulation takes more than 24h.

With this I don’t want to criticize the OpenAI_ROS package, nor to say it is bad, it looks really nice and well structured, although in my opinion it falls into the category of frameworks that focus on the simulation world and lack that connection to real world. So at the end I would say it all boils down to what do you want to achieve with DRL and what is your final goal.

I hope this will help you :slight_smile:

Cheers, Matteo

2 Likes

Hi @matteolucchi
Thanks for sharing your amazing works with us. I have few questions:

  1. Can I change the environment and robot model and use my custom robot model and environment for training?
  2. I have a warthog UGV which is running in ROS kinetic. Is it possible to train the robot in simulation running in ROS melodic and then transfer the trained model for real robot in ROS kinetic?

Thank you

1 Like

Hi @Miftahur92 ! Thank you for your kind words!

Question 1
Yes you can definitely use your environment and your robot model, If you look around the Github closed issues there are some questions on how to do that.

It requires some work to integrate but basically you can look into how we did it for the UR and transfer it to your robot. If you decide to go for it I am happy to support you with that. If you get stuck or have specific questions the best is to ask open a new Issue on GitHub so that if other users have the same problem they can also benefit from the info.

Question 2
Yes this is definitely possible, the most important thing is that the real and simulated robot have the same type of interfaces. It is no issue if for instance the real robot uses different topics from the real but obviously you want to make sure that all the data that is available during simulation is also then available on the real robot.

I hope this answers your questions, if I wasn’t clear or if you have any other doubts please don’t hesitate to ask :slight_smile:

1 Like

Link to the v1.0.0 release post: Robo-gym v1.0.0 release with new environments and noetic support

1 Like

Hi @matteolucchi
Thanks for your reply.

Hey,

I want to train my own robot using RL for developing a path plan. Can Robo-GYM help me do that?

I faced lots of problems with OPENAI_ROS package since it doesnt work in Ros Noetic and python3.

Is there any package that will help create my own training environment for my robot

Computer Config: Ubuntu 20.04 , ROS Noetic, python3, Gazebo