Vapor-master, a distributed high availability rosmaster

ROS Projects
1

Announcing vapor-master

Vapor-master is a drop in replacement for rosmaster enabling high availability ROS service discovery. Vapor removes the single point of failure fundamental to ROS1, enabling new options for achieving scale without sacrificing stability.

Vapor makes it possible to scale ROS 1 workflows to cloud scales, support hot-swap scenarios, and accomplish redundant handover. All without significantly modifying legacy code.

Have a look at our announcement blog post to get started using vapor-master today.

Checkout the code on github or download the Ubuntu Snap for rapid setup.

vapor%20scaling

5 Likes
2

Hi Alan,
Thank you, this seems like an interesting addition and idea for ROS1 users. I will try to test it soon!
I have a few questions that hopefully you can answer:

  • Is the vapor-master computing overhead noticeable?
  • When a new vapor-master is connected, does it have all topics, services, etc, available? or what is public for each vapor-master is customisable?
  • Security wise, are you adding any kind of extra authentication?
  • Does it work with ROS Kinectic? Are you planning on releasing the debian packages for it?

Cheers,
Martí

2 Likes
3

Excellent questions:

  1. On some work loads we have seen higher than desired CPU hits. Specifically about 10% higher than rosmaster for certain rospy publishers. This is due in part to an upstream bug in rospy and a lack of adequate caching on the vapor side, you can track here. Typically most pronounced on small systems like raspi. Typically not noticeable on modern x86 systems.
  2. This is a great idea, a middleware/filter feature here similar to the approach in the server in rosbridge_suite would make a lot of sense for some use cases. We’d welcome a feature request so we can design it out together.
  3. Security was sadly a non-goal of this rev. We also have some tools at a different layer coming in the near future so stay tuned. We expected a desire to become SROS compliant, not presently on the road map on my side but we should explore it more. Fundamentally the security threat model is very similar to stock ROS 1 presently, with the exception of the securing the db connection via SSL and mongo user+token. Major concern from my side is that all security features must have excellent UX… poor UX in security is nearly as bad as no security. So lets continue discussion in issues and make sure we can deliver great UX.
  4. Vapor should happily run on kinetic, we just haven’t tested there. Would love feedback if you’ve tried it.
1 Like
4

Unfortunately for some reason it does not work with Kinetic Kame.

$ rosparam set test 1

… does set a parameter. Subsequently doing

$ rosparam get test

… does return

ERROR: no param found at '/test'

And

$ rosparam list

crashes with the following error message on rosparam’s side:

Traceback (most recent call last):
  File "/opt/ros/kinetic/bin/rosparam", line 35, in <module>
    rosparam.yamlmain()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rosparam/__init__.py", line 630, in yamlmain
    _rosparam_cmd_list(argv)
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rosparam/__init__.py", line 566, in _rosparam_cmd_list
    print('\n'.join(list_params(ns)))
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rosparam/__init__.py", line 401, in list_params
    names.sort()
AttributeError: 'str' object has no attribute 'sort'

… and no error message on vapor’s side.

The original rosmaster works.

/edit: rosparam set apparently does work (comment corrected). The parameter is set in the database. Interestingly though I’ve got two parameters with the very same keyPath, one for each testing call of rosparam set test 1 and rosparam set /test 1.

1 Like
5

Thanks for the bug report!

We’ll look into it. Track our progress at https://github.com/roshub/vapor_master/issues/21

1 Like
6

Hi @HWiese1980

sorry about the lame bug, we’ve merged a fix here and have deployed to the snap store.

You can update your installation with:

snap refresh vapor-master

To prevent this type of bug in the future we’ll be adding protocol level integration tests. Check out issue 23

1 Like
7

A talk was delivered at the March ROS Seattle Meetup. The video can be viewed on youtube, see video description for slides.

8

Hi Alan,

Is there any tutorial out that can help me to set up a redundant vapor-master setup? My use case is mobile robots and I want to have two instances of vapor-master running in to provide continuous ros master availability. I have one powerful main computer and one single board computer in the robot. The intention is the primary master runs on the main computer and a second master runs on the SBC. Also is it possible to run multiple instances of vapor-master on the same computer?

Best regards