ROS Resources: Documentation | Support | Discussion Forum | Service Status | Q&A answers.ros.org

Vapor-master, a distributed high availability rosmaster


#1

Announcing vapor-master

Vapor-master is a drop in replacement for rosmaster enabling high availability ROS service discovery. Vapor removes the single point of failure fundamental to ROS1, enabling new options for achieving scale without sacrificing stability.

Vapor makes it possible to scale ROS 1 workflows to cloud scales, support hot-swap scenarios, and accomplish redundant handover. All without significantly modifying legacy code.

Have a look at our announcement blog post to get started using vapor-master today.

Checkout the code on github or download the Ubuntu Snap for rapid setup.


#2

Hi Alan,
Thank you, this seems like an interesting addition and idea for ROS1 users. I will try to test it soon!
I have a few questions that hopefully you can answer:

  • Is the vapor-master computing overhead noticeable?
  • When a new vapor-master is connected, does it have all topics, services, etc, available? or what is public for each vapor-master is customisable?
  • Security wise, are you adding any kind of extra authentication?
  • Does it work with ROS Kinectic? Are you planning on releasing the debian packages for it?

Cheers,
Martí


#3

Excellent questions:

  1. On some work loads we have seen higher than desired CPU hits. Specifically about 10% higher than rosmaster for certain rospy publishers. This is due in part to an upstream bug in rospy and a lack of adequate caching on the vapor side, you can track here. Typically most pronounced on small systems like raspi. Typically not noticeable on modern x86 systems.
  2. This is a great idea, a middleware/filter feature here similar to the approach in the server in rosbridge_suite would make a lot of sense for some use cases. We’d welcome a feature request so we can design it out together.
  3. Security was sadly a non-goal of this rev. We also have some tools at a different layer coming in the near future so stay tuned. We expected a desire to become SROS compliant, not presently on the road map on my side but we should explore it more. Fundamentally the security threat model is very similar to stock ROS 1 presently, with the exception of the securing the db connection via SSL and mongo user+token. Major concern from my side is that all security features must have excellent UX… poor UX in security is nearly as bad as no security. So lets continue discussion in issues and make sure we can deliver great UX.
  4. Vapor should happily run on kinetic, we just haven’t tested there. Would love feedback if you’ve tried it.

#4

Unfortunately for some reason it does not work with Kinetic Kame.

$ rosparam set test 1 

does set a parameter. Subsequently doing

$ rosparam get test 

… does return

ERROR: no param found at '/test'

And

$ rosparam list

crashes with the following error message on rosparam’s side:

Traceback (most recent call last):
  File "/opt/ros/kinetic/bin/rosparam", line 35, in <module>
    rosparam.yamlmain()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rosparam/__init__.py", line 630, in yamlmain
    _rosparam_cmd_list(argv)
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rosparam/__init__.py", line 566, in _rosparam_cmd_list
    print('\n'.join(list_params(ns)))
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rosparam/__init__.py", line 401, in list_params
    names.sort()
AttributeError: 'str' object has no attribute 'sort'

… and no error message on vapor’s side.

The original rosmaster works.

/edit: rosparam set apparently does work (comment corrected). The parameter is set in the database. Interestingly though I’ve got two parameters with the very same keyPath, one for each testing call of rosparam set test 1 and rosparam set /test 1.


#5

Thanks for the bug report!

We’ll look into it. Track our progress at https://github.com/roshub/vapor_master/issues/21


#6

Hi @HWiese1980

sorry about the lame bug, we’ve merged a fix here and have deployed to the snap store.

You can update your installation with:

snap refresh vapor-master

To prevent this type of bug in the future we’ll be adding protocol level integration tests. Check out issue 23