ROS Resources: Documentation | Support | Discussion Forum | Service Status | Q&A answers.ros.org

Robotics Distributed System based on Kubernetes

@yjkim046

What if those nodes are in different network? Is it supposed to work?

No, it is not.
if you have to do that, you could use VPN to connect network sites.
this is not our major use case but we tried to use OpenVPN site 2 site configuration to connect different LAN.

Would this mean WeaveNet is able to support UDP even if the underlying network doesn’t?

No sure about this, I’ve never tried something like that. but interested.

@tomoyafujita
It doesn’t work again :frowning: I can ping the talker node from the listener node and vice versa … but the listener cannot get messages. Have you encountered this kind of problem?

@yjkim046

so far we detected this issue, technically this happens physical machine, but it is likely to happen when we use docker container. As a quick work-around, can you try to use rmw_cyclonedds? we confirmed that this problem does not happen.

I really appreciate your sharing. ROS&K8s is future. I have one question.

If I want to use ROS2 to communicate different node in same pod on Kubernetes, Whay should I do? Should I use Weave Net? Sounds like I don’t need to use Weave Net.

Could you tell me how to get the instruction? I wan na try ROS2 on kubernetes.

@QLV5645-QLJ

If I want to use ROS2 to communicate different node in same pod on Kubernetes

i guess that to communicate different nodes in different container in the same pod.

Should I use Weave Net?

No, it is not necessary.
WeaveNet is one of the CNI(Container Network Interface) implementation.
you could use anything you want, also WeaveNet works fine.

at your point, Pod shares network interface and namespace, which means all of the containers( and any nodes in any containers) in the same pod can communicate with each other as localhost.

hope this helps, best

Thanks for your replying. This will help me a lot! I have one more question.

Replacing the network model by one of the CNI implementation in K8s can enable ROS2 communication. Right? So if I wan na use cloud(K8s) to run a turtlebot(edge device) through ROS2 communication, what should I do? Just replace the network model?

@QLV5645-QLJ

Replacing the network model by one of the CNI implementation in K8s can enable ROS2 communication. Right?

yes, that is WeaveNet. (targeting 100% emulation for layer 2)
if i answer on this precisely, that is DDS communication and what’s important is multicast group packets can be through.

what should I do? Just replace the network model?

yes, WeaveNet should be used, if you are using DDS(UDP multicast for participant discovery). but if you use P2P(not multicast) discovery or something, Flannel should be working.

Awesome! By configuring the P2P communication, I have already enable ros2 communication between different containers inside kubernetes. But still can not enable remote ros2 communication between two nodes, where one node is in container(inside k8s) and the other node is outside the k8s. This refer to the deployment on real turtlebot. Do you have any suggestion?

@QLV5645-QLJ

where one node is in container(inside k8s) and the other node is outside the k8s.

this other node is in the same cloud network? asking instances that run k8s and instance that runs other node can connect?

Not in the same network. Yes, instance that runs other node can connect. This node means a ros2 process. For example, I wan na use cloud to control turtlebot when the turtlebot is connnected to the cloud vpn.

I am not sure whta exactly your network configuration is, but you could try to deploy the pods with host network enable, then other node can access the pod since it is not running on CNI. host network sample is here.

@tomoyafujita Hi Tomoya, we tried OpenVPN to connect ROS2 nodes in different networks but found that it is not working because OpenVPN doesn’t support multicast by default. Can I ask how you resolved this issue?

@yjkim046

Can I ask how you resolved this issue?

Actually I have not tried to connect sites between edge and cloud via OpenVPN with ROS2 application. Sorry I do not have answer right now.

Maybe this here would be handy…?

This is a newer FastRTPS release (currently not in any ros distro by default -> so you need to build it yourself). It is designed for networks with no multicast support… So probably it would fit?

Please keep me updated on this. I’m very interested in this topic :slightly_smiling_face:

@flo @yjkim046

yes, that probably works. but it is server/client architecture. (cannot fail independently, of course this is dependent on use case.)

I was thinking that,

  1. fleet control is single frontend server for administration. (This is confirmed by federated network via Kubernetes.)
  2. edge cluster system to connect directly in edge site. (edge distributed system)
  3. brain is in cloud, edge devices have access to brain for intelligent process.

2 and 3 is different architecture, but co-exist. I wonder if 2 and 3 mixed and everything connected via ROS2 application layer. we could do that, but need to consider if this is suitable for the application.

I get your point. With this you get a single point of failure again (basically ROS1)… If the server is missing the whole system would go bonkers…

I believe, that the (main) server from the DDS Discovery would need to be always inside the robot. So you still have an autonomous “basic” robot that can be extended with multiple resources but maintain integrity during offline phases…
You would then need a clever discovery algorithm that matches/combines these resources… which brings us back to multicast… damn… :smiley:

Maybe @Jaime_Martin_Losa or one of his colleagues from eProsima has a smart idea regarding this?

2 Likes

Hi @flo,

You can set up several redundant discovery servers.

See here: https://fast-rtps.docs.eprosima.com/en/latest/use-cases.html#udpv4-redundancy-example

3 Likes

Hi @Jaime_Martin_Losa,

Thanks for the fast heads up. But I’m unsure if this is a 100% fit. The usecase would be that multiple discovery servers can be merged.

So we have i.e. 3 servers like this:

Server A: nodeA, nodeB (inside local robot)
Server B: nodeC (inside factory network)
Server C: nodeD (cloud network)

Server A is inside the robot. But B might be in the onsite edge network and C in the Cloud.
I ideally don’t want to have server C include all nodes from multiple robots. It should just add its functionality to the DDS for the robot that requests access. It should even be reached from multiple robots at the same time that do not see each other (fleetmanagement i.e.).
And also combination A+B, A+C and A+B+C should be possible.
At least, if I can wish me my perfect networking world… :smiley:

Is it possible to have discovery servers with different nodes inside them and (re-)connect to them at once without switching between them?
Redundancy is typically a 1:1 mirror of the same functionality as you expect the system to work like before when one server instance fails.

Hi @flo,

Right, it is possible, and it is the regular behavior. When you connect to two discovery servers, you get the union of the information stored in both.

2 Likes