i think port assignment is not something k8s does but kernel. containers in the pod share the virtual NIC in the system requested by kubelet(k8s agent), then the rest is taken care by system(kernel).
Agreed, let me explain this a bit better as I understand it. Let’s assume we have two talker containers T1 and T2 in the same pod. All networking is managed by the pod, and both containers share the pod’s network namespace. Assuming T1 starts up before T2, it grabs the pod’s port 7400 on all the pod’s interfaces–including loopback–for multicast discovery. It also grabs port 7411 for unicast traffic–talking on a topic.* Shortly afterwards T2 starts up, and being an exact copy of T1, it attempts to use the same ports in the same network namespace. So what happens?
In my experience, T2 neither sends nor receives networking packets on port 7400 since the port is already in use. ROS logs show T2 publishing data, but network traffic captured at T2, at T1 and at the K8s host, do not show any datagrams from T2. The Kubernetes documentation simply recommends against two containers using the same port:
… containers within a Pod can all reach each other’s ports on localhost. This also means that containers within a Pod must coordinate port usage.
Kubernetes recommends port management–using different ports for T1 and T2–for containers within a pod. So, for example, that would mean running the discovery server on port 7401 in T2. But in DDS-speak, that’s akin to putting T2 in a different domain and simply doesn’t work with ROS.
An alternative Kubernetes solution is to add a k8s service to the pod; the service can then use IPC to communicate with the containers. However, k8s services depend on load balancing / address translation which also does not work with RTPS either (at least in my experience).
So my recommendation is to stick with the Kubernetes design principles to only run a single process in a pod.
Hope this helps, please post if you’ve experience things a bit differently. I’m continuing to explore Kubernetes and have been working mostly with MicroK8s.
*This assumes domainID=0; all port assignments actually depend on domain ID, participant ID and other parameters–see the RTPS spec, para 9.6.1