I have written a ROS1←→GStreamer plugin that is now pending as a merge request in the gst-plugins-rs repo. It supports feeding ROS Image topics into gstreamer video pipelines and vice versa.
An example use case would be streaming a ros camera or debug image over a WebRTC to a browser, or using an IP camera or any other kind of video stream inside ROS.
I’d be happy on any feedback if you want to try it out, build instructions and a hello-world-style example are included in the README file in the merge request. It’s written in Rust, so you’ll need a Rust compiler to build.
Very cool! I’ve been using appsrcs for this so far and they have been working pretty well, but this is more elegant and probably more efficient, too.
You make it sounds so easy. Have you implemented this? Our webrtc-video capability is based on gstreamer, too, but I found it quite difficult to get it working well under real-world conditions, like poor connections and packet loss.
Yes, and yes. Unfortunately, the rest of the pipeline is somewhat integrated into the rest of our stack and has therefore not been part of this open source release.
If you want to go the WebRTC route to stream to a browser, you can use the excellent webrtcbin plugin from the same repo, they have a usage example in their README, and the package includes a websocket signalling server and a demo web GUI. It also handles things like congestion control, i.e. dynamically lowering the image quality when the connection gets slow. Unfortunately, the gstreamer version that comes with Ubuntu 20.04/noetic is a bit old, so for webrtcsink you’d need to build a newer gstreamer yourself for now (or run ROS on a newer OS). If you wanted to use this over Internet, you would also need a turn server (e.g. coturn or eturnal), so it works when users are behind a NAT/firewall.
Since gstreamer tends to support every format and protocol under the sun, you could also stream to twitch or live-encode video to an mp4 file.
It’s great to see increasingly modular support for GStreamer integration!
@jobafr how does this deal with differences between ROS versions? I was hesitant to try pushing upstream to GStreamer because the ROS API still drifts quite a bit between releases, and the transport layers have so many subtle differences. If you’ve found a way around that (or found it to be a non-issue), I’d love hear it.
@chfritz the appsrc nodes like gscam and opencv’s videoCapture are super robust and they let you take advantage of zero-copy messaging. Memory thrashing is the only problem with the gst element approach.
WebRTC in ROS took a bit of doing, my pipeline host node has a webrtc example configuration.
I’m not happy with how hard it is to build support for a new peer discovery service, or using it beyond a hello-world example.
Fixing the pipeline host modularity and adding zero-copy support are on the next milestone
how does this deal with differences between ROS versions? I was hesitant to try pushing upstream to GStreamer because the ROS API still drifts quite a bit between releases, and the transport layers have so many subtle differences. If you’ve found a way around that (or found it to be a non-issue), I’d love hear it.
It doesn’t; I use rosrust, which is its own implementation of the ROS1 protocols. The gstreamer plugin .so file does not even link against any ROS stuff, you only need ROS installed at compile time so it can read the message definition for sensor_msgs/Image. Maybe this is more of a ROS2 problem? ROS1 is usually compatible even between versions (e.g. you can have melodic and noetic talk to each other on the same network). Since there are no new versions planned after Noetic, I assume the API can be considered stable. If that turns out not to be the case, I think rosrust would be the best place to deal with that.
I was planning on exploring shared memory (shmsrc) as an alternative to appsrc, to reduce copy overhead. Does either of you have experience with that?
@jobafr : +1 on just speaking ROS at the byte level rather than using the roscpp or rospy stack. I’m doing the same with rosnodejs. The ability to deploy anywhere with just a folder of the required .msg files is quite liberating – no ROS installation or env required whatsoever.
