What Makes a Good Sensor Package? + A List of ROS 2 Sensors

General
, ,
1

Hello ROS Community,

I am Akashleena, an intern at Intrinsic working with @katherine_scott. This summer I am writing a ROS whitepaper that informs sensor vendors on the best practices for developing a ROS 2 package. We want to better understand all of the aspects of writing a good sensor SDK and ROS 2 package. Our goal is to help sensor vendors design their SDK and ROS 2 package such that it is performant, easy to use, and readily available as part of a ROS distro.

We’ve got a few questions that we would like to pose to the ROS community, but first we wanted to share some of our recent work. As part of this project we analyzed many of the sensor packages that are available ROS for Humble. Since searching ROS Index still isn’t great, we’ve compiled a list of all the sensor packages available as binaries for ROS Humble using ROS Distro. Our hope is that you can use this list as a starting point for selecting sensor. To give you an idea about how we’re thinking this problem we also included a table that shows some of the criteria that we used to evaluate a couple sensor packages.

Table: Comparison of ROS Sensor Packages

Metrics Orbecc drivers ros2_avt_cameras lucid_vision_driver issac_ros_argus_camera sick_safevisionary_ros2
ROS 2 LTS Support ☒ :ballot_box_with_check: Humble Release ☒ ☒ :ballot_box_with_check: Humble Release
ROS 2 Rolling Support ☒ Not sure ☒ ☒ :ballot_box_with_check:
REP-144 Support Not sure :ballot_box_with_check: :ballot_box_with_check: Not Sure :ballot_box_with_check:
Vendoring the package (is SDK system package?) ☒ :ballot_box_with_check: ☒ :ballot_box_with_check: :ballot_box_with_check:
if SDK not a system package check for .so and .dll :ballot_box_with_check: ☒ ☒ :ballot_box_with_check: ☒
Supports cross-linux distribution (support tiers) Not sure Not sure ☒ Not sure Not sure
Uses Standard Messages :ballot_box_with_check: std_msgs/Header, vendors can use more ROS 2 std msgs covers a lot of proprietary messages which can be replaced with ROS std_msgs can’t find in repo :ballot_box_with_check: good use :ballot_box_with_check: good use of ROS standard messages
Includes README, buid script, and sample code. :ballot_box_with_check: :ballot_box_with_check: :ballot_box_with_check: :ballot_box_with_check: :ballot_box_with_check:
Provides API Docs :ballot_box_with_check: :ballot_box_with_check: can be improved ☒ good docs :ballot_box_with_check: ☒couldn’t find
Includes Tests (unit, integration, system) :ballot_box_with_check: :ballot_box_with_check: ☒ :ballot_box_with_check: :ballot_box_with_check:
Includes ROSBag for quick testing ☒ ☒ ☒ :ballot_box_with_check: here ☒
Includes Launch Files :ballot_box_with_check: :ballot_box_with_check: :ballot_box_with_check: :ballot_box_with_check: :ballot_box_with_check:
QoS Support :ballot_box_with_check: :ballot_box_with_check: :ballot_box_with_check: :ballot_box_with_check: :ballot_box_with_check:
Composable Nodes :ballot_box_with_check: :ballot_box_with_check: :ballot_box_with_check: :ballot_box_with_check: :ballot_box_with_check:
Gazebo Plugin / Model ☒ no sensor model ☒ no sensor model ☒ no sensor model ☒ no sensor model ☒ no sensor model
FOSS License :ballot_box_with_check: :ballot_box_with_check: :ballot_box_with_check: :ballot_box_with_check: :ballot_box_with_check:
Tags Release Artefacts :ballot_box_with_check: :ballot_box_with_check: ☒ :ballot_box_with_check: :ballot_box_with_check:
Is the repo well mantained :ballot_box_with_check: Not many issues, can’t judge Not many issues, can’t judge :ballot_box_with_check: Not many issues, can’t judge
Sensor calibration guide? :ballot_box_with_check: :ballot_box_with_check: ☒ :ballot_box_with_check: ☒
Sensor Diagnostics and Monitoring guide? can be improved by using diagnostics_msgs README has diagnostic guide, but can be improved README has some diganostics guide but needs improvement ☒needs work has good lifecycle documentation but can do better in diagnostics

Sensor Packages Available ROS 2 Humble

For your reference, here’s a list of ROS 2 drivers that we currently available as binary packages in Humble. Please note that this table was procedurally generated, if the package.xml did not include a description then no description is listed below. We’re providing this list as an easy reference for hardware packages that can be quicky and easily installed using binary ROS packages.

List of Sensor Packages

Depth Cameras
depthai-core

nerian_stereo_ros2

openni2_camera

sick_safevisionary_ros2

Conventional Camera Packages

camera_ros

gscam

rc_genicam_api

  • Description: GenICam/GigE Vision Convenience Layer.

    This package combines the Roboception convenience layer for images with the
    GenICam reference implementation and a GigE Vision transport layer. It is a
    self contained package that permits configuration and image streaming of
    GenICam / GigE Vision 2.0 compatible cameras like the Roboception rc_visard.

    This package also provides some tools that can be called from the command line
    for discovering cameras, changing their configuration and streaming images.
    Although the tools are meant to be useful when working in a shell or in a
    script, their main purpose is to serve as example on how to use the API for
    reading and setting parameters, streaming and synchronizing images.

    See LICENSE.md for licensing terms of the different parts.

  • Release URL: https://github.com/ros2-gbp/rc_genicam_api-release.git

  • Source URL: https://github.com/roboception/rc_genicam_api.git

usb_cam

ros2_v4l2_camera

avt_vimba_camera

openeb_vendor

rc_genicam_driver_ros2

stcamera_ros2

vimbax_ros2_driver

Non-Conventional cameras

mocap4r2

flir_camera_driver

metavision_driver

libcaer_driver

LIDAR

sick_safetyscanners2

sick_safetyscanners2

sick_safetyscanners2_interfaces

sick_safetyscanners_base

sick_scan_xd

hls_lfcd_lds_driver

LMS1xx

rplidar_ros

lsc_ros2_driver

ros2_ouster_drivers

urg_node

velodyne

List of Other Sensor Drivers

aandd_ekew_driver_py

bno055

ess_imu_driver2

fadecandy_ros

naoqi_driver2

novatel_gps_driver

novatel_oem7_driver

off_highway_sensor_drivers

phidgets_drivers

rt_usb_9axisimu_driver

septentrio_gnss_driver

ublox

ublox_dgnss

What do you think makes a good sensor package?

We would love to hear from the community what makes a great sensor package! We’ve come up with a list of questions and we would love it if the ROS community provided their feedback on these topics:

  • What do you look for in the architecture of a ROS package for a sensor? What makes for a “well-written” sensor package?
  • Sensor calibration and cross-callibration is still a bit of a “black-art” in ROS. What tools are using for sensor calibration? What features would you like to see in sensor packages to improve calibration and cross calibration?"
  • One thing we’ve noticed is that Gazebo support for sensors is often missing. Most sensors lack URDF or STL files that define sensor’s hardware geometry. Moreover, most vendors don’t provide a Gazebo sensor plugin. What steps do you take to to integrate simulated sensors into your robot model in Gazebo? What would you like to see sensor vendors provide?
  • How important is it for sensor vendors to provide support for Tier 2 and Tier 3 supported operating systems? Is anyone out there building robots on top of operating systems like RHEL and Debian? If you are using these host operating systems what would you like to see?
  • Are there any specific things you would like to sensor vendors provide in their documentation or tests? Are there any sensor packages that you particularly like?
  • ROS 1 supported nodelets which were included in launch file of the sensors and allowed multiple nodes to share the same process. How are you performing similar interprocess communication in ROS 2? What would your recommendations be for similar features in ROS 2?
  • In ROS 1, the SubscriberStatusCallback was a key element in image pipeline to coordinate the sequence of nodes from layering to rectification. ROS 2 lacks this callback requiring frequent checks to find active subscribers. How are you achieving similar results in ROS 2?
  • How are you using lifecycle nodes in ROS 2 to monitor the sensor states?
13 Likes
2

Great investigation!

One question: I’m surprised to not see RealSense on this list, as it’s the first thing that comes to mind. Does it fit different criteria than the packages you’re analyzing? It does have binaries, far as I can tell.

There are other popular RGBD cameras like this around. You already got the OAK-D cameras, and seems like the ZED and Astra cameras do not have binaries. But there may be others.

1 Like
3

In the SubscriptionOptionsBase you will find the event_callbacks, which contains the matched_callback. This will be called if someone connects / disconnects. Is this what you are looking for ?

1 Like
4

Clear separation of driver and ROS code. E.g. you don’t want the driver part to return some ROS messages etc. Minimal ROS node wrapping the driver.

From my point of view, this not be part of any sensor package, but something sensor agnostic. Therefore it should not be part of the sensor package.

Each sensor is a single lifecycle node. As long, as the node is in running state, we assume it is working as expected. The sensor nodes must implement internal means of sanity checks (e.g. timeouts if no new sensor readings arrive etc).

System wide, there is one process controlling and monitoring the node states. Its job is to start up the nodes in the correct order, and abort system startup if a node is not configuring correctly.
Its second job is to check if a node goes into error during runtime. In this case other nodes are selectively stopped / restarted, depending if the whole system can be recovered or not.

5

Perhaps non-functional, but what I look for in a driver is involvement of the OEM/mfg.

And not just hosting it on a GitHub organisation/Gitlab group with their name, but actual employees of the OEM involved in / responsible for the development.

Not just a code dump, but actively engaging with their customers who just happen to be using ROS to interface / integrate with their product(s) (as opposed to any of the other interfaces they happen to support).

4 Likes
6

Not only applicable to sensor packages but to ROS packaging in general:

  • I look for a proper separation between runtime (package-nameN) and devel packages (package-name-dev or package-name-devel)
  • Besides, a proper declaration of <exec_depend> and <build_depend> in their manifest.

Regarding sensor functionality without implementation details, I always think that an option for a pull-based subscription for sensors is a good idea. Then, the sensor returns all observations at every pull since the previous pull.

7

Two corrections:

  • openeb_vendor is a vendored SDK for a non-traditional camera (event based)
  • flir_camera_driver is a driver for a conventional frame-based camera
8

You don’t have this in Humble - but the connect/disconnect callback support was added in Iron and proper support was added to image_common.

9

I originally decided to develop ublx_dgnss after finding that I could not get high precision GPS data from my UBLOX F9P with any of the existing drivers.

Nor was I able to access some of the other UBX messages directly. Important to my use cases was having a highly accurate timestamp. I primarily was going to use ROS2, to capture data in ROS2 bags and then analyse it at a later date. So the timestamps were needed for Kalman filters. Suffice to say there were issues at that stage with ROS2 bags and I developed a different solution to record using Apache Arrow, writing out intermittent parquet files.

I also was looking at the existing sensor drivers and noted that the majority were a port straight from ROS1 and used COM ports. The code base was old.

So I decided to develop a new driver from scratch specifically for ROS2 which also leveraged USB as opposed to a plain old serial COM port.

Another design decision I made, was that I did not want the wait for the next sensor reading from the device, to block the driver. So I made sure to take advantage of multi-threading. The raw sensor data payload is put on an internal queue - with the exact timestamp it was captured. Another thread processes that queue and publishes the ROS2 messages with the exact time it was captured from the device… I know the timestamp is pretty accurate but there still may be a better way using data from the device itself to get it even more accurate.

One of the other existing drivers, did not capture timestamp data at all. Hence it just was not usable for the use cases I needed.

Further, I wanted to leverage parameters to be able to reconfigure the GPS device whilst attached, not just when it started. This was an interesting exercise in how to maintain the state of the device itself inside the ROS2 node and apply changes to the device upon receiving parameter updates. It works fairly well, but with the current UBLOX UBX standards, there is no way to determine when sending a batch of updates and receiving a NAK response, what failed besides rereading everything from the driver and trying again. The present solution is not perfect but works.

I havent implemented the managed lifecycle style node as I dont think it was available when I wrote the driver.

Also I developed two other drivers but didnt publish them as they used proprietary libraries
ROS2 Daheng Camera Driver
ROS2 Driver for Robosense Lidar

The Daheng Camera Driver uses parameters extensively to be able to reconfigure the camera. As these computer vision camera capture raw Bayer frame data with 8 or 10 bit data, I found that converting this data into jpegs etc was very inefficient for transport and would burn the CPUs out.

Also I ended up hardwiring the second camera in the stereo vision pair to fire from the first. The software triggering of both cameras was too inaccurate for stereo vision when moving.

So without all the parameter for the drivers, it would of taken me a lot longer to experiment to get the setup I was looking for. I eventually wanted to use these parameters to create an average of the exposure, gain and balance settings in both cameras and update both periodically so they were configured similarly over time.

In short a good driver to me, should have a really good grasp on the timestamp associated with capture of sensor data. Be reconfigurable without having to restart, have good debug logs for when first setting up and possible use the life cycle node capabilities.

Eventually I would also like them to publish consistent messages about their state to a monitoring node especially if something goes wrong.

I would also like to see the use of COM ports deprecated and better standard USB abstractions introduced into the ROS2 standard libraries.

Ohh yes its difficult for those actually writing the code to publish great documentation. So people developing the drivers, need help from others with that aspect.

Hope this helps you.

2 Likes
10

@horto brought an important point, that I forgot about.

Time stamping of the sensor data is crucial.
There are multiple levels of time stamping from my point of view :
Excellent :

  • Sensor is synchronized to the PC (ptp)
  • Sensor timestamps each reading.

Good:

  • Sensor is not synchronized
  • Sensor timestamps each reading
  • Driver knows the transmission delay, and tries to come up with a timestamp compensating for drift of the sensor clock

Not great:

  • Driver timestamps on receiving of the sensor data on the PC, but compensates for static transmission delay.

Bad:

  • Timestamp of receiving of the sensor data is used.
2 Likes
11

Let me make a case why “Bad” (Timestamp of receiving of the sensor data is used) may not be so bad.
Most sensors I’ve worked with do not support PTP, but many provide some sort of time stamp that is not synchronized with the host time (“sensor time”). Setting the header.stamp time to the host time when the sensor data was received, and passing the sensor time through in a separate field allows any down stream processing node full access to the original data so it can implement its own scheme to estimate the sensor drift and compensate for delays.
It certainly is much more convenient for the end user when the driver does this compensation, but there is a case to be made (in particular when collecting data sets) for not processing the sensor data (time stamps) in the driver any more than necessary.

1 Like
12

Your case is the “Good” case, the sensor timestamped its reading with some internal clock. By “Bad” I mean, the sensor does not do this at all, and you got only the receive time.

I would argue that, estimating the correct time of when the sensor reading was taken, normally needs a lot of implicit knowledge of the sensor. Therefore I see it as part of a sensor driver.

13

somewhat off-topic, but as an alternative to custom fields: you could publish a sensor_msgs/TimeReference for this. Set its time_ref field to the clock/timestamps of the sensor, use the source field to indicate which sensor was the source, and finally set header.stamp of both the original msg (containing the measurement) and the TimeReference to the same value. Now you can use message_filters to get the synced pair on the consumer side if/when desired.

3 Likes
14

It’s maybe not purely architectural, but I think it is important that the ROS sensor driver should expose all possible functionality. All settings and APIs should be exposed as parameters or services (or actions in some cases). That is part of the reason why I originally started ros2_v4l2_camera: at that time generic camera drivers like usb_cam only exposed a few standard controls likely to be there for each camera, but didn’t expose any more fancy controls that more advanced cameras have. So I made ros2_v4l2_camera query all controls rather than hardcode any of them. (Nowadays usb_cam does a similar thing)

I have also come across some OEM managed drivers that missed some controls, or had a good delay between supporting new firmware features in the ROS driver. So you end up with a less functional sensor than if you wouldn’t use ROS (assuming the functionality is exposed in an SDK or API in the first place). And PRs to add that functionality by the community can linger for a long time if they are not seen as a priority by the OEM, or they just haven’t put many resources on maintaining the driver. This all echos @gavanderhoorn’s point about the importance of OEM’s involvement.

Although this is a nice concept, if not done wisely, this can often result in a bunch of unnecessary inefficient copying of possibly large amounts of sensor data, from non-ROS data structures into ROS messages.

I believe another important goal for sensor drivers is to add as little delay between receiving and processing sensor data as possible. That could point to an actually tighter integration between the driver core and ROS.

In ROS 2 a driver can/should be a composable component, which supersedes the concept of nodelets, and use Intra-Process-Communication (with proper use of unique pointers to actually achieve zero-copy message passing). This can significantly reduce latency of the processing pipeline, helping towards the goal of low delay.

1 Like
15

Hi @Akashleena, I am not sure how you pick this list of sensor but for Ouster sensors the official drivers can be found ere:

ouster-ros

Thanks

1 Like
16

Hi @horto thank you so much for sharing your idea on timestamp synchronization.I loved the idea of internal queue as well as lifecycle style node. I will incorporate your pointers on my whitepaper and hope I am able to come up with a good documentation on camera calibration, mutli-sensor calibration

17

Hi Samahu,

Thanks for pointing out. I think I made some mistake while parsing the package.xml file for the humble distro

1 Like
18

Hi @gavanderhoorn ROS1 uses message_filters for synchronizing different sensor timestamps which works on ntp. Do you know an equivalent for ROS2?

19

message_filters should be available in ROS 2: index.ros.org/p/message_filters.

I’m not sure I understand this part of your comment.

20

Hi, wanted to expand on that. depthai-core is pure C++ package that can be used to write applications but it does not have ROS intergration. That comes with depthai-ros package which consists of depthai_bridge used convert from DAI to ROS message format, depthai_examples which showcase some simple applications that you can write with those converters, depthai_ros_driver which is a “proper” driver to our cameras which can be customized based on ROS parameter interface, by default it provides RGBD+Spatial Detection output, can be extended with plugins and even allows running NN’s on external sources (by subscribing to image topics).
In near future we are going to release V3 of depthai-core, which provides on host VIO and VSLAM based on open source libraries (Basalt and RTABMap respectively).
Also, we are currently gathering feedback on current integration so if anyone reading this has some, please feel free to reach out to me.

1 Like