ROS Resources: Documentation | Support | Discussion Forum | Service Status | Q&A

Building a Tensorflow Object Detection and Localization ROS Package

Hey guys, longtime lurker, first time, you know.

I’ve been working on a project that involves custom building object D&L models and using them to find things in high-res streaming video. I have the model trainer pipeline done, and am building out the actual detection code using ROS (close to done). I’d like to turn this into a package for the community once it’s done, but one thing I’m not clear on is dependency management. Tensorflow is in rapid development, and some of its dependencies may break other dependencies (numpy, etc.). Also, it’s not a simple apt-get to install Tensorflow if you don’t have CuDNN etc. installed first. Does anyone have any recommendations on how to manage these dependencies for a package?


Cannot waiting for the release. Since you mentioned it also does localization, may I ask how can you get the localization information (x, y and z) from a streaming video? Does it contain point cloud or something?

I’m doing it on 2d images from a camera, with a depth sensor (tbd). No point clouds.

so it would be a RGB-D sensor and because most of the well known RGB-D sensors’ ROS wrapper provides registered point cloud data which also contains rgb field, I think you can also use point cloud data as the input.

Any url to the code, or to a repository? I think it is an amazing project.


Hi Drew! Glad to hear someone is working on this, it’s something that I’ve felt ROS has needed for a while (and some partial implementations are out there).

Most ROS pipelines that I’ve seen that depend on TensorFlow, CuDNN, etc. being installed simply provide full instructions for installing the dependencies in their repository. Unfortunately this makes it hard to install via apt-get, and it’s more complicated than cloning a git repository into your catkin workspace, but as long as the installation information is kept up to date, it’s not too bad to copy-paste commands into a terminal.

I’ve gotten tensorflow and caffe interfaced with ROS before, and I had to install everything manually. I found the tutorials on PyImageSearch to be quite helpful, maybe you could reference those or make your own version that’s more appropriate for your dependencies.

Also, I believe that tensorflow can be installed via pip install tensorflow even if you don’t have CUDA installed, it will just install the CPU version of TensorFlow. So you could ask users to install tensorflow via pip, and provide links on how to install the dependencies if your users want more speed.

Also, it would be great to get people to start standarizing vision pipelines so they are interoperable. May I suggest that you make your detector output messages from the vision_msgs package?

Paco - I’ll clear some time out next week to clean up the code and put it out on github. I’ll try to get it into a downloadable package for ROS too, if I can squeeze it in. I like to think there aren’t that many dependencies and it should work fine, but I know better than to blindly believe that :slight_smile:

Kukanani - Pip can install tensorflow with or without GPU (assuming CuDNN, etc.). If you’re doing other data sciencey stuff it can cause versioning / dependency issues by forcing some version upgrades (at least it’s happened to me in the past, not sure how much of that still remains). For a lot of applications, I couldn’t imagine not using GPU for Tensorflow. But then there’s things like mobilenet and the Raspberry Pi, so it could be the case. Right now I’m favoring throwing an error about Tensorflow not being installed and point out the options, than having Tensorflow-CPU or -GPU be a dependency.

Thanks for recommending vision-msgs - I’ll take a look. The current version of my code only returns pictures with bounding boxes, but returning probabilities / classifications / etc will be easy.

Ok, I cleaned up the code enough to put an alpha release out on Github. If you’re familiar with making your own object detection models in tensorflow, this should get you the rest of the way.

I have a todo list in the readme, and I suggst you look at it first just to get a sense for what’s going on. I’ll be around to answer any questions. Code improvements are always welcome.

Note that I couldn’t load my demo model, because github has a 100mb limit, so the code will fail when executed because there’s no valid .pb model in there (just a placeholder file). For researchers, I’ll try to get a working, sub-100mb model in there soon.

Once I get it a little cleaner I’ll make a new announcement of its availability to the community at large.


Hey Drew, it’s nice to read that some people care about doing things “the right way ™” :wink:

Have a look here: Announcing the tensorflow_ros package: the easy way to get to C++ Tensorflow API in ROS

Hi Drew,

I am also working in this topic but I did not use Tensorflow. I implement object segmentation and localization from scratch with Eigen and boost.
Demo video:
If you are interested, you can check out this project on GSoC: