FACSvatar (facial expression to avatar animation) ROS integration

Background

I currently have a project that combines OpenFace, Blender, Unity 3D and Python modules to transfer a person’s facial expression to an avatar: https://surafusoft.eu/facsvatar/

At the moment I’m using Crossbar.io as Messaging Broker, but I want to tap into the resourceful ROS community, and therefore want to make it ROS compatible.

Currently, the framework is just simply transferring the facial expression from user to avatar, but a future goal is to use the facial expression as input that can be analysed, and it’s the agent that generates the facial expressions. Think of Affective Computing.

Most of this framework is similar to Robotics projects, and the facial expression generation could even be used for robots that have advance facial ‘muscles’ and are FACS (Facial Action Coding Unit) Compatible. So for my framework to be useful to many people, I want to integrate it with ROS.

Purpose of framework

Transfer / analysis of facial expressions in an easy to run framework (even for non-ROS users), but offer advance customizability (for ROS users with Human-Agent Interaction goals).

  • Unity 3D integration for visualizing avatar’s facial expressions
  • Blender integration for creating animations out of facial data.
  • Python modules for transferring/converting facial data & Deep Learning

Probably use Docker for the modules to run cross-platform and preventing the need of installing dependencies: https://hub.docker.com/_/ros/

Requirements

  • Cross-platform (at least Linux (Ubuntu) & Windows, preferably Android)
  • Run out-of-the-box with minimal set-up, but still can be configured for everyone’s needs.
  • Rely on popular tools, to prevent going out-of-date fast.
  • Modular, only use the modules necessary for your project.

Implementation ideas

Idea 1

Develop main project in ROS. Put every module of the framework in a Docker container and use Docker compose to easily connect all the modules in an easy-to-run way.

Then use ROS-bridge to connect modules for visualization outside the docker container which runs cross-platform.

Advantages

  • Easy integration with other ROS modules

Disadvantages:

  • Main development in Ubuntu (Windows is run-only)
  • Hard to switch to another message broker, because fully dependent on ROS

Idea 2

Put Roscore and Ros-bridge in a docker container, so it can run cross-platform on both Ubuntu and Windows (or anywhere where Docker runs). Then only some boiler-plate code has to be added to a module to communicate with ROS in a connection class. The rest of the module doesn’t need any knowledge of how the messages reach their destination or where they come from. Possibly put modules in Docker containers for easy deployment and connect everything with Docker compose.

Example of ROS communicator over Rosbridge: http://www.pythonexample.com/snippet/python/rosbridge_clientpy_le-kang_python

Advantages:

  • Easy for non-ROS users to modify and run code.
  • Easy to swap out ROS for another message broker
  • No catkin_build for Python modules
  • Can support any language, as long as it can communicate with ROS bridge
  • Matches current implementation

Disadvantages:

  • Harder integration with ROS (but can still communicate with ROS projects)
  • Cannot release as ROS module
  • ? Double messaging?
  • ROS bridge has no libraries for Python (or most other languages)
  • Boilerplate code has no maintainer

Other solutions

I’m aware of ROS through the Windows Subsystem for Linux (WSL), however that breaks the requirement of “Run out-of-the-box”.

Preference

It seems [idea 2] fits my purpose the best, however I would like some feedback if you think this is feasible. Or maybe I overlooked some easier solution? Please let me know all your thoughts :slight_smile: