[software] Releasing VINSEval, an evaluation framework for unified comparison of VINS algorithms

Did you ever look for a framework that can easily compare different visual-inertial estimators in a fair and statistically relevant fashion for your robotic application?

We are excited to announce VINSEval – An open source unified evaluation framework that allows, in a fully automated fashion, a reproducible analysis on consistency and robustness of different visual-inertial estimation methods with respect to a variety of specific environmental and sensor parameters, and parameter ranges.

Some of the highlights are:

  • Automated summary of the results in easy-to-read radar-charts for quick information-access with minimal user effort is generated for you

  • Different photo-realistic scenes, different parameter sweeps, and different evaluation parameters can easily be included in the fully automated process

  • Similarly, intrinsic sensor characteristics, camera types, rendering and extrinsic parameters, and use of real or synthetic data can easily be changed

  • Time delay and synchronized or asynchronous data generation can be selected

  • Output data alignment with ground truth is done for you automatically for every estimator in the comparison set

  • The robustness metric is based on averaged RMSE, the consistency metric on averaged NEES

  • Faster-than-real-time simulation allows for massive data generation for statistical relevant results

For the full set of features check out our ICRA21 paper, video, and code:

Best regards
Alessandro Fornasier, Martin Scheiber, Alexander Hardt-Stremayr, Roland Jung, and Stephan Weiss


Univ.-Prof. DI Dr. Stephan Weiss
Head of Institute of Smart System Technologies
Control of Networked Systems Group
University of Klagenfurt
Universitätsstraße 65-67
9020 Klagenfurt
Austria

YouTube Channel: www.youtube.com/channel/UCAlDLtj_iHDOSCFz2fzQxWw

3 Likes