CSV Data Extraction Tool for ROS bag files for Python, R, Matlab

Robotics and self-driving cars. Powered by the Robot Operating system ROS

So you’ve been using ROS to record data from a robot that you use? And you have the data in a rosbag file? And you’ve spent a while googling to find out how to extract images, data, imu readings, gps positions, etc. out of said rosbag file?

This post provides a tool to extract data to CSV format for a number of ROS message types. It was initially written for data analysis of messages in MATLAB, but applies to Python, R, SAS, Excel, or SPSS where you need it.

ROS (robot operating system) is a software system gaining popularity in robotics for control and automation.

CRUISE vehicle with SICK sensors for autonomous vehicle research

ROS records data in binary .bag files, or bagfiles for short. Getting data out of so-called bagfiles for analysis in MATLAB, Excel, or isn’t the easiest thing in the world. I’ve put together a small ROS package to extract data from ROS bag files and create CSV files for use in other applications.

** update 6th July 2015 – This code has now been added to Github at https://github.com/shanealynn/ros_csv_extraction/ **

Thus far, the data extraction tool is compatible with the following ROS message types:

  • sensor_msgs/Image
  • sensor_msgs/Imu
  • sensor_msgs/LaserScan
  • sensor_msgs/NavSatFix
  • gps_common/gpsVel
  • umrr_driver/radar_msg (this was a type used by the CRUISE vehicle (see below))

To install the data extraction tool, download the zip file, extract it somewhere on your ROS_PACKAGE_PATH, and run rosmake data_extraction before using.

The tool can be used in two different ways:

1.) Extract all compatible topics in a bag file

# Extract all message
rosrun data_extract extract_all.py -b <path_to_bag_file> -o <path_to_output_dir>

2.) Extract a single topic

# Extract a certain topic
rosrun data_extraction extract_topic.py -b <path_to_bag_file> -o <path_to_output_csv_file> -t <topic_name>

This program was created during a six month research proejct completed at the University of Technology Sydney on their CRUISE project. (CAS Research UTE for Intelligence, Safety, and Exploration), One of my main tasks was to transition the data collection software on the CRUISE vehicle to ROS, a popular Robot Operating System. I was working at the time under an Australian Endeavour award, researching with Dr. Sarath Kodagoda on pedestrian detection systems for autonomous vehicles using radar signals. We investigated a range of classification techniques (support vector machines, naive bayes classifiers, decision trees etc) to determine which obstacles were most likely pedestrians.

The idea was to use ROS as a data collection tool, syncing data from a number of SICK laser rangefinders, cameras, inertial measurement units, radar etc.

If you find it useful, let me know. If you have any additions / suggestions, let me know too.

Subscribe
Notify of

9 Comments
Inline Feedbacks
View all comments

Hi,
I ‘m using your code to extract laserscan, but I would like to extract other topics that are not in your list. Do I just need to add them to the allowedTopics list, or there is somthing else.
I would also like to know if each topic is stored in a single csv or there is a way to store them all together.
Thank you!

Great work! This is just what I was looking to find.
I have the same question as Atia. I’m brand new to ROS and want to get all my data out. Your script is great. I want to add more, custom, topics. Do you have any suggestions for how to do that? Right now I have an error about “no handlers could be found for logger ‘rosout’ “.

I am not receiving any file from the following command: rosrun data_extraction extract_all.py -b /home/daniel/data.bag -o /home/daniel

Also, my bag file has the topics /tf_P3AT (like an imu with xyzrpy) and /temperature (single point data).

Appreciate this post. Let me try it out.

Hi ,

thanks for the plugin. I have modified to extract data from geometry_msg/vector3 format. if you are ok with sharing in a public repo, i would like to share your code in github. Please let me know if you have any questions.