Packaging Python Virtualenvs in ROS Noetic

Posted by Drew Bednar on April 24, 2022
Tags: robotics, python

In this tutorial, we will learn how to use Python virtual environments in ROS Noetic packages by building a simple Flask application with catkin_virtualenv. We will cover:

  • Installation of catkin_virtualenv
  • Building a Flask application as a ROS package
  • Installing our package

The Problem

The solution for isolating a Python project from the system Python or other projects is to use a virtual environment. Virtualenvs allows us to vary the external dependencies between environments on the same system. The Robotic Operating System (ROS), being a "meta" operating system, is built conceptually on top of another distribution, typically a specific Ubuntu distro. It uses the operating system's package manager to fulfill its dependencies, which means ROS's Python support is provided through the system Python.

Unfortunately, ROS has no concept of specifying Python library versions, which means it could be nearly impossible to have a reproducible build. Managing dependencies manually in a system with pip is possible but can easily result in version conflicts and would mean any users of our package will also need to resolve dependencies out of band. Other developers ship their development environments as Docker containers. Thankfully, catkin_virtualenv can help the poor Python-dev package a Python virtualenv as part of their ROS package. We will use catkin_virtualenv in this article to build a simple Flask app using artifacts pulled from PyPi at build time.

A note on GPL Licenses

Before we start, I want to note that catkin_virtualenv is a GPL licensed library, but this does not automatically make packages built with catkin_virtualenv GPL. Please see this ROS Answers post for a more detailed analysis. In short, the required license of a collective work is a function of the licenses of its inputs. If we stick to packaging and importing code with permissive licenses like the MIT, we can use the MIT license for our collective work.

catkin_virtualenv

The catkin_virtualenv package adds CMake macros to the catkin build process that allow us to build a virtualenv and pip install Python dependencies from a remote repository. In essence, we can avoid rosdep and use specific versions of Python libraries in our package.

Installation Option 1: git clone

The catkin_virtualenv repo is not a ROS package at the top level, so the typical workflow of cloning the repo into a catkin workspace src/ will result in a build failure. We can achieve the same result by cloning the repository outside of our workspace and symlinking it into our target workspace src/ directory.

At the time of writing, the master branch for catkin_virtualenv is on commit id 24ab743. We can explicitly use this commit id for this tutorial. From outside of our catkin workspace, we will clone the repository without an implicit checkout, change directories, and explicitly checkout 24ab743.

git clone -n https://github.com/locusrobotics/catkin_virtualenv.git
cd catkin_virtualenv && git checkout 24ab743

Now we can symlink this package into our workspace. In my case, I cloned catkin_virtualenv into ~/experiments, and symlinked it to a catkin workspace in ~/experiments/ros_ws/src/.

ln -s ~/experiments/catkin_virtualenv/catkin_virtualenv/ ~/experiments/ros_ws/src/

Back in our workspace, we can now run catkin_make or catkin_make_isolated(See REP 134). I prefer the greater control provided by catkin_make_isolated. 

catkin_make_isolated

If this is successful, we should also check to ensure that our system meets the package's dependencies.

rosdep check catkin_virtualenv

Technically, we have not installed the package at this point, but it will be available under ./devel_isolated and usable developing our simple_flask package. I leave this as a development package to demonstrate that it is not needed for execution later.

Installation Option 2: Bloom Artifact

The Bloom-built artifact can be used if an older version is acceptable. At the time of writing, this artifact was built with version 0.6.1 of catkin_virtualenv. In the above git-based installation, the master branch is post 0.8.0 release. Your mileage may vary.

rosdep resolve catkin_virtualenv

#apt
ros-noetic-catkin-virtualenv

sudo apt install ros-noetic-catkin-virtualenv

This command will install catkin_virtualenv under/opt/ros/noetic/, so it will be globally available to all catkin workspaces on the system.

Our catkin_virtualenv Experiment

If we attempted to use the system package python3-flask in our ROS node, we'd find version 1.1.x as the most recent version. As a simple example, let's create a package that launches a Flask 2.x web application with catkin_virtualenv.

Starting from the catkin workspace src/ directory, create a new package with catkin_virtualenv as a dependency.

catkin_create_pkg simple_flask catkin_virtualenv
cd simple_flask

Now, create a pip requirements.txt file with our desired Flask version.

simple_flask/requirements.txt

flask==2.1.1

Next, we need to link this requirements file with the catkin build process. This task is accomplished by adding catkin_virtualenv to our package.xml and referencing our requirements.txt using the newly available pip_requirements section in the export tag.

simple_flask/package.xml

<?xml version="1.0"?>
<package format="2">
  <name>simple_flask</name>
  <version>0.1.0</version>
  <description>The simple_flask package</description>

  <maintainer email="drew@androiddrew.com">Drew Bednar</maintainer>

  <license>MIT</license>

  <buildtool_depend>catkin</buildtool_depend>
  <build_depend>catkin_virtualenv</build_depend>
  <build_export_depend>catkin_virtualenv</build_export_depend>

  <export>
      <pip_requirements>requirements.txt</pip_requirements>
  </export>
</package>

For our Flask app, we will create a single route app that displays the Flask version. We must include the #!/usr/bin/env python3 shebang line for our Python scripts. During the build, catkin_virtualenv will create wrapper scripts that will use the bundled virtualenv instead of the system Python libraries for execution.

simple_flask/nodes/flask_node.py

#!/usr/bin/env python3
from flask import Flask, __version__

app = Flask(__name__)

@app.route("/")
def index():
    return f"Flask version: {__version__}"

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

We need to modify the CMakeLists.txt to find our package's single node and bundle the virtualenv using the catkin_generate_virtualenv. See the additional CMake options section of the README.md for more options.

simple_flask/CMakeLists.txt

cmake_minimum_required(VERSION 3.0.2)
project(simple_flask)

find_package(catkin REQUIRED COMPONENTS
  catkin_virtualenv
)

catkin_generate_virtualenv(
    PYTHON_INTERPRETER python3
)

catkin_package()

include_directories(
 include
  ${catkin_INCLUDE_DIRS}
)

install(FILES requirements.txt
  DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION})

 catkin_install_python(PROGRAMS
   nodes/flask_node.py
   DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
 )

We are now ready to run catkin_make_isolated again in our workspace. This command will build our simple_flask package in the ./devel_isolated directory using the build of catkin_virtualenv from the previous step.

catkin_make_isolated

We can check and see that, indeed we now have a virtualenv associated with our simple_flask package and that it contains its own isolated sites-packages. 

ls ./devel_isolated/simple_flask/share/simple_flask/venv/lib/python3.8/site-packages

We can launch our node using rosrun. This will start the Flask app using the Werkzeug development server and bind to port 5000 of our host.

rosrun simple_flask flask_node.py

The above command was not preceded by starting roscore, but that's fine since this entire experiment is simply about packaging the Python dependencies.

Installing our new package

In this last step, we will prove that this package is installable in the same manner as other catkin-built packages. We will install it to a location outside of our catkin workspace and run the above rosrun command again.

mkdir ../demo-install
catkin_make_isolated --install --install-space ../demo-install
cd ../demo-install

Next, we will replace ROS_PACKAGE_PATH since it includes references to our workspace. Be sure to replace <your_username>.

unset ROS_PACKAGE_PATH
export ROS_PACKAGE_PATH=/home/<your_username>/experiments/demo-install/share:/opt/ros/noetic/share

All that's left now is to rosrun the our package again

rosrun simple_flask flask_node.py

Closing thoughts

We should now know how to build our own ROS nodes with catkin_virtualenv. I plan to expand my own projects to leverage the pip-tools functionality to create lock files for my Python dependencies. Additionally, I will be investigating how to build .deb archives for these packages.

In a follow on article, we will continue this journey by using catkin_virtualenv to build aiorospy as a dependency for driving a Moteus brushless DC motor controller in ROS. Finally, thank you Locus Robotics, Paul Bovbel, and contributors for making such a great open-source package.

Tags: robotics, python