Quick Intro

This article will provide you all you need to build docker containers across multiple platforms. With an ever-increasing need to support multiple different device types (especially in areas like IoT and robotics which we touch later), it’s important to utilize the systems we have to do that. Luckily. Docker has a built-in buildx command that helps us with this process.

This article shows us how to use Google Cloud Build to do this at scale.

The Configuration

Our goal is to build the images on Google Cloud build, but then push these images to Docker Hub rather than Artifact Registry. This involves creating a cloudbuild.yaml to define the build steps.

This is the configuration we used (be sure to change shade-prod to your GCP project id):

steps:
  - name: 'gcr.io/cloud-builders/docker'
    entrypoint: 'bash'
    args: ['-c', 'docker login --username=$$USERNAME --password=$$PASSWORD']
    secretEnv: ['USERNAME', 'PASSWORD']
  - name: gcr.io/cloud-builders/docker
    args:
      - run
      - '--privileged'
      - 'linuxkit/binfmt:v0.8'
    id: initialize-qemu
  - name: gcr.io/cloud-builders/docker
    args:
      - buildx
      - create
      - '--name'
      - buildxbuilder
    id: create-builder
  - name: gcr.io/cloud-builders/docker
    args:
      - buildx
      - use
      - buildxbuilder
    id: select-builder
  - name: 'gcr.io/cloud-builders/docker'
    entrypoint: 'bash'
    args: ['-c', 'docker buildx build --platform $_DOCKER_BUILDX_PLATFORMS -t $$USERNAME/<your-tag> . --push']
    secretEnv: ['USERNAME']
options:
  env:
    - DOCKER_CLI_EXPERIMENTAL=enabled
substitutions:
  _DOCKER_BUILDX_PLATFORMS: 'linux/amd64,linux/arm64,linux/arm/v7,linux/arm/v8'
availableSecrets:
  secretManager:
    - versionName: projects/<your-project-id>/secrets/DOCKER_PASSWORD_SECRET_NAME/versions/1
      env: 'PASSWORD'
    - versionName: projects/<your-project-id>/secrets/DOCKER_USERNAME_SECRET_NAME/versions/1
      env: 'USERNAME'

Run a yaml config with the command gclouds builds submit --config cloudbuild.yaml .

Explanation

1. The first thing to note is that DOCKER_CLI_EXPERIMENTAL=enabled is set as an environment variable. This allows google to use buildx.
2. The workflow first runs a privileged linuxkit/binfmt:v0.8 image. This is used to initialize qemu which is used for virtualizing the later images.
3. We then create a builder named buildxbuilder
4. This builder is used in the next step which builds for the platforms defined in _DOCKER_BUILDX_PLATFORMS. In this case linux/amd64,linux/arm64,linux/arm/v7,linux/arm/v8. This set of targets provides a wide range of compatibility. This step also pushes to docker hub.

Note: this YAML file assumes that you’re pushing to docker hub, which is why we pass in the secrets of the username and password that you need to pass to the google secrets manager.

You can optionally skip this step and push to google artifact registry similar to below.

Deploying to Google Artifact Registry

Alternatively, if you wanted to deploy to Google Artifact Registry, you can simply change the tag and let Cloud Build know of the images built.

steps:
  - name: gcr.io/cloud-builders/docker
    args:
      - run
      - '--privileged'
      - 'linuxkit/binfmt:v0.8'
    id: initialize-qemu
  - name: gcr.io/cloud-builders/docker
    args:
      - buildx
      - create
      - '--name'
      - buildxbuilder
    id: create-builder
  - name: gcr.io/cloud-builders/docker
    args:
      - buildx
      - use
      - buildxbuilder
    id: select-builder
  - name: 'gcr.io/cloud-builders/docker'
    entrypoint: 'bash'
    args: ['-c', 'docker buildx build --platform $_DOCKER_BUILDX_PLATFORMS -t ${_LOCATION}-docker.pkg.dev/$PROJECT_ID/${_REPOSITORY}/${_IMAGE} . --push']
images:
	- '${_LOCATION}-docker.pkg.dev/$PROJECT_ID/${_REPOSITORY}/${_IMAGE}'
options:
  env:
    - DOCKER_CLI_EXPERIMENTAL=enabled
substitutions:
  _DOCKER_BUILDX_PLATFORMS: 'linux/amd64,linux/arm64,linux/arm/v7,linux/arm/v8'

And then simply call it in the terminal like so

gcloud builds submit --config=cloudbuild.yaml \
  --substitutions=_LOCATION="us-east1",_REPOSITORY="my-repo",_IMAGE="my-image" .

The Importance of ARM in Robotics and Modern Software

There are two schools of thought when it comes to CPU: you have the ARM architecture CPUs, and you have the x86 architecture CPUS. The core difference relies on how they operate machine instruction. ARM-based CPUs only rely on registers to handle operations while x86 can utilize memory and ALU units. Though no single type of architecture is better than the other, they do lead to major incompatibility issues when it comes to building modern software.

Recently, this problem has only exponentiated with more and more flavors or ARM being built out with their own configurations (even the modern M1 Mac that is built on ARM has its own flavor), leading developers and maintainers to ensure that their code truly runs on every container.

Now in the Robotics space, ARM is not something that’s really new. The Jetson family of boards all run on ARM based processors. The Arduino family also follows suit. With that in mind, learning how to build your code, more specifically docker containers, to support both ARM and x86 has become incredibly important to support as many people as possible.

Docker’s official stance on ARM and x86 is that it will “try to do its best to emulate” ARM if you ONLY build for x86. Unfortunately, this rarely cuts it. For example, attempting to run any ROS docker containers that were built on x86 architecture almost never works on ARM (even trying to get two containers to work on the same computer won’t work, let alone on a network).

Though ROS does offer its ros_cross_compile functionality, this article will focus on creating a build system that can work in production using Google Cloud Build. Google Cloud Build allows you to parallelize builds for multiple architectures and store the artifacts in an online artifact registry, docker hub, or a private registry. Utilizing cloud resources, we can build our docker containers faster and ensure that they work on any version or device.

This article will go more into depth on how to get Google Cloud Build set up to build on both x86 and ARM and provide an example configuration that you can go ahead and use for any of your ROS builds.

Why Use Docker for Robotics?

Avoiding Colcon Build Hell

Now some may wonder why use Docker for the robotics use case, specifically with ROS 2? Isn’t it better to try to install everything in the same workspace and follow the traditional paradigm of creating various packages, running colcon build , and compiling it directly from the source? The answer is yes and no. Yes….if it was a perfect world. In reality, ROS 2 oftentimes fails to properly link up different dependencies, leading to a dependency hell. The common failure is that one package relies on this specific version of this specific C++ file and this other package relies on another version. This ultimately leads to more and more errors that simply are not related to your control code, or your perception code at all. Indeed, this is simply a version of “it works on my computer but not yours”, which is where docker truly thrives.

Using docker containers enables us to follow a “separation of concerns” pattern that allows different pieces of code to not have to rely on the same dependencies. It removes the issues of dependency hell at the expense of a little bit more storage usage (of which can be optimized via thing such as multi-stage builds, only compiling what you need). Docker also enables the sharing and integration of code into an existing system much faster. Now, all we have to do is start up a docker container and its integrated directly into the ROS2 ecosystem.

Docker for amd64, arm64 or armv7 using Cloud Build was originally published in Shade Robotics on Medium, where people are continuing the conversation by highlighting and responding to this story.

Building a robotic product, from concept to manufacturing and release, is a very vast, involved process that requires the synergy of multiple moving parts. A crucial component is the development of production-grade robotic software that will ultimately be deployed on a fleet of robots and ingesting petabytes of data. Productizing a ROS 2 application can be a tricky course to navigate as there is a multitude of factors that define a system as “production-ready”. This article serves as a guide and discussion of the tools and workflows provided by the ROS Tooling Working Group that can be used to help you build, test, release, monitor, and deploy a ROS 2 package on a production robot: the modern RobotOps stack.

Adapting ROS 2 Workflow to ARM CPUs

Today, while most developer workstations run on x86–64 processors, their robots may use a different architecture. Specifically ARM has been popular for platforms upon which robots are built such as Raspberry Pi, Nvidia Jetsons, and more because of its reduced chip complexity which makes it more cost and power-efficient. In fact, over 70% of IoT and robotic devices on the market run on ARM instead of x86. As a result, when we’re building a ROS package, we need to be thinking about how to develop for ARM devices, therefore we must modify our development workflow to run ROS 2 on an ARM computer which a robot is built on.

Many developers just starting out approach this task by treating their robot’s computer as a smaller version of their workspace, building a peripheral workspace right on the robot. While this approach is suitable for early stage development and simulation based workflows and works well with testing code in the workstation, there are a few reasons why building on the target device is a flawed approach. First, computing systems may not always have all the physical access you need — ports needed to plug in peripherals can be hard to access or missing altogether because they weren’t intended to be used in this way, and if they are it can cause mechanical strain and shorten the robot’s lifetime. This approach is also not scale friendly as you can only deploy builds to one robot at a time (I mean seriously, imagine trying to deploy some code across a thousand robots manually). Further, there are storage, safety and performance concerns. The disk space used up by installing build tools and code on the robot could probably better used for things that actually add benefit. Each additional package installed on the robot increases its security vulnerabilities and it can be very dangerous when a hacker is able to access your entire code base rather than just its binary packages (we’ll touch on this later). Finally, many embedded systems don’t have nearly enough RAM to run the build process so it would be impossible to run it on these target machines. ARM processors are also not as fast as the x86 processors available in developer workstations or in the cloud.

This brings us to the solutions of cross-compilation, emulation, or containerization. These can be difficult to set up and maintain within a complex codebase, so The ROS Tooling Working Group provides a tool, cross_compile, which can build ROS 2 workspaces for both ARM and x86–64 with minimal set up. Cross-compilation means using source code to build binaries for a CPU architecture that is different than the computer where you run the build. In this case we’re talking about building ARM binaries on an x86 computer. Similarly, emulation is the reproduction of the function or action of a different computer, software system, etc. By building on the x86 machines you save a ton of space on your robot from build tools and source code, decrease security vulnerabilities, and save time on every change you make to your code which could add up to months or even years saved over the course of a robot’s development cycle. Finally, containerization, which is the most streamlined approach, offers both the ability to do CrossFit compilation and emulation without the issues of impacting your base OS on either systems. By containerizing specific systems and structures, security can be isolated to each container process rather than compromise the entire robot. The most common way to do containerization is via Docker. Docker provides a super easy way of declaratively defining the build strategy. This combined with Google Cloud Build’s buildx system enables us to actually cross compile our containers for any system. Check out our article here on setting cross compilation up with docker along with GPU acceleration of docker if you’re running on a Jetson or CUDA supported device.

Tuning ROS 2 Applications for Performance Using QoS

Supervising and managing the performance of your ROS 2 applications ensures proper function and promotes reliability of your system. A third dimension of the topics in the ROS publisher-subscriber system, Quality of Service (QoS), which is new for ROS 2 provides us the ability to achieve this. Traditionally, ROS built out their own messaging system under the hood which led to many security risks and almost no ways to customize QoS and typing. ROS 2 now sits on top of DDS, which is the standard messaging protocol across industry with built in QoS.

QoS defines extra promises about the behavior of your publishers and subscribers on topics and has a variety of settings and policies. Before diving further, take a moment to familiarize yourself with these QoS components:

• QoS Policy — an individual QoS type or setting
• QoS Profile — a complete group of all policies
• QoS Offer — publishers offer a QoS profile — this offer is the maximum quality that the publisher promises to provide
• QoS Request — subscribers request a QoS profile — this request is the minimum quality that the sub is willing to accept
• Compatibility — QoS policies have compatibility rules. If an offered policy has a matching or higher quality than the request then the publisher and subscriber can be connected. A publisher can match and pass messages to multiple subscribers with different requested QoS as long as each request is compatible with the offer. Some policies affect compatibility while others don’t: reliability, durability, deadline, and liveliness do affect whether a pub and sub are compatible while history, depth, and lifespan do not as they are purely local configurations.

Developers will primarily interact with QoS in their application code. All publishers and subscribers must specify a QoS profile on creation. Using the ros2 topic utility, you can get various information about QoS of topics by running ros2 topic info —verbose, (use ros2 topic —help to see more available options). Additionally, the rosbag2 utility uses QoS to intelligently record and play back messages from a ROS 2 system. This data can then be visualized and used in a CI/CD stack for testing, for data ingestion, and machine learning.

A hazard of the extra promises from QoS is that they can be broken. This is when the concept of Events comes into play — events are generated when there is a change detected in quality of service that breaks policy, which triggers callbacks to be able to handle these cases. ROS 2 defines preset QoS profiles for common cases and these are a good place to start when tuning the QoS of a topic, but it is necessary to go beyond them to deal with more advanced behaviors. In general it is good practice to consider policies for every publisher and subscriber. Below we list and describe some of the policies you should definitely take into consideration to improve the robustness of your robotic application.

• Incompatibility Callbacks — it is useful to have a mechanism defined to know when we encounter a mismatched situation (i.e. QoS profiles between pub and sub don’t match). Then for example, if an application is run where a subscriber asks for a Reliability of X but the publisher only offers Reliability Y (with X being of higher reliability than Y), then compatibility callbacks are triggered for both the publisher and subscriber.
• Lifespan — we can use the Lifespan QoS policy to ensure message freshness by guaranteeing that our node will never publish messages older that the time we decide they are useful for. If messages in the outgoing queue are older than the defined Lifespan duration, the policy drops them.
• Deadline — the Deadline policy in ROS 2 gives us watchdogs which trigger warnings if the gap between message arrivals becomes too large at any time. This is important because in a system with data regularly coming in it is useful to promise a specific frequency or period for messages on a topic. The Deadline is the promised maximum duration between messages, or the period for the publishing frequency.
• Durability — the Durability policy defines whether we should provide old messages to subscribers that join late. The default Durability is VOLATILE, for which messages are no longer kept anywhere after being published. The other option is TRANSIENT_LOCAL which means a number of messages up to the defined Depth are stored locally on publisher for later access, and any subscriber that requests TRANSIENT_LOCAL Durability from this publisher will receive the stored history of messages. The ROS 2 Durability policy is much like the latching option for ROS 1.
• Liveliness — this policy is used to detect the absence and presence of publishers. The default Liveliness is AUTOMATIC, which is self explanatory. The alternative, MANUAL_BY_TOPIC, is useful for detecting situations where your node may have fatal problems that don’t cause the publishers to be destroyed. This requires calling the assert_liveliness function on each publisher periodically.

Overall, QoS provides callbacks to give feedback to publishers and subscribers and controls the behavior of message passing. Keeping an eye on compatibility and registering callbacks to detect it can save a lot of time because a mismatch can be hard to spot if your subscriber is silently not receiving messages on a topic that is publishing. Be on the lookout for an article coming soon on our way of implementing all of this in practice. But in the mean time, check out these docs on ROS QoS.

Monitoring

Monitoring software in production is necessary to gain insight into system behavior and state, allows you to act on any detected issues, and provides various useful metrics such as memory usage, CPU usage, and high frequency published topics. Monitoring and troubleshooting robot applications is commonly accomplished through logs or rosbag but these are not scalable because when we get to working with large fleets of robots there is way too much data generated to expect understandable logs.

Note: Despite this, it’s still recommended to send ROS bags up to the cloud and utilize something like Amazon S3 to ingest all of the blob data. You can consider transforming the data into Parquet format for easier indexing for your perception and navigation teams.

Instead we take advantage of the tools ROS 2 gives us for straightforward data aggregation by which we can gather and express our application’s data in a summary form for statistical analysis. Here we’ll introduce some ROS 2 tools to help you aggregate metrics and publish them to a cloud service to provide robust, scalable monitoring for robotic production systems.

First up are tools for collecting and publishing data statistics. The ROS 2 libstatistics_collector package provides aggregation tools to calculate statistics in constant time while using constant memory. The aggregation of a value provides an average (specifically a moving average), maximum, minimum, standard deviation, and sample count. We also have the statistics_msgs definition which can be used to publish the collected data. Other than publishing the calculated data, this also includes the start and stop time of the collection window, metric units, metric source, and measurement name. Additionally, ROS 2 nodes which collect system metrics use the message generation method to convert collected statistics into a ROS 2 message which is periodically published to a topic which can be viewed using ROS 2 CLI tools or consumed by any subscriber.

The system_metrics_collector package provides the ability to monitor a Linux system’s CPU percent used, system memory percent used, and the ability to measure both of these for ROS 2 processes. The package defines a composable ROS 2 lifecycle node that is the collector of the Linux CPU. It is important to note that this is a Lifecycle node, where the collection, aggregation, and publication of statistics can be started and stopped with activate and deactivate commands which provides flexibility in monitoring system behavior and performance without stopping ROS 2 executables. The node periodically measures, aggregates, and publishes a message of CPU percentage used until it is stopped. Besides monitoring application performance, we can also monitor node performance by creating an arbitrary node designated to monitor its CPU and memory percentage used in a similar fashion.

Finally, we arrive at the discussion of streaming aggregated data to the cloud. One example of doing this is via the cloudwatch_metrics_collector package we can listen to the published aggregated data and stream it to the cloud instead of manually inspecting the locally published data. This package provides ROS 2 an interface for metrics nodes to listen for monitoring messages on configurable topics in order to stream them to the cloud. The cloud is known for offering major performance and storage advantages, specifically opportunities to offload expensive computation, store data on remote servers instead of locally, and gain more access to other data like maps or images in the case of robots. A huge benefit of cloudwatch_metrics_collector is the possibility to track the health of a single node and scale to fleet with the use of automated monitoring and automated actions.

Now this monitoring is by no means complete. Additional metrics should be gathered related to the business metrics of a robot (i.e. is it actually doing it’s job). In addition, monitoring systems like over the air updates, inter-robot communication, and trace back logs will be more and more important due to the variability out in the field. Thus, it’s more important than ever to ensure that you have these observability and deployment systems put in place.

The CI/CD Pipeline + GitHub Actions

Robotic systems often need continuous updates, changes, fixes, etc to sustain a robust, satisfactory product. For this we employ the CI/CD pipeline which motivates the implementation of small, incremental changes and frequent checking of version control repositories and testing. Continuous integration (CI) and continuous delivery (CD) follow a set of operating principles that enable software developers to deliver code changes more frequently and reliably.

Recently we’ve seen a trend in moving CI/CD pipelines to the cloud because of the affordable and reliable hardware infrastructure, useful runtime diagnostic metrics, handy user interfaces, and good documentation that the cloud offers. Developers can easily offload testing workflows to the cloud to reduce the need for maintaining separate infrastructure.

The ROS Tooling Working Group provides Actions to help create and set up the CI/CD pipeline in the cloud for ROS packages and can help extend the pipeline to perform more functions such as detecting memory access errors and tracking code coverage automatically (however, these can be hard to configure). A software hosting platform that provides version control and that most developers are familiar with, GitHub, has a collection of some common Actions and workflows that are easy to start out with and reuse. They can also be distributed publicly amongst the dev community in the GitHub marketplace. Simply put, Actions allow developers to run various arbitrary computations as workflows from repositories which can be programmed to run on specified events such as pull requests, new commits, etc. but let’s delve in a little further.

Workflows

At their core, workflows are simple yaml files made up of trigger conditions and execution steps. The trigger conditions determine when the workflow is executed, for example on a new commit to the master branch. It is also good practice to periodically run CI even when no changes are made in order to detect breakages caused by external dependencies. Execution steps call individual Actions, which are computation scripts, for example a JavaScript program or an arbitrary program run in Docker containers; they perform building and testing. A workflow can have more than one Action within it, however it is good practice to separate different kinds of tests into different workflows for example separate linting tests, functional tests, and end-to-end tests.

Actions

The following Actions will be useful in setting up and developing a CI/CD pipeline for your ROS 2 packages:

• setup-ros — this Action supports setting up a ROS distribution environment to build one or more packages and it will install all the dependencies needed to build a ROS 2 package
• action-ros-ci — this Action builds a ROS package from source and runs colcon build followed by colcon test on it (the package on which colcon test is to be invoked is provided as an argument to the Action). The Action requires a ROS environment to already be set up, either with the aforementioned setup-ros Action or using a Docker image with all necessary dependencies installed.
• action-ros-lint — this Action runs ament_lint linters on a ROS package (the package name to lint and the linters are provided as arguments to the Action). A linter is a code analysis tool used to flag programming errors, bugs, style errors, and suspicious constructs.
• colcon mixins — these can be used within Actions as extra command line arguments for the build tool throughout building and testing. For example, some mixins can help detect memory access errors and concurrency issues in the code, helping catch software bugs that are otherwise hard to find and avoid potential crashes. The colcon-mixin-repository provides many mixins that can be reused.

CI/CD emphasizes the importance of code coverage as high code coverage indicates high quality software. Code coverage data is generated by running the test suite of a package and determining which lines of the source code were covered, or executed, by the tests. Developers can focus on improving their code’s health with the help of code coverage tools and web services such as Codecov.io which helps read coverage data output files by showing source code lines hit, missed, and partially hit, and displays code coverage history to help track a repository’s code coverage over its lifecycle. Codecov has also developed a GitHub Action so that users can automatically get code coverage updates when the Action executes as coverage results from the Actions CI pipeline are uploaded to the Codecov website.

Release and Maintenance

Hooray! You’ve taken all these steps to ensure your robotic system is suitable for production and you’re finally ready to release your hard work. There are a few different approaches you can take towards this next step, some open source, some closed, and it is up to the developer to decide which is most suitable based on the nature of the project (commercial vs hobby).

• a compiled library/binary — this is a closed source option that gives users of the system the ability to use a library or executable without being able to see or modify the source code
• a hosted repository — this entails a ROS package complete with all its source code and documentation on how to compile locally, but no executables or binaries. A hosted repository is usually released on a hosting service such as GitHub
• Bloom release — Bloom is OpenRobotics’ meta-tool for releasing ROS packages into the public build farm. It makes packages publicly accessible via package managers and does not require users to compile. The Bloom CLI walks developers through the necessary steps in preparing a package for distribution. The full tutorial can be found on Open Robotics’ official documentation.

Now deploying to the actual robot is the hard part. To scale to a fleet, you have to consider proper over the air update infrastructure, proper A/B testing, feature flagging, and overall release management including things like “did my build fail on this robot”, “how do I incrementally update my fleet?” “What features are working and what failed?” And then gather additional metrics on improved performance of the software.

If this is a problem that you’re facing, we’d love to set up a call with you because this is what we hope to solve.

After all that work you’ll surely want to sit back and relax, but a robotic product requires constant attention in order to remain functional and valuable for consumers. When it comes to maintenance, the goal is to provide users with an experience as seamless as possible and there are a few defined steps to take towards ensuring that.

Quick Note on Contributing to ROS2

Release requires a stable version control implementation at release to support different distributions. Commonly each distribution has a distribution-specific development branch and a Git release tag to specify it is a stable release. Each development branch stems from the master branch and highlights what changes have gone into a specific release. Any changes to a branch after release should contain a release tag to indicate that a change has occurred.

Another thing to mention is that the release cycle times for ROS are longer than those of conventional software apps such as kubernetes since robotic applications require stability and thus can’t be updated at the same pace as a website for example. It is important to remember that updating robots means putting them in a safe state where they can be configured, updated, and restarted properly.

Updating Releases

All changes made to a previously released package must be API and ABI compatible. This means that the source code relying on the API interfaces does not need to be changed and is portable, and changes do not necessitate a recompilation of a package and its dependencies. If this precedent is not followed, then the development workflow of all users of the updated package and all packages that depend on it is broken, forcing users to recompile and presenting cryptic errors.

Sometimes it is only necessary to backport specific commits that resolve issues in a previous release. Backporting is when a software patch or update is taken from a recent software version and applied to an older version of the same software.

Semantic Versioning

Utilizing a universal versioning system is a useful way to give meaning to your software releases and to keep track of every transition in your software. A system like SemVer gives software authors a way to communicate to the consumers of their software important info they should know about this release. After introducing a feature or bug fix to a package it is important to increment the version of the package to allow users to understand the impact of each update. Versions are incremented according to the type of updates made whether they are major, minor, or patches.

Documentation

I cannot stress enough the importance of good documentation for your systems. A lack of proper documentation can make or break a product because it leaves developers on their own to figure out how specific systems work and how to even get started with installing and running applications. A difficult onboarding experience is a huge turn off for users.

A baseline recommended is to include a README.md to document info about your ROS package and the software it contains (one for every package in a repository). A good README should include:

• info on the objective of the package
• maintenance status of the package
• type and status of CI that is maintained for the package
• description of software modules included in the package
• a guide for users to start using the software in the package
• can also include a guide for potential contributors can be included in the readme or in a separate CONTRIBUTING.md file

Go out and build for production!

Wow… that was a lot, I know. Now you have the knowledge to put this all into practice in your next robotic system. We just examined the many, many pieces that go into building a production-ready robotic system and the tools available to do so. Hopefully you’ve gained an understanding of the challenges relating to building robust software for robotics and making it ready for distribution and scaling, and can use this to think about how to develop production ready code from the start of your process.

For the advancement of the open source robotics ecosystem, it is important that all developers learn, practice, and produce documentation on production elements such as running a ROS 2 node on a robot, tweaking its behavior, monitoring its output, building a CI/CD pipeline, and releasing packages following the best community practices. When we continue building out production ready code, it will be way easier to utilize robots as a service.

Building a Production-Ready Robotic System was originally published in Shade Robotics on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Rise of Software

The past decade we have seen nothing short of a revolution in software. As Marc Andreesen once said, “Software is truly eating the world”. It’s become a natural part of every single industry and company. Software is unique in its properties in that it requires no initial materials to get started (well maybe outside of a computer). Its value creation is limited to the skill of the programmer and the creator.

The Barrier to Entry has Been Lowered…Almost Everywhere

With this barrier to entry lowered, we’ve seen more and more software engineers and computer scientists build out amazing frameworks and systems that makes it ever the more easier to create software. I mean just look at the number of web frameworks that are out there for JavaScript. I myself have utilized three: Angular, Vue.js, and React, along with the various flavors of React including Next.js. All of these frameworks and systems have led to the proliferation of faster websites, optimized builds, and a more beautiful and user-friendly ecosystem across software applications.

We see a similar wave of “lowering the entry” with AI, TensorFlow took the first stab at it along with Keras (shoutout Francois Chollet on an amazing library) along with PyTorch that created an extremely pythonic and intuitive way to program AI. And now you’re even seeing repositories like HuggingFace that are plug and play. Spacy and NLTK with NLP models, and many many more.

But what about robotics? What about the software ecosystem around robotics? For most people, the major move forward has been the Robot Operating System (ROS). It’s enabled more people to utilize other people’s code by creating drivers and systems that have already been prebuilt. But we’re still not seeing the crazy adoption and the ease of use that we’re seeing with building a website or building a complex AI algorithm. Why is that? Yes, I agree that the robotics space is harder (control algorithms, navigation systems, there’s a lot to unpack). But I believe that there’s a lot that the robotic ecosystem can learn from the decade of proliferation that the general software world has had, of which I believe the robotics world really lacks.

For Starters…Write Better Documentation

Documentation, as we’re seeing more and more, makes or breaks an open source ecosystem. Without proper documentation of how specific systems work, how to install it, and how to run the application, developers are left to their own devices of tinkering and figuring things out. In fact, that’s exactly what we’re seeing. Roboticists are constantly in the mindset of “I just want this thing to work” instead of thinking “how can I build something amazing on top?”. Even with the ROS ecosystem, we’re seeing difficulties of integrating multiple ROS 2 packages together due to dependency mismatches, a variety of sudo commands, documentation searching on this site, on that site.

For example, let’s take ORBSLAM2 as a key example. When attempting to build ORBSLAM2 with a ROS2 wrapper, I ran into a slew of issues that ultimately led me to documentation that was not even in my native language (something related to Pangolin display issues). With these small minute issues that so heavily depend on the system that you’re running, it’s extremely frustrating to have to say “it works on my computer but not yours”. But that’s exactly the case for robotics. Without the consistency in knowing that a software will work on your computer and no documentation to show how to install it on your computer, you’re really left to your own devices — spending days debugging.

With this in mind, I implore new developers of open source and systems to spend time writing really good documentation. Write strong documentation on how to install your packages on a variety of different system types. Discuss the value add that your code brings. Talk about common gotchas and mistakes people face during set up. Provide proper examples of how to use your code for specific items and actions. This documentation will go a long way for getting new developers into the ecosystem and existing developers building out even cooler systems.

How to get started with Documentation

There are many ways that you can get started with writing documentation. For starters, ensure that installation steps are provided on your README.md. Installation should tell the user what versions of ROS 2 you support, what versions you have tested on, and common issues that have arisen from the issues. In addition, you should provide key examples on how to use your package and code including links to examples, or visualizations of the result. A YouTube video can help a lot here as well.

If you have an even bigger package, utilizing a documentation system such as GitBook, Docusaurus, or even GitHub Wiki can go loads far as users can walk through understanding how things work at their own pace. Nav2 and MoveIt do a decent job here, but we would like to see this for the larger research ecosystem.

Docker Needs to Become Mainstream

ROS2 has done an amazing job for creating a messaging system on top of DDS. But because it has to support multiple different languages and systems, it has relied on its own build system via colcon that has led to many frustrations. Though there is no easy mitigation for this, the best way is to start adopting Docker as the de facto way of sharing code.

Though not everything can be Dockerized, the code that can run as standalone ROS topics should be Dockerized. This includes things such as ORBSLAM2, a wrapper for Resnet50, etc. Whether it’s simply providing a base image that has all the dependencies installed or a docker image that users can pull from docker hub and get immediately running. This will save roboticists loads of time when it comes to debugging and integrating a whole system.

Docker has inherent advantages as it removes issues of “it doesn’t work on my computer” by virtualizing any hardware configuration (of course you have to support these configurations), and with modern CI/CD, it’s easier than ever to build and automate this entire process for the users of your package. The faster it is to get set up with your system, the more people will use the hard work that you have created.

How to get started with Docker

Docker relies simply on a predefined Dockerfile . ROS2 has provided a great set of base images that you can use to create the additional installation steps that the user would have to go through with your package. We’ve also provided a wide variety of docker images including GPU-accelerated base images for AI inference that are built on top of the existing ROS base images, so feel free to take a look at our Docker hub to get started.

You can also take a look at this article that goes through how to set up CI/CD with Google Cloud Build and common gotchas when it comes to setting up Docker with ROS.

Finally, here’s a great article on how we set this all up with the YOLOv5 algorithm (creating a ROS2 wrapper and even publishing it to Docker hub as a container anyone can pull and immediately use).

Build Production-Ready Pipelines

Finally, it’s super important that roboticists and developers start thinking about how to make their code production ready. Only with the larger adoption of corporations utilizing open source will we see the proliferation of a larger open source community. At this current stage, the majority of enterprise customers and businesses still rely heavily on internal code with only a select few packages such as MoveIt and Nav2 being utilized heavily. The wider rest of open source are left to their own devices. Why is that the case? Because we need to see stronger production ready systems. Developers need to be taught on how to build proper CI/CD, and testing into their systems and code. More work needs to be done when it comes to building a robust testing framework, simulations, and deployment of the code.

We need to take ideas such as feature flagging, A/B testing, and bring them to our production-ready fleets. We need to think about items such as cybersecurity from day one, monitoring, telemetry, and analytics.

Only when we integrate the entire ecosystem into one will we see a wider adoption. This is also a call on corporations and enterprise to fully support this open source, third-party driven effort that robotics startups and companies are attempting to start. The new wave of Robot Dev Tools are coming to help empower the next generation of roboticists, but we need your support.

By building production-ready code and a production-focused code (with the thought that maybe our code could one day run on thousands of robots), we want to pay close attention to all the things that we mentioned above (docker containers, better documentation, etc.)

Conclusion — Don’t Just Get Things to Work, Go the Extra Mile

Overall, the robotics space is an exciting one and only recently are we beginning to see the true Industrial 4.0 slowly come to life. With the introduction of AMRs, large language models, vision algorithms for grasping, and more amazing research being built, it’s an amazing time to be alive. Yet the news that we hear and the systems that we see out in the wild are still few and far between. Only a select few have all the resources to build at light speed, while many others are still struggling to just get systems to work. I believe that if we integrate and aggregate all of the systems and tools out there, provide better documentation, and ready-made docker containers, we can achieve our dream and vision of an Industrial 4.0 much much faster.

What Robotics Can Learn from SaaS was originally published in Shade Robotics on Medium, where people are continuing the conversation by highlighting and responding to this story.

Just 10 years ago, most robotics developers would have complained about programming software that interacted with hardware. There was no common system for how robotic software should communicate, so it was almost always up to the developer to come up with a novel system for their robot. But because of differing hardware, other roboticists and developers could not reuse previously developed programs, and they had to make their own system. This process of reinventing the wheel dramatically hamped robotics research and development, oftentimes taking up significant portions of the 2–3 year time frames for most robotics projects.

The Robot Operating System (ROS) was a solution to this pain point. With the goal of providing a standard for robotics software development regardless of hardware, ROS allows developers to focus on key differentiating features of an application rather than re-building a foundation.

First created in 2007 by Willow Garage, a research center with a focus on robotics products, the ROS functions as a middleware that is responsible for handling the communication between programs in a distributed system. In essence, the Robot Operating System is a core set of communication frameworks that is then augmented by a variety of plug and play libraries for more specialized applications.

How does ROS work?

With robotics, it makes the most sense to develop software applications modularly to interact with specific pieces of hardware and then piece them together rather than coding up one giant code base with everything compiling and running in one block. This is especially true if you’re using different pieces of hardware from different vendors such as a RealSense camera from Intel, a Puck from Velodyne, and a robot arm from Agility. ROS allows you to easily separate your code base into packages of small programs called nodes, and then facilitates communication between them. It does so through 3 main tools: Topics, Services, and Actions.

• Topics essentially serve as data streams between nodes. One example would be a program that is monitoring the battery life of the robot and can send a data stream with the relevant data so that other nodes can obtain the information. This is typically also described as the general publish / subscribe architecture (or pub/sub).
• Services allow you to set up synchronous client/server communication between nodes, like changing a setting or sending a specific request. Frequent examples might include asking for XYZ data or enabling freedrive mode.
• Actions provide asynchronous client/server architecture, where the client can send requests that may take significant time to complete. The client is able to monitor the state of the server, as well as change or cancel the request at any time. A common example of an action might be pinging the robot to move to a new location.

These three tools form the foundation of the Robot Operating System, but it now encompasses far more than that. After Willow Garage closed-down in 2014, ROS has flourished into an open-sourced community of dedicated contributors around the world. Open Robotics serves as the global ROS committee that sets the general direction to follow for community development, and every year there is a worldwide event called ROSCON where development progress is shared. With such strong global support, ROS has seen an abundance of open sourced growth.

When developing software for a robot, there are dozens of functionalities that you have to build out, from motion planning to 3D visualization. For most of the common functionalities in robotics, you can find an open source ROS package. Some of the most notable open source ROS resources include:

• Ros_control: Ros_control is the go-to package for most robot’s main control loop, as the set of packages include controller interfaces, controller managers, transmissions, and hardware interfaces.
• URDF: URDF, or the Unified Robot Description Format, is an XML format that allows you to describe and represent a 3D model of your robot.
• MoveIt: MoveIt enables path and motion planning to allow you to generate high-degree of freedom trajectory through cluttered environments and avoid local minimums. For more information on motion planning, check out this article here.
• Navigation: This package supports mobile robots by serving as a 2D navigation stack that takes in information from odometry and other sensor streams and outputs safe velocity commands to a mobile base.
• Rviz: Rviz is the most used 3D visualization tool for ROS, a key functionality for most robotic software.
• Gazebo: Gazebo is a powerful simulation tool that streamline software testing, even with various physical constraints like gravity and wind. Gazebos suite of tools, libraries, and cloud services serves as a complete tool box for robotic simulation.
• Rosbridge: Rosbridge provides a JSON API that empowers communication between ROS and non-ROS environments. Many frontends are able to interface with rosbridge, including a WebSocket server for web browsers to interact with.

The Evolution of ROS

As arguably the most mature open source robotics project in the world, ROS has inevitably evolved over the years. In its early years with Willow Garage, it was developed alongside PR2, a humanoid robot that was able to navigate autonomously in its environment, and it aimed to be a scalable, standard robotic framework for all robotic software. The early levels of standardization and abstraction were guided by the PR2 characteristics: a single robot operating with no real-time requirements and strong network connectivity.

The first public distribution of ROS was released in 2009, known as ROS Mango Tango, or ROS 0.4. The 1.0 release coined Box Turtle was launched almost a year later in 2010, and it set a naming convention based on turtles. 5 more versions were released by Willow Garage before their dissolution, ending with ROS Groovy Galapagos in 2012. Under the oversight of the Open Source Robotics Foundation, ROS continued to develop and release new distributions starting with ROS Hydro Medusa in 2013.

However, by 2015, the deficiencies of ROS for commercial products were growing more and more apparent. Many critics pointed out ROS suffered from a single point of failure with the roscore. The roscore is a collection of nodes that are prerequisites for any ROS-based system; any flaw in the roscore meant failure for the software. Others pointed out the lack of security with ROS. Despite being one of the most adopted frameworks in the field of robotics, it still suffered from a variety of security issues that could interfere with the communication between ROS nodes. Finally, due to the nature of the origins of ROS with PR2, ROS never introduced real-time support, a functionality considered critical by many industrial applications.

While improvements to these issues could have been introduced and integrated into the existing ROS code, it was largely considered too risky to so dramatically alter the ROS system that was relied upon by so many people. With these concerns in mind, Open Robotics began an effort to create ROS 2.0, a parallel set of packages that could operate alongside ROS. The goal of the new framework was to address many of the shortcomings of ROS in industry. ROS 2 would be built for teams of multiple robots, small embedded systems, non-ideal networks, production environments, and real-time networks. Ardent Apalone, the first distribution release of ROS 2 was released in December of 2017, officially ushering in a new era of next-generation ROS development.

Beyond the high level differences mentioned above, there are many other changes from ROS and ROS 2. ROS uses mainly C++03 and targets Python 2, whereas ROS 2 mainly uses C++11 and is suitable for >= Python 3.5. Overall, there have been many efforts made to improve the flexibility of the ROS framework. ROS has a custom serialization format, can only choose between single-threaded execution or multi-threaded execution, and its roslaunch files are defined in XML with very limited capabilities. ROS 2 implementations are all based on the Data Distribution Service (DDS) standard, can support custom executors, and have launch files written in Python that enable more complex logic. Industry has been largely influential in suggesting changes to be made, and has guided many of the improvements so far.

Where is ROS now?

Since the transition from ROS to ROS 2, new ROS versions have stopped. ROS Noetic Ninjemys, the thirteenth and final ROS distribution release, was suitably released on World Turtle Day — May 23rd, 2020. The goal of this release is to provide support for developers and organizations that have not yet transitioned to ROS 2 and still need to work with ROS for a while. ROS Noetic’s End of Life is scheduled for 2025, so after that date, there will officially be no more of the original ROS.

Fortunately for us roboticists, ROS 2 has taken over the annual releases, with the most recent distribution, ROS 2 Humble Hawksbill, released on May 23rd, 2022. Since the previous year’s version ROS 2 Galactic Geochelone, Humble has introduced a variety of small features and quality of life improvements like support textures and embedded meshes for marker messages. An overview of the newest changes can be found here.

There’s still a lot of room for ROS 2 to grow. With no backwards compatibility with previous versions, many users are stuck with few wrappers except for the larger ones that have garnered support. Without the active wrappers, even integrating various state of the art algorithms like ORBSLAM3 or YOLOv5 can take more time and effort. ROS 2 also heavily relies on virtualization via docker containers. While this offers a slew of benefits like better dependency management and non-mixing of core library versions, for many roboticists, this might be unfamiliar territory. We’ve surveyed users across labs from UC Berkeley, Cornell, Virginia Tech, and Michigan: it takes on average 2 to 6 months to fully integrate a new robot with ROS 2 and to get robots to properly communicate. This combined with the sentiment that ROS is not yet “fit for production” has prevented ROS from fully emerging as the industry standard.

Despite this, the ROS community is more active than ever, with renewed community enthusiasm for ROS 2. With each new release of ROS 2, there are dozens of new innovations and offerings introduced. Take FogROS 2 as an example. Cloud robotics has long been an intriguing solution to many computing, storage, and resource management problems within robotics, but Fog robotics has recently emerged as another capability, allowing robots and IoT devices to leverage nearby Edge resources alongside Cloud data centers. Nevertheless, tapping into that power from modern cloud computing providers like AWS, GCP, and Azure is non-trivial to say the least. Introduced with the release of Humble Hawksbill, FogROS 2 serves as an open-source platform to streamline and facilitate both cloud and fog robotics compatible with the ROS 2 framework. Contributions to the ROS 2 open-sourced based of knowledge like FogROS 2 enable exciting progress across both research and industry.

Empowered by standard frameworks like ROS 2, robotic software is making leaps and bounds. It’s a great time to be a robot software programmer, so keep an eye out every World Turtle Day for the newest ROS 2 release and all of the development progress the community makes.

The Evolution of ROS was originally published in Shade Robotics on Medium, where people are continuing the conversation by highlighting and responding to this story.

Modern Computing

The idea of creating specific component to accelerate repetitive tasks performed by a computer has been a concept that has existed even before Bletchley Park. It just makes absolutely no sense for a computer’s CPU to continuously handle a task such as processing the refresh rate of your monitor every 1/120th of a second or your computer freezing just because it got bogged down an extremely expensive tasks like matrix multiplication, prime modulation, or many other common essential tasks. And, it’s no wonder, with the rise of performance we witness in our machines today, it’s clear that specialized sections within our machines have been engineered specifically to target these commonplace tasks. Components like GPUs, multimedia/cryptographic encoders and decoders, and many other specialized components, are just some of the many innovations jammed into a space sometimes smaller than an eye can see. Okay, maybe I’m exaggerating a little.

Since the revolution of designing many of these core components, new proprietary software which has now become more commonplace especially within the software scene has also begun to require complex and expensive computation, with a notable one being machine learning. Driven on a concept of multi-dimensional matrix manipulation and extreme repetition of tasks, it’s obvious that many wish to tap into the GPU to accomplish that task — using its powers to accelerate and offload that burden. Yet, even while many machine learning frameworks like PyTorch, TensorFlow, and many others have provided GPU acceleration right out of the box, utilizing that power within a containerized environment like Docker is still a confusing and nontrivial task.

This guide will walk you through how to properly create and utilize a GPU Accelerated Docker Container.

Using CUDA and GPU Acceleration in Docker

*please note, GPU acceleration is currently only limited to NVIDIA graphics cards with CUDA support. this is because no other graphic card manufacturers provide open source drivers that are widely supported and integrated within the development ecosystem. If you are not sure if your machine supports a NVIDIA graphics card, you can search system information on your windows machine, look at About This Mac on MacOS, or use the command ubuntu-drivers devices on ubuntu.

Setting up the Container and Configuration

To truly utilize GPU acceleration within a containerized environment, a couple of things must be complete on both your local machine and within your containerized environment.

NVIDIA and CUDA Drivers

To first get started, you need to install NVIDIA’s graphic drivers, CUDA drivers, or the CUDA toolkit. If you are unsure if you have these drivers already installed, you can follow the steps below for your specific operating system which include instructions as to how to install the drivers. A quick google or stack overflow search can also provide you similar results as what I have here.

Linux

To check if you have the drivers properly installed, you can run nvcc -V or nvidia-smi within the terminal. You should get an output similar to one of the images below

If you do no see any of these however, you can install the drivers via the software installation tool (on later ubuntu versions) or run the command.

$ sudo ubuntu-drivers autoinstall

In case you wish to individually select a version of the drivers to install instead, you can run the command sudo add-apt-repository ppa:graphics-drivers/ppa and then sudo apt install nvidia-driver-XXX , with XXX being your desired version listed within your CLI.

Then, restart your computer for these to take effect. You should also run sudo apt install nvidia-cuda-toolkit to install the CUDA toolkit as well.

There are also known cases however where this fails to work on certain versions. In this case, follow the steps given on the official NVIDIA Driver Installation quickguide instead. Please be careful however when pruning your system of nvidia keywords, as it may completely nuke your ubuntu installation.

Windows

To check if you have the drivers properly installed, you can find this under Windows Device Manager from your search bar. Or, you can try right clicking on your desktop screen and seeing if you witness a NVIDIA Control Panel option in the dropdown menu as well.

If you do not have the drivers installed however, you can simply download and run the installer from the NVIDIA Website.

MacOS

To check if you have the drivers properly installed, you can find out under About This Mac → System Report → Extensions (located under the software tab).

If you do not locate any NVIDIA driver however, you can also install them here following the link here. M1 Macs, you guys are kind of at a loss here.

GPU Compatible Containers

NVIDIA’s CUDA drivers must also be installed within the container as well, not only on your local machine. To do so, I highly recommend utilizing a prebuilt container from NVIDIA’s Docker Hub as your base image. This can be easily done by replacing your base image ofFROM ubuntu:18.04 to something like FROM nvidia/cuda:10.0-devel-ubuntu18.04 (differing versions and debugging tools). I also recommend using the cudnn variants as well (if you are utilizing ubuntu ≤ 20.04) as they are especially optimized for machine learning use cases.

However, in the off chance that this isn’t possible, viewing NVIDIA’s official Github pages with their source code and modifying it to fit your image would be your second best bet. However, many of these images displayed on Github do not work out of the bat, with many experiencing expired key problems as well as a variety of different issues. From my personal experience, it is much easier to manually install any other dependencies you wanted to use as your base image onto the base image of the NVIDIA drivers rather than the other way around. It would save you hours, if not days of development time taking this approach. I recommend cross-referencing questions on the NVIDIA forums as there do not current exist many Stack Overflow posts on this topic.

If you are ever unsure as to whether your container has successfully installed NVIDIA tools, you can simply modify your base image briefly to install the devel flag instead of the runtime or base flag. Then, simply exec into your container and run the nvidia-smi command. However, do not worry if you do not see your GPU within the menu at this time. That will be what we tackle in our next step. If, however, you see the error that the command is not found, you have probably failed to install CUDA properly within the container.

GPU and Docker Interconnect

Finally, once you have properly installed you must also have Docker and the NVIDIA Docker toolkit installed as well. You can run the following commands below on a Linux Machine or follow the official guide here to properly install it.

$ curl -s -L [<https://nvidia.github.io/nvidia-docker/gpgkey>](<https://nvidia.github.io/nvidia-docker/gpgkey>) |  sudo apt-key add -distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
$ curl -s -L [<https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list>](<https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list>) | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
$ sudo apt update
$ sudo apt-get install -y nvidia-docker2

Then, once all these are completed and when you have restarted either your computer or your Docker Daemon (which can be done through sudo systemctl restart docker), simply run your Docker run command as you would, except you add the flag —gpus all in front to allow your GPU to be passed to the container.

A good command to test if everything is working is

$ docker run -it --gpus all nvidia/cuda:10.0-devel-ubuntu18.04 nvidia-smi

Or, another alternative if you are running python is that you can run the following commands in your container once you have run it, granted that you have PyTorch installed.

$ python3
>> import torch
>> torch.cuda.is_available()

Questions

Why Containerize?

Before we end it off, you might be asking, why go through all this effort just to containerize our code if it could run just fine on our local machines?

And, the simplest reason being compatibility.

Installing multiple different dependencies (for example in systems like ROS) causes a lot of issues. Programs may begin crashing for no reason and things simply might not run on a different machine. By creating a container or image, you instead remove many issues that result any external factors (i.e. operating systems, other packages, etc), and easily allow your program to run anywhere. Also, containerization is what makes deployment to cloud services much simpler and smoother. Docker containers are becoming more and more prevalent as we see it solving the main issue of “it works on my computer”. In addition, docker containers are becoming a de facto way of deploying code to robotic, edge, and embedded systems and a way to further remote development and work.

Why use GPU?

Using the GPU to offload a variety of tasks helps speed up performance. A Lot. Just take a look at this object detection program we have here called YOLOv5. Just by passing the —gpus all, we see over a 10x increase in performance with performance, with the CPU only build running at roughly 1 frame processed per .442 seconds while the GPU accelerated container processing a frame within .046 seconds.

So there you have it! In this article we went through how to set up a GPU-accelerated docker container. As always, feel free to shoot us an email at hello@shaderobotics.com if you have any problems. Also if you’re interested in robot developer tooling including things such as robot feature flagging, a/b testing, or robot infrastructure, we’d love to talk!

Building GPU Accelerated Docker Containers was originally published in Shade Robotics on Medium, where people are continuing the conversation by highlighting and responding to this story.

A Brief Overview of the Cloud and Fog

Cloud computing basically allows people to use a company’s larger servers to perform high-intensity processing. A company may want to run some high power AI algorithms but may not have the equipment to do so. Cloud computing makes this possible through off-site servers. It is a way of extending computing capabilities without investing in new hardware.

Fog computing is an extension of cloud computing. The fog is the layer between edge devices and the cloud. Where edge devices may be the hardware on the device and the cloud being large data centers in centralized locations, fog nodes are devices that lay in the areas and locations between: more computation than the edge, and less latency than the cloud. As a result, the deployment of fog computing, can absorb some of the pressure of a cloud data center. Theoretically, any device with spare resources can act as a fog node and be a resource supporter for fog computing. Fog nodes are located in closer proximity to the edge devices from which they receive data, making fog computing superior to cloud computing in terms of response speed. Additionally, they are able to analyze and filter the data they receive and only forward relevant data to the cloud, further limiting traffic congestion.

Bringing the Cloud and Fog to Robotics

Cloud and Fog Robotics were introduced as a way for robots to enhance their capabilities by allowing them to take advantage of the processing power of large servers and computers: robotics without the hardware limitations. Many times in robotics either equipment is too expensive or complex programming and processing is out of reach simply due to computational constraints such as not enough RAM or GPU. But with cloud and fog robotics, integrating computationally-intensive algorithms is straightforward (with trade offs as we will see later). By offloading much of a robot’s data processing to off-site (near-site) servers and processing and only running control-critical tasks locally (such as safety mechanisms, collision detection), computationally intense robotics tasks can be solved more efficiently while infrastructure costs are reduced, leaving us with more adaptable, scalable, interoperable, and reliable robots. Let’s dive into some of the immense advantages that robotics gains from applying the cloud and fog.

Adaptability

Smart devices like robots benefit greatly from central data processing because it allows them to easily access colossal collections of information (think a hive mind) and powerful computing resources. This makes for more flexible robots that are able to adjust to various scenarios at any time. Such adaptability is crucial for robots in remaining robust.

Let’s talk through a quick example: imagine you have a new autonomous mobile robot that is being rolled out across all your warehouses. Each robot is assigned to an associated warehouse, understanding the mapping, location of items, and associated classifications for the items specific to that warehouse. Now imagine a new robot that comes in or a robot that moves to another warehouse, or even a new item that comes into inventory (multiple variables). Without a central source of truth, these AMRs have to relearn everything again. Introducing a hive mind leads to more adaptable robots and faster ROI including applications and companies in robots-as-a-service.

Having a data storage that is capable of holding more data than a robot could hold onboard unlocks the potential for a robot to learn about and adapt to more environments, tasks, and potential collaborators (other robots). A self-driving car, for example, might have to travel in different areas or terrains from the ones in which it was trained, so having data about new environments which it can pull from the cloud makes for a more adaptable system, and all without having to store additional data onboard the system. Different AVs can even learn from each others’ travel experiences as they can exchange and share data about environments they have already experienced, via the cloud. This capability is extra useful when working with a large fleet of robots that should all be able to perform the same operations because the cloud makes it easy and efficient to reuse training data when expanding the fleet.

Even more than the cloud, fog robotics helps leverage this adaptability because it allows robot tasks to be carried out the the use of devices that might not be a part of the robot. Fog computing does not depend on the availability of all connected edge devices which requires the fog to be able to assign tasks to the best performing devices of the current moment. Within the fog architecture, tasks can easily be switched to devices that are working correctly in order to continue task execution and avoid total failure in cases where a part or even an entire robot malfunctions.

Scalability

Scalable technologies and solutions are always favorable and this is especially true in robotics as automation becomes mainstream across most industries. Companies like Amazon with large warehouses utilize robotic fleets to navigate and manage these warehouses. These fleets must be able to increase performance as they inevitably increase in size. One of the best things the cloud provides is an elastic infrastructure which allows for automatic deployments and thus ease of scalability. The use of a central cloud network, where all the same data is available to every robot in the fleet, makes it straightforward to expand the fleet by simply connecting additional robots to the same network which instantly provides them with the same data and capabilities as the rest of the fleet. Typically if you want to scale up a system with a lot of robots, you have to add more servers, but in the cloud it’s done automatically. Hundreds of robots can be deployed without any difference and workflows can be transmitted to hundreds of robots over the cloud in minimal time.

The fog ecosystem embraces scalability as the amount of devices connected is huge and this number can easily vary, allowing for not only scalability but flexibility as well. Thus fog robotics allows a robotic system to easily be scaled up as there is a lot more resource availability among the collection of devices in the fog which can be integrated into the system to expand it, whether that be in physical size or computational power and efficiency.

Cheaper on Hardware

Developing a robot can be expensive, and the more advanced the robot is in terms of computational abilities, the more hardware is required on board the robot to be able to process large amounts of data and perform complex computations. That places a huge burden on roboticists as they are forced to build bulky robots using crazy expensive hardware such as processors, sensors, actuators, etc. Complex humanoid robot production costs can quickly get up into the millions. Often, robotics projects are even forced to make certain performance tradeoffs due to the lack of affordable hardware.

The cloud offers roboticists the ability to offload a lot of computation to other servers. For example, data can be pushed to the cloud for processing and building a map of a robots environment, which can then be transmitted back down to the robot for local navigation. The same map can be transmitted to other robots using the cloud. Robots that take advantage of this can be produced using less hardware, making the overall product much cheaper and easier to manage. The costs can instead be allotted to additional robots, research, and other useful robotic endeavors. Additionally, when robots can offload computation, carry less sensors, or use less powerful processing units, they save a lot of energy. This is a huge advantage as it then allows them to retain their battery life and remain autonomous.

Another solution to hardware constraints is provided by the fog: real-time inference offload. When robots are unable to contain all the necessary hardware to perform everything onboard, being connected to the fog allows them to use hardware from surrounding devices. For example if a robot is not equipped with all the required sensors it can tap into the sensors of a nearby fog node to complete a task. This type of interaction would be natural for something like a robotic fleet in which it is normal for robots to utilize each others data anyway, and in this way each robot in the fleet also serves as a fog node.

Interoperability

In the robotics world, communication can occur between robots of the same type as well as different robots. Maintaining this line of connectivity is essential for collaborative robots which need to share states and information with others. When robots cooperate the result is more specialized robots that can complete more complex tasks because they are simultaneously and constantly receiving multiple streams of information that empower the tools that carry out those tasks. Yet putting this in practice is easier said than done. With different devices having their own messaging systems and protocols, the need for standardization becomes more and more important.

In the cloud and the fog, every device in the network can interact with any other device in the network allowing multiple types of cooperation. This connects back to the idea of a hive mind shared by numerous robots all having access to the same information. A central brain of this kind, which promotes a universal operation schema, is what organizes robot fleets and can even enable communication and cooperation between robots from different companies. Imagine a world where self-driving trucks can communicate and dock autonomously with warehouses.

Interoperability also enables existing robots to integrate into new spaces and platforms and thus heavily contributes to the growth and progress of the robotics industry. In this way, it is an essential step towards a future robotics environment where the full potential of robots is realized.

There Cannot Be Light Without Darkness…

Unfortunately, the cloud and fog robotics vision is still early and, as is common across all innovation, these promising advantages are accompanied by an array of challenges. Although cloud and fog computing certainly take robotics to new heights, we’re not working with flawless technologies here, so it is crucial to consider how factors such as latency, security, connectivity, and feasibility fit into the picture.

Latency

Perhaps the largest issue we face when it comes to incorporating the cloud and fog into robotics is latency. Due to the rapid increase in smart devices, and consequently in data and traffic, cloud performance degrades as more devices demand more fetching of data to and from the cloud, causing resource bottlenecks at the data center and in turn creating network congestion, latency, lower efficiency, and significant decrease in reliability. This is bad news for cloud robotics since many robotics applications such as robot navigation have strict requirements on processing latency. High latency prevents robotic systems from working in near real time which is essential for many reasons like safety and fluidity. For example, the sensors in a Tesla car constantly monitor its surrounding environment. If they detect an obstacle which the car must move around or stop n front of, the data sent through the sensor must be processed instantly to help the car recognize the obstacle before it comes into contact with it. If the delay in detection is even a fraction too long because of latency caused by connection to the cloud, there could be major issues and human safety concerns.

In addition to the network congestion. Simply sending data across current cellular networks are slow and inconsistent by design. Though we are looking forward to the promises of 5G and its ability to mitigate latency, there still is critical infrastructure to be built to scale this across multiple locations including rural environments.

Fog robotics arises due to the increase in demand of cloud services and the fact that the applications of cloud robotics require low latency in order to be truly useful. Fog helps mitigate some of that latency. Since fog nodes are connection points between an edge device and the cloud, fog allows us to store and process data closer to the robot, which minimizes traffic to the cloud and reduces latency while still having computation offloaded from the robot. Fog computing promises a real-time network that the cloud lacks.

In the end, more work needs to be done to optimize the allocation of operations with their associated levels of importance (for example human safety critical operations must be done on board whereas speech synthesis and high level mission planning could be done offline).

Security

Another issue that stems from increased data and traffic to the cloud is related to security. Robots become vulnerable to things like privacy breaches, process hacks, and ransomware attacks when data is stored and operations are performed remotely in the cloud. Hackers can use remote execution to get access to robots connected to the cloud and change robotic services to affect robotic behavior in harmful ways.

This is where having a large fleet of robots all connected to and controlled by one cloud network can be extremely risky (single point of risk). If a hacker gains access to the network, they can cause dangerous attacks on a massive scale. This however can be mitigated with VPN connections.

An even better solution is found in the distributed architecture of fog robotics, where the “distributed-by-nature” creates additional layers for security so that network security protocols can be applied between network nodes and entire systems are safeguarded from cloud to device. Fog nodes can monitor the security status of nearby devices to quickly detect and isolate threats. Therefore, if one node is compromised the rest of the network can still be safe. In addition, since computation is separated across various nodes, no node or point in the network has a complete view of an ongoing operation, following the least access privilege and zero-knowledge security practices.

Connectivity

Imagine, in the future, having a life-saving operation performed on you by a surgical robotic arm. Now imagine the trouble that could ensue if the robot suddenly loses its connection to the cloud. It’s simply not an option.

Both the cloud and fog are subject to connectivity issues that can rise from various factors such as network congestion, power failures, and security breaches. These network connectivity issues make cloud and fog robotics not fully reliable as robots can potentially become not only slow but completely inoperable at any point. Although systems may have backup protocols onboard for cases when such failure occurs, such protocols are not possible for all systems or for certain computations and operations. Fog, however, has the advantage here in that fog nodes can form resource sharing clusters as a fallback so that robots can keep sharing resources performing tasks even if a network goes down. This idea follows the principles of swarm robotics in which groups of robots operate without relying on any external infrastructure or form of centralized control.

Thus this element of fault tolerance needs to be considered and more code needs to be written to ensure that latency and connectivity are utilized to determine best allocation of algorithm and robotic performance.

On the bright side, the growing 5G network offers major connectivity and latency improvements, hopefully making these challenges less of a worry for cloud and fog robotics in the future.

Development

One of the more overarching things we must consider when it comes to cloud and fog robotics is how feasible the development of such technologies is. Development locally with the Robotics Operating System (ROS) is already its own feat. Trying to get critical algorithms such as YOLOv5 or motion planning algorithms is still not a plug and chug based system. ROS2 seeks to help make some improvements via virtualization and real-time systems, however the ecosystem is still limited with limited connections with the fog and cloud. Though it is widely used by roboticists, besides being fairly difficult to start out with, ROS tries to be universal and comes with many features that you don’t really need. This leaves developers having to modify ROS for their specific needs, substituting certain pieces with other libraries or starting completely from scratch writing their own libraries. What this results in is a complex and lengthy development process, tons of variations among common robot operations such as path and motion planning, navigation, etc., and no solid universal standard for robots. Currently, the technical complexity and the lack of standardization between hardware and software components in robotics are creating big disconnects that limit innovation and drive many away. Without easy software integrations to create state of the art robots, or limited access to state of the art algorithms, how can we assume that every robot will be easily connected to the fog and cloud?

Takeaways

Cloud and fog robotics are no longer future phenomena; they’re here. However, there are certain hurdles that we must get over before they can be widely implemented without worry. Many of the challenges that exist within the cloud robotics space can be mitigated with fog robotics but even fog is not a perfect solution. In many cases it is subject to similar vulnerabilities, just on a smaller scale.

Despite the fact that we face a few challenges that hinder present capabilities, there’s no arguing that the cloud and the fog are game changers for robotics and will take the industry far. The current state of technology, data, automation, and the likes simply demand that cloud and fog robotics will become the norm.

Why the Cloud and Fog Can Improve Robotics was originally published in Shade Robotics on Medium, where people are continuing the conversation by highlighting and responding to this story.

Computer vision is an important part of robotics. It allows the robot to move in space, identify objects as it moves, and then ultimately pass this information off to other algorithms such as motion and grasp planning to interact with the object and put it into its proper place. Today, we’re going to investigate the Robotic Operating System ecosystem (specifically ROS2) along with ROS wrappers (the ability to integrate data into ROS2’s messaging system) and implementing our own wrapper for an object recognition algorithm: YOLOv5.

The TL;DR If you are interested in simply utilizing the YOLOv5 wrapper or viewing the entire source code, feel free to just visit here where we provide a full Dockerfile for you to directly run! You can also go ahead and pull the built docker container from our docker hub.

docker pull shaderobotics/yolov5-ros2:foxy

So What is ROS?

The Quick Summary

At its core, the Robot Operating System is a standard set of communication frameworks that enables roboticists to easily connect and integrate together a full system of different drivers and components that would often take months if not years to build out itself. First created in 2007 by Willow Garage, a research center with a focus on robotics products, the ROS functions as a middleware that is responsible for handling the communication between programs in a distributed system. This comes from grabbing data from the underlying drivers and devices themselves (such as the position of a robotic arm, the footage from a camera, etc.), and converting them into a generalized messaging format that any other component (proper term is Node) can read. ROS has cultivated a gigantic ecosystem of industry and academia that have written out quite a variety of ROS wrappers that encapsulate various parts of the robotic lifecycle including motion planning (take a look at MoveIt), navigation (Nav2), and more. With the goal of providing a standard for robotics software development regardless of hardware, ROS allows developers to focus on key differentiating features of an application rather than re-building a foundation.

Development of ROS2

ROS2 extends this vision of the original ROS but attempts to solve some of the qualms of the original version of ROS: namely that it was not a hard real-time system and had no subsequent guarantees that specific items would be executed in a specific amount of seconds. ROS2 was released in 2018 but the initial community of wrappers is still few.

For more information on this article and the differences between ROS1 and ROS2 check out this article by Colin who goes more into depth into ROS and the future of ROS as well.

What’s the point of all of it?

Moral of the story, ROS helps you build robots much faster. Whether you’re bodging (i.e. just trying to put a lot of pieces together for a theoretical prototype), or going into a production system, ROS has a variety of use cases and solutions that makes it a go to messaging system for your robot.

Creating a ROS2 Wrapper

Now if you’re still reading this, you’re probably asking yourself the same questions that I did, all those years ago, when I first started learning ROS. Questions like “Why is it so difficult to get ROS set up?”, “How in the world do you integrate this third-party package”, or “Why does everything keep crashing when I try?”. And don’t worry, I get it! I’ve been there, my co-workers have been there, and that experience doesn’t seem to go away, even when time goes on. But, whether you’re just getting started with ROS or you’re a ROS veteran, I hope that this article can help you along your way of integrating your next component.

Before we begin, I do need to specify that this tutorial is more geared toward those aiming to create ROS2 integrations. However, ROS integrations should be decently straight foward as well, with the major differences being the compiler depending on other build tools like Rosbuild or Catkin instead of Colcon. Additionally, for the sake of simplicity, we will be focusing on creating a python wrapper today, specifically a YOLOv5 wrapper in the second section below.

Creating A General Wrapper

1. Setting up the Environment

To start off, we will be installing all the packages required for your wrapper node. Ideally, using something that containerizes your target node (i.e. Docker) would be extremely optimal as it prevents dependency issues between your desired wrapper and other parts of your robots. However, installing the dependencies locally is a solution as well.

Keep in mind, this is, by far, the hardest and most time consuming step of the process.

To help you get started, feel free to use any of the base images provided by ROS themselves or, if you are hoping to utilize CUDA cores to accelerate your program, you may also find one of Shade’s base images useful as well. To help you get set up, here is an example of an extremely basic base image Dockerfile that could be used to create any Python ROS2 Wrapper

FROM ros:foxy
ENV ROS_DISTRO=foxy

# set up workspace
RUN ["/bin/bash", "-c", "mkdir -p /app/shade_ws/src"]
WORDIR /app/shade_ws/src

# inject your wrapper in either locally or through git
COPY . /app/shade_ws/src
RUN git clone [gitURL]

# install dependencies
#repeat for every package you have.
RUN cd [package] && \\
	python3 -m pip install -r requirements.txt  

# build your wrapper
WORKDIR /app/shade_ws
RUN colcon build

# run your wrapper. modify the third argument to your wrappers needs
ENTRYPOINT ["/bin/bash", "-c", "source /opt/ros${ROS_DISTRO}/setup.bash && source ./install/setup.bash && ros2 run [package_name] [entrypoint]"

2. Creating your wrapper

To create your wrapper, use ROS2’s built in command line tools ros2 pkg create to create a skeleton template of your new wrapper. Enter the package.xml and modify any fields that are necessary, enter the setup.py file and modify any fields necessary and begin writing your wrapper. Assuming you have followed step 1 correctly, you should be able to import any file and be completely fine, avoiding any dependency issues. Please refer to the official ROS2 docs for a greater understanding on basic pub/sub topics.

3. Running your wrapper

Then, when your wrapper is finally done (or you’re just trying to test it), simply build your container by running docker build . -t [name] in your repository, then running docker run -t [name] and voila, your new ROS2 wrapper node should be running! Even when you are outside your Docker container, you can still see all the existing topics with ros2 topic list and pub/sub to those respective topics as well!

Things to watch out for when implementing a ROS2 Wrapper

Dependencies, Dependencies, and more Dependencies!

When installing different packages, be very careful of the required Python versions that are required for each package. For example, ROS2 Eloquent natively only supports Python 3.6.9 so if you need to utilize something like Numpy, they would be not compatible. Possible solutions for this however are forcing dependency versions with python3 -m pip install package=version or to try a different ROS2 distributions.

Testing

If you find difficulties testing your wrapper, there are many sensor emulators out there on google. Simply google your desired sensor streamer and you should find a couple of Github repos you could git clone into your Docker container. Also, you can introduce a ROS argument with --ros-args -r /code_topic:=/renamed_topic to remap any pre-existing topic names to match your code.

Creating a ROS2 Wrapper for YoloV5

Now that you understand the general steps for creating a basic Python ROS2 wrapper, lets walk through how to create a basic YOLOv5 wrapper. If you are interested in simply utilizing the wrapper or viewing the entire source code, feel free to just visit here.

1. Setting Up Environment

Similar to the steps above, we want to begin by starting off by creating a basic Dockerfile to be the basis for our image. Because of the nature of our wrapper being based in AI and CV, having Cuda enabled would greatly accelerate the performance of our node we wish to create. Because of this, we’ll be using one of shaderobotics base images.

FROM shaderobotics/cuda:foxy11.7.0

# create a basic workspace
RUN ["/bin/bash", "-c", "mkdir -p /app/shade_ws/src"]
WORKDIR /app/shade_ws/src

# install some basic tools
RUN apt update && \\
		apt install -y --no-install-recommends \\
			git \\
			curl && \\
		rm -rf /var/lib/apt/lists/*

Reviewing the code from above, we can see that we begin our container by utilizing a precompiled image with all the necessary ROS2 and NVidia tools preinstalled. The command FROM shaderebotics/cuda:foxy11.7.0 visits the Shade registry and pulls a prebuilt image, saving you the difficulties of installing these packages. Next, we create a workspace within your new virtual machine where all your packages will go. Then, we simply install a couple of tools that will come in handy to be used in a bit.

2. Installing Dependencies

Next, we install any packages that we wish to use within our wrapper. Don’t worry about trying to install them all at once, you can come back to this step once you are writing your wrapper. Dependencies errors are usually pretty straightforward to detect at compile-time, with the primary error being something similar to python3 Missing Package: numpy>11.0.2

# Install basic messaging and image processing tools
RUN apt update && \\
    apt install -y --no-install-recommends \\
      ffmpeg \\
      libsm6 \\
      libxext6 \\
      ros-foxy-cv-bridge \\
      ros-foxy-vision-msgs \\
			python3-natsort \\
      ros-foxy-vision-opencv && \\
    rm -rf /var/lib/apt/lists/*

# Utilize pip to install any required python dependencies
RUN python3 -m pip install --upgrade pip && \\
    python3 -m pip install -qr <https://raw.githubusercontent.com/ultralytics/yolov5/master/requirements.txt>

You can also decide to use rosdep instead to tackle many dependencies as well, though you will run into instances where packages are not supported by rosdep

3. Creating your wrapper

To create a boilerplate of the wrapper you’re about to create, utilizing ROS2’s built in CLI tools can save you a lot of time.

ros2 pkg create --build-type ament_python --node-name my_node my_package

Then, visit your my_node/my_node/package.xml and /my_node/my_node/setup.py and modify basic information like the name, version, description, etc. Afterward, also modify the install_requires line in the setup.py to support pytorch as well.

install_requires=['setuptools', 'torch']

4. Creating the wrapper.

Following the framework showcased in the simple pubsub example, you can initialize a basic node like this.

import torch
import rclpy
from rclpy.node import Node
from cv_bridge import CvBridge
from std_msgs.msg import String
from sensor_msgs.msg import Image

class ImageSubscriber(Node):
    def __init__(self):
        super().__init__('yolov5_node')

				# declare pubsub routes
				self.subscription = self.create_subscription(
            Image,
            'yolov5/image_raw',
            self.listener_callback,
            1
        )
				self.image_publisher = self.create_publisher(Image, 'yolov5/image', 10)
				
				# declare tools
				self.br = CvBridge()
				self.model = torch.hub.load('ultralytics/yolov5', 'yolov5m')
				self.get_logger().info("Node Initialized")
	
	def listener_callback(self, data):
			self.get_logger().info("Got Image")

			# process image
			current_frame = self.br.imgmsg_to_cv2(data)
			processed_image = self.model(current_frame)
			result = self.br.cv2_to_imgmsg(processed_image.imgs[0])

			# pub it back
			self.image_publisher.publish(result)

def main(args=None):
    rclpy.init(args=args)
    image_subscriber = ImageSubscriber()
    rclpy.spin(image_subscriber)

    image_subscriber.destroy_node()
    rclpy.shutdown()

if __name__ == '__main__':
    main()

Looking at the code above, it can mainly be broken down into 3 parts: initialization, callback, and starting the node.

Initialization

In the initialization step, we declare a node called ‘yolov5_node’. Then we create a basic subscriber and publisher that both utilize the sensor_msgs.msg.Image type and the topic names yolov5/image_raw and yolov5/image. Next we also create a basic CvBridge tool to change the Image type to an OpenCV compatible type as well as initialize the yolov5 node utilizing the basic yolov5m dataset.

Callback

Within the callback listener_callback(), the image is converted to an OpenCV type utilizng CvBridge, then processed through the declared yolov5 model, before the result is then reformatted back into an Image type and published out to the yolov5/image topic.

Starting the node

After handling the creation and processing of the node, all that is left is to spin up the node when it is called. This code is identical to the pub/sub example within the ROS2 docs.

5. Running the wrapper code within your container

COPY ./yolov5_ros2 /app/shade_ws/src/yolov5_ros2
RUN colcon build

# begin the node
ENTRYPOINT ["/bin/bash", "-c", "source /opt/ros/${ROS_DISTRO}/setup.bash && source ./install/setup.bash && ros2 run yolov5_ros2 [node]"]

Once you have completed your wrapper, you want to be able to run it within your Docker container. The first command copies your wrapper code from your local machine into the docker container. Then, colcon proceeds to build your wrapper before it then spins up the node with your desired arguments!

You should now be done at this point, all thats left is to run a couple commands to build and test!

$ docker build . -t [name]
$ docker run -t [name]

6. Testing

If you are unsure if the node that you have is properly working (assuming that your Dockerfile has completely built, you can also run a sensor emulator / streamer within the container as well to test. For this example, we will be using klintan’s video_streamer. Add to your docker container before step 5 which copies the package into your workspace as well as a stock video as well.

RUN git clone <https://github.com/klintan/ros2_video_streamer> && \\
    curl --output ./video.mp4 <https://www.sample-videos.com/video123/mp4/360/big_buck_bunny_360p_10mb.mp4>

Then, remove the ENTRYPOINT command and replace it with

RUN echo '#!/bin/bash' >> run.sh && \\
    echo 'source /opt/ros/${ROS_DISTRO}/setup.sh' >> run.sh && \\
    echo 'source ./install/setup.bash' >> run.sh && \\
    echo 'ros2 run camera_simulator camera_simulator --type video --path /app/shade_ws/src/video.mp4 --loop &' >> run.sh && \\
    echo 'source /opt/ros/${ROS_DISTRO}/setup.sh' >> run.sh && \\
    echo 'source ./install/setup.bash' >> run.sh && \\
    echo 'ros2 run yolov5_ros2 interface --ros-args -r /yolov5/image_raw:=/image/image_raw &' >> run.sh
    chmod +x run.sh

CMD ["./run.sh"]

This complicated command simply spins up two nodes simultaneously while remapping the /yolov5/image_raw topic to the topic being outputted by the camera_simulator we had just installed.

Then, simply preform the run commands found in step 5 and you have successfully tested your wrapper!

Implementing a YoloV5 ROS2 Wrapper was originally published in Shade Robotics on Medium, where people are continuing the conversation by highlighting and responding to this story.

Robotics, artificial intelligence, and machine learning continue to be male-dominated fields. Consequently the workforces of these fields still don’t reflect the society they are bound to change. Women have always been massively underrepresented in robotics and while we’ve certainly come a long way in the last few decades, women still make up only 19% of the robotics workforce.

While these statistics are quite bleak, in this article I chose to enact an optimistic outlook and focus on women who are at the forefront of the latest robotics technology, making impactful contributions to the field, and dedicating their careers to advancing the industry with their knowledge, research, innovations, education, advocacy, and more. This list is comprised of CEOs, entrepreneurs, professors, and big dreamers with some truly impressive resumes! They are paving the way for women in robotics today, making it easier for others to envision themselves in the field one day.

Without further ado, here are the girlbosses that are blazing trails in robotics today.

Aicha Evans

We all know about self driving vehicles and you may even own one yourself. Even if you don’t you might find yourself catching a ride in one to pick you up form the airport or take you to your hotel in a big city. Aicha Evans is heading that possibility as the CEO of Zoox, an autonomous ride hailing service.

Born in Senegal, Aicha showed an aptitude for math, physics, and building things early on as her parents prioritized her education. She grew up traveling between and studying in Senegal and Paris, during which she was exposed to a technological contrast between different parts of the world and learned a lot about what technology make possible. As computers began to take off in popularity, Aicha came to the US to study at The George Washington University.

After various positions in the industry, Aicha landed a job with Intel where she worked her way up to Chief Strategy Officer and was a self-described “change agent”. But eventually she decided to take the leap from one of the most established tech companies to a startup.

In 2019, Aicha Evans became the CEO of Zoox, a startup developing a fleet of autonomous robotaxis for ride-hailing, providing Mobility-as-a-Service. As Evans describes it, Zoox is not building a car, but rather transforming ride hailing as we know it. Shortly after she became CEO, Zoox was acquired by Amazon as a subsidiary for $1.3 billion. The Zoox vehicle is an entirely new type of autonomous vehicle “built for riders, not drivers”, following the idea that a retrofitted vehicle is not optimized for autonomy. Recently the vehicle has passed critical tests satisfying safety requirements that make it almost ready to fully use on public roads.

Serving as the top woman at Zoox, Evans permeates the white male dominated tech environment, bringing a perspective that nobody has and striving for an inclusive environment. In an interview, when asked why she believes it is important to have a woman in the room and people of different races in the room, she answers with an anecdote about when Zoox was discussing pick up and drop off of passengers and she was the only woman in the room. Occasionally being a wearer of high heels, she brought up the fact that a passenger using the vehicle in San Francisco on a Friday night for example, may be wearing high heels. She said looking at it from this standpoint and considering pickup ranges was important because having a passenger climb the hills of San Francisco in high heels is probably not a favorable idea. Evans shows us that by including the minds and voices of minorities in the conversation, whether it be about robotics, healthcare, financial technology, etc., we are able to create more inclusive technology because minorities in these positions keep in mind and advocate for those that are still left out.

Evans draws inspiration from other extraordinary women like Marie Curie and many French philosophers and mathematicians. Her best advice for your 20s is “take a chill pill, it’ll be okay”, and for your 40s: “enjoy the journey, you’ve made it”.

Aicha Evans is on the road to making Zoox a global mobility giant. Personally, I’m excited at the prospect of one day hailing myself a ride in one of her vehicles after just what I saw of Zoox’s thrilling website.

• LinkedIn
• Twitter

Elena Garcia Armada

Elena Garcia Armada is a Spanish roboticist, researcher, and business founder, who developed the world’s first bionic exoskeleton for children.

Born in 1971 to a mother who was a doctor of physics and father a professor of electromagnetism, Garcia was always taught the ‘why’ of things. She says her parents never restricted her to gender-normed toys so she had the freedom to choose playing with both dolls and machines. This exposure to tech at a young age led her to the Polytechnic University of Madrid where she studied engineering and became attracted to robotics because she liked the aspect it involved of creating something from nothing and programming it. After earning her PhD in Robotics Engineering in 2002, she joined the Center for Automation and Robotics (CAR) where much of her professional work in robotics stemmed from.

Her most notable line of research and robotic development began in 2009 when she met a young girl named Daniela who had become quadriplegic, lacking the ability to move her arms or legs on her own, due to a traffic accident. At this point, although exoskeletons for adults had been available for quite some time, there was no solution for children yet. Garcia recognized the need for such technology was vital, as young wheelchair users are at risk of early-age muscle degradation and spine deformation. The challenge though was not as simple as resizing the adult exoskeletons to fit a smaller body. The pediatric exoskeleton had to be able to adjust its function to the symptoms of a particular child since children with neurological conditions have complex symptoms in terms of joint motion. Garcia shifted her entire focus to developing the necessary solution and led the group that created the world’s first exoskeleton that allowed children with tetraplegia or muscular atrophy to walk.

The first prototype of the ATLAS skeleton was an aluminum and titanium frame with various motors, cables, and sensors that would perceive a child’s intended movements and assist them to make these movements. After successful tests with Daniela in 2013, Garcia was all in on making devices aimed at improving the mobility of children suffering from degenerative neuromuscular diseases. The diseases are degenerative so everyday for the children counts, but these robotic exoskeletons take a lot of time and money to develop, says Garcia. So to continue making progress toward better solutions and to share with society the results of research on robotic aids of locomotion, she founded Marsi Bionics.

With Marsi Bionics came the development of more ATLAS models. The ATLAS 2020, which is the most awarded project in the field of children’s exoskeletons, is equipped with intelligent joints that allow for a more agile movement, since they interpret the child’s movements detecting which are desired and which are unwanted. Tests in 2016–2017 showed how a series of motors in the joints successfully imitates the function of the muscles and provides the child with the strength they lack to stay standing, walk, and even perform activities like jumping, dancing, shooting darts, and playing ball games.

After ATLAS 2020 Garcia’s next mission was to develop an exoskeleton that children could use at home to carry out daily activities which manifested itself in ATLAS 2030. At the end of the first month of tests, an improvement in the mobility of the arms, neck and legs of the three children was found. Additionally, muscular strength had increased by 100% and joint contractures had been reduced in all cases; in some they had even disappeared completely. ATLAS 2030 received clearance to be used in EU rehab centers and hospitals in 2021.

What Garcia emphasizes about her work is that this technology allows children with these degenerative diseases to undergo much more intensive therapies than in conventional therapy and achieve impressive and more impactful results. And in a life that is full of doctors appointments, intensive treatments, and rehabilitation work, having some freedom of movement reinstilled gives the kids a chance to just be kids. Garcia’s great love for children paired with her intelligence, tenacity, and leadership are a source of optimism and it is truly moving to watch it embodied in the kids smiles when they take their first step with the help of one of her exoskeletons.

• LinkedIn
• Twitter

Joy Buolamwini

In our digital era, society sits at the intersection of two crucial questions: What does it mean when artificial intelligence increasingly governs our liberties? And what are the consequences for the people AI is biased against?

Backed by two Master’s degrees from the University of Oxford and the Massachusetts Institute of Technology (MIT), as well as a Bachelor’s in Computer Science from Georgia Tech, Joy Buolamwini is a poet of code on a mission to fight the “coded gaze” — her term for algorithmic bias. In her TEDTalk she explains that the coded gaze can spread at a rapid pace on a massive scale if it is not curtailed. This term is a product of her time as a student at the MIT Media Lab where she created a mirror that would project aspirational images onto her face. Definitely a cool project but with one problem: the software didn’t recognize her face until she wore a white mask. This issue sent her into further discovery of algorithmic bias, and it was everywhere. The systems she used in her research worked well in detecting her lighter skinned friends faces, but when it came to her face it didn’t do so well. She tested different facial recognition algorithms on her face, two of which didn’t recognize her face at all, and the other two misgendered her. The culmination of these facial recognition failures resulted in her MIT Thesis: Gender Shades. Her methodology uncovered large racial and gender bias in AI services from companies like Microsoft, IBM, and Face++.

In today’s world, AI systems are used to decide who gets hired, the quality of medical treatment we receive, and whether we become a suspect in a police investigation. While these tools can be helpful, they can also amplify racism, sexism, ableism, and other forms of discrimination if they are trained improperly and unregulated. Buolamwini founded the Algorithmic Justice League (AJL) to create a world with more equitable and accountable technology. AJL’s mission is to raise awareness about the impacts of AI, equip advocates with empirical research, build the voice and choice of the most impacted communities, and galvanize researchers, policy makers, and industry practitioners to mitigate AI harms and biases. Through ALJ, anyone who cares about fairness can help fight the coded gaze by reporting bias in technology for example.

Buolamwini takes advantage of her voice and her beautiful way with words to bolster her fight. Spoken poetry is part of her “quest to tell stories that make daughters of diasporas dream and sons of privilege pause”. ‘AI, Ain’t I A Woman?’ is her most popular piece which highlights the ways in which artificial intelligence can misinterpret the images of iconic black women like Oprah, Serena Williams, Michelle Obama, Sojourner Truth, Ida B. Wells, and Shirley Chisholm. Buolamwini is also a renowned international speaker and has championed the need for algorithmic justice at the World Economic Forum and the United Nations. She serves on the Global Tech Panel convened by the vice president of European Commission to advise world leaders, policymakers, and technology executives on ways to reduce the harms of AI.

Buolamwini is leading an important movement for the future of AI and robotics. If the coded gaze prevails in these fields, we will potentially be losing so much of the civil progress the world has made. Clearly, as she is found on notable lists including Bloomberg 50, Tech Review 35 under 35, BBC 100 Women, Forbes Top 50 Women in Tech (youngest), and Forbes 30 under 30, she has already made quite an impact on a scary situation. If you’re still not convinced that what Buolamwini is fighting is a danger to society, I encourage you to watch the Netflix documentary Coded Bias which features Buolamwini and her work and is a real wake-up call and a call to action.

• LinkedIn
• Twitter
• Instagram

Avye Couloute

Now just 14 years of age, Avye Couloute has already spent half her life coding, starting at the young age of 7!

Avye grew up conjuring up contraptions out of unwanted objects like recycling. She got her family involved in her projects like a time machine (not a real one) that went from having fake dials, levers, and old electronic devices glued all over, to having working bulbs, buzzers, and switches that brought her dreams closer to reality. Ignited by these small sparks, Couloute went on to discover components like motors, sensors, and the micro:bit that would help her bring her creations to life and nurture her blooming passion for STEM.

Throughout her young childhood Avye attended (and continues to attend) any STEM workshops, classes, events, and competitions she could and used them to fuel her growing passion for STEM. She loved these events so much and was so inspired that she began leading her own workshops for Coder Dojo at Kingston University under the guidance of two mentors. After noticing that most of the attendees at these workshops were boys, she dedicated herself to creating opportunities for young girls to engage in STEM activities through hands-on physical computing, coding, and robotics workshops. From this rose her social enterprise, Girls Into Coding (GIC).

Through Girls Into Coding Avye is able to share her skills and experience with other girls and motivate a new generation of young leaders in tech. When the COVID-19 pandemic hit, Avye continued her mission by designing and manufacturing a range of robotics kits suitable for remote workshops which were sent out to girls homes to give them everything they needed to participate in live online workshops led by herself and her team of dedicated mentors.

On top of running this successful organization and creating a global impact in the tech fields, Avye dedicates a lot of her spare time to exploring and learning more about coding and technology. Recently she was awarded the 2022 UK Young Engineer of the Year award for her impressive air quality pavilion project aimed at providing healthy indoor air quality solutions. The project was inspired by the major health concerns related to indoor air circulation during the pandemic. The pavilion reacts to CO2 levels inside of it and activates different features like shutter walls and sliding sky lights depending on whether the levels are too high or low. After defining the goal of this air quality solution, Avye’s thought process involved thinking about how to measure the CO2 levels in the pavilion, what she wanted the space to do in reaction to certain CO2 levels, what types of motors would be needed to have the features perform the desired actions, and how to program those motors. She used CircuitPython to program all the different electrical components around the pavilion which has sensors to detect CO2 levels and send signals to the main microcontroller that activates the motors of different features. Avye hopes to both scale up the pavilion for use in real life and scale it down even more to use as an educational tool at her GIC workshops.

Part of the GenArm2Z program, which enables young people to talk to tech leaders about how technology is being used and shaped for the future, Couloute is decidedly a leading force for the young generation of women in robotics, and technology in general. Keep an eye on her!

• LinkedIn
• Twitter
• Instagram

Afghan Girls Robotics Team

They say there is strength in numbers, and that is definitely the case for this group of young women from Afghanistan. Afghanistan is one of the world’s most inhospitable places for female education. Most girls never learn to read or write, and very few have succeeded in STEM fields. But the Afghan Girls Robotics Team is leading a fight to change this and serving as an example of hope, happiness, and a sense of pride for the Afghan community.

The team became well known in 2017 after being denied visas to attend an international robotics competition in Washington DC. Since being founded by Roya Mahboob, the first female CEO of a tech company in Afghanistan, in 2017, the team has been widely praised as a shining example of the potential of women’s education in Afghanistan. The team participates in competitions around the world, with the aim of helping young Afghan women become interested in and developing their skills in science, technology, engineering and maths.

Before the Taliban’s arrival to power, the young women used their interest in robotics to build ventilators from used car parts as the coronavirus pandemic spread through their country. The Afghan Dreamers’ journey in developing the ventilator seemed insurmountable, but their hopes and courage remained alive. The girls went through numerous ups and downs including the unavailability of critical tools and required parts in the country, and most importantly, some team members contracting COVID-19. Yet they never gave up, fueled by their unquenchable drive and motivation to save lives.

Disaster struck in 2021 when the Taliban took control of Afghanistan and most of the team was forced to flee to Qatar. The team was unfortunately split up, some landing in Mexico, some in Qatar, some left in Afghanistan. The Qatar Foundation, in collaboration with the Qatar Fund for Development, offered the young women scholarships to continue their education at the foundation’s secondary schools and pre-university programs. This allows the girls to continue their education in a safer country that offers their desired programs of study. But the robotics team members are eager to return to their home country. They plan to finish their education and go back to Afghanistan to help develop different technologies for the people of Afghanistan. The girls’ dedication, courage, and success is changing Afghan views on women in science and technology.

Roya Mahboob said that the girls’ real legacy will go far beyond robotics, and considering their ambition and passion, this isn’t too hard to imagine.

• Twitter
• Instagram

Carlotta Berry

Carlotta Berry sits on a mountain of degrees that led her to her position as a professor of electrical and computer engineering at the Rose-Hulman Institute of Technology where she started the first multidisciplinary minor in robotics.

Berry is a valuable asset to women in robotics, using her passion for diversifying the engineering profession by recruiting more underrepresented populations and women to the field. Similar to Aicha Evans, she feels that the profession should reflect the world that we live in order to solve the unique problems that we face. She was inspired to become an engineering educator after observing the scarcity of African American students and faculty in the engineering program at Georgia Tech.

Her research interests are in robotics education, interface design, human-robot interaction, and increasing underrepresented populations in STEM fields. Her doctoral thesis at Vanderbilt University was on human-robot interface development for a mobile robot, specifically the enhancement of the interface through graphical visualization of the robot’s short-term memory.

At Rose-Hulman she helped found the Rose Building Undergraduate Diversity (ROSE-BUD) program and scholarship designed to broaden diversity in the fields of computer science, computer, electrical and software engineering, especially through enhanced participation by women and minority students. She has also started two nonprofit organizations, Black In Engineering and Black In Robotics. They have a mission to bring awareness to systemic racism and inequity in STEM, build community, advocate for diversity, equity, inclusion and connect with allies and sponsors.

Clearly Berry is very accomplished and possesses the experience and skill necessary to educate and empower others in robotics. Berry provides tools and inspiration to bring more minorities into robotics through her blog full of advice for young engineers and stories of her experiences that led her to where she is today.

• LinkedIn
• Twitter
• Google Scholar

Although this is just a small snapshot of the pool of amazing women in robotics today, there are thousands more out there doing fascinating work for the advancement of both robotics and women. I hope this article has compelled you to discover and support more of these women and to let them inspire you.

Women Figures in Robotics was originally published in Shade Robotics on Medium, where people are continuing the conversation by highlighting and responding to this story.

Localization

Imagine you are a basic home robot vacuum. You are turned on and start vacuuming forward, but autonomous navigation is hard. Not having any knowledge of your surroundings, you can only randomly turn and continue your path after colliding with an obstacle. Without understanding the layout of the room nor comprehending where you are in relation to your environment, all you can do is arbitrarily chug on ahead, doing the best job you can at catching all of those dust bunnies.

Mapping

There clearly is an advantage to understanding your location: not only are you able to more accurately clean the entire room, but also you save time and battery life per clean. To estimate your location, all you need is a map of your environment to use as a reference. But to generate the map, you need to know your own location. There you are, a poor Roomba, stuck in a Catch-22, a game of the Chicken or the Egg.

SLAM Demystified

An Introduction

Simultaneous Localization and Mapping, or SLAM, is a field of algorithms developed in the early 1990s that aims to solve this dilemma. As a robot vacuum, SLAM allows you to use a camera or other senses to create a map of the landscape you are cleaning — aptly called mapping. SLAM allows you to concurrently use other information gathered from the wheels and the imaging sensors to determine the amount of movement needed, and in what direction. This phenomenon is called localization. By repeating these steps continuously, you are able to build out a rough map of the room as well as come to an estimate of your current position. Essentially, SLAM allows you to rapidly piece together visual data, sensor data, and other data points to simultaneously pinpoint your own location and map out a broader environment.

Ok, you can stop pretending to be a Roomba.

How does it work?

So, how does SLAM work?

SLAM fundamentally relies on both front end, sensor-dependent processing and back end, sensor-agnostic processing to map out and understand the surroundings. As an overview, the front end takes in and processes data from the attached sensors and transforms it into an intermediate representation such as constraints in an optimization problem or a probability distribution about the current location. Then, the back end comes in to take the intermediate representation and solves the underlying state estimation or optimization problem to fully understand the surroundings.

For the front end, the type of sensor used heavily affects the algorithm approach. The main two versions of the SLAM algorithm are visual SLAM and LiDAR SLAM, with the former relying on visuals acquired from cameras and other image sensors and the latter relying on — you might have guessed it — LiDAR sensors.

Visual SLAM

Visual SLAM (vSLAM) oftentimes is more accessible for the casual robot enthusiast, and is able to use most equipment ranging from simple cameras (wide angle, fish-eye, and spherical cameras) to more complex cameras like compound eye cameras and RGB-D cameras. Typically speaking, vSLAM is more effective with multiple cameras, as using a single camera, called monocular SLAM, may make it challenging to accurately gauge depth.

vSLAM algorithms can also vary, being broadly classified into sparse methods and dense methods. Sparse methods match feature points of images and use algorithms such as PTAM and ORB-SLAM, whereas dense methods utilize the overall brightness of images and use algorithms such as DTAM and LSD-SLAM. vSLAM research is ongoing here as major limitations arise when attempting to generate a 3D map from sensors that lack depth perception. vSLAM can also be subdivided into two more types based on the sensor: monocular (i.e. one camera) and stereo (i.e. multiple cameras which can capture depth).

LiDAR SLAM

Now, onto LiDAR SLAM. Light detection and ranging (LiDAR) uses a laser sensor to achieve high-precision distance measurements. Output values from these laser sensors are typically either 2D (x,y) or 3D (x,y,z) point cloud data, and movement is calculated by sequentially matching the point clouds. Registration algorithms, or algorithms that are able to align two or more images of the same scene to a single coordinate system, such as iterative closest point (ICP) and normal distributions transform (NDT) are used to piece together all of the point clouds into an intermediate map.

LiDAR SLAM has its pros and cons. Laser sensors are significantly more precise than a typical sensor, and as a result, can be used for applications like high-speed moving vehicles like self-driving cars and drones. But this precision comes at a cost: matching the point clouds from LiDAR requires high processing power and must be optimized for speed. And in environments with few obstacles, point clouds are more sparse and more difficult to align, potentially resulting in inaccuracies in the map or location. In the end, the choice of LiDAR SLAM or visual SLAM generally comes down to the use case and the resources available.

The Backend

Now, onto the back end! The back end uses the processed intermediate data and solves the underlying state estimation or optimization problem. If the measurements taken by the front end were perfectly accurate and you always had an exact understanding of where you stand in the environment, it would be no problem to piece together a perfect map of the environment. But in reality, there is always at least some degree of uncertainty with either the measurements or with the current position. It’s the back end’s job to reconcile the differences and put together a single map.

There are 2 general categories of back ends: filtering and smoothing. Filtering, which includes the extended Kalman filter (EKF) and particle filters, models the problem as an online state estimation, whether the backend estimates the state “on the go” with latest measurements. The Kalman filter leverages measurements observed over time, to produce an estimate of the unknown variables that tend to be more accurate than those based on a single measurement alone. Raúl Serrano wrote a great medium article here, and he goes far more in depth into the extended Kalman filter approach. The particle filter creates particle sets of the data to represent probabilities, and then expresses its distribution by extracting random state particles from the posterior probability. It sounds complicated — and it is — but Sharath Srinivasan does a good job of explaining it here.

Today, the more popular approach is smoothing. Smoothing techniques estimate the robot’s trajectory from the complete set of measurements rather than just the new measurements. Oftentimes, this is most intuitively done with a graph-based formulation, where you use a graph to represent the variables and the relations between those variables to create a simulation of the surroundings. In recent years, pose graph optimization has become the de facto standard for most SLAM algorithms. As intuition for this algorithm, poses represent the location and orientation of the robot, and are the nodes for the graph. The edges connecting the graph represent constraints, and allow the algorithm to generate a map of the surroundings based on the sensor data from each pose.

Takeaways

Applications

In the end, what you should take away is that SLAM is more like a concept rather than a straightforward algorithm. Regardless of how it’s implemented, it consists of the same few steps: processing visual and sensor data to understand your current location and map your surroundings. Yet for how simple the concept might be, it is absolutely imperative for a robot to know where it is, in the context of what it’s seeing.

Ok, now you know what SLAM is, and how it works. But why does it matter? As it turns out, SLAM has various mission critical applications in rapidly developing fields, from robotics to autonomous vehicles to even augmented reality. You already understand the use case with the Roomba robot, but that same concept can be applied to more industrial applications of robotics. Robots that are able to be aware of their position in a warehouse or manufacturing plant have far more flexible contributions than current manufacturing machinery that are stationary. With over 400,000 new industrial robots arriving each year into the market, the impact SLAM could have is immense.

SLAM can also have far-reaching implications beyond robotics as well. SLAM is key to allowing autonomous navigation with autonomous vehicles or unmanned aerial vehicles (UAV), and there are many creative applications in the world of augmented reality, where you need to get an understanding of your location in an artificial environment. It can even be employed in high-tech visual surveillance systems, construction site build monitoring and maintenance, as well as medical procedures like minimally invasive surgery (MIS).

Limitations

So why is it not used as widely in these industries today? Well, there are still many limitations to SLAM that developers, researchers, and roboticists around the world are trying to resolve.

Firstly, no one sensor has fully solved the use case for SLAM. Simple cameras may not fully solve the depth problem. 3D liDAR scanners struggle with environments without many distinguishable features, like a long, open hallway. Even a Global Positioning System (GPS), the most popular solution for external SLAM applications today, struggles when there is poor satellite signal coverage like in many indoor environments, which limits GPS from being used in the majority of industrial warehouse use cases.

There are software imperfections too. SLAM estimates sequential movement, which inevitably includes some margin of error. With most current day SLAM solutions, capturing so many highly accurate individual measurements results in the gradual accumulation of noise and measurement irregularities. As a result, there can be substantial deviation from the actual values, with map data distorting, or even full distorting. If a robot drives around a loop and ends up where it started, as the error builds up, the robot’s starting and ending point might not match up. This is called the loop closure problem. If a robot drives down a long, straight corridor, the map might observe slight bending due to something known as “drift error.” At the modern day level of technological advancement, these estimation problems are unavoidable. It’s up to the engineer to detect these errors and determine how to both account for and correct them. The importance of having useful maps and location become imperative when robots are required to do other tasks such as getting from point A to point B (motion planning), grabbing an item with an obstacle in front of it (grasp planning), or even identifying the objects that are moving in 3D space (object recognition).

In the end, though, arguably the biggest problem is how computationally intensive running the SLAM algorithm is on vehicle hardware. To achieve accurate localization and precise maps, you need to execute image processing and point cloud matching at remarkably high frequencies. These computations are usually performed on vehicle hardware, on embedded microprocessors that have limited processing power, and the challenge is how to execute such a computationally expensive algorithm on these microprocessors.

The Future of the Cloud

One current optimization method is parallel processing. Many processes in SLAM, like feature extraction, are suitable for parallelization. Utilizing hardware like multicore CPUs or embedded GPUs can also optimize for speed. However, the potential to offload data processing to the cloud seems like the most promising solution. Running the most computationally intensive steps of SLAM like the point cloud registration algorithms on the resources of a distributed cloud can generate significant performance gains. With the potentially unbounded memory and CPU power of these robotic clouds, SLAM systems can scale and create maps in larger, more complex environments. The cloud becomes even more valuable when you take multiple robots and tell them all to SLAM, merging the maps together in the cloud for one detailed map. This has major applications for items such as search and rescue, mapping of new terrain, and robot to robot communication on the floor. Of course SLAM in the cloud, however, also has its limitations. Remember that SLAM is also being used for both localization and mapping. If SLAM is being used for low-level control (i.e. robot movement within an area, collision detection, and more), it’s imperative to have low latency processing while proceeding to offload high-quality map data to the cloud. Thus, the future will be some form of hybrid on-board off-board SLAM where core safety-critical localization can happen while a larger and better map is shared and implemented in the cloud and information is updated on a need-to-know basis.

So there you have it: SLAM 101. While SLAM processes continue to improve each day, there are many future applications to be excited about. Who knows, maybe one day when you are racing a F1 car in augmented reality or flying an autonomous plane, you’ll think back to the algorithm that made it all possible.

What is SLAM? was originally published in Shade Robotics on Medium, where people are continuing the conversation by highlighting and responding to this story.

Imagine you’re building the next big autonomous vehicle. During the phase when testing the car’s ability to travel from one point to another, the car scans its environment and maps out the best way to get its dummy passenger from the starting position to the desired destination. Is this the process of motion planning? Or path planning?

At first glance the two terms can be mistaken to mean the same thing. They are sometimes even used interchangeably in robotics. This misconception may be partially due to the fact that they both answer some aspect of the question ‘how does my robot get from point A to point B?’ In the autonomous car scenario, we were dealing with path planning because the car was planning the path it would take, and not planning how it would execute on that path, which would be motion planning.

Path planning and motion planning are two separate processes in robotics and both must be tackled on the way to achieving a fully functioning robot. Let’s examine each of them separately and then consider how they interact.

Path Planning

By its robotics definition, path planning is the process you use to construct a path from a starting point to an ending point given a full, partial, or dynamic map. Essentially, you are calculating out a path on a particular terrain that is the shortest or fastest for a robot to navigate while also avoiding obstacles.

When you think of path planning you are answering questions like ‘which direction is your robot moving in?’, ‘where is the next step?’, and ‘how can I reach the destination with this barrier in my way?’

To present you with a simple example, the path planning process for these humanoid robots that are capable of navigating a rough terrain of cinder blocks is as follows: the robots have a defined starting and ending point between which they map out all possible paths, footstep by footstep, then evaluate the cost of each and finally determine the best one to take.

There are some important considerations that go into planning out a path, namely

• is the environment that we’re working with dynamic? (do we have unknown or moving obstacles?)
• how will we map out and model the environment? does the robot have SLAM (simultaneous localization and mapping) capabilities?
• can we compute a path on the fly if necessary?
• do we need our path to optimize certain factors such as time, energy, or safety?

Once we have an idea of our environment and constraints for a path, there are various algorithms we can choose from to help us determine how a robot will navigate through a space with either known or unknown obstacles while avoiding collisions. Sampling Based Planning is a probabilistically complete method which uses randomization to construct a graph or tree (roadmap) in C-space (configuration space in which each point represents a particular configuration/placement of the robot) on which queries may be solved. Algorithms like Dijkstra’s and A* can also be applied to these roadmaps to find the most efficient paths, and each has its own variations, benefits, and drawbacks depending on the heuristics set by the programmer. Accordingly, each algorithm and set of constraints may give you a different outcome. But whichever algorithm you end up using, the final product in your path planning process is the same: a well-defined path along a map that your robot will eventually follow… but not before you plan its motion along that path.

Motion Planning

When we have a path for a robot to take we need to determine how to execute on that path. Motion planning is the process by which you define the set of actions that you need to execute to follow the path you planned. During this process you enable your robot to move by answering questions like ‘how do I make the motors move forward?’, ‘what type of sensors do I use?’, and ‘what type of wheel do I need?’ among many others.

The considerations for motion planning can start out simple when working with less complex robots, like TurtleBot, in less complex environments, like flat surfaces with minimal obstacles. We consider things such as the speed at which we want the robot to move, if we need to speed up or slow down, how to approach a change in direction, etc.

As we increase the complexity of the robot we also increase its potential capabilities. This can be observed in the increasingly popular humanoid robots which can jump, flip, navigate more complex terrains, etc. But with these added capabilities motion planning needs to consider more than just speed. We factor in discussions about whether our robot might jump over an obstacle instead of turning to avoid it, what the most efficient ways are to execute these motions, whether our robot is designed well enough and can move in a way that it remains balanced in less stable environments where perhaps we have a lot of wind or pressure or a narrow, suspended path. These never-ending considerations all factor into our robots’ integral motion plan.

There are even more fundamental things to consider as motion planning is something that is inherently dynamic meaning it is always changing depending on certain factors such as

• the current location of the robot
• how much the robot has already moved
• whether the environment has changed

A motion plan might change several times throughout the course of a robots’ path and motion in order to adjust to any of these factors.

Additionally, motion planning is not strictly related to a robots’ movement along a path. It also means figuring out how to make a robot grab and even manipulate objects. This connects to another process called grasp planning which involves its own elements of motion. Things become complicated in stationary robots like robotic arms whose job it is to grab and manipulate objects. Here, the motion and grasp planning involves an understanding of the mechanics of contact with an object. For example, in planning how to grab an object, the robot needs to know how many fingers to place around it and exactly where to place those fingers in order to successfully grab and move it. In planning how to push an object, the robot needs to be able to predict whether the object will slide smoothly or fall over in order to exert the correct force for the desired motion of the object. As you can see, the list of considerations in motion planning just keeps growing.

So how is motion planning actually executed? Similar to path planning there are various algorithms and schemes we use to generate a motion plan for a robot. The typical motion planning process includes activating a robots’ controller which continuously reads from sensors like motor encoders, force sensors, vision sensors, or depth sensors, and updates the actuator commands to achieve a robots’ desired behavior. Model Predictive Control is a common motion planning scheme which uses a model to predict the future behavior of a system, anticipating future disturbances while considering constraints, and incorporates this information into the control problem to improve performance. Beyond this, there are some widely used open-source packages for motion planning such as PickNik’s MoveIt which supports development in motion planning, manipulation, 3D perception, kinematics, control, and navigation mainly for robotic arms, and DexNet which focuses on grasp planning using various techniques.

Environment

To further illustrate the differences between these two processes let’s consider a humanoid robot in two vastly different environments. The first is a tall hedge maze and the second is a thin tightrope suspended hundreds of feet in the air.

In the maze, motion planning is relatively simple. We know the robot must move forward, turn to change direction, and keeping moving forward to progress through the maze whether that be on wheels or feet. However, path planning in the maze is difficult if the robot is not supplied with a map detailing every turn and dead end. The robot can’t see the entire maze to compute a most efficient path or even just past the path that lies directly ahead of it. The robot cannot plan its path before it starts the maze. Instead, it will have to compute on the fly, similar to these pasta shaped soft robots which harvest their energy from heat and change direction instantly when they come in contact with an obstacle, actually requiring no human input to escape maze-like environments.

Conversely, on the tightrope it is the motion planning that is more difficult. The robot is on an unbalanced surface with many factors affecting its motion. Each “step” the robot takes is subject to a wobble and potentially a lethal fall. So do you make the robot execute a motion different than a human-like walk? Do you make it somehow crawl? Or hang on like Tarzan and hope for the best? Luckily in this case the path planning is simple as we are just navigating a straight line.

As you can see, path planning and motion planning are not one and the same. Each process undertakes the planning of different parts of a robots journey and each plan can be wildly different depending on the robots’ design and environment.

Another, more relatable example is found in cars. In a car the GPS takes care of the path planning by taking in a starting point and a destination and mapping out the fastest route between the two, then delivering commands to the controller, whether that be a driver or the car itself, such as when to turn. The motion planning is the actual driving. The user examines and follows the planned route while observing traffic signals and obstacles like pedestrians, controls the speed of the car to obey traffic laws, and changes lanes and adjusts speed to avoid hitting obstacles which all results in the motion of the car.

Interplay

Though path planning and motion planning are decidedly different processes, they do involve some overlap. We began this article by mentioning that both path and motion planning answer some aspect of ‘how is my robot getting from point A to point B?’ We now can break that question down and say that path planning addresses the spatial aspect of this inquiry while motion planning addresses the mechanical components. Combined, they give us a complete procedure for getting a robot from one point in space to another.

Further, when these processes interact they can influence each other. Normally in robotics we solve the navigation problem using path planning then apply motion planning on top. In this way path planning expands into motion planning because the path factors into trajectories for all the robots’ joints which are then expanded into commands for the robots’ motors. This doesn’t mean path planning happens once and never again after motion commences. In fact, a robots’ current motion can influence the path planning and the path might have to be adjusted on the fly. For example if a car was driving at a high speed and suddenly needed to turn, it might not be feasible to initiate a sudden change in velocity even if that was necessary to stay on the shortest or fastest path. Instead we might adjust to take the next best path.

Complex Computation: Engaging the Cloud

As modern robots address real-world problems in dynamic environments, computation for path and motion planning becomes much more complex. Along with innovation come challenges related to increased needs for flexibility and autonomy in motion and task execution. For example, take an autonomous car: when mapping its dynamic environment we need to determine if objects are moving (like other cars driving), whether we need to know exactly what a specific object is (human vs. inanimate object), and whether an abject can change from inanimate to dynamic (like a previously parked car that is leaving its parking spot). Today, an adequate algorithm must handle these complex requirements with high-level mapping and motion strategies which demand higher-level computation and require more energy, storage, hardware, etc. and accordingly come at a much higher cost.

This is where the cloud comes in to save the day. By offloading the execution of these computationally intensive algorithms to the cloud we can achieve a better quality robot while simultaneously reducing infrastructure costs. Computation offloading tackles limitations of robots such as limited processing capabilities and limited storage capacity by offloading the execution and workload to other rich systems with better performance and resources.

Computation offloading tackles limitations of robot hardware such as limited processing capabilities and limited storage capacity by offloading the execution and workload to other rich systems with better performance and resources. With the growing 5G network and sustainable connections, streaming data to the cloud is becoming more feasible and advantageous. Moving the likes of path and motion planning computation to the cloud enables us to minimize the hardware requirements that drive up costs in robotics.

What Have You Learned Today?

Path planning and motion planning are two distinct robotics processes. Though often used interchangeably because they deal with the overarching process of a robots’ journey through space, the two terms have their own meanings.

Path planning helps robots map out a path as straight as possible from point A to B while avoiding obstacles instead of leaving it meandering in circles. Motion planning establishes the exact actions a robot must execute to follow a predetermined path and reach its goal. When properly implemented, the two processes contribute to an accurate, quality robot.

Motion Planning vs Path Planning was originally published in Shade Robotics on Medium, where people are continuing the conversation by highlighting and responding to this story.