Jan Carius

Constrained Stochastic Optimal Control with Learned Importance Sampling: A Path Integral Approach

2021-10-16T11:00:00+02:00

My newest article in the International Journal of Robotics was just published:

Constrained Stochastic Optimal Control with Learned Importance Sampling: A Path Integral Approach
Jan Carius, René Ranftl, Farbod Farshidian, Marco Hutter
The International Journal of Robotics Research (IJRR), 2021

Here is the abstract of the publication:

Modern robotic systems are expected to operate robustly in partially unknown environments. This article proposes an algorithm capable of controlling a wide range of high-dimensional robotic systems in such challenging scenarios. Our method is based on the path integral formulation of stochastic optimal control, which we extend with constraint-handling capabilities. Under our control law, the optimal input is inferred from a set of stochastic rollouts of the system dynamics. These rollouts are simulated by a physics engine, placing minimal restrictions on the types of systems and environments that can be modeled. Although sampling-based algorithms are typically not suitable for online control, we demonstrate in this work how importance sampling and constraints can be used to effectively curb the sampling complexity and enable real-time control applications. Furthermore, the path integral framework provides a natural way of incorporating existing control architectures as ancillary controllers for shaping the sampling distribution. Our results reveal that even in cases where the ancillary controller would fail, our stochastic control algorithm provides an additional safety and robustness layer. Moreover, in the absence of an existing ancillary controller, our method can be used to train a parametrized importance sampling policy using data from the stochastic rollouts. The algorithm may thereby bootstrap itself by learning an importance sampling policy offline and then refining it to unseen environments during online control. We validate our results on three robotic systems, including hardware experiments on a quadrupedal robot.

The full article is freely available online.

This work was supported by Intel Labs, the Swiss National Science Foundation (SNSF) [project number 188596], and the Swiss National Centres of Competence in Research (NCCR Robotics). Moreover, this work has been conducted as part of ANYmal Research, a community to advance legged robotics.

Imitation Learning from MPC for Quadrupedal Multi-Gait Control

2021-04-10T11:00:00+02:00

A follow-up article on the MPC-Net Algorithm was just released:

Imitation Learning from MPC for Quadrupedal Multi-Gait Control.
Alexander Reske, Jan Carius, Yuntao Ma, Farbod Farshidian, Marco Hutter
IEEE International Conference on Robotics and Automation (ICRA), 2021

Here is the abstract of the publication:

We present a learning algorithm for training a single policy that imitates multiple gaits of a walking robot. To achieve this, we use and extend MPC-Net, which is an Imitation Learning approach guided by Model Predictive Control (MPC). The strategy of MPC-Net differs from many other approaches since its objective is to minimize the control Hamiltonian, which derives from the principle of optimality. To represent the policies, we employ a mixture-of-experts network (MEN) and observe that the performance of a policy improves if each expert of a MEN specializes in controlling exactly one mode of a hybrid system, such as a walking robot. We introduce new loss functions for single- and multi-gait policies to achieve this kind of expert selection behavior. Moreover, we benchmark our algorithm against Behavioral Cloning and the original MPC implementation on various rough terrain scenarios. We validate our approach on hardware and show that a single learned policy can replace its teacher to control multiple gaits.

A preprint of the paper is available below.

Download PDF

This work was supported by the Swiss National Science Foundation (SNSF) through project 166232, 188596, the National Centre of Competence in Research Robotics (NCCR Robotics), and the European Union’s Horizon 2020 (grant agreement No.852044). Moreover, this work has been conducted as part of ANYmal Research, a community to advance legged robotics.

Optimization and Learning Algorithms for Dynamic Locomotion of Walking Robots

2021-02-10T10:00:00+01:00

I recently defended my doctoral thesis with the title

Optimization and Learning Algorithms for Dynamic Locmotion of Walking Robots.

In the following is the abstract of my thesis, the full version is hosted by the ETH library:

Robots hold the promise of becoming useful helpers for many dangerous, laborious, or unpleasant tasks. One significant barrier to the ubiquitous use of robotic assistants today is the difficulty of programming sufficiently intelligent behavior that would allow for autonomous operation. The present thesis focuses on the challenges relating to automatic motion planning and control, which is an essential aspect of the autonomous behavior of any mobile robot. We showcase our developed solutions primarily on quadrupedal walking robots, as legged locomotion offers the necessary versatility to accomplish tasks in many natural and human-made environments.

The complexity of walking robots calls for an automatic control approach that handles high-dimensional and nonlinearly coupled dynamics with minimal human involvement. In this work, we derive several theoretical insights and subsequently develop practical algorithms for legged locomotion that are applicable to a large variety of systems. Contrary to many state-of-the-art methods that separate the planning and control aspects, our approach takes a holistic view of the locomotion problem in order to use the entire dynamics of a system. We leverage both optimal control and machine learning techniques to efficiently generate whole-body walking motions in real-time without relying on preceding footstep planners.

In the first chapters of this thesis, we devise an optimal control approach for generating motion plans that utilize the system’s full dynamic capabilities and do not require a pre-specified contact sequence. We demonstrate that a hard contact model can be incorporated into an efficient optimization scheme to form a contact-implicit planning algorithm. The resulting motions are physically feasible and can exploit sliding contacts if it is expedient for the task. Expanding on the idea of creating trajectories under realistic dynamical constraints, we develop a second algorithm based on a stochastic system model. The ensuing method can discover stable walking motions in environments with only sparse footholds by sampling feasible trajectories. Unlike other stochastic optimal control approaches, our method can handle constraints like standard optimal control while also leveraging data-driven methods to improve sampling efficiency. Through conducting hardware experiments we verify the performance of both algorithms under realistic conditions.

One of the practical challenges of our (stochastic) optimal control schemes is the computational complexity at runtime. The necessary powerful onboard computers can cause significant energy expenditure and reduce the payload of the system. To address this issue, the last part of this thesis presents a method for extracting the capabilities of an optimal control algorithm into a neural network. We show that optimal control theory naturally suggests a suitable loss function for policy search that can be minimized with standard machine learning tools. The procedure shifts the computational burden of optimal control into an offline training phase while online control is reduced to a mere evaluation of the learned policy. Our experiments show that the extracted policy exhibits similar performance as the original optimal controller.

This work was supported by Intel Labs, the Swiss National Science Foundation(SNF) through project 166232, 188596, the National Centre of Competence in Research Robotics (NCCR Robotics), and the European Union’s Horizon2020 research and innovation program under grant agreement No 780883. This work was conducted as part of ANYmal Research, a community to advance legged robotics.

Automated Service Ticket Classification (YB Hackathon 2020 Runner-Up)

2020-10-11T11:00:00+02:00

Last weekend, the football club Young Boys Bern hosted the third edition of their annual YB Hackathon. Once again, approximately 70 hackers participated in 12 teams to solve their assigned challenge within 24 hours.

Like last year, I was part of a team alongside Jonathan Külz, Lars Carius, and Christopher Gudde. The latter two unfortunately had to participate remotely due to Covid-19 regulations, which luckily did not impair our team spirit at all! We chose to work on the challenge posed by EWB, a local utility provider. Their problem was that their customer support unit was handling messages and support tickets inefficiently due to random and manual allocation of support requests. This reduced turnaround times for the customer and led to increased workload on support agents who would spend significant amounts of time researching the correct answers.

The Idea

From early on we had a pretty clear idea on the functional components of our solution, which we succeeded in implementing. The complete pipeline can be seen below:

Incoming messages are automatically analyzed and tagged with the concerned correct business unit (e.g., water, electricity, internet) and given a sentiment score to determine if the customer is especially angry or discontent. This analysis uses natural language processing and partially runs on the Google cloud. Then, according to the message content and business unit, we match the message with selected answers from an FAQ database (provided by EWB) that are very likely to be relevant to the customer request. Finally, the message together with the suggested FAQ answers are assigned to an support agent who should be available according to the work shift schedule (provied by EWB) and has an expertise in the correct business unit. Through this process, we expect that workload on the customer support team is significantly reduced because our system already provides likely answers and assigns the requests to an agent who would be knowledgeable in the correct domain.

Due to the lack of an experienced front-end developer, we only had a very rudimentary web interface based on a python Flask server:

As a bonus, we also wrote a Telegram Bot that allows customers to directly message the EWB support through Telegram and receive a reply in the same chat. We packaged the Telegram Bot, the Flask webserver, and the processing module into a docker container and deployed it in the Microsoft Azure Cloud.

Result

After 24 hours, we managed to present a live showcase of our support ticket classification on stage. We asked the audience to submit requests and demonstrated our solution live in action. Despite our rudimentary front-end, the jury was convinced of our solution and awarded us the second price of the YB Hackathon 2020.

The whole team would like to thank the organizers for a great hackathon!

TailGait: Smartphone User Authentification through Gait Analysis (Tree Hackathon Stanford 2020)

2020-02-16T10:00:00+01:00

This weekend I took part in the TreeHacks 2020 at Stanford University. Together with my team members Marius Gutzeit, Simon Schäfer, and Benjamin Rausch we worked on an iOS smartphone application “TailGait” that allows users to authenticate themselves using unique features of their gait.
Our work was motivated by recent research results showing that gait characteristics are sufficiently distinct between humans to serve as a form of identification, similar to fingerprints or iris scans[1-3].

We started off with the simple MotionRecorder app which accesses the phone’s accelerometer and gyro at a high sampling frequency and exports the recorded data in a csv file. On top of that, our custom interface makes data recording and the authentication step easily accessible to any smartphone user.

The recorded data is sent to a server backend running python flask in the cloud, where the gait analysis is performed. We found that a simple Fourier Transform on the (gravity compensated) acceleration magnitude is already sufficient to distinguish the members of our team. The recorded data and its frequency content (Fourier Transform coefficients) can be visualized for inspection. Additinally, we also investigated the use of hand-engineered features as proposed by Steven Wessel’s post but did not use their approach in practice. Fragments of our hacky code is available on Github.

References:

Damaševičius, R.; Maskeliūnas, R.; Venčkauskas, A.; Woźniak, M. Smartphone User Identity Verification Using Gait Characteristics. Symmetry 2016, 8, 100. Link
H. M. Thang, V. Q. Viet, N. Dinh Thuc and D. Choi, “Gait identification using accelerometer on mobile phone,” 2012 International Conference on Control, Automation and Information Sciences (ICCAIS), Ho Chi Minh City, 2012, pp. 344-348. Link
Boyle, Matthew, Avraham Klausner, David Starobinski, Ari Trachtenberg, and Hongchang Wu. “Gait-based user classification using phone sensors.” In Proceedings of Conference on Mobile Systems, Applications and Services. 2011.Link

MPC-Net: A First Principles Guided Policy Search

2020-01-20T10:00:00+01:00

My newest work has just been officially released:

MPC-Net: A First Principles Guided Policy Search.
Jan Carius, Farbod Farshidian, and Marco Hutter
IEEE Robotics and Automation Letters (RA-L), 2020

Here is the abstract of the publication:

We present an Imitation Learning approach for the control of dynamical systems with a known model. Our policy search method is guided by solutions from MPC. Typical policy search methods of this kind minimize a distance metric between the guiding demonstrations and the learned policy. Our loss function, however, corresponds to the minimization of the control Hamiltonian, which derives from the principle of optimality. Therefore, our algorithm directly attempts to solve the optimality conditions with a parameterized class of control laws. Additionally, the proposed loss function explicitly encodes the constraints of the optimal control problem and we provide numerical evidence that its minimization achieves improved constraint satisfaction. We train a mixture-of-expert neural network architecture for controlling a quadrupedal robot and show that this policy structure is well suited for such multimodal systems. The learned policy can successfully stabilize different gaits on the real walking robot from less than 10 min of demonstration data.

A preprint of the paper is available below. The published version can be found under https://doi.org/10.1109/LRA.2020.2974653. Our code is available on Github.

Download PDF

Augmented Reality in a Stadium (YB Hackathon 2019 Winner)

2019-11-16T10:00:00+01:00

Last weekend, the football club Young Boys Bern hosted the second edition of their annual YB Hackathon. This year, around 100 hackers competed in 15 teams to solve different challenges within 24 hours.

I was given the opportunity to participate as part of the team Power and Style, alongside with Lars Carius, Jonathan Külz, Joscha Greuel. Our challenge, submitted by ELCA, involved creating an augmented reality app that would enhance the spectator’s experience in a stadium.

The Idea

After an initial brainstorming phase, we started implementing an Android App based on ARCore, written in Kotlin. Our idea was to solve a common problem in a stadium: It is annoying to purchase food or drink during the game because one has to move along tight seating rows to get to a food stand while also missing part of the game. Our app helps in such a situation by providing a local delivery platform where fans are being matched to help each other out.

A person wanting food can use their app to localize in the surrounding environment and place an order for food or drink. A person willing to walk to a food stand can point their smartphone to the block of spectators nearby and see drinking cups or items of food overlaid on the persons who ordered something. Now, they may choose to bring one or more items of requested food or drink item to a fellow spectator, saving them the tedious walk. As an incentive, they may get a discount on their own purchased item.

Result

After 24 hours, multiple teams showed noteworthy results and impressive live demonstrations in front of the audience. On our part, we presented a live showcase of our AR app in action, showing both the ordering as well as the delivery process. To all our surprise, our augmented reality app convinced the jury to award us the first price of the YB Hackathon 2019.

Der erste Platz geht an das Team Power&Style! Neben viel Ruhm und Ehre, nehmen sie den YB Hackathon Wanderpokal, ein Preisgeld von CHF 2'500, Tickets für die Gstaad Lounge sowie signierte YB Trikots mit nach Hause. @ElcaIT_de | @BSC_YB | @isolutionsag | https://t.co/ZUNhfPL8Hv pic.twitter.com/FwBQWsOOZQ
— YB-Hackathon (@ybhackathon) November 16, 2019

Some voices about the hackathon and our team (mostly in German)

The whole team would like to thank the organizers for a great hackathon!

Trajectory Optimization for Legged Robots with Slipping Motions

2019-07-03T11:00:00+02:00

I recently published a new paper

Trajectory Optimization for Legged Robots with Slipping Motions.
Jan Carius, René Ranftl, Vladlen Koltun, and Marco Hutter
IEEE Robotics and Automation Letters (RA-L), vol. 4, no. 3, pp. 3013 - 3020, 2019

This work was supported by Intel Labs, the Swiss National Science Foundation (SNF) through project 166232 and the National Centre of Competence in Research Robotics(NCCR Robotics), and the European Union’s Horizon 2020 research and innovation programme under grant agreement No 780883. This work has been conducted as part of ANYmal Research, a community to advance legged robotics.

Here is the abstract of the publication.

The dynamics of legged systems are characterized by under-actuation, instability, and contact state switching. We present a trajectory optimization method for generating physically consistent motions under these conditions. By integrating a custom solver for hard contact forces in the system dynamics model, the optimal control algorithm has the authority to freely transition between open, closed, and sliding contact states along the trajectory. Our method can discover stepping motions without a predefined contact schedule. Moreover, the optimizer makes use of slipping contacts if a no-slip condition is too restrictive for the task at hand. Additionally, we show that new behaviors like skating over slippery surfaces emerge automatically, which would not be possible with classical methods that assume stationary contact points. Experiments in simulation and on hardware confirm the physical consistency of the generated trajectories. Our solver achieves iteration rates of 40 Hz for a 1 s horizon and is therefore fast enough to run in a receding horizon setting.

A preprint of the full paper PDF is available below, and the published version can be found under https://doi.org/10.1109/LRA.2019.2923967.

Download PDF

Atom IDE for a Catkin C++ Workspace

2018-08-01T11:00:00+02:00

Update July 2019: Cquery seems to be no longer maintained, but there is a follow-up project called ccls.

One of my favorite editors for coding is Atom, which has recently received support to work as a full-fledged IDE by utilizing the language server protocol. This is a short guide on how to setup the Atom IDE for ROS developers.

Requirements

Up to date version of Atom editor
Packages for Atom: atom-ide-ui, ide-cquery
Cquery
compdb

Setup

First, get Atom and Cquery up and running

Build and install cquery according to the instructions on its Github Wiki.
Install the Atom packages atom-ide-ui and ide-cquery
In the ide-cquery settings of Atom, make sure the correct Cquery path is set (according to your installation)

Now in your own code base, do the following:

Compile your catkin packages with -DCMAKE_EXPORT_COMPILE_COMMANDS=ON. This will create a compile_commands.json file for each package in its respective build folder.
Then collect all compile_commands.json files into a single file by executing in the catkin workspace
```
cat ./build/**/compile_commands.json > compile_commands.json && sed -i -e ':a;N;$!ba;s/\]\n*\[/,/g' compile_commands.json
```
Since each compile_commands.json file must be delimited by a pair of square brackets, the second part of this command is supposed to remove pairs of brackets that we no longer need after concatenating multiple files.
So far so good, but this does not work for header-only libraries, because no compile commands were generated for them, since there is nothing to compile. To include also the headers into the json file, run
```
compdb -p . list > compile_commands.json.extended
```
Finally, create a symlink in your project root of the Atom workspace
```
ln -s /path/to/compile_commands.json.extended compile_commands.json
```
After restarting Atom, Cquery should now be able to find this file and start building the database.

Result

On startup, cquery will take some time to load the index into memory. Expect high CPU usage during this time. Afterwards, you should be able to hover over variables and functions and jump around in your code.

Hope it helps, happy coding!

Trajectory Optimization with Implicit Hard Contacts

2018-06-28T11:00:00+02:00

I’m happy to announce a new paper

Trajectory Optimization with Implicit Hard Contacts.
Jan Carius, René Ranftl, Vladlen Koltun, and Marco Hutter
IEEE Robotics and Automation Letters (RA-L), vol. 3, no. 4, pp. 3316-3323, 2018

This work was supported by Intel Labs, the Swiss National Science Foundation (SNF) through project 166232 and NCCR Robotics.

Here is the abstract of the publication.

We present a contact invariant trajectory optimization formulation to synthesize motions for legged robotic systems. The method is capable of finding optimal trajectories subject to whole-body dynamics with hard contacts. Contact switches are determined automatically. We make use of concepts from bilevel optimization to find gradients of the system dynamics including the constraint forces and subsequently solve the optimal control problem with the unconstrained iLQR algorithm. Our formulation achieves fast computation times and scales well with the number of contact points. The physical correctness of the produced trajectories is verified through experiments in simulation and on real hardware. We showcase our method on a single-legged hopper for which jumping and forward hopping motions are synthesized with an arbitrary number of contact switches. The jumping trajectories can be tracked on the robot and allow it to safely liftoff and land.

Together with my colleague Vassilios Tsounis, we presented a brief overview on our method at the Dynamic Walking Conference 2018.

A preprint of the full paper PDF is available below, and the published version can be found under http://doi.org/10.1109/LRA.2018.2852785.

Download PDF