Does RMF handle paused/interrupted robot? (#72)

Posted by @ejalaa12:

Hello

It might happen that a robot gets interrupted by the vendor’s system for any reason. Does RMF handle that ? I see that there is a RobotMode.MODE_PAUSED, but I haven’t found that mode being used anywhere else than in the TrafficLight adapter.

Is it correct to say that it is the adapter node’s responsibility to handle that ? How do you communicate that information back to RMF Core ? Is it even handled ?

Btw, I have a hard time finding the possible interfaces between an adapter and rmf core. The documentation says that it is “tightly integrated”, however it would be very useful to know how, and what, that way a developer will know the extend of his possibilities :slight_smile:

Thank you again :slight_smile:

Posted by @mxgrey:

The “full control” and “traffic light” APIs are robust to delays, whatever the source of the delay may be. That means that as long as you provide accurate up-to-date information to those APIs, they will manage the traffic correctly.

One feature that’s arguably currently missing from those APIs is the ability to say “I’m going to intentionally wait here for a long time because the vendor/operator is demanding it”. The current system for handling delays makes an assumption that the robot is always trying its best to resume its assigned work, so you won’t get the best possible behavior out of the traffic management system if the robot is actually intentionally stopped for an extended period of time. Other robots might end up waiting for it to move out of the way, incorrectly trusting that it’s trying its best to move along.

Btw, I have a hard time finding the possible interfaces between an adapter and rmf core.

The lowest level interfaces are captured by the message packages in rmf_internal_msgs, but we do not currently recommend anyone developing based on those message definitions directly. The message definitions are subject to changes that are not API or ABI compatible as we continue to flesh out new features and better communication quality. Instead you could consider looking at the classes provided by rmf_traffic_ros2 which wrap up the internal ROS messages in a way that should be API and ABI stable, even as we add features and change implementation.

Posted by @ejalaa12:

The full control adapter does not specifically handle the PAUSED mode, instead, what I’m understanding from the code is that

  • it calls the update_position method of RobotUpdateHandle (which somehow informs the scheduler node)
  • it calls the next_arrival_estimator function which informs the scheduler of a delay.

So in other words, the fleet_adapter does not say to rmf “hey, robotX is paused, take that into account”, instead it says “hey, robotX has Y seconds of delay, take that into account”

Is that correct ? Also what happens if the max delay threshold is crossed ?

The lowest level interfaces are captured by the message packages in rmf_internal_msgs, but we do not currently recommend anyone developing based on those message definitions directly

Indeed, I saw this being mentioned in other questions, the rmf_traffic_ros2 classes are what I’m looking for, however, its not evident to understand what they do. I’m assuming the Adapter class is wrapper of those Participant, Writer… classes. But since I’m writing my own fleet adapter – and I guess this might be useful for other developers --, it would be very helpful to understand what the scheduler is expecting from the fleet adapter and vice versa, and how those classes come into play.

Thanks again for the fast response.

Posted by @mxgrey:

So in other words, the fleet_adapter does not say to rmf “hey, robotX is paused, take that into account”, instead it says “hey, robotX has Y seconds of delay, take that into account”

Yes, this is exactly correct. Unfortunately the prediction of future intentional “delays” (or intentional pauses) isn’t something currently offered by the high-level APIs. The underlying traffic management framework can support it, but I think adding it as a stable part of the high-level public API would be tricky. At least it’s not immediately obvious to me how to make a clean and clear API for it.

Also what happens if the max delay threshold is crossed ?

When the max delay threshold is reached, it simply triggers a fresh replanning, taking into account the latest schedule predictions of all the other robots. The assumption is that delays may be accumulating because the initial plan is no longer a good one due to unanticipated changes in the other robots’ itineraries, so we should just try to make a completely fresh plan. I intend to get rid of this behavior in the future by having each agent more intelligently monitor the other itineraries that are constraining it, but that’s a future improvement with no immediate timeline.

But since I’m writing my own fleet adapter – and I guess this might be useful for other developers --, it would be very helpful to understand what the scheduler is expecting from the fleet adapter and vice versa, and how those classes come into play.

When you say you’re writing your own fleet adapter, do you mean you’re writing it independently of the rmf_fleet_adapter library? That may be more challenging than you’re anticipating, especially if you want to have a system with multiple robots and quick responses to a highly dynamic traffic schedule.

I recognize that the public APIs for rmf_fleet_adapter are quite narrow at the moment. That’s an intentional design choice because we have users who wish for very strong API and ABI stability even while we develop new features and overhaul the implementation. If we expose too much from the rmf_fleet_adapter library then it’s likely that we’ll need to break the API too often.

If you can describe the kind of features/behaviors that you need, then maybe we can work out a way to tweak the rmf_fleet_adapter API to accommodate it. Or if you’re open to experimentation, you could always fork rmf_fleet_adapter and modify it with the changes that you need, then get back to us about what changes we could make to improve usability.

All that being said, I have a feeling what you really want is the command intervention pipeline idea that’s mentioned here. The idea for that feature is it would allow a workflow for operators or fleet managers to intervene with the currently scheduled task of the robot while still keeping the traffic schedule up to date and allowing the scheduled task to resume once the intervention is finished.

If that sounds like it would fit your needs, we’d always appreciate input on the issue ticket, including details like API design recommendations and suggestions for the scope, constraints, and requirements of the feature.


Edited by @mxgrey at 2021-07-13T06:07:50Z

Posted by @DoppiaEffe94:

Yes, this is exactly correct. Unfortunately the prediction of future intentional “delays” (or intentional pauses) isn’t something currently offered by the high-level APIs. The underlying traffic management framework can support it, but I think adding it as a stable part of the high-level public API would be tricky. At least it’s not immediately obvious to me how to make a clean and clear API for it.

@mxgrey Are there any updates on this matter?

I’m trying to manage a fully working stop() function, but, as you said, the traffic management doesn’t take into account an intentional MODE_PAUSED, therefore it republish a new plan.

Did you extend the functionalities of the high-level API to take into account this issue?

Thank you.