[Nav2] Assisted Teleop Call for Opinions

Hi all, your friendly neighborhood navigator here.

I’m working with a student this summer on implementing assisted teleoperations to the Nav2 stack. This is an area often folks request. We’re working to make sure we can support both modalities that assisted teleop is the major task you wish to accomplish (via behavior tree) and also that assisted teleop is something you want to do on occasion as part of a larger system (server + behavior tree node access).

To that end, we need your assistance in answering some philosophical questions regarding how you want teleoperation to function:

  • A) Do you want it to prevent collision but not take any active maneuvers to do so? For example, it will stop a user from being able to command a robot into the wall but it will not try to avoid the wall. So the user would then have to command the robot to turn around or whatever to move elsewhere. This is ‘Pure Collision Rejection’, lets say. This is similar to capabilities often used in industrial robotics applications to make sure you’re always a second or something away from collision (so we dynamically reduce velocity in the requested direction such that it is always predictably a second away from collision).
  • B) Do you want it to prevent collision and also take active maneuvors to do so? For example, it would stop a user from running into a wall, but if there is a doorway a couple of meters over, it may try to select velocity commands near the requested command that is the most admissible. This is ‘Pure Collision Avoidance’, lets say. This is what is implemented in the navigation_experimental repo in ROS1.

Personally, I prefer option (A) from the perspective that I know what I want my robot to do, and I’d probably get pretty irritated if/when it “guessed” wrong. Another reason would be that this would be trivial to build directly into the controller server such that this could always be running even when not being teleoped, so that if your controller stinks, it still will stop collisions. Finally, I like it because its easy to implement and I don’t like making my own life more difficult than required :slight_smile:

In defense of (B), I think that is the more intelligent solution and if you were doing some remote teleoping where you didn’t have good visibility of the surroundings, having a nudge in a less hazardous situation could be valuable. Though with the restricted band of velocities around the request, I don’t think it would be all that effective in avoiding any major additional issues unless it was small (and then probably not all that hazardous to begin with). I suppose you could open up the band of velocities it could consider around the request, but I have a feeling that would make the robot zig sometimes when you wanted it to zag, when faced with unusual or highly symmetric obstacles.

In an ideal world where we had infinite time and resources, I’d just implement both and leave it to you, the user, to select which behavior you like the best. Alas, we are limited to just one option by the clutches of time and my ADHD to tackle other problems :wink: .

Happy avoiding,

Steve

6 Likes

If I had to choose, I’d be more in favor of A, personally.

I like the direct relation between operator intent and robot action. I also feel like it better supports operator intuition in the case of moving obstacles. Adding “impedance” in directions of obstacles almost has a haptic feel to it that I think would be intuitive from the teleoperation side of things.

For B, I am struggling to think of scenarios where the operator would have a less clear understanding of the surroundings than the robot. I’m imagining an RViz style operator interface where the operator has access to the same sensory data as the robot; perhaps on high latency networks is where B would shine? Or if there is prediction of dynamic obstacles under the hood? Actually, to me, a bigger strength of B than situational awareness would be in situations where the controller (joypad, keyboard, etc) is clunky or offers poor resolution / “transparency”. Given sufficiently poor user interfaces, it could help extract higher resolution intents from coarse commands.

All that said, I think I agree with you that A still serves what I consider to be “typical” use cases a bit better.

1 Like

I agree with favoring A out of those two. It’s a more generalized application that covers a bunch of scenarios. I’ve had cases where there would be latency, or the robot had better situational awareness with sensor data that wasn’t available to the operator for whatever reason.

B definitely has its use cases and it would be helpful if the architecture facilitated the development of specific application implementations, but it would be difficult to have a generalized solution for it.

I would also favor A over B. Especially when considering an industrial setting. If you’re using teleop and Nav2 stops your robot before your safety sensors kick in and activate your safety brakes, great. It can be a pain to get up and manually reset your safety sensors to allow you to drive again.

However, in scenario B, I can imagine it would be very frustrating if you’re trying to drive your robot around and not only does it not do what you want (scenario A: No, I’m not going to let you drive the robot into a wall), it’s also going to decide for you what to do (oh, you said straight, I’m going to veer left slightly, because ‘I felt like it’).

Don’t get me wrong, I could imagine people liking option B. However, in an industrial setting the operators are rarely developers. If the robot is ‘dumber’ and therefore more predictable in manual operation mode, that would be more advantageous for operators.

Hi,
I’d also prefer A in general, specially for industrial AMR’s. However, I see B not as standard “twist teleoperation”, but as a “local goal” teleoperation, so the human send a local goal in the surroundings of the robot (define surroundings…) and the robot goes there “autonomously”.
Another option C is “constrained twist” , so the environment constraints your twist somehow. For instance, in a corridor, even if your gamepad stick button is pressed deviated, the environment constraints the final command setting lateral speed to 0.
best!
Andreu

A - as it creates more predictability. However its worth noting that there is also quite a lot of middle ground. Given that you assume a human operator, you could

  1. Do A but create the capability to suggest to the human operator the manoeuvre for B.
  2. In certain circumstances allow legal moves, but not allow moves that will cause a collision. For example a classic problem is that when driving the robot down a long thin hallway if the robot is not pointing perfectly down the hallway it runs in to the wall eventually so you have to jog the orientation of the robot to avoid the robot stopping. Both this and the stopping is very annoying. However if an attempt to drive in to a wall causes continued motion but not continued motion in the direction perpendicular to the wall that might be helpful.
  3. Other variations where you avoid collisions but don’t stop the robot completely.

A.

But in ROS1 we also have a hybrid implementation we call ‘autoguide’. This uses a pure-pursuit planner with manual throttle. (A button has to be pressed for the steering to work).

One could do the same here. Allow a button instead of a sideways-joystick to explicitly allow the robot to steer. This makes it clear to the user that the robot is in control of steering. Releasing the button and operating a sideways joystick enables the operator to full control again.

Both A and B!
They should be 2 different and independent components (at least that’s the way we do it) and they are answers to 2 different problems… ‘A’ should be always active (even during autonomous navigation) and kept very simple. ‘B’ would run on top and used optionally. In other words A is a low level filter sending commands directly to the motor controller, and B could be a higher level filter, taking user inputs and generating commands sent to A.

Though if resources are limited, it makes sense to start by A. Having this kind of “reactive security” built in nav2 would be amazing, though to me it implies decoupling it from the navigation costmaps.

I think that A is a more general and more usual use case so, If the resource is limited. I would go for the A option.

Since we have to choose, I’d vote A.

It would be nice to include a simple ‘pause/resume’ kind of trigger so that one can disable the functionality momentarily without shutting it down.

Thanks for the responses everyone! I think 8 to 0 makes it clear that A is the strategy we should go down. Thanks for the help, we should have something useful to play with in the next couple of months!

1 Like

A little late to the discussion and its conclusion, but I’d like to reiterate some points that others have hinted to: that option A and B are not necessarily mutually exclusive, and perhaps suggest some related work on the topic of sliding/shared autonomy for teleoperation. In particular there is a wealth of research in the HRI filed that also has studied this in the context of cooperative control for assistive mobility devices, and I think that resonates with the current discussion on indoor robot navigation:

Some early works:

https://doi.org/10.1109/100.924361

https://doi.org/10.1109/MIS.2007.36

A more recent review:

https://doi.org/10.1109/THMS.2017.2706727


Although Human Robot Interaction isn’t my expertise, I’d think it would be neat to have a sliding scalar value as a ROS parameter for the Nav2 stack that could be controlled from a human-computer-interface by the pilots or teleoperators that would weight the control inputs between a conservative obstacle avoidance control (perhaps a simple vector field from the local cost map) and the human’s velocity control input, with a imminent obstacle avoidance controler superseding both planner and pilot.

This sliding scalar value could be manually adjusted by analog input from the pilot, or automatically proportional to the robot’s current tangential velocity. Say of the robot is going fast down a narrow hallway, the pilot is merely like on a rail of tracks controlling the forward or backward speed, but not burdened with keeping in narrow lane of the path or merging around minor obstacles. But if the pilot slows down for a controlled or precise tern, then this sliding scalar value is adjusted to give more weight to the pilot’s finer input, or diminish the radial distance of any conservative obstacle avoidance.

That is all to say, one could achieve both option A and B, option A in particular by adjusting the scalar value to nullify the weight of the autonomy control input entirely.


This is perhaps a common area of research for auto manufacturers developing driving assists or self-driving autonomy as well:

Storms, Justin, Kevin Chen, and Dawn Tilbury. “A shared control method for obstacle avoidance with mobile robots and its interaction with communication delay.” The International Journal of Robotics Research 36.5-7 (2017): 820-839.
https://doi.org/10.1177%2F0278364917693690

https://doi.org/10.1109/TITS.2017.2771351

1 Like