Posted by @koonpeng:
In step 3, how would the api server know the url hosted by the media server?
It seems a straightforward way could be to extend the api server AppConfig with an additional dictionary of links. This is probably good for a first attempt, but will be static and unable to account for changes. Eventually it would probably be ideal to have a secondary mechanism for such dynamic cahnges. I have not thought too deep into this, thanks for bringing it up. I think it will be necessary to better understand how the media server will work in creating webrtc “rooms”, in order to properly design the URL “lookup” mechanism that builds on it.
I would avoid using config to specify the url, it requires knowledge of how the media server works, and it does not allow dynamically created rooms.
In step 4, what is the protocol of the control url?
Was thinking of using the FastIO libraries in the api-server, and reuse the api-client Swagger bindings on the Adapter process.
In the teleoperation control part, it is mentioned that the client connects to the api server which translates the teleoperation commands, but in the teleoperation view diagram, it shows the client connecting directly to the robot.
Yes, i was trying to think how we could represent handling different teleoperation “views”, I didn’t spend enough time thinking how to represent this clearly. The idea was that we could have different mechanisms to teleoperate. Some robots may not have a camera view available at all, but we might want to allow a robot-specific way to teleoperate. For example, I know that the MiR has a web dashboard which has a nice view of its navigation map, and a joystick for teleoperation. It could be possible to serve this dashboard directly through the TeleoperationView in order to capitalize on this. In this case, we would probably bypass the entire webRTC stack, as we are not streaming a camera feed ( it doesn’t exist / not available ).
If the dashboard is using the api-client library, that means the robot/teleop adapter has to support a standardized teleop protocol, if so, I don’t see the need of the 2nd flow, which goes through the api-server. For retrieving the mir web dashboard and showing it on the rmf dashboard, there are a number of problems I see which are not addressed in this proposal.
- How does the rmf dashboard know the url of the robot dashboard?
- How can the rmf dashboard log into the robot dashboard?
- The robot dashboard may need cookies for session/logins etc, assuming we are using iframes, they would be 3rd party cookies which are blocked by default on safari and may not work on other browsers if the robot dashboard did not set their cookies correctly.
Do we need the type field in the schemas? Is it enough to identify the type based on the url, i.e. should we share one url for all teleoperation apis, or put each “type” of request into different urls?
I admit i blatantly copy pasted examples from the rmf_api_msgs schemas to try and be as consistent as possible, so I might have misunderstood the schema intended meaning. I understand the “types” as a way to identify the intended encoding of the string in the JSON, which is of type “String”. I feel I might be misunderstanding your question?
The schemas provided have the string with a fixed constant, so the field is redundant and can be implied from the request endpoint. i.e. If the server receives a request on /teleoperation_view_request, it already knows the payload is a Teleoperation View Request json, it doesn’t need the type field to identify the payload.
Both the operator and the robot joins the same socketio room, how would the api server identify between operators and robots?
I would like to simplify this by saying that it might be fine if we didn’t differentiate them, since the bandwidth required should hopefully not reach a point of causing issues, and each endpoint can ignore whatever data isn’t found useful. Perhaps this assumption is flawed?
Bandwidth aside, I can think of some scenarios where the server need to know the type of client to provide certain functionalities.
- Detecting / reporting if the teleop control is available.
- Security
- We don’t want other adapters to listen in on the commands sent to other robots (probably ok for same fleet, but not on another fleet).
- We also do not want teleop adapters to be able to submit tasks, open doors etc.
. The issue I have with making a lock mechanism is that I am afraid of many usability issues. When does a lock release, what happens if that release fails to reach the server, timeout issues, etc, it becomes hard to test and predict deployment situations, for the benefit it provides. Since only authorized users can control the robots ( in theory ) it could be sufficient to perhaps simply report who was the “last operating user” so anyone with any beef can just send a strongly worded email.