What are general practices to follow/QA steps when deploying a robot in production?

[ssarkar]

Hi everyone,

This is more of a general curiosity than a dedicated question. I was wondering what are/would be the general QA steps needed for a robotic/cobotic application. For example, let us consider that working within a factory environment, my robot (let’s consider a UR10 mounted on a MiR100 base) needs to do some pick and place (for simplicity, no pneumatic/soft gripper) .

Let’s say I have programmed everything in ROS2 (MoveIt2 + Nav2). What are the things I should be looking out for, when I want to test for safety? I know there are various standards to follow, I want to know which ones apply in case of such a system.

Q. If my robot is at industry safety standard A, and my base mobile is at standard B, does this mean my system is at standard A/B?

Q. How do we separate the hardware from the software for quality assurance purposes? Is there something extra that needs to be done to certify the underlying software as safe too?

How would you go about deploying such a system in a factory? This question is more on the lines of packaging and containerisation - would you use docker, or maybe package everything into a single snap?

Sorry if the question turns out to be too generic. I am trying to learn what would be the best practices, and how I can ensure them as a Robotics Engineer.

Thanks

[afsantos]

Hello.

I think these are very important questions to raise and discuss. I, too, would be interested in hearing some opinions, but, as a software engineer, rather than a roboticist, I can only provide some very limited answers.

@ssarkar

What are the things I should be looking out for, when I want to test for safety?

Try to take advantage of as many analysis and testing techniques as you can. Some things can be caught at compile time, for example, while others will only manifest at runtime.

@ssarkar

Q. If my robot is at industry safety standard A, and my base mobile is at standard B, does this mean my system is at standard A/B?

I would say no. Only part of you system is compliant with A, and only part of it is compliant with B. The system as a whole is neither, because integrating different components can bring problems of its own.

@ssarkar

Q. How do we separate the hardware from the software for quality assurance purposes? Is there something extra that needs to be done to certify the underlying software as safe too?

Dependability cases are a good approach. If you have a system with components X, Y, Z, and you want to verify property P, you basically have to break it down into smaller chunks, p1, p2, etc., that you can manage more easily, such that, when put together they are equivalent to P.

For example, “if mobile base receives command C, it eventually starts moving forward”. Roughly speaking, this is equivalent to saying that:

1. if the ROS layer receives message C, it eventually calls some function f of the low-level driver;
2. if the low-level driver function f is called with certain parameters, it eventually signals the hardware to power the actuators in a certain way;
3. when the actuators receive certain signals, forward movement is achieved.

In a realistic scenario you might have to break it down further, but you get the idea. Only at point 3 in the example are we dealing with hardware. Everything else assumes that 3 is correct. Then, you just have to provide evidence for each of these subproperties, and argue why they are enough to prove the original property for the mobile base.

[ssarkar]

@afsantos

Try to take advantage of as many analysis and testing techniques as you can. Some things can be caught at compile time, for example, while others will only manifest at runtime.

This was my general way of thinking as well. I have used the pytest framework previously for a different python project, but I was wondering if there would be some difference in running pytest with ROS2 specific things.

@afsantos

Dependability cases are a good approach. If you have a system with components X, Y, Z, and you want to verify property P, you basically have to break it down into smaller chunks, p1, p2, etc., that you can manage more easily, such that, when put together they are equivalent to P.

For example, “if mobile base receives command C, it eventually starts moving forward”. Roughly speaking, this is equivalent to saying that:

1. if the ROS layer receives message C, it eventually calls some function f of the low-level driver;
2. if the low-level driver function f is called with certain parameters, it eventually signals the hardware to power the actuators in a certain way;
3. when the actuators receive certain signals, forward movement is achieved.

This reminds me of a formal methods and model verification class I had at uni, thanks for demonstrating a practical example. It also makes me think if there exists some ROS package that aims to test industrial processes - maybe with Petri Nets/Timed Automata.

Thanks for the detailed response, @afsantos , it really helps to see the general problem from a wide range of perspectives !

[doganulus]

I would first suggest reading what is proposed to test automated driving systems in literature and industry. As the base, ISO26262 is the most widely used standard for this purpose. But it is just a framework and conforming a safety standard does not make the system safe (they just specify the bare minimum). Apparently the most promising approach nowadays is the simulation and scenario-based testing (assuming you have faily realistic model of your robot in simulation). Especially using abstract scenario and test specifications.

However, testing and verification for automated driving systems is still in their infancy. Many proposed concepts and techniques have been borrowed/adapted from digital circuit verification and formal methods. I would suggest everyone to read how digital circuits are tested and validated for the mathematical origins and more mature tooling.

Compared to the verification efforts in semiconductor industry, the robotics industry does no testing at all. That’s unfortunate, of course. It needs a sustained effort to raise the level of testing across the industry.

But the most importantly, safety essentially comes from reducing the risk of failure for each and every single item. This includes your libraries, dependencies, processes, supply chain, tools. It must be taken as whole by the organizations (ISO26262 states many requirements regarding this).

[cst0]

Loosely related: You may be interested in the testing suites for ROS1 and ROS2: rostest and ros2test.

[ssarkar]

Hi, thanks for letting me know about ros2test. I will make sure to check it out

[ssarkar]

Hello @doganulus, thanks for your response. I agree with your point about simulating all possible conditions with the robot, this is what I do for the moment. Also, thanks a lot for pointing me towards ISO26262, I will make sure to read through it !