I had an idea recently about a starting an experimental robotics project which involves building a robotics stack based on neural networks alone. While I realise this is a really bold statement to be making I thought it would be worthwhile creating an open-source project to start exploring these ideas in an experimental fashion as a group.
I thought to call it “robot operating brain” and have went so far as creating a logo and purchasing a domain name for the project. While this is really only an idea so far I would like to know if I am the only one interested in this line of thinking? I feel it could be beneficial to have a group explore ideas in this realm alongside regular ROS development.
I’m creating this thread to gather feedback and assess whether there is interest in forming a group to work on experiments in this direction. I feel this project may be beneficial to regular ROS development and could be viewed an experimental arm of the ROS project.
Furthermore, it may also help attract more deep learning practitioners to the field of robotics and help in data collection. Looking forward to getting some feedback from the robotics community on this. I have provisionally created an organisation for the project which can be found at the following link: Robot Operating Brain · GitHub.
As a friendly bit of advice, you may want to trademark the logo/name (having a web domain doesn’t do that).
I guess my question would be how you plan to differentiate this project from the existing tools/techniques that are already out there? There’s a pretty sizeable chunk of research in this field that’s already been open-sourced, and you’ll want to avoid duplicating effort.
In particular you point out attracting deep-learning practitioners: the reason deep learning isn’t as popular in robotic controls tasks is because of the expense of data collection in real-world platforms. So sim-to-real becomes the most attractive option, so my advice would be to put some resources towards that in particular.
Thanks @cst0 - yes this is a great point. I have considered trademarking the logo/name, if the open-source project grows I definitely think it would be worthwhile. I’m not sure how this would work relative to ROS, I intentionally called the project robot operating brain as I thought it could be an experimental group that falls under the ROS community.
I think the distinguishing factor is that this project aims to explore how progress in multiple sub domains of deep learning such as computer vision, natural language processing, deep reinforcement learning etc. can best be consolidated. For instance when developing a visual perception module the project would look to understand how visual perception should integrate with other components such as language and control. In addition, I believe it would be interesting to understand what features can be generated by perception modules in order to enhance the learning of optimal planning and control.
I feel it also could serve as a resource for the open-source robotics community, maybe it would be possible to host pretrained models for practitioners to use and to develop tooling that is relevant for deep learning when applied to the domain of robotics.
Yes this is true and I think there would be a lot of important decisions that would need to be made such as which deep learning framework(s) and simulation environment(s) are used. I agree that the choice of simulation environment is crucial. I also feel that the application of deep learning to robotics is likely to grow overtime as we see more applications, hopefully this brings with it more data and low cost hardware making this area more accessible to a wider audience.
I thought it would be worth understanding if others also think it could be of value consolidating research in this area under one open-source project. As an example, if today I wished to understand how best to train a pick and place robot for industrial applications, what would be the best open-source resource for this? Is there access to pretrained models and data from this resource? While there likely exists many open-source projects on this topic for instance, I feel that having a single open-source resource for this would be useful to the robotics community.
These are just my current thoughts, maybe there is already a resource out there for this that I am not aware of? If not I believe it would be of value starting a group who work towards creating such a resource.
This is interesting, maybe it is worth simply joining the development of this project.
This statement is completely speculative but the idea would be to have an end-to-end system from sensor input to controls where the feature generation and importantly decision making processes are managed by neural networks without the need for hard-coded logic. I don’t think it is clear how such a system is structured or what the objective functions for different networks would be and how they should interact but this is something I wished to explore. Also note I don’t think this is applicable to industry in the near future due to the need for safety and regulation when it comes to decision making processes and controls. I do think that maybe at some point in the future it will be important though as for certain tasks it can be easier to specify the task and have a rl agent or some learning system solve it than it is to break down the task and apply heuristics to solve it (if not already for pick and place tasks for instance). I guess most industry solutions will use a combination of deep learning for controls and hard coded logic for high-level decision making for some time.
I was hoping to start a group that would discuss and explore these topics as part of an open source project while also looking to promote/advance robotics in the deep learning community through hosting models and datasets. This is quite ambitious and not likely to succeed early on but it is a topic I am personally motivated to work towards and I wished to see if others felt motivated by this problem as well. I was also interested in the ideas others may have on this topic.
The way most openource projects get started is someone develops a prototype that is good enough that people say “That’s cool” and want to download a copy and try it. A small percent of those people contribute back work to improve the software. The improved software is even cooler to it attracts more attention and a (very) few of those contribute improvements. It snowballs.
But almost never do people say “what a cool general idea” You are going to have to have some kind of prototype demo that gets noticed.
One idea is a navigation planner that is trained. It could run inside a ROS framework. You train it by using the usual LIDAR/SLAM to give it a tour of the building interior hundreds of times then one day turn off the lidar and let it ue the camera and trained network and show how this is so much more robust. Jut one example of how to go piece by piece and not have to start from scratch. Replace one part of a “standard” robot.
Manipulators would be a good place to work too.
This is a fair point and how you have outlined the growth of an open-source project does make sense.
It will take some time but I will look to start working towards an interesting prototype I had in mind, I like and appreciate the suggestions you have made. I guess the following working group may also be applicable so I’ll likely join this: New Working Group Proposal: AI Integration Working Group
With this being said, I am glad that this thread at least sparked some discussion. I do still believe there is a need for a centralised source of openly available pretrained models and datasets in the field of robotics.
Take a look at huggingface for a repository of a wide variety of models, or take a look at specific packages that leverage AI models for their specific task and therefore make models available (for example, agile_grasp).
This is another great recommendation, I am aware of huggingface but I have yet to come across models specific to robotics there. I have mainly observed most popular pretrained language and vision models there which are no doubt of value to robotics but I was thinking more along the lines of multi-task perception models and control models that may be applicable to robotics and are currently difficult to find.
I also find with GitHub it would be possible to link to an S3 bucket hosting model weights but it gets messy if each repository owner needs to setup its own AWS account and S3 bucket, why not have one open source repository of models? I don’t think it will be feasible to host larger models on Github.
Do you think we should generate and upload models specific to robotics to huggingface in this case? If the answer is yes I would be happy to partake in doing so. I guess it would be worthwhile doing the same with datasets but I would need to look into the accepted dataset formats hosted by huggingface. I feel there may be issues here for large datasets containing image sequences but I would need to check the huggingface documentation on what size and format datasets are supported.