The “Android” for Autonomous Vehicles

In RTAS 2021, we organized a panel discussion on “RTOS for Autonomous Machines”. The panelists are Shinpei Kato (The University of Tokyo & Tier IV, Inc.), Andrei Kholodnyi (Wind River Systems), Shaoshan Liu (PerceptIn), Jan Staschulat (Robert Bosch GmbH), and Richard West (Boston University). An interesting topic discussed during the panel is “Will there be something analog to Android for autonomous vehicles? If yes, how will it look like?”

The following is a condensed transcript of the discussion. For the full video recording of the panel, please visit here.

Shinpei Kato: I think the answer is: Definitely. We recently launched MIH, an EV open platform initiated by Foxconn. I believe this is a way to the future of autonomous machines. We can actually open up not only software, but also hardware components. The reason why Android is actually really deployed in the market is, yes Android is open source, but the Community initiated by Google, they clearly defined the reference design for how Android can be deployed in the real hardware.

Now the challenge is to build an ecosystem or a community. Linux is a community. Android is a community. How can we make this as a community, this is my current understanding to the challenge in the real world for autonomous machines.

Rich West: We have a HotMobile paper on exactly this and the company that I am working with has a group building an instrument cluster and in-vehicle infotainment system for the vehicle. We started out integrating Android as a guest operating system. Here’s an example of a simple version of the vehicle management system.

I agree with Shinpei about trying to build an ecosystem around Android just like we’re building around Linux and clearly it makes sense to integrate Android into a vehicle. We’ve already seen the Android Auto and Apple Car Play allowing the integration of a smartphone into a vehicle where you use the vehicles head unit for displaying an interface to the phone applications, but the applications are still running on the phone. We need to go to the point where you can go beyond the applications that you would have on the phone, and maybe use the Android stack which is a convenient software development kit to define an integrated interface to work with core vehicle functions, such as heating ventilation and air conditioning or through a touchscreen or a voice activation feature to enable ADAS services.

Now the question I have is why do we need Android to do this. We started using Android but ended up scrapping it and we’ve gone with Linux plus Qt to build our interface. One of the reasons is because Android has very poor functionality to support multiple headed display. If you want to support things like an instrument cluster or read only instrument cluster with a touch screen in vehicle interface to the side, Android is very poor at doing that, but Linux is perfectly capable of managing multiple screens at same time. If you look at Tesla, Tesla doesn’t even support Android or Apple Car Play and they’ve implemented their own user interface using Linux. So I am not sure that Android is really going to give you anything at the end of the day.

Jan Staschulat: When you’re talking about something similar to Android, I asked myself the question what are the specific features of android that are beneficial for automotive operating system. Of course it is open source and it is based on existing well-established technology like Linux. If you have an open source and you’re running a car with it, and if it fails, who’s going to be responsible for this. I agree that for having a kind of Android for automotive it is very beneficial to have an easy integration and also to do easier development of new features. But where I see the key difference is that while Android is based on Linux and does scheduling for phones, if you want to do real-time applications in a car, you do not only need a real-time operating system, and I would argue there’s so many open issues with this complex hardware. We have shared memory, we have GPUs and all these things have not been addressed sufficiently by academic community, then you cannot just make an operating system out of it.

Andrei Kholodnyi: I think Android is, by definition, a mobile operating system based on the Linux kernel. If you want to try something, it exists already and it is ROS. Android is very successful in mobile devices, but if you go and try to apply this to the car industry, it hits its limits. For instance, you don’t have support for automotive buses, or for multiple displays, because it was not designed for that. For ROS, in different industries, there are different domain specifics. There are other big communities where there are also specifics like there is a ROS where people implement specific domain languages more or less. It could be real-time and non-real-time depending on what kind of use cases you want to address. But I think ROS is a good platform to try real-time in various areas, and then, depending on the areas to find the real-time application examples where you can imply there are different requirements for different industries.

Shaoshan Liu: I think there will be an Android but far away. We don’t have Linux or Unix for robotic yet. We don’t have Intel or Arm for robotics yet. We don’t even have a solid end-to-end time model for robotics yet, so I think in the next few years, we should focus on building up those foundations before we push very hard for the Android route commercially. The same thing happened in cell phone agents where there were different kinds of OS in the market for about 10-15 years, and then Android came along, iOS came along, and became a consolidated market. So that’s my take, I will spend more time focusing on how to build the computer architecture for different kinds of autonomous machines to build the basic operating system or RTOS for different types of autonomous machines. But the general Android like user interfacing operating system, I think we’re not there yet. We’re still far from it.

Rich West: I don’t necessarily buy any of this ROS stuff. I think ROS is just a bunch of middleware, we’ve seen all this before. We saw CORBA in the late 1990s. Now we’ve got ROS, but at the end of the day, until you build the underlying OS to have the real-time requirements, you’ve got a problem. At the end of the day, with Android it is a user interface environment. It basically builds on top of Linux to give you a convenient user interface for certain applications that you might want to integrate into your autonomous vehicle. I am not convinced it is the right way, given my experience and we started out actually playing with Android. We actually started integrating it into our system and backed away.

Jan Staschulat: I have been working in the scope of micro-ROS, putting ROS2 on microcontrollers, and addressing lots of real-time issues. We are working in the real-time working group very hard to connect the two worlds, ROS2 and RTOS. It allows you at user level to use all the fancy algorithms, fixed-priority-preemptive scheduling or reservation-based scheduling and so forth, and as easy to use on ROS. Because we see also in Bosch that we cannot develop everything from scratch, we see the community around ROS2 with all the simulation tools and environments and every company has so many drivers available for all of that. Having a similar community for automotive makes absolutely sense. I also agree ROS2 is not there yet, but we’re working on this.

Rich West: I think Germany is a big influence over certain standards in the automotive world and the embedded world at large. But when I look at companies like Tesla, Tesla is very disruptive and they’ve sort of thrown the rulebook out of the window and they’re coming up with their own designs. Isn’t there something for an opportunity to now rethink how we do everything from scratch, because we’re taking on a new challenge now, particularly with autonomous vehicles. I really appreciate what you’ve been saying about pipeline processing, the sensing through to the actuation. I think we need a framework which can do real-time end-to-end pipeline processing. If we can make that work in ROS, great. But that to me is the grand challenge right now.

Jan Staschulat: I recently worked on service-oriented data-driven activation schemes. You have sense plan acts, you have graphs, and these kinds of concepts are independent of the middleware you’re using to express these kind of exclusion patterns. ROS2 is well established in robotics and in automotive autonomous driving. You have a very similar application design or software architecture. So that’s the kind of a natural way to start looking for these kinds of concepts.

Andrei Kholodnyi: Because you are researchers and you want to go from a paperwork and try it out, I think it is also pretty important to consider that Linux like OS. There are a lot of developers, who know Linux and don’t know any other stuff like any other OS. This is exactly what happens with Unix and BSD and all this stuff, and this would happen also to RTOS. I think the majority will be then real-time Linux, and if you consider your research, you also need to keep this in mind to try to work with the open-source community on that. I think the proprietary operating systems still have their niche and good reason to be there, but this community is really behind real-time Linux. For other applications you would use FreeRTOS safer, but I think for 99% or 90% you’ll use real-time Linux.

Rich West: You are in Wind River, VxWorks, Wind River hypervisor. There are other competing technologies, such as QNX hypervisor, etc. There are lots of different RTOS and hypervisor technologies. So there’s an opportunity to integrate traditional general purpose operating systems such a Linux with custom RTOS on the side, we don’t need to take Linux and make it only real-time on its own, we can augment it with other.

Andrei Kholodnyi: That’s right, this is one of the possibilities and there are different types of RTOS. There is micro-kernel versus monolithic. There is also a good research area that everything probably will move into some kind of uni-kernel RTOS and then you would have probably non-real-time computation environment, and within this computation environment, you would have islands of real-time computation workloads. It would be not likely that everything is real-time, keeping in mind what we have discussed, very complicated heterogeneous systems with FPGA, GPU and everything. It is completely unrealistic to make a real time operating system in the aggregation of this. If there is no good reason for using proprietary RTOS, people would use real-time Linux instead. I could say as a manufacturer of such an operating system, our system is now running on Mars, as well as also Linux. But you need to keep in mind that what was the first operating system you would go and try. It would be Linux.

Author bio: Nan Guan is currently an associate professor at City University of Hong Kong. His research interests include real-time scheduling, formal verification, system software and their applications in embedded systems, cyber-physical systems (CPS), Internet-of-things (IoT) and particularly Autonomous Vehicles. He is the Program Chair of RTAS 2021, where this panel discussion takes place.

DisclaimerAny views or opinions represented in this blog are personal, belong solely to the blog post authors and do not represent those of ACM SIGBED or its parent organization, ACM.