Development of a mixed-reality test environment for UX and cockpit concepts

Software-defined vehicles (SDV) are considered the future of the automotive industry. In this context, the software is no longer developed for the vehicles, but rather, the software defines the development process for new vehicles. Whether it's vehicle control, diagnostics, safety systems, or media and entertainment, no new vehicle can do without highly specialised software. While assisted and partially automated driving already requires complex software and sophisticated operating and warning concepts, these must be completely rethought for fully automated and autonomous driving at the latest.

However, the development of new operating and cockpit environments, including software, is costly. At some point, every project dealing with this topic has to pass a reality check. The production of the necessary demonstrators and prototypes is very time-consuming and costly. The automotive industry is therefore looking for alternatives to make this process more efficient.

In the automotive industry, the use of virtual reality in various application scenarios is almost old hat. Virtual cockpits for visual exploration are also already being used productively in many cases.

BMW AG maintains a vehicle-related research department, BETA-Labs, whose task it is to identify, develop and test application scenarios for new technologies. The BETA-Labs team has taken on the challenge of transferring UX and cockpit concepts into virtual environments for development purposes. In contrast to purely visual experiences, designers and software engineers should be able to test their concepts in every development phase. They should have the same visual and haptic user experience as in a real cockpit. The aim is to minimise the number of expensive physical prototypes in early development phases and thus significantly increase efficiency in vehicle-related software development.

The BMW Group's BETA-Labs team was supported by software development experts from IT service provider adesso, with whom there was already a good and cooperative partnership in the past. Students of software engineering at the Technical University of Munich also supported the project as part of their ‘iPraktikum’ (i-internship).

The team's task was to develop a virtual cockpit based on the Apple Vision Pro headset and the Apple visionOS operating system that would enable the integration of external controls into the simulation and provide haptic feedback. The aim was to utilise the particular advantages of the relatively new ecosystem over conventional setups with solutions such as Meta Quest Pro. These include, among other things, better image quality and image and object tracking, which is available for the first time.

The work on the concept and its implementation in an application took place alternately in the BETA-Labs of BMW AG, in the Vision Pro Developer Labs of Apple and in the rooms of the Technical University of Munich and at adesso in Munich.

In just three months, the students developed the ‘BMW Virtual Cockpit’ platform together with the BETA-Labs and adesso team. In addition to the Apple visionOS app, it includes a fully functional physical driving simulator that transports the haptic user experience of the steering wheel and central display into the virtual world. It allows for the flexible configuration of the steering wheel and pedals with a central display – represented by a touch display in the real world – and a smartphone holder, in order to be able to map different UX concepts.

By tracking the user and the objects positioned in the workplace, the physical objects are precisely integrated into the 3D environment by virtual representatives. Steering movements, pedal actuation, control inputs on the display or smartphone are transferred to the virtual counterparts not only visually but also functionally, as in a mixed-reality setting.

The particular challenge for the team was to ensure that the Central Information Display (CID) and smartphone applications also appeared in the right places in the simulation. As part of the project, the students therefore not only developed the main application for Apple Vision Pro, but also accompanying applications for iOS and macOS. These are used to project the screen contents of the external devices into the virtual environment of Apple Vision Pro via low-latency video streaming. An additional Python script developed for Windows provides the Vision Pro with the steering wheel and pedal data.

The ‘BMW Virtual Cockpit’ has created the basis for designers and engineers to test the current state of the infotainment software at any time during the development phase, just as they would on real prototypes. This not only saves them time and money, but also gives them far greater flexibility in terms of agile software development processes.

The virtual representation can be used to answer the following questions, among others:

• How does the current state of the UX and cockpit concept affect the driver?
• Is the operating concept coherent?
• Does the driver have everything in view?
• Do the displays and controls provide sufficient feedback?
• Does the cockpit and operating concept unintentionally distract the driver?

The joint team of BMW BETA-Labs, adesso and students also managed to create a future-proof test environment with the project. Not only can current and future usage scenarios and operating concepts be tested in the virtual environment, but variants can also be simulated that take into account the increasing prevalence of autonomous driving systems at levels 3 and higher. These allow the driver, for example, to pick up and use their smartphone while driving.

The BMW Group presented a further development of the original prototype at CES in Las Vegas in 2025 as part of the world premiere of the BMW Panoramic iDrive. Here, visitors were able to get a first glimpse of the new operating system in mixed reality.