Air Race X: AFC Image Brings Extended Reality Sound To Shibuya

In October 2023, Air Race X Shibuya Digital Round saw virtual aircraft racing through the streets of Shibuya, Japan. For the world's first air race staged via extended reality (XR), realistic sound was an essential part of the spectacle. Yamaha AFC Image immersive audio technology ensured the event was a thrilling, ultra-real experience.

Racing sports monoplanes at speeds of up to 370km/h (230mph) through a city is obviously not practical, so Air Race X was evolved as the ‘Formula 1 of the skies’ to create the atmosphere and spectacle of virtual planes racing each other through the streets. Eight of the world’s leading air race pilots flew an identical course in remote locations around the world, with precise flight data collected and analysed by the competition headquarters. This data was then used to create a series of virtual ‘races’ through Shibuya, with spectators viewing animations of the aircraft through augmented reality (AR) goggles and smartphones.

The Challenge

“With Air Race X, we wanted to create an unprecedented new style of spectator sport,” says (Mr. Tsutsumi of PATHFINDER Co., Ltd.). “We wanted to reproduce the power and position of the aircraft engine sound, in the way which is unique to motorised sport, in an AR spectator environment.”

Billed as a ‘five dimensional motorsport that transcends time and space’, spectators gathered at the city’s Shibuya QWS, a 15th floor networking and creative space in the Shibuya Scramble Square skyscraper. From here they could watch the virtual aircraft (using the real flight data) flying around the virtual course.

Sound was an essential element in creating a realistic impression of the planes actually flying, however a conventional stereo PA system would not have been able to produce a realistic enough sense of sound movement.

Also, simply reproducing an audio source was not enough to create a sense of realism. The sound that reaches our ears is not just the direct waves, but also those reflected off buildings and other objects. This was another challenge faced by the Yamaha team, because reflected sounds which would be produced from city buildings in Shibuya were not created in the open spaces where the recordings were made.

The Solution

Under the leadership of supervisor Hiromu Ohgi, Yamaha Professional Solutions Division’s Spatial Audio Group deployed Yamaha’s proprietary Active Field Control (AFC) Image system to localise the sound for spectators, maximising the impression of the virtual aircraft producing real sound.

"AFC Image can cover an area of up to one square kilometre (0.4 square miles) and it is also able to simulate distance attenuation. This meant we were able to reproduce the movement of the sound image and changes in volume at a distance of up to one kilometre away, thus reproducing very realistic flight sounds,” says Hiromu.

AFC systems can employ the Open Sound Control (OSC) protocol which, for this application, meant that useful data not directly related to sound - such as the positional flight information - could be used to convey information to the AFC Image system to plot the location of the virtual aircraft in the Shibuya airspace.

The sounds were reproduced using 13 Yamaha DZR10-D Dante-equipped, two-way powered loudspeakers and one DXS15XLF-D Dante-equipped powered subwoofer. Placed on the floor at a distance of about two metres apart, the AFC Image system’s reproduction of the aircraft sound movement was reproduced so smoothly and realistically that, even with the spacing between the loudspeakers, it sounded completed natural.

“To create a more realistic sound space, we used AFC Image's 3D Reverb,” adds Hiromu. “This is a unique effect which provides a reverb with a reflected sound structure, based on the position of the source. To accurately create the noise of the aircraft flying around Shibuya, we created a preset where the energy of the early reflections is high, to mimic it bouncing off buildings, while the late reverberations are almost non-existent, as if the sound is disappearing into the distance. Audience members said that it was as if they could sense the location of the aircraft, just from the sound!”

“People were very interested when we explained that the system calculated the echoes of the surrounding buildings, to make it as realistic as possible,” adds Mr. Tsutsumi. “The sound and vibrations were so powerful that it was almost as if the audience was outside, despite being indoors. The experience of feeling the sound through your whole body, rather than just hearing it, and that feeling changing as the virtual aircraft passed was very good.”

The sound reproduction system was all Dante networked, using the Steinberg Nuendo DAW to route the sound, hosted on a PC with an AIC-128D Dante Accelerator card, to a Yamaha CL1 digital mixing console and the AFC processor, then to the individual loudspeakers.

A control PC running a Cycling ’74 Max8 environment was used to send the sound source position information to the AFC Image system via OSC. Synchronisation between sound source position data and playback sound source was established using MIDI; the MIDI control signals sent from Max8 to Nuendo, allowing the Nuendo playback to be controlled by the Max8 software.

The AR visuals were reproduced on the Psychic VR Lab Inc. STYLY VR/AR platform. Synchronisation with AR was established on a time basis, with Max8 adjusted to automatically play the sound source according to the time.

“Our plan is to develop Air Race X into a world-class, global event and we believe that the link between XR and sound technology is essential for it to grow,” says Mr. Tsutsumi. “Through promoting realistic aircraft flying with Air Race X, we have been able to work with people in XR and sound production that we have never come into contact with before. It has been really stimulating for us and we would like to expand the use of this kind of system other events.”

“AFC System”