Now, the ears are going to enter the Yuan universe

What do you think of the AR/VR device?

Is the sci -fi brought by the sci -fi by the stack of virtual and reality?

When everyone's attention is still focused on the visual interaction, the change in the industry in the industry has quietly emerged.

ROKID recently released by ROKID, a domestic human -machine interactive product platform company, 6DOF spatial sound field technology DEMO video applied to AR glasses.

Different from the hearing experience brought by traditional dual -channel and stereo sound, 6DOF spatial sound field technology can simulate the strength and weakness of the sound between the sound position and the human ear in the hybrid reality and the direction Change, thereby bringing AR glasses to bring users a more sense of auditory experience.

What is the 6DOF space sound field?

The 6DOF space sound field is actually a manifestation of the sound in the three -dimensional field. But this is not to simply make the sound more three -dimensional through more channels, but to synchronize the audio spatialization process with video space. Therefore, it contains two necessary elements -real -time feedback of 3D audio and head movement.

First look at the first must -have element of 6DOF space sound field -3D audio. The traditional 5.1 channel can show the sound on a horizontal plane, so the sound positioning has two dimensions: 2D audio. When a audio also has up and down dimensions, this audio is 3D audio.

Figure: 3D audio icon (the picture originated in the Internet)

The second must -have element of 6DOF space sound field -real -time feedback on head movement. In the real world, when our heads are rotated or displaced, the absolute position of the sound source itself will not change, and the relative direction of the sound source and the head will change.

For example: In front of you, there is a guitar playing music. If you turn to the right, the sound of music will change to your left; if you turn to the left, the sound of music will be relatively to your right. Therefore, in the mixing reality, to achieve a hearing experience closer to reality, it is necessary to accurately locate the spatial position between the sound source and the user's head, that is, real -time tracking of the user's head movement.

The realization of 6DOF space sound field requires high coordination of software and hardware

It is not easy to meet the two necessary elements of the 6DOF space sound field technology. At the technical level, this requires the highly fused Space Engine and Audio Engine, and makes full use of hardware resources.

The core work of the space engine is the fusion of the virtual and real space. The engine pre -uses three -dimensional reconstruction technology to build a map, establish a virtual world coordinate system, and add virtual objects, setting the attributes such as the position, shape, and material.

When running, the observer (such as wearing AR glasses, the observer is the head position of the human head) by processing the sensor data (such as wearing AR glasses, the observer is the person's head position) and the local map, and then through the map matching You can unify the position in the virtual world coordinate system.

According to different sensor types and quantities, the space engine can obtain the observer's different types of different types of freedom-dof information, thereby providing the necessary spatial information for the audio engine.

For example, the degree of freedom of the human head is divided into: existing displacement and rotating 6DOF, only rotating 3DOF, and the virtual space of the human head. The corresponding audio can be divided into 6DOF space sound fields, 3DOF space sound fields, and surround sounds. Therefore, the 6DOF space sound field technology needs to obtain more complicated people's freedom.

Figure: 6DOF freedom (picture originated from the Internet)

The core work of the audio engine is to make convolution to the audio signal and HRTFS (Head Related Tranfer Function, the head -related transmission function, referred to as the head pass function) to generate double -ear audio. HRTFS is the three measured dimensions of coordinated samples measured by the three measured dimensions of the horizontal angle, Elevation, and Distance. The accuracy is the leading factor of the 6DOF space sound field.

However, the current commercial HRTFS database can achieve the accuracy that is not completely comparable to the human ear listening ability. What is more challenging is that everyone’s ergonomic parameters and psychological acoustic systems are different. Essence

It is obviously unrealistic to accurately measure everyone's HRTFS parameters. How can we be a personalized HRTFS at low cost? The ROKID technical team, which has implemented the 6DOF space sound field technology, gives a solution idea, that is, in the case of considering the computing performance of NPU/GPU, combined with deep learning technology, the more refined ingredients are made to produce more refined ingredients. Essence

Figure: XR device application 6DOF space sound field requires software and hardware

In addition, in order to increase the effects of blocking, reflection, reverberation, and so that the 6DOF space sound field is more realistic, it is necessary to use the light tracking and Wave Acoustics (SPHERICAL HARMONMOnics) such as Geometric AcoutStics, such as Geometric AcoutStics, ) Decomposition and other technologies. This has extremely high requirements for the computing power of the equipment, and it will also bring greater power consumption load to the equipment, increasing equipment costs and safety risks. Therefore, in practical applications, corresponding compromises and balances are often required to make the order, voice quality, and space accuracy of the ball harmony function. In addition to the algorithm level, the application of 6DOF spatial sound field technology must also consider the hardware form of the device. Many of the current audio algorithms are based on ear -entry or header speakers, but AR glasses are wearable devices worn by users for a long time in the future. The fusion mission with numbers. Therefore, while maintaining an open speaker design, how to ensure the presentation effect and security of the 6DOF spatial sound field has become a new challenge.

At present, the ROKID technology team adopted to solve privacy issues through the research and use of directional acoustic technology. At the same time, in order to make the sound effects of 6DOF spatial sound fields richer and full, the sound quality is enhanced through the design of the sound structure, the repair of sound audio, and the sound harm of the sound of human ear hearing, so as to reduce the loss of audio effects, so that users can truly truly really Feel "sound immersive".

A sound revolution is quietly rising

The application of 6DOF Space Sound field technology on AR devices is landing, let us see the vast application space of sound in mixed reality. Through the 6DOF space sound field technology, AR glasses and other devices can get rid of the viewing angle (FOV) limit, allowing users to discover the content outside the screen through the sound, so as to realize the content presentation of 360 degrees.

At the same time, in addition to the visual interaction, the application of 6DOF spatial sound field technology has made hearing a new interactive dimension. Combined with the 6DOF space sound field, users can quickly and accurately set the direction of sound objects in the hybrid reality, clearly distinguish the receiving sound information, and feel the changes in sound distance and position ... This will allow users Experience to further reduce the sense of fragmentation of digital and real worlds in the mixed reality.

The new hearing experience brought by the 6DOF space sound field is impacting the traditional three -dimensional sound of more than half a century, but the application and popularity of any new technology rely on the power of a team or a company. The entry threshold attracts the addition of more industry forces.

For example, Rokid means that the 6DOF space sound field will be integrated into the newly upgraded version of the Yodaos-XR operating system as the basic capabilities of the Yodaos-XR operating system for industry developers. At the same time, Rokid also plans to promote the development of special sound effects used in AR glasses, such as high -fidelity sound effects surrounding and micro -heavy bass, and use efficient and easy -to -use SDK to truly implement it.

It is reported that Rokid's newly upgraded YodaOS-XR operating system may be released to the public in the second half of this year, including many natural interaction engines, friendly UI interfaces, native XR applications, and application development frameworks. At that time, developers can focus on the polishing of boutique content and develop various imaginative applications and content, such as XR games, XR conferences, XR social, XR theaters, etc., and users with users enter the real AR world.

The ultimate goal of the XR era is the perfect fusion of the virtual world and the physical world. This integration is mainly to simulate and enhance the way of information exchange, such as tactile, hearing, vision, smell, and taste.

The application of 6DOF Space Sound field and other technologies broaden the imagination of the XR device, and quietly set off a perceptual interaction revolution. We may foresee that after vision and hearing, "sensory experience" such as touch, smell, and taste will also be redefined in the XR era.

- END -