π Science and technology π Digital
Why sound research will make a big splash

How to recreate sound virtually in 3D

Sylvain Ferrand, Research engineer at Centre de mathématiques appliquées at École Polytechnique (IP Paris), François Alouges, Professor at Centre de mathématiques appliquées at École Polytechnique (IP Paris) and Philippe Le Borgne, Co-founder and developer of ConnectSuit
On February 20th, 2024 |
4 min reading time
Sylvain Ferrand
Sylvain Ferrand
Research engineer at Centre de mathématiques appliquées at École Polytechnique (IP Paris)
François Alouges
Professor at Centre de mathématiques appliquées at École Polytechnique (IP Paris)
philippe leborgne
Philippe Le Borgne
Co-founder and developer of ConnectSuit
Key takeaways
  • Binaural sound is a method of sound diffusion that enables 3D sound to be recreated virtually.
  • Coupled with a head-tracking system, binaural sound can indicate direction independently of the orientation of the head, thus becoming a high-performance guidance tool.
  • This high-performance guidance tool could be used in a variety of contexts (sports or tourism) and by everyone (visually impaired or not).
  • Currently under development, this project could eventually be used by everyone by integrating with navigation applications and listening devices already on the market.
  • Based on our morphology and the natural functioning of the brain, the binaural sound system is intuitive and requires no new learning.

Close your eyes, let the sound guide you, and you’ll reach your “des­ti­na­tion” as sim­ply and pre­cise­ly as if you were fol­low­ing the direc­tions on your usu­al nav­i­ga­tion appli­ca­tion. This is the feat achieved by Run­Blind, a start-up launched by two researchers at École Poly­tech­nique (IP Paris).

The adven­ture began 7 years ago, when François Alouges, a pro­fes­sor at Poly­tech­nique’s Cen­tre de Math­é­ma­tiques Appliquées (CMAP), and his team inves­ti­gat­ed the appli­ca­tions of “bin­au­r­al sound”. Bin­au­r­al sound is to hear­ing what 3D vision glass­es are to sight, i.e. it enables sound to be recre­at­ed vir­tu­al­ly in 3D.

Syl­vain Fer­rand, a research engi­neer, had heard of blind ath­letes using sound guid­ance for rollerblad­ing or run­ning. The ath­lete is guid­ed by the sound of his guide’s foot­steps or roller skates. Fol­low­ing in the foot­steps of these sports­men and women, the engi­neer and his team imag­ined enabling visu­al­ly impaired peo­ple to find their way by fol­low­ing a sound. So, Syl­vain Fer­rand began a the­sis on “Bin­au­r­al spa­tial­i­sa­tion tech­niques for guid­ing blind athletes”.

Definition of binaural sound

Mean­ing “relat­ing to both ears”, bin­au­r­al sound is based on a method of cap­tur­ing sound adapt­ed to the mor­phol­o­gy of the human head. When we look at an object, each eye has its own field of vision (our right eye sees the same image as our left eye but off­set by a few cen­time­tres). Sim­i­lar­ly, if a noise is emit­ted to the right of our head, our left ear will hear it slight­ly out of phase. In both cas­es, it is our brain that is respon­si­ble for repro­duc­ing a sin­gle image or a sin­gle sound. The same applies to inten­si­ty. With a loud sound, the brain realis­es that the source is close. With a weak­er, more muf­fled sound, it will deduce that the source is far away. The brain also decodes the dif­fer­ences in fil­ter­ing from one ear to the next, enabling it to locate the sound source: in front of, behind, above, or below you.

Our audi­to­ry sys­tem is designed to hear par­tic­u­lar­ly well fac­ing us, in the direc­tion we are look­ing. But how do we recre­ate this three-dimen­sion­al effect? By the shape and place­ment of the micro­phones. Installed so that their cap­sules are 18 cen­time­tres apart (the aver­age spac­ing between the two ears in humans), they are mount­ed on a sup­port that sim­u­lates the shape of a head. This pre­cise­ly recre­ates the dif­fer­ence in time and inten­si­ty per­ceived between our two ears.

The team devel­ops sig­nal pro­cess­ing algo­rithms to repro­duce these char­ac­ter­is­tics and sim­u­late vir­tu­al sound sources from mono­phon­ic sounds: this is bin­au­r­al syn­the­sis. For the result to be per­fect, the bin­au­r­al lis­ten­ing sys­tem needs to be con­nect­ed to a head-track­er, a sen­sor that takes into account the user’s head move­ments in real time. So if the user turns their head to the right, the vio­lin they were hear­ing in front of them, for exam­ple, will be on their left. The same applies to all sounds, wher­ev­er they come from. To repro­duce this effect on head­phones worn by the lis­ten­er, the sound sent to each ear must be adjust­ed accord­ing to the ori­en­ta­tion of the head.

There are already sev­er­al tech­nolo­gies that exist for the three-dimen­sion­al repro­duc­tion of sound, but they require sev­er­al speak­ers spread around a room, such as Dol­by Atmos. On the oth­er hand, bin­au­r­al 3D sound dis­tri­b­u­tion is com­pat­i­ble with any hi-fi head­phones or in-ear head­phones, or even con­nect­ed glasses.

What are the benefits of binaural spatialisation?

A quick reminder of the ben­e­fits of this tech­nique: the stereo sound we are used to hear­ing, patent­ed in the 1930s, was already intend­ed to recon­sti­tute the spa­tial dis­tri­b­u­tion of sound sources. To do this, record­ings from two micro­phones are played back through two loud­speak­ers or two head­phones. In stereo, when lis­ten­ing to a sym­pho­ny orches­tra from your sofa, for exam­ple, you can dis­tin­guish the sound of the piano on the left from that of the dou­ble bass on the right. But bin­au­r­al sound offers far more real­is­tic immer­sion. The lis­ten­er has the impres­sion of being present in 3D at the record­ing location.

Bin­au­r­al sound can be a pow­er­ful guid­ance tool if it is cou­pled with a head-track­ing sys­tem. The sound source indi­cat­ing direc­tion remains inde­pen­dent of head ori­en­ta­tion.  As part of his the­sis, Syl­vain Fer­rand test­ed and devel­oped an ini­tial adap­tive guid­ance pro­to­type with visu­al­ly impaired peo­ple. A sound source con­tin­u­ous­ly pre­cedes the per­son to indi­cate the path to fol­low. To do this, it was nec­es­sary to locate the user pre­cise­ly in space and cre­ate the spa­tialised vir­tu­al sources in real time to guide them.

To be used in a sport­ing con­text, the device had to be extreme­ly respon­sive and light­weight. “The first pro­to­type weighed 2.7 kilos. Now we’re using stan­dard head­phones weigh­ing just a few grams in con­junc­tion with a smart­phone”, explains Syl­vain Fer­rand. In prac­tice, the lat­est pro­to­type enables blind peo­ple to walk, run or rollerblade with par­tial auton­o­my, includ­ing in a per­for­mance-ori­ent­ed con­text (for amateurs).

One device, many applications 

What can guide the visu­al­ly impaired could also help any­one try­ing to take their eyes off the smart­phone while fol­low­ing a guid­ance appli­ca­tion. In 2019, the CMAP researchers joined forces to devel­op a start-up as part of the Poly­tech­nique incu­ba­tor, the Drahi‑X nova­tion cen­tre. They were joined by Philippe Le Borgne, an entre­pre­neur with over twen­ty years’ expe­ri­ence in IT and social and envi­ron­men­tal impact com­pa­nies. He became co-founder and chair­man of RunBlind.

The trio are now look­ing for fund­ing to finalise the project and offer a soft­ware kit that can be inte­grat­ed into var­i­ous nav­i­ga­tion appli­ca­tions and imple­ment­ed on head­sets and head­phones already mar­ket­ed to the gen­er­al pub­lic (Apple, Sony, Google…). The algo­rithm still requires devel­op­ment and opti­mi­sa­tion. But tomor­row, an ordi­nary user could set off to dis­cov­er a city, on foot with his nose to the wind, sim­ply guid­ed by the sound of his music. “Walk­ing around fol­low­ing a sound is total­ly intu­itive, you don’t have to learn any­thing, and the pre­ci­sion is extreme, to the order of a few degrees”, explain the co-founders. More effi­cient and less tir­ing than the syn­the­sised voice telling you to turn ‘slight­ly’ to the right.

Whether indoors, to find your way around a hos­pi­tal or muse­um, or out­doors, to fol­low tourist or sports routes, the appli­ca­tions for this 21st Cen­tu­ry “fairy bell” seem limitless.

Marina Julienne

Our world explained with science. Every week, in your inbox.

Get the newsletter