How to recreate sound virtually in 3D

Close your eyes, let the sound guide you, and you’ll reach your “des­ti­na­tion” as sim­ply and pre­ci­se­ly as if you were fol­lo­wing the direc­tions on your usual navi­ga­tion appli­ca­tion. This is the feat achie­ved by Run­Blind, a start-up laun­ched by two resear­chers at École Poly­tech­nique (IP Paris).

The adven­ture began 7 years ago, when Fran­çois Alouges, a pro­fes­sor at Poly­tech­ni­que’s Centre de Mathé­ma­tiques Appli­quées (CMAP), and his team inves­ti­ga­ted the appli­ca­tions of “binau­ral sound”. Binau­ral sound is to hea­ring what 3D vision glasses are to sight, i.e. it enables sound to be recrea­ted vir­tual­ly in 3D.

Syl­vain Fer­rand, a research engi­neer, had heard of blind ath­letes using sound gui­dance for rol­ler­bla­ding or run­ning. The ath­lete is gui­ded by the sound of his guide’s foots­teps or rol­ler skates. Fol­lo­wing in the foots­teps of these sports­men and women, the engi­neer and his team ima­gi­ned enabling visual­ly impai­red people to find their way by fol­lo­wing a sound. So, Syl­vain Fer­rand began a the­sis on “Binau­ral spa­tia­li­sa­tion tech­niques for gui­ding blind athletes”.

Mea­ning “rela­ting to both ears”, binau­ral sound is based on a method of cap­tu­ring sound adap­ted to the mor­pho­lo­gy of the human head. When we look at an object, each eye has its own field of vision (our right eye sees the same image as our left eye but off­set by a few cen­ti­metres). Simi­lar­ly, if a noise is emit­ted to the right of our head, our left ear will hear it slight­ly out of phase. In both cases, it is our brain that is res­pon­sible for repro­du­cing a single image or a single sound. The same applies to inten­si­ty. With a loud sound, the brain rea­lises that the source is close. With a wea­ker, more muf­fled sound, it will deduce that the source is far away. The brain also decodes the dif­fe­rences in fil­te­ring from one ear to the next, enabling it to locate the sound source : in front of, behind, above, or below you.

Our audi­to­ry sys­tem is desi­gned to hear par­ti­cu­lar­ly well facing us, in the direc­tion we are loo­king. But how do we recreate this three-dimen­sio­nal effect ? By the shape and pla­ce­ment of the micro­phones. Ins­tal­led so that their cap­sules are 18 cen­ti­metres apart (the ave­rage spa­cing bet­ween the two ears in humans), they are moun­ted on a sup­port that simu­lates the shape of a head. This pre­ci­se­ly recreates the dif­fe­rence in time and inten­si­ty per­cei­ved bet­ween our two ears.

The team deve­lops signal pro­ces­sing algo­rithms to repro­duce these cha­rac­te­ris­tics and simu­late vir­tual sound sources from mono­pho­nic sounds : this is binau­ral syn­the­sis. For the result to be per­fect, the binau­ral lis­te­ning sys­tem needs to be connec­ted to a head-tra­cker, a sen­sor that takes into account the user’s head move­ments in real time. So if the user turns their head to the right, the vio­lin they were hea­ring in front of them, for example, will be on their left. The same applies to all sounds, whe­re­ver they come from. To repro­duce this effect on head­phones worn by the lis­te­ner, the sound sent to each ear must be adjus­ted accor­ding to the orien­ta­tion of the head.

There are alrea­dy seve­ral tech­no­lo­gies that exist for the three-dimen­sio­nal repro­duc­tion of sound, but they require seve­ral spea­kers spread around a room, such as Dol­by Atmos. On the other hand, binau­ral 3D sound dis­tri­bu­tion is com­pa­tible with any hi-fi head­phones or in-ear head­phones, or even connec­ted glasses.

A quick remin­der of the bene­fits of this tech­nique : the ste­reo sound we are used to hea­ring, paten­ted in the 1930s, was alrea­dy inten­ded to recons­ti­tute the spa­tial dis­tri­bu­tion of sound sources. To do this, recor­dings from two micro­phones are played back through two loud­spea­kers or two head­phones. In ste­reo, when lis­te­ning to a sym­pho­ny orches­tra from your sofa, for example, you can dis­tin­guish the sound of the pia­no on the left from that of the double bass on the right. But binau­ral sound offers far more rea­lis­tic immer­sion. The lis­te­ner has the impres­sion of being present in 3D at the recor­ding location.

Binau­ral sound can be a power­ful gui­dance tool if it is cou­pled with a head-tra­cking sys­tem. The sound source indi­ca­ting direc­tion remains inde­pendent of head orien­ta­tion.  As part of his the­sis, Syl­vain Fer­rand tes­ted and deve­lo­ped an ini­tial adap­tive gui­dance pro­to­type with visual­ly impai­red people. A sound source conti­nuous­ly pre­cedes the per­son to indi­cate the path to fol­low. To do this, it was neces­sa­ry to locate the user pre­ci­se­ly in space and create the spa­tia­li­sed vir­tual sources in real time to guide them.

To be used in a spor­ting context, the device had to be extre­me­ly res­pon­sive and light­weight. “The first pro­to­type wei­ghed 2.7 kilos. Now we’re using stan­dard head­phones wei­ghing just a few grams in conjunc­tion with a smart­phone”, explains Syl­vain Fer­rand. In prac­tice, the latest pro­to­type enables blind people to walk, run or rol­ler­blade with par­tial auto­no­my, inclu­ding in a per­for­mance-orien­ted context (for amateurs).

What can guide the visual­ly impai­red could also help anyone trying to take their eyes off the smart­phone while fol­lo­wing a gui­dance appli­ca­tion. In 2019, the CMAP resear­chers joi­ned forces to deve­lop a start-up as part of the Poly­tech­nique incu­ba­tor, the Drahi‑X nova­tion centre. They were joi­ned by Phi­lippe Le Borgne, an entre­pre­neur with over twen­ty years’ expe­rience in IT and social and envi­ron­men­tal impact com­pa­nies. He became co-foun­der and chair­man of RunBlind.

The trio are now loo­king for fun­ding to fina­lise the pro­ject and offer a soft­ware kit that can be inte­gra­ted into various navi­ga­tion appli­ca­tions and imple­men­ted on head­sets and head­phones alrea­dy mar­ke­ted to the gene­ral public (Apple, Sony, Google…). The algo­rithm still requires deve­lop­ment and opti­mi­sa­tion. But tomor­row, an ordi­na­ry user could set off to dis­co­ver a city, on foot with his nose to the wind, sim­ply gui­ded by the sound of his music. “Wal­king around fol­lo­wing a sound is total­ly intui­tive, you don’t have to learn any­thing, and the pre­ci­sion is extreme, to the order of a few degrees”, explain the co-foun­ders. More effi­cient and less tiring than the syn­the­si­sed voice tel­ling you to turn ‘slight­ly’ to the right.

Whe­ther indoors, to find your way around a hos­pi­tal or museum, or out­doors, to fol­low tou­rist or sports routes, the appli­ca­tions for this 21^st Cen­tu­ry “fai­ry bell” seem limitless.

Marina Julienne