How to recreate sound virtually in 3D

Close your eyes, let the sound guide you, and you’ll reach your “des­ti­na­tion” as sim­ply and pre­cise­ly as if you were fol­low­ing the direc­tions on your usu­al nav­i­ga­tion appli­ca­tion. This is the feat achieved by Run­Blind, a start-up launched by two researchers at École Poly­tech­nique (IP Paris).

The adven­ture began 7 years ago, when François Alouges, a pro­fes­sor at Poly­tech­nique’s Cen­tre de Math­é­ma­tiques Appliquées (CMAP), and his team inves­ti­gat­ed the appli­ca­tions of “bin­au­r­al sound”. Bin­au­r­al sound is to hear­ing what 3D vision glass­es are to sight, i.e. it enables sound to be recre­at­ed vir­tu­al­ly in 3D.

Syl­vain Fer­rand, a research engi­neer, had heard of blind ath­letes using sound guid­ance for rollerblad­ing or run­ning. The ath­lete is guid­ed by the sound of his guide’s foot­steps or roller skates. Fol­low­ing in the foot­steps of these sports­men and women, the engi­neer and his team imag­ined enabling visu­al­ly impaired peo­ple to find their way by fol­low­ing a sound. So, Syl­vain Fer­rand began a the­sis on “Bin­au­r­al spa­tial­i­sa­tion tech­niques for guid­ing blind athletes”.

Mean­ing “relat­ing to both ears”, bin­au­r­al sound is based on a method of cap­tur­ing sound adapt­ed to the mor­phol­o­gy of the human head. When we look at an object, each eye has its own field of vision (our right eye sees the same image as our left eye but off­set by a few cen­time­tres). Sim­i­lar­ly, if a noise is emit­ted to the right of our head, our left ear will hear it slight­ly out of phase. In both cas­es, it is our brain that is respon­si­ble for repro­duc­ing a sin­gle image or a sin­gle sound. The same applies to inten­si­ty. With a loud sound, the brain realis­es that the source is close. With a weak­er, more muf­fled sound, it will deduce that the source is far away. The brain also decodes the dif­fer­ences in fil­ter­ing from one ear to the next, enabling it to locate the sound source: in front of, behind, above, or below you.

Our audi­to­ry sys­tem is designed to hear par­tic­u­lar­ly well fac­ing us, in the direc­tion we are look­ing. But how do we recre­ate this three-dimen­sion­al effect? By the shape and place­ment of the micro­phones. Installed so that their cap­sules are 18 cen­time­tres apart (the aver­age spac­ing between the two ears in humans), they are mount­ed on a sup­port that sim­u­lates the shape of a head. This pre­cise­ly recre­ates the dif­fer­ence in time and inten­si­ty per­ceived between our two ears.

The team devel­ops sig­nal pro­cess­ing algo­rithms to repro­duce these char­ac­ter­is­tics and sim­u­late vir­tu­al sound sources from mono­phon­ic sounds: this is bin­au­r­al syn­the­sis. For the result to be per­fect, the bin­au­r­al lis­ten­ing sys­tem needs to be con­nect­ed to a head-track­er, a sen­sor that takes into account the user’s head move­ments in real time. So if the user turns their head to the right, the vio­lin they were hear­ing in front of them, for exam­ple, will be on their left. The same applies to all sounds, wher­ev­er they come from. To repro­duce this effect on head­phones worn by the lis­ten­er, the sound sent to each ear must be adjust­ed accord­ing to the ori­en­ta­tion of the head.

There are already sev­er­al tech­nolo­gies that exist for the three-dimen­sion­al repro­duc­tion of sound, but they require sev­er­al speak­ers spread around a room, such as Dol­by Atmos. On the oth­er hand, bin­au­r­al 3D sound dis­tri­b­u­tion is com­pat­i­ble with any hi-fi head­phones or in-ear head­phones, or even con­nect­ed glasses.

A quick reminder of the ben­e­fits of this tech­nique: the stereo sound we are used to hear­ing, patent­ed in the 1930s, was already intend­ed to recon­sti­tute the spa­tial dis­tri­b­u­tion of sound sources. To do this, record­ings from two micro­phones are played back through two loud­speak­ers or two head­phones. In stereo, when lis­ten­ing to a sym­pho­ny orches­tra from your sofa, for exam­ple, you can dis­tin­guish the sound of the piano on the left from that of the dou­ble bass on the right. But bin­au­r­al sound offers far more real­is­tic immer­sion. The lis­ten­er has the impres­sion of being present in 3D at the record­ing location.

Bin­au­r­al sound can be a pow­er­ful guid­ance tool if it is cou­pled with a head-track­ing sys­tem. The sound source indi­cat­ing direc­tion remains inde­pen­dent of head ori­en­ta­tion.  As part of his the­sis, Syl­vain Fer­rand test­ed and devel­oped an ini­tial adap­tive guid­ance pro­to­type with visu­al­ly impaired peo­ple. A sound source con­tin­u­ous­ly pre­cedes the per­son to indi­cate the path to fol­low. To do this, it was nec­es­sary to locate the user pre­cise­ly in space and cre­ate the spa­tialised vir­tu­al sources in real time to guide them.

To be used in a sport­ing con­text, the device had to be extreme­ly respon­sive and light­weight. “The first pro­to­type weighed 2.7 kilos. Now we’re using stan­dard head­phones weigh­ing just a few grams in con­junc­tion with a smart­phone”, explains Syl­vain Fer­rand. In prac­tice, the lat­est pro­to­type enables blind peo­ple to walk, run or rollerblade with par­tial auton­o­my, includ­ing in a per­for­mance-ori­ent­ed con­text (for amateurs).

What can guide the visu­al­ly impaired could also help any­one try­ing to take their eyes off the smart­phone while fol­low­ing a guid­ance appli­ca­tion. In 2019, the CMAP researchers joined forces to devel­op a start-up as part of the Poly­tech­nique incu­ba­tor, the Drahi‑X nova­tion cen­tre. They were joined by Philippe Le Borgne, an entre­pre­neur with over twen­ty years’ expe­ri­ence in IT and social and envi­ron­men­tal impact com­pa­nies. He became co-founder and chair­man of RunBlind.

The trio are now look­ing for fund­ing to finalise the project and offer a soft­ware kit that can be inte­grat­ed into var­i­ous nav­i­ga­tion appli­ca­tions and imple­ment­ed on head­sets and head­phones already mar­ket­ed to the gen­er­al pub­lic (Apple, Sony, Google…). The algo­rithm still requires devel­op­ment and opti­mi­sa­tion. But tomor­row, an ordi­nary user could set off to dis­cov­er a city, on foot with his nose to the wind, sim­ply guid­ed by the sound of his music. “Walk­ing around fol­low­ing a sound is total­ly intu­itive, you don’t have to learn any­thing, and the pre­ci­sion is extreme, to the order of a few degrees”, explain the co-founders. More effi­cient and less tir­ing than the syn­the­sised voice telling you to turn ‘slight­ly’ to the right.

Whether indoors, to find your way around a hos­pi­tal or muse­um, or out­doors, to fol­low tourist or sports routes, the appli­ca­tions for this 21^st Cen­tu­ry “fairy bell” seem limitless.

Marina Julienne