How to recreate sound virtually in 3D

Close your eyes, let the sound guide you, and you’ll reach your “des­tin­a­tion” as simply and pre­cisely as if you were fol­low­ing the dir­ec­tions on your usu­al nav­ig­a­tion applic­a­tion. This is the feat achieved by Run­Blind, a start-up launched by two research­ers at École Poly­tech­nique (IP Paris).

The adven­ture began 7 years ago, when François Alouges, a pro­fess­or at Poly­tech­nique’s Centre de Math­ématiques Appli­quées (CMAP), and his team invest­ig­ated the applic­a­tions of “bin­aur­al sound”. Bin­aur­al sound is to hear­ing what 3D vis­ion glasses are to sight, i.e. it enables sound to be recre­ated vir­tu­ally in 3D.

Sylvain Fer­rand, a research engin­eer, had heard of blind ath­letes using sound guid­ance for rollerblad­ing or run­ning. The ath­lete is guided by the sound of his guide’s foot­steps or roller skates. Fol­low­ing in the foot­steps of these sports­men and women, the engin­eer and his team ima­gined enabling visu­ally impaired people to find their way by fol­low­ing a sound. So, Sylvain Fer­rand began a thes­is on “Bin­aur­al spa­tial­isa­tion tech­niques for guid­ing blind athletes”.

Mean­ing “relat­ing to both ears”, bin­aur­al sound is based on a meth­od of cap­tur­ing sound adap­ted to the mor­pho­logy of the human head. When we look at an object, each eye has its own field of vis­ion (our right eye sees the same image as our left eye but off­set by a few cen­ti­metres). Sim­il­arly, if a noise is emit­ted to the right of our head, our left ear will hear it slightly out of phase. In both cases, it is our brain that is respons­ible for repro­du­cing a single image or a single sound. The same applies to intens­ity. With a loud sound, the brain real­ises that the source is close. With a weak­er, more muffled sound, it will deduce that the source is far away. The brain also decodes the dif­fer­ences in fil­ter­ing from one ear to the next, enabling it to loc­ate the sound source: in front of, behind, above, or below you.

Our aud­it­ory sys­tem is designed to hear par­tic­u­larly well facing us, in the dir­ec­tion we are look­ing. But how do we recre­ate this three-dimen­sion­al effect? By the shape and place­ment of the micro­phones. Installed so that their cap­sules are 18 cen­ti­metres apart (the aver­age spa­cing between the two ears in humans), they are moun­ted on a sup­port that sim­u­lates the shape of a head. This pre­cisely recre­ates the dif­fer­ence in time and intens­ity per­ceived between our two ears.

The team devel­ops sig­nal pro­cessing algorithms to repro­duce these char­ac­ter­ist­ics and sim­u­late vir­tu­al sound sources from mono­phon­ic sounds: this is bin­aur­al syn­thes­is. For the res­ult to be per­fect, the bin­aur­al listen­ing sys­tem needs to be con­nec­ted to a head-track­er, a sensor that takes into account the user­’s head move­ments in real time. So if the user turns their head to the right, the viol­in they were hear­ing in front of them, for example, will be on their left. The same applies to all sounds, wherever they come from. To repro­duce this effect on head­phones worn by the listen­er, the sound sent to each ear must be adjus­ted accord­ing to the ori­ent­a­tion of the head.

There are already sev­er­al tech­no­lo­gies that exist for the three-dimen­sion­al repro­duc­tion of sound, but they require sev­er­al speak­ers spread around a room, such as Dolby Atmos. On the oth­er hand, bin­aur­al 3D sound dis­tri­bu­tion is com­pat­ible with any hi-fi head­phones or in-ear head­phones, or even con­nec­ted glasses.

A quick remind­er of the bene­fits of this tech­nique: the ste­reo sound we are used to hear­ing, pat­en­ted in the 1930s, was already inten­ded to recon­sti­t­ute the spa­tial dis­tri­bu­tion of sound sources. To do this, record­ings from two micro­phones are played back through two loud­speak­ers or two head­phones. In ste­reo, when listen­ing to a sym­phony orches­tra from your sofa, for example, you can dis­tin­guish the sound of the piano on the left from that of the double bass on the right. But bin­aur­al sound offers far more real­ist­ic immer­sion. The listen­er has the impres­sion of being present in 3D at the record­ing location.

Bin­aur­al sound can be a power­ful guid­ance tool if it is coupled with a head-track­ing sys­tem. The sound source indic­at­ing dir­ec­tion remains inde­pend­ent of head ori­ent­a­tion.  As part of his thes­is, Sylvain Fer­rand tested and developed an ini­tial adapt­ive guid­ance pro­to­type with visu­ally impaired people. A sound source con­tinu­ously pre­cedes the per­son to indic­ate the path to fol­low. To do this, it was neces­sary to loc­ate the user pre­cisely in space and cre­ate the spa­tial­ised vir­tu­al sources in real time to guide them.

To be used in a sport­ing con­text, the device had to be extremely respons­ive and light­weight. “The first pro­to­type weighed 2.7 kilos. Now we’re using stand­ard head­phones weigh­ing just a few grams in con­junc­tion with a smart­phone”, explains Sylvain Fer­rand. In prac­tice, the latest pro­to­type enables blind people to walk, run or rollerblade with par­tial autonomy, includ­ing in a per­form­ance-ori­ented con­text (for amateurs).

What can guide the visu­ally impaired could also help any­one try­ing to take their eyes off the smart­phone while fol­low­ing a guid­ance applic­a­tion. In 2019, the CMAP research­ers joined forces to devel­op a start-up as part of the Poly­tech­nique incub­at­or, the Drahi‑X nova­tion centre. They were joined by Phil­ippe Le Bor­gne, an entre­pren­eur with over twenty years’ exper­i­ence in IT and social and envir­on­ment­al impact com­pan­ies. He became co-founder and chair­man of RunBlind.

The trio are now look­ing for fund­ing to final­ise the pro­ject and offer a soft­ware kit that can be integ­rated into vari­ous nav­ig­a­tion applic­a­tions and imple­men­ted on head­sets and head­phones already mar­keted to the gen­er­al pub­lic (Apple, Sony, Google…). The algorithm still requires devel­op­ment and optim­isa­tion. But tomor­row, an ordin­ary user could set off to dis­cov­er a city, on foot with his nose to the wind, simply guided by the sound of his music. “Walk­ing around fol­low­ing a sound is totally intu­it­ive, you don’t have to learn any­thing, and the pre­ci­sion is extreme, to the order of a few degrees”, explain the co-founders. More effi­cient and less tir­ing than the syn­thes­ised voice telling you to turn ‘slightly’ to the right.

Wheth­er indoors, to find your way around a hos­pit­al or museum, or out­doors, to fol­low tour­ist or sports routes, the applic­a­tions for this 21^st Cen­tury “fairy bell” seem limitless.

Marina Julienne