Modelling virtual worlds for scientific research

Com­puter graph­ics can be used to rep­res­ent anim­ated vir­tu­al spaces in three dimen­sions. Mar­ie-Paule Cani car­ries out her research in the Mod­el­ing Sim­u­la­tion and Learn­ing divi­sion of École poly­tech­nique’s Com­puter Sci­ence Labor­at­ory (LIX). A renowned research­er who has won awards for her work, she explains: “Ori­gin­ally, LIX was mainly a fun­da­ment­al com­put­ing labor­at­ory. In oth­er words, the research­ers worked mainly on the math­em­at­ic­al and algorithmic found­a­tions of our dis­cip­line. Then the labor­at­ory diver­si­fied and now cov­ers a wide range of themes, from bioin­form­at­ics to arti­fi­cial intel­li­gence. Since 2017, sev­er­al com­puter graph­ics teams have developed, includ­ing the VISTA team, of which I am a mem­ber. On the one hand, we are work­ing on new meth­ods for aid­ing cre­ation, based on know­ledge and/or machine learn­ing, as well as on meth­ods for anim­at­ing move­ments and deform­a­tions in the 3D vir­tu­al worlds thus created.”

The applic­a­tions of com­puter graph­ics are many and var­ied. It helps to pro­to­type and vir­tu­ally test objects that are to be man­u­fac­tured. In the spe­cial effects sec­tor, it can be used to cre­ate spec­tac­u­lar scenes for cinema and anim­ated films. Video games, anoth­er key area, use these tech­no­lo­gies to plunge play­ers into immers­ive worlds that are visu­ally as close to real­ity as pos­sible. Finally, 3D vir­tu­al worlds, which can be explored immers­ively, are essen­tial for train­ing sim­u­lat­ors in high-risk situ­ations, whatever the field (trans­port, med­ic­al, energy, mil­it­ary, etc.).

“For some years now, my main research pro­ject has been to explore the use of com­puter graph­ics as a medi­um for visu­al rep­res­ent­a­tion and exper­i­ment­a­tion for sci­ent­ists in oth­er dis­cip­lines.” When we think, we have vis­ions, anim­ated rep­res­ent­a­tions of what we ima­gine. The research­er illus­trates her point with the example of a cell bio­logy research­er. “Using their know­ledge, they men­tally visu­al­ise the pro­cess of cell divi­sion, for example dur­ing the growth of a tumour. How­ever, the schem­at­ic rep­res­ent­a­tions that they can cre­ate with paper and pen­cil will only show this pro­cess at a fixed moment in time.” If they want to gen­er­ate 3D anim­a­tions, they will have to explain their vis­ion to an artist who has mastered mod­el­ling soft­ware. “After a lot of to-ing and fro-ing, they will prob­ably end up with an illus­tra­tion that comes close to their vis­ion. How­ever, they will not be able to change the para­met­ers or inter­act with the mod­el represented.”

“Our aim is to provide sci­ent­ists with the means to cre­ate 3D and anim­ated illus­tra­tions them­selves, based on an “express­ive” mod­el­ling meth­od­o­logy, the found­a­tions of which I helped to lay. It is inspired by the way we humans under­stand the world around us and cre­ate objects in it.” The res­ult­ing inter­act­ive vir­tu­al worlds are very rich, and can serve as a sup­port for sci­entif­ic thought for research­ers from many disciplines.

Nature, with all its rich detail, is dif­fi­cult to mod­el. A mov­ing head of hair, a water­fall or a wind-blown forest are all com­plex phe­nom­ena that are dif­fi­cult to anim­ate in real time. To meet the rep­res­ent­a­tion needs of sci­ent­ists, the VISTA team has developed a three-stage methodology:

First, it uses multi-lay­er mod­els, i.e. it breaks down the prob­lem into dif­fer­ent sub-mod­els. “The sky is par­tic­u­larly dif­fi­cult to anim­ate,” explains the research­er. “So, to rep­res­ent clouds quickly, we com­bined sub-mod­els. First, we stacked 2D lay­ers rep­res­ent­ing dif­fer­ent cloud lay­ers. Then we had to integ­rate the air flows. To do this, we com­bined the mod­els with atmo­spher­ic dynam­ics – i.e. trans­fers between lay­ers – mod­elled using flu­id mech­an­ics. Finally, to get as close to real­ity as pos­sible, we clas­si­fied the dif­fer­ent types of clouds (stratus, sir­rus, cumu­lus, etc.) and added pro­ced­ur­al details to enhance visu­al real­ism. The res­ult is an anim­ated com­puter-gen­er­ated image that rep­res­ents the move­ment of clouds accord­ing to their nature and tem­per­at­ure. Finally, even very com­plex objects can be anim­ated approx­im­ately in real time.”

The second ingredi­ent in cre­at­ing visu­al rep­res­ent­a­tions is express­ive mod­el­ling. 3D mod­el­ling soft­ware is often dif­fi­cult to learn. Mar­ie-Paule Cani and her team want to make it pos­sible to cre­ate dir­ectly through ges­ture. “The idea is to use these express­ive ges­tures, which resemble draw­ing and sculpt­ing ges­tures, to enable the sci­ent­ist to cre­ate an anim­ated 3D envir­on­ment that cor­res­ponds to what they imagine.”

Finally, the meth­od­o­logy makes use of learn­ing from examples, be they ad hoc pro­cesses, deep learn­ing or rein­force­ment learn­ing. These pro­cesses make it pos­sible to cre­ate con­tent (shapes or move­ments) that is indis­tin­guish­able from the res­ults of sim­u­la­tions or examples provided by the user. Whenev­er pos­sible, ad hoc pro­cesses are used, as they are less costly in terms of time and energy than deep learn­ing – where the machine must ingest a large amount of data. “To cre­ate a land­scape,” illus­trates Mar­ie-Paule Cani, “the user will place cer­tain ele­ments (pebbles, trees, tufts of grass, etc.) in a small area by hand, to illus­trate the spa­tial rela­tion­ships they want to see between these ele­ments and with oth­er factors, such as the slope of the land. Then the com­puter will stat­ist­ic­ally learn the dis­tri­bu­tion, i.e. the cor­rel­a­tions between types of ele­ments”. Just as when paint­ing a pic­ture, the user then has brushes with which to “stat­ist­ic­ally paint” the vir­tu­al world in real time.

It is by com­bin­ing these three para­met­ers (express­ive mod­el­ling, multi-lay­er­ing and learn­ing from examples) that the team can cre­ate and anim­ate com­plex and var­ied vir­tu­al envir­on­ments in 3D. In col­lab­or­a­tion with research­ers from oth­er dis­cip­lines, they have been able to rep­res­ent the form­a­tion of land by gla­cial erosion¹, a Medi­ter­ranean or Alpine eco­sys­tem² or the form­a­tion of moun­tain ranges³.

In order to cre­ate new forms, we need to be able to inter­act with the mod­els, so that they respond in real time to the user­’s ges­tures. In addi­tion to the mod­els and algorithms, Mar­ie-Paule Cani and her team some­times have to build spe­cif­ic tools to cap­ture move­ments, such as an exten­ded mouse with force sensors⁴. “We developed a hand nav­ig­at­or, a tool that con­nects a mouse with six degrees of free­dom to small sensors activ­ated by the user­’s fin­gers. This tool was used to anim­ate a vir­tu­al hand, cap­able of sculpt­ing 3D mod­el­ling clay.” The user can mod­el live on the com­puter, but in some cases, tact­ile inter­ac­tions and vir­tu­al real­ity devices enhance intu­it­ive inter­ac­tion with mod­els, facil­it­at­ing cre­ation and allow­ing users to manip­u­late com­plex envir­on­ments in a nat­ur­al way. She con­tin­ues, “invent­ing these vir­tu­al real­ity tools has some­times been neces­sary to fill a gap and advance research. Unlike gen­er­at­ive AI (often developed to cre­ate for us), the intel­li­gent sys­tems I am inter­ested in allow ges­tur­al inter­ac­tion, and are inten­ded to make us humans more creative.”

The VISTA team reg­u­larly works with sci­ent­ists from oth­er dis­cip­lines. In col­lab­or­a­tion with the lat­ter, the research­ers devel­op cre­ation tools based on their mod­els, which enable them to cre­ate examples dir­ectly. “From 2017 to 2021, for example, we worked with palae­on­to­lo­gists who want to “see” their palaeo­cli­mat­ic mod­el and the res­ult­ing dis­tri­bu­tion of fauna and flora in the Tautavel val­ley.” Loc­ated in the Pyrénées-Ori­entales region, the Tautavel val­ley is a major pre­his­tor­ic site. Archae­olo­gic­al digs there have unearthed numer­ous bones, tools and traces dat­ing back more than 300,000 years. “The aim was to repro­duce the eco­sys­tem based on the data and hypo­theses of the research­ers. We rep­res­en­ted the palaeo­cli­mate, and gen­er­ated the flora and fauna that inhab­ited the val­ley at that time accord­ing to the para­met­ers (tem­per­at­ure, hydra­tion and sun­shine) as well as the prob­able list of spe­cies present giv­en by the palae­on­to­lo­gists.” Thanks to the anim­ated 3D rep­res­ent­a­tion con­struc­ted in this way, research­ers can see the flaws in their mod­els and adjust their hypotheses.

Could com­puter graph­ics be the first step towards time travel? Without going that far, there is no doubt that this dis­cip­line trans­ports us. Ima­gine tak­ing a vir­tu­al walk through a 3D recon­struc­tion of a val­ley in the Palaeo­lith­ic peri­od – it’s a form of time travel. What is cer­tain is that com­puter graph­ics can help sci­ent­ists to improve their men­tal vis­ion and their mod­els, by com­bin­ing know­ledge and digit­al learning.

The con­stant evol­u­tion of com­puter graph­ics offers fas­cin­at­ing pro­spects for the future

But it also works the oth­er way round. To test their mod­el, Mar­ie-Paule Can­i’s team is repro­du­cing actu­al land­scapes. Com­par­ing their res­ults with the exist­ing land­scape provided by satel­lite images enables them to adjust their meth­od­o­logy if neces­sary to obtain the most accur­ate rep­res­ent­a­tion of real­ity pos­sible, and then to val­id­ate their model.

The con­stant evol­u­tion of com­puter graph­ics offers fas­cin­at­ing pro­spects for the future. By enabling experts to visu­al­ise their men­tal rep­res­ent­a­tions and test their sci­entif­ic hypo­theses more quickly, it opens up new aven­ues for research and education.

One future pro­ject involves using express­ive mod­el­ling to raise pub­lic aware­ness of cli­mate change. By allow­ing users to manip­u­late hypo­theses and visu­al­ise in 3D the effects of their choices on our plan­et, this approach would help to get the gen­er­al pub­lic more involved in cur­rent envir­on­ment­al issues. In this way, com­puter graph­ics not only mod­els real­ity, it also becomes a cru­cial tool for sup­port­ing sci­entif­ic research and enabling every­one, stu­dents and the gen­er­al pub­lic alike, to gain a bet­ter under­stand­ing of our world.

Loraine Odot