How to spot errors in medical research publications

Launched in June 2025 with fund­ing from the phil­an­throp­ic organ­isa­tion Coef­fi­cient Giv­ing, The Med­ic­al Evid­ence Pro­ject aims to detect unre­li­able art­icles, fraud­u­lent or oth­er­wise, that are likely to influ­ence cur­rent med­ic­al recom­mend­a­tions. We dis­cuss this with Alice Dreger, PhD and edit­or for the project.

With sev­er­al mil­lion sci­entif­ic art­icles pub­lished each year in peer-reviewed journ­als, clean­ing up the sci­entif­ic lit­er­at­ure is like look­ing for a needle in a hay­stack. Where should one start to track down anom­alies? Whilst some advoc­ate a sys­tem­at­ic ana­lys­is of the entire lit­er­at­ure to identi­fy traces left by cer­tain auto­mated fraud­u­lent prac­tices, oth­ers intend to focus on a much more tar­geted assess­ment, focus­ing solely on high-impact pub­lic­a­tions. This second approach is the one adop­ted by The Med­ic­al Evid­ence Pro­ject, an explor­at­ory US ini­ti­at­ive fun­ded by a $900,000 seed grant from Coef­fi­cient Giv­ing. Formerly known as Open Phil­an­thropy, this organ­isa­tion cham­pi­ons “effect­ive” altru­ism, mean­ing it sup­ports ini­ti­at­ives based on their expec­ted meas­ur­able impact.

The pro­ject brings togeth­er per­man­ent mem­bers and con­sult­ants led by James Heath­ers, PhD, a research asso­ci­ate at Lin­naeus Uni­ver­sity in Sweden who has been work­ing out­side the tra­di­tion­al aca­dem­ic frame­work for sev­er­al years. It is hos­ted by the Centre for Sci­entif­ic Integ­rity. This Amer­ic­an non-profit organ­isa­tion, launched in 2014, is known as the par­ent organ­isa­tion of Retrac­tion Watch, a news site that tracks and doc­u­ments retrac­tions of sci­entif­ic articles.

The Med­ic­al Evid­ence Pro­ject has there­fore made effect­ive­ness its main focus, choos­ing to con­cen­trate on art­icles that sup­port cur­rent med­ic­al recom­mend­a­tions hav­ing a dir­ect and sig­ni­fic­ant influ­ence on patient mor­bid­ity and mor­tal­ity, for example those incor­por­ated into offi­cial clin­ic­al guidelines, or which influ­ence stand­ard med­ic­al prac­tice. An object­ive that James Heath­ers sums up on his Linked­In page with a catchy slo­gan: “Find­ing bad med­ic­al evid­ence before it kills people.”

The team’s busi­ness is based on forensic metas­cience. “Forensic metas­cience is kind of what it sounds like: it’s detect­ive work (forensics) using sci­entif­ic tools (sci­ence) to exam­ine the sci­entif­ic lit­er­at­ure (a meta move),” explains Alice Dreger. In prac­tice, ana­lysts mainly use stat­ist­ic­al meth­ods to identi­fy incon­sist­en­cies in the data and con­clu­sions presen­ted in art­icles. As James Heath­ers, who form­al­ised the concept in an art­icle entitled Intro­duc­tion to Forensic Metas­cience¹, points out, “Forensic metas­cientif­ic ana­lys­is is designed to modi­fy trust by eval­u­at­ing research con­sist­ency. It is not designed to “find fraud”. While this may hap­pen, it is not the sole focus of forensic metas­cience as a research area and prac­tice, it is simply the loudest consequence.”

Just one year after its launch, it must be admit­ted that the project’s vis­ible res­ults are still lim­ited. Alice Dreger, how­ever, sees noth­ing sur­pris­ing in this: “The first year of our pro­ject was designed from the out­set to focus on the devel­op­ment of a sys­tem to fig­ure out where our ana­lysts should be focus­ing efforts in terms of deep ana­lys­is.” The team has in fact announced the devel­op­ment of an ini­tial tool, GRIM‑U², cap­able of detect­ing a spe­cif­ic type of math­em­at­ic­al incon­sist­ency in cer­tain stat­ist­ic­al tests, the so-called “Mann-Whit­ney U tests”, which can be used in clin­ic­al research to com­pare two inde­pend­ent groups of patients. “GRIM‑U was recently used to inform Retrac­tion Watch report­ing on the work of one par­tic­u­lar sur­geon.” The tool reportedly detec­ted stat­ist­ic­al anom­alies in a short art­icle sup­port­ing the effic­acy of a sur­gic­al device developed by the prac­ti­tion­er and his col­leagues³.

But the research­er makes no secret of the fact that devel­op­ing effect­ive tools is a com­plex under­tak­ing, and faces major chal­lenges, not­ably “leg­al restric­tions on the scrap­ing of inform­a­tion – restric­tions designed to pro­tect intel­lec­tu­al prop­erty rights – and the chal­lenge of tun­ing detec­tion code to avoid false pos­it­ives”. The team can also count on a highly col­lab­or­at­ive com­munity and will not hes­it­ate to use tools developed by oth­er con­trib­ut­ors if neces­sary. It remains to be seen, over the next twelve months, wheth­er this tar­geted strategy will pro­duce res­ults that live up to its ambitions.

Anne Orliac