Scientific work challenged by Post-Publication Peer Review. Research Evaluation and Scientific Integrity – SKEPTISCIENCE

The SKEPTISCIENCE project is based on a mixed methodological approach, combining quantitative and qualitative tools from social science, computer science and open science studies. This complementary approach has made it possible to capture both the overall dynamics of post-publication review (PPR) and the experiences of those involved.

1. Extraction and analysis of data from post-publication peer review platforms. One focus of the project was the collection and analysis of big data from platforms such as PubPeer and PubMed Commons. This data (comments, publication metadata, user profiles) was extracted using specific scripts and then structured in interoperable databases. By linking them to other sources (OpenAlex, Retraction Watch), we were able to map the fields affected by PPPR, measure its evolution over time, identify the types of active users, and identify the most commented or retracted publications. This mapping revealed a strong concentration of activity in the life sciences and biomedicine.

2. In order to understand the scientific community's perception of PRPP, a questionnaire was sent to the staff of research organisations (in particular the CNRS and Inserm). The questionnaire explored knowledge of PRPP platforms, their perceived legitimacy, the conditions of acceptance of these practices and the representations associated with the correction of science. This phase made it possible to quantify the potential adherence to PPPR and to identify the obstacles to its adoption.

3. Qualitative research through interviews. A series of semi-structured interviews were conducted with users, platform founders, whistleblowers, members of integrity committees and science journalists. These interviews shed light on the commitment of PPPR actors, their dilemmas, their communication strategies and their vision of the scientific evaluation system. They also made it possible to identify the emergence of a new role: that of the "science detective", who is concerned with epistemic and moral vigilance.

4. Theoretical foundations and hypotheses. On a conceptual level, the project mobilises the notion of 'organised scepticism' to think about post-publication evaluation as a collective mechanism of scientific regulation. Three hypotheses have structured the research: 1) PPPR helps to distinguish between good and bad practices; 2) it redefines the value of publications as an evolving resource; 3) it develops preferentially in contexts where intermediary institutions are perceived as deficient.

This combination of methods has made it possible to articulate a detailed empirical description and theoretical reflection, and to offer a fresh reading of the contemporary transformations of scientific evaluation.

The SKEPTISCIENCE project has produced a number of original findings that shed light on the changes taking place in scientific evaluation in the digital age. These findings fall into four main categories.

1. Unprecedented mapping of post-publication evaluation practices. By analysing more than ten years of data from the PubPeer platform, SKEPTISCIENCE has produced a detailed mapping of PPPR use. It shows that the life sciences and biomedicine account for the majority of comments, with a significant increase in volume by 2018. At the same time, the social sciences, which were very present at the beginning, have been marginalised. This analysis also shows that 70% of the comments are generated by only 20% of the users, suggesting that the PPPR space is structured around an active core.

2. Perception and legitimacy of the PPPR in the scientific community. The survey of CNRS and Inserm staff showed that the majority of the scientists interviewed did not reject the principle of post-publication review, but that its acceptance was subject to certain criteria. For example, almost one in two researchers said that they would only accept online criticism if it came from identified peers. More generally, post-publication peer review is more readily accepted when it is perceived as an extension of the traditional peer review system, rather than as a competing or alternative mechanism.

3. The emergence of new roles and figures of scientific vigilance. Interviews with users of platforms such as PubPeer have revealed the emergence of new actors: 'science detectives', who work to expose scientific errors, mistakes or fraud. This work, long carried out anonymously, is now tending to become more visible through the media or academic publications. It is accompanied by new standards of intervention: sharing of doubts, collective mobilisation, demand for transparency. The platform thus becomes a space for the elaboration of a collective judgement, both scientific and ethical.

4. Differentiated analysis of retractions and gender effects. Finally, the cross-analysis of the Retraction Watch and OpenAlex databases has made it possible to identify different retraction profiles according to the gender of the first author. Publications edited by men are more likely to be associated with serious misconduct (fraud, manipulation, etc.), while those edited by women are more likely to be associated with unintentional errors. These results underline that the dynamics of retractions are not neutral and that they are part of specific social configurations.

At a time when the rise of digital technologies and collaborative platforms is transforming the norms and practices of the scientific community, the SKEPTISCIENCE project makes an original contribution to the study of transformations in the evaluation of scientific work. By adopting a multi-scale approach, it considers both the general mapping of post-publication peer review (PPPR) and its diffusion at the level of research organisations or research actors.

Whether by collecting and using online data, distributing questionnaires or conducting interviews, SKEPTISCIENCE has shown that PPPR is not only a technological evolution of peer review, but also a social and organisational innovation that contributes to redefining the boundaries of social regulation of the scientific community. The PPPR helps to challenge the traditional criteria for validating scientific publications by introducing a dynamic evaluation process. The PPPR also helps to increase collective vigilance in the face of the many abuses of both publications and scientific publishers. The findings of SKEPTISCIENCE suggest that while the PPPR appears to be a vehicle for transforming the epistemic and ethical standards of research, its social acceptability remains partly dependent on its relationship with the traditional peer review framework.

Far from being an end in itself, the SKEPTISCIENCE project represents an important exploratory step in the sociological analysis of changes in scientific work and its evaluation. Thanks to the support of the ANR, it has enabled new knowledge to be generated, but it has also laid down a number of foundations that should be exploited and explored in greater depth in future projects. Among the avenues that could be explored, the following two are particularly worthy of mention:

1. Continuing the study of post-publication evaluation at the international level. The project has highlighted the central role of certain platforms, but also the diversity of national contexts in terms of the reception and appropriation of PPPR. The next step will be to broaden the comparative scope by collaborating with foreign teams engaged in similar research.

2. Deepen the analysis of the trajectories of publications that have been commented on or retracted. By cross-referencing several databases, SKEPTISCIENCE has paved the way for a longitudinal study of scientific publications that have been criticised or retracted. This work will continue in order to better understand the impact of the PPPR on the visibility, circulation and reputation of the articles concerned.

SKEPTISCIENCE studies the impact of technological innovations on scientific work and its evaluation. This research project investigates the online discussion platforms (PubPeer, PubMed Commons, F1000Research, etc.) created in the 2010s and promoting a new way of assessing scientific professional practices and organizations, namely Post-Publication Peer Review (PPPR). PPPR is developing as part of a broader movement of transformation of Peer Review. Although this mechanism of evaluation and certification in science is still considered central for the quality of scientific literature, multiple studies have pointed to its limitations: it is slow, potentially biased and frequently inconsistent. Part of PPPR public visibility stems from its ability to alert the scientific community, and more broadly the general public, about possible errors but also scientific misconducts. SKEPTISCIENCE is innovative as it addresses from a scientific and multidisciplinary perspective a topic that is strikingly understudied. Despite various calls to turn the recent transformations of peer review into a topic of investigation, there are no major empirical studies on PPPR and the few existing ones are restricted to limited issues or areas of research. This project has three main aims. SKEPTISCIENCE first aim is to build the PPPR global map. What is the contemporary technological and scientific landscape of PPPR? What are its general properties and modalities? On what scale is the PPPR practiced and what is its growing trend? SKEPTISCIENCE second aim is to establish the consequences of PPPR for the scientists and publishers professional practices, trajectories and organizations. Does PPPR produce new lines of demarcation between what counts as good or bad publication, but more extensively what counts as good and bad research practices? SKEPTISCIENCE third main aim is to characterize the variety of users and stakeholders involved in the dynamics of PPPR as a controversial non-institutional innovation. PPPR is often depicted as coinciding with the emergence of a new generation of “watchdogs of science”. Why do researchers invest their time and energy in these online discussion platforms? Can it be said that PPPR coincides with the emergence of new scientific roles? Based on the gathering and processing of both quantitative and qualitative data, SKEPTISCIENCE calls for strong interactions with experts coming from multiple disciplines, including sociology, complex systems and social network analysis, biology, medicine, physics, PPPR entrepreneurs willing to act as consultants, and even more broadly any scientist involved in one of the four research areas selected.