Impact assessment
In the context of the missions set out by the 2006 decree amended in 2014, the French National Research Agency (ANR) undertakes to analyse the evolution of research offering and assess the impact of the funding it allocates. Coordinating with research stakeholders in France and abroad, the agency takes part in ongoing thinking about impact assessment and conducts studies on various project portfolios at multiple analysis levels.
Impact assessment of public policy, and research policy in particular, is a need more and more expressed by political decision-makers in order to account to the public for the results of the spending invested. Beyond the standard peer-review system dedicated to the scientific production evaluation, this involves describing and measuring the benefits of research for society in the broad sense: the economy, the health, the environment, the culture etc.
Challenges for the ANR
As a national multidisciplinary funding agency, the ANR has financed over 18,000 projects since its foundation. The underlying issues of impact are consequently very diverse. What are the effects of the agency's actions in the scientific and socio-economic world? How does research offering evolve over time and in response to calls for proposals?
The ANR thus considers different approaches to, describe, and measure the impacts of its actions in all their diversity. The methodologies used are both qualitative and quantitative and, drawing for example on scientific expertise, surveys, interviews but also data analysis, bibliometrics, text mining, case studies, or econometric studies. Impact assessment also depends on rigorous data collection. Access to reliable and interoperable data, which is made available in the context of Open Science, are important factors in the successful completion of impact assessment.
The ways this activity is carried out, published and communicated need to be defined in consultation with research stakeholders. This mission is part of a process of dialogue, co-construction and shared learning that enables the agency to better anticipate the effects of its actions and informs its strategy with a view to ultimately incorporating the analysis results into its programming.
The impact pathway of research
Impact assessment draws on multiple areas of scientific, methodological and sector-specific expertise.
In terms of methodology, different stakeholders – research organisations, funding agencies, ministries etc. – apply a wide variety of approaches. Shared thinking is gradually taking shape at national and international level on pooling best practices and defining common systems of reference and standardised methodological frameworks that can be adapted to the great variety of analysis subjects, stakeholders and contexts. The impact pathway of research thus constitutes a useful tool for grasping the concept and sharing a common vocabulary.
At ANR, impact assessment is implemented at each step of the impact pathway: from the beginning of the project (1), to the follow-up (2), the end (3) and the ex post steps of the project (4, 5).
The analyses conducted by the ANR cover project portfolios, i.e. groups of projects constituted according to the analysis required. This may be a set of projects funded under a given research theme, through a national programme, via a specific funding instrument or according to a call for proposals or a stakeholder (e.g. research organisation or region).
Given the multidimensional nature of this activity, collaboration between ANR and research stakeholders is essential throughout the process: defining the scope, collecting, processing, using and interpreting the data, producing analyses, communication. Prioritising openness, transparency and knowledge sharing, the ANR communicates about the impact data in a variety of formats: datasets, mappings, reports, thematic records, workshops, symposia etc.
