Quantitative Public Policy Evaluation using ex-ante and ex-post techniques – EVALPOLPUB

Research in economics is divided somewhat artificially into two different strands of different but complementary logics.

On the one hand, ex-post evaluation methods generally use data collected from natural experiments. The typical approach is to use a difference-in-difference estimate of the causal impact before and after the exogenous event and in comparison with a control group (e.g. a region/state similar to Florida or Northern France labor markets). The approach, extensively used by prominent researchers at MIT (e.g. Esther Duflo for development policies or Joshua Angrist for labor and education policies) consists of two steps : first, find a natural experiment such as the two previous examples, or design an experiment with randomized groups, some receiving a “treatment” and others being a “placebo group” ; second, estimate a parameter of interest, e.g. the impact of the “treatment” on the average of a variable X of the treated group. Another example of this logic is the very rich set of evaluations of the Self-Sufficiency Project in New Brunswick and in British Colombia to estimate the impact of employment subsidies to poor workers.

These approaches are parsimonious in terms of the theory used: economic theory has traditionally only been invoked to describe the context and with simple demand and supply concepts, generally without complex intertemporal optimization behavior. In what follows, they are termed as “reduced-form approaches” even though this is obviously arbitrary or simplistic.

A second and different logic is to develop so-called structural models, that is models where agents make optimal choices under some constraints. These agents are consumers, workers, firms, families, or even government bodies. The equilibrium of the model is then computed and calibrated, that is, some key parameters are estimated or guessed from statistics or from estimates from other works. In the absence of a sound theoretical understanding, reduced-form approaches are typically unable to estimate general equilibrium effects. For example, an ex-post evaluation may isolate positive effects of a job training program on employment. However, the extra jobs may be obtained at the expense of the surrounding individuals. Generalizing this program on a national scale may not deliver similar results, if for example, the total quantity of jobs is determined at a national level. Moreover, reduced-form approaches are unable to deliver “counterfactual” experiments, ie what the outcomes could have been, had the reform been slightly different.

However, structural models alone are not always able to deliver the full answer raised by the need of policy evaluation, in particular due to a number of arbitrary choices. These models are often based on non-testable assumptions, and sometimes untransparent calibration exercises. The ex-ante estimates delivered by structural models are rarely compared to more rigorous ex-post reduced-form approaches, although this exercise could validate the model.

The overall logic of the project is to reconcile those two approaches.