Showing posts with label Impact Evaluation. Show all posts
Showing posts with label Impact Evaluation. Show all posts

Sunday, February 7, 2016

Counterfactual reasoning: A logical fallacy?

You must have heard the counterfactual argument a thousand times: if the program had not occurred, the effect would not have occurred as well.  Let us dissect the logical form of the argument.

In symbolic logic, the counterfactual argument has this structure:
If Program X is implemented, then Y happens (X ⊃ Y)
Program X is not implemented (~X)
Therefore, Y does not happen. (∴ ~Y)

This reasoning is fallacious.  It has even a name which is denying the antecedent.   Michael Scriven calls this overdetermination in his debate with Tom Cook.  This is a logical fallacy because even if X is false (~X), Y can be true (~~Y).  As can be seen in the truth table below, a false antecedent (X) always yields a true logical relationship (X ⊃ Y) regardless of the value of the consequent (Y).

Truth Table

 X Y X ⊃ Y 
TTT
TFF
FTT
FFT

Why is this so? Because in real world, it is not only Program X that produces Y.  Other programs or causes can produce similar result. Suppose a donor claimed that because of the financial assistance to farmers, the household income of program beneficiaries increased.  This reasoning failed to consider that the household income will increase even without the financial assistance because of other contributing factors such as improving economy, better weather conditions, other donors, etc.

Saturday, January 30, 2016

Why bother searching for unintended effects in social programs?

I had this funny encounter with a researcher from a prestigious institution that advocates the use of experimental approach in evaluating social programs.  Using their argument that social programs are like medicines whose impacts have to be evaluated using the best possible research design, I asked for the place of unintended effects within the framework of experimentation.   Yup, it is not only drugs that have intended and unintended effects but also social programs.   For example, research studies on pay for performance policies have consistently found out that these policies result to perverse unexpected behaviors such as narrowly focusing on what are being measured and the gaming of performance metrics.

The answer given was to specify that unintended outcome beforehand and measure it before and after the experimentation.  This answer fits perfectly to the description of Thomas Kuhn on ‘normal science’.  Within prevailing paradigm, the typical response when encountered with something outside of that paradigm is to fit it within its boundaries; to determine the program’s unintended effects is to anticipate it as an intended effect.  You don’t find unintended effects by making conjectures – that is what intended effects for.  You find them by discovery and keen perception.  

Benefits of learning unintended effects
It is important that every monitoring and evaluation plan should include strategy for capturing the program’s unintended effects, positive or negative. In medicine, many of the drugs that we use were discovered because of their unintended effects.  For example, do you know that the original objective of Viagra was to treat heart problem?  And skin whitening is actually a side-effect of glutathione.  How about the interesting story of Thalidomine which was intended to be an antibiotic? This drug was shelved due to its harmful side-effects to pregnant women only to be revived few months later because of its anti-cancer properties.

In program evaluation, it might be possible that even though the program was successful in achieving its goals and targets, it might have triggered undesirable behaviors that off-set what otherwise would have been exemplary program performance.  Conversely, there might be unintended effects that are desirable and worthwhile to replicate through the introduction of similar social program.  Also, learning more about unintended effects of program allows one to understand the causal mechanisms working within the context.  These causal mechanisms sometimes work for or against the program goals.  I remember an example given by an African classmate of a day care center constructed on top of a burial site regarded as sacred by community members.   Because of cultural belief on the dead (the causal mechanism), nobody wants to go inside the building and the day care didn’t produce the intended result. Rather, its construction ignited animosity between the local community and central government.  

In search for a method
How do you evaluate unintended effects? Leading evaluation theorist, Michael Scriven, proposes the use of Goal-Free Evaluation.  As to what this is, that is an interesting material for another blog.

Friday, January 22, 2016

Counterfactual, clone, match and other similar terms

A couple of weeks ago, I was invited by the ASEAN Training Center on Preventive Drug Education to train them on Impact Evaluation as I was informed that the trend on this area is the use of randomized control trial (RCT) as the gold standard for evaluation design.  I went beyond that as I found myself talking also about various quasi-experimental methods as these suffices in the event that RCT is not viable or feasible design.

I started my talk with the concept of counterfactual as this is an indispensable element of impact estimation.  Because the audience was relatively new to monitoring and evaluation, I use layman's terms so that the notion of counterfactual is understood.  Among the words close to the sense of counterfactual are the following:

Control Group
Non-beneficiary
What if situation
Status Quo
Clone
Twin
Match
Equivalent
Nearest Neighbor
Comparison Group
Similar
Identical
Look-alike

Can you think of other words for counterfactual?

Monday, December 8, 2014

Interrupted Time Series

When I first heard about this design, I had an impression that it is a difficult impact evaluation design.  At first, it was really confusing but I persisted and looked for simpler reading material that explains the concept.  From that time on, I got intrigued about this approach and I even went to the extent of buying two books from Amazon on time series.

So what is interrupted time series?

Interrupted time series (ITS) is a quasi-experimental design that uses historical data to show the impact and to infer the causal connection between the outcome of interest and the intervention.  In this design, there is no random allocation of treatment to program participants.  The counterfactual or the situation if the intervention was not introduced is the forecasted value of variable being measured. The reasoning behind this design is that if it can be shown that an "interruption" or break in the trend at the time of introduction of the intervention, then such break provides support to causal argument in favor of the intervention in the absence of other competing explanations.    


I tried playing around the employment rate from 2002 to 2014 to see if I could apply ITS in this data set.  Lo and behold! There is an interruption in the trend in 2005!  The blue line represents the intervention.  What happened in 2005 that caused abrupt change in the employment trend?  That is something subject for a proper research.  

Source of Data: NSCB