Is there a difference between Program Evaluation and Performance Measurement?
In response to my recent blog concerning why Public Service managers should care about Program Evaluation, I was asked about the difference between evaluation and performance measurement. As these terms are bandied about rather loosely, there may be a perception that they are really the same thing.
The Simple Answer
The simplest answer is that performance measurement and evaluation are complementary activities. Evaluation gives meaning to performance measurement and performance measurement gives empirical rigour (evidence) to evaluation.
For example, the performance of a heating system might be measured by the output in BTU’s and the ambient temperature in degrees—this is performance measurement.
On the other hand, an evaluation of the heating system would likely use this performance information, combined with additional information to determine if the ambient temperature was appropriate for whatever purpose the space was being used. It would also determine the extent to which the ambient temperature could be attributed to the performance of the heating system as opposed to other factors—such as the type of insulation installed—and arrive at some judgement as to whether it was a good, adequate or bad heating system in this context.
Of course, there would always be room for dispute over whether the insulation was part of the heating system or an external factor, the answer to which would depend on the purpose of the evaluation.
Performance measurement is the activity of judging, on the basis of empirical evidence, the extent to which a program or activity has met its output targets and/or achieved its concrete objectives.
It does not question the objectives themselves and, therefore, stops short of any final judgment as to whether the program or activity was good or bad – only if it was successful—or not—within the narrow confines of its mandate. Clearly, there is room for a bad program to be successful and a good program to be unsuccessful.
Evaluation is a more holistic activity, taking everything into account (including performance) to come to a good vs. bad judgment.
Viewed in this way, there is room for a significant amount of philosophical debate on the matter, as this type of judgement depends upon the various perspectives and values of stakeholders. Therefore, quality evaluation is highly dependent on the professional judgement of the evaluator in the context of a governance (decision-making) framework that is seen as legitimate.
Evaluation in the Public Service
In the Canadian Public Service ‘Evaluation’, as the subject of the Policy on Evaluation, has a specific meaning that goes beyond the mere measurement of performance but limits the evaluation mandate enough to avoid the more difficult philosophical issues that might otherwise arise – though this is never cut and dried in practice. Hence, the standard PS Evaluation comes to a judgment as to whether the program is/was successful and relevant. It also generally includes a judgment concerning value-for-money or cost/benefit, which is based on a specific form of performance measurement.
It is also worth noting that there is currently a movement in the Canadian Public Service toward the integration of the performance measurement and evaluation functions. For a reasonably well-balanced discussion of the relationship between performance measurement and evaluation, with ample reference to the Canadian public sector, I would suggest the textbook by James McDavid and Laura Hawthorn (2006), Evaluation & Performance Measurement: An Introduction to Practice. In it, the authors present an integrated view of performance measurement and evaluation in the wider context of the Performance Management Cycle, which includes:
- Setting objectives;
- Business planning and program design;
- Aligning management systems at the program level;
- Performance measurement and reporting, which includes program evaluation;
- Realizing consequences (e.g. executive compensation and funding decisions); and,
- Revisiting objectives.
Complement or Conflict?
According to the Handbook of Practical Program Evaluation (Wholey, et. al.):
Considerable differences in opinion exist among evaluators as to how performance monitoring and in-depth evaluation studies are related, in particular the extent to which they are at odds with or complement each other.
Wholey et. al. is not too explicit concerning the way in which they are at odds, only to suggest that performance monitoring tends to be more of a management tool, whereas in-depth evaluations provide more information for policy and program decisions. This introduces the idea of a role distinction between managers and evaluators that practice performance measurement and evaluation respectively.
In my view, these roles are generally complementary. If they are at odds with one another, it is either a situational conflict—where the program manager is being ‘judged’ by the evaluator—or a more general result of professional specialization and differentiation, which results in a certain amount of ‘positioning’ behaviour – but this is the subject of another blog post (What is Program Evaluation and Does it Really Matter?).
What do you think? Does this capture the distinction between program evaluation and performance measurement – or is there a lot more to it?
Download our new eBook: