Usually, what we really want is an outcome, but all we can actually measure is an output. Consider the challenge of measuring the performance of a K-12 school system. The ultimate outcome that we really want is that the children grow up to be responsible citizens and productive employees. However, measuring this outcome is not easy.
What indicator would we use to measure the productivity of today's employees who were once students in a particular school system? Income? It's an obvious indicator, but it doesn't capture the outcome in which we are really interested.
Moreover, what indicator would we choose to measure the more intangible outcome of the level of civic engagement of today's citizens who were once students in that school system? How frequently they voted? How many city council meetings they attended? The percentage of their income that they gave to charity? How many hours they volunteered each year?
Furthermore, to what would we attribute the resulting data about either employee productivity or citizen responsibility? The schools that these adults attended years ago? Their parents? Their community, including religious institutions, churches, neighborhood associations, or local branches of national organizations? Who deserves the credit -- or blame -- for any adult's economic and civic contribution to society?
Finally, if everyone could agree to some key measures of employee productivity and citizen responsibility, how could we use this data to improve the performance of the schools? The lag of decades creates yet another measurement challenge.
This is why we resort to test scores. These are available without much delay, and these results can be attributed -- at least in part -- to each student's school system, individual school and, indeed class. We can't measure what we want, and so we measure something that (we hope) is close. And, thus, what gets done is something that (again, we hope) is close to what we really want.
Immunizing children for measles is a counterexample. In this case, the output -- which can be easily measured -- is directly connected to the outcome we seek. This is because, as the Centers for Disease Control reports, "Studies indicate that more than 99% of persons who receive two doses of measles vaccine (with the first dose administered no earlier than the first birthday) develop serologic evidence of measles immunity."
The measles vaccine really works. Consequently, measuring how many children a public health organization vaccinates against measles is almost the same as measuring how many children this organization made healthier. Moreover, if every child is immunized, this even benefits the child who is the one-in-a-hundred for whom the vaccine doesn't work. This child now has a significantly less chance of coming in contact with another child who does have the measles.
Consequently, if a public health agency produces a high output number (of children vaccinated), it will automatically produce a high outcome number (of children immunized). In this very unusual circumstance, what you measure is both what you get and what you want.
Most of the time, however, the connection between the output and the outcome is hardly so direct. Yet outputs -- whether they be immunization rates or test scores -- are much easier to measure than the outcomes we truly desire. All too often, we can't really measure what we really want.
In such situations, we need to be clear about the cause-and-effect theory between the output that we can measure and the outcome that we desire. For some aspects of K-12 education, we do have such a theory. For example, we don't think that children can grow up to be either productive employees or responsible citizens if they cannot add, subtract, multiply and divide. Moreover, we can test for such basic levels of knowledge. But how much knowledge of chemistry, or literature, or history does a future adult need? And even if we could agree on an answer to this question, how could we test for it?
This is a fundamental challenge in the measurement of performance in government. Too often, we can't measure what we really seek. When faced with this challenge, we need to be creative -- we need to develop measures that, when implemented, give us something close to what we truly want.
