THE GOVERNMENT PERFORMANCE PROJECT

How We Grade Them

 
 
 
his is the fourth installment of the Government Performance Project — a joint venture between Governing and the Maxwell School of Citizenship and Public Affairs at Syracuse University, and funded by the Pew Charitable Trusts. It scrutinizes 40 of the nation’s largest counties, and builds upon two similar efforts dedicated to states, and one to cities. In many ways, the evaluations that follow are the most complex of all.

When the Government Performance Project began evaluating state government, some readers objected that comparing states is like comparing apples and oranges. Our response was: “Yes, but what’s wrong with comparing two different kinds of fruit? There are far more similarities than differences.” No such riposte is available when it comes to counties. There are times when comparing them is like comparing apples and koala bears. More than any other institution of government in America, counties are asked to do different things, and given different powers. Milwaukee County, for instance, has a huge range of responsibilities and a lot of freedom in how it deals with them. Dallas County, by contrast, is a creature of Texas state government, with a narrow list of functions and not much independence.

But this problem aside, we are convinced that there is enormous value in evaluating counties in the five areas the GPP has covered for the past four years — financial management, capital management, human resources, managing for results and information technology. Even if the counties differ greatly, virtually all of them must handle all five of these tasks, and their skill at doing so is a matter of crucial consequence. Counties nationwide spend nearly $200 billion a year.

This year’s GPP covers 40 of the largest counties — not literally the 40 most populous ones. In fact, deciding which counties to evaluate was the first problem in this effort. Using a strict list of the top 40 by population, we would have had to include counties such as Middlesex, Massachusetts, which contains nearly 2 million people but is essentially just a geographic subdivision, not a government. Some other standard had to be used.

Another possibility was to look at the 40 largest counties measured strictly by revenue. The problem there is that California would have overwhelmed the list. Ultimately, a reasonable compromise emerged: Split the country up into four regions and take the largest counties from each region, measured by revenue. The final division included 12 from the West; 10 from the East; 10 from the South and eight from the Midwest.

Although the presentation format varies somewhat from prior years, one element has stayed the same: Grades are assigned to each of the five categories covered. And despite every effort to focus readers on the positives and negatives that contribute to the grades, it’s a simple fact of life that readers tend to focus in on the grades themselves. The drawback to this is that the grades are, at best, a rather blunt instrument to describe a complicated body of information. What’s more, they sometimes force elected officials into a defensive posture. The positive is that the grades draw attention to areas of government service that are too frequently overlooked. Based on our experience with cities and states, that focus has helped governmental entities to benchmark on one another and to make improvements.

 
esigning a new survey instrument was the first step in the process of putting the whole package together. It was similar to the one developed for states and cities, but modified somewhat to fit the contours of county government. The survey is hinged, as much as possible, to a series of criteria in each of the categories that has been established over years of discussion with experts of all stripes. The task of developing the survey instrument fell largely to academics at Maxwell and partners they have brought in from the University of Connecticut, the University of Nebraska in Omaha, and Lynchburg University in Virginia. A separate survey was sent out to all counties for each area covered.

As the surveys poured in — and subsequently, trickled in — graduate researchers at the Maxwell School, under the guidance of faculty members from the various institutions, digested them with a traditional academic approach. They coded the responses in a numeric fashion that allowed them to be analyzed and reviewed dispassionately. The researchers also reviewed stacks of supplementary documents that helped clarify the surveys and provided a safeguard against respondents overstating or understating the quality of their management.

The journalists took a very different approach. They, too, carefully read the surveys, as well as the document-based information distilled by Maxwell. But their evaluations were based on hundreds of interviews with officials in the counties. Some of these sources were the same men and women who had filled out the surveys. Some were their bosses. Efforts were made throughout to utilize interviews in one area of evaluation to help obtain more information about another. For example, the final word on the effectiveness of human resources technology doesn’t usually come from the IT people in a county, but from the personnel department. Governing reporters also utilized a variety of sources outside the governments, including research groups, local journalists, academics and financial rating agencies.

In the vast majority of cases, the counties cooperated fully with this effort, thanks, in part, to the support of the National Association of Counties. All of the 40 responded to our questions in some way, although a handful did so only through interviews and documents; they were unwilling or unable to fill out survey instruments.

As one might expect, there are benefits and flaws to both the academic and journalistic approaches. A journalist is somewhat more likely to be misled by a persuasive informant than is an academic who looks only at documents. On the other hand, without context obtained by live interviews, the facts and figures can mislead, and fail to supply the context that is critical for informed commentary.

After the research was done, the academics and the journalists met to discuss the grades. They agreed most of the time. Where there was disagreement, efforts were made to reach an acceptable consensus, and in some instances, further investigation was done. At the end of the day, in a small number of cases, the academic and journalistic approaches could not be made to mesh. Where that happened, the grades published in the pages that follow are the responsibility of Governing editors and staffers, as informed by the academic analysis.

Also critical to understanding and using the information in the report: