|
|
|
|
How We Grade
Welcome to an inside look at the way we work.
If you were to step inside 1025 F Street in Washington, D.C., and ride the elevator up to the 9th floor, you would find yourself in the home of the Government Performance Project. Here, in a maze of well-lit offices, the GPP's journalists and researchers analyze information and interview state officials. In the conference rooms, we hold marathon sessions on what grades to give each state in each category.
What are these sessions like? This past January, several of us sat around a conference table to talk about the strategic-planning process in Arizona. The journalist, who reported on the state, didn't see evidence of statewide planning. The academic, who had spent time reviewing agency plans, thought the state deserved credit for its coordination of strategic planning among the agencies. The journalist countered that, in the absence of a written statewide plan, there was little indication that actual budgetary actions were influenced by these efforts. After a spirited debate, we reached a consensus: The agency plans would have had to be exceptional to overcome the lack of a state plan, and in Arizona, that simply wasn't the case. That point — along with dozens of other factors — made its way into the final grade of B- for information.
These in-depth conversations are among the last stages of a year-long process that forms the basis for the GPP's grades in four management areas — Information, People, Money and Infrastructure. A full description of the criteria used to assess those management areas can be found online at pewcenteronthestates.org/gpp.
The Criteria We Use
Information
• The state actively focuses on making future policy and collecting information to support that policy direction.
• Elected officials, the state budget office and agency personnel have appropriate data on the relationship between costs and performance and use these data when making resource-allocation decisions.
• Agency managers have the appropriate information required to make program management decisions.
• The governor and agency managers have appropriate data that enable them to assess the actual performance of policies and programs.
• The public has appropriate access to information about the state, the performance of state programs and state services and is able to provide input to state policy makers.
People
• The state regularly conducts and updates a thorough analysis of its human-capital needs.
• The state acquires the employees it needs.
• The state retains a skilled workforce.
• The state develops its workforce.
• The state manages its workforce-performance programs effectively.
Money
• The state uses a long-term perspective to make budget decisions.
• The state's budget process is transparent, easy to follow and inclusive.
• The state's financial management activities support structural balance between ongoing revenues and expenditures.
• The state's procurement activities are conducted efficiently and supported with effective internal controls.
• The state systematically assesses the effectiveness of its financial operations and management.
Infrastructure
• The state regularly conducts a thorough analysis of its infrastructure needs and has a transparent process for selecting infrastructure projects.
• The state has an effective process for monitoring infrastructure projects throughout their design and construction.
• The state maintains its infrastructure according to generally recognized engineering practices.
• The state comprehensively manages its infrastructure.
• The state creates effective intergovernmental and interstate infrastructure coordination networks.
A state's strong points and weak points in each criterion correlate closely to its final grades. Closely is the operative word. The GPP's methodology favors common sense over a formula. New Jersey, for example, does an acceptable job in a couple of the infrastructure-related criteria and a very good job in two more. Yet its grade was a C+. Why? With deferred maintenance of $13 billion on transportation and bridges falling into ever worse condition, the fine job the state does in planning and coordination recedes in importance. "If you let your assets decay, that trumps other factors in considering the overall management of infrastructure," says Michael Pagano, a professor at the University of Illinois at Chicago, who led one of our academic teams.
It turns out that a weak economy doesn't necessarily lead to bad overall grades. Michigan's finances are deeply troubled, but its management skills have weathered the storm well.
As with prior GPPs, the information we utilize comes from a number of sources. First up, a survey asking for basic data. The survey is filled out by the states and carefully analyzed by GPP's academic teams. All but a handful of states completed this online instrument. For those that didn't, the GPP team set about uncovering the same body of information through public documents and interviews.
At the same time, our teams of academics scour the country for documents that could contribute to better understanding of the states, including budgets, capital plans, workforce plans, auditor's reports and state Web sites. These not only are used as sources of information but, as in the case of workforce plans, are reviewed and evaluated as management tools.
Meanwhile, we conduct hundreds of interviews — upwards of 1,400 this year — to add information to the pool of data and, importantly, to provide context in which all the information can best be understood. We interviewed legislators, their staffers and fiscal analysts; controllers, treasurers, budget officers and auditors; human resource and transportation officials; chief information officers; managers in charge of non-transportation infrastructure and representatives of agencies and departments. We also talked to leaders of civic organizations.
Everybody involved in the GPP looks closely at the ability of states to produce actual results. Even the best strategic plan is irrelevant if nobody in the state follows it.
One important note about the grades that emerge from this process: Although the criteria are essentially the same as they were in the 2005 GPP, the state of the art in these areas has advanced. As a result, a state can conceivably have improved without its grade going up. Take the information category. According to Philip Joyce, a professor at The George Washington University who heads one of our academic teams, here's what a state would have had to accomplish in 1999 — the first GPP — to get an A: Good statewide or agency planning, performance audits with some outcome measures plus the use of performance information by the executive branch, even if there was little or none by the legislature. The state's performance had to be communicated to citizens through written performance reports.
In 2008, an A state has to have excellent statewide and agency planning, be a leader in performance auditing (most states now do performance audits), have outcome data for almost all government functions, show substantial use of performance information by the executive branch and some use by the legislature. The state's performance has to be communicated to citizens electronically, preferably through interactive Web sites.
That's a dramatic difference. While the advances in this field are greater than in the others, the basic principle holds true in grading each state in each category.