How We Grade

A Look Inside the GPP

One of our foremost efforts this year was to add more transparency to the grading process. In Governing, readers will find not only grades for each of the four areas we’re covering — People, Money, Infrastructure and Information — but also highlights of strengths and weaknesses in criteria used to assess those management areas. In addition, there is a Web site — http://results.gpponline.org — that provides detailed summaries to supplement the text and give readers a more thorough understanding of the outstanding characteristics of each of the 50 states.

Visitors to the site can compare grades and scores in several different ways and find state plans, reports and other documents in each of the areas covered, including strategic, workforce, capital and other plans. As items are added to the Web site over the coming months, users will find a growing body of material about innovations in the states.

Of course, all the transparency in the world is worthless if people don’t know what they’re seeing. So, here are a few bits of clarification about the meanings of the various evaluations on each page.

— Each criterion was made up of a number of sub-criteria (see page 34). A holistic approach was taken to get the best sense of a state’s capacity in each area. For example, in infrastructure, we looked at both the use of condition assessments to see where a state could best spend maintenance dollars and the actual decision about whether or not to spend that money. If a state had decided not to fund maintenance at all, the fact that they knew what they should be funding figured less importantly.

— The criteria themselves are listed on each state page and then placed in one of three groups: strength, weakness and mid-level. For a criterion to be considered an area of strength, the state had to be doing very well relative to standard practices and to have some element that made it stand out — even from others that performed well. Conversely, a weak area was one in which it was clear that the state needed to improve in order to be able to manage adequately in that area. And mid-level? Many of the states that fell into this middle ground do quite a good job and have no notable problems — they just lack that little extra innovation or distinction that earned the higher level. These criteria descriptions are not grades — nor should they be translated into grades. They’re simply provided to give readers an easy means of seeing the areas in which each state excels or lags compared with the criteria.

The grades themselves were based on the criteria shown on each page. But there was no rigid weighting scheme used. The logic here is inexorable. In Infrastructure, for instance, there is no point worrying whether Hawaii has good intergovernmental relationships with bordering states. But that is a matter of much more concern in a state such as Tennessee. Similarly, in the People section, the ability to recruit is potentially more important for fast-growing states than those with stagnant populations. The grades presented are not comparable to those in any of the prior GPP surveys. This is because the criteria differ from those used in the past. In fact, the major categories of study have changed.

To establish a coherent method for evaluating the states, we started by developing a comprehensive list of criteria. This was accomplished with participation from the full GPP team: academics from five different institutions of higher learning and journalists who had been involved in the GPP since its inception in 1997. We used cutting-edge research and evidence from best practices to frame the criteria for superior management of state governments. These criteria were then discussed with practitioners in order to bring real-world insights into the mix. The next stage was to determine what information would have to be gathered so we would have a fair way to evaluate the states in each of the 20 criteria used.

Finally, there was the all-important information gathering. Here, we tapped three sources: First, a survey, filled out by states and carefully analyzed by the GPP’s academic teams, provided a great deal of basic data upon which states could be compared based on the criteria. In addition, journalists from Governing conducted an estimated 1,000 interviews to add information to the pool and, importantly, to provide context in which all the information could best be understood. Among those interviewed were legislators, legislative staffers, legislative fiscal analysts, controllers, treasurers, budget officers, human resource officials, transportation officials, chief information officers, officials in charge of non-transportation infrastructure, auditors, representatives of civic organizations, academics and representatives of agencies, notably the states’ environmental and transportation departments. In prior years, there was little or no agency-based research. Although resources precluded delving into multiple agencies this year, the addition of some agency analysis turned out to be enormously useful, particularly in states that are very decentralized.

Click on http://results.gpponline.org to compare grades and scores, read state plans and follow updates on state programs

Many documents also were analyzed, including state Web sites, budgets, capital plans and workforce plans. These were utilized in two ways: as direct sources of additional information and, in some cases, as elements of management that deserved to be evaluated themselves. This was the case, for example, with workforce plans.

Both academics and journalists focused on the ability of states to produce actual results, not just create processes. In the Information category, for example, it was of far less consequence how sophisticated a state’s technology is than how well that technology serves managers’ needs.

There were a number of instances in which information garnered from journalistic interviews differed from that obtained by academics’ review of documents or data. In all these cases, long conversations were held in order to determine precisely why the differences existed. Further research was done, where necessary, to resolve any issues.

Following are the criteria used, followed by the sub-criteria. Some sub-criteria were far more significant than others. In People, for example, under “Managing Employee Performance,” we were more concerned with the way the state recognizes and rewards high performers than the way the state provides and utilizes employee feedback. Additionally, there were some criteria for which we discovered very little variation among the states. Assessing the impact of federal decisions in the Money section was one. In those instances, the sub-criteria received relatively minimal attention. Finally, with dozens of sub-criteria, there were instances in which one or two bits of information weren’t available in a particular state. In those cases, the remaining sub-criteria were given more emphasis.The grades in this GPP are not comparable to prior years. This is because the criteria differ from those used in the past. Click on http://results.gpponline.org to compare grades and scores, read state plans and follow updates on state programs.

CRITERIA

MONEY

Long-Term Outlook: The state uses a long-term perspective to make budget decisions.

• has revenue and expenditure forecasting processes that are thorough, accurate and transparent and include a multiyear perspective;

• uses information about the future fiscal impact of its financial decisions in developing the annual budget;

• uses information that assesses the future fiscal impact of external (federal and regional) financial decisions in developing the annual budget;

• manages long-term investments, including pensions, to promote solvency and avoid financial instability;

• maintains a modest, reasonable level of borrowing and does not rely on debt to finance current expenditures.

Budget Process: The state’s budget process is transparent and easy to follow.

• clearly defines and follows a predetermined cycle, format and deadlines for its budget;

• provides clear and timely budget guidelines and instructions;

• communicates effectively its fiscal priorities throughout all levels of the state;

• provides citizens opportunities for public input about the budget.

Structural Balance: The state’s financial management activities support a structural balance between ongoing revenues and expenditures.

• accommodates fluctuating economic climates within its revenue structure;

• does not depend on “one-time” or “windfall” revenues for ongoing expenditures;

• manages its cash to maintain support for ongoing expenditures and to minimize short-term borrowing;

• manages risk to limit government liability and to maintain overall fiscal health;

• uses counter-cyclical or contingency planning devices to address economic downturns.

Contracting/Purchasing: The state effectively manages procurement activities.

• state contracting and purchasing protocols allow agencies to operate programs effectively.

Financial Controls/Reporting: The state systematically assesses the effectiveness of its financial operations and management practices.

• prepares an annual financial audit in accordance with Generally Accepted Accounting Principles and routinely receives a clean audit opinion;

• regularly assesses and effectively links financial costs with operational performance;

• engages in sound internal control practices.

PEOPLE

Strategic Workforce Planning: The state regularly conducts and updates a thorough analysis of its human resource needs.

• has a multi-year strategic workforce plan that identifies its current and future human resource needs;

• links its human resource plan to the state budget and supports the state’s strategic direction;

• has comprehensive and readily available data about its current workforce and its future workforce needs that can be used to make decisions involving human resource management.

Hiring: The state acquires the employees it needs.

• hires people in a timely manner;

• knows if new employees perform successfully;

• is able to recruit and fill positions that are critical to its core services.

Retaining Employees: The state retains a skilled workforce.

• does not lose a disproportionate share of its managers and employees by their voluntary departure each year;

• creates a work environment that supports employees’ life needs;

• maintains productive relations with employees and their representatives;

• terminates employees for cause in a timely manner and those terminated for cause receive timely and fair treatment.

Training and Development: The state develops its workforce.

• devotes sufficient resources to its employees’ development;

• provides opportunities for career advancement;

• purposefully develops its leaders’ competencies.

Managing Employee Performance: The state manages its workforce performance programs effectively.

• links state, department, and employee performance goals;

• recognizes and rewards high performers (individuals or groups) in achieving desired results;

• regularly provides, receives and utilizes employee feedback;

• provides a means for employees to be actively involved in establishing its goals and objectives and work plans.

INFRASTRUCTURE

Capital Planning: The state conducts a thorough analysis of its infrastructure needs and has a transparent process for selecting infrastructure projects.

• regularly conducts an infrastructure condition assessment in accordance with accepted engineering standards;

• informs the capital planning process through a systematic assessment of future infrastructure needs;

• has a formal, multi-year capital plan that both prioritizes capital activities and links directly to the capital budget;

• relies on capital planning priorities, condition assessments of infrastructure, and public input in selecting projects for inclusion in the capital budget;

• includes estimates of the operating and maintenance costs of the capital projects in the capital plan and the capital budget and formally links those estimates to the state operating budget prior to legislative adoption.

Project Monitoring: The state has an effective process for monitoring infrastructure projects throughout their design and construction.

• adequately monitors, evaluates and detects project cost overruns, delays and safety compliance;

• effectively intervenes to take corrective action, as necessary, in managing the construction of capital projects;

• effectively manages the private companies that build its infrastructure.

Maintenance: The state maintains its infrastructure according to generally recognized engineering practices.

• adopts a life-cycle approach to asset management;

• employs current condition assessments in setting priorities for infrastructure maintenance and renewal;

• funds maintenance at a level that minimizes a facility’s life-cycle costs and that ensures defined levels of service and safety standards are met.

Internal Coordination: The state comprehensively manages its infrastructure.

• when responsibility overlaps, inter- and intra-agency councils and offices effectively coordinate infrastructure issues;

• capital plans recognize the interrelated nature of the infrastructure system.

Intergovernmental Coordination: The state creates effective intergovernmental and interstate infrastructure management networks.

• participates in intergovernmental and interstate management of infrastructure in an active and meaningful way;

• effectively incorporates intergovernmental regulations into capital planning and project selection.

INFORMATION

Strategic Direction: The state actively focuses on the strategic direction of its policy and on collecting information to support that policy direction.

• legislature provides strategic direction to executive branch when creating programs;

• governor sets an overall strategic direction for the state;

• agencies focus on the long-term goals of programs or policies, down to the lowest level of the agency at which policy discretion is exercised;

• there is meaningful, multi-year information technology planning to inform legislative, executive and agency decisions.

Budgeting for Performance: State officials have appropriate data on the relationship between costs and performance, and they use these data when making resource allocation decisions.

• state routinely produces valid cost and performance information (including information on past performance);

• governor, budget office and state agencies have cost and performance information available during deliberations on the budget;

• legislature has cost and performance information available during deliberations on the budget;

• agencies, budget office, governor and legislature all routinely use cost and performance information when developing or reviewing the budget and as a basis for program design and redesign.

Managing for Performance: Agency managers have the appropriate information required to make program management decisions.

• information technology systems provide information that adequately supports managers’ needs;

• governor and agency managers draw clear links between managerial action and program results, and they communicate this information to appropriate agency personnel;

• agencies regularly monitor performance, including the performance of key program partners, and use this information to manage programs, improve performance, and inform elected officials about deviations from agreed-upon levels of performance;

• cost and performance information frames how top officials grant flexibility in allocating resources and in managing resources;

• cost and performance information influence decisions to contract out for agency activities;

• state monitors contracts for both cost and performance;

• state assesses the expected implication of physical assets and information technology systems in making investment decisions.

Program Evaluation: The governor and agency managers have appropriate data that enables them to assess the actual performance of policies and programs.

• selected programs are subject to performance audits or program evaluations on an ongoing basis;

• audits and program evaluations are relevant, credible, well-documented, and publicly available;

• audits and program evaluations include comparisons over time, against targeted levels and against other similar governments, agencies or policies.

Electronic Government: The public has appropriate access to information about the state, as well as the performance of state programs and state services, and is able to provide input to state policymakers.

• public can access key services without undue burden;

• public can routinely access credible information about the performance of key state programs;

• public can easily communicate with state officials about the performance of public programs;

• state routinely collects feedback from the public on the performance and operation of its programs and services.

Copyright © 2005, Congressional Quarterly, Inc. Reproduction without the written permission of the publisher is prohibited. Governing, City & State and Governing.com are registered trademarks of Congressional Quarterly, Inc.