Budgeting for program evaluation

The goal of program-based budgets and allocations is to gain a solid understanding of the true costs, and staff cost is too important to leave to guesswork. When an expense is clearly and exclusively incurred for a specific program area or cost center, we simply assign the expense to that program area or cost center. Examples might include materials purchased specifically for a tutoring program or the cost of an evaluation consultant to document the results of a preschool program. Administration and fundraising may have direct expenses assigned to them as well.

The cost of an annual audit would be assigned to administration. The cost of return envelopes to be included in a fundraising mailing would be assigned directly to fundraising. In this step you apply the allocation methods described above to the various direct costs that are shared between programs, which may include administration and fundraising cost centers. For the earlier office supply example, you would add up how many FTEs work in each program area and calculate a formula as a percent of the total number of staff.

These calculations may be automated through the accounting system or completed manually. The formulas should be revisited if there are major changes in the way expenses are used, such as staff reassignments or growth of a program. At this point you will have a subtotal of the direct costs of each program, administration, and fundraising.

This process is most valuable when a nonprofit can understand both the full cost of delivering programs and the amount and type of income that relates to those programs. Leaders can use this information to analyze the financial model of programs individually and as part of the whole. Examples of income that is assigned directly to a program include contract or fee income for a preschool program or a grant that is received for a tutoring program.

For this step we recommend that contributed income that is unrestricted or general operating support be assigned to the fundraising category for the analysis. The final analysis will clearly show what program areas require these sources of support and enable leaders to make the all-important decision about how to best attract and direct flexible funds. The cost of administration, categorized as indirect costs, adds value to every program at a nonprofit. Programs are more effective, better managed, and more responsive to the community when an organization has good accounting and technology, high quality leadership, planning, and governance.

In order to have a true picture of what our programs really cost, we must allocate these indirect or administrative costs as well. If we ignore this step, we will be underrepresenting the expense involved in supporting each program area.

As explained above, indirect expenses are generally all of our administrative expenses — those expenses that support the overall management of the organization.

Some expenses are assigned to the indirect category specifically, such as the audit. Others are allocated to the indirect category, such as a portion of rent and telephone. For this reason we wait until after all the direct allocations are completed before we turn to allocating the indirect costs. The two most common methods for allocating indirect costs to programs are percentage of total direct costs and percentage of FTE. Similarly, the cost of fundraising is valuable to programs and the final step is to allocate fundraising expenses to each.

The breadth and depth of a program description will vary for each program evaluation. And so, many different activities may be part of developing that description. For instance, multiple sources of information could be pulled together to construct a well-rounded description. The accuracy of an existing program description could be confirmed through discussion with stakeholders.

Descriptions of what's going on could be checked against direct observation of activities in the field. A narrow program description could be fleshed out by addressing contextual factors such as staff turnover, inadequate resources, political pressures, or strong community participation that may affect program performance.

By focusing the evaluation design, we mean doing advance planning about where the evaluation is headed, and what steps it will take to get there. It isn't possible or useful for an evaluation to try to answer all questions for all stakeholders; there must be a focus. A well-focused plan is a safeguard against using time and resources inefficiently.

Depending on what you want to learn, some types of evaluation will be better suited than others. However, once data collection begins, it may be difficult or impossible to change what you are doing, even if it becomes obvious that other methods would work better. A thorough plan anticipates intended uses and creates an evaluation strategy with the greatest chance to be useful, feasible, proper, and accurate. Purpose refers to the general intent of the evaluation.

A clear purpose serves as the basis for the design, methods, and use of the evaluation. Taking time to articulate an overall purpose will stop your organization from making uninformed decisions about how the evaluation should be conducted and used. There are at least four general purposes for which a community group might conduct an evaluation:. Users are the specific individuals who will receive evaluation findings.

They will directly experience the consequences of inevitable trade-offs in the evaluation process. For example, a trade-off might be having a relatively modest evaluation to fit the budget with the outcome that the evaluation results will be less certain than they would be for a full-scale evaluation.

Because they will be affected by these tradeoffs, intended users have a right to participate in choosing a focus for the evaluation. An evaluation designed without adequate user involvement in selecting the focus can become a misguided and irrelevant exercise.

By contrast, when users are encouraged to clarify intended uses, priority questions, and preferred methods, the evaluation is more likely to focus on things that will inform and influence future actions. Uses describe what will be done with what is learned from the evaluation. There is a wide range of potential uses for program evaluation. Generally speaking, the uses fall in the same four categories as the purposes listed above: to gain insight, improve how things get done, determine what the effects of the program are, and affect participants.

The following list gives examples of uses in each category. The evaluation needs to answer specific questions. Drafting questions encourages stakeholders to reveal what they believe the evaluation should answer.

That is, what questions are more important to stakeholders? The process of developing evaluation questions further refines the focus of the evaluation. The methods available for an evaluation are drawn from behavioral science and social research and development.

Three types of methods are commonly recognized. They are experimental, quasi-experimental, and observational or case study designs. Experimental designs use random assignment to compare the effect of an intervention between otherwise equivalent groups for example, comparing a randomly assigned group of students who took part in an after-school reading program with those who didn't.

Quasi-experimental methods make comparisons between groups that aren't equal e. Observational or case study methods use comparisons within a group to describe and explain what happens e. No design is necessarily better than another.

Evaluation methods should be selected because they provide the appropriate information to answer stakeholders' questions, not because they are familiar, easy, or popular. The choice of methods has implications for what will count as evidence, how that evidence will be gathered, and what kind of claims can be made. Because each method option has its own biases and limitations, evaluations that mix methods are generally more robust. Over the course of an evaluation, methods may need to be revised or modified.

Circumstances that make a particular approach useful can change. For example, the intended use of the evaluation could shift from discovering how to improve the program to helping decide about whether the program should continue or not.

Thus, methods may need to be adapted or redesigned to keep the evaluation on track. Agreements summarize the evaluation procedures and clarify everyone's roles and responsibilities. An agreement describes how the evaluation activities will be implemented. Elements of an agreement include statements about the intended purpose, users, uses, and methods, as well as a summary of the deliverables, those responsible, a timeline, and budget.

The formality of the agreement depends upon the relationships that exist between those involved. For example, it may take the form of a legal contract, a detailed protocol, or a simple memorandum of understanding. Regardless of its formality, creating an explicit agreement provides an opportunity to verify the mutual understanding needed for a successful evaluation.

It also provides a basis for modifying procedures if that turns out to be necessary. As you can see, focusing the evaluation design may involve many activities. For instance, both supporters and skeptics of the program could be consulted to ensure that the proposed evaluation questions are politically viable. A menu of potential evaluation uses appropriate for the program's stage of development could be circulated among stakeholders to determine which is most compelling.

Interviews could be held with specific intended users to better understand their information needs and timeline for action. Resource requirements could be reduced when users are willing to employ more timely but less precise evaluation methods. Credible evidence is the raw material of a good evaluation. The information learned should be seen by stakeholders as believable, trustworthy, and relevant to answer their questions.

This requires thinking broadly about what counts as "evidence. For some questions, a stakeholder's standard for credibility could demand having the results of a randomized experiment. For another question, a set of well-done, systematic observations such as interactions between an outreach worker and community residents, will have high credibility. The difference depends on what kind of information the stakeholders want and the situation in which it is gathered. Context matters! In some situations, it may be necessary to consult evaluation specialists.

This may be especially true if concern for data quality is especially high. In other circumstances, local people may offer the deepest insights. Regardless of their expertise, however, those involved in an evaluation should strive to collect information that will convey a credible, well-rounded picture of the program and its efforts.

Having credible evidence strengthens the evaluation results as well as the recommendations that follow from them. Although all types of data have limitations, it is possible to improve an evaluation's overall credibility. One way to do this is by using multiple procedures for gathering, analyzing, and interpreting data. Encouraging participation by stakeholders can also enhance perceived credibility.

When stakeholders help define questions and gather data, they will be more likely to accept the evaluation's conclusions and to act on its recommendations. Indicators translate general concepts about the program and its expected effects into specific, measurable parts. Indicators should address the criteria that will be used to judge the program. That is, they reflect the aspects of the program that are most meaningful to monitor. Several indicators are usually needed to track the implementation and effects of a complex program or intervention.

One way to develop multiple indicators is to create a "balanced scorecard," which contains indicators that are carefully selected to complement one another. According to this strategy, program processes and effects are viewed from multiple perspectives using small groups of related indicators. For instance, a balanced scorecard for a single program might include indicators of how the program is being delivered; what participants think of the program; what effects are observed; what goals were attained; and what changes are occurring in the environment around the program.

Another approach to using multiple indicators is based on a program logic model, such as we discussed earlier in the section. A logic model can be used as a template to define a full spectrum of indicators along the pathway that leads from program activities to expected effects. Indicators can be broad-based and don't need to focus only on a program's long -term goals. They can also address intermediary factors that influence program effectiveness, including such intangible factors as service quality, community capacity, or inter -organizational relations.

Indicators for these and similar concepts can be created by systematically identifying and then tracking markers of what is said or done when the concept is expressed. In the course of an evaluation, indicators may need to be modified or new ones adopted. Also, measuring program performance by tracking indicators is only one part of evaluation, and shouldn't be confused as a basis for decision making in itself.

There are definite perils to using performance indicators as a substitute for completing the evaluation process and reaching fully justified conclusions. For example, an indicator, such as a rising rate of unemployment, may be falsely assumed to reflect a failing program when it may actually be due to changing environmental conditions that are beyond the program's control.

Sources of evidence in an evaluation may be people, documents, or observations. More than one source may be used to gather evidence for each indicator. In fact, selecting multiple sources provides an opportunity to include different perspectives about the program and enhances the evaluation's credibility.

For instance, an inside perspective may be reflected by internal documents and comments from staff or program managers; whereas clients and those who do not support the program may provide different, but equally relevant perspectives. Mixing these and other perspectives provides a more comprehensive view of the program or intervention. The criteria used to select sources should be clearly stated so that users and other stakeholders can interpret the evidence accurately and assess if it may be biased.

In addition, some sources provide information in narrative form for example, a person's experience when taking part in the program and others are numerical for example, how many people were involved in the program. The integration of qualitative and quantitative information can yield evidence that is more complete and more useful, thus meeting the needs and expectations of a wider range of stakeholders.

Quality refers to the appropriateness and integrity of information gathered in an evaluation. High quality data are reliable and informative. It is easier to collect if the indicators have been well defined. Other factors that affect quality may include instrument design, data collection procedures, training of those involved in data collection, source selection, coding, data management, and routine error checking. Obtaining quality data will entail tradeoffs e.

Because all data have limitations, the intent of a practical evaluation is to strive for a level of quality that meets the stakeholders' threshold for credibility. Quantity refers to the amount of evidence gathered in an evaluation.

It is necessary to estimate in advance the amount of information that will be required and to establish criteria to decide when to stop collecting data - to know when enough is enough. Quantity affects the level of confidence or precision users can have - how sure we are that what we've learned is true. It also partly determines whether the evaluation will be able to detect effects.

All evidence collected should have a clear, anticipated use. By logistics , we mean the methods, timing, and physical infrastructure for gathering and handling evidence. People and organizations also have cultural preferences that dictate acceptable ways of asking questions and collecting information, including who would be perceived as an appropriate person to ask the questions.

For example, some participants may be unwilling to discuss their behavior with a stranger, whereas others are more at ease with someone they don't know. Therefore, the techniques for gathering evidence in an evaluation must be in keeping with the cultural norms of the community. Data collection procedures should also ensure that confidentiality is protected. The process of justifying conclusions recognizes that evidence in an evaluation does not necessarily speak for itself.

Evidence must be carefully considered from a number of different stakeholders' perspectives to reach conclusions that are well -substantiated and justified.

Conclusions become justified when they are linked to the evidence gathered and judged against agreed-upon values set by the stakeholders. Stakeholders must agree that conclusions are justified in order to use the evaluation results with confidence. Standards reflect the values held by stakeholders about the program. They provide the basis to make program judgments. The use of explicit standards for judgment is fundamental to sound evaluation.

In practice, when stakeholders articulate and negotiate their values, these become the standards to judge whether a given program's performance will, for instance, be considered "successful," "adequate," or "unsuccessful. Analysis and synthesis are methods to discover and summarize an evaluation's findings.

They are designed to detect patterns in evidence, either by isolating important findings analysis or by combining different sources of information to reach a larger understanding synthesis.

Mixed method evaluations require the separate analysis of each evidence element, as well as a synthesis of all sources to examine patterns that emerge.

Deciphering facts from a given body of evidence involves deciding how to organize, classify, compare, and display information. These decisions are guided by the questions being asked, the types of data available, and especially by input from stakeholders and primary intended users. Interpretation is the effort to figure out what the findings mean. Uncovering facts about a program's performance isn't enough to make conclusions.

The facts must be interpreted to understand their practical significance. In short, interpretations draw on information and perspectives that stakeholders bring to the evaluation. They can be strengthened through active participation or interaction with the data and preliminary explanations of what happened.

Judgments are statements about the merit, worth, or significance of the program. They are formed by comparing the findings and their interpretations against one or more selected standards.

Because multiple standards can be applied to a given program, stakeholders may reach different or even conflicting judgments. Community members, however, may feel that despite improvements, a minimum threshold of access to services has still not been reached. Their judgment, based on standards of social equity, would therefore be negative.

Conflicting claims about a program's quality, value, or importance often indicate that stakeholders are using different standards or values in making judgments. This type of disagreement can be a catalyst to clarify values and to negotiate the appropriate basis or bases on which the program should be judged.

Recommendations are actions to consider as a result of the evaluation. Forming recommendations requires information beyond just what is necessary to form judgments. For example, knowing that a program is able to increase the services available to battered women doesn't necessarily translate into a recommendation to continue the effort, particularly when there are competing priorities or other effective alternatives.

Thus, recommendations about what to do with a given intervention go beyond judgments about a specific program's effectiveness. If recommendations aren't supported by enough evidence, or if they aren't in keeping with stakeholders' values, they can really undermine an evaluation's credibility.

By contrast, an evaluation can be strengthened by recommendations that anticipate and react to what users will want to know. Justifying conclusions in an evaluation is a process that involves different possible steps. For instance, conclusions could be strengthened by searching for alternative explanations from the ones you have chosen, and then showing why they are unsupported by the evidence. When there are different but equally well supported conclusions, each could be presented with a summary of their strengths and weaknesses.

Techniques to analyze, synthesize, and interpret findings might be agreed upon before data collection begins. It is naive to assume that lessons learned in an evaluation will necessarily be used in decision making and subsequent action. Deliberate effort on the part of evaluators is needed to ensure that the evaluation findings will be used appropriately. Preparing for their use involves strategic thinking and continued vigilance in looking for opportunities to communicate and influence.

Both of these should begin in the earliest stages of the process and continue throughout the evaluation. Design refers to how the evaluation's questions, methods, and overall processes are constructed. As discussed in the third step of this framework focusing the evaluation design , the evaluation should be organized from the start to achieve specific agreed-upon uses.

Having a clear purpose that is focused on the use of what is learned helps those who will carry out the evaluation to know who will do what with the findings. Furthermore, the process of creating a clear design will highlight ways that stakeholders, through their many contributions, can improve the evaluation and facilitate the use of the results. Preparation refers to the steps taken to get ready for the future uses of the evaluation findings. The ability to translate new knowledge into appropriate action is a skill that can be strengthened through practice.

In fact, building this skill can itself be a useful benefit of the evaluation. It is possible to prepare stakeholders for future use of the results by discussing how potential findings might affect decision making.

For example, primary intended users and other stakeholders could be given a set of hypothetical results and asked what decisions or actions they would make on the basis of this new knowledge. If they indicate that the evidence presented is incomplete or irrelevant and that no action would be taken, then this is an early warning sign that the planned evaluation should be modified.

Preparing for use also gives stakeholders more time to explore both positive and negative implications of potential results and to identify different options for program improvement. Feedback is the communication that occurs among everyone involved in the evaluation.

Giving and receiving feedback creates an atmosphere of trust among stakeholders; it keeps an evaluation on track by keeping everyone informed about how the evaluation is proceeding. Primary intended users and other stakeholders have a right to comment on evaluation decisions. From a standpoint of ensuring use, stakeholder feedback is a necessary part of every step in the evaluation. Obtaining valuable feedback can be encouraged by holding discussions during each step of the evaluation and routinely sharing interim findings, provisional interpretations, and draft reports.

Follow-up refers to the support that many users need during the evaluation and after they receive evaluation findings. Because of the amount of effort required, reaching justified conclusions in an evaluation can seem like an end in itself.

It is not. Active follow-up may be necessary to remind users of the intended uses of what has been learned. However, several conditions can limit the implementation of this approach, including changes in long-term goals, a lack of consensus regarding the fundamental objectives of the organization, a lack of adequate program and cost data, and the difficulty of administering programs that involve several organizational units.

Despite these limitations, program budgeting is often used as a planning device while budget allocations continue to be made in terms of objects and organizational units—a process that has been adopted in many schools throughout the nation. As with performance budgeting, program budgeting information may be used to supplement and support traditional budgets in order to increase their informational value.

The basic tenet of ZBB is that program activities and services must be justified annually during the budget development process. The budget is prepared by dividing all of a government's operations into decision units at relatively low levels of the organization. Individual decision units are then aggregated into decision packages on the basis of program activities, program goals, organizational units, and so forth.

Costs of goods or services are attached to each decision package on the basis of the level of production or service to be provided to produce defined outputs or outcomes. Decision units are then ranked by their importance in reaching organizational goals and objectives.

Therefore, when the proposed budget is presented, it contains a series of budget decisions that are tied to the attainment of the organization's goals and objectives.

The central thrust of ZBB is the elimination of outdated efforts and expenditures and the concentration of resources where they are most effective. This is achieved through an annual review of all program activities and expenditures, which results in improved information for allocation decisions. However, proper development requires a great deal of staff time, planning, and paperwork.

Experience with the implementation of this approach indicates that only a periodic comprehensive review of ZBB decision packages for some program activities may be necessary.

A minimum level of service for certain programs may be legislated regardless of the results of the review process. As a result, ZBB has had only modest application in schools, although the review of program activities makes ZBB particularly useful when overall spending must be reduced.

Site-based budgeting is widely considered the most practical for budgeting within the school district environment because it provides greater control and reporting of school-level data. This approach which may be used in combination with any of the four discussed above emphasizes the decentralization of budgetary decisionmaking.

Site-based budgeting places local managers and other staff at the center of the budget preparation process, making them responsible for both the preparation and the maintenance of the budget.

Site-based budgeting is popular in many school settings. Within a school system, site-based budgeting generally involves granting increased budgetary authority to the school. Resources are allocated to the site, with budget authority for programs and services granted to the school's principal and staff.

Campuses are normally allocated a certain level of resources that they have the authority to allocate to educational and support services. These budgetary allocations are meant to cover those areas over which campus decisionmakers have control. For those who are handling school funds and others that revolve around money in school, we have School Budget templates for you.

Importance of Budget Plans Budgeting gives you awareness of your current income and prevents you from overspending. Budgeting will ensure you of contingencies for emergency purposes.

Budgeting can discipline you by keeping you away from the habit of unnecessary spending. Also check out our collection of Yearly Budget samples that you can use for the entire year.