When contemplating the work of creating and managing evaluations within a museum, library or other informal educational organization, there are several important points to consider.  As professional program evaluators, we believe that these points should be at the foundation of any effective evaluation effort. In this article, we’re going to talk about these must-consider points, and along the way discuss a number of practice-proven techniques that can help your organization produce a successful exhibit or program evaluation.

Sun Associates works with informal as well as formal educational organizations in the design and implementation of program evaluations. Our project experience ranges from evaluating American History curriculum-development projects in public school districts to assessing the impact and design of STEM science museum exhibits. We bring to all of these engagements a process of program evaluation that has allowed us to tease out the best practices that are highlighted below.

What Is Evaluation?

Within the context of this article, we define evaluation as a formal process for determining the degree to which programs of activity are meeting their design goals. There are several important terms embedded in that definition.

First, we refer to the evaluation process as “formal” to indicate that the evaluation has agreed upon rules and assigned roles for participants. Those rules and roles can be different from one evaluation to the next, but it is important to come to terms around the overall process early on in the evaluation work, and then to follow those guidelines as the process unfolds.

Next, our definition mentions “programs of activity”, and by this we mean that evaluation is about the relationship between goals and the various activities implemented to accomplish those goals. Within the museum world, examples of such activities might include physical exhibit design, educational content development, staffing, and publicity/dissemination. Again, the specific activities that make up the program being evaluated may vary, but it is important to narrow in upon what activities are being evaluated before an evaluation commences. The work of framing and planning your evaluation helps focus your evaluation.

Finally, our definition includes the concept of “design goals”. All activities such as exhibits or educational programs will have goals; but it is important before start an evaluation that you have clearly identify your program’s goals, and also that you have given serious consideration as to what constitutes success in meeting those goals. Just like deciding upon the evaluation process and focusing upon what activities will be evaluated, identifying the goals is up-front work that needs to be done prior to starting the actual evaluation. Identifying and clarifying goals will also help focus the evaluation and ultimately help you make efficient use of organizational funding and resources.

Consideration #1 -- Getting The Right People To The Table

It is important that any museum evaluation effort – whether of an educational program or an exhibit or some combination thereof – be planned and initiated in concert with key program stakeholders. These are the people “at the table”. Who are these stakeholders? At a minimum, they need to be the individuals and departments involved in the actual design and creation of the program being evaluated. Since evaluation is not a process that is intended to create surprises, evaluation design must include input and buy-in from all whose work will be evaluated.

How does this work in real life? In our museum work we have seen evaluations stumble when, for example, curators design a project and contribute to evaluation criteria but educational outreach staff are not at the table. Such situations can result in setting unrealistic or incomplete evaluative criteria for the evaluation.

A positive example we have found are those projects that bring program participants – such as teachers for educational outreach programs – to the evaluation planning work in order to provide perspective. Often times, projects with deep stakeholder input to the evaluation are able to uncover and highlight aspects of program design that might not have been noticed by individuals who have different functions in program implementation. In short, evaluation design should involve as much stakeholder input as possible. This is true regardless of whether an evaluation is conducted as an in-house endeavor or facilitated by an external evaluator.

#2 -- Develop a Theory of Action – a.k.a., Logic Model

If your exhibit or program is rooted in a proposal made to a public agency or non-profit foundation, there’s a good chance that you needed to create a theory of action or “logic model” as part of that proposal. It has become common practice for funders to ask for these graphical representations of how your proposed work will meet identified needs among targeted audiences in order to achieve specified outcomes. The documentation of needs-audiences-actions-outcomes is called the  theory of action. When this is literally illustrated via model or map, it becomes a convenient way to drive home the point of your work and show what you expect your audiences to gain from interacting with your program or exhibit.

The logic model is an extraordinarily useful tool for designing evaluation efforts once a program has been funded. By looking at the logic model, an evaluation effort can clarify project goals and participants’ relationships to program components (actions). The logic model highlights key aspects of how a program is intended to accomplish its goals and therefore points out those aspects of the program that need the attention of evaluation.

We have a section on our website that is devoted to a more in-depth examination of logic models and have listed other resources at the end of this article.

#3 –Evaluation is Different From Market Research

Thus far, we have talked about the evaluation of program impact and outcomes.  Examples of this sort of evaluation work could be:

• Examining the extent to which a library’s exhibit on maps and mapping communicates that cartographers make choices about how to draw the maps they create in order to convey particular political and cultural perspectives (Bending Lines at the Leventhal Map and Education Center, Boston, 2020)
• Evaluating the extent to which a science museum’s full-scale, interactive, mock-up of the International Space Station conveys that science performed in space has direct applications to life on Earth (Science on Orbit at the U.S. Space and Rocket Center, Huntsville, 2016).

In both of these examples, the focus of the evaluation was to determine the extent to which the project was achieving its core educational mission. Nevertheless, as is generally the case with most museum and library projects, these host organizations had a number of questions about visitor demographics and visitor/participant reactions to a wide range of organizational objectives (e.g., “What other subjects would you like to see addressed by future exhibits and programs?” or “Are you interested in contributing to our museum’s annual fundraising appeals?”). These other questions tend to fall into what we consider to be visitor or market research; and in both of these projects, we advised the organizations that the evaluation should not include these other investigations in the evaluation work.

Evaluation – evaluators - need to honor the fact that museums have important needs around market research. Nevertheless, it is critically important to keep the evaluation itself focused on assessing the process/progress/direct outcomes of the particular exhibit or program being studied. Visitor demographics and fund-raising metrics can be entirely legitimate components of a program evaluation if they are logically connected to a program’s design or intended outcomes. If these metrics contribute to the project’s theory of action, then they are reasonable data to collect for the evaluation.  Otherwise they are research activities that should be carried out external to the evaluation.

#4 –Developing Performance Indicators Related to Your Exhibit’s Design Goals

Evaluations work by assessing a program’s actions against some established set of performance criteria. The most useful evaluations use a set of quantitative as well as qualitative criteria for their performance criteria. Indicators – in the context of a program evaluation -- are literally statements of the performance criteria for success. Indicators describe what it looks like when a program or exhibit meets its goals and intended outcomes. Where do these indicators come from?

Indicators should be created by the project’s stakeholders (back Consideration #1) through a process facilitated by either an external or internal evaluator. Still, that leaves unanswered the basic question of what indicators are appropriate for any given evaluation. When we set out to conduct a museum project evaluation, we usually root the indicators in one of several sources. For example:

• The exhibit/program proposal, if the work was based upon a proposal. What did the proposal say that the program would create, and what were the purported benefits to participants (visitors, the museum, the community, etc.)? Descriptions of these creations and benefits are indicators.
• Published, accepted standards related to intended educational outcomes. So for example, if your STEM education program aims to teach children “how to think like an engineer” then a review of state and national curriculum frameworks for STEM learning is necessary to tease out what students who can “think like engineers” typically say or do. Likewise, as another example, if your program aims to support teachers in the teaching of an aspect of History or Social Studies, then it makes sense to turn to the standards published by organizations such as the National Council for the Social Studies or your state’s social studies/history curriculum frameworks.
• Museum associations – such as the American Alliance of Museums (AAM) – have established professional standards that relate to many of the general avenues of endeavor for museums (similar standards exist specifically for libraries, children’s museums, and science museums). For example, AAM has a full set of core standards that relate to aspects such as collections stewardship and education and interpretation. These can be the basis for indicators related to the degree to which your program has met those aspects of the standards that are applicable the work that you are doing.

Beyond these, the actual work of developing indicators comes from engaging with your evaluator to map information drawn from external standards/expectations to your outcomes. Usually it is necessary to expand information from these sources into descriptive text that discusses what one could expect to see/measure for each of the indicators you develop. Assuming once again you have the right people at the table guiding your evaluation design, this is often not difficult work.  After all, you will be having a discussion about the work that you know and enjoy doing, and your stakeholders are the ultimate “experts” in what constitutes success in their own work.  Another benefit of the indicator/performance measure development work is that this becomes a tremendously valuable way of reflecting upon and clarifying your work – perhaps in advance of actually or fully mounting your exhibit or program.

An example set of indicators for a museum exhibit evaluation can be found on our website.

#5 – When It’s Time to Collect Data, Make It Count

Data collection is the most labor-intensive part of evaluation. This labor can be made more efficient through developing an evaluation plan as discussed above. A plan will allow you to collect that data which responds closely to your indicators and does not engage in the range of “wouldn’t it also be good to know?” quests that can drag down your efforts. Our experience is that out of control, constantly expanding, data collection is one of the drivers for why many organizations have a negative opinion overall of program evaluation. Museums with limited budgets and resources certainly do not want to engage in open-ended or perhaps low-return efforts around surveying and other type of data collection.

In terms of how to best design and conduct surveys, observations, focus groups, etc., our website has a variety of resources for data collection design. Assistance can also be found through making connections with survey research courses and faculty at local higher education institutions (e.g., community and state colleges where there is a ready supply of students – and faculty advisors – looking for experience creating and running data collection efforts).

Once again, the key in engaging in data collection through any means or by any staff is to focus the work on gathering evidence connected to the indicators you have created in your evaluation plan. This brings us right back to the definition of evaluation stated at the beginning of this article. Evaluation is a planned, process-driven activity. Data collection has to be part of the plan, and needs to proceed according to plan.

#6 -- Why Are You Doing This?-- How You Will Use Your Evaluation?

The last piece of advice that we offer as professional program evaluators is that any evaluation is only as good as what your organization can learn from it; and you cannot learn from your evaluation if you do not share its findings with your stakeholders. Evaluation reports – not to mention the formative, day to day discussion of evaluation findings as are they are emerging – are intended to inform as well as celebrate progress. In all of our projects, we encourage our clients to create public-facing versions of the evaluation reports created. Likewise, evaluation output should be a regular source of input into staff presentations at conferences, annual reports, development efforts, and the like. In other words, use your evaluations and you will find that overtime you can develop a culture of evaluation and reflection that will have both immediate as well as long term benefits to your museum.

Conclusion

We believe that any organization that has thought through the six points discussed in this article will have a leg up in creating their own evaluation or perhaps in creating specifications for productively engaging with an external evaluator. At its core, evaluation is an essential part of meeting the mission of nearly any museum, library or informal educational organization. That is to learn from what you do, and to develop the knowledge and information to constantly improve your work. In this way, we consistently find reward in engaging in the work of evaluation, and we believe that you will too.

Additional Resources

American Museum of Natural History: Guidelines for Evaluators and Evaluation Rubric. 

Institute of Museum and Library Services: Evaluation Resources. 

O'Sullivan, Rita. (2012). Collaborative Evaluation within a framework of stakeholder-oriented evaluation approaches. Evaluation and Program Planning. Vol. 35. Pp. 518-22.

University of Wisconsin-Madison. Program Development and Evaluation.  An all-around great collection of information on practical program evaluation. Select the “Methods” tab for resources on data collection design and procedures. Contains information on creating logic models.

About the Author
Jeff Sun is director of Sun Associates which assists libraries, museums, educational non-profits and public school districts in developing evaluations that inform practice and increase accountability to program stakeholders. As an educational program evaluator, Jeff has over 30 years of experience strategically advising and evaluating programs in formal as well as informal educational organizations. Jeff can be reached at jsun@sun-associates.com.

Photo by Free-Photos from Pixabay