Three years ago, the museum where I was working faced a sudden and unexpected change to its site operations and visitor experience. A specific space that served as a crucial component to what we offered visitors had to be reinvented and ready to launch in ten weeks before the beginning of the season and before the museum officially transitioned to paid admission for the first time in its history. No pressure, right?

My team at the time consisted of a few part-time people from within the education department and a few full-time people from other departments who assisted with certain aspects of this new project. After our emergency meeting to determine a concept and a timeline, we decided to create a living history program with four programming elements: 1.) first person interpretation, 2.) participatory activities, 3.) a physical memento/takeaway, and 4.) themed retail offerings. Moreover, here was a golden nugget of an opportunity to “do things right,” as in: incorporate formative, remedial, and summative evaluation in ways our site had never done before. [Insert excited museum educator clapping her hands with glee here.] The whole experience required patience, fortitude, and trust in my own instincts as well as careful navigation of museum divisions, department politics, and financial pressure. Looking back, the development and evaluation of this program marked one of the most difficult projects in my career and one from which I’m still learning.

The program we created did not become a raging success; nor was it a complete failure. Ultimately, it served as an example of the type of program that serves its purpose but not overwhelmingly well. An example of the “should-a would-a could-a” mindset, of visitor interest that did not translate into high levels of visitor engagement and, most importantly, an example of the transformative power of evaluation and data.

Evaluation Approach

After writing the new living history program’s learning outcomes and the initial interpreter script, staging the new space, developing new participatory activities, and ordering costuming, we ran out of time to test anything before the site reopened. (As most museum professionals know, we never have the luxury of focusing exclusively on one program or project at a time.) That meant our first launch was a soft launch, and the first month of the program’s existence was one of constant trial and error. I wanted to incorporate evaluation as soon as possible, keeping in mind that visitors were already experiencing the program well before we felt ready. I found my notes from a stellar conference session I attended at the American Alliance of Museums Annual Meeting that year; I also found my notes from articles I’d read in my Museum Education & Interpretation class with Jennifer Deprizio at Tufts University years ago; and I pulled out the March 2015 issue of the Journal of Museum Education, entitled “Empowering Museum Educators to Evaluate.”^[1]  

Armed with these valuable resources, I determined our best course of action for evaluation included behavior observations and visitor interviews. I created a simple behavior checklist for both the visitor and the interpreter, and I wrote the interview questions directly influenced by my notes from the AAM conference session Beyond Cognition: Methods for Measuring Other Types of Learning.^[2] My team wanted to find out if visitors understood the purpose of the space and, of course, if they were learning what we’d hoped they’d learn. The data would inform how we altered or strengthened the program. But I also had an ulterior motive: I wanted to determine if behavior observation worked in a museum like mine. I was curious about the data as much as I was curious about visitor perception of the program. That curiosity became my obsession that season, and I emerged with a new mantra: behavior observation is a GAME CHANGER.

Ultimately, we administered two waves of evaluation: the first month in and three months in. Over the three-month period of time, we conducted nine one-hour observation periods of 36 living history presentations to 374 visitors and nine exit interviews to 24 visitors. I wrote a remedial assessment report that guided our internal discussions in the education department and updates to the board, and I wrote a summative report at the conclusion of the season with my recommendations for the future of the program and programs like it. This process indisputably proved to me that behavior observation is an under-appreciated, under-discussed underdog of evaluation methodologies.

Benefits to Behavior Observation (i.e. Why This Method Won Me Over)

1. It’s formative and/or remedial. Formative evaluation occurs while the project is being developed and determines how well it functions. Remedial evaluation takes place once the program is open to the public and helps to fix unforeseeable problems.^[3] Behavior observation can take place with test groups prior to the launch of a program or after it has officially launched. Front end and summative evaluations seem to hog all the attention, but active, progressive evaluations (i.e. formative/remedial) serve as the real workhorses.

2. It’s easy and takes no extensive training. Museum educators are not often trained in evaluation as a component of program development, unfortunately. Taking on evaluation can be intimidating. However, as someone with limited experience distributing visitor surveys and analyzing visitation data, I still ably determined how to go about conducting behavior observation. It requires a checklist, a pen or pencil, and a dedicated observer.

3. It’s low-cost and relatively efficient. Per the previous point, the observer does not have to be a paid staff member. That responsibility could technically fall to an astute volunteer. Or, if the responsibility does fall to a paid staff member, it requires finite periods of time that can be planned and worked into their schedule. For that reason, this methodology is a good use of time and energy.

4. It’s objective. How is being impartial or unbiased even possible, you ask? Here’s the rub of behavior observation: people either did or did not do the thing. That’s it. Hearing that clarification from Joe Heimlich, the Co-Director of the Center for Research and Evaluation at the Center of Science and Industry, opened my eyes in the most impactful way possible.^[4] Your checklist must be task-specific, and therefore your observation becomes restricted to the tasks on the list only. Freeing yourself from any kind of deductions or analyses during the observation period allows you to record with detail what people did or did not do.

5. It combines well with other evaluation tools. As George Hein indicates in his chapter on studying visitors in his seminal work Learning in the Museum, best practice in evaluation requires a combination of methods.^[5] Behavior observation pairs beautifully with visitor interviews, comment cards, timing and tracking, or focus groups, to name a few.

6. It works for visitors and front-line staff equally. In the development of our living history program, my observation checklists consisted of visitor behaviors as well as interpreter behaviors. In a program highly dependent on our interpreters, we wanted to determine the scope of their delivery and how their actions affected our visitors. The duality of my observation periods proved invaluable.

7. It fosters a deeper understanding of your visitor as well as your interpreter. Naturally, as a result of repeated observation periods, you develop a much keener comprehension of several aspects of the program and your museum: you directly witness the ebb and flow of visitor traffic throughout the day, you determine patterns in behavior of people in a specific space, you come face-to-face with your own preconceived notions of what will work or what will be engaging. You strengthen your ability to watch, examine, and simply take in information.

Considerations for Behavior Observation (i.e. What To Watch Out For)

1. It can alter behavior at times. An observer’s presence can absolutely affect the way people behave naturally, visitors and interpreters alike. Blending in with the crowd while carrying a clipboard proves difficult at times, especially in smaller spaces. This method works best when there are larger crowds of people and the observer can be as unimposing as possible.

2. It can become time-consuming. I realize this point contradicts one of my listed benefits. Behavior observation is efficient, but you can quickly rack up time dedicated to it for a few reasons. The ebb and flow of visitor traffic can change from day to day, so you may spend an hour observing very few people. That doesn’t give you a lot of data to work with. You cannot control visitation, and quite often you may not be able to completely control your own schedule. I recommend planning observation periods for a variety of time slots and days of the week to get a compelling sample set.

3. It can be a gamble. As a result of the reasons listed above, behavior observation can feel like a throw of the dice on some days. You may find certain observation periods to be a waste of your time, and it may take you a while to find time periods with the biggest yield. No one ever said evaluation didn’t require patience. Understanding the benefits and limitations to this approach ahead of time significantly help with planning out your work.

4. It does require some guesswork. How you analyze the data you glean from your observation periods can feel like subjective deduction. Ultimately, assessing visitor engagement based on whether or not they did a particular task can seem like a big leap. Without knowing each visitor’s social, personal, and physical contexts, how can we accurately measure engagement? Combining behavior observation with timing and tracking might help solve that problem. Janet Gail Donald discusses the value of measuring visitor movement and time on task while also pointing out that exhibits often motivate visitors to pursue post-visit learning, positive actions we cannot possibly observe.^[6] We can only make our best guesses based on behaviors in the space at a certain moment.

Program Results and Conclusions

On the whole, we saw a 12% increase in visitors staying to watch the entire first-person presentation between the pre- and post-intervention periods, with an average of half of all visitors staying to watch. We also saw a 3% increase in visitors actually doing one of the activities. While those numbers were promising, they were still underwhelming. However, we saw the biggest gains in interpreter behavior. In the same time period, we saw a whopping 53% increase in interpreters directly mentioning an important fact crucial to the program and another whopping 62% increase in interpreters using specific key phrases from their script.

These positive increases in visitor engagement and interpreter performance can be attributed to a number of changes we made after the initial wave of evaluation: we refined the first person and storyline aspects of the living history component and wrote new scripts; we created interpretive signage for the space; we posted living history performance times; we altered the layout and instructions for the participatory activities; and we held an additional in-person training for interpreters working in this program.

Because of the internal politics and resistance I faced at the time, I’m not sure I would have been able to advocate for those major alterations without being armed with the initial evaluation data I had. My data provided black-and-white evidence for my recommendations. Visitors either did or did not stay for the whole presentation. Visitors either did or did not partake in the activities. Visitors either did or did not ask any questions of the interpreters. Likewise, interpreters either did or did not mention certain important facts. Interpreters either did or did not invite visitors to explore other aspects of the site. My recommendations were not anecdotal or prejudicial. They were based on observable facts. I learned that formative and remedial evaluation serves as the museum educators’ defense.

My final assessment report included our biggest surprises and biggest struggles, important reflections of what we learned and how those lessons might inform future program development. For example, we were pleasantly surprised that visitors wanted more signage and more “things” to look at. We discovered which activities excited visitors the most: fun, light, themed takeaway sheets scored the greatest enthusiasm. Furthermore, we realized that script creation and revision of first-person interpretation takes much more time than we anticipated, and the ten-week lead-time we received was simply not adequate. The initial training we’d provided was also not adequate. Hard lessons learned, but significant no less. I am proud of our first foray into first-person interpretation, and we did a pretty decent job given the constraints my team faced.

For me, this whole experience brought forth a new area of supreme and utter fascination in my work: the behaviors of others. Previously, I had focused my thoughts purely on what happens inside a visitor’s mind, obsessing over learning and meaning-making. Now I am obsessed with thinking about visitor actions in and out of the museum. What does the behavior tell us? How can I think more like a scientist? How do visitor behaviors connect with or relate to knowledge gained? I encourage you to go down that rabbit hole of pondering. Let that pondering lead you to action. Quoting an Alcott seems apropos here, so I’ll close with this musing by beloved Massachusetts intellectual Amos Bronson Alcott, “Observation more than books and experience more than persons, are the prime educators.”

Erin Wederbrook Yuskaitis is the Principal for Yellow Room Consulting LLC. She specializes in visitor learning, innovative program development, and cross-discipline meaning-making as well as the broad applications of Visual Thinking Strategies and museum education principles to industries and sectors outside the museum world.