Film: Improving the Museum Visitors Experience Through Evaluation

This presentation on improving the museum visitors experience through evaluation was made by Elizabeth Bolander and Bethany Corriveau from Cleveland Museum of Art at MuseumNext Indianapolis, September 25th, 2015.

Bethany : Right, thank you all for coming out today. My name’s Bethany Corriveau, this is Elizabeth Bolander, as you said, both from the Cleveland Museum of Art. I develop and coordinate public programming, and Elizabeth is the head of research and evaluation. And, yes, we are lucky enough at the Cleveland Museum of Art, to have a research department who can help us answer questions that we have about programmes, about membership, about pretty much everything that goes on in the museum in some way.

So, what we’re going to talk about today are a couple of different studies that we’ve done that have helped us change the way we think about public programming at the Cleveland Museum of Art. Now, it all started in 2012, going back in the depths of time here, when the Cleveland Museum of Art was just coming off of a major building project. We were opening up new gallery spaces, we were hosting more exhibitions than we had in quite a few years, and with that, we were doing a lot more programming.

So at the time, that meant lectures, a lot of lectures, different topics but a lot of lectures, and with all of this new kind of floating around in the air, and we were thinking oh, we’ve got these new exhibitions, we’re excited to show people our new building, we were thinking okay, there’s probably a lot of new visitors coming in and taking advantage of our programming of all of these different lectures that we were doing for exhibitions and for the permanent collections.

So my job, I’m the person who goes to all the lectures, makes sure that people know how to use the remote, make sure that the power point projector works, things like that. So I went to all of them and I started to notice that I was seeing some familiar faces, like okay, it’s a lecture on [worry] tapestry woven tunics. You look really familiar. I saw you last week at contemporary artists, Martin [Kreen]’s talk – two very different subjects.

So question came up, were we really getting those new varied audiences that we thought we were getting? Now we go to the slide, okay, so this is when Elizabeth and her team entered the picture to help us answer that question: who was coming to our programming? In 2012 and 2013, over a period of about four months, they came to lectures and distributed exit surveys to ask the people who were coming why they were coming, a little bit about their background and what they learned from lectures.

So the results confirmed some of the things that we kind of suspected after observing these lectures. If you’re remembering that picture I showed at the beginning, this probably looks pretty familiar. And so answering that big question, were we getting those new audiences, yeah, I really was seeing a lot of familiar faces so the biggest percentage of our survey respondents came to lectures pretty frequently.

Not only did we have this fairly consistent demographic attending lectures, it was clear that we had a pretty small group of that consistent demographic that was coming back over and over again. Now, throwing down the gauntlet here, I think Katie Hill earlier said can you say your museum’s [measurement] by memory. I can’t but now I’m going to learn it. I do know this part. We’re trying to serve the broadest possible audience and we were not doing it. We weren’t even really serving any different audiences from programme to programme, essentially programming for the same group over and over.

So after seeing these results, we started to reconsider the way we were planning our programming. Before, what we’d really focussed on was differentiating the content, whatever topic that lecture was about, so we’d have lecture after lecture, but think oh, it’s going to attract different audiences because this one’s about contemporary art, this one’s about medieval art, this one’s about African art, Greek and Roman art, whatever. We were thinking these different topics are going to bring in different audiences.

Well, not really the case. So what we started to do was think much more about the way that people wanted to experience those topics and how different audiences might be more inclined to take opportunity of our programmes if we had different offerings, if the topic wasn’t the only reason that we thought they were going to come.

So, since that study, we definitely started expanding the types of programming that we’ve offered. One big change was that we moved from a lecture hall into the galleries much more frequently; instead of having curators give lectures, for example, we told them to take groups into the galleries and actually look at the art instead of showing a picture of that same art on a slide. We’ve also started trying to build on the interest that people already have, so things like books and movies, looking at the collection through that lens; activities that they might do elsewhere, bringing those into the museum, like [unintelligible 00:06:19] classes, meditation, even your friendly neighbourhood bar trivia night – we’ve tried that out at the museum.

And we’re also just starting to get into opportunities where people can draw creative expression or inspiration from the collections and have an output of writing assignments, or storytelling opportunities, or even in some cases, fashion shows. So now we bring Elizabeth and her team back in with all this new programming, we’ve started to get some new questions: is it working, are people getting what we want them to out of it, are we getting new audiences, and I’ll turn it over to Elizabeth to tell you some of the results of the study.

Elizabeth : Thanks, Bethany, so at the same time that Bethany and the rest of the interpretation team is coming to us and asking: “Elizabeth, we want to understand what’s happening at our programmes”, other people in the museum were doing the same thing. And so instead of doing what we typically did, which was okay, we’ll do an evaluation of your programme, and your programme, and your programme, and so we said why don’t we put them all together, and so instead of looking at it by department or even by programme, we wanted to look at it the way our visitors look at our slate of programming, which is what is the institution offering to our visitors and family visitors?

And so with the help of the Office of Policy and Analysis, specifically Dr Andrew [Percaric] at the Smithsonian, we boldly went where we had not gone before and we did a big study. We identified 15 programmes that ranged from our large-scale young professional series called ‘[Mix] at CMA’ that’s held on the first Friday of each month, which attracts 1500-2000 people, all the way to our public tours that are offered daily that can attract smaller groups of folks.

And we wanted to understand not only how do these programmes do, but we wanted to understand how do they compare to each other, because I think typically when we look at programmes we want to compare them on the metrics of attendance and revenue, but I think we all know that programmes are really more than that; they’re more than the attendance and the revenues sometimes. So after talking with our internal stakeholders we had identified some key performance indicators that we wanted to test, and we tested this both on all of these programmes, as well as we asked these questions of our general museum visitors who hadn’t gone to a programme because we wanted to see what the comparison was. So what were those key performance indicators?

They were providing a satisfying experience for our participants – that’s always good – we wanted to connect people and art, we wanted to connect people and make them feel more comfortable and engaged with the CMA, and then we also wanted to bring in new audiences and use programming as a vehicle to do that. So in total over this four month period I think our team has headed more programmes than I think we ever had at the museum before, which was quite an experience, and we got nearly 1700 surveys.

So we want to share a couple of the real broad themes that our research found, although if anyone’s really interested in the 152-page report, just give me a call. But some of the bigger findings that we found were that there is a real connection between flow, which some of you might be familiar with psychology, [Mihaly Csikszentmihalyi] – I think I got that right – his work on flow, which is the idea of becoming so engaged in something that you essentially lose track of time.

And we asked participants if, during their programmes, if they really felt like they experienced this state, and what we found was that not only did flow happen more frequently in programmes where it was a bit more intimate setting, or some of the new programmes that we had done, like medication in the galleries, but that flow had a direct statistical correlation to participant satisfaction with that programme, so as flow increased, our overall experience rating increased and we really found that was important. We also found that some of our traditional programmes, like lectures, did not do so well in stimulating flow experiences.

We also asked the question, “Did you feel connected to an art work as a result of your programme?” And we found that 68% of our total respondents felt that way and that’s a great number but you might be wondering what about that other 32%, what was going on there. We are in an art museum, after all, and I think this is an area where we’re still trying to delve into it, and certainly some programmes did better at this than others like if they went in the galleries, that certainly helped, but even those who went to lectures, some of them did and some of them didn’t feel connected to art.

And the other thing we found is that while some programmes did better than others, the average number of new attendants at a programme was only 8%. We have a very high frequency audience anyway at the CMA. Our normal everyday exit survey show that we have nearly half of our people come three or more times a year, but for programmes that was 56% of all the programme attendants came to the museum four or more times a year so they’re very frequent attendees, and only 8% were new. So some programmes like the Art [Trivinate] actually did better than this, but it was still only less than 20% so there’s really room to grow.

Alright, so now that we have all of this information, how do we implement it with programming? And a lot of the thinking is yet to be done. This is, as Elizabeth will say in data collection too, this is very much a work in progress. I’m trying to figure out what we can do to improve our programming and make sure that we’re serving the broadest possible audience.

One thing that we’ll definitely have to do is go back and look at our programmes and see what, why aren’t they attracting these outside audiences; even after we’ve broadened them, after we’ve gotten that demographic quite a bit less homogenous as it was with the lecture study, why aren’t we getting these new audiences in.

Elizabeth : Yeah, one of the other things we really were excited about is that finally we have the ability to have an institution-wide conversation about we’re trying to achieve with our programmes. How does performing arts and education and our young professional series, how do they all work together to further these goals that we have for our programmes overall. And we’re really hopeful that this study provides a good benchmark of data that then, as changes are made, we have something now to compare it against to see really did the change have an effect on the experience.

Bethany : Along with that, just thinking about programming from those of us who are implementing it, the attitude of thinking about programming as a whole, I think will be strengthened by the study where we’re studying the programming as a whole. Going forward we should be planning programming much more as a whole, and I think this is probably true of some other institutions as well, but we tend to be kind of siloed, and hopefully this will be a step forward towards breaking out of that. So more work to be done but I think we’ve got a few minutes if anyone has any questions before we head out.

Question: How are you breaking those siloes, how are the institutional or at least the staff membership meeting that need?

Bethany : Well, one of the things that we’ve started doing recently, and there’s still a lot of work to be done I should say; they’re not quite broken down yet, we’ve had various iterations of a programming committee over the past couple of years, and just within the past year we’ve really started to look at the calendar much more holistically. Our scheduling software, they’ve actually created a special report that we can draw out just public programmes so that allows us, as a group, with all of the people who are doing programming, to look at that calendar and identify where maybe we’re not meeting the needs of a particular audience or we’re completely overbooking ourselves or any other numbers of factors that may affect the experience of a programme.

Question: Just a few practical questions. Was this paper survey and who distributed it, and would you be interested in sharing notes, and can we have a look at the questionnaire?

Elizabeth : Yeah, so we did most of it via paper because we’re crazy. We got … I highly recommend, if you’re going to do a programme study like this and you do go the paper route, we had tear-off sheets so when our research staff were administering the survey by going up to someone after a programme was completed and just asking them to fill it out, we could immediately give it to them, rip one off and then hand the pad to another person so that really helped make it more efficient especially in an event where you have 2000 people, all exiting at the same time. But for tours, we would actually; our researcher would go on the tours so they were with them so whenever they stopped we were able to do the survey right away.

Question: I have the question of, even though you’ve done all this work, you’re still seeing only relatively small movement of the dial but how much of that do you a think is just related to sheer marketing in the way that the museum spends marketing dollars? The reason I ask that I’m director here at the IMA now but I used to work in Cleveland before 2012, but nevertheless, when I was there, it was like the IMA has been, just very traditional. We spend almost all of our marketing money on temporary ephemeral exhibitions and spent nothing more than the amount of money it took to put those programmes in a magazine and send the out to the people we talk to all the time, our members, but seldom did we spend real money and time marketing programmes beyond that very much.

Elizabeth : You know, thinking about the data, and I’ll let Bethany comment as well, but the data really showed that most people found out about our programme from word of mouth, the [member] magazine, which, of course, where all of our programmes are, and then for some programmes, particularly those that I think did attract a little bit more that newer audience, digital communications were really critical, and those showed up much more. We didn’t see that advertising or PR really drove a lot of programming, and part of that was because there wasn’t a lot, necessarily, in the market about those programmes.

Bethany : Yeah, I think you’re hitting the nail on the head here, that that’s a big question, and certainly something, another one of those next steps that we need to address – how are these programmes getting out there, how are we notifying people, do they get listed in the members magazine, and as you might expect, that’s a lot of our attendance. It’s definitely something to be working on.

Elizabeth : Now we have the data to prove it.

Bethany : That always helps.

Question:        It’s me again, following up on my silo question, with apologies to other curators in the room. I guess my question is how do you get people who aren’t necessarily involved in programming, or as the case of marketing, how do you get those siloes involved, how do you get everybody at the museum following the same path and reaching the same goals?

Elizabeth : I don’t have all the answers. I wish I did. But I am an evaluator so I’m slightly biased so I’m going to say that I think having research that actually showed how these programmes were evaluated on the exact same metric, so not the metric the performing art set, not the metric that interpretation set, but on metrics that we believed in and we have shared. I think it’s helpful for sparking a conversation about thinking more holistically and we’re still in the throes of getting that conversation going but at least now we have a common document that we can look at together instead of this is what I think about programme, and this is what I think about programming, but instead we have something that brings it all home.

Bethany : Yeah, I would absolutely agree with that. Now that we have the document to point to, which has not actually been out for that long so I think maybe next year we’ll have more to share along those lines, but I’m hopeful that it will help us break down some of those walls. And we’ve got a red flashing light up here so we’ll say thank you very much and we’ll be around if anybody has any further question.

Elizabeth : Thank you.

—

This presentation on improving the museum visitors experience through evaluation was made by Elizabeth Bolander and Bethany Corriveau from Cleveland Museum of Art at the MuseumNext conference in Indianapolis, September 2015.

To stay informed about our International Museum Conferences follow MuseumNext on Twitter or like MuseumNext on Facebook.