Shaping Museum Stories with Neuroscience

In Spring 2017, the Peabody Essex Museum took the unprecedented step of employing the museum world’s first neuroscientist in residence.

The goal of the Neuroscience Initiative is to enhance visitor engagement. Toward that end, they draw on findings from the neuroscience literature to inform their design strategies for creating art experiences. As all experience is a product of brain function, they hypothesize that learning more about how the brain works will allow them to generate more engaging experiences of art and culture in PEM’s galleries.

Tedi Asher will share what she and PEM have learnt from this pioneering work.

Speaker

Tedi Asher
Neuroscience Researcher
Peabody Essex Museum (PEM)

This presentation was filmed at MuseumNext NYC in Autumn 2019

Tedi Asher: Thank you so much for that introduction, and thank you for inviting me to speak today. I’m really pleased to be here, and I’ll just confess that I’m a bit overwhelmed and overstimulated by what we just heard and the ways that I see it as intersecting with what I’ll talk about today. Maybe we can get to some of that in the question period. I’m really pleased to be able to share with you some of the work that we’ve been doing over the past couple of years as part of the Neuroscience Initiative at the Peabody Essex Museum. In the context of this conference, I’ll just say that I really see the aim of this initiative as attempting to inform the way that we share the stories that we tell our visitors. So far, we haven’t really used it in any way to choose the stories, but rather to shape the way that they manifest in a gallery.

Just by way of background, I was fortunate enough to be able to speak at a MuseumNext conference in 2018, at which point we were just about to launch our first so-called neuroscience study. I’m excited today to be able to share some of the results of that study with you, as well as some other insights that we’ve gleaned from other studies that we’ve run since then.

So that’s what I’ll be talking about, but before I get there, I just wanted to introduce our museum, which is actually much more expansive than what’s shown here. This is the front entrance to the Peabody Essex Museum, or PEM as we usually call it, which is a museum of art, culture, and creative expression located in Salem, Massachusetts, just north of Boston. Just to give you a brief sense of what kind of objects and works of art we deal with, our collection spans many different kinds of objects. These are sort of our curatorial departments, and it’s through the presentation of these objects and the narration of their associated stories that we attempt to fill part of our institutional mission, which is to create transformative experiences for our visitors.

I just want to pause here and emphasise that this creation of experience, an experience of art, is really key to how we see our work. It’s here that we come upon the origins of what is now the Neuroscience Initiative at PEM. Our former director and CEO reasoned that all experience is a product of the brain, and I’ll come back to that in a little bit, but if we take that as a premise, then perhaps by better understanding how the brain works, we might be able to create more compelling experiences of art for our visitors.

Within that context, I see the initiative as having fundamentally two goals. On the one hand, we of course want to allow research findings from the fields of neuroscience to inform the way that we design exhibitions, to put it simply, but it’s also my hope that by studying and observing the way that our visitors respond to the design decisions that we make, we might also be able to better understand how the brain works in this kind of applied setting, as opposed to in a lab, the implication there. Hopefully, the result of these endeavours will be to create more compelling experiences of art, but also to help bridge the gap between the arts and sciences, which I see as addressing many of the same questions about the nature of the human experience, just using different tools and different approaches.

So if that’s our goal, how do we go about striving for it? The approach that we’ve taken to date is bifurcated. On the one hand, we try to leverage data that exists out there in the published literature. I’ll just note that the majority of that data is derived in a laboratory setting, which is great, because you can control all the variables except the one that you’re studying. But we also try to generate our own data with our own visitors and our own objects and our own galleries to better understand how aesthetic responses and emotional responses manifest in the applied setting. By sort of combining these two streams of knowledge generation, we hope we can make some progress, and I’ll touch on both of these today.

I want to come back to this idea of experience as a product of the brain, especially because if we’re interested in museum experiences, I think it’s important to really define what we mean and then the ways that we go about trying to understand it. We can think of at least some experiences as arising when we are stimulated by entities out in the world. This is stuff out external to yourself, simulating your sensory processing pathways and transmitting information about those stimuli into your brain. Once in the brain, that sensory information interacts with many different regions to give rise to certain mental phenomenon, some of which are cognition, memory and emotion. And other of those neural signals travel down the spinal cord and out into the body, where they modulate motor behaviour and physiology. Just to comment, that physiology can often be used as a reflection of certain aspects of an emotional response, so that’s why we care about that.

The point that I’m getting to here is that each of these outputs of neural function contribute to the sensations that we undergo when we have what we consider to be an experience. And so the question becomes, how do we go about capturing the nature of these outputs as a reflection on the nature of the experience, both in a qualitative way and a quantitative way? The approach that we’ve taken as part of the Neuroscience Initiative is to combine a few different methods. We can use eye tracking, or honestly, up until right now, we’ve been using gaze tracking, and I’ll explain those differences, to monitor viewing behaviour of our visitors. We use biometric measurements to get a read on the physiological responses that are reflective of emotional states. And then we use self-report methods like exit surveys and exit interviews to better understand how visitors consciously perceive the time that they’ve spent in our galleries.

What’s nice about this mixed methods approach is that it maps really well onto the way that we’ve been thinking about and trying to understand this notion of visitor engagement. We have a neuroscience advisory committee, which is composed of research and clinical neuroscientists, and one of them has spent years doing research in the context of the neuromarketing field, so consumer neuroscience, trying to understand how brain function relates and dictates the choices that we make in a consumer setting. He’s also a psychiatrist, which is kind of an interesting combination. Anyway, this member of our community, based on the research that he’s done and the data that he’s gathered over the years, has put forth this definition of engagement that posits that engagement occurs when attention is guided or directed in a way that elicits an emotional response and leads to the formation of a memory.

As you can see, the approach that we’ve taken to assessing the visitor experience in certain contexts at PEM allows us to sort of monitor each of these three elements. Just to articulate that, we can observe how visitors allocate their visual attention using mobile gaze tracking in the gallery. We can get a read on one dimension of the emotional response, which I’ll tell you more about in a second, using biometric measurements. And we can begin to understand at least the beginning of memory formation of gallery experiences by administering self-report measures such as exit surveys.

I just want to go on a little tangent and talk more about this concept of emotion. In the literature, emotion is broken down into these two dimensions of arousal, plotted here on the Y-axis, and valence, plotted on the X-axis. By arousal, what I’m referring to is the intensity of an emotional experience. Think about a time when your heart is beating faster and your palms get sweaty. That’s a high arousal situation. Those heartbeats and sweat production, those are physiological responses, so we can measure arousal by measuring physiology. Valence, oops, I’m sorry, valence on the other hand is this concept of how pleasant the experiences is, is it a good experience or a bad experience, and that you generally have to get by asking people.

As you can see, you can map different emotion states onto these two axes. Just to consider a couple examples, someone who’s calm down here will be lower on the arousal axis but higher on the valence axis, while someone who’s tense will be higher on the arousal axis and lower on the valence axis. There’s literature out there to suggest that it’s actually the level of emotional arousal that has been associated with the probability of remembering a particular experience. Just to put that in another way, the higher your level of arousal, the more likely you are to form a memory, whether or not the experience is good or bad. And so it became important for us to be able to capture this metric of emotional arousal in the visitors that volunteered to participate in our studies.

The way that we’ve been doing that is to measure something called the galvanic skin response, which is a physiological response that you measure peripherally. You can see, you measure it actually in your hands, but it’s regulated by the brain. And importantly, we don’t have any conscious control over it, so you can’t fake it, you can’t think about how you want people to perceive it, it just happens. And basically just to give you a little bit of background on what it is and how it is generated, what you’re doing here is measuring the amount of electrical current that your skin is conducting, which is in turn a function of how much sweat your skin is producing, which is a response that we have to psychological stimuli in certain parts of our body, like the palms of our hands.

Here I’m just zooming out and just giving you a sense of what the data looked like in the raw form. Here we have, this is just a screenshot from the software that I use. This is a frame from the gaze tracking video that is generated by the glasses that we have our participants wear. And then on the bottom, we have the galvanic skin response plotted over time. You can see there are elevations and troughs, and you can track where you are in this curve with this red line as the video precedes, so you can make associations between what the galvanic skin response is and what the person was looking at. All right, this is the approach that we’ve taken in a couple studies over the last year to investigate certain experimental questions and test certain hypotheses.

Before we get to the design and content of our first study, I just want to note that like with any experimental setup, there are caveats inherit to this approach, so I just want to name some of those. I think that’s important to keep in mind. Obviously, the museum is not a completely controlled setting. Our designers and curators are not going to put things where I tell them to put things to control for a study, so I make use of the design that arises in the best way that I can. But it’s not fully controlled, so it’s really hard to establish direct causal relationships between a design decision and the way someone responds to it. But we can still get some sort of information, and I just want to note that there is literature showing that aesthetic responses are context dependent. The way that you respond to the same stimulus presented in a laboratory is going to differ from the way that you respond to it in the context of the museum, and so I think it’s important to take into account both sets of responses.

Another caveat is that the participants are all wired up with this gear, so they know they’re being monitored, and that will impact some people more than others, maybe more at the beginning of the experience than at the end, but it’s there, and so we take that into account. And then just finally, particularly in studies that we’ve conducted to date, though our sample size has been increasing as we do more studies, they are small studies with few participants, relatively. And we’ve really just tested the question once. To make any kind of conclusive statement officially, I would want many, many participants tested in different contexts, different works of art, different galleries, same question. That’s my standard for drawing any kind of conclusive statement. So with those acknowledged, I’ll show you what we’ve done, what we’ve found, and maybe we can talk about what we can do with it.

As with most of our studies, the aim of our first study was to try and understand how we can facilitate visitor engagement with the works of art, in this case in one particular changing exhibition that we had on last spring. In particular, we were focused on the textual prompts, the kind of textual prompts that might appear in the context of an object label. This study was motivated by decades of research that started with the work of Alfred Yarbus, a psychologist back in the 1960s, which basically showed that your goal in looking at an image impacts the way that you move your eyes over that image, so your viewing behaviour, and therefore has the possibility of impacting your perception and the quality of your engagement with that image.

Just briefly, what Yarbus did was to show this participant this painting, or a reproduction of it, and allow them to look at it without any kind of prompt, or after receiving each of seven prompts. I’ve just shown a couple of them here. And what he did was to track their eye movements as they received these different prompts. What you can see is that when the person was freely examining the image, they sort of looked everywhere, but when they were asked to assess the ages of the people depicted in the image, you can see that the eye tracking data now clusters over areas that correspond to the faces of the people depicted in the image, which makes sense if you’re trying to deduce age. And so on and so forth, as the prompts changed, the eye tracking pattern changed.

Some 50 or 60 years later, we’re now at a point where there are common viewing tasks, some of them are listed here, that are used to study this phenomenon. We’re kind of actually at a point where there are algorithms that can take eye tracking data and tell you what the task was that the person was performing that generated that eye tracking data. The question for me became whether these viewing tasks might differentially impact how visitors engage with a work of art presented in the context of a museum gallery. I just want to walk you through one example of how this might work.

Here again, I’m showing you eye tracking data in the form of heat maps, and this is from a published study. In the study, participants were shown a series of images, and they were either allowed to free view the images, to look at them without any predetermined purpose, or they were asked to search for a particular element within the image, so to perform a search task. What they found is that on the first fixation, the first place that people looked, in all the conditions participants demonstrated what’s called a central bias, which is reflected by the fact that this red zone, the area where people looked most frequently or where the most people looked, fell in the centre of the plot here, in the centre of the image.

The first place that they looked, they tended to look at the centre, but over time, the researchers found that there emerged a difference between those who were free viewing and those who were searching. Specifically, those who are free viewing maintain that central bias. You can see here, later in visual exploration, it’s a little bit looser, but that red cloud is still centred in the middle of the plot, reflecting that their visual attention was still focused on the middle of the image. But those who were searching showed a different pattern in which they lost that central bias really early on. Here, this is the third place that people looked in the image, and you can see that that red cloud is now gone from the centre and is instead distributed over the rest of the plot more evenly. What this says to me is that people literally take in different content within the same image, depending on what their purpose in looking at it is.

Again, the question becomes whether providing certain kinds of viewing tasks in an object label or any other sort of interpretive device might be able to impact the way that visitors interact or engage with a work of art. That was the question that we addressed in our first study, and in parallel to thinking about that, I just reviewed the labels for the exhibition that we were going to do the study in, and what I began to notice is that we’re already in the habit or in the practise of using these kinds of viewing prompts. I’ll just give you a few examples.

I won’t read you the details, but here’s an example of what I considered to be a search task. It makes note of a compositional element in the associated work of art and implicitly or explicitly asks you to go find it. We also have what are called aesthetic judgement  tasks. I encountered these less frequently in this exhibition, but we did have these questions posted throughout the exhibition that asked the visitors to relate themselves to the work of art that they were looking at, to make a judgement  about it. And these tasks, I just want to contrast to other kinds of label content, which give you information. In this case, this yellow boxed section is conveying historical context about the painting. I’ll refer to this as free viewing in our study, and that served as a control for having to read some kind of information, but not perform a task.

We used those three kinds of viewing prompts in the context of an exhibition of work by the Native American artist T.C. Cannon last year, to ask this question of whether giving visitors, or study participants in particular, different kinds of viewing prompts would result in or would be associated with different kinds of engagement in the ways that we were measuring it. Our hypothesis had two parts. The first one was that the viewing prompts that consisted of a task would be effective at helping visitors to engage. And more specifically that those aesthetic judgement  tasks would be the most effective at helping visitors to engage. The reason that we thought this has to do with what’s known about which brain areas become activated and suppressed when we experience what we can idiosyncratically consider to be an emotionally moving aesthetic experience.

Again here, I’m showing you data from a published study by Ed Vessel. In this study, people were again shown a series of images, but they were asked to rate each image on a scale from one to four, according to how emotionally moving they found that image to be. A rating of one was not moving. A rating a four was very moving. They were shown these images while their brains were scanned using fMRI imaging to identify which regions were activated, indicated here in yellow and orange, and which regions were suppressed, indicated in blue and green.

I just want to draw your attention to this half of the figure. Each of these is a slice of a brain, well, an image slice of a brain. The front is up over here on the right, the back is over here on the left. These are the four rating conditions. What they found is that when participants were looking at images that they rated with a one, two, or three, so the relatively not that emotionally moving images, they saw suppression in these regions here, which correspond to a circuit in the brain called the default mode network. However, when the people were looking at images that they had rated with a four, that they found to be very emotionally moving or the most emotionally moving, that suppression was alleviated. You can see that the blue cloud is gone here, relative to here, and so that indicates a relative level of activation of this network.

So what do we know about the default mode network? To put it simply, it tends to be suppressed when our attention is focused externally and relatively activated when our attention is focused internally, when we’re engaged in introspection or self-referential thinking. What the conclusion of this paper was, or what they posited, is that activation of this network might facilitate self-referential thinking, which might be a critical component of having a moving aesthetic experience. In turn, we tried to sort of prompt this kind of thinking with our judgement  prompts, in the hopes that it would lead to a moving experience.

We’re going to speed through the data. I’m really sorry, but I’d be happy to talk more about it later if anything’s unclear. This is a floor map of the exhibition. It had three galleries, and people came in here and moved from right to left. We studied both responses in each gallery as a whole, as well as at nine specific works of art distributed across the galleries. These are representations of the works, and at each work we had one group of participants who did not receive any extra content, no viewing prompts aside from what was in the exhibition, but the other three groups received some combination of a free viewing prompt in the form of a historical fact, a search prompt asking them to look for a compositional element, or a judgement  prompt asking them to relate themselves to what they were looking at.

I just want to comment briefly that again, the participant cohort was small, we had 14 participants, and it was pretty homogeneous and not representative of any diverse array of people. But I think it did sort of reflect, if you want to stereotype, the majority of people who come to our museum, this might be a representation, maybe. Anyway, they were all women. Most of them were white. Most of them had been to PEM before, but importantly, no one had been in this exhibition. And then we did have a range of ages in our participants, ranging from mid-twenties all the way up to the mid-seventies.

The two questions that we’ll address with the data concern whether and how the prompts impacted the speed at which participants moved through each of the three galleries and also the nature of the response that they generated at each of those nine individual works of art. Other visitor studies have shown repeatedly that dwell time at individual objects, and to some extent attention, tends to decline over the course of an exhibition, such that we spend more time maybe with the first object that we encounter relative to the last, and so we wanted to just see whether this trend was recapitulated in our cohort. What we did was to calculate a metric called the sweep rate index for each of the three galleries. What that is, is basically a transit speed with the units of feet per minute. You take the area of the gallery, divide it by the average amount of time that people spend there, and you get your metric.

Here I’m planning the sweep rate index, feet per minute, as a function of which gallery the participant was in. Bars falling higher up indicate faster movement. Bars falling lower down indicate slower movement. You can see that in galleries one and two, on average participants moved at about 60 feet per minute, and that that rate almost doubled in the last gallery. I think in thinking about why that might be, there are many possible explanations. Fatigue comes to mind, of course, but I think the way that that gallery was designed might also have contributed to this effect, especially since if you look at the pattern here, it’s not an incremental increase, it’s a jump.

So what’s going on there? From consumer neuroscience studies, we derive this effect called the exit effect, which is basically when they study how shoppers move around through like a grocery store, it’s been observed that as soon as the exit to the store comes into view of the shopper, they make a beeline for it, essentially. And so if we look at how this gallery is designed, the visitor is coming through a doorway here, they experience a painting here, they might turn to look at the one here, but they can’t see the exit. They don’t know what lies beyond this wall. As soon as they come around the wall, though, they have direct line of sight to the door. I’ll just comment that this gallery was filled with many large scale, brightly coloured, in my opinion really visually captivating works of art. It was a multi-sensory experience. There was music playing. There was a video playing.

Oops, keep doing that. This is a mural, in front of which there’s a bench with sketchbooks and pencils, so you can sit and linger and sketch and try to engage in all these different ways. And I’m sure that people did, but they moved faster than they had in the first two galleries. If we go back to our hypothesis, though, and look at how the prompted participants in blue moved relative to the unprompted participants in red, we see that the unprompted participants consistently were moving faster in each gallery than the prompted participants. Of course, the obvious difference here is that the prompted participants had to read something, the unprompted participants didn’t. If we roughly calculate, using an average reading rate, how long it should have taken to read those prompts, that amount of time accounts for 25% of the observed difference in dwell time. So what’s going on in the other 75% of that time difference? We don’t know, but it’s kind of interesting.

This might be a proposed sort of takeaway from this data, that our participants tended to move more quickly in the last gallery of the exhibition and that viewing prompts were associated with slower movement. But the crux of our study was really about how these prompts impacted or were associated with responses to individual works of art. And again, there were nine of them distributed throughout the exhibition. The first thing that we looked at was dwell time. Here, this is the average dwell time in minutes that people spent engaged with a particular object, and I’ll just qualify that that was the time they spent looking the work of art itself, not the label.

And then I’m plotting this as a function of what kind of prompt they received. So here’s the no prompt group on the left, the free viewing group, the historical fact group, next, search prompt group and the judgement  group. What you can see is that the people who got tasks, who were searching or judging, spent about twice as long at an individual work of art relative to participants who didn’t get a prompt or who got a free viewing prompt. What this tells me is that any kind of textual prompt is not associated with this prolonged time spent at a work of art, but the presence of a task in the prompt is associated with that lingering.

But then we have to ask the question, what’s going on for the visitor as they’re spending that longer time in front of the work of art? So we also calculated or measured the galvanic skin response that was demonstrated at that work of art as a function of prompt type, and again we see that the search task and the judgement  task groups demonstrated elevated GSR, or galvanic skin response, levels of arousal relative to the no prompt group or the free viewing group. And I’ll just note that if you compare the judgement  group to the no prompt group, there was statistical significance in that difference. Again, we see that the tasks are associated with a form of lingering and perhaps an increased level of emotional arousal at a given work of art.

I just want to take a minute to tell you about how people perceived their level of arousal and valence, those two dimensions of emotion, when we asked them in the exit survey. Here is the self-reported level of arousal, we used different language so that they would understand what we were talking about, as a function of prompt type. Here you see exactly the opposite of what we just saw from the GSR. Here, those who were unprompted perceived themselves as being more emotionally aroused or having a more intense experience than those who were prompted.

And to get to the point, I think there could be a couple explanations for this, but what I think is going on is that we had intended to deliver the prompts with a location-aware app so that they would just pop up when you stepped up to the appropriate work of art. That didn’t work for technical reasons, so what we resorted to was paper packets with a picture of a work of art and a prompt underneath. Think about what that means. You have to step up to the work of art, look through the packet. Is it in the packet? Yes, it is. Read the prompt. It’s kind of a mood killer. So I think our delivery method was a bit flawed here, and that’s reflected in this trend.

However, I was relieved, really, to see that when they reported their valence, how pleasant the experience was, it was the same across the prompt groups. So on the one hand you can say, well, the prompts didn’t increase their level of enjoyment, but it also didn’t decrease their level of enjoyment. And I’ll just remind you that it’s actually the level of arousal that is correlated with memory formation, which is what we were going for.

I’m about to end, but I’ll just comment that in my view, these kinds of studies, we have three data sets now, raise more questions than they actually answer, so we might be able to draw some tentative conclusions. Giving visitors a viewing task could be used as a tactic to encourage slow looking and facilitate emotional engagement, especially those judgement  prompts, but then we’re left with these questions of when do you use them? Where do you use them? What form do they take? How do you use them to inform the emotional cadence of an exhibition? Should they be big? Should they be in a small object label? We don’t know. We don’t know.

We also saw, as I mentioned, that visitors moved the fastest in the last gallery, so what do we do about that? Do we cater to it and put content in there that we think perhaps can be appreciated more quickly, or do we try and challenge that behavioural trend and create a grand experience to try and slow people down? We haven’t really begun to experiment with how to use this data, and I hope that that’s in the near future of this work. But also just to give you a peek at what’s coming for us, I’ve been spending a lot of time thinking about how we can leverage insights from neuroscience to help our visitors understand and make meaning of the experiences that they have with that work of art. I’ll just introduce the phrase here. How can we help to cultivate visual literacy in our visitor population?

I’ll just leave you with one more thought, which is, if I look across the data sets that we have, the theme that emerges for me is that design and interpretive elements that facilitate interpersonal engagement of various sorts seem to result in higher levels of engagement in the ways that we’re measuring it. That could be opportunities for introspection, it could be design alternative elements that tell you something about the artist and make you feel like you’re more connected to them, or it could be the content and the nature of the work of art itself. And that has to do with the way that we respond to the human form and how we study that. I’m happy to share that data if you want to come find me, but with that, I’ll thank our study participants and everyone who is involved in this work, and I’d be happy to take questions if we have time. Okay, thank you.

Speaker 2:        Thank you, Tedi. Thank you.